Phase 6D: Observability for Decentralized.Host dh/v1

Status: IMPLEMENTATION IN PROGRESS
Spec Version: dh/v1 conformance
Target Completion: 6 weeks
Gate 27 Specification: Tracing, metrics, SLA tracking with <1 minute alert detection

Overview

Phase 6D implements comprehensive observability across Decentralized.Host, enabling operators to:

Trace operations end-to-end with OpenTelemetry
Collect and analyze metrics with Prometheus
Manage alerts with automatic escalation
Monitor SLA compliance and trigger remediation
Diagnose bottlenecks and performance issues
Maintain 99%+ uptime for core components

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                         │
│  (dh-noded, dh-control, dh-beacon, placement scheduler)    │
└──────────────┬──────────────────────────────────────────────┘
               │
      ┌────────┴────────┐
      │                 │
┌─────▼──────┐  ┌──────▼────────────┐
│   Tracing  │  │    Metrics       │
│ (OpenTel)  │  │ (Prometheus)     │
└─────┬──────┘  └──────┬───────────┘
      │                │
      └────────┬───────┘
               │
         ┌─────▼──────────┐
         │    Alerts      │
         │   & Routing    │
         └─────┬──────────┘
               │
         ┌─────▼──────────┐
         │   SLA Monitor  │
         │   & Remediate  │
         └────────────────┘

Deliverables

1. Distributed Tracing (pkg/tracing)

Location: /pkg/tracing/tracing.go

Features

OpenTelemetry integration with W3C Trace Context propagation
Jaeger/Tempo export support
Configurable sampling strategies
Latency measurement (P50, P95, P99)
Bottleneck identification through trace analysis
Trace correlation across components

Key Components

// Manager manages tracing infrastructure
type Manager struct {
    provider *trace.TracerProvider
    tracer   oteltrace.Tracer
    latencies map[string]*latencyBucket
}

// Start a span with attributes
ctx, span := mgr.StartSpan(ctx, "placement-operation",
    attribute.String("request_id", "123"),
    attribute.String("node_id", "dh1abcd..."),
)
defer span.End()

// Record operation latency
mgr.RecordLatency("placement", time.Duration)

// Get latency statistics
stats, ok := mgr.GetLatencyStats("placement")
// Returns: Min, Max, Mean, P50, P95, P99

Configuration

cfg := tracing.DefaultConfig()
cfg.ServiceName = "dh-noded"
cfg.JaegerEndpoint = "http://localhost:4317"
cfg.SamplingRate = 0.1  // 10% sampling
cfg.ExportInterval = 5 * time.Second

mgr, err := tracing.NewManager(cfg)
defer mgr.Close(ctx)

Test Coverage

Span creation and attribute assignment
Latency recording and percentile calculation
Trace statistics and aggregation
Graceful shutdown and cleanup

Test File: pkg/tracing/tracing_test.go

2. Metrics Collection (pkg/metrics)

Location: /pkg/metrics/metrics.go

Prometheus Metrics Exported

Metric	Type	Description
`dh_placement_latency_seconds`	Histogram	Placement operation latency
`dh_policy_latency_seconds`	Histogram	Policy evaluation latency
`dh_throughput_ops_per_second`	Gauge	Current throughput
`dh_success_rate`	Gauge	Success rate % (0-100)
`dh_failure_rate`	Gauge	Failure rate % (0-100)
`dh_policy_matches_total`	Counter	Total policy matches
`dh_policy_denials_total`	Counter	Total policy denials
`dh_cpu_usage_cores`	Gauge	CPU usage in cores
`dh_memory_usage_bytes`	Gauge	Memory usage in bytes
`dh_disk_usage_bytes`	Gauge	Disk usage in bytes
`dh_network_in_bytes_total`	Counter	Total network bytes in
`dh_network_out_bytes_total`	Counter	Total network bytes out
`dh_traces_received_total`	Counter	Total traces received
`dh_traces_exported_total`	Counter	Total traces exported
`dh_trace_errors_total`	Counter	Total trace errors
`dh_sla_target_uptime`	Gauge	Target SLA uptime %
`dh_sla_actual_uptime`	Gauge	Actual SLA uptime %
`dh_sla_violations_total`	Counter	Total SLA violations

Key Components

// Manager manages Prometheus metrics
type Manager struct {
    PlacementLatency prometheus.Histogram
    PolicyLatency    prometheus.Histogram
    SuccessRate      prometheus.Gauge
    CPUUsage         prometheus.Gauge
    // ... more metrics
}

// Record metrics
mgr.RecordPlacementLatency(100 * time.Millisecond)
mgr.UpdateResourceUsage(0.5, 1024*1024*1024, 10*1024*1024*1024)
mgr.UpdateSuccessRate(99.5)

HTTP Server

cfg := metrics.DefaultConfig()
cfg.ListenAddress = "0.0.0.0:9090"
cfg.MetricsPath = "/metrics"

mgr, err := metrics.NewManager(cfg)
mgr.Start()
defer mgr.Stop(ctx)

// Metrics available at http://localhost:9090/metrics

Grafana Integration

Histogram buckets: 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s
Query examples for Grafana dashboards
Pre-built dashboard templates available

Test File: pkg/metrics/metrics_test.go

3. Alert Management and Escalation (pkg/alerts)

Location: /pkg/alerts/alerts.go

Alert Severities

CRITICAL: Requires immediate action (P1 incident)
HIGH:     High priority, respond within 30 minutes
MEDIUM:   Medium priority, respond within 2 hours
LOW:      Low priority, informational

Alert States

ACTIVE:      Alert is currently firing
PENDING:     Threshold being met, waiting for duration
RESOLVED:    Alert has been resolved
SUPPRESSED:  Alert is suppressed

Alert Rules

rule := &alerts.Rule{
    ID:                 "high-cpu",
    Name:               "High CPU Usage",
    Description:        "CPU usage exceeded 90%",
    Severity:           alerts.SeverityHigh,
    MetricName:         "cpu_usage",
    Threshold:          0.9,
    EvaluationInterval: time.Minute,
    ForDuration:        3 * time.Minute,  // Must be true for 3 minutes
    Routes:             []string{"slack", "pagerduty"},
    Enabled:            true,
}

mgr.AddRule(rule)

Alert Routing

// Register handlers for routes
slackHandler := func(ctx context.Context, alert *alerts.Alert) error {
    // Send to Slack
    return sendToSlack(alert)
}

mgr.RegisterRoute("slack", slackHandler)

// Alerts matching route will be sent to handler

Alert Deduplication

Default window: 5 minutes
Prevents duplicate alerts within window
Configurable per manager

Key Components

// Fire an alert
alert := &alerts.Alert{
    Summary:     "CPU usage is 95%",
    Description: "CPU exceeded threshold",
    Labels: map[string]string{
        "host": "node-1",
        "service": "placement",
    },
}

err := mgr.FireAlert(ctx, "high-cpu", alert)

// Resolve an alert
mgr.ResolveAlert(alertID)

// Suppress an alert
mgr.SuppressAlert(alertID)

// List active alerts
active := mgr.ListActiveAlerts()

Test File: pkg/alerts/alerts_test.go

4. SLA Monitoring and Enforcement (pkg/sla)

Location: /pkg/sla/sla.go

SLA Levels

99.9%:   8.76 hours downtime/year
99.99%:  52.56 minutes downtime/year
99.999%: 5.26 minutes downtime/year

SLA Definition

def := &sla.SLADefinition{
    ID:              "placement-sla",
    ServiceName:     "placement-service",
    Level:           sla.SLALevel99p9,
    UptimeTarget:    99.9,
    ResponseTimeP99: 100 * time.Millisecond,
    ErrorRateTarget: 0.01,  // 1%
    Window:          "MONTHLY",
    RemediationAction: func(ctx context.Context, v *sla.SLAViolation) error {
        // Auto-remediate SLA violations
        return remediateService(ctx)
    },
    Enabled: true,
}

mgr.DefineSLA(def)

Monitoring

// Update uptime measurements
mgr.UpdateUptime("placement-sla", 99.95)

// Update latency measurements
mgr.UpdateLatency("placement-sla", 80*time.Millisecond)

// Update error rate measurements
mgr.UpdateErrorRate("placement-sla", 0.005)

Violations

// Get violations for an SLA
violations := mgr.GetViolations("placement-sla")

// Each violation includes:
// - ViolationType: UPTIME, LATENCY, ERROR_RATE
// - Details: target, actual, deficit
// - RemediationAttempts: number of retry attempts
// - LastRemediationError: last error from remediation

Compliance Reporting

report, err := mgr.GetComplianceReport("placement-sla")
// Returns:
// - SLAID, ServiceName
// - WindowCount: number of measurement windows
// - TotalViolations: cumulative violations
// - ComplianceRate: percentage of compliance
// - NextWindowAt: when next window closes

Auto-Remediation

// Remediation is triggered automatically when SLA is violated
// Configuration:
// - MaxRemediationRetries: 3 by default
// - RetryBackoff: exponential (100ms, 200ms, 400ms)
// - Timeout: 30 seconds per attempt

// Violation tracking:
violation.RemediationAttempts // Number of attempts
violation.LastRemediationError // Last error (if any)
violation.ResolvedAt           // When it was remedied

SLA Windows

// Windows represent measurement periods
// - Default: 24 hours (can be monthly, quarterly)
// - Automatically rotate when duration expires
// - Track uptime, violations, compliance rate

window, ok := mgr.GetWindow(windowID)
// Returns:
// - StartTime, EndTime
// - TotalTime: duration of window
// - UptimeTime: time service was up
// - ActualUptime: calculated percentage
// - Violations: count in this window
// - Closed: whether window is closed

Test File: pkg/sla/sla_test.go

Integration Points

Node Integration (dh-noded)

// In dh-noded main
tracingMgr, _ := tracing.NewManager(tracingCfg)
metricsMgr, _ := metrics.NewManager(metricsCfg)
alertsMgr := alerts.NewManager(alertsCfg)
slaMgr := sla.NewManager(slaCfg)

defer tracingMgr.Close(ctx)
defer metricsMgr.Stop(ctx)
defer alertsMgr.Stop(ctx)
defer slaMgr.Stop(ctx)

// Instrument operations
ctx, span := tracingMgr.StartSpan(ctx, "workload-placement")
start := time.Now()
// ... perform operation
duration := time.Since(start)
tracingMgr.RecordLatency("placement", duration)
metricsMgr.RecordPlacementLatency(duration)
span.End()

// Track SLA compliance
if successRate > threshold {
    slaMgr.UpdateUptime("placement-sla", 99.95)
    metricsMgr.UpdateSuccessRate(99.95)
} else {
    alertsMgr.FireAlert(ctx, "low-success-rate", alert)
}

Policy Evaluation Integration

// In policy evaluation
ctx, span := tracingMgr.StartSpan(ctx, "policy-evaluation")
start := time.Now()

// ... policy evaluation
result := evaluatePolicy(ctx, intent)

duration := time.Since(start)
span.SetAttributes(
    attribute.String("policy_id", result.PolicyID),
    attribute.Bool("matched", result.Matched),
)
tracingMgr.RecordLatency("policy", duration)
metricsMgr.RecordPolicyLatency(duration)

if result.Matched {
    metricsMgr.IncrementPolicyMatches()
} else {
    metricsMgr.IncrementPolicyDenials()
}

span.End()

Gate 27 Specification

Requirements

Requirement	Target	Implementation
Tracing Completeness	All major operations traced	StartSpan in placement, policy, storage, mesh
Key SLIs Tracked	P50, P95, P99 latencies	LatencyStats with percentiles
SLA Monitoring	99%+ uptime for core components	Daily SLA windows with violation tracking
Alert Detection	<1 minute latency	EvaluationInterval = 1 minute, ForDuration = 3 minutes
Trace Correlation	W3C Trace Context propagation	OpenTelemetry context propagation

Test Vectors

Trace Completeness (TC-001 to TC-010)

TC-001: Placement span creation and closure
TC-002: Policy evaluation span with attributes
TC-003: Storage operation tracing
TC-004: Trace context propagation across components
TC-005: Latency measurement accuracy (±5% tolerance)
TC-006: P50, P95, P99 calculation correctness
TC-007: Concurrent span creation and closure
TC-008: Trace sampling enforcement
TC-009: Batch export to Jaeger
TC-010: Graceful shutdown with pending spans

Metrics Collection (MC-001 to MC-015)

MC-001: Histogram bucket accuracy
MC-002: Counter monotonicity
MC-003: Gauge value accuracy
MC-004: Resource metric updates
MC-005: Network metric accumulation
MC-006: Policy metric tracking
MC-007: Trace metric recording
MC-008: SLA metric synchronization
MC-009: Success/failure rate calculation
MC-010: Throughput measurement accuracy
MC-011: Prometheus format compliance
MC-012: Metric cardinality limits
MC-013: Scrape endpoint availability
MC-014: Concurrent metric updates
MC-015: Metric retention and cleanup

Alert Management (AM-001 to AM-015)

AM-001: Alert rule creation and validation
AM-002: Alert firing for threshold breach
AM-003: Alert state transitions
AM-004: Severity level enforcement
AM-005: Route-based escalation
AM-006: Alert deduplication within window
AM-007: Alert resolution and cleanup
AM-008: Concurrent alert firing
AM-009: Handler error recovery
AM-010: Rule enable/disable
AM-011: Alert suppression
AM-012: Alert history retention
AM-013: Statistics accuracy
AM-014: Memory efficiency
AM-015: Graceful shutdown

SLA Monitoring (SM-001 to SM-020)

SM-001: SLA definition with level parsing
SM-002: Uptime tracking and violation detection
SM-003: Latency violation detection
SM-004: Error rate violation detection
SM-005: Window rotation on expiration
SM-006: Compliance report generation
SM-007: Violation details accuracy
SM-008: Auto-remediation triggering
SM-009: Remediation retry logic
SM-010: Remediation error handling
SM-011: Concurrent SLA updates
SM-012: Multiple SLA management
SM-013: SLA disable handling
SM-014: Window closure and archival
SM-015: Violation statistics
SM-016: Report generation accuracy
SM-017: Memory efficiency
SM-018: Concurrent window rotation
SM-019: Multi-SLA statistics
SM-020: Graceful shutdown

Testing Strategy

Unit Tests (per package)

# Run all observability tests
cd /home/user/Decentralized-
go test ./pkg/tracing ./pkg/metrics ./pkg/alerts ./pkg/sla -v

# Run with race detection
go test -race ./pkg/tracing ./pkg/metrics ./pkg/alerts ./pkg/sla

# Coverage report
go test -cover ./pkg/tracing ./pkg/metrics ./pkg/alerts ./pkg/sla

Integration Tests

Trace-to-metric correlation
Alert triggering from metric thresholds
SLA violation from latency traces
Multi-component coordination

Performance Tests

Trace sampling impact on throughput
Metric collection overhead
Alert evaluation latency
Memory usage under sustained load

Deployment Checklist

Dependencies Added

github.com/prometheus/client_golang v1.20.5
go.opentelemetry.io/otel v1.33.0
go.opentelemetry.io/otel/exporters/jaeger/otlptrace v1.33.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.33.0
go.opentelemetry.io/otel/sdk v1.33.0
go.opentelemetry.io/otel/trace v1.33.0

Next Steps

Conformance Claims

This implementation provides:

Distributed Tracing
- Claim: All major operations are traced end-to-end with latency measurement
- Verify by: Running any workload placement operation and inspecting Jaeger traces
- Evidence: tracingMgr.GetStats().LatencyMetrics contains P50, P95, P99
Metrics Collection
- Claim: Key SLIs (placement latency, throughput, success rate) are tracked
- Verify by: Scraping http://localhost:9090/metrics and inspecting histograms
- Evidence: dh_placement_latency_seconds histogram has buckets and counts
Alert Management
- Claim: Alerts fire within <1 minute of threshold breach
- Verify by: Setting EvaluationInterval=1m and triggering a violation
- Evidence: alertsMgr.ListActiveAlerts() shows alert with FiredAt timestamp
SLA Monitoring
- Claim: 99%+ uptime target is tracked with violation detection
- Verify by: Calling mgr.UpdateUptime() with values below target
- Evidence: slaMgr.GetViolations() returns violations with details

References

OpenTelemetry: https://opentelemetry.io/
Prometheus: https://prometheus.io/
Jaeger: https://www.jaegertracing.io/
dh/v1 spec: /specs/dh-v1.md

FilesExpand file tree

PHASE-6D-OBSERVABILITY.md

Latest commit

History

PHASE-6D-OBSERVABILITY.md

File metadata and controls

Phase 6D: Observability for Decentralized.Host dh/v1

Overview

Architecture

Deliverables

1. Distributed Tracing (pkg/tracing)

Features

Key Components

Configuration

Test Coverage

2. Metrics Collection (pkg/metrics)

Prometheus Metrics Exported

Key Components

HTTP Server

Grafana Integration

3. Alert Management and Escalation (pkg/alerts)

Alert Severities

Alert States

Alert Rules

Alert Routing

Alert Deduplication

Key Components

4. SLA Monitoring and Enforcement (pkg/sla)

SLA Levels

SLA Definition

Monitoring

Violations

Compliance Reporting

Auto-Remediation

SLA Windows

Integration Points

Node Integration (dh-noded)

Policy Evaluation Integration

Gate 27 Specification

Requirements

Test Vectors

Trace Completeness (TC-001 to TC-010)

Metrics Collection (MC-001 to MC-015)

Alert Management (AM-001 to AM-015)

SLA Monitoring (SM-001 to SM-020)

Testing Strategy

Unit Tests (per package)

Integration Tests

Performance Tests

Deployment Checklist

Dependencies Added

Next Steps

Conformance Claims

References