OpenTelemetry · Jaeger · Grafana Tempo — namespace: nutripaw-prod
generate_nutrition_plan acumula el 73% del tiempo total de la request. Causa: llamada bloqueante a modelo ML sin caché + N+1 queries a PostgreSQL (se detectaron 47 SELECT individuales por llamada).
Grafo generado automáticamente por Jaeger a partir de las trazas. Las líneas rojas indican el critical path.
| Prioridad | Servicio | Fix | Impacto estimado |
|---|---|---|---|
| P0 | nutrition-engine | Batch SQL (47 → 1 query), async ML | −6.8s |
| P0 | nutrition-engine | Cache resultados ML (Redis TTL 1h) | −5.2s (hit) |
| P1 | pet-profile | Aumentar TTL Redis: 30s → 24h | −630ms |
| P1 | postgres | Índice en pets.owner_id + EXPLAIN | −200ms |
| P2 | api-gateway | Connection pooling (asyncpg) | −28ms |
from opentelemetry import trace from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import SERVICE_NAME, Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased import asyncio # ── Inicializar TracerProvider con 5% sampling (prod) ── resource = Resource(attributes={SERVICE_NAME: "nutrition-engine"}) sampler = ParentBased(root=TraceIdRatioBased(0.05)) # 5% prod provider = TracerProvider(resource=resource, sampler=sampler) exporter = OTLPSpanExporter(endpoint="http://tempo:4317") # → Tempo/Jaeger provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) tracer = trace.get_tracer("nutrition-engine") async def generate_nutrition_plan(pet_id: int, owner_id: int) -> dict: with tracer.start_as_current_span("generate_nutrition_plan") as span: # Tags semánticos para filtrado en Jaeger UI span.set_attribute("pet.id", pet_id) span.set_attribute("owner.id", owner_id) span.set_attribute("model.version", "nutripaw-ml-v2.3") # FIX P0: batch query en lugar de N+1 with tracer.start_as_current_span("db.fetch_pet_data_batch") as db_span: db_span.set_attribute("db.system", "postgresql") db_span.set_attribute("db.statement", "SELECT * FROM pets JOIN nutrition_history ... WHERE pet_id=$1") pet_data = await db_fetch_batch(pet_id) # 1 query, no 47 # FIX P0: ML async, no bloqueante with tracer.start_as_current_span("ml.infer_nutrition_plan") as ml_span: ml_span.set_attribute("ml.framework", "torch") ml_span.set_attribute("ml.cached", "false") plan = await asyncio.to_thread(ml_model_infer, pet_data) ml_span.set_attribute("ml.tokens_used", plan["tokens"]) span.add_event("plan_generated", {"calories": plan["kcal"]}) return plan
version: "3.8" services: jaeger: image: jaegertracing/all-in-one:1.62 ports: ["16686:16686", "4317:4317"] environment: COLLECTOR_OTLP_ENABLED: "true" tempo: image: grafana/tempo:2.7 ports: ["3200:3200", "4317:4317"] volumes: ["./tempo.yaml:/etc/tempo/tempo.yaml"] command: "-config.file=/etc/tempo/tempo.yaml" otel-collector: image: otel/opentelemetry-collector-contrib:0.113 volumes: ["./otel-config.yaml:/etc/otel/config.yaml"] command: "--config=/etc/otel/config.yaml" depends_on: [jaeger, tempo]
apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: nutripaw-jaeger namespace: nutripaw-prod spec: strategy: production sampling: options: default_strategy: type: probabilistic param: 0.05 # 5% — restr. costes S3 service_strategies: - service: nutrition-engine type: probabilistic param: 1.0 # 100% — debug activo storage: type: elasticsearch options: es.server-urls: http://elasticsearch:9200
Instalar el operador v1.51 con la configuración de sampling: 5% global, 100% para nutrition-engine durante la fase de debug. Conectar Elasticsearch para persistencia de trazas 30 días.
Añadir opentelemetry-sdk, opentelemetry-exporter-otlp y el código de tracing.py. Envolver las funciones críticas con spans: generate_nutrition_plan, db_fetch_batch, ml_model_infer.
Inyectar cabeceras W3C TraceContext (traceparent) en todas las llamadas entre api-gateway → pet-profile → nutrition-engine → notification-service usando los propagadores de OTel.
Tempo escribe a bucket GCS nutripaw-traces con retención de 90 días. Jaeger envía trazas a Tempo via OTLP gRPC. Grafana 10 datasource apunta a Tempo para correlación con métricas Prometheus.
Configurar alerta en Grafana: si el span generate_nutrition_plan supera 500ms en P95 durante 5 minutos → PagerDuty. Segunda alerta para error rate en trazas > 1%.