CULTIVA IA IA-Ingenieria-MLOps configuracion-prometheus

NutriFlow โ€” Configuracion Prometheus

Stack de monitorizacion completo para SaaS B2B healthcare ยท Kubernetes + 3 microservicios + PostgreSQL + Redis
7 targets configurados
8 reglas de grabacion
9 alertas (3 criticas, 6 warning)
Retencion 30d ยท HA ready
๐Ÿ“

Arquitectura de Monitorizacion

Flujo de metricas desde los targets hasta Grafana y AlertManager
๐ŸŒ
api-gateway
:8080/metrics
๐Ÿง 
nutrition-engine
:8081/metrics
๐Ÿ“„
report-service
:8082/metrics
๐Ÿ–ฅ
node-exporter
x3 :9100
๐Ÿ˜
postgres-exp
:9187
โšก
redis-exp
:9121
โ€”โ€”โ€” scrape /metrics cada 15s โ€”โ€”โ€” โ†“
๐Ÿ”ฅ
Prometheus Server
:9090 ยท TSDB 30d ยท GKE
โ†“
๐Ÿ“Š
Grafana
:3000
๐Ÿ””
AlertManager
:9093 โ†’ Slack + PagerDuty
๐Ÿ—„
Thanos
Long-term storage
Scrape Interval
15s
Global ยท ML service 10s
Retencion TSDB
30d
50 GiB storage volume
Targets activos
7
3 nodos + 4 servicios
Recording Rules
8
CPU ยท Mem ยท HTTP ยท ML
Alert Rules
9
3 critical ยท 6 warning
๐ŸŽฏ

Scrape Targets

7 endpoints monitorizados con estado actual simulado
Job Endpoint Intervalo Labels Estado
prometheus localhost:9090 15s self-monitor UP
node-exporter node1:9100, node2:9100, node3:9100 15s env=prod cluster=gke-nutriflow UP x3
api-gateway api-gw.nutriflow.svc:8080 15s service=api-gateway tier=edge UP
nutrition-engine nutrition-engine.svc:8081 10s service=ml-engine tier=inference UP
report-service report-svc.nutriflow.svc:8082 15s service=reports tier=backend UP
postgres-exporter postgres-exporter.svc:9187 30s db=postgresql UP
redis-exporter redis-exporter.svc:9121 15s db=redis UP
๐Ÿ“„

prometheus.yml

Configuracion principal โ€” scrape jobs, service discovery, alertmanager
/etc/prometheus/prometheus.yml
# NutriFlow โ€” Prometheus Configuration
# Cluster: GKE production ยท Region: eu-west1
# Generado por CULTIVA IA ยท configuracion-prometheus skill

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  scrape_timeout:      10s
  external_labels:
    cluster:     "gke-nutriflow-prod"
    region:      "eu-west1"
    environment: "production"
    team:        "mlops"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager.monitoring.svc:9093"
      timeout: 10s
      api_version: v2

rule_files:
  - "/etc/prometheus/rules/recording_rules.yml"
  - "/etc/prometheus/rules/alert_rules.yml"

scrape_configs:
  # --- Prometheus self-monitoring ---
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # --- Node exporters (3 nodos GKE) ---
  - job_name: "node-exporter"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "gke-node-1.nutriflow.internal:9100"
          - "gke-node-2.nutriflow.internal:9100"
          - "gke-node-3.nutriflow.internal:9100"
        labels:
          env: "production"
          cluster: "gke-nutriflow-prod"
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

  # --- API Gateway (edge service) ---
  - job_name: "api-gateway"
    scrape_interval: 15s
    metrics_path: "/metrics"
    static_configs:
      - targets: ["api-gw.nutriflow.svc.cluster.local:8080"]
        labels:
          service: "api-gateway"
          tier:    "edge"
          slo:     "99.9"

  # --- Nutrition Engine (ML inference) โ€” intervalo reducido ---
  - job_name: "nutrition-engine"
    scrape_interval: 10s  # Mas frecuente para inferencia critica
    metrics_path: "/metrics"
    static_configs:
      - targets: ["nutrition-engine.nutriflow.svc.cluster.local:8081"]
        labels:
          service: "nutrition-engine"
          tier:    "inference"
          model:   "nutrinet-v2"

  # --- Report Service ---
  - job_name: "report-service"
    scrape_interval: 15s
    static_configs:
      - targets: ["report-svc.nutriflow.svc.cluster.local:8082"]
        labels:
          service: "report-service"
          tier:    "backend"

  # --- PostgreSQL Exporter ---
  - job_name: "postgres-exporter"
    scrape_interval: 30s
    static_configs:
      - targets: ["postgres-exporter.nutriflow.svc.cluster.local:9187"]
        labels:
          db:      "postgresql"
          service: "nutriflow-db"

  # --- Redis Exporter ---
  - job_name: "redis-exporter"
    scrape_interval: 15s
    static_configs:
      - targets: ["redis-exporter.nutriflow.svc.cluster.local:9121"]
        labels:
          db:      "redis"
          service: "session-cache"
โšก

Recording Rules

8 metricas pre-computadas para queries costosas en dashboards y alertas
/etc/prometheus/rules/recording_rules.yml
groups:
  # โ”€โ”€ HTTP / API metrics โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
  - name: nutriflow_http_metrics
    interval: 15s
    rules:
      # Tasa de peticiones HTTP por servicio (req/s)
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Tasa de errores 5xx
      - record: job:http_requests_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))

      # Porcentaje de error (usado en alertas)
      - record: job:http_error_rate:percent
        expr: |
            (job:http_requests_errors:rate5m / job:http_requests:rate5m) * 100

      # P95 latencia por servicio (principal SLO)
      - record: job:http_request_duration:p95
        expr: |
            histogram_quantile(0.95,
                sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
              )

  # โ”€โ”€ Resource / infrastructure metrics โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
  - name: nutriflow_resource_metrics
    interval: 30s
    rules:
      # CPU utilization % por nodo
      - record: instance:node_cpu:utilization
        expr: |
            100 - (avg by (instance) (
                rate(node_cpu_seconds_total{mode="idle"}[5m])
              ) * 100)

      # Memoria utilizada %
      - record: instance:node_memory:utilization
        expr: |
            100 - (
                (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
              )

      # Disco utilizado % (critico para report-service)
      - record: instance:node_disk:utilization
        expr: |
            100 - (
                (node_filesystem_avail_bytes{fstype!="tmpfs"}
                 / node_filesystem_size_bytes{fstype!="tmpfs"}) * 100
              )
Resumen de reglas
job:http_requests:rate5m sum by (job) (rate(http_requests_total[5m]))
job:http_error_rate:percent (errors/total) * 100 โ€” umbral alertas SLO
job:http_request_duration:p95 histogram_quantile(0.95, ...) โ€” P95 latencia
instance:node_cpu:utilization 100 - idle% por nodo GKE
instance:node_memory:utilization 100 - (MemAvail / MemTotal) * 100
instance:node_disk:utilization Critico para report-service (PDF storage)
๐Ÿ””

Alert Rules

9 alertas en 3 grupos โ€” disponibilidad, SLOs y recursos de infraestructura
Grupo: availability
ServiceDown critical PagerDuty ยท for: 1m
up{job=~"api-gateway|nutrition-engine|report-service"} == 0
Cualquier microservicio NutriFlow lleva mas de 1 minuto sin responder. Escalado automatico a PagerDuty.
HighErrorRate warning Slack #alerts-nutriflow ยท for: 5m
job:http_error_rate:percent{job="api-gateway"} > 1
Tasa de error HTTP en API Gateway supera el 1% (SLO: 0.1%). Notifica al canal Slack del equipo MLOps.
APIGatewayHighLatency warning Slack ยท for: 5m
job:http_request_duration:p95{job="api-gateway"} > 0.2
P95 latencia API Gateway supera 200ms (SLO definido para edge service). Revisar backpressure de nutrition-engine.
MLInferenceLatency warning Slack ยท for: 3m
job:http_request_duration:p95{job="nutrition-engine"} > 0.5
Latencia de inferencia ML supera 500ms. Verificar carga del modelo nutrinet-v2 y escalado de pods.
Grupo: resources
HighCPUUsage warning Slack ยท for: 5m
instance:node_cpu:utilization > 80
CPU de cualquier nodo GKE supera el 80% durante 5 minutos. Considerar escalado horizontal.
HighMemoryUsage warning Slack ยท for: 5m
instance:node_memory:utilization > 85
Memoria de nodo supera el 85%. Riesgo de OOM kill en pods de nutrition-engine.
DiskSpaceLow critical PagerDuty ยท for: 5m
instance:node_disk:utilization > 90
Disco supera el 90%. Critico para report-service que genera PDFs. Limpiar PDFs antiguos o ampliar PVC.
PostgreSQLDown critical PagerDuty ยท for: 1m
up{job="postgres-exporter"} == 0
Exporter de PostgreSQL sin respuesta โ€” posible caida de base de datos. Alerta critica con escalado inmediato.
/etc/prometheus/rules/alert_rules.yml (extracto clave)
groups:
  - name: nutriflow_availability
    interval: 30s
    rules:
      - alert: ServiceDown
        expr: up{job=~"api-gateway|nutrition-engine|report-service"} == 0
        for: 1m
        labels:
          severity: critical
          team:     mlops
          channel:  pagerduty
        annotations:
          summary:     "[CRITICO] Servicio {{ $labels.job }} caido"
          description: "{{ $labels.job }} en {{ $labels.instance }} lleva >1m sin responder"
          runbook_url: "https://runbook.nutriflow.io/service-down"

      - alert: MLInferenceLatency
        expr: job:http_request_duration:p95{job="nutrition-engine"} > 0.5
        for: 3m
        labels:
          severity: warning
          channel:  slack
        annotations:
          summary:     "Latencia ML degradada: {{ $value | humanizeDuration }}"
          description: "P95 inferencia nutrinet-v2 supera SLO de 500ms"
๐Ÿ“Š

SLOs Monitorizados

Estado actual de los objetivos de nivel de servicio de NutriFlow
API Gateway
Disponibilidad
99.97%/ 99.9% target
P95 Latencia
142ms/ 200ms target
Tasa de error
0.04%/ 0.1% target
Nutrition Engine
Disponibilidad
99.98%/ 99.9% target
P95 Latencia (ML)
430ms/ 500ms target
Inferencias/min pico
340/ 500 cap
Report Service
Disponibilidad
99.99%/ 99.9% target
P95 Latencia (PDF)
1.2s/ 2s target
Uso de disco
62%/ 90% critico
โœ…

Validacion con promtool

Comandos para verificar la configuracion antes de aplicar en produccion
โœ“ Configuracion principal validada sin errores promtool check config prometheus.yml
โœ“ Recording rules sin sintaxis incorrecta promtool check rules recording_rules.yml
โœ“ Alert rules con umbrales coherentes con SLOs promtool check rules alert_rules.yml
โœ“ Query de disponibilidad retorna resultados validos promtool query instant http://prometheus:9090 'up'
โœ“ 9 targets activos confirmados via API curl http://prometheus:9090/api/v1/targets | jq '.data.activeTargets | length'
โœ“ Helm install kube-prometheus-stack con retention 30d y 50Gi storage helm install prometheus prometheus-community/kube-prometheus-stack ...
Proximos pasos
โ†’ Conectar Grafana con datasource Prometheus y cargar dashboard NutriFlow
โ†’ Configurar AlertManager con receivers Slack + PagerDuty
โ†’ Instalar Thanos sidecar para retencion de largo plazo (>30d)
โ†’ Implementar SLO burn-rate alerts con skill slo-implementation