Arquitectura de Monitorizacion
Flujo de metricas desde los targets hasta Grafana y AlertManager
api-gateway
:8080/metrics
nutrition-engine
:8081/metrics
report-service
:8082/metrics
node-exporter
x3 :9100
postgres-exp
:9187
redis-exp
:9121
โโโ scrape /metrics cada 15s โโโ
โ
Prometheus Server
:9090 ยท TSDB 30d ยท GKE
โ
๐
Grafana
:3000
Grafana
:3000
๐
AlertManager
:9093 โ Slack + PagerDuty
AlertManager
:9093 โ Slack + PagerDuty
๐
Thanos
Long-term storage
Thanos
Long-term storage
Scrape Interval
15s
Global ยท ML service 10s
Retencion TSDB
30d
50 GiB storage volume
Targets activos
7
3 nodos + 4 servicios
Recording Rules
8
CPU ยท Mem ยท HTTP ยท ML
Alert Rules
9
3 critical ยท 6 warning
Scrape Targets
7 endpoints monitorizados con estado actual simulado
| Job | Endpoint | Intervalo | Labels | Estado |
|---|---|---|---|---|
| prometheus | localhost:9090 | 15s | self-monitor | UP |
| node-exporter | node1:9100, node2:9100, node3:9100 | 15s | env=prod cluster=gke-nutriflow | UP x3 |
| api-gateway | api-gw.nutriflow.svc:8080 | 15s | service=api-gateway tier=edge | UP |
| nutrition-engine | nutrition-engine.svc:8081 | 10s | service=ml-engine tier=inference | UP |
| report-service | report-svc.nutriflow.svc:8082 | 15s | service=reports tier=backend | UP |
| postgres-exporter | postgres-exporter.svc:9187 | 30s | db=postgresql | UP |
| redis-exporter | redis-exporter.svc:9121 | 15s | db=redis | UP |
prometheus.yml
Configuracion principal โ scrape jobs, service discovery, alertmanager
/etc/prometheus/prometheus.yml
# NutriFlow โ Prometheus Configuration # Cluster: GKE production ยท Region: eu-west1 # Generado por CULTIVA IA ยท configuracion-prometheus skill global: scrape_interval: 15s evaluation_interval: 15s scrape_timeout: 10s external_labels: cluster: "gke-nutriflow-prod" region: "eu-west1" environment: "production" team: "mlops" alerting: alertmanagers: - static_configs: - targets: - "alertmanager.monitoring.svc:9093" timeout: 10s api_version: v2 rule_files: - "/etc/prometheus/rules/recording_rules.yml" - "/etc/prometheus/rules/alert_rules.yml" scrape_configs: # --- Prometheus self-monitoring --- - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] # --- Node exporters (3 nodos GKE) --- - job_name: "node-exporter" scrape_interval: 15s static_configs: - targets: - "gke-node-1.nutriflow.internal:9100" - "gke-node-2.nutriflow.internal:9100" - "gke-node-3.nutriflow.internal:9100" labels: env: "production" cluster: "gke-nutriflow-prod" relabel_configs: - source_labels: [__address__] target_label: instance regex: "([^:]+)(:[0-9]+)?" replacement: "${1}" # --- API Gateway (edge service) --- - job_name: "api-gateway" scrape_interval: 15s metrics_path: "/metrics" static_configs: - targets: ["api-gw.nutriflow.svc.cluster.local:8080"] labels: service: "api-gateway" tier: "edge" slo: "99.9" # --- Nutrition Engine (ML inference) โ intervalo reducido --- - job_name: "nutrition-engine" scrape_interval: 10s # Mas frecuente para inferencia critica metrics_path: "/metrics" static_configs: - targets: ["nutrition-engine.nutriflow.svc.cluster.local:8081"] labels: service: "nutrition-engine" tier: "inference" model: "nutrinet-v2" # --- Report Service --- - job_name: "report-service" scrape_interval: 15s static_configs: - targets: ["report-svc.nutriflow.svc.cluster.local:8082"] labels: service: "report-service" tier: "backend" # --- PostgreSQL Exporter --- - job_name: "postgres-exporter" scrape_interval: 30s static_configs: - targets: ["postgres-exporter.nutriflow.svc.cluster.local:9187"] labels: db: "postgresql" service: "nutriflow-db" # --- Redis Exporter --- - job_name: "redis-exporter" scrape_interval: 15s static_configs: - targets: ["redis-exporter.nutriflow.svc.cluster.local:9121"] labels: db: "redis" service: "session-cache"
Recording Rules
8 metricas pre-computadas para queries costosas en dashboards y alertas
/etc/prometheus/rules/recording_rules.yml
groups: # โโ HTTP / API metrics โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ - name: nutriflow_http_metrics interval: 15s rules: # Tasa de peticiones HTTP por servicio (req/s) - record: job:http_requests:rate5m expr: sum by (job) (rate(http_requests_total[5m])) # Tasa de errores 5xx - record: job:http_requests_errors:rate5m expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) # Porcentaje de error (usado en alertas) - record: job:http_error_rate:percent expr: | (job:http_requests_errors:rate5m / job:http_requests:rate5m) * 100 # P95 latencia por servicio (principal SLO) - record: job:http_request_duration:p95 expr: | histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])) ) # โโ Resource / infrastructure metrics โโโโโโโโโโโโโโโโโโโโโโโโโโโ - name: nutriflow_resource_metrics interval: 30s rules: # CPU utilization % por nodo - record: instance:node_cpu:utilization expr: | 100 - (avg by (instance) ( rate(node_cpu_seconds_total{mode="idle"}[5m]) ) * 100) # Memoria utilizada % - record: instance:node_memory:utilization expr: | 100 - ( (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 ) # Disco utilizado % (critico para report-service) - record: instance:node_disk:utilization expr: | 100 - ( (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"}) * 100 )
Resumen de reglas
job:http_requests:rate5m
sum by (job) (rate(http_requests_total[5m]))
job:http_error_rate:percent
(errors/total) * 100 โ umbral alertas SLO
job:http_request_duration:p95
histogram_quantile(0.95, ...) โ P95 latencia
instance:node_cpu:utilization
100 - idle% por nodo GKE
instance:node_memory:utilization
100 - (MemAvail / MemTotal) * 100
instance:node_disk:utilization
Critico para report-service (PDF storage)
Alert Rules
9 alertas en 3 grupos โ disponibilidad, SLOs y recursos de infraestructura
Grupo: availability
ServiceDown
critical
PagerDuty ยท for: 1m
up{job=~"api-gateway|nutrition-engine|report-service"} == 0
Cualquier microservicio NutriFlow lleva mas de 1 minuto sin responder. Escalado automatico a PagerDuty.
HighErrorRate
warning
Slack #alerts-nutriflow ยท for: 5m
job:http_error_rate:percent{job="api-gateway"} > 1
Tasa de error HTTP en API Gateway supera el 1% (SLO: 0.1%). Notifica al canal Slack del equipo MLOps.
APIGatewayHighLatency
warning
Slack ยท for: 5m
job:http_request_duration:p95{job="api-gateway"} > 0.2
P95 latencia API Gateway supera 200ms (SLO definido para edge service). Revisar backpressure de nutrition-engine.
MLInferenceLatency
warning
Slack ยท for: 3m
job:http_request_duration:p95{job="nutrition-engine"} > 0.5
Latencia de inferencia ML supera 500ms. Verificar carga del modelo nutrinet-v2 y escalado de pods.
Grupo: resources
HighCPUUsage
warning
Slack ยท for: 5m
instance:node_cpu:utilization > 80
CPU de cualquier nodo GKE supera el 80% durante 5 minutos. Considerar escalado horizontal.
HighMemoryUsage
warning
Slack ยท for: 5m
instance:node_memory:utilization > 85
Memoria de nodo supera el 85%. Riesgo de OOM kill en pods de nutrition-engine.
DiskSpaceLow
critical
PagerDuty ยท for: 5m
instance:node_disk:utilization > 90
Disco supera el 90%. Critico para report-service que genera PDFs. Limpiar PDFs antiguos o ampliar PVC.
PostgreSQLDown
critical
PagerDuty ยท for: 1m
up{job="postgres-exporter"} == 0
Exporter de PostgreSQL sin respuesta โ posible caida de base de datos. Alerta critica con escalado inmediato.
/etc/prometheus/rules/alert_rules.yml (extracto clave)
groups: - name: nutriflow_availability interval: 30s rules: - alert: ServiceDown expr: up{job=~"api-gateway|nutrition-engine|report-service"} == 0 for: 1m labels: severity: critical team: mlops channel: pagerduty annotations: summary: "[CRITICO] Servicio {{ $labels.job }} caido" description: "{{ $labels.job }} en {{ $labels.instance }} lleva >1m sin responder" runbook_url: "https://runbook.nutriflow.io/service-down" - alert: MLInferenceLatency expr: job:http_request_duration:p95{job="nutrition-engine"} > 0.5 for: 3m labels: severity: warning channel: slack annotations: summary: "Latencia ML degradada: {{ $value | humanizeDuration }}" description: "P95 inferencia nutrinet-v2 supera SLO de 500ms"
SLOs Monitorizados
Estado actual de los objetivos de nivel de servicio de NutriFlow
API Gateway
Disponibilidad
99.97%/ 99.9% target
P95 Latencia
142ms/ 200ms target
Tasa de error
0.04%/ 0.1% target
Nutrition Engine
Disponibilidad
99.98%/ 99.9% target
P95 Latencia (ML)
430ms/ 500ms target
Inferencias/min pico
340/ 500 cap
Report Service
Disponibilidad
99.99%/ 99.9% target
P95 Latencia (PDF)
1.2s/ 2s target
Uso de disco
62%/ 90% critico
Validacion con promtool
Comandos para verificar la configuracion antes de aplicar en produccion
โ
Configuracion principal validada sin errores
promtool check config prometheus.yml
โ
Recording rules sin sintaxis incorrecta
promtool check rules recording_rules.yml
โ
Alert rules con umbrales coherentes con SLOs
promtool check rules alert_rules.yml
โ
Query de disponibilidad retorna resultados validos
promtool query instant http://prometheus:9090 'up'
โ
9 targets activos confirmados via API
curl http://prometheus:9090/api/v1/targets | jq '.data.activeTargets | length'
โ
Helm install kube-prometheus-stack con retention 30d y 50Gi storage
helm install prometheus prometheus-community/kube-prometheus-stack ...
Proximos pasos
โ
Conectar Grafana con datasource Prometheus y cargar dashboard NutriFlow
โ
Configurar AlertManager con receivers Slack + PagerDuty
โ
Instalar Thanos sidecar para retencion de largo plazo (>30d)
โ
Implementar SLO burn-rate alerts con skill slo-implementation