GitOps Pipeline — NeuralDeploy

📊 DORA Metrics — últimas 30 días

Deployment Frequency

4.7

deployments/día/servicio

↑ +56% vs baseline manual · Elite

Lead Time for Changes

8min

commit → producción

↓ −91% (era 1h 35min) · Elite

Mean Time to Recovery

3.2min

auto-rollback via selfHeal

↓ −82% · objetivo <5min ✓

Change Failure Rate

1.4%

fallos en producción

↓ −76% · objetivo <2% ✓

🔄 Flujo CI/CD — deploy inference-engine v2.8.1

✅

git push main

hace 14m

🔨

GitHub Actions CI

build + test ✓

📦

Docker → ECR

sha256:3a9f…

📝

Update GitOps repo

image tag bump

🔁

ArgoCD detects diff

syncing…

🐦

Canary 20%

en curso

🚀

Producción 100%

pendiente

🐦 Canary Deployment en curso — inference-engine

⚡ inference-engine · v2.8.0 → v2.8.1 (hotfix latencia P99)

Iniciado hace 8min · cluster eu-west-1 · 5 réplicas totales

0%50%100%

20%

Canary inicial

✓ 1 réplica · hace 6min

50%

Pausa 2 min

⏳ en 4 min (auto)

100%

Promoción completa

estimado +8 min

📊 Métricas v2.8.1 (20% tráfico): P50: 42ms · P99: 187ms (era 340ms) · Error rate: 0.08% · Auto-rollback umbral: >2% errores

📋 Estado de Aplicaciones

Aplicación	Sync	Cluster
api-gateway v3.4.2 · 4/4 pods	● Synced	eu-west-1
inference-engine v2.8.0→2.8.1 · canary 20%	⚡ Canary	eu-west-1
model-registry v1.9.0 · 2/2 pods	● Synced	eu-west-1
api-gateway v3.4.2 · 3/3 pods	● Synced	us-east-1
inference-engine v2.8.0 · 5/5 pods	● Synced	us-east-1
model-registry v1.9.0 · 2/2 pods	● Synced	ap-southeast-1

☸️ Multi-cluster Overview

● EU Primary

aws / eu-west-1

Apps: 3/3 synced

1 canary activo

● US East

aws / us-east-1

Apps: 3/3 synced

sin incidencias

● APAC

aws / ap-southeast-1

Apps: 3/3 synced

sin incidencias

🔔 Notificaciones Slack recientes

🐦

Canary 20% completado — inference-engine eu-west-1

P99 latencia: 187ms (↓45%). Error rate estable 0.08%

hace 2m

✅

Sync OK — api-gateway v3.4.2 (3 clusters)

Desplegado sin incidencias, selfHeal activo

hace 47m

⚠️

Auto-rollback activado — model-registry us-east-1

Error rate 3.1% superó umbral. Revertido a v1.8.9 en 2m 18s

ayer

📁 Estructura GitOps Repo — App of Apps

gitops-neuraldeploy/

argocd/

root-app.yaml # App of Apps raíz
applications/

api-gateway.yaml
inference-engine.yaml
model-registry.yaml

apps/

production/

api-gateway/

kustomization.yaml
deployment.yaml
hpa.yaml

inference-engine/

rollout.yaml # Canary
analysis-template.yaml
kustomization.yaml

staging/

infrastructure/

external-secrets.yaml
cert-manager.yaml
monitoring.yaml

📄 ArgoCD — inference-engine Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-engine
  namespace: argocd
spec:
  project: neuraldeploy-prod
  source:
    repoURL: https://github.com/neuraldeploy/gitops
    targetRevision: main
    path: apps/production/inference-engine
  destination:
    server: https://eks-eu-west-1.neuraldeploy.io
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  info:
    - name: slack-channel
      value: #deploys-prod
        

🐦 Canary Rollout YAML — inference-engine

# apps/production/inference-engine/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: inference-engine
  namespace: production
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: inference-svc-canary
      stableService: inference-svc-stable
      analysis:
        templates:
          - templateName: error-rate
        args:
          - name: service-name
            value: inference-svc-canary
      steps:
        - setWeight: 20
        - pause: {duration: 1m}
        - setWeight: 50
        - pause: {duration: 2m}
        - setWeight: 100
        

# AnalysisTemplate — auto-rollback
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate
      interval: 30s
      successCondition: result[0] <= 0.02
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{
                service="{{args.service-name}}",
                status=~"5.."
            }[2m])) /
            sum(rate(http_requests_total{
                service="{{args.service-name}}"
            }[2m]))
        

🔐 Gestión de Secretos — External Secrets + AWS SM

🏛️

AWS Secrets Manager

prod/neuraldeploy/*
API keys, DB creds

→ sync 1h

🔄

External Secrets Operator

ESO en cluster
reconcilia automático

→ K8s Secret

📦

Pod env vars

montado como env
0 secretos en Git

# infrastructure/external-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: model-api-keys
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
  target:
    name: model-api-keys
  data:
    - secretKey: openai-key
      remoteRef:
        key: prod/neuraldeploy/openai
    - secretKey: anthropic-key
      remoteRef:
        key: prod/neuraldeploy/anthropic
        

✅ Best Practices implementadas

🌿

Repos separados por entorno

staging y production en branches aisladas con PR gates

🛡️

RBAC Git + ArgoCD Projects

dev puede hacer push a staging, prod requiere aprobación

🔔

Notificaciones Slack

#deploys-prod para sync OK/fail, #alerts para auto-rollback

❤️

Health checks custom

Lua scripts para Argo Rollouts + Prometheus AnalysisTemplate

🔑

0 secretos en Git

External Secrets Operator + AWS SM; pre-commit hook bloquea

🏗️

App of Apps pattern

1 root-app despliega todo; staging = espejo exacto de prod

🏷️

Tags semánticos para rollback

v{semver} en cada merge; argocd rollback --revision en <30s

📈

Prometheus DORA dashboard

Grafana board tracking deployment freq, MTTR, CFR en tiempo real