CompeteRadar — Scrapfly Webhook Handler

Webhooks recibidos hoy

2.847

callbacks procesados

+18% vs ayer

Firmas verificadas

100%

HMAC-SHA256 validas

0 rechazos

Precio mas bajo detectado

€47.99

AirMax 90 — Rival Store ES

-22% vs ayer

Idempotencia

duplicados ignorados

reentregas Scrapfly

Flujo de procesamiento

🔒

Verificar
firma HMAC

›

♻️

Idempotencia
webhook-id

›

🔀

Router por
resource-type

›

✅

Handler
especializado

›

💾

BD / S3
actualizado

Regla critica: despachar por X-Scrapfly-Webhook-Resource-Type ANTES de JSON.parse() — las capturas de pantalla llegan como bytes binarios, no JSON.

Enrutamiento por resource type

Header value	Handler	Destino	Estado
scrape	handleScrape()	scrape_snapshots (PG)	OK
extraction	handleExtraction()	price_snapshots (PG)	OK
screenshot	handleScreenshot()	S3 /screenshots/	OK
crawler_*	handleCrawler()	crawl_jobs (PG)	OK

Verificacion HMAC-SHA256 (JS)

// Scrapfly firma el raw body — NO el JSON parseado
function verifyScrapflySignature(rawBody, sig, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)       // <-- raw bytes
    .digest('hex')
    .toUpperCase();       // Scrapfly emite MAYUSCULAS

  const received = sig.toUpperCase();

  return crypto.timingSafeEqual(
    Buffer.from(received, 'hex'),
    Buffer.from(expected,  'hex')
  );
}

// CRITICO: usar express.raw(), no express.json()
app.post('/webhooks/scrapfly',
  express.raw({ type: '*/*', limit: '10mb' }),
  async (req, res) => { /* ... */ }
);
    

Headers Scrapfly relevantes

X-Scrapfly-Webhook-Signature A3F8...9C12 (HMAC-SHA256 hex upper)

X-Scrapfly-Webhook-Resource-Type scrape | extraction | screenshot

X-Scrapfly-Webhook-Job-Id job_01J9RX… (clave idempotencia)

X-Scrapfly-Webhook-Id wh_01J9RX… (clave deduplicacion)

X-Scrapfly-Webhook-Env live test

X-Scrapfly-Crawl-Event-Name crawler_finished (solo crawler)

Log de actividad reciente (ultimas 8 entregas)

14:37:52 🔍

[scrape] url=https://rivalstore.es/products/airmax-90 status=200 job=job_01J9RXF2K live → Guardado en scrape_snapshots, extraccion encolada

14:37:54 💶

[extraction] job=job_01J9RXF2K price=47.99 EUR in_stock=true product="Nike Air Max 90 (talla 42)" → price_snapshots actualizado ⚠ ALERTA precio bajo umbral: €47.99

14:37:55 📸

[screenshot] job=job_01J9RXG8P bytes=184320 → s3://competeradar-screenshots/screenshots/job_01J9RXG8P.jpg

14:38:01 🕷️

[crawler] event=crawler_url_visited crawl_id=crwl_01J9RX pages_visited=142 → crawl_jobs actualizado (status: running)

14:38:15 🔍

[scrape] url=https://rivalstore.es/products/ultraboost-22 status=200 job=job_01J9RXH1M → Guardado, extraccion encolada

14:38:16 ♻️

[webhook] Duplicado ignorado: webhookId=wh_01J9RXF2K — reentrega de Scrapfly ignorada correctamente

14:38:30 💶

[extraction] job=job_01J9RXH1M price=129.00 EUR in_stock=false product="Adidas Ultraboost 22 (talla 41)" → price_snapshots actualizado · precio dentro de umbral

14:39:02 🏁

[crawler] event=crawler_finished crawl_id=crwl_01J9RX pages_visited=389 pages_discovered=412 → crawl_jobs actualizado (status: completed)