Engineering labRunnable locally

GCP Ingestion Lab

What happens when the same event arrives twice?

Break a file-ingestion pipeline, replay events, and inspect the rows and operations that survive.

The problem

A collected file is not necessarily a published file. Once published, events can repeat or arrive late, and a change to identity rules can duplicate data that already exists.

The approach

Compare one loader rule at a time using the same synthetic input, with source listings, operation counts and warehouse output visible together.
Use a completed receipt to skip repeated downloads and merges. Read the exact object generation and compare observation time rather than arrival order.
Expose the identity trap: name-based keys collapse listings; replaying with new keys can double-count them. Inspect an explicit partition repair.

What you can inspect

Start with “Send it twice”, then try a late arrival or “Two URLs, one name”. The local browser invokes the loader core and reports its actual in-memory adapter operations.

Scope & limitations

Synthetic fixtures and in-memory adapters. Cloud timing, billing and IAM are not emulated. The cloud deployment has separate setup and operational requirements.

Run it locally

Node.js 22 or newer. The default lab needs no API key or cloud account.

git clone https://github.com/alejandrofrank/gcp-ingestion-lab.git
cd gcp-ingestion-lab
npm run dev

Then open http://127.0.0.1:4313. Follow the repository for optional integrations and verification commands.