Bakiano
How do you make scattered market data useful?
A market intelligence platform for Venezuela, from daily collection to historical prices and an AI analyst.
The problem
Prices are scattered across stores, categories and currencies. A useful comparison needs both a trustworthy current price and the history behind it.
The approach
- Collect source snapshots and preserve the original observations before transforming them for the product.
- Keep catalog enrichment separate from prices, then join attributes to the current data. Derive USD prices using the relevant BCV exchange rate.
- Serve dashboards, alerts and an AI analyst through scoped tools and bounded warehouse queries.
What you can inspect
Explore the live product and the public engineering case study. The three labs that follow isolate ingestion, product identity and query controls so those decisions can be inspected separately.
Scope & limitations
The product source is private. The public case study documents the system; the companion labs are standalone examples, not a copy of the production platform.
Technical pipeline
From infrastructure to the product. Open a stage to go deeper.
InfrastructureGCP · Terraform · Docker · GitHub Actions
Terraform defines the jobs, schedules, storage, warehouse, IAM, and networking. Changes ship through pull requests; the deployment workflow builds the changed Docker images and applies infrastructure changes after merge. Cloud Run hosts the collection jobs and the product.
Network accessTailscale · gateway VMs
Tailscale gateways support the collectors that need Venezuelan network access. The gateway VMs start and stop on a schedule around the collection window. This is supporting infrastructure for scraping, not a stage that transforms the data.
Scraping & orchestrationCloud Scheduler · Cloud Run Jobs · Scrapy / nodriver
Collectors are organized by vendor and domain. Cloud Scheduler triggers the Cloud Run jobs, and each run produces a CSV snapshot. Scrapy or browser automation handles collection depending on the source; refresh cadence varies by domain.
Archive & ingestionGoogle Cloud Storage · Cloud Functions
CSV snapshots land in GCS, organized by domain, vendor, and date. An object-finalize event triggers the loader, which appends records to BigQuery and records the run’s health. The archive retains the source snapshots for reprocessing and backfills.
Data warehouseBigQuery · raw / staging / product layers
Raw tables preserve the collected values and history. Staging holds cleaned, deduplicated data; product tables and serving views support analysis and the application. Operational tables track vendors, collection runs, and exchange rates. Supermarket and retail raw data is partitioned by ingestion date.
Transformations & currencyScheduled SQL · idempotent MERGE · BCV rates
Scheduled queries process a rolling three-day window, allowing recent missed runs to recover on the next execution. They clean the data and produce daily summaries. Raw prices stay unchanged: serving views derive USD values from vendor currency rules and the BCV rate associated with the ingestion date.
AI enrichmentGemini batch · Vertex AI · BigQuery
A separate branch from raw data classifies new supermarket products: product type, brand, units, weight, and other catalog attributes. Classification is stored separately from prices. Current prices are joined from the latest raw partition, so enrichment does not freeze yesterday’s price into the catalog.
Serving & analystNext.js · Cloud Run · Gemini on Vertex AI
The product combines warehouse views and catalog enrichment for chat, dashboards, and a scoped data API. The AI analyst uses a defined set of tools rather than unrestricted SQL. Metered, byte-capped queries and a shared result cache control warehouse usage; run-health monitoring closes the operational loop.