News Scraper MX + INE Importer
Data engineering over public sources: press scraping and a geospatial ETL of open INE data.
- Year
- 2025
- Role
- Solo developer (data engineering)
About the project
A data engineering platform over Mexican public sources, with two components. News Scraper MX: a Scrapy-based scraper with 9 spiders (Google News RSS, several national outlets, one using Selenium for Cloudflare/JS sites, and a configurable generic one), with a structured article model and deduplication and persistence pipelines. INE Importer: a geospatial ETL that ingests aggregated, open INE data and persists it to PostgreSQL + PostGIS, with temporally versioned geography, idempotent and incremental ingestion (ETag/Last-Modified, streaming hashing, retries), and a Typer CLI. Designed with privacy as a core principle: a PII guard rejects any file that looks like personal data; it only processes aggregated, public data.