Xebia
Full-time · 2 yrs 3 mos
Gurugram, India
Junior Consultant, Data Engineer · Client engagement · Global apparel retail brand
Jul 2026 – Present · 4 mos
- Data correctness: The site marketing team’s twice-weekly dashboard lost its campaign field in September 2026. I fixed the pipeline and ran a chained dbt backfill over 6 months of events, repairing 1,455 records, so their reports are complete again.
- Silent failures: Batches stopped at the 4-hour Dataproc limit were still reported as successful. I added time limits to 600+ DAGs, sized from each job’s worst recent runs, so a timed-out job now fails and raises an alert.
- Pipeline framework: Owned per-task time limits in the shared Vertex AI pipeline framework: each limit comes from measured run times, and the alert names the task that failed. Live on 59 tasks across 25 e-commerce pipelines.
- Cost: A high-cost Hybris Spark pipeline was cancelled at the 4-hour limit on 9 of 9 days. After I pruned its lookup joins and partition scans, 11 of 16 runs succeeded and daily cost fell 19.7%. I also right-sized clusters for 29 pipelines, cutting their cost 17.6%.
- Shared component: Merged duplicated Dataproc pipeline templates into one framework component. 524 production schedules now run on it with no change in output.
- Source migration: Led preprod validation of 7 Hybris pipelines on the new SQL Server Hyperscale source; all 7 matched the old output.
- Team practice: Reviewed 102 pull requests across 7 repositories, triaged production incidents and wrote 7 design and analysis docs.
- Dataproc
- Vertex AI
- BigQuery
- DBT
- Dataplex
Technical Trainee · Centre of Excellence department · Client-focused project
Aug 2024 – Jun 2026 · 1 yr 11 mos
- Cloud migration: Led the move of 13 e-commerce sources (OMS, Hybris, Shopify, Salsify, Cordial, Zendesk and more) from Databricks to Dataproc and BigQuery, checking each one against the old output.
- Order reporting: Designed change data capture for the OMS order tables with Datastream, so order reporting reads one source of truth in BigQuery (snapshot and change views), to replace Qlik.
- Data quality: Built Dataplex quality scans for 2 data products, with thresholds tested against years of production data, and a CI gate that blocks dbt models shipped without a scan.
- Client work: Turned the Looker Studio sales and demand reporting needs of the brand’s product owners and BI analysts into data models.
- Generative AI: At the Centre of Excellence, built RAG document extraction (LlamaIndex chunking into Weaviate and ChromaDB) and a Terraform linter that uses Claude on AWS Bedrock to check all 6 Well-Architected pillars.
- Apache Spark
- Dataproc
- BigQuery
- Datastream
- Dataplex
- LLMs
- RAG
