Data & Analytics
Overview
Data Engineering Copilot helps you design, debug, recover, and improve production data systems across Spark, Databricks, Delta Lake, SQL, CDC, Structured Streaming, Airflow, backfills, data quality, observability, and lakehouse architectures. It can diagnose failed or slow pipelines, reason about duplicate and late data, plan safe replay and backfill strategies, review checkpoint and streaming-state risks, tune Spark from execution evidence, and create production-oriented SQL, PySpark, and orchestration patterns. It prioritizes correctness, recoverability, SLA, security, and cost before scaling or redesigning.
Capabilities
Design Reliable Batch, Cdc, Streaming, And Lakehouse ArchitecturesDebug Failed, Slow, Duplicated, Stale, Or Backlogged Production PipelinesPlan Safe Replay, Backfill, Recovery, And Checkpoint StrategiesReview Delta Merge Logic For Keys, Ordering, Duplicates, Updates, And DeletesDiagnose Spark And Databricks Performance Using Plans, Shuffle, Spill, Skew, And File EvidenceDesign Structured Streaming State, Watermark, Checkpoint, And Sink BehaviorCreate Production-Oriented Sql, Pyspark, Databricks, And Airflow PatternsBuild Data-Quality Gates, Reconciliation, Freshness Checks, And Operational RunbooksEvaluate Retry Safety, Idempotency, Schema Evolution, And Downstream ContractsUse Current Official Docs For Spark, Databricks, Delta Lake, Airflow, And Runtime-Specific Behavior
Example Prompts
Click any prompt to copy it.
