In A Nutshell
Candid is a nonprofit that provides the most comprehensive data and insights about the social sector. We get you the information you need to do good. Candid currently has an opportunity for a Senior Data Pipeline. The Senior Data Pipeline Engineer is a senior technical role responsible for the day-to-day development, and delivery of Candid’s data pipelines. This role blends deep SQL and ETL engineering expertise with hands-on experience across modern cloud data platform technologies to design, build, and tune pipelines that move data reliably from source systems into Candid’s data warehouse and lakehouse.
Responsibilities
- Pipeline development: Architect, develop, and maintain ETL/ELT and C#/.NET pipeline services (SSIS, Airflow, AWS Lambda, enhancer and message-processing services) that move data reliably from source systems into Candid’s warehouse and lakehouse.
- Cloud platform & performance: Support Candid’s evolving cloud data platform, including Iceberg ingestion and transformation pipelines, DynamoDB, and query performance tuning via Trino/Starburst.
- Data quality & incident resolution: Monitor data quality end-to-end and investigate production pipeline issues at the root cause, documenting findings and driving fixes through to reprocessing.
- Source onboarding & schema coordination: Scope and onboard new external and regulatory data sources end-to-end, and contribute to schema and architecture decisions, coordinating changes across ingestion and transformation layers.
- Documentation, mentorship & process: Maintain SOP and technical documentation, mentor and review the work of Associate-level engineers, and participate in the team’s release and deployment process.
- Partnership & support: Partner with the Data Operations Manager on architecture and roadmap decisions, and provide escalation-level support for mission-critical pipeline issues, including after-hours as needed.
Skillset
- 5+ years of experience in database engineering, data engineering, or ETL development, with demonstrated ownership of production data pipelines.
- Expert-level Transact-SQL skills (stored procedures, views, indexing) and hands-on C#/.NET development experience.
- Proven experience architecting ETL processes with SSIS; experience with orchestration platforms such as Apache Airflow preferred.
- Experience with AWS and open table formats (Apache Iceberg, Parquet); AWS Lambda, DynamoDB, and Trino/Starburst a plus.
- Strong data quality and root-cause troubleshooting skills, with experience resolving production data incidents.
- Familiarity with source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments.
- Ability to mentor and review the work of other engineers; excellent written and verbal communication.
- Good business judgment, independent prioritization, and the ability to work well in a distributed team.
- Willingness to perform other duties and special projects as needed/requested.
- Sensitivity and respect for racial, gender, sexual orientation, and cultural differences.
- Commitment to Candid’s values: driven, direct, accessible, curious, and inclusive.
