About the role
A mid-sized software development company in Sunnyvale is hiring for this role. You will own the health of large-scale Spark-based ingestion pipelines, so that data reaches annotation, data science and model training teams reliably and in the right order.
What you'll do
- Debug and improve large-scale Spark-based ingestion pipelines.
- Resolve corrupt data, stalled jobs, queue problems and complex workflow failures.
- Increase pipeline resilience, throughput, reliability, cost efficiency and operability.
- Prioritize datasets for downstream annotation, data science and model training.
What you bring
- Hands-on experience building and troubleshooting Spark-based data pipelines.
- Strong software engineering skills applied to data ingestion.
- Skill at diagnosing failures across jobs, queues and workflows.
- A focus on reliability, cost efficiency and operability.
Details
- Location: Sunnyvale, United States
- Type: Full-time
Apply below and a Traverse recruiter will be in touch.