Software Systems Engineer, Data Pipelines

Here's the day to day:

• Automate, patch and otherwise develop data processing systems written in Python/C++ and orchestrated using Docker/Kubernetes
• Maintain data storage and retrieval systems, while consistently responding to needs for enhanced sorting, searching, and transformation
• Work with embedding spaces, keeping vector search operations fast and insightful
• Ensure the integrity and security of all proprietary data using industry best practices and up-to-date standards
• Track progress in model training pipelines and ensure that benchmarks are being met and exceeded
• Maximize the productivity vs cost for compute resources, both cloud and on-prem
• Troubleshoot ML inference by identifying training bottlenecks and outliers in training data
• Present analysis and insights with clear descriptions and visual aids as needed

Here's what we're looking for:

• 5+ years of experience in data engineering, data pipelines, ML Ops, or similar field
• Proficiency in Python, working knowledge (or better) of C++
• Solid programming fundamentals: programming paradigms, design patterns, data structures and algorithms, complexity analysis, concurrency
• Experience with data indexing and retrieval toolsets (example: ELK stack).  You can explain in great detail how these tools work under-the-hood and the tradeoffs associated with their use
• Experience using containers and with container orchestration (Docker/Kubernetes preferred)
• Knowledge of ML model training techniques, trade offs, and pitfalls
• Knowledge of vector search: similarity metrics, encoding embeddings, ANN search algorithms
• Software development best practices: version control (Git), designing and maintaining automated tests, giving and receiving code review, taking an active role in regular release cycles

Bonus points for:

• Math fundamentals: statistics, linear algebra, calculus
• Experience with data science tools: SciPy, R, Matlab or similar
• Open source contributor for a relevant tool