Distributed Compute Engineer

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pre-training, domain-specific RL, ultra-long context, and test-time compute to achieve this goal.

About the role: As a distributed systems engineer for compute, you will build the stack and systems that enable 1T+ parameter model training and efficient inference on Magic’s GPU clusters. 

What you might work on: 

• Develop and maintain the software stack to support large-scale, highly available AI training and inference infrastructure

• Implement and optimize systems for data processing and inference using technologies like Ray, Redis, Message Queues (Kafka), distributed communication libraries (gRPC, ZeroMQ) and HPC technologies

• Orchestrate fine-grained data movement using Rust, C++ and NCCL or UCX

• Design and manage high-performance storage and caching solutions to support data-intensive applications

• Build with an eye towards fault-tolerance, performance and observability

• Hack on the internals of deep learning frameworks (PyTorch, Jax) in a distributed setting

• Troubleshoot and resolve complex issues across GPU resources, networking, OS, drivers, and cloud environments. Automate fault detection and recovery processes

What we’re looking for: 

• Deep knowledge of distributed systems design and cloud platforms (AWS, GCP, Azure)

• Extensive experience designing and operating high-availability, data-intensive systems

• Specific experience in operating large-scale storage or networking solutions

• Experience with the internals or operation of distributed DBMS (Clickhouse, Snowflake, BigQuery, vector DBs), batch and stream processing (Spark, Flink), file/storage systems (RocksDB, Lustre/NFS), and distributed ML systems (Deepspeed, torch.distributed, Ray, Dask) or HPC workloads

• Exceptional problem-solving skills across complex infrastructure up and down the stack 

Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.

Our culture:

• Integrity. Words and actions should be aligned

• Hands-on. At Magic, everyone is building 

• Teamwork. We move as one team, not N individuals

• Focus. Safely deploy AGI. Everything else is noise

• Quality. Magic should feel like magic

Compensation, benefits and perks (US):

• Annual salary range: $90K - $900K

• Equity is a significant part of total compensation, in addition to salary

• 401(k) plan with 6% salary matching

• Generous health, dental and vision insurance for you and your dependants

• Unlimited paid time off

• Option to work in-person in SF or remotely

• Visa sponsorship and relocation stipend to bring you to SF

• A small, fast-paced, highly focused team