Software Engineer - Inference Performance

Baseten is a rapidly growing company that specializes in powering inference for AI applications, serving clients like Notion and OpenEvidence. The team focuses on building infrastructure and tools that enable cutting-edge models to be deployed efficiently. The company recently secured significant funding, emphasizing its expansion and leadership in AI infrastructure. The environment is dynamic, with a focus on innovation and supporting AI companies at the forefront of technology.

The role involves working on optimizing AI inference performance across the stack, from runtime internals to scheduling and routing. The engineer will implement advanced techniques like speculative decoding and cache management, aiming to improve speed and cost-efficiency. The position requires expertise in GPU architecture, programming languages like Python or C++, and familiarity with ML libraries such as PyTorch and TensorRT.

The work is hybrid, with locations in major North American cities, and is a full-time contract role. Candidates should have a background in computer science or related fields, with experience in LLM optimization and inference engine contributions being advantageous. The role offers opportunities to influence AI model deployment, contribute to open-source projects, and work on innovative projects that impact AI performance and cost savings.