Applied AI Engineer

The role

Orchestra helps teams find the right model for their recurring work and build a specialist when the task warrants it. That begins with understanding what a successful result looks like, turning that definition into an evaluation, and improving the system against it.

You will take real workloads from examples and failure reports to tested improvements. You will work across prompts, schemas, tool contracts, model comparisons, and selective training, then help turn successful experiments into reliable production behavior.

What you'll work on

• Translate a workflow and its expert feedback into concrete acceptance criteria, test cases, and a held-out evaluation set.
• Build and debug LLM applications that produce structured outputs, use tools, and carry out multi-step work.
• Improve prompts, context, parsers, and tool contracts before reaching for a more complex intervention.
• Compare model routes on the same tasks, measuring outcome quality alongside latency and the cost of completing the work.
• Prepare accepted examples and corrections for specialist training, then evaluate the resulting model against the baseline.
• Investigate production failures and build regression checks that prevent them from returning.
• Work with research and infrastructure engineers to turn experiments into maintainable integrations, with observable behavior and a fallback when a candidate falls short.

What you bring

• Experience building software around language models and taking responsibility for how it behaves beyond a demo.
• Strong programming skills, with comfort in Python and production APIs, data processing, and debugging.
• Practical experience with evaluation, structured outputs, tool calling, or agent workflows.
• The ability to read messy examples, identify the actual failure, and choose a useful next experiment.
• Care for software reliability and for measuring improvements honestly, including the tradeoffs a new route introduces.
• Clear communication and a willingness to work through unfamiliar domains with the people who understand them.

Experience with fine-tuning or model serving is helpful. You do not need to have worked on every part of the stack; show us where you have gone deep and how you learn.

Tell us what you’ve built.

Tell us about an AI system you built, a failure you had to understand, and how you decided whether your fix worked. A repository, technical write-up, or concise walkthrough is welcome. Let us know what you would like to build at Orchestra, and leave out confidential information.

Opens your email app. You can also write to founders@understudylabs.com.