AI Benchmark Engineer | Native Language Specialist - Portuguese (Brazil) - Remote
LILT
- КомпанияLILT
- СтранаБразилия
- ФорматУдалённо
- ГрафикRemote
- АдресBrazil (Remote)
Описание
ABOUT THE OPPORTUNITY
We are building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. Our goal is to measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows.
We are seeking experienced native-speaking software engineers to design, build, and validate these benchmarks. You will create high-signal, high-quality tasks that genuinely test a model's ability to handle multilingual environments without relying on English translation crutches.
Note this is a remote, freelance opportunity
WHAT YOU’LL DELIVER
- Task Engineering: Evaluating Coding Agents.
- Asset Creation: Build realistic task environments using datasets and files in your native language. Crucially, these assets must remain in the target language to genuinely measure multilingual handling.
- Prompting & Translation: finding failure points where AI does not work, in your native language
- Implementation & Verification: Support the development of robust solutions (reference implementations) and write highly reliable, deterministic verifier scripts (using rubric-based judging only when strictly necessary).
- Calibration & Execution: Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus).
- Quality…
Вакансии этой компании
- Medical Translators - Nepali - US-based — LILT, США
- Medical Translators - Haitian Creole - US-based — LILT, США
- Program Manager, Life Sciences — LILT, США
- Medical Translators - Thai - US-based — LILT, США
- Medical Translators - Farsi - US-based — LILT, США
Похожие вакансии
- Ocean Dispute Resolution Specialist — Maersk, Индия
- Performance Ads Specialist (Meta & Google) — Platinumlist, Индия
- Technical Talent Acquisition Specialist — Platinumlist, Казахстан
- AdOps Trafficking Specialist — Criteo, Индия
- Payroll Specialist — Criteo, Франция
- IT Procurement Specialist — Criteo, Испания
- Payroll Specialist — Criteo, Бразилия
- Sales Transformation Lead International, Specialist Sales — Rubrik, Великобритания