Python Software Engineering LLM Evaluator

About OpenTrain

OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. It helps contributors discover projects, build a professional profile, and apply to opportunities that match their skills. Creating an OpenTrain account is free.

  • Build a lasting portfolio of AI training and evaluation experience.
  • Find flexible contractor work connected to the rapidly growing AI industry.

About AI Training and LLM Evaluation

AI training is the human side of building artificial intelligence. For language models, skilled contributors review outputs, design realistic tasks, and assess whether systems can produce accurate, useful code. Your software engineering judgment can help improve how models handle real development challenges.

  • Work with realistic programming and bug-fixing scenarios.
  • Evaluate model behavior against real codebases and software engineering expectations.
  • Contribute to cutting-edge AI development through practical technical review.

The Role

OpenTrain is recruiting Python software engineers for a three-month contractor assignment focused on LLM evaluation and training. You will work with public GitHub repositories and realistic software engineering tasks to assess how language models handle real code and bug-fixing scenarios.

This is a fully remote, part-time assignment requiring 20+ hours per week. The role is available to candidates based in India, Pakistan, Nigeria, Egypt, Ghana, Bangladesh, Turkey, or Mexico, and pays $100 per task.

  • Role type: Remote contractor, part time
  • Assignment length: Three months
  • Time requirement: 20+ hours per week
  • Payment: $100 per task
  • Working language: English

What You'll Do

  • Analyze and triage GitHub issues across open-source libraries.
  • Configure repositories and Docker-based development environments.
  • Run, modify, and test real codebases locally.
  • Evaluate unit-test coverage and test quality.
  • Assess LLM performance on software engineering and bug-fixing tasks.
  • Collaborate with researchers to identify repositories and issues that provide meaningful challenges for LLMs.

Requirements

The structured role level is entry level, while the assignment specifically requires at least three years of software engineering experience. Candidates should be comfortable working independently with complex public codebases and evaluating both software quality and LLM performance.

  • At least three years of software engineering experience.
  • Strong Python software engineering experience.
  • Proficiency with Git, Docker, and basic software pipeline setup.
  • Ability to understand and navigate complex codebases.
  • Experience running, modifying, and testing real-world projects locally.
  • Judgment in evaluating unit-test coverage, test quality, and LLM bug-fixing performance.
  • English proficiency for technical evaluation work.

Preferred Experience

  • Experience contributing to or evaluating open-source projects.
  • Previous LLM research or evaluation experience.

Why Build an AI Training Career With OpenTrain

AI training and data-labeling work gives people a way to contribute directly to state-of-the-art AI systems. Many projects are remote and flexible, while specialized technical assignments let experienced professionals apply their existing expertise to emerging model-development work.

Through OpenTrain, you can build a profile that showcases your AI training experience, discover projects aligned with your skills, and develop a longer-term portfolio in this fast-growing field.

  • Work remotely with a flexible weekly commitment.
  • Apply software engineering expertise to advanced AI evaluation.
  • Grow experience in LLM testing, code assessment, and AI training.

How to Apply

Create or use your free OpenTrain account, review the assignment details, and submit your application. Make sure your profile reflects your Python, Git, Docker, software engineering, and code evaluation experience.

  • Confirm that you are based in an eligible country.
  • Highlight experience with complex codebases and real-world testing.
  • Apply through OpenTrain in minutes.
Back to blog

Other Jobs To Apply

No other job posts for this day.

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...