Skip to main content

Train a model to write code with TRL’s GRPOTrainer. Each training step generates a batch of completions, and every completion runs against a test suite in its own Upstash Box. The share of tests it passes is its reward.


1. Set up#

Get a Box API key from the Upstash Console.

Tested with trl 1.18 and upstash-box 0.4.


2. Define the tasks#

Each task is a prompt the model continues, a reference solution, and words a completion may not use. The harness is the program that runs in the box. It prints the function’s output for each test.

tasks.py

3. Score completions in boxes#

score keeps the function body from a completion, builds the harness, and runs it in a box with a 2 second timeout. The outputs are compared with the reference on your machine, so a completion cannot change its own grade. A completion that crashes, times out, or uses a banned word gets -1.

rewards.py

4. Train#

The pool has one box per completion in a step, 64 with these settings, so a whole batch is scored at the same time. It is created once, right before training, and reused at every step. The boxes are deleted when training ends.

train.py

TRL logs the mean reward at every step, and the checkpoints are saved to grpo-box/. The 10 steps only check that everything works. Raise max_steps to train for real.


5. Results#

A run with 500 completions per step (BATCH_SIZE = 20, ACCUMULATION_STEPS = 25, NUM_GENERATIONS = 50) on one 48 GB GPU, scored in a pool of 500 boxes:

StepMean rewardCompletions passing every test
10.2078 of 500
20.37172 of 500
40.85408 of 500
60.98489 of 500
81.00500 of 500

Creating the 500 boxes took 24 seconds, and scoring the 500 completions of a step took 2 to 4 seconds.


6. Customize#

  • Scale up: raise BATCH_SIZE and ACCUMULATION_STEPS to fit your GPU. The pool grows with them.
  • Add a task: add a prompt to TASKS with its function name, reference function, and banned words.
  • Harder tests: raise TIMEOUT_MS when correct solutions need more time.
  • Bigger model: change MODEL. The prompts are written for a base model, which continues the code instead of answering in chat.

Troubleshooting#

  • Every reward is -1: print run.result in score to see the error from the box.
  • bf16/gpu error: the machine has no GPU. Add use_cpu=True, bf16=False to GRPOConfig.
  • num_generations error: NUM_GENERATIONS has to divide the number of completions in a step, which is BATCH_SIZE * ACCUMULATION_STEPS.
  • Boxes left after a crash: delete them from the Upstash Console.