project writeup · october 2026 · 5 min read

KnightGym: an order-book RL environment, and how two bots can game its reward

I took the matching engine from the Rutgers live trading competition, wrapped it as a Gymnasium and PettingZoo environment, and audited the reward. Under the competition’s rules, two teammate bots can raise their marked equity by $1,077 an episode while actually losing money.

Aryan MalikOctober 2026Rust · PyO3 · Gymnasium · PettingZoo · Stable-Baselines3

Code

The hole

Reward hacking in trading environments is usually about the accounting, not the model. In the competition this engine was built for, holdings were marked at the last traded price, and self-trade prevention keyed on the individual bot rather than the team. So two teammates can trade with each other in a thin asset at an absurd price. Nobody loses money on the trade, the last price jumps, and the team’s marked equity goes up. Any RL environment that pays “change in marked equity” inherits the same hole.

What I built

The engine. The competition’s order book, with the web and database layers removed and the matching code unchanged. On one thread it matches 5.5M resting limit orders a second (p99 0.85µs). The live competition ran at about 420 orders a second, limited by the bots and the API.
A deterministic simulator in Rust on top of it, exposed to Python with PyO3. Time only advances in step(), and a seed fixes the fair-value walk, noise flow and action ordering. The full Python loop runs about 80k steps a second. A house market maker requotes every step and can pick off stale agent orders.
The environments. A Gymnasium env (book and inventory in, limit, market, cancel and two-sided requote orders out) and a PettingZoo env for teams, with switchable marking (last, mid, VWAP) and self-trade prevention by bot or by team.

The audit

Two teammates each hold 100 units of a thin asset, over 100 paired seeds against a no-attack baseline. “Marked” is what the reward pays. “True” marks inventory at the hidden fair value.

Bar chart of marked versus true gain for teammate wash trading under six combinations of marking rule and self-trade prevention, from +1077 marked under competition rules to −297 marked and −298 true when patched — **Wash trading under each rule set.** Blue is what the reward pays, orange is what actually happened. Only team-level self-trade prevention plus mid marking makes them agree.

Neither half of the fix works alone. Mid marking alone ignores the wash prints, but team self-trade prevention alone still leaves +$372, because the house lifts the attacker’s leftover bid at the inflated price. VWAP marking is weak because wash volume swamps real volume in a thin asset. After the full patch, two smaller leaks remain: a lone agent can buy out the few real asks to push the last price (+$48 under last or VWAP marking), and quote spoofing under mid marking is bounded by half the spread.

Training an agent

PPO over 2M steps made $2.10 per 200-step episode on 50 held-out seeds, against $1.80 for a scripted market maker and −$26.80 for random. With three training seeds I wouldn’t claim it beats the market maker, and it holds about four times the inventory. Its first runs collapsed to doing nothing. A combined cancel-and-requote action wasn’t enough on its own; reward normalization and more steps were what got it trading. I tuned hyperparameters once after seeing eval results on the same seeds, so treat the PPO number as mildly optimistic.

Limits

The market is stylized: a random-walk fair value and synthetic flow, nothing calibrated to real data.
The competition’s marking and self-trade rules come from my notes, not from re-reading the live leaderboard code.
Next: does a learned attacker find the wash trade on its own in the team environment? And price collars as a third patch.

The repo has the benchmark method (results/benchmark.md), the full attack table (results/hack_results.md) and tests that check every claim above.