Wenyue Hua, Zachary Huang, Tyler Payne, Safoora Yousefi, Saleema Amershi, Asli Celikyilmaz

Increasingly, we ask AI agents to act for us: to schedule our meetings, compare offers, settle the terms of a deal, or haggle in a marketplace. We want them to represent our interests against a counterpart who has their own. However, the qualities that make an assistant pleasant to talk to, such as agreeableness, transparency, an eagerness to find common ground, become liabilities at the bargaining table: an agent that always says yes makes a pleasant conversational partner but a poor delegate to fight for a principal’s interests.  

For example, even frontier models give away the game before anyone pushes. In negotiation, when each side is expected to guard private item values it ought to guard, models routinely announce them by opening with lines like “hats value 3 to me,” telegraphing exactly where to squeeze. And when resistance does come, they retreat: a greedy opening, met with a single rejection, collapses into surrendering the contested items. The result is an agent that’s pleasant to negotiate against and easy to beat.

Social reasoning is the missing skill: the ability to model what the other side wants, decide what to reveal, and steer toward an outcome that serves the principal. This post describes SocialRL, our effort to train that capability directly, and how far we can push a 4-billion-parameter small language model. Our early exploration has shown that we are able to push its negotiation capability in the range of or even surpass frontier models.

What social reasoning actually requires

We decompose agent social reasoning into four interlocking capabilities:

• Intent modeling: inferring what the counterpart wants and what constraints they face. When an opponent fights hardest over books, that’s data: books are worth more to them than they’re letting on.
• Information disclosure: deciding what to disclose and what to keep private. Instead of announcing “books value 3 while laptops value 5 to my user”, let the other side reveal where they’re flexible first.
• Strategic pushback: answering a lopsided offer with a counter, not always a direct concession. A single rejection shouldn’t collapse your opening position; hold the line and make them move.
• Value-based bargaining: trading away what you value little for what you value a lot, so the final split is good for your principal, and ideally Pareto-efficient.

Learning social reasoning from games

SocialRL trains social reasoning capability in multi-agent interactive game environments, where every interaction has feedback, and we use reinforcement learning to turn those consequences into better policy.  It translates the abstract goal of “faithfully represent your principal” into concrete, robust behavior against real counterparts.

We train by games for three reasons. They are diverse: spanning bargaining, cooperation, and competing preferences. They are scalable: each game is a template that spawns endless instances. A fresh draw of hidden values in Deal-or-No-Deal or a new listing in Craigslist is a new scenario, so we generate as much training data as we need. In addition, the lightweight logic runs in-process with no sandbox to maintain. They are scorable: each game ends in a concrete number, whether points captured or a final price, so every rollout yields a reward to learn from. Notice that this reward is a proxy-based reward: unlike math or coding, social reasoning has no ground-truth answer to check against, so the score is only meaningful relative to the opponent the agent faced, i.e. beating a weak opponent and a high score proves nothing. So we fix the yardstick: agents play against strong frontier models, which keeps a given score comparable from one rollout to the next and ties it to real social-reasoning skill rather than a lucky matchup.

Four games anchor the work:

• Deal-or-No-Deal: two agents divide a shared pool of items including books, hats, balls, each valuing them differently and in private. Doing well means inferring what the other side cares about, claiming what is worth most to you, and conceding the rest.
• CaSiNo: two campers split a fixed stock of firewood, food, and water. Each has a private priority order over the three justified by a backstory and argues to secure more of what matters most.
• Job Interview: a candidate and a recruiter bargain over a five-issue employment package: salary, company, vacation, location, and job title. The issues carry different weight for each side, so the best agreements come from trading away low-priority issues to win high-priority ones.
• Craigslist: a buyer and a seller haggle over the price of a single listed item. Each holds a private target price, and the final number lands wherever the back-and-forth of offers and concessions pushes it.

Training infrastructure

Our training infrastructure is fully decoupled: environment, agent harness, inference engine, and trainer are independent layers behind clean interfaces, such that each can be swapped on its own. This design brings us three benefits: (1) It speeds up development, since each layer can be built, tested, and changed by itself, so adding a game or switching RL algorithms is a local edit rather than a stack-wide rewrite. (2) It maximizes reuse, since every layer is task-agnostic, so one harness drives any agent and one trainer serves every game instead of a fresh pipeline per task. (3) It enables ecosystem-level flexibility, since components depend only on interfaces, so they can be scaled independently, and run on heterogeneous hardware, even in setups like training for black-box agents.

Training recipe

Our recipe is simple: optional supervised distillation to give the policy reasonable priors, then PPO to turn those priors into robust behavior. How much distillation we need depends on the game. For Deal-or-No-Deal, CaSiNo, and Job Interview, PPO works from the base model directly, so distillation is optional. Craigslist is the exception: PPO from a cold start never gains enough traction, so we first distill from Qwen3-30B-A3B and then run PPO on top.

Reward design has one subtlety: scenarios differ in difficulty, so a high raw score can mean the agent played well or simply drew an easy setup. In Deal-or-No-Deal, when two players want different items, both score high just by taking what they want; when they want the same items, only sharp negotiation yields a good outcome. To correct this, each game grades the agent against a per-scenario reference point, the Pareto-efficient, envy-free division an agent could have reached, and shapes reward around that bar. The result reflects skill rather than a lucky draw and stays comparable across games of very different stakes.

We train with PPO and a learned value function rather than group-relative methods like GRPO. GRPO’s baseline normalizes reward within a group of trajectories from the same prompt, which assumes they are comparable; against a stochastic opponent they are not, since each rollout faces different replies, so a high return can reflect a lucky draw rather than better play. A learned critic avoids this: it conditions on the actual state, including what the opponent has revealed so far, and supplies a per-state baseline rather than a per-group average.

A 4B model reaches frontier negotiation capability

Here is the part that surprised us: a 4B model after in-domain training can reach frontier negotiation capability by matching, and in places beating, much larger models across the GPT-5 family on held-out scenarios (10 unseen scenarios × 3 opponents × 5 trials).

Negotiation reward of the 4B SocialRL agent vs. frontier models across the four games (held-out eval). “SLMs close the capability gap.”

What did it actually learn?

The aggregate number is satisfying, but the trace-level behavior is where social reasoning becomes visible. Consider a single Craigslist negotiation over a vintage drafting lamp listed at $75:

The baseline agent opens by announcing its own target: “my target is $45, so I’m offering that”. It then concedes upward under light pressure, and accepts $72, far above where it wanted to land. It is transparent, and therefore exploitable.

The SocialRL agent plays a different game entirely. It opens with a low anchor, invokes outside options such as “I’m looking at three listings”, applies a deadline such as “only if we close today”, states a credible ceiling, and walks the counterpart down to a close near its own target.

Baseline vs. SocialRL traces on the Luxo lamp task — caving above target vs. anchoring, outside options, deadline pressure, and a credible walk-away.

This is not a one-off. Measurable shifts appear across games:

• It anchors the whole trajectory, not just the final number. 78% of trained openings start below target, versus 3% untrained.
• It diversifies its actions. Blind proposing drops sharply (~45%) and is replaced by a mix of messaging, accepting, and rejecting at the right moments.

Does social skill transfer?

We hoped social reasoning learned in one game would carry to others. The honest answer is: unevenly. Deal-or-No-Deal and CaSiNo form a natural transfer pair. Job Interview is a generous donor that lifts performance elsewhere. Social skills generalize to out-of-domain environments that are configurably similar.

Reading the opponent’s mind

If an agent is negotiating on your behalf, it is fair to ask what it is thinking. We instrumented the agent with explicit theory-of-mind: each turn it infers the opponent’s preferences, takes an action, and predicts the reply, forming the three steps of Infer → Act → Anticipate.

We distilled Qwen3-4B from GPT-5.2 on the four games under two supervision formats: (1) a standard prompt, and (2) a ToM-reasoning prompt carrying the explicit Infer → Act → Anticipate annotations, then evaluated both on two domains the model never trained on: Marketplace and Calendar. The ToM format generalizes markedly better. Trained across the mixed set, it scores 0.399 on held-out Marketplace and 0.535 on Calendar, against 0.226 and 0.317 for standard SFT, a relative gain of roughly 77% and 69%. On Calendar the ToM format matches or beats standard supervision from every training source; on Marketplace the picture is less uniform, standard SFT transfers better from a Craigslist-only source, but the mixed-source result is decisive.

ToM-reasoning-prompt SFT transfers to held-out domains (Marketplace, Calendar) better than standard-prompt SFT; the gap is widest when distilled across the mixed game set.

The reason is structural. An ordinary trace supervises only the observable move and leaves the opponent model latent, to be reconstructed for free from outcomes. The ToM format promotes that latent variable to a token-level target, i.e. naming the opponent’s likely values and constraints, acting on the estimate, forecasting the reply, which is denser supervision than the action alone, with a closing forecast that is falsifiable against what the opponent actually does. Representations learned that way travel: they generalize to unseen social tasks instead of memorizing the surface form of the training games.

In addition, we also probe which of these correlates with success gave a sharp result. Anticipating the opponent’s next move correlates positively with reward in all four games (average +0.19). Modeling the opponent’s hidden preferences/beliefs barely correlates at all (average +0.03). Put plainly: acting on what they will do helps; scoring what they prefer does not.

Correlation of each theory-of-mind probe with final negotiation reward, per environment. Next-action prediction tracks reward; preference (belief) modeling is near zero.

The road ahead

The larger aim of training LLMs to have social skills is principal-aligned agents: delegates you can trust to represent you faithfully when the other side has an agenda of its own. Getting there means agents that don’t just talk about strategy, but act on it.

Opens in a new tab