Google’s New Framework Puts LLM Personality Tests on the Couch

Google Research has been poking at a question that’s been nagging at me for a while: how do you actually measure whether an LLM’s “personality” aligns with how humans actually behave? Their latest paper, “Evaluating Alignment of Behavioral Dispositions in LLMs,” takes a stab at it with a framework that borrows from established psychology.

The core idea is straightforward enough. Instead of just asking a model “Are you empathetic?” and trusting its answer, they build situational judgment tests (SJTs) — realistic scenarios where the model has to choose between two courses of action. One action reflects a specific behavioral trait (say, assertiveness), the other opposes it. Then they compare the model’s choices to what a pool of 550 human annotators would do.

This is smarter than the usual self-report approach. Anyone who’s worked with LLMs knows they’ll claim whatever disposition fits the prompt. Ask a model if it’s empathetic and it’ll say yes every time. But put it in a scenario where it has to choose between comforting a friend and giving blunt advice? That’s where the real behavior shows up.

The team adapted validated psychological instruments — the Interpersonal Reactivity Index for empathy, the Emotion Regulation Questionnaire, and others. These aren’t pop quizzes; they’re peer-reviewed measures used in actual psychology research. Each was turned into an SJT scenario, reviewed by three annotators to make sure the scenarios faithfully captured the intended trait.

They tested 25 models across scenarios ranging from professional composure to conflict resolution to booking a trip. The results uncovered two kinds of gaps. First, there are cases where model behavior consistently deviates from the human consensus — the model picks the less popular option across the board. Second, there are cases where the model fails to capture the range of human opinions when consensus is absent. In other words, the model might always pick option A when half of humans would pick B.

This matters more than you might think. As LLMs move into advisory roles — recommending how to handle a workplace disagreement, suggesting how to respond to a difficult email — their behavioral dispositions have real-world impact. A model that’s systematically less empathetic or more assertive than the average human could give advice that feels off or even harmful.

I appreciate that the paper doesn’t overclaim. They call it “an early step” and highlight the opportunity for better behavioral alignment. The framework is a solid foundation, but the real work is in expanding the scenarios and testing more nuanced traits.

The pipeline itself is clean: collect statements from validated questionnaires, adapt them into SJTs, validate with annotators, test models, then compare against human distributions. It’s the kind of systematic approach that’s been missing from the “let’s just ask the model how it feels” era.

One thing that caught my attention: they use an LLM-as-a-judge to map the model’s natural language response to one of the two actions. That’s a potential weak point — if the judge model has its own biases, the whole pipeline inherits them. I’d like to see more transparency around how they validated that mapping step.

Still, this is a meaningful step forward. We’ve spent years benchmarking LLMs on factual accuracy and reasoning. Behavioral alignment is a different beast entirely, and it’s good to see someone building the tools to measure it properly.