Large language models (LLMs) are increasingly deployed as agents in high-stakes domains like healthcare where optimal actions depend on both uncertainty about the world and the utilities of different outcomes. However, their decision logic is difficult to interpret. We study whether LLMs act like rational utility maximizers with coherent beliefs and preferences. If so, those beliefs and preferences would provide a lens to interpret their behavior and inform model development. For example, models in medical settings are often too quick to draw conclusions -- is this because they hold overconfident beliefs about a patient's true diagnosis, or because they act as if there is a cost to asking questions?

We develop a framework that elicits probabilistic beliefs alongside decisions from a language model, tests whether beliefs and decisions are jointly rationalizable, and if so estimates a utility function which explains the model's behavior. We apply this framework to medical diagnosis tasks. On a cautionary note, we identify specific ways in which it is impossible for models' stated diagnosis probabilities to represent their "real" belief. Nevertheless, we show how revealed-preference analysis detects substantial regions of misalignment, where correcting models' preferences substantially improves their performance even when the accuracy of their beliefs is unchanged.