In human society, punishment can sometimes teach and enforce social norms of behavior, but other times backfires and undermines the authority's legitimacy. These seemingly contradictory effects of punishment can only be understood by considering the cognitive processes in the minds of human observers of punishment. One challenge is that in real situations, participants bring strong priors about every element of a punitive setting. Our experiments therefore use vignettes about hypothetical societies to measure what adults and children learn from observing punishment, with experimental control over all of the priors. A formal cognitive model, derived from a standard model of how peoplemake sense of one another’s actions (Inverse planning for Theory of Mind) precisely predicts people’s judgements. Our results show that polarized interpretations of punishment arise rationally. We also measured and modeled the effects of ideological authoritarianism on interpretations of punishment the model predicts that individual differences in authoritarianism may persist rationally and even deepen as people observe authorities using punishment. Our model illuminates a central tension faced by any authority, from university leaders to parents of toddlers: how the same punitive choice can communicate social norms to some people, yet cause loss of legitimacy in the eyes of others.