We demonstrate how the preference to be highly regarded by others as well as oneself can drive strong polarization and segregation. Being highly regarded by someone requires conforming to that person’s normative views. We assume that if normative views move towards one extreme, image costs increase for one type of individuals but decrease for the other type. Individuals trade off image benefits against effort costs in the short run; in the long run, normative views and peer group affiliations develop endogenously. Individuals gradually adapt their normative views to those of their peers, but more reluctantly so for views harming their self-image. Over time, people may also switch social groups, seeking peers regarding them highly. The steady state is extremely polarized and segregated: each social group consists of same-type individuals who agree on the most extreme implicitly self-serving normative view. However, if normative views are partially adopted across peer groups, not only within groups, equilibrium polarization is less extreme.