Rather than get into listing the typical culprits, I want to touch on a couple things that people may overlook if they're stuck in the mindset of singular parameters.
a) Tonal balance: In general, the overall shape of the frequency response. A bit more specifically, transitioning from region to region (any area, not just the fuzzy concepts of bass/mids/treble). For most people, FR is the first (and easiest) thing to notice with headphones.
b) Distortion and phase error: harmonic and intermodular distortion, diaphragm breakup, and phase deviations are all aspects that move the reproduced sound away from the input signal in a manner that isn't (directly) related to the frequency response. Being able to decipher things in this category is more difficult than tonal changes.
c) Channel matching: mismatched sides, even by one or two dB, can create a very large change in sonic perception and is often mistaken as a poor FR or distortion.