Call classifications by human observers are often subjective yet they are critical to studies of animal communication, because only the categories that are relevant for the animals themselves actually make sense in terms of correlation to the context. In this paper we test whether independent observers can correctly detect differences and similarities in killer whale repertoires. We used repertoires with different a priori levels of similarity: from different ecotypes, from different oceans, from different populations within the same ocean, and from different local subpopulations of the same population. Calls from nine killer whale populations/subpopulations were pooled into a joint sample set, and eight independent observers were asked to classify the calls into separate categories. None of the observers’ classifications strongly followed the known phylogeny of the analyzed repertoires. However, some phylogenetic relationships were reflected in the classifications substantially better than others. Most observers correctly separated the calls from two North Pacific ecotypes. Call classifications averaged across multiple observers reflected the known repertoire phylogenies better than individual classifications, and revealed the similarity of repertoires at the level of subpopulations within the same population, or closely related populations.