Predicting Within-Source Agreement in Multisource Feedback Ratings: An Examination of Characteristics of the Rater Group and the Focal Manager
Date of Degree
Karen S. Lyness
Multisource feedback (MSF) involves gathering information about a manager's effectiveness from his or her boss, peers, and subordinates. Researchers typically average MSF ratings within rating sources (e.g., peers or subordinates), which assumes that agreement within rating sources is relatively high. However, there is little prior MSF research that has addressed the issue of within-source agreement, and the extant studies have often used inappropriate statistical techniques such as reliability indices. Moreover, this research often focuses on assessing the mean level of agreement or reliability within rating sources but has ignored the variability surrounding these indices. The purpose of the present study was to identify the predictors of agreement among peer and subordinate rater groups. Based on Kenny's (1991) weighted-average model of consensus, it was hypothesized that within-source agreement would be higher (1) for groups with higher levels of acquaintance with the focal manager, (2) for groups that were less diverse in terms of gender, race, age, and education, (3) for peers rather than subordinates, (4) for rating dimensions that raters have a high opportunity to observe rather than a low opportunity to observe, (5) for focal managers who are more extraverted, agreeable, and conscientious, and (6) for focal managers who are more effective as rated by their supervisors. These hypotheses were tested with data from 33,696 focal managers who participated in the Benchmarks® multisource feedback program. The results indicated that peers had higher agreement than subordinates. Also, there were higher levels of agreement associated with more effective managers than less effective managers. Agreement was measured using awg and rwg indices. These two indices were highly similar. However, awg was only calculated for about one-third of rater groups because many groups were too small or the group mean was outside of the interpretable range. The implications of eliminating groups are discussed. About three-quarters of peer groups and almost two-thirds of subordinates had high levels of agreement, however, an average of 5% of peers and 10% of subordinate groups failed to agree with one another at an acceptable level. The relevance of within-source agreement for MSF administration and feedback are discussed.
Fernandez, Christine Schrader, "Predicting Within-Source Agreement in Multisource Feedback Ratings: An Examination of Characteristics of the Rater Group and the Focal Manager" (2008). CUNY Academic Works.
Digital reproduction from the UMI microform.