Predicting Within-Source Agreement in Multisource Feedback Ratings: An Examination of Characteristics of the Rater Group and the Focal Manager

Author

Christine Schrader Fernandez, The Graduate Center, City University of New YorkFollow

Date of Degree

2008

Document Type

Dissertation

Degree Name

Ph.D.

Program

Psychology

Advisor

Karen S. Lyness

Committee Members

Charles Scherbaum

Harold Goldstein

Walter Reichman

Allen Kraut

Subject Categories

Psychology

Abstract

Multisource feedback (MSF) involves gathering information about a manager's effectiveness from his or her boss, peers, and subordinates. Researchers typically average MSF ratings within rating sources (e.g., peers or subordinates), which assumes that agreement within rating sources is relatively high. However, there is little prior MSF research that has addressed the issue of within-source agreement, and the extant studies have often used inappropriate statistical techniques such as reliability indices. Moreover, this research often focuses on assessing the mean level of agreement or reliability within rating sources but has ignored the variability surrounding these indices. The purpose of the present study was to identify the predictors of agreement among peer and subordinate rater groups. Based on Kenny's (1991) weighted-average model of consensus, it was hypothesized that within-source agreement would be higher (1) for groups with higher levels of acquaintance with the focal manager, (2) for groups that were less diverse in terms of gender, race, age, and education, (3) for peers rather than subordinates, (4) for rating dimensions that raters have a high opportunity to observe rather than a low opportunity to observe, (5) for focal managers who are more extraverted, agreeable, and conscientious, and (6) for focal managers who are more effective as rated by their supervisors. These hypotheses were tested with data from 33,696 focal managers who participated in the Benchmarks® multisource feedback program. The results indicated that peers had higher agreement than subordinates. Also, there were higher levels of agreement associated with more effective managers than less effective managers. Agreement was measured using a_wg and r_wg indices. These two indices were highly similar. However, a_wg was only calculated for about one-third of rater groups because many groups were too small or the group mean was outside of the interpretable range. The implications of eliminating groups are discussed. About three-quarters of peer groups and almost two-thirds of subordinates had high levels of agreement, however, an average of 5% of peers and 10% of subordinate groups failed to agree with one another at an acceptable level. The relevance of within-source agreement for MSF administration and feedback are discussed.

Comments

Digital reproduction from the UMI microform.

Recommended Citation

Fernandez, Christine Schrader, "Predicting Within-Source Agreement in Multisource Feedback Ratings: An Examination of Characteristics of the Rater Group and the Focal Manager" (2008). CUNY Academic Works.
https://academicworks.cuny.edu/gc_etds/2235