College Rankings (continued)

Criticisms of rankings publications have proliferated as scholars, college administrators, and higher education researchers address what they perceive as methodological flaws in the rankings. After reviewing research on rankings publications, Stuart (1995) identified a number of general methodological problems: 1) Rankings compare institutions or departments without taking into consideration differences in purpose and mission; 2) Reputation is used too often as a measure of academic quality; 3) Survey respondents may be biased or uninformed about all the departments or colleges they are rating; 4) Rankings editors may tend to view colleges with selective admissions policies as prestigious; and 5) One department's reputation may indiscriminately influence the ratings of other departments on the same campus (pp. 17-19).

U.S. News and World Reports "America's Best Colleges"

The most specific criticism has been directed against U.S. News and World Report's, "America's Best Colleges," published since 1990 and the most popular rankings guide. Monks and Ehrenberg (1999) investigated how U.S. News determines an institution's rank, basing their study on statistics from U.S. News' 1997 publication. They found that U.S. News takes a weighted average of an institution's scores of in seven categories of academic input and outcome measures as follows: academic reputation (25%); retention rate (20%); faculty resources (20%); student selectivity (15%); financial resources (10%); alumni giving (5%); and graduation rate performance (5%) (Monks and Ehrenberg, 1999, p. 45). These categories were further divided and 16 variables were used as measurements. McGuire (1995) asserts that the variables U.S. News uses to measure quality are usually far removed from the educational experiences of students (McGuire, 1995, p. 47). For example, U.S. News measures the average compensation of full professors, a sub factor of the faculty resources variable mentioned above. McGuire argues that this variable implies that well-paid professors are somehow better teachers than lower-paid professors-an implication unsupported by direct evidence.. He says that "In the absence of good measures, poor measures will have to suffice because the consumer demand for some type of measurement is strong and the business of supplying that demand is lucrative" (McGuire, 1995, p. 47). Along the same lines, Hossler (2000) believes that better indicators of institutional quality are outcomes and assessment data that focus on what students do after they enroll, their academic and college experiences, and the quality of their effort (p. 23).

Monks and Ehrenberg (1999) found that U.S. News periodically alters its rankings methodology, so that "changes in an institution's rank do not necessarily indicate true changes in the underlying 'quality' of the institution" (p. 45). They contend note, for example, that the California Institute of Technology jumped from 9th place in 1998 to 1st place in 1999 in U.S. News, largely due to changes in the magazine's methodology (Monks and Ehrenberg, 1999, p. 44). Ehrenberg (2000) details how a seemingly minor change in methodology on the part of U.S. News can have a dramatic effect on an institution's ranking (p. 60). Machung (1998) states that "The U.S. News model itself is predicated upon a certain amount of credible instability" (p. 15). The number one college in "America's Best Colleges" changes from year to year, with the highest ranking fluctuating among 20 of the 25 national universities that continually vie for the highest positions in the U.S. News rankings (Machung, 1998, p. 15). Machung asserts that "new" rankings are a marketing ploy by U.S. News to sell its publication (1998, p. 15).