Dartmouth may have been hurt by a "special bias" in the college rankings that Forbes.com released last week, according to Ohio University economics professor Richard Vedder, who devised the research methodology. This potential bias stems from the prevalence of internal faculty evaluations and the relatively small number of students at the College who use Ratemyprofessors.com, which is a heavily-weighted factor in the analysis. Dartmouth, the Ivy League institution with the lowest ranking, placed 127th.
"We just assumed there was a generalized use of [the web site]," Vedder said.
College Provost Barry Scherr responded to the news on Monday, calling into question whether the five criteria used in the rankings are good indicators of quality and are weighted appropriately. The criteria include the number of alumni listed in "Who's Who in America," student evaluations on Ratemyprofessors.com, four-year graduation rates, the number of alumni and faculty receiving national awards and the average amount of debt students accumulate over four years.
"My general response would be that the methodology does not seem terribly valid in a lot of different ways," Scherr said.
Scherr pointed to Ratemyprofessors.com, explaining that the site has 144 Dartmouth professors listed, but many of them do not have evaluations. While one professor has 18 student assessments, most have fewer than five.
"I don't believe that is statistically significant," Scherr said, adding later, "Since Dartmouth has its own internal system of doing course evaluations, it seems Dartmouth students don't particularly use the Ratemyprofessors site."
Out of the 569 colleges and universities in the study, Dartmouth had the 23rd lowest rate of participation on Ratemyprofessors.com relative to the size of the student body, according to Jim Coleman, a research associate at the Center for College Affordability and Productivity in Washington, D.C., who worked on the rankings.
Vedder said Ratemyprofessors.com ratings proved to be the most damaging category for the College. Dartmouth could have been in the top 10 or 15 if the teaching ratings had been more in line with the College's peer institutions, he said, given that Dartmouth was ranked at or near the top in terms of listings in "Who's Who."
The researchers used Ratemyprofessors.com because it appeared to be representative of student sentiment nationally, Vedder said, as the site has over seven million evaluations.
"Maybe you have unusually low participation and an unusually biased sample within Ratemyprofessors.com in that the only [students] who fill it out are the disgruntled ones," Vedder said. "There is nothing perfect in the world."
James Felton, an economics professor at Central Michigan University who has studied the accuracy of Ratemyprofessors.com, said the low number of ratings for Dartmouth professors makes it is difficult to make statistically significant conclusions. His research relies on schools that have a few thousand ratings, while Dartmouth has about 230, he said.
Felton was unsure how the researchers responsible for the Forbes.com analysis could compensate for different levels of usage. A "bigger question," he said, is whether Forbes.com should use Ratemyprofessors.com at all.
"We could see that students are biased by the easiness of the class and the attractiveness of the professor," he said.
Several economics professors at the College agreed with Felton's assessment.
"There are a total of 29 evaluations of Dartmouth economics professors from the thousands of students who have taken classes in the economics department since [Ratemyprofessors.com] was launched," professor Elizabeth Cascio said in an e-mail. "These 29 evaluations do not appear to be representative: Many are 'blue sad faces,' even for professors which have good reputations on campus and by end-of-term student evaluations. Even if this sample were representative, it is also much, much smaller than statisticians would generally feel comfortable using to arrive at strong conclusions."
There is no reason to conclude, however, that the low level of participation at Dartmouth would introduce a damaging bias, Coleman said.
"There are schools with small participation rates that actually scored fairly well," he said.
There could be bias, he explained, if Dartmouth's evaluation procedures and culture were vastly different from other colleges, but this is not necessarily the case. Dartmouth currently requires students to complete an online evaluation or opt out to see their grades. Many departments also have in-class evaluations, and some students contribute to the Student Assembly Course Guide. Several of the College's peer institutions also have online evaluation systems, although most lack the redundancy present at Dartmouth. Yale University, like the College, requires students to complete an evaluation to access grades.
"In the data there is no variable on how universities internally evaluate professors," Coleman said. "There would be no way to get at that statistically and make adjustments in the data."
Although the College did well in the "Who's Who" category, faculty and administrators again questioned whether it is an appropriate measure for college rankings. Political pundit Tucker Carlson wrote in a 1999 article in Forbes magazine that gym teachers and small business owners have made their way onto the "Who's Who" list even though they may not be worthy of such a distinction. In addition, many influential individuals refuse to provide the publisher with biographical information. Vedder admitted that he has received requests from "Who's Who" and has never responded.
Forbes released the information as an alternative to the annual U.S. News and World Report analysis, according to Michael Noer, executive editor of special projects at Forbes.com. Dartmouth was ranked 11th in last year's U.S. News rankings.
"The U.S. News [ranking] has its own problems, but this one is far from solving those problems," Felton said.
