June 25, 2003
IOWA CITY, Iowa (AP) --Hunkered down in a windowless conference room, five scholars analyzed a high school student's essay with a scrutiny normally reserved for the likes of Hemingway or Dickens.
"We need to be more forgiving about the endings," counseled Roseanne Cook, a program manager for the ACT, examining the handwritten pages. "We need to recognize what happens when they run out of time, or what happens when the proctor says there are only three minutes remaining."
Cook and the ACT certainly know something about deadlines.
The clock has been ticking down for the testing service since last year, when the ACT announced that it would add an optional essay to the second-most popular college entrance exam in the nation starting in spring 2005.
Since then, test-makers on ACT's leafy campus have been working on the painstaking task of crafting precise, thoughtful questions that will inspire great writing from America's teenagers. The researchers also are figuring out how to grade an exam for hundreds of thousands of students that's not multiple choice.
"What we're interested in is measuring the writing skills," said ACT president Richard Ferguson. "Do the sentences flow logically? Do they hang together?"
A similar process is under way in Princeton, New Jersey, where the Educational Testing Service, which produces the SAT entrance exam for the College Board, also is preparing an essay to be introduced in 2005.
Pressured to expand the scope of its exam by the biggest public higher education system in the nation, the University of California, the SAT announced in June 2002 that it was adding an essay. The ACT announcement came two months later.
While the ACT decided to provide students with an option to forgo the essay -- because many colleges already require a writing sample with their application -- the SAT essay will be mandatory.
The ACT is unsure how many of the 1.1 million high school students who take its test each year will decide to write an essay. A survey found that, so far, about half of the colleges that accept the ACT have said they will not require students to take the writing assessment.
General interest essay topics
To make sure it's ready for students who do exercise the option, hundreds of
ACT employees and outside consultants are immersed in a labor-intensive process
that includes reviews of subject matter and drawn-out debate on the phrasing
of essay questions -- known as prompts.
"We've never had the perfect item," said Sherri Miller, ACT's director of elementary and secondary school measurement and research. "But we get pretty close."
David Duer, a senior language arts specialist, noted the change in going from a multiple choice exam to an essay: "It's a whole different experience to sit down with the writing. You get to understand who they are and what they are thinking."
To put all the students at the same starting point, the ACT is identifying general interest essay topics relevant to teenagers of every religion, race, ethnicity and state in the union.
"The bottom line is, whatever we wind up with needs to be a prompt that is clear to all students, fair to all students and gives each student a chance to show what they've learned in school," said Cyndie Schmeiser, the ACT's vice president of development.
Company officials stressed that students will not be asked to elaborate on religion, politics, cataclysmic events or personal issues. So, for example, the essay will not be a forum for students to share their views on the September 11 terrorist attacks.
"You certainly don't want to create a situation where a student will be distracted by a strong emotional reaction to a prompt," Duer explained.
Once the subject for a question is chosen, the fine-tuning begins.
Review process
Each of the ACT's multiple choice questions is subjected to a 21/2-year review
process that assesses content, phrasing, fairness and bias.
Similar benchmarks will be applied to the writing sample.
"Nobody in the country knows that we sometimes spend up to 12 hours working on a single word," Miller said. "But if my son was taking the test, I wouldn't want that one word to be vague."
Working from a pile of essays written by students in a high school pilot program earlier this year, Cook, Duer and a trio of academics that included college professors and an ACT testing expert brought the next phase of the process to the conference room.
Members of the ACT's "range-finding team," the scholars are responsible for establishing grading guidelines that will be used for the exam. Eventually, upward of 400 professional "graders" will score ACT essays on a scale of one to six.
Although the same principles of grammar, spelling and punctuation used in the English section of the multiple-choice exam will be applied to the essay, the writing will not be scored solely on those components.
"They won't get punched out if they put an apostrophe in 'its' when they shouldn't," Duer said.
Indeed, recognizing that a 30-minute timed essay allows for a rough draft at best, ACT officials emphasize it is what the students say -- and not how they say it -- that will count the most.
"We are looking for some evidence of critical thinking with an emphasis on writing to convey that thinking to someone else," Cook said.
Settling on a score
On a recent morning at ACT headquarters, the range-finding team's goal was to
reach agreement -- among its own members -- on the grading scale: In other words,
what makes one essay worth a "two" and another worth a "six."
To do that, the team turned its attention to a pilot essay written by a student who responded to a test question, which asked whether schools should intervene to stop nonviolent negative behavior, such as teasing.
To an outside observer, the essay was grammatically correct, but seemed to lack focus. There was no strong thesis.
The team split the first time they read the effort, unable to decide if the essay rated a score or four or five. Then, page-by-page, line-by-line and, sometimes, even word-by-word, they reviewed the content again.
"I don't know, I think this paper showed very haphazard movement," said Barbara Kroll, an English professor at California State University, Northridge.
Duer also noted a lack of "coherent logic."
"It started out fine, but became repetitious and stalled," Cook said. "I didn't have a sense of what she was trying to say."
Kroll said that, despite its faults, the essay demonstrated creativity.
"When you're faced with a topic that you don't know what to do with, or how to address, the only choice is to waffle. And she waffled pretty good," Kroll said.
When it came to the vote after 10 minutes of discussion, the team didn't waffle at all; they dropped the essay's score from five to four.
Then, Cook plucked another paper from a stack of test essays and announced the name of the author. Digging through their own piles of paper, each member of the team retrieved the essay.
With furrowed brows, they began the process again.
--------------------------------------------------------------------------------
Copyright 2003 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.
Find this article at:
http://www.cnn.com/2003/EDUCATION/06/24/act.essay.ap/index.html