******************************** From the New York Times [NYTimes.com], Monday, May 21, 2001. See http://www.nytimes.com/2001/05/21/business/21EXAM.html ; ------------------------------------ This is Part II of two parts - continued from Part I ******************************** The Results: School Districts Cope With Falling Scores While CTB stepped up its inquiry, its clients were dealing with the consequences of the test results they had been given. In Tennessee, the adjusted results were not distributed to teachers and principals until late summer, too late to play their customary role in many districts' decisions on summer school or student promotion. In Indiana, the districts' very public concerns about the accuracy of the scores led teachers and principals to be wary about how much stock, if any, to put in those numbers. And so, educators there grew reluctant to use the test results to shape their lesson plans. Nevada had voiced similar concerns to CTB. But state education officials nonetheless moved forward, branding a handful of schools as "inadequate" based on their poor scores. One of them was Cambeiro Elementary, in the shadow of the Las Vegas strip, which was put under the supervision of a state oversight panel and awarded over $100,000 for remedial programs. School administrators felt more than a little humiliation. "At bowling night and at church," Cenie Nelson, the school principal, said, "teachers were asked by other teachers and friends, `Why would you want to be associated with a school not doing a good job?' " But nowhere did CTB's scores have more impact than in New York City. Based solely on their performance on the test, Dr. Crew immediately ordered nearly 40,000 third and sixth graders to attend summer school. "Your child must attend summer school," the superintendent in one district wrote to parents. "We feel that your child would benefit from this enriching experience." Two weeks after releasing the test results, Dr. Crew took direct control of 43 failing schools, saying he intended to fire many of their principals. He also fired or eased out 5 of the 32 superintendents who preside over the city's neighborhood school districts, citing their failures as leaders as well as their students' test scores. One of them was Robert Riccobono, then 54, who had brought rigorous literacy programs to one of the poorest districts in the city, No. 19 in East New York, Brooklyn. After four years as superintendent, Mr. Riccobono says, his efforts were starting to bear fruit when Dr. Crew fired him. "Giuliani was talking tough," Mr. Riccobono said. "Crew felt the need to find victims." The day after Dr. Crew announced his firing at a news conference broadcast live on local television, Mr. Riccobono attended his son's graduation from high school. "I felt singled out and embarrassed," said Mr. Riccobono, who had known teachers at the school for a decade. "I was wondering where I had gone wrong." The Inquiry: An Error Is Found Deep in the Software While New York City was firing administrators and disrupting the summer vacations of students and teachers, CTB was closing in on evidence that would undermine those very decisions. The company's focus was again on the equating process, which allows test scores to be compared year over year. As it turned out, CTB - despite its assurances to Indiana and others - had done an incomplete job of reviewing test data. When a much larger sample was reviewed, a programming error surfaced. The error had - erroneously - made the current test appear easier than the previous year's. To make the tests equal in difficulty, the computer had then compensated by making it harder for some students to do as well as they had last time. The error did not change students' right and wrong answers, but it did affect their comparative percentile scores. On July 20, Wendy Yen, then the vice president of research for CTB, walked into the office of Mr. Taggart, the company president, and announced, "We have found something." Mr. Taggart decided not to tell schools just yet about the problem, because, he says, he did not yet know how bad it was. "Would it be a positive impact, a negative impact, no impact?" Mr. Taggart said. At the time the company found the error, New York City's students were just two weeks into a month-long summer-school program, sweltering in a heat wave. Even classrooms with air-conditioners routinely registered 90 degrees on indoor thermometers. Dr. Crew would later say that had he known what CTB knew - no matter how tentative - "we could have corrected the action midstream, and not put families through all that torment." A month later on Aug. 24, after summer school had ended, Mr. Taggart traveled to New York City to hear, in person, the city's lingering concerns about the spring results. "We're the largest school system in the country," Dr. Crew recalls saying. "You have got to get this right with us." Again, Mr. Taggart promised to look into the city's complaints. And again, he did not tell them what he knew about the error. Mr. Taggart had more to say when he called Mr. Tobias, the city's testing director, on the first day of school, Sept. 9, 1999. "We have done further analysis into your concerns about the scoring," Mr. Tobias recalls being told. "And we have found a problem." "It's a small problem," Mr. Tobias remembers the company president saying. "We don't believe it's going to have a huge impact on your scores." Mr. Tobias quickly did a few calculations of his own. It seemed, at first, that 3,000 students who had been sent to summer school in June had in fact scored well enough to have spent the summer as they wished. That number eventually grew to nearly 9,000 - almost a quarter of the mandatory summer-school roster. So much for "a small problem," Mr. Tobias thought. But the real shock came when school officials learned what the corrected test scores meant for the entire city. Instead of reading scores stagnating over all, the citywide average had actually risen five percentage points - a substantial jump, particularly for an urban school district. "I was feeling really horribly," Mr. Tobias said. "I realized that what was a bad story last spring really could have been a triumph for the chancellor." Dr. Crew agreed. "You've got the mayor and the political people saying you haven't done a damn thing," Dr. Crew said. "This was the beginning of the end for me. You can't go back and retrieve this." The following week, Mr. Taggart flew back to New York City to tell a packed meeting of the New York City Board of Education that he was sorry. His voice shaking, Mr. Taggart said that CTB had "worked diligently" to find the problem, and had notified New York "as soon as those calculations were complete and verified." Mr. Taggart also said it was not his company's idea to use CTB's test to decide who had to go to summer school. Even so, he said, "The test itself remains a valid measure of student performance." William C. Thompson Jr., the president of the board, was disbelieving. "Why would I use your company after this?" he asked. Two days later, Mr. Taggart appeared at the Indiana Board of Education, where he told a similar story and received a similar reception. It was his second trip to Indiana in six months, and he was armed with his company's third version of that state's test scores. But this time, the corrected percentile scores virtually eliminated the unexplained drops that had troubled Mr. Kline, the Fort Wayne testing director. "It was just good to know we were right," Mr. Kline said. Mr. Taggart did not travel to Nevada, but he called testing officials there. Careful readers of The Las Vegas Sun on Oct. 20, 1999, may have noticed the headline, "Cambeiro Elementary School Taken Off Academic Probation by State." When CTB recalculated the results of the Nevada tests, students at Cambeiro, and another school, in Reno, were found to have exceeded the state's criteria for the label "inadequate." They were, in fact, "adequate." The school was no longer entitled to the more than $100,000 in remedial money it had been given, but the money had already been spent. A cloud had lifted, but it was hard for the school to tell. "You can't undo an 'inadequate,' " Ms. Nelson, the school principal, said. "It's not something that goes away." CTB also called Tennessee, with word that it could finally explain the unexplainable dips in its rankings. Now the company could actually correct the percentile scores, rather than simply adjust them to meet what Professor Sanders thought they should have been. The Future: Most School Districts Have Few Options When Mr. Tobias first learned of the error, he says, he asked Mr. Taggart if any districts outside New York had been affected. Mr. Tobias was told that was proprietary information. The press release issued in New York, written by CTB's parent company, McGraw- Hill, mentioned only New York. And a release issued the same day in Indiana referred only to Indiana. While the company has since confirmed that in addition to Tennessee and Nevada, two other states were affected - Wisconsin and South Carolina - it has refused to identify two other school districts involved, or to say whether the districts ever alerted teachers and parents to the error. Subsequent audits by Indiana and New York criticized CTB for lax supervision in the research department - the department that had created the error, and then was charged with finding and correcting it. The auditors wrote that managers were only "informally involved in the day-to-day work of subordinates." Wendy Yen, the CTB official who oversaw the research department, has since left the company to work for the Educational Testing Service, which administers the SAT. Dr. Yen, through a spokesman at her current company, refused several interview requests. But Benjamin Brown, Tennessee's testing director, said the problem went beyond research: he said CTB's greatest error was in treating each customer as if its problem was isolated, even after the company knew otherwise. "It'd be like someone holding a barking dog and saying, `This dog won't bite,' knowing he's bitten three neighbors in the previous month," Mr. Brown said. Mr. Taggart said the company had since installed new quality controls to intercept such an error, and had put its employees through a customer relations course. The New York City Board of Education voted to renew CTB's contract despite its record, although the board did negotiate financial penalties that totaled $500,000 on a multimillion-dollar agreement renewable over four years. Dr. Crew supported retaining CTB because the city had already spent years working with the company to create tests specifically designed for city students. Also, CTB's competitors had experienced their own quality control problems. "There was no place else to go," Dr. Crew said. Dr. Crew did not fare as well as CTB. On Dec. 23, 1999, a board majority led by the mayor's appointees voted not to extend his contract, saying that after four years he had lost interest in his job. Though he lamented that no one noticed the city's vastly improved scores, Dr. Crew refused to rehire the superintendents and principals whom he had fired, saying their problems went beyond bad test scores. But New York State's education commissioner, Richard Mills, disagreed, at least in the case of Mr. Riccobono, the innovative superintendent from Brooklyn. Mr. Mills is taking steps to help Mr. Riccobono, who teaches part time at New York University, get his old job back. "I suppose I felt vindicated," Mr. Riccobono said. "I am certain that had the correct scores been reported initially, I wouldn't have been fired." But he says he still bears emotional scars from the experience. After his firing, he applied for at least 30 other superintendent jobs in New York State - and did not get one of them. "Clearly standardized tests are a valid way of providing part of the picture," Mr. Riccobono said. "But they should not be the ultimate determinant of success." New York City now uses multiple measures - teacher evaluations as well as test scores - to make summer-school assignments. Indiana's contract with CTB expires this year, and the state is soliciting bidders. For the first time, the state is requiring bidders to list all errors made over the last two years and to promise, if hired, to disclose any new errors immediately. The superintendent of education, Suellen Reed, has said she would consider rehiring CTB, particularly if it was the low bidder. But officials in Fort Wayne are not awaiting the outcome. Mr. Kline and his superintendent, Thomas Fowler-Finn, have instead written their own tests for the district's students, to be administered in grades 3 through 9. "I still believe in standardized testing," Mr. Kline said. "I just don't think the industry is ready to give us the tests we need." ****************************************** -- Jerry P.Becker Department of Curriculum & Instruction Southern Illinois University Carbondale, IL 62901-4610 USA Phone: (618) 453-4241 [O] (618) 457-8903 [H] Fax: (618) 453-4244 E-mail: [EMAIL PROTECTED] ---------------------------------------------------- This is the CPS Mathematics Teacher Discussion List. To unsubscribe, send a message to <[EMAIL PROTECTED]> For more information: <http://home.sprintmail.com/~mikelach/subscribe.html>. To search the archives: <http://www.mail-archive.com/science%40lists.csi.cps.k12.il.us/>
