[ for more; http://www.quine-duhem.com ]
[NYTimes] July 11, 2003 Data in Conflict: Why Economists Tend to Weep By DANIEL ALTMAN Is the economy pulling itself out of a slump, or is it sinking deeper? The answer could be either, based on the data from government agencies these days. Because they measure the same things - employment, incomes and prices - in different ways, it can be hard to tell what is really happening. Yet among all the alternative gauges, some of the most widely cited data is collected in a patchwork fashion, or checked against benchmarks only once a decade. What is more, frequent changes in the agencies' methods can make comparisons with earlier trends impossible. And a few economists complain that those revisions may be motivated by politics rather than science. "This is not a customer-driven enterprise," said Daniel J. B. Mitchell, a professor of management and public policy at the University of California at Los Angeles. "It's not always clear to me that we're getting the services that we'd hope for." Many Americans have been watching data from the job market, but those figures can sometimes be misleading. The Bureau of Labor Statistics reported last week that payrolls outside farming had dropped by 30,000 in June, on a seasonally adjusted basis. In the same report, it said that the total number of employed people had grown by 251,000. Was the labor market stagnating, or booming? Neither number represented an absolute truth. The figure for payrolls comes from reports by businesses and government that are checked once a year against unemployment insurance records, which are fairly comprehensive. But the figure for employed people comes from a sampling of households that is thoroughly benchmarked only once a decade, with the census. "Particularly if you are interested in employment as a measure of economic trends, you probably want to use the payroll survey more than the household survey," Professor Mitchell said. "Things that are proportions, percentages, ratios - that's what I would say you want to use the household survey for." Thomas J. Nardone, who supervises the survey of households as chief of the bureau's division of labor force statistics, said Professor Mitchell's approach was basically correct for short periods. "You have to be careful about interpreting things, especially if you're dealing with just one additional bit of information." Even a single data point can change, sometimes more than once, over time. The Bureau of Economic Analysis, a Commerce Department agency, releases three figures for economic growth for a quarter: an advance number, a preliminary number and a final number. For the first quarter of this year, the bureau announced annualized growth rates of 1.6 percent in April, 1.9 percent in May and 1.4 percent in June. Given that the economy averages growth around 3 percent, a swing of half a percentage point is not negligible. Yet it is not uncommon, either. "That's not an unusual or atypical swing," said Brent R. Moulton, the bureau's associate director for national economic accounts. "When you're working with a small base, it seems more noticeable." Much of the quarterly data is collected in dribs and drabs, Mr. Moulton said. For service industries, which represent two-thirds of economic activity, comprehensive data is only collected annually. For more frequent figures, some statistical guesswork is involved. "We have data on volume of stock market activity, which we can use to infer something about the amounts paid in commissions for financial transactions," Mr. Moulton said. "We have data on temperature - heating and cooling degree-days - that we can use to infer something about electricity and natural gas consumption." In some sectors, the economy has grown beyond the scope of statistics, leaving the government scrambling to fill the holes. "We've got an economy that has changed rapidly over the past 15 or 20 years that our statistical infrastructure hasn't really kept up with," said Richard B. Berner, chief United States economist at Morgan Stanley and a member of the Bureau of Economic Analysis's advisory committee. Because reporting is often voluntary, he added, it can be hard to persuade new companies to contribute. They may also drop in and out. The semiconductor industry, for instance, recently agreed to share information on its shipments and inventories for the first time in about a year. But how should officials measure, say, the output of an Internet search engine? "In a knowledge-based, information-based economy, one could argue that we don't have good metrics for measuring output," Mr. Berner said. Measures of income, which the Bureau of Economic Analysis releases alongside measures of production, can also be incomplete. Mr. Moulton said the agency received fairly reliable data on wages and profit, but "for most of the other pieces of income, we don't have very timely information coming in." Among the most controversial of government gauges are the indexes of prices. These determine the growth of Social Security checks, the interest paid on inflation-adjusted bonds and the positioning of tax brackets. They also feed into data on worker productivity and real interest rates. A true change in price, said Neal Soss, chief economist of Credit Suisse First Boston, should not reflect changes in the quality of a product - essentially, the evolution of a new product. And in the services sector, discerning those distinctions can be difficult. "I don't know how you'd measure the quality of haircuts, or whether today's rock music is better than the golden oldies of the past," Mr. Soss said. "But in some parts of the service sector, there are objective measures of quality that could be applied, and I don't think that the Bureau of Labor Statistics has applied nearly enough effort to this." Adjusting prices of services for changes in quality can be especially challenging, said John S. Greenlees, acting associate commissioner for prices and living conditions at the Bureau of Labor Statistics, because services are less likely than goods to have observable characteristics. Comparing the features on television sets with different prices is easy, he said, but not so for movie tickets. "You don't have different movies with different admissions prices based on their acting," he said. The scope of a statistical agency's work can also be limited by its budget. For instance, the federal government has been gradually replacing its classification of industries in the last few years - a move that has affected thousands of figures collected by many agencies. Yet while the Federal Reserve painstakingly reconstructed decades of old data to match the new framework, Mr. Berner said, the Census Bureau has been unwilling or unable to follow because of a lack of money. The experts all said that almost constant revisions in methods and data sources made continuity a recurring problem in government statistics. Drawing comparisons of trends in inflation, employment and growth could be tricky, even, Mr. Berner said, for periods as short as five years. "The statistical agencies are a little too cavalier about saying, `Well, we want to have a very accurate snapshot right now, and to hell with continuity,' " Professor Mitchell said. "If they break the series, the users of that information - the clientele - don't have any alternative. We're at their mercy." Putting aside budget shortages, Mr. Berner said that the government's statisticians and economists were able to work in an independent, scientifically minded way. But Professor Mitchell, who has consulted for several federal economic agencies, said they also faced political pressures. As an example, he said, researchers making revisions to price indexes may have been pushed to understate inflation, as such a change would result in smaller liabilities in the Social Security system when the baby-boom generation retires. "In reality, those kinds of things do play some role." Mr. Soss agreed. "It's sort of in the air," he said. "For 35 years, countries have wanted their inflation rates to be lower, and I think there's a natural tendency of anyone living in the society to breathe the air. The air surrounding these processes was something that led to looking for ways that inflation might be lower."
