How the Jennings/ CEP Report Is Balanced -- But Not Valid
I asked Greg Forster (see below) to comment on the current conversation about the NCLB report and the larger issue of think tanks and the press.
Like the article he wrote, his answers include some points that I and others would do well to consider (and some that seem a little bit over the top).
He doesn't think the CEP reports are skewed towards anti-NCLB findings, but his thinks their scrupulous balance is still misleading. He doesn't much care that Jennings is an advocate, it's research that's not solid that irks him so. He doesn't know of any other folks who in his view are purporting to do academic research like this, but he knows for sure why the press don't call traditional academics.
Interesting stuff.
READ MORE>>
Do controversial or anti-NCLB findings get highlighted in CEP NCLB reports?
GF: I think just the opposite is the case – the CEP reports almost go out of their way to say something good and something bad on just about every aspect of No Child. And I don’t think they “highlight” findings that are more controversial or are explicitly anti-No Child. For example, the finding about schools cutting back other subjects to make room for more reading and math was not particularly highlighted in the most recent report.
Then what's the problem with what CEP does?
GF: There’s no need for CEP to go out of its way to highlight certain findings, since (as you point out) the structure of the data collection is already designed in a way that ensures the results will reflect the public school party line. Once you’ve stacked the deck, there’s no need to peek at your opponents’ cards. You already know what he’s holding. Similarly, once CEP has designed a study whose results are guaranteed to reproduce the public school party line, there’s no need to cherry-pick the results.
What about the findings themselves, then -- they're limited to what educators have to say, but otherwise balanced and representative, right?
GF: The dynamic that’s at work here is somewhat obscured because CEP says both good and bad things about No Child. A lot of people assume that if you say both good and bad things about something, you must be a reasonable and objective observer – after all, if you were biased, you would only say one or the other. But this premise is false. You can say both good and bad things and still not be making valid observations.
How can the report be balanced but not valid?
GF: The main reason CEP’s reports on No Child say both good and bad things is precisely because they do no more than reproduce the public school party line, and for some time now the public school system (as distinct from, say, the teachers’ unions) has adopted a sort of jujitsu strategy to attacking No Child – first they say lots of warm, fuzzy, positive-sounding things about it, and then they say, “but of course this commitment to accountability should be accompanied by an adequate investment in our schools,” etc. etc. So CEP says both good and bad things about No Child because the public school system does, too, and in these reports CEP is basically serving as their p.r. firm.
Are there any other education shops that exhibit the problems you describe in terms of political background and quality of research?
GF: I’m not aware of anyone else who fits that description in the same way Jennings does. There are plenty of education researchers whose work does not live up to rigorous scientific standards, but they are still career academics who have been trained in social science. Jennings, by contrast, has no training as a social scientist. His only graduate degree is in law, and his background in education is having spent 27 years promoting the Democratic party line. That’s not exactly the kind of background that prepares you to follow the evidence wherever it leads, or even to know what the scientific rules of evidence are. There are different rules for what counts as evidence when you’re doing science than when you’re doing advocacy, and Jennings doesn’t seem to have picked them up.
So is your main objection that he's an advocate -- or that he's not a scientist?
GF: My real objection to Jennings and CEP is that his work fails to live up to scientific standards. The only really important question about any piece of research is whether it follows the rules of science. If it does, then it doesn’t matter who produced it. Looking at people’s backgrounds only becomes relevant after it has been shown that the research isn’t good science – once you’ve proven that it’s propaganda, then you’re allowed to talk about whose propaganda it is and what agenda it’s serving. But not before; otherwise, all science can be equally dismissed as propaganda, because the findings will always serve someone’s interest. In Jennings’ case, however, it isn’t even a close call. His work is so severely compromised as science that it just doesn’t make sense for us to ignore the obvious explanation for those failings.
Then why does the press go to him -- and why do funders fund him -- when there are so many academic and university experts on hand?
GF: I don’t know anything about where he gets his funds, so I won’t comment on that. But as for why the media cover him, I think it’s mainly because he’s very skilled at putting himself forward as being exactly what they’re desperately looking for – a voice of disinterested expertise. Most of the academics who study No Child are really not very good at this; they clumsily flaunt their hostility to No Child and their ideological biases. This is hardly surprising, since they’re mostly located in schools of education where everyone around them shares their prejudices. By contrast, Jennings’ career of policy advocacy in D.C. has obviously provided him with an excellent skill set when it comes to self-presentation. He’s very good at coming across as exactly the kind of figure the media is looking for – someone who can stand above the political fray and objectively evaluate the facts. That’s why it serves him so well to produce these reports that carefully say something good and something bad about every aspect of No Child; it feeds into this assumption that anyone who says both good and bad things must be doing so because he’s looking at all the facts objectively, rather than because it’s in someone’s interest to come up with both good and bad things to say.
Like the article he wrote, his answers include some points that I and others would do well to consider (and some that seem a little bit over the top).
He doesn't think the CEP reports are skewed towards anti-NCLB findings, but his thinks their scrupulous balance is still misleading. He doesn't much care that Jennings is an advocate, it's research that's not solid that irks him so. He doesn't know of any other folks who in his view are purporting to do academic research like this, but he knows for sure why the press don't call traditional academics.
Interesting stuff.
READ MORE>>
Do controversial or anti-NCLB findings get highlighted in CEP NCLB reports?
GF: I think just the opposite is the case – the CEP reports almost go out of their way to say something good and something bad on just about every aspect of No Child. And I don’t think they “highlight” findings that are more controversial or are explicitly anti-No Child. For example, the finding about schools cutting back other subjects to make room for more reading and math was not particularly highlighted in the most recent report.
Then what's the problem with what CEP does?
GF: There’s no need for CEP to go out of its way to highlight certain findings, since (as you point out) the structure of the data collection is already designed in a way that ensures the results will reflect the public school party line. Once you’ve stacked the deck, there’s no need to peek at your opponents’ cards. You already know what he’s holding. Similarly, once CEP has designed a study whose results are guaranteed to reproduce the public school party line, there’s no need to cherry-pick the results.
What about the findings themselves, then -- they're limited to what educators have to say, but otherwise balanced and representative, right?
GF: The dynamic that’s at work here is somewhat obscured because CEP says both good and bad things about No Child. A lot of people assume that if you say both good and bad things about something, you must be a reasonable and objective observer – after all, if you were biased, you would only say one or the other. But this premise is false. You can say both good and bad things and still not be making valid observations.
How can the report be balanced but not valid?
GF: The main reason CEP’s reports on No Child say both good and bad things is precisely because they do no more than reproduce the public school party line, and for some time now the public school system (as distinct from, say, the teachers’ unions) has adopted a sort of jujitsu strategy to attacking No Child – first they say lots of warm, fuzzy, positive-sounding things about it, and then they say, “but of course this commitment to accountability should be accompanied by an adequate investment in our schools,” etc. etc. So CEP says both good and bad things about No Child because the public school system does, too, and in these reports CEP is basically serving as their p.r. firm.
Are there any other education shops that exhibit the problems you describe in terms of political background and quality of research?
GF: I’m not aware of anyone else who fits that description in the same way Jennings does. There are plenty of education researchers whose work does not live up to rigorous scientific standards, but they are still career academics who have been trained in social science. Jennings, by contrast, has no training as a social scientist. His only graduate degree is in law, and his background in education is having spent 27 years promoting the Democratic party line. That’s not exactly the kind of background that prepares you to follow the evidence wherever it leads, or even to know what the scientific rules of evidence are. There are different rules for what counts as evidence when you’re doing science than when you’re doing advocacy, and Jennings doesn’t seem to have picked them up.
So is your main objection that he's an advocate -- or that he's not a scientist?
GF: My real objection to Jennings and CEP is that his work fails to live up to scientific standards. The only really important question about any piece of research is whether it follows the rules of science. If it does, then it doesn’t matter who produced it. Looking at people’s backgrounds only becomes relevant after it has been shown that the research isn’t good science – once you’ve proven that it’s propaganda, then you’re allowed to talk about whose propaganda it is and what agenda it’s serving. But not before; otherwise, all science can be equally dismissed as propaganda, because the findings will always serve someone’s interest. In Jennings’ case, however, it isn’t even a close call. His work is so severely compromised as science that it just doesn’t make sense for us to ignore the obvious explanation for those failings.
Then why does the press go to him -- and why do funders fund him -- when there are so many academic and university experts on hand?
GF: I don’t know anything about where he gets his funds, so I won’t comment on that. But as for why the media cover him, I think it’s mainly because he’s very skilled at putting himself forward as being exactly what they’re desperately looking for – a voice of disinterested expertise. Most of the academics who study No Child are really not very good at this; they clumsily flaunt their hostility to No Child and their ideological biases. This is hardly surprising, since they’re mostly located in schools of education where everyone around them shares their prejudices. By contrast, Jennings’ career of policy advocacy in D.C. has obviously provided him with an excellent skill set when it comes to self-presentation. He’s very good at coming across as exactly the kind of figure the media is looking for – someone who can stand above the political fray and objectively evaluate the facts. That’s why it serves him so well to produce these reports that carefully say something good and something bad about every aspect of No Child; it feeds into this assumption that anyone who says both good and bad things must be doing so because he’s looking at all the facts objectively, rather than because it’s in someone’s interest to come up with both good and bad things to say.
posted by Alexander Russo at 4/04/2006 12:15:00 PM
0 Comments:
Post a Comment
<< Home