Research Project for Statistics 151A, Fall 2006

[Description] [Details] [Roles and Responsibilities] [Grading Criteria] [Due Dates] [Examples of News Articles] [Resources]

Description.

Broadly, the project is to unearth the science behind a current scientific news story, to determine whether the news report was accurate, and to assess the credibility of the underlying science.

25% of your grade in this course is based on this project. I expect the project will take you 40 to 60 hours to complete—so start early.

The research project has an individual component and a group component. (Your grade does too.) For the group component, you need to work with four other students. One of you will be the team leader. The responsibilities of the team leader differ from those of the other team members; see Roles and Responsibilities, below.



Details.

  1. Read Huff, D., 1954. How to Lie with Statistics, W.W. Norton & Co., NY.

  2. Pick your team. Teams should have five students, all from the same section of the class. (Advice for forming study groups.)

  3. Pick a team leader.

  4. Read the New York Times, the Wall Street Journal, the Los Angeles Times, or another premier newspaper daily until you find an article that reports a scientific finding that interests you. “Scientific” is meant broadly: astronomy, biology, chemistry, earth science, economics, epidemiology, medicine, public health, political science, psychology, sociology, etc., are all fine. There are examples of suitable articles at the end of this document.

    Scientific news items tend to fall into a few categories. One category is a research announcement, tantamount to a press release. The journalist covered a conference where a new result was announced, or had a preview of an article about to appear in a scientific journal. Such items usually cite the scientific literature, which gives you a toehold to start your research project. Another category is more like a review. Such articles tend to quote individual scientists, but often do not cite the scientific literature. It will be a little harder to do the background work for such an article, but by searching the academic literature for the names of the scientists who were quoted, you should be able to make progress. If an article does not cite the literature and does not quote experts by name, it will probably be hard to do the rest of the project.

  5. Find the scientific source on which the popular article is based, or relevant articles by people cited in the news story, if the news story does not itself cite the scientific literature. The scientific source might be a book, a journal article, or a newly released study, for example. There might be more than one source cited by the news story; if so, pick one you find central to the claims. There are many library tools you might use; see [link to resources]. To help you get started, [someone] will give a tutorial on library search tools on [date]. Make sure you can read the source.

    DUE 28 SEPTEMBER 2006: The names, student ID numbers and email addresses of the people in your team, and the name of the student who will be the team leader, by email. Team members should be in the same section of the class.

    DUE 12 OCTOBER 2006: A copy of the news item you propose to use, and a copy of the principal scientific source. You will hear within a week whether your proposed topic and source are approved. I will hold a special session for those whose topics or sources are not approved to help them select a topic and source.

  6. Each team member—other than the team leader—should find at least one scientific article he or she thinks is particularly relevant to the claims made in the source article. (Start with the bibliography in the main source article.) At least one of the articles should contradict the primary scientific source you are studying. The team leader is responsible for making sure at least one article is in this category.

  7. The team as a whole should write several paragraphs summarizing the primary source article. Each team member (other than the leader) should also write several paragraphs summarizing the supporting article he or she chose. The summaries should include:

    1. the scientific question the article addresses

    2. the names of the authors, and their institutional affiliations

    3. the source of funding for the research, if known, and the nature of that entity (government? industry? private foundation?)

    4. a description of the data

    5. a brief description of how the data were collected

    6. an overview of how the data were analyzed

    7. the main conclusions

    8. a synopsis of the argument that connects the data to the conclusions

    The team leader should collect the paragraphs from the team members and circulate them within the team for peer editing. The team leader should assemble these paragraphs into a well formatted document. The names and SIDs of the students on the team should be on the cover page. Each set of paragraphs should start on a new page, labeled with the name and SID of the student who drafted them.

    DUE 2 NOVEMBER 2006: A document containing

    (i) The summary paragraphs described above in (7).

    (ii) A preliminary bibliography in APA style (www.apastyle.org/) containing the news item, the source article, and the four other scientific articles. If there are other materials you expect to rely on, list them in the bibliography too.

    The document should be submitted to me by email, in Rich Text Format (rtf), HTML or Adobe pdf format. I will not accept Microsoft Word files.

  8. Now the real fun begins. Read the source article more carefully. Determine whether the news item correctly reported the findings of the source article, including any disclaimers or qualifications. Note any gaps, overstatements or distortions. Read the other four articles carefully. Assess whether their data and methodology are trustworthy, and examine their arguments. Decide whether you believe the claims in the source article.

    DUE 30 NOVEMBER 2006:

    An 8-12 page group paper based on your team's work in steps (7) and (8). The paper should include citations in APA style and a bibliography in APA style listing all materials you relied on. The paper should be submitted by email in Rich Text Format (rtf), HTML or Adobe pdf format. I will not accept Microsoft Word files.



Roles and Responsibilities.

The team as a whole is responsible for:

The team leader is responsible for:

Other team members are responsible for:



Grading Criteria.

All writing will be graded for content, accuracy, logic, clarity, brevity, spelling, grammar and style. Individual grades will be based 50% on the quality of the paragraphs that student drafted, and 50% on the quality of the team report as a whole.

An A+ team paper would address the following:

  1. Did the news report present the science accurately? Were there errors, distortions, or important omissions? Did the journalist see the big picture? Or did he or she cherry-pick, perhaps to make the results seem more sensational?

  2. Who says? Is the source reliable on its face? Is the research funded by industry or government or private foundation or ...? For whom do the authors work? Is an axe being ground here?

  3. Does the result pass the "sniff test?" Is it prima facie plausible? What experience or knowledge do you have have that could be used to check whether the conclusion is reasonable?

  4. What evidence do the authors have? What are the data? Was the data collection fair, or possibly biased? Was sampling used? What was the population? The frame? Might frame bias be an issue? What were the sampling units? The sampling design? Was it a sample of convenience? A quota sample? A systematic sample? Was the sample self-selected? Was the sample drawn at random? How big was the sample? What was the response rate? Is nonresponse bias an issue? Are the authors extrapolating beyond their sample? Beyond the sampling frame?

  5. Is this an experiment or an observational study? Is it a controlled experiment? If there were controls, were they historical? Is this a longitudinal study? A cross-sectional study? If there were human subjects, was the experiment blind? Double-blind? Were subjects assigned at random to treatment and control? What was the actual wording of any survey questions? Do the questions seem to be worded to elicit particular responses?

  6. What methods were used to analyze the data? Were the methods appropriate? Are the methods generally reliable? Are there situations in which those methods tend to distort the picture? What are the assumptions of the methods? Were the assumptions justified in this case? Would violations of the assumptions matter much? If so, can you figure out how plausible violations of the assumptions would affect the conclusions?

  7. Is confounding likely to be an issue? What variables might be confounders? Did the authors control for the confounders? If so, how? Might aggregation cause confounding here? (Is Simpson's paradox an issue?)

  8. What is the logical structure of the argument? Is it sound? Are there obvious gaps or errors in the logic? Is causation confused with association? Did somebody change the subject? Are there realistic possibilities not contemplated by the argument?

  9. Would it be possible to reproduce the research from the description in the article? If not, what's missing? Were ad hoc choices made that might influence the results?

  10. Do the authors confuse statistical significance with practical significance? Do they report uncertainties? Are their uncertainty calculations appropriate? (E.g., do they calculate confidence intervals or P-values using a sample that is not random?) Do they take into account sampling error? Bias? Specification error (error from using the wrong variables or the wrong model)? Do the uncertainty calculations depend on questionable assumptions about the origin of the data (e.g., independence, normal errors, linearity, homoscedasticity, ...)?

  11. Are there other explanations of the data just as plausible as those advanced by the authors? Did the authors consider those explanations? Did they have good arguments against those competing hypotheses?

  12. Were the results presented in a neutral way, or were technical devices used to make things look more impressive (gee-whiz graphs, one-dimensional pictures, ...)?

  13. Did the authors cite papers that disagree with their conclusions? Do the authors seem circumspect, or do they seem to be stacking the deck?

  14. What study or experiment might be done to confirm or dispute the research? What would be involved in conducting that research? Is it feasible?


Due Dates.



Examples of suitable articles.



Resources.

Newspapers:

Article Databases:

Other resources:


Copyright 2006, P.B. Stark. All rights reserved.

Last modified 8 October 2006