Sampling to Adjust the U.S. Census

Miller Institute for Basic Research in Science

Lunchtime Colloquium

12 January 1999

P.B. Stark
Department of Statistics
University of California, Berkeley

 


Outline

  1. Why adjust the census?

  2. What's happening right now?

  3. What happened in 1990?
    Sampling error and bias

  4. What's the proposal to adjust the 2000 census?
    Model and assumptions

  5. Why won't it work?
    1990 evidence that the assumptions are seriously wrong
    Ad hoc decisions drive the results
    Prima facie evidence of failure in 1990
    Conflict with Demographic Analysis in 1990

  6. Supporting arguments are poor

 

 


Why adjust the census?

Census can make two principal kinds of counting errors about individuals:

Gross omissions reduce the count, erroneous enumerations inflate the count.

The same individual can contribute both kinds of error.

For example, if a person's address is recorded incurrectly, he can be a gross omission at his correct address and an erroneous enumeration at his incorrect address.

The two kinds of errors cancel to some extent, but overall, the census misses some people: net undercount.

The undercount is different in different places, and for different demographic groups, e.g.:

1990 Undercount rate estimated by Demographic Analysis
Group estimated undercount rate
Total 1.8%
Non-Black 1.3%
Black 5.7%

Uneven undercount yields errors in state population shares, which determine congressional representation and allocation of Federal funds.

If the undercount were even, wouldn't affect state shares.

Unevenness is unfair (and politically incorrect!)

It would be wonderful to know how many people the census missed, and where.

Then could add them where they belong to improve state shares.

 


What's happening right now?

Census Bureau has proposed for 2000 a "one number census" using sampling based adjustments---raw counts would not be available.

The plan also proposes using sampling to select only a fraction of households that do not mail back their questionaires for follow-up. (In the past, attempts were made to follow up all non-responders.)

On 30 November 1998, the Supreme Court heard arguments regarding the use of sampling in the 2000 Decennial Census.

Suit brought by Speaker of the House (then Gingrich) against Department of Commerce to bar using sampling for apportionment.

Similar suit brought by Southeastern Legal Foundation

The Constitution requires an "actual enumeration" of the population.

The 1976 amendment of the Census Act of 1957 states that

"except for the determination of the population for purposes of apportionment of Representatives in Congress among the several States, the Secretary shall, if he considers it feasible, authorize the use of the statistical method known as `sampling' in carrying out the provisions of this title."

This would seem to prohibit sampling for some purposes, as was held by lower courts.

Supreme Court's decision is expected in March.


What happened in 1990?

1990 Census counted nearly 249 million people in the US.

The Census Bureau used sampling (Post-Enumeration Survey, PES) to estimate the 1990 undercount, to adjust for it.

(For the 2000 Census, there is a proposal to use sampling to adjust for undercount, and to follow up some people who do not mail back their census forms. This talk is only about adjusting for undercount.)

The sampling-based estimates come from an idea in wildlife management, called "capture-recapture."

To estimate the number of fish in a pond, could

Problems using this for people are outlined below.

Secretary of Commerce Mossbacher decided not to adjust the official 1990 census numbers.

Led to litigation by the City of New York, et al., against the Federal government.

In July 1991 (when Mossbacher had to decide), undercount estimate was about 5.3 million people.

Then found a programming error that had inflated the estimate by about 1 million.

More careful matching of records decreased the estimate by about another 300,000.

Now acknowledged that about 2-3 million of the remaining estimate is error in the estimate, not error in the census:

60% to 80% of the proposed 1990 adjustment was erroneous!

Most of the adjustment is "measured bias in the adjustment" rather than measured undercount

 


Sampling Error and Bias

Adjustment has two kinds of error:

Bias is a technical term: it does not mean someone is intentionally skewing the results.

Sampling errors tend to average out. Bias does not.

Nobody says that the Census is perfect. The question is whether adjustment makes the Census better.

Because of bias, adjustment almost certainly makes the Census worse, at the level of state shares and smaller geographic regions.

Estimating from a sample is like shooting a rifle

Each shot hits the target in a different place.

Sampling error is the scatter in the shots.

Bias is a tendency for all the shots to be off in the same direction.

Can fix bias in a rifle by sighting it in.

Possible, because you can see where the shots land.

Fixing statistical bias in a census adjustment is hard.

Only get one shot (because you only take one sample).

Cannot see where the shot lands (because you do not know the true undercount).


Proposed Census 2000 Adjustment Procedure

Example. Black male renters age 30-44 living in the central city of a major metropolitan area in New England were one 1990 post-stratum.

In the 1990 procedure, there were 1,392 post-strata in all.

Sample was about 380,000 people in 169,000 households in 5,290 block groups.

For 2000, propose about 12,600 post-strata (50 states 6 race/origin 7 age/sex 2 tenure 3 geography)

Sample to be about 1.7 million people in 750,000 households in 60,000 block groups.

Sample to be about 5 times larger for 2000, but taken in much less time.

Increased number of post-strata outweighs any improvement the larger sample might afford.

Decreasing the time to collect the data means using more poorly trained staff: more errors, lower data quality.

 


Adjustment Model and Assumptions

Basic idea:

Just a sketch--the details are extremely complex.

Definition of the Dual System Estimate (DSE) sampling-based adjustment in each post stratum:

NDSE number in post-stratum estimated by DSE
NC number  in the Census count
NEE number determined to be "erroneously enumerated"
NP number in the post-stratum found in the post enumeration survey
NM number in the post-stratum in the post enumeration survey that can be matched to the Census

 

NDSE = (NC - NEE)(NP/NM).

Subtract the estimated number of erroneous enumerations from the putative count to estimate the number correctly enumerated; inflate the result by the fraction of matches to account for those missed.

Every term on the right has error.

Assume:

 


None of these Assumptions is true.

The failures add bias, enough to make the adjustments worse than doing nothing.

Effect of heterogeneity within post strata is not negligable.

1990 Match/rematch mismatch rate 1.8%

1990 Fabricated interview rate estimated to be from about 0.03% to nearly 9%.

Just 13 (detected) fabrications identified in 1990 added about 50,000 to the undercount estimate.

A 1% fabrication rate would inflate the undercount by about 1.7 million.

A single unmatched family of 5 in the sample added 45,000 to the undercount estimate.

Garbage in, Garbage out:
Small errors in the sample give huge errors in the adjustment.

 


Ad hoc choices in the adjustment procedure drive the results.

Address errors and geocoding errors: searched over a ring of one or two blocks.

Without search, estimated undercount would have been twice as large.

If search area were larger, Census Bureau estimates that 75% of the unmatched cases would become matches.

Ad hoc choices drive the estimate.

Unresolved match status: 4,000,000 (weighted) people in census, 4,000,000 in sample survey.

Undercount estimate ranges from 9,000,000 and -1,000,000 (overcount) depending on how unresolved cases are treated.

Adjustment depends on a dubious statistical model (hierarchical logistic regression) for the unresolved cases. The model has no theoretical or empirical justification, and was calibrated on incommensurate cases where the match status was resolved.

The undercount estimate does not fix error in the census---it just adds new errors.

 


The Adjustments Don't Make Sense Prima Facie

New York, Pennsylvania, and Illinois lose shares in the 1990 adjustment. Texas and Arizona gain shares.

Probably easier to count in Dallas and Phoenix than in the Bronx, Philadelphia, and Chicago.

Taking shares from New York, Pennsylvania and Illinois might be right --- or it might be bias from bad assumptions.

Illustration of the effect of the proposed 1990 adjustment on State shares.

Figure courtesy of Brown et al. 1998.

 

Sex ratios of children under 10 is well determined demographic parameter across cultures: about 51% boys.

Essentially what was observed in 1990 before adjustment. Extremes of fraction of boys 50.3% to 52.1%.

Adjustment corrupts the ratios: after adjustment, get 48% to 56% boys. (See p17 of Darga, 1998.)


Demographic Analysis

There is another way to estimate the total population, called Demographic Analysis:

population = births - deaths + immigration - emigration

Because inter- and intra-state migration is not tracked, Demographic Analysis estimates only national totals (not state shares).

The 1990 adjustment adds more people than Demographic Analysis says were missed, including about a million extra women.

The Census and the PES miss many of the same people, including homeless, and people who do not want to be found.

The only way that the DSE can approach Demographic Analysis at the national level, therefore, is if it misclassifies "matches" as "non-matches." There is no reason to think that the geographical or demographic distribution of the misclassification mirrors the true undercount.

Because of bias, the adjustment probably puts the people in the wrong place, making state shares worse.

National-level estimates of the 1990 census undercount from the Post Enumeration Survey (July 1991 revision) and Demographic Analysis.
Table taken from From Brown et al., 1998.
Group PES Demog. Anal. Difference
Black Male 804,000 1,338,000 -534,000
Black Female 716,000 498,000 +218,000
Other Male 2,205,000 2,142,000 +63,000
Other Female 1,544,000 706,000 +838,000

 


Statistical Analysis Supporting Adjustment is Poor

For adjustment to make the census better, systematic errors (biases) need to cancel.

Random errors to cancel; systematic errors don't.

Arguments that systematic errors in the adjustment cancel depend on statistical models.

The models are false, and have bizarre consequences.

E.g., the model for “correlation bias” says that the 1990 census missed nearly 900,000 white males overall, but only 13 between 20 and 30 years old.

Model also says census missed >750,000 black males, but counted almost 30,000 too many black males under age 10.

Relying on that model, the Census Bureau claimed that some of the adjustment biases cancel, to give a net bias of 38%.

Without the model, the bureau estimated the bias at 57%, almost 20% higher.

Best study (in my opinion) finds the bias over 80%.

 

Current articles and manuscripts by Census Bureau personnel regarding the 2000 plan neglect bias---they report sampling error as if it were the entire source of error in sampling-based adjustments.

In 1990, bias was a more serious problem than sampling error, and was sufficient to invalidate the results.

Nothing in the new plan would mitigate the bias.


References

Bell, W.R., 1993. Using Information from Demographic Analysis in Post-Enumeration Survey Estimation, J. Amer. Statist. Assoc., 88, 1106-1118.

Breiman, L., 1994. The 1991 Census Adjustment: Undercount or Bad Data? Statistical Science, 9, 458-537.

Lawrence D. Brown, Morris L. Eaton, David A. Freedman, Stephen P. Klein, Richard A. Olshen, Kenneth W. Wachter, Martin T. Wells, and Donald Ylvisaker. Statistical Controversies in Census 2000, Technical Report 537, Department of Statistics, U.C. Berkeley, November 1998.

Bureau of the Census, 1993. Decision of the Director of the Bureau of the Census on Whether to Use Information From the 1990 Post-Enumeration Survey (PES) To Adjust the Base for the Intercensal Population Estimates Produced by the Bureau of the Census ACTION: Notice of final decision. Federal Register 58 FR 69 .

Committee on Adjustment of Postcensal Estimates, 1992. Asessment of Accuracy of Adjusted Versus Unadjusted 1990 Census Base for Use in Intercensal Estimates, Bureau of the Census (C.A.P.E. Report).

Census 2000 Operational Plan, U.S. Department of Commerce, Economics and Statistics Administration, Bureau of the Census, April 1988 (revised).

Darga, K., 1998. "Straining Out Gnats and Swallowing Camels: The Perils of Adjusting for Census Undercount," and "Quantifying Measurement Error and Bias in the 1990 Undercount Estimates." (submitted as testimony to the US House of Representatives Subcommittee on the Census on 5/5/98.)

Freedman, D. and Wachter, K., 1994. Heterogeneity and Census Adjustment for the Intercensal Base, Statistical Science, 9, 476-485.

Freedman, D., and Wachter, K., 1994. Rejoinder, Statistical Science, 9, 527-537.

Hogan, H., 1993. The 1990 Post-Enumeration Survey: Operations and Results, J. Amer. Statist. Assoc., 88, 1047-1060.

Waite, P.J., and Hogan, H., 1998. Statistical Methodologies for Census 2000: Decisions, Issues, and Preliminary Results, submitted to Amer. Statist. Assoc. July 1998.

Miscellaneous House of Representatives testimony on the Census.