                                                              1391

 1    UNITED STATES DISTRICT COURT
      EASTERN DISTRICT OF NEW YORK
 2    ------------------------------x
      THE CITY OF NEW YORK, et al.,
 3    
                         Plaintiffs,          88 Civ. 3474 (JMcL)
 4    
                 v.                           
 5    
      UNITED STATES DEPARTMENT OF COMMERCE,
 6    et al., 
      
 7                       Defendants.
      ---------------------------------x
 8    CITY OF ATLANTA, et al.,
                     
 9                       Plaintiffs,
      
10                v.                          92 Civ. 1566 (JMcL)
                                              
11    MOSBACHER, et al.,
      
12                       Defendants.
      ---------------------------------x
13    FLORIDA HOUSE OF REPRESENTATIVES, 
      et al.
14    
                         Plaintiffs,
15                v.                          92 Civ. 2037 (JMcL)
      
16    BARBARA H. FRANKLIN, Secretary of
      Commerce,
17    
                         Defendant.
18    --------------------------------x
      
19    
                                              May 20, 1992
20                                            9:30 a.m.
      
21    
      
22    Before:
      
23               HON. JOSEPH M. McLAUGHLIN,
      
24                                            Circuit Judge
      
25
                                                              1392

 1               MR. GOLDIN:  Good morning, your Honor.  

 2               THE COURT:  Good morning.  

 3               I gather we are going to hear from Dr. Tukey.  

 4               MR. GOLDIN:  We are, indeed, your Honor.

 5               Shall I call him?  

 6               THE COURT:  Yes.  

 7               MR. GOLDIN:  Plaintiffs call Professor John W. 

 8    Tukey.

 9    JOHN WILDER TUKEY, 

10             called as a witness by the plaintiff, having

11             first been duly sworn, was examined and

12             testified as follows: 
                                     

13    DIRECT EXAMINATION 

14    BY MR. GOLDIN:

15         Q.    Dr. Tukey, where are you currently employed?

16         A.    I am currently employed part-time in Princeton 

17    University, where I have the active title of senior research 

18    statistician and where I am a professor of statistics 

19    emeritus and donner professor of science emeritus.

20         Q.    When did you receive emeritus status?

21         A.    In 1985, when I turned 70.

22         Q.    Before that, what position did you hold at 

23    Princeton?

24         A.    I was professor of statistics and donner 

25    professor of science.
                                                              1393

 1         Q.    Have you served as Chair of the department of 

 2    statistics in Princeton?

 3         A.    Yes.

 4         Q.    Have you taught in any departments at Princeton 

 5    other than statistics?

 6         A.    I taught in the mathematics department for quite 

 7    a long time before there was a statistics department.

 8         Q.    When did you join the faculty at Princeton?

 9         A.    In 1939, when I got my Ph.D..

10         Q.    That was in the mathematics department?

11         A.    Yes.

12         Q.    When was the statistics --  

13               THE COURT:  Hieroglyphics in those days.  

14               MR. GOLDIN:  And still.

15         Q.    When was the statistics department created?

16         A.    I would have to look at the record, which I think 

17    is correct.

18               It seems as though it's 1965 is the correct date.

19         Q.    Do you currently hold a position with AT&T Bell 

20    Laboratories?

21         A.    No.  I am retired from AT&T Bell Laboratories, 

22    where I was an associate executive director in research.

23         Q.    What were your responsibilities in that position?

24         A.    To assist the executive director, who reported to 

25    the vice president for research in the area that was labeled 
                                                              1394

 1    communications principles, at least at one time.

 2         Q.    How long, approximately, did you hold that 

 3    position?

 4         A.    Roughly, 20 years, I think.

 5         Q.    And how long overall were you associated with 

 6    AT&T Bell Laboratories?

 7         A.    Well, under the previous name of Bell Telephone 

 8    Laboratories, beginning in 1945, when I went there to engage 

 9    in work part-time from late 45 on until retirement.

10         Q.    From what institution did you receive your 

11    doctorate?

12         A.    Princeton.

13         Q.    That was in mathematics?

14         A.    Yes.

15         Q.    Have you received any honorary doctorates?

16         A.    About five.

17         Q.    From which institutions have you received 

18    honorary doctorates?

19         A.    Chicago, Yale, Case Western, Temple, Brown.  I 

20    don't know whether I left any out or not, but it doesn't 

21    really matter.

22         Q.    It sounded like five to me.

23               Let me ask you a few questions about prior 

24    federal government service.

25               Apart from the special advisory panel in this 
                                                              1395

 1    case, have you served on any panel or committee created by 

 2    or under the auspices of the United States Department of 

 3    Commerce?

 4         A.    I served as a member of the National Commission 

 5    on Oceans and Atmosphere, to which I was appointed by 

 6    President Ford, and this operated out of the National 

 7    Oceanographic and Atmospheric Committee with general 

 8    responsibility to the President and the Congress.

 9         Q.    What did that commission involve?

10         A.    I'm sorry, can you clarify that a little?

11         Q.    What did your work on that committee involve?

12         A.    I think a realistic description of things is that 

13    it tried to understand and help resolve those problems which 

14    could be resolved without going to legislative questions.

15               We had a very mixed group of people, including 

16    former legislators, state legislators and a large group of 

17    scientific and technical people.

18         Q.    Have you engaged in any other federal government 

19    service?

20         A.    Yes.

21         Q.    Have you performed any federal government service 

22    on behalf of or under the auspices of the United States 

23    Department of State?

24         A.    I was a member of the U.S. delegation to the 

25    Technical Working Group 2 of the test ban negotiations and 
                                                              1396

 1    to the U.N. Conference on the Human Environment.  That's two 

 2    delegations.

 3         Q.    What did your work in connection with the test 

 4    ban conference involve?

 5         A.    Participating in the meetings when appropriate 

 6    and participating in the preparation for the meetings, which 

 7    took you would expect more time than the meetings 

 8    themselves.  I was in Geneva for about a month.

 9         Q.    And what did your work in connection with the 

10    conference on human environment involve?

11         A.    Being a working member of the delegation and 

12    trying to help in whatever was being done.

13         Q.    Apart from the committees or panels that you have 

14    already testified to, have you performed in connection with 

15    any other federal government service?

16         A.    Yes.

17         Q.    I am not going to go through all of those.

18               Let me turn to another area.

19               Apart from honorary degrees, have you received 

20    any awards or other honors during your career?

21         A.    Surely.

22         Q.    Did you receive the National Medical of Science?

23         A.    I did from President Nixon.

24         Q.    Could you tell us, what is the National Medal of 

25    Science?
                                                              1397

 1         A.    It's a medal to recognize scientific performance 

 2    and it's been given since roughly 1970 or 71 and to an 

 3    increasing number of people.

 4         Q.    To how many people was it being given in 1973, if 

 5    you know?

 6         A.    I'm vague on this, but I would think five to 

 7    nine, something of that sort.

 8         Q.    Are you a member of the National Academy of 

 9    Sciences?

10         A.    Yes.

11         Q.    What is the National Academy of Sciences?

12         A.    It is a body that was established by President 

13    Lincoln in order that he might have a group of scientists to 

14    which he could refer government questions.  

15               THE COURT:  You were the first chairman? 

16               THE WITNESS:  No, I'm afraid not, sir.

17               I have served as chairman of Class 3, however.

18         Q.    Approximately how many members does the academy 

19    have?

20         A.    I think now it's got close to a thousand.

21         Q.    And what is the procedure for selecting or 

22    electing a member of the academy?

23         A.    Nomination and things are reviewed first in the 

24    sections and then to some degree in the classes and then 

25    they go to a vote of the membership.
                                                              1398

 1         Q.    Are you a member of the Royal Society?

 2         A.    I was elected a foreign member last year.

 3         Q.    What is the Royal Society?

 4         A.    It's probably the premier scientific organization 

 5    in the world.  It was founded in the early 1600s and it 

 6    covers all fields of science and engineering.

 7         Q.    Do you know how many foreign members of the Royal 

 8    Society there are?

 9         A.    Approximately a hundred.

10         Q.    Do you know how many of them are from the United 

11    States?

12         A.    Approximately half of them.

13         Q.    Within statistics, do you have any particular 

14    fields of specialization?

15         A.    I have a collection of fields of specializations, 

16    really, rather than one.

17         Q.    What are they?

18         A.    Well, some of them are time series, robust 

19    techniques, exploratory techniques, graphical techniques, 

20    probably I should add analysis of variance, seasoned with 

21    other topics to taste, such as regression.

22         Q.    Roughly speaking, and I think that we have 

23    established in prior proceedings here that we are counting 

24    multiple volumes of collected papers in separate books, how 

25    many books have you published?
                                                              1399

 1         A.    It seems to be 22 on the page.

 2         Q.    Roughly speaking, how many articles have you 

 3    published?

 4         A.    Several hundred.

 5         Q.    Have you published any articles on the subject of 

 6    census undercount and adjustment?

 7         A.    I have co-authored at least one paper in that 

 8    area.

 9         Q.    Did you serve as an expert witness in connection 

10    with litigation concerning the 1980 census?

11         A.    Yes.

12         Q.    Did you submit an affidavit in connection with 

13    that litigation?

14         A.    I did.

15         Q.    Are you aware of whether that affidavit has been 

16    and continues to be cited in the statistical literature?

17         A.    I believe that to be the case, but by secondhand.

18         Q.    Let me call your attention to the document which 

19    has been marked for identification as Plaintiff's Exhibit 

20    724, a copy of which should be before you and a copy of 

21    which, I believe, is before the court.

22               Is that a copy of your curriculum vitae?

23         A.    It is.  

24               MR. GOLDIN:  Your Honor, Mr. Millet has asked me 

25    to clarify, as I am happy to do, that this is an update of 
                                                              1400

 1    the copy of Dr. Tukey's CV previously supplied to 

 2    defendants.  It was not prepared specially for purposes of 

 3    this litigation and is simply his most current CV.

 4               With that representation, I would offer into 

 5    evidence Plaintiff's Exhibit 724.  

 6               MR. MILLET:  Your Honor, reserving my right to 

 7    object, your Honor, I did ask Mr. Goldin to make one further 

 8    representation, and that is the changes from the current 

 9    Exhibit 724 from the CV that was produced in discovery, 

10    which is two pages shorter, that those changes in addition 

11    are not relevant to the issue of adjusting the 1990 

12    decennial census.

13               For example, Professor Tukey's membership in the 

14    Royal Society was not identified in the CV produced in 

15    discovery.

16               We have no objection to this, subject to that 

17    representation.  

18               MR. GOLDIN:  Let me say, your Honor, I'm not 

19    really sure where this is going.  

20               THE COURT:  The exhibit is admitted, 724.  

21               MR. GOLDIN:  Thank you, your Honor.

22               Your Honor, I would proffer Dr. Tukey as an 

23    expert witness.  

24               MR. MILLET:  In the fields of?  

25               THE COURT:  On the skin and all its contents?  
                                                              1401

 1               MR. GOLDIN:  In the field of statistics.  

 2               MR. MILLET:  No objection, your Honor.  

 3               THE COURT:  Dr. Tukey is qualified to express his 

 4    opinion on matters of statistics.

 5               (Plaintiff's Exhibit 724 marked for 

 6    identification was received in evidence.) 

 7    BY MR. GOLDIN:

 8         Q.    Dr. Tukey, are you receiving compensation from 

 9    plaintiffs in connection with your work in this litigation 

10    or for your testimony today?

11         A.    No.

12         Q.    Do you expect to receive any such compensation?

13         A.    No.

14         Q.    Dr. Tukey, did you serve as a member of the 

15    special advisory panel created under the stipulation and 

16    order in this litigation?

17         A.    I did.

18         Q.    When did that service begin?

19         A.    I guess it must have begun sometime in the fall 

20    of 89, whenever the panel was constituted.

21         Q.    Approximately how much time have you devoted to 

22    your work as a member of the panel?

23         A.    At least 20 full days and various other bits and 

24    pieces.

25         Q.    Did you submit a recommendation as part of your 
                                                              1402

 1    work on the panel to the Secretary of Commerce on the 

 2    question of whether or not to correct the 1990 decennial 

 3    census?

 4         A.    I did.

 5         Q.    What did you recommend?

 6         A.    That it be adjusted.  

 7               MR. GOLDIN:  Your Honor, if I might approach the 

 8    witness.  

 9               THE COURT:  Yes.

10               (Handing to the witness)

11               (Pause)

12         Q.    Dr. Tukey, would you please look at the document 

13    that we have marked for identification as Plaintiff's 

14    Exhibit 728.

15               Is that a copy of your recommendations submitted 

16    to the Secretary?

17         A.    Yes.  

18               MR. GOLDIN:  I would offer Plaintiff's Exhibit 

19    728.  

20               MR. MILLET:  No objection, your Honor.  Even 

21    without the stamps I can tell this is from the so-called 

22    administrative record.  

23               THE COURT:  728 is admitted.

24               (Plaintiff's Exhibit 728 marked for 

25    identification was received in evidence.) 
                                                              1403

 1    BY MR. GOLDIN:

 2         Q.    Dr. Tukey, in reaching the conclusion reflected 

 3    in this recommendation, did you determine that adjusting the 

 4    census would improve the accuracy of the census counts at 

 5    higher levels of geography?

 6         A.    Yes.

 7         Q.    In reaching your conclusion, did you determine 

 8    whether the issue of block level accuracy is a significant 

 9    one to address in resolving the question of whether to 

10    adjust the census counts?

11         A.    I considered this carefully and decided that it 

12    was not a significant issue.

13         Q.    What was the basis for that conclusion?

14         A.    May I be allowed to use the flip chart, please?  

15               THE COURT:  Please do.

16               (Pause) 

17         A.    If we consider -- this is all schematic, we all 

18    know Congressional districts don't turn out to be squares -- 

19    if we consider a congressional district that has a 

20    population these days that have something like 550,000 and 

21    we consider the influence of a one percent change in this, 

22    to meet the same target figure we would either add or 

23    subtract a strip whose population is one percent of the 

24    total.

25               If this was undercounted by one percent, you 
                                                              1404

 1    would decide if it was larger then you thought you would 

 2    take a strip off, if this were overcounted, you would say 

 3    it's not as big as you thought, you have to add a strip, but 

 4    in each case if you are talking about a one percent change, 

 5    you are going to talk about a strip with one percent as many 

 6    people, and in this are some little blocks with typically 50 

 7    people.

 8               Now, whether or not one percent is undercounted 

 9    or overcounted here is what determines whether this strip 

10    will be added or not.

11               The fact that you were one percent off on the 

12    5,500 is only 55 people, and that's not really statistical 

13    going to influence the decision at all.  So even for the 

14    whole strip of 5,500 people, a few percent of undercount or 

15    overcount are not going to influence what you do to adjust 

16    the district to allow for the undercount or overcount which 

17    mainly occurs on the part of the district that was there 

18    anyway, and in particular the fact that you might be one or 

19    two people over or under on a particular block is going to 

20    have absolutely no effect on the sort of decisions that go 

21    on when you do congressional redistricting.  They have no 

22    effect on the question of is this population equal to that 

23    of other districts to whatever tolerance the courts may have 

24    decided is appropriate and also to all the other 

25    considerations that arise in redistricting.
                                                              1405

 1         Q.    Dr. Tukey, you are talking about a situation in 

 2    which, as a result of determining that there is an 

 3    undercount, the size of a congressional district is to be 

 4    adjusted to take into account that fact?

 5         A.    Yes.  What the difference is between what you 

 6    would do if you accepted the raw counts and what you would 

 7    do if you accepted the adjusted counts.

 8         Q.    And in order to be able to make the appropriate 

 9    modification, one's concern is with a strip of people --

10         A.    Well, the concern is with the effect of adding on 

11    or subtracting a small part of the district, which I have 

12    sketched as a strip so that it would be easiest to see what 

13    is going on.

14         Q.    But in any event, it's a portion of people which 

15    is going to be substantially unaffected by the question 

16    whether particular blocks have been counted with accuracy, 

17    with extreme accuracy?

18         A.    Yes, and even the question of whether the strip 

19    has been provided with a more accurate population by 

20    adjustment.

21               The basic issue is, is the body of the proposed 

22    district, do we improve that by adjustment.

23         Q.    And the improvement we obtain here is one which, 

24    if there is an undercount, then the district is adjusted by 

25    contracting it, if there is an overcount it is adjusted by 
                                                              1406

 1    expanding its boundaries, is that right?

 2         A.    The changes in the district due to using the 

 3    adjusted counts are as you describe.

 4         Q.    From what you have just said about block level 

 5    accuracy, that would mean that the adjustment represents in 

 6    this illustration an improvement, but not a perfection, is 

 7    that correct?  

 8               MR. MILLET:  I object to form.

 9         A.    It is certainly generally true that the purpose 

10    of adjustment is improvement and not perfection.

11               We would be glad to get perfection if we could, 

12    but in almost every situation that I know, probably every 

13    one, where adjustment is what one is considering, you are 

14    glad to improve things and you know perfection is never 

15    going to arise.

16         Q.    Was that a consideration that you took into 

17    account in framing your recommendation to the Secretary?

18         A.    Certainly.  I mean, I was suppose to make a 

19    recommendation to the Secretary based on the real situation 

20    and not on any idealized form of it.

21         Q.    Does it concern you that adjustment leads to 

22    improvement, but not necessarily to perfection?

23         A.    No.  You can hardly be a statistician who is 

24    involved in practice as well as theory without realizing in 

25    the real world things are that way and are going to stay 
                                                              1407

 1    that way.  

 2               MR. GOLDIN:  Your Honor, I would like to mark and 

 3    offer as an illustrative exhibit the chart that Dr. Tukey 

 4    drew a moment ago.  

 5               MR. MILLET:  No objection.  

 6               THE COURT:  You will arrange for its 

 7    reproduction?  

 8               MR. GOLDIN:  Thank you, your Honor.  

 9    BY MR. GOLDIN:

10         Q.    Dr. Tukey, in making your recommendation and 

11    reaching your conclusion about adjustment, did you consider 

12    whether the adjustment procedure was prespecified?

13         A.    Yes.

14         Q.    Can you tell me what you mean by prespecified in 

15    this context?

16         A.    Certainly.

17               In almost any situation where statistical 

18    analysis is to be applied to carefully collected information 

19    and another convenient example besides the question of 

20    adjusting the census arises with when you have clinical 

21    trials of medicines or other therapies, for example.

22               There are always a wide variety of analyses 

23    differing usually in the small but perhaps in the medium 

24    that could be used and there is always a question whether, 

25    since most of these things turn out to some degree or other 
                                                              1408

 1    to be adversarial, whether the choice of the method of 

 2    analysis has been unduly influenced by the desires of one 

 3    party or another.

 4               The usual technique for dealing with this is to 

 5    prespecify the analysis in as much detail as one reasonably 

 6    can.  Now, this may mean prespecifying how you set about 

 7    choosing the details of the analysis, it does not 

 8    necessarily mean prespecifying the details.  But once you do 

 9    that, you have a clear prespecified answer which, in the 

10    clinical trial situation, may well be agreed on as the 

11    answer to the analysis of that trial.  

12               In the census situation the prespecification left 

13    two answers, the unadjusted enumeration and the adjusted 

14    enumeration, and an important reason for prespecifying 

15    things is to avoid dispersed attention and irrelevant 

16    controversy about other analyses that could have been done, 

17    in the case of the census other adjustments that could be 

18    made.  

19               It's pretty widely understood with statisticians 

20    with experience in this sort of thing that tying down the 

21    alternatives in advance as far forward as possible is 

22    routinely a good thing may not produce the absolute best 

23    analysis.

24               I told a meeting of the Society of Clinical 

25    Trials last week that any one of those of us who are on that 
                                                              1409

 1    particular program could undoubtedly do a better job in a 

 2    particular trial than the prespecification, but it did not 

 3    seem wise to let people do such things.

 4               So the importance of prespecification is that it 

 5    really does greatly reduce the impact of prejudice and bias.  

 6    The more you understand about the process, the more 

 7    completely you are able to prespecify things.  Almost 

 8    always, however, there are minor adjustments that need to be 

 9    made.

10               In the case of the clinical trial there are 

11    requirements in the protocol, for example, and a few people 

12    will be put into the trial who don't meet those requirements 

13    and have to be edited out at a late stage.

14         Q.    Do you consider the procedure followed by the 

15    Census Bureau in connection with adjustment of the 1990 

16    census to be prespecified?

17         A.    I consider that it was as fully prespecified as 

18    our knowledge and understanding at the time before the data 

19    was analyzed would permit.

20               As I say, there are always things of some degree 

21    or other in terms of follow-on and we don't always think of 

22    all the things we should in the beginning.

23         Q.    Do you consider it important that a procedure 

24    such as this would be prespecified?

25         A.    Yes.  I think it is important in removing 
                                                              1410

 1    unnecessary discussion of alternative analyses, particularly 

 2    alternative analyses very close to the one selected.

 3               One can never be sure that the one selected or 

 4    the one agreed to by some after the fact discussion is the 

 5    best that could have been done.  If you do not prespecify, 

 6    you waste a lot of time discussing alternatives that are not 

 7    really helpful.

 8         Q.    Are you familiar with the statistical concept of 

 9    robustness?

10         A.    Yes.

11         Q.    How do you come to be familiar with that concept?

12         A.    Well, in the more technical sense of the word, 

13    it's probably true that this goes back to a technical report 

14    that Ted Harris and I wrote in 1958, and it's a field in 

15    which I have been involved at intervals rather deeply ever 

16    since.

17         Q.    What for a statistician is the idea of 

18    robustness?

19         A.    I will need to answer that, I think, in two 

20    pieces, if I may.

21               In the narrow technical sense, the idea is that 

22    rather than seeking a procedure that is optimum under 

23    narrowly specified conditions, one tries to find procedures 

24    which are very satisfactory under a variety of alternative 

25    specified procedures.
                                                              1411

 1               The most used case of this arises where the 

 2    alternative specifications have to do with distribution 

 3    shapes, and in this case the consequence of trying to be 

 4    robust is almost always, I think I can safely say always, 

 5    arranging not to put too much weight on any one piece of 

 6    evidence and that this is the central phenomenon for the 

 7    statistician's technical idea of robustness.

 8               It's a word in the common language and so it's 

 9    been used in a variety of senses.

10         Q.    In making your recommendation to the Secretary 

11    and in reaching your conclusion about adjusting the 1990 

12    census, did you consider whether the adjustment was robust?

13         A.    In a different sense than the one I've just 

14    discussed, yes.

15               One of the guidelines describes robustness as not 

16    being subject to too much change when you replace one 

17    analysis by other reasonable ones.

18               Now, if we interpret that in the sense which I 

19    said the practice of robustness usually corresponds to, that 

20    would mean that you do not have undue sensitivity to changes 

21    in the analysis, you are going to have some changes.

22               There were some alternative analyses conducted by 

23    the Census Bureau, there were some alternative analyses, 

24    incomplete but very illuminating, conducted by Dr. David 

25    Hoaglin and colleagues.  All of these showed small changes 
                                                              1412

 1    in the results.

 2               My view was that all of these clearly indicated 

 3    an absence of undue sensitivity, and so it was my judgment 

 4    in the sense proposed in the Secretary's guidelines that the 

 5    adjustment was robust.

 6         Q.    Does the adjustment depend upon a number of 

 7    assumptions?

 8         A.    You will pardon me if I sigh, because the word 

 9    quote assumptions close quote in statistics is misleading 

10    and gives rise to undue difficulty and it's I suppose all 

11    our fault.

12               You would be very wrong to think of assumptions 

13    as something like hypotheses in elementary geometry as 

14    things which were supposedly essential to reach the 

15    conclusion.

16               The situation in statistics, and this is in all 

17    areas that I am at all familiar with, is that people talk 

18    about quote assumptions close quote as the basis on which 

19    one is led to a particular style, type and perhaps details 

20    of analysis.

21               If these analyses only worked when these quote 

22    assumptions held, there would be no statistical analyses 

23    around, because I don't really know of any situation where 

24    the quote assumptions do hold, and I do know that the 

25    analyses are useful and broadly used.
                                                              1413

 1               So I think the conclusion is there are quote 

 2    assumptions which have helped to guide the choice of 

 3    analysis.  If the real world were too far from those, the 

 4    procedures would not be satisfactory.  But typically the 

 5    real world is not too far from those and the procedures are 

 6    satisfactory.

 7               An extreme example of this is the question of 

 8    homogeneity within some collection of entities that are 

 9    combined, for example, homogeneity among the undercount 

10    rates for all the blocks that enter into a single 

11    post-strata.

12               It is a mistaken view to say that these 

13    undercount rates are assumed to be equal when we adjust the 

14    census in the way that the Census Bureau proposes.  It would 

15    be wonderful if they were equal, but the procedure is 

16    effective when they are unequal.  The more equal they are 

17    the better, but we are, thanks to prespecification, facing a 

18    simple binary choice to adjust or not to adjust and we can 

19    be happy that we expect improvement from adjustment in the 

20    face of heterogeneity.

21         Q.    The assumption that you are referring to there, 

22    the assumption of homogeneity, is the assumption that the 

23    undercount rate for every member of a geographically and 

24    demographically defined group called the post-stratum is the 

25    same?
                                                              1414

 1         A.    That the undercount rates, the true undercount 

 2    rates for all the blocks that make up the post-stratum are 

 3    the same.  We, fortunately, don't even need to think about 

 4    individuals.

 5         Q.    Given what you have just said about that 

 6    assumption, is there a meaningful question that ought to be 

 7    asked about whether there have been significant violations 

 8    of the assumption?

 9         A.    That might have to be answered separately for 

10    each of the quote assumptions unquote.

11               For heterogeneity and homogeneity the example is 

12    simple.  The analysis used by the Census Bureau to provide a 

13    standard error for the adjustment factor for a post-strata 

14    makes use of the differences between the observed 

15    undercounts in the different blocks involved, as a matter of 

16    fact, in pieces of blocks, because the post-strata are 

17    divided by age and sex and, therefore, only a part of a 

18    block will contribute to one.

19               But the analysis looks at the observed 

20    differences which reflect both sampling error and 

21    heterogeneity, and this is adequately taken count of in the 

22    calculation of the standard error.

23         Q.    And I take it from your testimony, as long as the 

24    post-strata are satisfactorily differentiating between areas 

25    and group of different undercount rates, then the adjustment 
                                                              1415

 1    is still providing an improvement in accuracy?  

 2               MR. MILLET:  I object to the form of the 

 3    question.  

 4               THE COURT:  Overruled.

 5         A.    I'm going to answer this in the affirmative.  I'm 

 6    just trying to decide how I should best answer it carefully.

 7         Q.    It's not a trick question. 

 8         A.    No, no, not an unfair question at all.

 9               The usefulness of the post-strata in allowing for 

10    undercount differences from one geographic and 

11    demographically defined group to another can be seen from 

12    the relation of the results of these post-strata to that 

13    standard error and it's not a question of assuming that they 

14    are doing a job in separating out different areas, it's a 

15    question of observing that there is evidence that they have 

16    done a job.  

17               MR. GOLDIN:  Your Honor, I have no further 

18    questions for this witness at this time.  

19               THE COURT:  All right, Mr. Goldin.  We will take 

20    a five minute break before we start cross.  

21               MR. MILLET:  Fine, your Honor.

22               (Recess)

23               (Continued on the next page) 

24    

25    
                                                              1416

 1    CROSS-EXAMINATION 

 2    BY MR. MILLET:

 3         Q.    Hello again, Professor Tukey.  Professor, you 

 4    discussed prespecification in your direct testimony.  I 

 5    believe you said that the adjustment procedures for 1990 

 6    were prespecified as much as possible except for some 

 7    smaller details.  Is that a fair statement?

 8         A.    That is approximately what I said.
                                                   

 9         Q.    Not all aspects of presmoothing were 

10    prespecified, were they, Professor Tukey?

11         A.    As I said also, we don't always know all the 

12    things we should at the right time.

13         Q.    Isn't it true that the choice of whether to 

14    include or exclude the various outliner points was one of 

15    the details that was not prespecified?

16         A.    Yes.

17         Q.    Isn't it also true --

18         A.    It should have been.

19         Q.    Isn't it also true that that detail made a 

20    difference in the apportionment of the House of 

21    Representatives?

22         A.    I'm not sufficiently cognizant of the 

23    consequences to answer that question.

24         Q.    Dr. Tukey, isn't it your opinion that reasonable 

25    statisticians and demographers can differ on the question of 
                                                              1417

 1    adjusting the 1990 decennial census?

 2         A.    I think if you added the word "otherwise," so it 

 3    is otherwise reasonable statisticians and demographers, I 

 4    would agree.  

 5               THE COURT:  Reasonable in all other respects.

 6         Q.    Let me ask you this, Professor.  Aren't there 

 7    statisticians and demographers who you consider to be 

 8    reasonable who at least seem to disagree on the question of 

 9    adjusting the 1990 decennial census?

10         A.    Yes.  I found that a very interesting phenomenon.

11         Q.    In fact, you reviewed some of the views and 

12    opinions of some other statisticians and demographers, isn't 

13    that true?

14         A.    In a mild way.

15         Q.    You reviewed Professor Wachter's recommendation 

16    to the Secretary, didn't you?

17         A.    Certainly.

18         Q.    Isn't it true that you don't consider the views 

19    that Professor Wachter expressed in his recommendation to be 

20    irrational?  Isn't that correct?

21         A.    There is a question of fact that I think I am not 

22    clear about.  It is my belief that Professor Wachter has 

23    said that in his judgment the adjustment was an improvement 

24    for large areas, like 500,000.  I do not know whether that 

25    was in the recommendation or not.  But in the light of the 
                                                              1418

 1    fact that for Congressional apportionment that is the level 

 2    at which you want to have good performance, then I think I 

 3    have to be surprised that he didn't come down for 

 4    adjustment.

 5         Q.    Professor, my question to you was a limited one, 

 6    and I think it was a little different from the question you 

 7    answered.  Let me restate it again.

 8         A.    Please.

 9         Q.    My question to you is, albeit you may disagree 

10    with Professor Wachter's ultimate conclusions to the 

11    Secretary, isn't it true that you do not consider the views 

12    he expressed in his recommendation to the Secretary to be 

13    irrational?

14         A.    I am trying to answer this as honestly as I can.  

15    It is going to take a moment's reflection.

16         Q.    Please.

17         A.    I think I should ask for a clearer definition of 

18    the word "irrational."

19         Q.    Let me try it this way, propose, Professor.  Do 

20    you recall my asking you in your deposition whether you 

21    considered Professor Wachter's views on adjustment being 

22    irrational, and do you recall stating, and I am reading from 

23    page 18:  "I don't think that there is any reason why I 

24    should do that.  I don't -- that doesn't mean that I agree 

25    with them."  
                                                              1419

 1               Is that what you said in your deposition?

 2         A.    I don't know.  But if that is what the record 

 3    shows, I must have.  I have no strong feeling that I need to 

 4    change it.

 5         Q.    Fair enough.  

 6               Professor, you have reviewed a recent draft paper 

 7    by Dr. Fay, Dr. Robert Fay of the census bureau, concerning 

 8    the estimating the variances for the smooth adjustment 

 9    factors, isn't that correct?

10         A.    Yes.

11         Q.    Isn't it true that you disagree with Dr. Fay to 

12    the extent that you believe that the estimates for the 

13    variances that he proposed in that draft paper are too 

14    large?  Is that right?

15         A.    That is accurate.

16         Q.    You also agree, do you not, that there should be 

17    some increase in the allowance for the variability of the 

18    smoothed adjustment factors, isn't that correct?

19         A.    That is probably correct.  I found the analysis 

20    sufficiently incomplete -- and I don't mean that this was in 

21    any sense not made obvious -- sufficiently incomplete that I 

22    am not sure how much adjustment is needed.  I believe it to 

23    be considerably less than Dr. Fay proposes.

24         Q.    But you would agree with the proposition that 

25    some increase in the allowance for the variability of the 
                                                              1420

 1    smoothed adjustment factors is probably warranted in this 

 2    case?

 3         A.    Probably.

 4         Q.    Isn't it true that you also agreed with the 

 5    general points of the Fay paper, and it is your opinion that 

 6    Dr. Fay's draft is, in general, reasonable and important?  

 7    Isn't that correct?

 8         A.    I think he raises an important issue.  I am not 

 9    sure to what extent it is so.

10         Q.    If a statistician were to conclude that the 

11    variances for smoothed adjustment factors should be doubled 

12    from what was originally calculated by the bureau, isn't it 

13    true that you certainly want to examine the reasons behind 

14    that conclusion?

15         A.    I think I would like to examine the reasons 

16    behind any conclusion that calls for substantial changes in 

17    the results.

18         Q.    Professor, you are aware that there was an error 

19    in the computer coding process using in mashing in the 1990 

20    PES process, aren't you?

21         A.    I have been told this.

22         Q.    It is your knowledge that that coding error 

23    actually inflated the net national undercount, isn't that 

24    right? 

25         A.    It could very well be that that is what I was 
                                                              1421

 1    told.  I have been concerned with the question of 

 2    comparative quality of estimates much more than with what 

 3    the consequences are.

 4         Q.    My question, Professor, however, is limited to 

 5    the coding error of which you have been informed.  Isn't it 

 6    true that that coding error resulted in a net national 

 7    undercount that was too large, in fact too large by a 

 8    million people?

 9         A.    I probably have been told this.  I do not know of 

10    my own knowledge. 

11         Q.    Let me, since I interrupted you, Professor --

12         A.    No, I think I have answered.

13         Q.    Also you have been told that it inflated the net 

14    national undercount by a million people?

15         A.    I believe I have been told, and I do not know of 

16    any more direct knowledge.

17         Q.    Professor, didn't you determine that the coding 

18    error inflated the differential undercount for blacks by 

19    about 10 percent.  

20         A.    Did I personally?

21         Q.    Yes.

22         A.    No.

23         Q.    Professor, do you recall stating in your 

24    deposition that you looked at the standard errors and you 

25    determined that making an allowance for the coding error 
                                                              1422

 1    reduced the black/white differential by perhaps 10 percent?  

 2    Do you recall stating that?

 3         A.    I certainly recall stating that that is what I 

 4    had been told.

 5         Q.    But even with the coding error, Professor, isn't 

 6    it true that you would favor an adjustment of the census 

 7    based on the PES over the unadjusted counts?

 8         A.    Yes.

 9         Q.    Wouldn't you also agree, Professor, that an 

10    adjustment with the coding error corrected would be --

11         A.    I'm sorry.  I misinterpreted your earlier 

12    question.

13         Q.    Okay.

14         A.    -- to mean with the coding error corrected.

15         Q.    No.  I was asking first without the coding error 

16    corrected.  So let me state that question so we are clear.

17         A.    Yes.

18         Q.    Isn't it your view that even knowing about the 

19    coding error, you would prefer an adjustment based on the 

20    1990 PES, without an allowance for the coding error, over 

21    the unadjusted enumeration?  Isn't that true?

22         A.    That is a question I don't think I have ever 

23    examined.

24         Q.    You certainly would favor an adjustment with the 

25    coding error corrected over one that was not corrected, 
                                                              1423

 1    isn't that true?

 2         A.    That is certainly true.

 3         Q.    Isn't it also true, Professor, that you didn't 

 4    make any calculations as to the impact of the coding error 

 5    on the shares of any areas?

 6         A.    That is correct.

 7         Q.    Professor, if almost all of the estimated 

 8    undercount at the national level and at the level of the 

 9    evaluation poststrata was found to be due to nonsampling 

10    error, moving the estimated undercount close to zero, 

11    wouldn't you agree that that would be a strong case for 

12    reexamining the question of adjustment?

13         A.    You are positing that some unspecified analysis 

14    for nonsampling error would bring all 13 of the evaluation 

15    poststrata results close to zero as far as the differential 

16    undercount went?

17         Q.    I am positing that both at the national level and 

18    at the level of the evaluation poststrata that we would move 

19    the undercounts close to zero.

20         A.    If someone put that view forward, I would say it 

21    required very careful analysis.

22         Q.    Professor, you told the Judge a little bit about 

23    your views on assumptions in statistics.  Is it a fair 

24    statement to say that it is your view that statistical 

25    procedures don't have assumptions, they just have 
                                                              1424

 1    circumstances where they work better and circumstances where 

 2    they don't work as well as others?

 3         A.    I am in print saying that.

 4         Q.    Isn't it also your view, Professor, that the 

 5    statistical procedures involved in the 1990 dual system 

 6    estimates have no assumptions?

 7         A.    If you want to take me that far, I will go that 

 8    far.

 9         Q.    Let me take you one step farther, Professor.  

10    Wouldn't you, in fact, agree that that view would be a 

11    fairly controversial viewpoint among statisticians?

12         A.    I think "somewhat controversial" is perhaps more 

13    accurate

14         Q.    Professor, isn't it true that in calculating the 

15    final adjustment factors for a particular poststratum, data 

16    from the other poststrata in the same region affect the 

17    final calculation?  Isn't that right?

18         A.    We are talking about what the census is doing in 

19    terms of the empirical basis?  

20         Q.    Correct.

21         A.    Yes.

22         Q.    Professor, do you have Defendant's Exhibit 68?

23         A.    Pardon?

24         Q.    Do you have any notebooks with Defendant's 

25    Exhibit 68 in front of you there?  If not, let me offer you 
                                                              1425

 1    one.  

 2               MR. MILLET:  Your Honor, may I approach?  

 3               THE COURT:  Yes.  

 4               MR. MILLET:  Does your Honor need a copy?  

 5               THE COURT:  I don't know.  Defendant's Exhibit 

 6    68?  

 7               MR. MILLET:  Your Honor, it looks a lot like 

 8    this.  

 9               THE COURT:  Everything looks a lot like that.  

10               MR. MILLET:  I don't think it is in your 

11    notebook, your Honor.  

12               THE COURT:  I can see the blackboard.  I'm all 

13    right.  

14               MR. MILLET:  Let me give you a copy.  I have an 

15    extra here.

16         Q.    Professor Tukey, I think all the mystery has been 

17    stripped away from Exhibit 68 at this point.  I believe you 

18    may recall this from your deposition.  This is, as testimony 

19    has already indicated, a document that has been prepared by 

20    one of defendants' experts concerning an alternative 

21    assumption in smoothing.  

22               Professor, my question to you is simply this.  

23    Taking a look at Exhibit 68, and particularly the second 

24    equation on the third line from the bottom, the equation 

25    including the term V sub 3 -- pardon me, formula, not the 
                                                              1426

 1    equation -- formula including the term V sub 3.  Do you have 

 2    that, Professor?

 3         A.    Yes.

 4         Q.    If you were involved in prespecifying the 

 5    procedures for the year 2000 census, isn't it true, 

 6    Professor, that this is an alternative that you would be 

 7    interested in exploring?

 8         A.    Let me say yes and no and then clarify things, if 

 9    I may.

10         Q.    All right.

11         A.    If I had a hand in prespecifying things for the 

12    year 2000 and if I was following the general line of 

13    analysis that the bureau followed in 1990, then I might 

14    include something that was more flexible and include it as 

15    special cases, both this with V3 and the corresponding one 

16    with capital I, which is what the census bureau used, and 

17    prescribing a procedure to decide what mixture of these 

18    seemed appropriate.  

19               I would not, if I had to choose one or the other, 

20    I don't think would, not in my understanding, want to 

21    replace capital I by V3.

22         Q.    You would be interested in exploring some kind 

23    of, if I can use the term, some kind of hybrid treatment 

24    that would use V sub 3?

25         A.    There were various things that involved 1990 that 
                                                              1427

 1    I would be interested in.  This is not one of the largest 

 2    ones.

 3               MR. MILLET:  We are done with Defendant's Exhibit 

 4    68, your Honor.  

 5               THE COURT:  Go ahead.

 6         Q.    When you made your recommendation to the 

 7    Secretary, Professor, it was your view at that time that the 

 8    correlation bias was sufficient to offset the effects of 

 9    nonsampling error identified by the census bureau in the 

10    total error model, isn't that correct?

11         A.    It was.

12         Q.    Isn't it true, Professor, however, that you have 

13    no measurement of correlation bias?

14         A.    "The correlation bias" is another rather 

15    inappropriate term, and it is supposed to cover everything 

16    that could be represented by people who are essentially 

17    impossible to count.  Measurement of it without a third 

18    system of estimates has no hope of being accurate enough to 

19    help.

20         Q.    So the answer to my question is yes?

21         A.    The answer to your question as put is yes.

22         Q.    Isn't it true, Professor, that you can't do 

23    anything other than conjecture as to how correlation bias is 

24    distributed geographically in the sense or the PES?  Isn't 

25    that correct?
                                                              1428

 1         A.    It depends on what the strength of the word 

 2    "conjecture" is.  There is evidence.  The evidence is 

 3    plausible.  It is not stringent or confining, but it is the 

 4    sort of thing on which I am used to making professal 

 5    judgments.  It would be my professional judgment that there 

 6    is a reasonable amount of evidence that correlation bias 

 7    follows other measures of difficulty to count.

 8         Q.    Professor, I want to use the term "conjecture" in 

 9    a fairly precise way and I want to differentiate it from a 

10    professional opinion.  Isn't it true that you can only 

11    conjecture as to how correlation bias is distributed 

12    geographically in the census or the PES?

13         A.    If you do not include professional opinion in 

14    "conjecture," then the answer is no.

15         Q.    The answer is no, that you can do something other 

16    than conjecture?

17         A.    Yes.  You can express a professional opinion 

18    based on evidence.

19         Q.    Professor, with all due respect, that is not what 

20    you testified to in your deposition, is it?

21         A.    I don't know.

22         Q.    Let me read to you from page starting at page 

23    341.  See if this refreshes your recollection.

24              "Q.    Professor, do you have an opinion as to how 

25    correlation bias is distributed geographically?  
                                                              1429

 1                    "MR. GOLDIN:  I object to the form.  In 

 2    something?  

 3                    "MR. MILLET:  In the United States of 

 4    America.  

 5                    "MR. GOLDIN:   No, I'm sorry.  In some 

 6    survey?  

 7                    "MR. MILLET:  In the decennial census.

 8              "A.    The census and the PES it takes too?

 9              "Q.    And the PES, right, does?

10              "A.    I would have a conjecture.

11              "Q.    That something was?

12              "A.    Rather than reaching the level of an 

13    opinion.

14              "Q.    Well, what would be your conjecture?

15              "A.    And the conjecture is that where ones that 

16    are hard to reach are hard to reach in correlation bias, the 

17    percentage is probably higher.  In other words, it seems to 

18    exist on different levels of hard to reach as far as we 

19    know.  Now, that's a conjecture."  

20               Weren't you differentiating between an opinion 

21    and a conjecture in that answer?  

22               MR. GOLDIN:  I would ask, given the length of the 

23    reading of counsel and the deposition, that counsel provide 

24    a copy of the quoted portion to the witness.

25         Q.    Would you like to take a look at it, Professor?
                                                              1430

 1         A.    Yes.

 2         Q.    We start on the first page and we carry over to 

 3    the next.

 4         Q.    Did you make a --

 5         A.    It seems that the precise definition of 

 6    "conjecture" which I tried to answer today and tried to 

 7    answer then were not the same.

 8         Q.    Professor, at the end of your deposition I asked 

 9    you a lot of questions on statements about regression.  We 

10    discussed those, and when you got done with the deposition 

11    you asked me to promise to tell you where they came from.  I 

12    am going to try to do that for you right now.  

13               I ask you to take a look at what has been marked 

14    for identification purposes as Defendant's Exhibit 604.  

15               MR. MILLET:  Your Honor will find that under Tab 

16    4 of your binder.

17         Q.    Professor, what I have handed you is a copy of a 

18    law review article published in the Columbia Law Review 

19    entitled "Multiple Regression in Legal Proceedings," 

20    authored by Franklin M. Fisher, professor of economics, 

21    Massachusetts Institute of Technology.  

22               Professor, I am going to direct your attention to 

23    the first page and the first line of that article.  I would 

24    like you to review that.  I would ask you to the following 

25    question.  
                                                              1431

 1               Does multiple regression make precise 

 2    quantitative estimates of the effect of different factors on 

 3    some variable of interest?

 4         A.    Always, no.  Sometimes, yes.

 5         Q.    Let me direct your attention to page 706.  I am 

 6    looking at the first paragraph on that page.  I am looking 

 7    at the second sentence.  Certainly feel free to review the 

 8    entire paragraph.  

 9               My question to you is this, Professor.  Is 

10    multiple regression the best available substitute for 

11    controlled experimentation?

12         A.    Often this is true.

13         Q.    Aren't there occasions when if controlled 

14    experimentation is not available, jumping to multiple 

15    regression can be dangerous?

16         A.    Yes, and so can all the other techniques.  I 

17    don't know anything that is more bulletproof.  That the 

18    doesn't say it isn't vulnerable.

19         Q.    Looking at the next to last sentence of that 

20    paragraph, which reads, "The results of multiple regressions 

21    can be read as showing the effects of each variable on the 

22    dependent variable, holding the others constant."  

23               Professor, do you agree that the results of 

24    multiple regression show the effect of each independent 

25    variable, holding the other independent variables constant?
                                                              1432

 1         A.    This is conventional wisdom in statistics books, 

 2    but there are some important caveats that need to be 

 3    considered.

 4         Q.    Isn't it true that you said in your deposition 

 5    that that statement is not always true, even if most of the 

 6    other things you might ask for happen to be true in an ideal 

 7    situation?

 8         A.    Yes, and I think that is equivalent, or 

 9    noncontradictory at least, with what I just said.

10         Q.    Professor, I would like to read to you a 

11    statement from the deposition of Franklin Fisher in this 

12    case.  

13               MR. MILLET:  Counsel, I am referring to page 130.

14         Q.    Professor Fisher was asked the following 

15    question.

16              "Q.    Is it also fair to say that what multiple 

17    regression does is to take advantage of the fact that the 

18    variables which explain the dependent variables do not 

19    always move together?"  

20               Do you agree with that statement?

21         A.    Yes.

22         Q.    Professor, didn't you disagree with that 

23    statement in your deposition?

24         A.    I'm sorry if I did.  Again, we are down at the 

25    cutting edge of what the words mean.  "Take advantage of," 
                                                              1433

 1    and the answer is yes.  "Do a good job of taking advantage 

 2    of," and the answer is sometimes or maybe no.  So if I 

 3    answered loosely in the deposition, I'm sorry.

 4         Q.    Didn't you state in the deposition, "how can 

 5    anybody take an affirmative answer to that question 

 6    seriously"?  

 7               MR. GOLDIN:  If you are going to read the answer, 

 8    counsel, I would ask, A, that you provide a copy of the 

 9    deposition transcript to the witness and, B, that you read 

10    it in its entirety.  

11               MR. MILLET:  I am looking for it now.

12         Q.    I am reading from page 231.

13              "Q.    Do you agree that the multiple regression 

14    takes advantage of the fact that the variables which may 

15    explain the dependent variables do not always move together 

16    and produce precise estimates of the independent effect of 

17    each of the variables?

18              "A.    How could anybody really take an 

19    affirmative answer to that question seriously, I guess is my 

20    reaction."  

21               Is that what you said in your deposition?

22         A.    Could I see that, please?

23         Q.    Sure can.  Right here, Professor.

24         A.    Thank you.  

25               Can I have your last question reread, please?  
                                                              1434

 1               MR. MILLET:  I have forgotten what my last 

 2    question was.

 3         A.    You had a question, do you agree with multiple 

 4    regression, I think.

 5         Q.    Do you agree that multiple regression takes 

 6    advantage of the fact that the variables which may explain 

 7    the dependent variables do not always move together and 

 8    produce precise estimates of the independent effect of each 

 9    of the variables?

10         A.    I'm sorry.  I did not hear the latter part of 

11    that question.  I was responding to the part that ended "do 

12    not always move together."

13         Q.    Okay.

14         A.    To that, the answer might be one thing.  But when 

15    you add "produce precise estimates of the independent 

16    effects of each of the variables," then I revert to my 

17    answer in the deposition:  How could anybody really take an 

18    affirmative answer to that question serious?

19         Q.    Let me ask you this, Professor.  Isn't it your 

20    view that Franklin Fisher is better known as an economist 

21    than as a statistician?

22         A.    Certainly.

23         Q.    Professor, when you perform or make a 

24    presentation about statistical findings in a professional 

25    meeting or in a courtroom, is it your custom to normally do 
                                                              1435

 1    your own calculations?

 2         A.    Probably not.

 3         Q.    Is it your custom to normally check those 

 4    calculations?

 5         A.    Yes.

 6         Q.    Professor, are you familiar with the imputation 

 7    process?  

 8               THE COURT:  Imputation?  

 9               MR. MILLET:  Imputation.

10         A.    Yes.  Only in the most general terms.

11         Q.    Do you know whether imputation uses a formal 

12    model?

13         A.    I am not sure how that question should be 

14    answered.  Imputation is a process.  If you are asking did 

15    people who invented the imputation, particular imputation 

16    process, use a formal model in the invention, I would have 

17    to say I don't know.

18         Q.    Isn't it also true, Professor, that you don't 

19    know the matched probabilities of cases sent to imputation?

20         A.    That is a part of the issue that I have not been 

21    concerned with.

22         Q.    Isn't it also true that you don't have any real 

23    knowledge of the impact of imputation on the final dual 

24    system estimate results?  Isn't that true?

25         A.    Except as it is reflected in the corresponding P 
                                                              1436

 1    studies, which were the bureau's procedures to try to assess 

 2    various things of this sort.

 3         Q.    Professor, isn't it also true that you don't have 

 4    any real knowledge of fabrication rates in PES?

 5         A.    Of which rate, please?

 6         Q.    Fabrication rates.

 7         A.    Fabrication rate.  I have seen a variety of 

 8    reports concerning the fabrication rate, yes.

 9         Q.    Isn't it true that the fabrication rate was 

10    determined by the bureau to potentially range from 2.8 

11    percent to 8.79 percent?

12         A.    I don't consider that I am informed about that.

13         Q.    If it did fall within that range, that would have 

14    been an interesting number to you, wouldn't it have been, 

15    Professor?

16         A.    Certainly.

17         Q.    In fact, if it fell within that range, couldn't 

18    that have affected your recommendation to the Secretary?

19         A.    I don't think it would have.

20         Q.    Could it have?

21         A.    How do I know?  Whether something of this sort 

22    would affect the recommendation is not just a matter of 

23    numbers, it is a matter of the nature of the support for the 

24    numbers and things of that sort.  I am not going to try to 

25    answer could or could not.
                                                              1437

 1         Q.    Wouldn't a fabrication rate of that range 

 2    indicate that the fabrication rate could have equaled or 

 3    exceeded the net national undercount rate?

 4         A.    If the numbers are as stated, presumably the 

 5    answer has to be yes.

 6         Q.    Professor, isn't it your understanding that, on 

 7    average, at the national level, each person in a PES sample 

 8    block weighted up to roughly about 600 people?

 9         A.    We are saying that the scale-up from the survey 

10    size in the PES to the national population?

11         Q.    Right.

12         A.    Is a factor of about 600?

13         Q.    Right.  Does that sound about right to you, on 

14    average?

15         A.    Could well be.

16         Q.    But there is some variation in that ratio, isn't 

17    there, Professor?

18         A.    Certainly.

19         Q.    Do you know whether it ranges as high as 1 to 

20    10,000?

21         A.    I do not.

22         Q.    If it did range that high, in fact, wouldn't you 

23    be rather surprised?

24         A.    No.  Having seen rather similar cases where there 

25    were some exceptional weights in the national assessment of 
                                                              1438

 1    educational progress, rather surprised is, I think, not what 

 2    I would have felt.

 3         Q.    Didn't you testify in your deposition that if it 

 4    ranged that high, you would be very surprised?

 5         A.    My degree of surprise clearly depends on the day 

 6    on which you ask me the question.  

 7               THE COURT:  And the state of your digestion.  

 8               THE WITNESS:  I hope not, but you may be right.  

 9               MR. MILLET:  After a while, I don't think 

10    anything surprises me in this case.

11         Q.    Professor, in your view, did sampling error in 

12    the P studies seriously affect their estimates of 

13    nonsampling error?

14         A.    Your question is as to whether sampling error in 

15    the conduct of the P studies seriously affected the P 

16    studies' estimate of nonsampling error?

17         Q.    Correct.

18         A.    I haven't reviewed the P studies from this point 

19    of view, but based on my knowledge of the census bureau and 

20    its statisticians, I would be doubtful.

21         Q.    In that same vein, Professor, isn't it true that 

22    you didn't review the total error model in any great detail?  

23    Isn't that right?

24         A.    In great detail, no.  In moderate detail, yes.

25         Q.    In your view, Professor, the census bureau could 
                                                              1439

 1    have used the raw adjustment factors without smoothing and 

 2    presmoothing, isn't that right?

 3         A.    You are saying, I believe, that a possible 

 4    alternative prespecification would have been to do this?

 5         Q.    That is a fair statement.

 6         A.    To which I would say yes and say this would have 

 7    then called for a careful reconsideration of what you 

 8    thought the standard errors were for the poststratum 

 9    adjustment factors.  I am not offering an opinion on that.

10         Q.    Professor, you are familiar with the post- 

11    stratification scheme used in the 1990 PES, are you not?

12         A.    Sorry?

13         Q.    You are familiar with the post-stratification 

14    scheme used in the 1990 PES?

15         A.    The 116 times 12 plus 12, yes.

16         Q.    Your math is faster than mine.  

17               Isn't it true that the post-stratification 

18    involved post-stratificatio,n among other elements, an 

19    element of geography?

20         A.    You are saying it involved other things than 

21    geography?

22         Q.    No, it involved among other things -- I phrased 

23    that poorly.  Let me try that again.  You're right.  

24               Post-stratification in the 1990 PES was done, in 

25    part, geographically, is that correct?
                                                              1440

 1         A.    Certainly.

 2         Q.    Those geographic groupings corresponded to the 

 3    census bureau's divisions, isn't that right?

 4         A.    With minor exceptions.  There were some cases 

 5    where divisions were combined within region.

 6         Q.    If the groupings of states were changed so that 

 7    they did not correspond to the census bureau divisions with 

 8    the modification that you just stated, wouldn't that result 

 9    in a new set of estimates at the state level?

10         A.    Almost anything you do differently will produce a 

11    new set of estimates at the state level.

12         Q.    Isn't there more than one possible grouping of 

13    the states for post-stratification that would be sensible, 

14    in your view?

15         A.    Certainly.  That is why prespecification is so 

16    important.

17         Q.    Professor, I am going to show you what has been 

18    received in evidence for illustrative purposes as 

19    Defendant's Exhibit 201.  

20               MR. MILLET:  Your Honor, I think you will find it 

21    under Tab 1.  

22               Just to clarify, Mr. Goldin has asked me a 

23    question, your Honor.  This was originally offered as part 

24    of a larger document in Dr. Wolter's testimony, and it was 

25    received with just the last page marked as Exhibit 201 for 
                                                              1441

 1    illustrative purposes only, so that we are all clear.

 2         Q.    Could you take a look at what has been marked as 

 3    Defendant's Exhibit 201, Professor.

 4         A.    I am.

 5         Q.    Does that reconfiguration of the states represent 

 6    a reconfiguration that, in your view, would be sensible?

 7         A.    I don't think I know enough about detailed 

 8    state-by-state behavior to answer that in the strong sense 

 9    that this is sensible enough that one might want to take it 

10    seriously.  In terms of my general knowledge of the states 

11    where I do have to consider various sorts of breakdowns, 

12    this is certainly one of many that might be sensible.

13         Q.    Professor, I am handing you what has been marked 

14    for identification purposes as Defendant's Exhibit 802.  

15               MR. MILLET:  Your Honor will find it under Tab 2 

16    of your binder.

17         Q.    Professor, this is Appendix H to the report that 

18    was authored under the names of your and your three 

19    colleagues to the Secretary, is that correct?

20         A.    Correct.

21         Q.    This appendix concerns small area accuracy, is 

22    that right?

23         A.    It is, I think, officially entitled "Errors for 

24    Smaller Groups in Relation to Errors for Larger Ones." 

25               MR. MILLET:  For the clarity of the record, this 
                                                              1442

 1    is contained in Plaintiff's Exhibit 195.  I won't be 

 2    introducing it separately.  I just have copies here for the 

 3    convenience of all concerned.  

 4         Q.    Let me direct your attention to the first page, 

 5    Professor, particularly page 1 at the bottom, the text 

 6    following the designation Result 2, and it carries over to 

 7    the top of page 2. 

 8               Professor, isn't it true that Result 2 depends on 

 9    the combined error measurement expressed in the text just 

10    above Result 1, where we see the sum of paren error over 

11    size --

12               This was a little complicated.  Take it in steps.  

13    First of all, Professor, if you would, just review Result 2.  

14    Tell me when you finish doing that.

15         A.    Yes, I have reviewed Result 2.

16         Q.    Isn't it true that Result 2 depends upon the 

17    combined error measurement expressed in the line just above 

18    the line beginning "Result 1"?

19         A.    As far as a precise result goes, yes.  I am 

20    trying to put in as carefully as I can.  As far as a 

21    reasonable anticipation goes, it certainly would apply in a 

22    broader range of cases.  I do not at this time have any way 

23    to try to tell you how broad this broader range would be or 

24    what it would include.

25         Q.    Isn't it true that Result 2 is applicable to the 
                                                              1443

 1    process of going from poststratum to poststratum block 

 2    groups in the 1990 decennial census?  Strike that.  Let me 

 3    rephrase it.  

 4               Isn't it true, Professor, that Result 2, in your 

 5    judgment, is applicable to the process of going from 

 6    poststratum-to-poststratum block groups?

 7         A.    We will have to make sure we are on the same 

 8    wavelength here.  When you say poststratum block groups, 

 9    what am I to interpret that as meaning?

10         Q.    Precisely that.  Let me see if I can refresh your 

11    recollection.

12         A.    Do you mean what the census called poststratum 

13    groups that consist of 12 poststrata.

14         Q.    I am referring to precisely the language that you 

15    used in your deposition, Professor, on page 201, where you 

16    said in response to this question:

17              "Q.    Is Result 2 directly applicable to the dual 

18    system estimate adjustment?

19              "A.    As I indicated in a previous answer, it is 

20    applicable from going to poststratum-to-poststratum block 

21    groups, which is something people, in effect, didn't do in 

22    the 1990 census adjustment."  

23               Maybe I can collapse all that into one.

24         A.    The answer to that is yes -- 

25               MR. GOLDIN:  Your Honor, let me just object to 
                                                              1444

 1    this question, since Mr. Millet has read only one sentence 

 2    out of a considerably longer answer that the witness 

 3    provided at the time of the deposition.  I would ask that if 

 4    he is going to use the answer as part of this question, that 

 5    he provide the full answer that the witness supplied at the 

 6    deposition.  

 7               MR. MILLET:  I think I am entitled to ask my 

 8    questions and Mr. Goldin can ask what he wants on redirect.  

 9    If the witness would like me to show him this and let him 

10    read it, I would be happy to do that.  

11               THE COURT:  I think that might facilitate 

12    matters.  

13               MR. GOLDIN:  Thank you, your Honor.

14         Q.    I am going to ask you to ignore my markings on 

15    this, Professor.

16         A.    That's easy.

17         Q.    I'm easily ignored.

18         A.    I didn't say that.

19         Q.    Again I want to direct your attention where you 

20    see my markings on the bottom of page 201.

21         A.    Yes.

22         Q.    Let me ask you, I am going to paraphrase it 

23    because I don't have the language in front of me, do you 

24    still agree with the statement that the result in Result 2 

25    in Appendix H is applicable to the process of going from 
                                                              1445

 1    poststratum-to-poststratum block groups, but that isn't what 

 2    was done in the 1990 census adjustment process?

 3         A.    Yes.

 4         Q.    Professor, isn't it true that block level data 

 5    are used in Congressional redistricting?

 6         A.    I presume the answer is yes.  I can't speak from 

 7    my own knowledge.

 8         Q.    Isn't it also true that you did not calculate the 

 9    accuracy of adjustment at the Congressional district level?

10         A.    Specifically, no.

11         Q.    Professor, I am showing you what has been marked 

12    for identification purposes as Defendant's Exhibit 803.  

13               MR. MILLET:  Your Honor will find that under 

14    Tab 3.  Again, this is an excerpt from the group report.  I 

15    don't intend to introduce it.  It is just here for ease of 

16    reference by the parties right now. 

17         Q.    Professor, you were the primary drafter of 

18    Appendix F, were you not?

19         A.    I think so.  

20               THE COURT:  The primary?  

21               MR. MILLET:  Drafter.  

22               THE WITNESS:  Yes. 

23         Q.    Isn't it true that Appendix F is not intended to 

24    be an analogy to the census or the post-enumeration survey?  

25    Isn't that correct?
                                                              1446

 1         A.    It is not intended to be a strict analogy.  It is 

 2    supposed to try to help people understand what the usual 

 3    assumptions about measurement and remeasurement amount to in 

 4    terms of how often, if you measure lots of different things, 

 5    the remeasurement result will be better than the measurement 

 6    result.  I have shortcut a few things, but that is the 

 7    purpose.

 8         Q.    Doesn't Appendix F, in your opinion, demonstrate 

 9    that in a fairly simple case and under good conditions, 

10    improvement from a remeasurement can be expected in only 

11    two-thirds of the cases?

12         A.    If we have the case of one measurement with 

13    random errors and a second measurement of the same entities 

14    with independent random errors, then it tells us that using 

15    the means of the two measurements, which I think generally 

16    speaking everybody assumes is better than using just the 

17    first one, will only improve the results for individual 

18    entities in rather less than two-thirds of the cases.  That 

19    is, it is an attempt to try to offer people a chance to 

20    calibrate their understanding of what to expect when you 

21    improve a measurement.

22         Q.    Doesn't Appendix F assume that there is no bias 

23    in either the measurement or the remeasurement?

24         A.    That is true.  I said random error just now when 

25    I tried to describe it.
                                                              1447

 1         Q.    The PES is biased, wouldn't you agree?

 2         A.    Are we saying that the PES is biased because of 

 3    nonsampling errors?

 4         Q.    Correct?

 5         A.    Right.  There certainly are nonsampling errors, 

 6    they certainly can be recorded as biases.

 7         Q.    Isn't it your view that the census also has bias 

 8    in the same sense?

 9         A.    Oh, yes, lots of bias.  But if I might add, the 

10    situation in this appendix was not intended to be a direct 

11    analogy.  It was intended to help people understand what 

12    their thoughts had been.

13         Q.    It is an illumination, right?

14         A.    It is an illumination, yes.

15         Q.    So Appendix F doesn't mean that in an adjustment 

16    all parts of the United States would be improved, is that 

17    correct?

18         A.    Adjustment is supposed to be improvement and not 

19    perfection.  Very good improvements will still not improve 

20    everybody.  Nothing that I know of improves everybody except 

21    where you have a meter that has a bad calibration and you go 

22    and change the calibration.  Everything else, you hope it 

23    helps more than it hurts.

24         Q.    So the answer is yes?

25         A.    Yes.
                                                              1448

 1         Q.    Professor, isn't it your opinion that there are, 

 2    indeed, some states whose shares will be improved from an 

 3    adjustment?

 4         A.    Yes.

 5         Q.    You can't name them by name, though, can you, 

 6    Professor?

 7         A.    I haven't tried to look at them from this point 

 8    of view.

 9         Q.    Aren't there some states whose shares will be 

10    made worse by adjustment?  Isn't that inevitable?

11         A.    It is essentially inevitable.  And this is true 

12    for any form of adjustment anywhere.

13         Q.    The states whose shares are going to be made 

14    inevitably worse cannot be identified either, can they, by 

15    you, Professor?

16         A.    They have not been identified by me.

17         Q.    There are some states whose shares would be moved 

18    by an adjustment for which we cannot tell whether their 

19    shares are being improved or worsened, wouldn't you agree 

20    with that?

21         A.    Certainly.

22         Q.    In fact, Professor, isn't it fair to say that you 

23    don't think the question of which states are improved is a 

24    very important issue?

25         A.    No, I don't think it is an issue that has 
                                                              1449

 1    concerned me.  And, if I might complete the answer to that, 

 2    the answerable sort of question, it seems to me, is states 

 3    showing which sort of behavior are likely to be improved and 

 4    states showing which sort of behavior are likely to be made 

 5    worse.  

 6               The states where there is a large change between 

 7    raw and adjusted will by and large be made better.  Those 

 8    where there is a very small change between raw and adjusted, 

 9    it is likely to be a toss-up as to whether they are better 

10    or not.

11         Q.    So there are going to be some states where an 

12    adjustment is going to be a toss-up as to whether it is 

13    going to be better or not?

14         A.    Where it is a toss-up for us as to determine 

15    whether it is an improvement or not.  Presumably for these 

16    states, if we had infinite knowledge, we would know what 

17    happened.  But we don't.

18         Q.    Professor, isn't it true that in reaching your 

19    recommendation to the Secretary, you did not rely on any 

20    loss function analysis?

21         A.    Any explicit loss function analysis, the answer 

22    is no, I did not.  The sorts of things that I looked at, 

23    however, could have been translated into loss functions, and 

24    I felt it was better not to.

25         Q.    In fact, isn't it fair to say, Professor, that 
                                                              1450

 1    you did not rely on any specific statistical test in 

 2    reaching your recommendation to the Secretary?

 3         A.    I felt the evidence was clear enough that one did 

 4    not need to rely on a specific statistical test.

 5         Q.    Isn't it fair to say that you relied a great deal 

 6    on your overall impressions from the data rather than any 

 7    particular calculations?

 8         A.    That I think is probably going too far.  It 

 9    depends on what a calculation is.  Looking at two numbers 

10    and seeing one was considerably larger than the other could 

11    be construed as a calculation.  If so, I used calculations.

12         Q.    Professor, didn't you testify in your deposition 

13    that you probably relied more on a rather careful overall 

14    impression than on a particular calculation?

15         A.    Yes.  As I say, this depends on what a 

16    calculation is.

17               MR. MILLET:  Thank you, Professor.  

18               Your Honor, I have no further questions.  

19               MR. GOLDIN:  Your Honor, could I ask that we take 

20    a break for a moment or two?  

21               THE COURT:  Five minutes?  

22               MR. GOLDIN:  Five minutes will be fine.  

23               THE COURT:  All right.  

24               (Recess)

25    
                                                              1451

 1    REDIRECT EXAMINATION 

 2    BY MR. GOLDIN:

 3         Q.    Dr. Tukey, during the cross-examination Mr. 

 4    Millet asked you a question that I think may have wound up 

 5    being rather complicated about otherwise reasonable 

 6    statisticians.  Let me ask you, are you aware, whether they 

 7    emanate from otherwise reasonable statisticians or 

 8    otherwise, of any plausible or rational reasons for not 

 9    adjusting the 1990 census?

10         A.    No.  
                      

11               MR. GOLDIN:  Nothing further, your Honor.  

12               THE COURT:  Any recross?  

13               MR. MILLET:  No, your Honor.  

14               THE COURT:  Thank you very much, Professor Tukey.  

15    It has been a pleasure listening to you.  

16               (Witness excused) 

17               MR. MILLET:  Your Honor, if I may, we need a 

18    couple of minutes to change personnel and documents between 

19    witnesses, if the Court will indulge us.  

20               THE COURT:  Why don't we take a substantial 

21    break, 15 minutes.  

22               MR. MILLET:  Thank you, your Honor.  

23               (Recess) 

24               (Continued on next page) 
                                          

25    
      
                                                              1452

 1               THE COURT:  Mr. Zimroth.  
                                           

 2               MR. ZIMROTH:  Your Honor, the plaintiffs' next 

 3    witness is Steven Fienberg, and with your Honor's 

 4    permission, my colleague, Rosemary Barry will examine the 

 5    witness.  

 6               THE COURT:  Very well.

 7    STEVEN ELLIOT FIENBERG, 

 8             called as a witness by the plaintiff, having

 9             first been duly sworn, was examined and

10             testified as follows: 
                                     

11    DIRECT EXAMINATION 

12    BY MS. BARRY:

13         Q.    Dr. Fienberg, where do you work?

14         A.    I am a professor of statistics and law at York 

15    University in Toronto, Canada, where I also serve as vice 

16    president for academic affairs.

17         Q.    And how long have you been employed in that 

18    position?

19         A.    Since July 1 of 1991.

20         Q.    And where did you work before that?

21         A.    Prior to that I was at Carnegie Mellon 

22    University, where I was dean of the College of Humanities 

23    and social sciences and a professor in the department of 

24    statistics and the department of social and decision 

25    sciences.
                                                              1453

 1         Q.    Have you concentrated your work in statistics in 

 2    any particular areas?

 3         A.    The work I do in statistics could probably be 

 4    roughly grouped into four somewhat overlapping areas:

 5               First, I work on the development of knew 

 6    statistical techniques or what we call statistical 

 7    methodology and, second, I work at the application of 

 8    statistical methodology in a variety of areas, especially in 

 9    the social sciences, but also in public health and in 

10    medicine.

11               In the third area I've done a considerable amount 

12    of research on the evaluation of federal statistical 

13    programs, and some of them involve statistical methodology 

14    in various ways.

15               And then finally I've done work on the design and 

16    analysis of sample surveys and how those techniques 

17    interrelate with the rest of statistical methodology.

18         Q.    What degrees do you hold in statistics?

19         A.    I received an undergraduate degree from the 

20    University of Toronto and did my graduate work at Harvard, 

21    where I received a master's degree, and then a Ph.D. in 

22    statistics, the latter being awarded in 1968.  

23               THE COURT:  Are you going to be putting a CV into 

24    evidence?  

25               MS. BARRY:  Yes, I intend to, your Honor.  
                                                              1454

 1               THE COURT:  Why don't you tell me what it is so I 

 2    can follow along?  It makes things easier.  

 3               MS. BARRY:  It has been marked for identification 

 4    as Plaintiff's Exhibit 1, and it should be in volume 1 of 

 5    the Fienberg exhibits.  

 6               THE COURT:  Thank you.

 7               (Handing to the court)

 8               (Pause) 

 9    BY MS. BARRY:

10         Q.    Dr. Fienberg, do you belong to any organizations 

11    in your field?

12         A.    I belong to several different statistical 

13    societies and organizations, among them the American 

14    Statistical Association, the Statistical Society of Canada, 

15    the Royal Statistical Society, the Institution of 

16    Mathematical Statistics, the International Statistical 

17    Institute and a number of more specialized ones.

18         Q.    Are you a fellow of the American Statistical 

19    Association?

20         A.    Yes, I was elected as a fellow to the American 

21    Statistical Association in 1972.

22         Q.    Have you published any books in the area of 

23    statistics?

24         A.    Yes, I have.  I published, actually published 13, 

25    the 14th is in press.
                                                              1455

 1         Q.    Have you published any textbooks in statistics?

 2         A.    Yes, I have.  I've co-authored a textbook with 

 3    Frederick --  introductory textbook in statistics with 

 4    Frederick Mosteller and Robert Rourke and I am the sole 

 5    author of a somewhat more advanced textbook on the analysis 

 6    of categorical data.

 7         Q.    Have you edited any statistical journals?

 8         A.    I've been on the editorial boards of a number of 

 9    different statistical journals, and from 1977 through 1979 I 

10    was coordinating and applications editor in the Journal of 

11    the American Statistics Association.  That's like the editor 

12    in chief.  And that's the principal research publication of 

13    the American Statistical Association.

14         Q.    Have you received any significant awards in the 

15    field of statistics?

16         A.    I guess the most significant award might be the 

17    President's Award from the Committee of Presidents of 

18    Statistical Societies for the outstanding statistician under 

19    the age of 40, which I received in 1983.  I was 39 and a 

20    half at the time, but they still gave it to me.

21         Q.    Have you taught any courses in Survey 

22    Methodology?

23         A.    Yes, I have.  At the University of Minnesota I 

24    helped to design a graduate sequence in applied statistics, 

25    and there was a quarter course on the analysis and design of 
                                                              1456

 1    sample surveys.

 2               I've also designed modules for introductory 

 3    courses and I taught from those modules several times.

 4               Then I've done a graduate seminar at Carnegie 

 5    Mellon on the design of sample surveys and the 

 6    interrelationships with other areas of statistics.

 7         Q.    Have you published any articles on the design and 

 8    analysis of sample surveys?

 9         A.    Yes, several.

10         Q.    Could you briefly describe your experience in the 

11    area of evaluating federal survey programs?

12         A.    I've been involved with the evaluation of federal 

13    survey and federal statistical programs going back into the 

14    early 1970s, and the two different activities during that 

15    period were work that I did as part of an evaluation of the 

16    design of the national crime survey, which is a survey that 

17    was done by the Bureau of the Census for the Bureau of 

18    Justice Statistics and later in the decade of the seventies 

19    I was part of a group that received a contract to actually 

20    formally redesign that survey.

21               And then in 1977, I worked as a special 

22    consultant to the National Commission on Employment and 

23    Unemployment Statistics, and in that work we were evaluating 

24    the two principal surveys that were being used to produce 

25    the nation's employment and unemployed data, and in 
                                                              1457

 1    particular the unemployed rate in those data were produced 

 2    as a product of the current population survey done by the 

 3    Census Bureau.

 4               We considered a variety of aspects of the current 

 5    population survey and in particular I analyzed the extent of 

 6    various sources of nonsampling error and the potential 

 7    implication for the quantities that were of interest to the 

 8    commission.

 9         Q.    Have you done any other work in the area of 

10    evaluating federal survey programs?

11         A.    Yes, I have.

12               As a member of the Committee on National 

13    Statistics, which is at the National Academy of Sciences, 

14    I've been involved with the review and analysis directly 

15    studying a number of different federal surveys, surveys like 

16    the survey of income and program participation and others 

17    both directly as a part of the committee's work and also as 

18    a part of the work of several of its panels, and in addition 

19    I have served as a consultant to government statistical 

20    agencies and to employees of cabinet departments carried out 

21    analyses, evaluations of statistical programs for them.

22         Q.    I believe you mentioned that you evaluated the 

23    sources of nonsampling error in connection with the CES?

24         A.    Yes, we did.

25         Q.    That was in connection with your work on the 
                                                              1458

 1    National Commission on Employment and Unemployment 

 2    Statistics?

 3         A.    Yes.  I was a member of a small panel of 

 4    statistical experts that was brought in to study these 

 5    issues on behalf of the commission and we reviewed the 

 6    various studies that the Census Bureau itself had done over 

 7    a 20 year period, studies done on the current population by 

 8    a number of people outside the Census Bureau, most of which 

 9    were collected in a compendium in integrated form in a 

10    volume by Barbara Bailar and Camelia Brookes, who were 

11    statisticians working at the Census Bureau at that time, 

12    that is, at the time we were doing the work for the National 

13    Commission.

14         Q.    Did the National Commission issue any report on 

15    nonsampling error in the CES?

16         A.    Well, the National Commission issued a very 

17    substantial report on the nation's employment and unemployed 

18    statistics.  Within it it commented on the current 

19    population survey and on the quality of that survey.

20               It, as does almost any other group, pointed to 

21    areas where things could be improved, but in general it, on 

22    the basis of the information on nonsampling errors, 

23    concluded that the data on unemployment that the CES 

24    provides was sufficiently high quality to be used for the 

25    various policy purposes to which it is put.
                                                              1459

 1         Q.    Did you participate in the drafting of the 

 2    National Commission report?

 3         A.    I drafted several sections of that report, others 

 4    I worked with the staff in the drafting and yet others I 

 5    reviewed after preliminary drafts were done by others.

 6         Q.    Do you recall what particular sources of error 

 7    were addressed in that report?

 8         A.    Oh, my goodness.

 9               It's a big book and the Bailar-Brookes compendium 

10    is a compendium of research looking at errors from different 

11    perspective that the Census Bureau had carried out beginning 

12    in the early 1950s throughout the fifties, sixties and 

13    seventies and even some that were still underway.  They 

14    looked at missing data, they looked at errors of nonresponse 

15    they looked at attrition.  This is a panel survey where, in 

16    fact, as you go from one interview to another you don't have 

17    the ability to go back and get all of the people who 

18    answered the first time so you lose some along the way.

19               There were a number of different sources, and in 

20    particular there was a report submitted as part of one of 

21    the background papers that the commission solicited that 

22    explicitly looked at the effect of undercount in the census 

23    on the current population survey and on the nation's 

24    estimates of unemployment.

25         Q.    Have you evaluated sources of nonsampling error 
                                                              1460

 1    in sample surveys in other contexts?

 2         A.    Yes, I have.  I evaluated, as I said when we were 

 3    doing the work to redesign the national crime survey, the 

 4    big issue was not what were the sources of sampling error, 

 5    although there was a group that focused on getting the 

 6    sampling error a little better, we spent virtually all our 

 7    time addressing issues of nonsampling error and how we could 

 8    improve the quality of the survey in light of what we had 

 9    learned from various studies of those sources.

10               In addition, I've reviewed a number of other 

11    detailed evaluations of sampling error in various large 

12    scale sample surveys, including another report done by the 

13    Census Bureau on the survey of income and program 

14    participation where it recast all of what it had learned in 

15    the context of the knowledge it had accumulated on 

16    nonsampling errors in SIPP.

17         Q.    Sipp is the --

18         A.    Survey of income and program participation.

19         Q.    Have you published any articles that address the 

20    subject of nonsampling error in sampling surveys?

21         A.    Yes, I have published several such articles.

22         Q.    Dr. Fienberg, have you studied the use of survey 

23    methodology for estimating the size of the population?

24         A.    Yes, I have.  Beginning in the late 1960s, 

25    shortly after I received my degree, there was a graduate 
                                                              1461

 1    student in Chicago who was working on this problem and in 

 2    reviewing her dissertation I got quite interested in it and 

 3    began to do my own research on the topic, and as a 

 4    consequence of that I published a paper in 1972 that 

 5    appeared in the Journal of Biometrica on the topic of 

 6    multiple recapture censuses.

 7         Q.    Is that method, multiple recapture censuses, 

 8    related to the PES methodology that was applied in the 1990 

 9    PES?

10         A.    Yes.

11               The techniques that I was looking at are 

12    extensions or modifications of what is known as 

13    capture/recapture methods, and the capture/recapture methods 

14    are essentially similar, in the same class as the dual 

15    systems estimation methods that serve as the basis of the 

16    post-enumeration survey.

17         Q.    You mentioned you published an article in 1972 on 

18    the topic of multiple recapture censuses.

19               Have you done any subsequent work?

20         A.    Yes, I have.  I actually worked at recasting that 

21    material, extending it and applying it to some other 

22    examples than the one I worked on originally, and that 

23    material became a chapter in a book I co-authored with Ivan 

24    Bishop and Paul Holland in 1975, and then subsequent to that 

25    I continued to work on this problem off and on in various 
                                                              1462

 1    ways, sometimes simply in an application of these 

 2    techniques.

 3               For example, I worked on a project with a 

 4    graduate student applying the class of techniques to 

 5    estimating the number of migrant workers in Idaho in the 

 6    late seventies, early eighties.

 7               And because of the direct applicability of these 

 8    methods in terms of census-like techniques, I continued to 

 9    work on them as I studied aspects of the U.S. census 

10    throughout the 1980s and had actually graduate students do 

11    work on the topic and ultimately prepared several additional 

12    papers dealing with extensions to the capture/recapture 

13    methodology.

14         Q.    Which, again, is related to the PES methodology 

15    applied in the 1990 PES?

16         A.    Yes.

17               You will have to excuse me, I use the word 

18    capture/recapture because it was what I first read when I 

19    read the methods.

20               The Census Bureau for a long period has used the 

21    word dual systems.  When you put the things down beside each 

22    other it is the same methods that they are being referred 

23    referring to, the same class of techniques.  

24               I tend to use them interchangeably now, but 

25    people who do things relating to census use the term dual 
                                                              1463

 1    systems, my graduate students tend to use the word 

 2    capture/recapture because they go off and read things in the 

 3    statistical literature that have those titles, but they are 

 4    used interchangeably in that sense.

 5         Q.    Thank you, Doctor.

 6               When did you first begin to study the issue of 

 7    census undercount?

 8         A.    Well, as I mentioned, the work that I did --  

 9    within the statistical profession people talked about the 

10    undercount.  It comes up and appears sort of indirectly in 

11    speeches in various ways so I had heard about it actually I 

12    guess early in the seventies, early on in my career.

13               The first time I directly considered the 

14    consequences was when I reviewed this background paper for 

15    the National Commission on Employment and Unemployment 

16    Statistics where it looked at the impact of the differential 

17    undercount, that is, the difference between the undercount 

18    of blacks and the undercount of others, on unemployment 

19    rates and tried to note the bias that that produces in the 

20    nation's estimate of unemployment.

21               Subsequently --

22         Q.    Doctor, when was that?

23         A.    That was in 1977.

24               Subsequently, when I joined the Committee on 

25    National Statistics actually the next year, the committee 
                                                              1464

 1    had a panel that was underway and that panel was reviewing 

 2    methodology to be used, to actually be taken in the field in 

 3    the 1980 census, and I reviewed that report for the panel 

 4    and actually wrote out some detailed comments and submitted 

 5    them which were ultimately taken into account, and that 

 6    panel focused on the issue of undercount, differential 

 7    undercount and possible methods for it's eradication.

 8               Subsequent to that, following the 1980 census, 

 9    when there was considerable discussion within the 

10    professional statistical community, and especially amongst 

11    those that were related to the federal government, as part 

12    of the Committee on National Statistics I began to review 

13    different aspects of the results of 1980, and I've been 

14    involved essentially continuously from that time forward 

15    evaluating, examining this issue in different ways, or at 

16    least studying it from different perspective.

17         Q.    Have you published any articles on census 

18    undercount?

19         A.    Yes, I have.

20         Q.    Have you ever given testimony before Congress on 

21    the issue of census undercount?

22         A.    Yes, on several occasions I was requested by the 

23    House Oversight Committee, the committee that reviews the 

24    Census Bureau's activities, to come before it and discuss 

25    aspects of the census, of errors in the census and of the 
                                                              1465

 1    use of the post-enumeration survey and its potential for 

 2    correcting the differential undercount.

 3         Q.    Have you ever taught any courses on issues 

 4    related to the census?

 5         A.    Yes.  In the winter of 1990 I taught an 

 6    undergraduate course in social science to students using the 

 7    census as the focal point.  It was called the American 

 8    census.  And in that course we looked at the history of 

 9    census-taking in the United States, we discussed the methods 

10    of census-taking that were about to be used in 1990.  

11               We actually, my students actually served as 

12    special enumerators and they went out and helped gather the 

13    data on behalf of the Census Bureau.

14               We had the director of the Census Bureau come to 

15    my class and lecture to them, and then we discussed issues 

16    like undercount, differential undercount and how the Census 

17    Bureau might address it.

18         Q.    Dr. Fienberg, have you been paid by the 

19    plaintiffs in this case for your services here?

20         A.    No.

21         Q.    Do you have any understanding with the plaintiffs 

22    about receiving any payment for your services?

23         A.    No, I have not.

24         Q.    And do you have any expectation of receiving any 

25    payment for your services?
                                                              1466

 1         A.    No, I do not.

 2         Q.    Dr. Fienberg, would you turn to the document 

 3    behind tab 1, which I believe is in your first binder, and 

 4    that document has been marked for identification as 

 5    Plaintiff's Exhibit 1.

 6               Could you please tell the court what it is?

 7         A.    Yes.  This is a copy of my curriculum vitae.

 8         Q.    Is that your most current CV?

 9         A.    Yes, it is.  

10               MS. BARRY:  At this time, your Honor, I would 

11    request that Plaintiff's Exhibit 1 be received into 

12    evidence.  

13               MR. SITCOV:  No objection.  

14               THE COURT:  Plaintiff's Exhibit 1 is admitted. 

15               (Plaintiff's Exhibit 1 marked for identification 

16    was received in evidence.) 

17               MS. BARRY:  And at this time I would also request 

18    that plaintiffs would also proffer Dr. Fienberg as an expert 

19    in the areas of statistics in general, in sample surveys and 

20    in the methodology and history of the census and the PES.  

21               MR. SITCOV:  Well, I'm not sure about the last 

22    part, your Honor, the history of the methodology of the 

23    census and the PES, but as to the first two, we don't 

24    object.  

25               THE COURT:  I will qualify him in all of those 
                                                              1467

 1    areas.

 2               You may proceed.  

 3               MS. BARRY:  Thank you, your Honor.  

 4    BY MS. BARRY:

 5         Q.    Dr. Fienberg, what is the Committee on National 

 6    Statistics?

 7         A.    First, I got to explain --  this is going to 

 8    sound complex.

 9               First there is the National Academy of Sciences, 

10    the honorific scientific organization located in Washington.  

11    It's an honorary society, they don't do much work, except 

12    that which they do back in their universities.

13               What they have put in place instead is something 

14    called the National Research Council, which is the research 

15    wing of the academy, and the National Research Council does 

16    its work through committees and panels.

17               The Committee on National Statistics is a 

18    standing committee of the National Research Council, and so 

19    when we talk about it, I normally talk about it in saying 

20    it's at the National Academy of Sciences.

21         Q.    Does the Committee on National Statistics focus 

22    on any particular issues?

23         A.    The committee was an outgrowth of a 

24    recommendation of a presidential commission in 1972, and it 

25    received essentially its charge, initial charge from that 
                                                              1468

 1    presidential commission, and that was to look at the broad 

 2    uses of statistics within the U.S. Government and for policy 

 3    purposes to assist federal statistical agencies and others 

 4    in evaluating national statistical programs and, third, to 

 5    consider and examine areas of statistical methodology that 

 6    have crosscutting interest to both the statistical 

 7    profession and to the application of statistics in a 

 8    national sense, an example of the latter being incomplete 

 9    data, and the committee had a panel that issued a three 

10    volume report on the analysis of incomplete data.

11         Q.    And by incomplete data, you mean a little more 

12    explicitly?

13         A.    Missing data and techniques like imputation.

14         Q.    How long were you a member of the Committee on 

15    National Statistics?

16         A.    I was selected as a member of the committee and 

17    asked to join it in 1978, and in 1981 I was selected by the 

18    National Research Council as chair of that committee.  

19               I served in that capacity through into 1987, 

20    until the end of June 1987, with the exception of a brief 

21    period of time in the middle when I took a sabbatical and my 

22    wife insisted that I not spend my time flying to meetings in 

23    Washington to attend things like committee meetings, and so 

24    then I just reviewed committee stuff from afar from 

25    California.
                                                              1469

 1         Q.    How does the Committee on National Statistics do 

 2    its work?

 3         A.    The committee does its work in a number of 

 4    different ways.

 5               First, through a substantial professional staff 

 6    that carries out projects initiated or suggested by the 

 7    committee.

 8               Secondly, through committee meetings, which are 

 9    held typically three times a year.

10               But most importantly, the committee's work is 

11    carried out by separately constituted panels, panels that 

12    are usually set up in response to specific requests from 

13    federal statistical agencies or even from Congress itself.

14         Q.    And did there come a time when the Committee on 

15    National Statistics focused on the issue of census 

16    undercount?

17         A.    Yes.  As I mentioned before, the committee was 

18    actually studying it, that is, it had a panel that was 

19    studying undercount in the 1980 census in advance as I 

20    joined the committee in 1978.

21               Shortly after the 1980 census results were in 

22    and there was this extensive discussion both within the 

23    statistical community and in the courtrooms about the 

24    appropriateness of correcting the census for differential 

25    undercount, the committee staff and I were approached by 
                                                              1470

 1    senior officials of the Census Bureau and we discussed with 

 2    them the possibility of creating a new panel at that time in 

 3    the near future --  I guess this was 1981, maybe late 81, 

 4    early 82 --  and we discussed how that panel might work and 

 5    the benefits it might produce for the Census Bureau.

 6         Q.    What senior officials were you referring to?

 7         A.    I explicitly remember having several extensive 

 8    conversations with then associate director Barbara Bailar, I 

 9    also had a number of discussions with the deputy director of 

10    the Census Bureau, and I seem to recall as we got toward the 

11    end of this process several discussions in addition with the 

12    director of the Census Bureau, who by this time I believe 

13    was Jack Keane.

14         Q.    And what discussions did you have about the 

15    benefits of such a panel?

16         A.    Well, the Census Bureau wanted to begin --  told 

17    us that they wanted to begin work on 1990 right away.

18               It's hard for people who aren't statisticians and 

19    don't do censuses to understand that the planning for the 

20    next census often begins before the last one was done, and 

21    if you wait too long and get toward the end of the process, 

22    then it's hard to develop the kinds of changes and 

23    improvements that the Census Bureau always tries to make in 

24    its work.  

25               Therefore, what they really wanted to do this 
                                                              1471

 1    time was to begin early in the decade to have an independent 

 2    panel that would review ideas and methodology on census 

 3    taking, consider alternatives that the panel thought it 

 4    would be worthwhile for the census to consider to carry out 

 5    research on and then, as we discussed how this panel might 

 6    work, we talked about it moving from that general kind of 

 7    phase into a more interactive phase where it would directly 

 8    review specific targeted research focused on evolving plans 

 9    for taking the 1990 census, and then --  and this was 

10    actually for me the amazing part --  we actually talked 

11    about a panel that might run over the course of its 

12    lifetime as long as eight or nine years, and then in the 

13    final phase that panel might actually review census results 

14    and provide direct feedback, especially in the context of 

15    what it had learned throughout the decade.

16         Q.    Dr. Fienberg, were you the primary representative 

17    of the Committee on National Statistics involved in the 

18    formation of this panel?

19         A.    Well, there was a lot of work done by staff at 

20    the Bureau and the staff of the committee, but I worked with 

21    the staff in actually drafting the statement of work for the 

22    panel that was submitted to the Census Bureau in response to 

23    its request that we present a work plan for such a study.

24         Q.    Are you aware that the Census Bureau has standing 

25    advisory committees?
                                                              1472

 1         A.    Yes, I am.

 2         Q.    And is the National Academy of Science or the 

 3    Committee on National Statistics one of those standing 

 4    advisory committees?

 5         A.    No, neither the Academy nor the Committee on 

 6    National Statistics is a standing committee for the Census 

 7    Bureau.

 8         Q.    And at the time the panel you are talking about 

 9    was formed, did you have any discussions with persons at the 

10    Census Bureau about why the Bureau wanted to establish a 

11    panel above and beyond its own standing advisory committees?

12         A.    Yes.

13               The National Academy of Sciences and the NRC and, 

14    in turn, the committee have always prided themselves on 

15    their independence and that independence is something that 

16    many federal statistical agencies truly value.  They want 

17    advice, they want good advice and they want that advice 

18    proffered at arm's length and perhaps much further than 

19    that, and the Committee on National Statistics has operated 

20    in that way, did operate in that way before it was 

21    approached by the Census Bureau in this instance.

22               And so I think that the independence of the 

23    committee and what would ultimately be the independence of 

24    the panel that it would create bringing together a diverse 

25    collection of experts was a primary, if not the primary, 
                                                              1473

 1    reason that we were told by the Census Bureau officials that 

 2    they wanted us to draft this report --  to create a proposal 

 3    for such a panel.  

 4               MR. SITCOV:  I am going to object and move to 

 5    strike.  That is full of hearsay and speculation on the 

 6    witness' part.  

 7               THE COURT:  I will take it.  The motion is 

 8    denied.  

 9    BY MS. BARRY:

10         Q.    Dr. Fienberg, I'm sorry --

11         A.    I wasn't done.

12               Secondly, the panel that was being proposed would 

13    have a permanent staff that would carry out independent 

14    activities under the directions of the panel members.  The 

15    standing advisory committees of the Census Bureau do not 

16    have such independent staff.

17               And then third, the panel, and I had indicated 

18    that the expected length that people had talked about, would 

19    have had a much --  a considerable accumulation of 

20    experience in the evaluation process, and that's typically 

21    different than the standing committees that gather once or 

22    twice a year and do no work directly in between and rotate 

23    people on and off on a regular basis.

24         Q.    Did there come a time when the panel was formed?

25         A.    Yes.  The panel was officially constituted in 
                                                              1474

 1    late 1983.

 2         Q.    And did you participate in the selection of the 

 3    panel members?

 4         A.    Yes, I did.

 5               The committee solicits advice on the composition 

 6    of a panel from a variety of sorts.

 7               Clearly some of it is structured by the mandate 

 8    of the panel, and in particular this panel had three 

 9    separate components to its mandate.

10               The first of these was to look at census 

11    methodology in the large, look at a spectrum of activities 

12    and statistical techniques, for example, that might be able 

13    to reduce or eliminate the differential undercount.

14               Secondly, the panel was specifically asked by the 

15    Census Bureau to focus on issues of sampling.  The Census 

16    Bureau is the foremost organization, government organization 

17    in the world in the area of sampling, and we were told that 

18    they wanted to consider other ways that sampling might be of 

19    value in the actual enumeration process, and so we needed to 

20    be sure that we had a number of people who could consider 

21    that issue.

22               And then third, we were asked to consider the 

23    possible role of administrative records in carrying out 

24    coverage evaluation and possible direct incorporation of 

25    those into census process.  
                                                              1475

 1               A number of other countries actually use 

 2    administrative records as part of their census-taking 

 3    activities and we were being asked to address those kinds of 

 4    issues as well.

 5               Then there is the way the Academy things about 

 6    panels.  So we had to cover these areas.

 7               The Academy wants to get a broad collection of 

 8    areas of expertise, including expertise in statistics, 

 9    expertise in the use of census data, knowledge about 

10    census-taking, but especially applications of census data in 

11    economics, in geography, in demography and other areas of 

12    the social sciences.

13               And then finally, overlaying all of those things, 

14    we had to think hard about the issue of undercount and 

15    whether or not we had a diversity of views amongst the panel 

16    members on the issue of correction or adjustment of 

17    undercount.

18               And so we tried to weigh all of these different 

19    considerations and put together a panel of approximately 13 

20    people to meet these different objectives.

21         Q.    Did this panel have a name?

22         A.    Yes.  This was known as the panel on decennial 

23    census methodology.

24         Q.    And would this be the same panel that Dr. John 

25    Rolph testified about earlier in this trial?
                                                              1476

 1         A.    Yes, it was.

 2         Q.    Dr. Fienberg, would you turn to tab 2 in your 

 3    binder number 1.

 4               If you could just take a look at the page that is 

 5    numbered three little i's.

 6               Actually, before you get there, I'm sorry, this 

 7    document has been previously admitted into evidence as 

 8    Plaintiff's Exhibit 2.

 9               Dr. Fienberg, can you identify it?

10         A.    Yes.  This is the first full report of the panel 

11    on decennial census methodology.

12         Q.    Would you turn to page three little i's, and is 

13    that a list of the original members of the decennial census 

14    panel?

15         A.    Yes, it is.

16         Q.    Could you just briefly tell us what the relevant 

17    expertise was of that group?

18         A.    Well, let me try to group them.

19               There is a whole --  there is a substantial 

20    subset of people who are in this list who I would label as 

21    people interested in statistical methodology with some 

22    knowledge of sample surveys and censuses, but that's not 

23    their principal area of focus, people like John Pratt, Jay 

24    Kadane, Albert Madansky and John Rolph, the person we 

25    mentioned a moment ago.
                                                              1477

 1               Then remember we talked about the need for people 

 2    with expertise in sample surveys, and we included Wayne 

 3    Fuller, who is an expert actually on measurement error in 

 4    general and on sample surveys and measurement error in that 

 5    context; Benjamin King, Ivan Fellegi from statistics Canada, 

 6    the deputy chief statistician there, and Joseph Waxberg, who 

 7    was involved with many aspects of the design and evaluation 

 8    of the current population survey when he served as a member 

 9    of the senior staff of the Census Bureau in the fifties, 

10    sixties and early seventies.

11               Then there were people with application interests 

12    in the social sciences.  We had a couple of demographers, 

13    Ansley Coale and Alberto Palloni.  Pastora Cafferty was 

14    interested in the application of census to study minority 

15    issues.

16               Donald Deskins is a geographer and geography is 

17    of particular interest and concern in the census and, in 

18    fact, the Census Bureau was developing new approaches to use 

19    geography in 1980, and actually the National Research 

20    Council insisted that we have at least one.  We had to go 

21    back and forth because they wanted actually to put a second 

22    geographer on and so on.

23               We had two experts, clear experts on 

24    census-taking in different settings who were also familiar 

25    with administrative records; Ivan Fellegi, who had actually 
                                                              1478

 1    worked on the application of administrative records in the 

 2    Canadian census context, and Joseph Waxberg.

 3               And there is also Courtney Slader, she is an 

 4    economist and she served as the assistant secretary at the 

 5    Department of Commerce during the taking of the 1980 census 

 6    and it was to her that the director of the Census Bureau 

 7    reported in 1980.

 8         Q.    On the adjustment decision in 1980?

 9         A.    On the adjustment decision in 1980 that was made 

10    in the Census Bureau, yes.

11         Q.    Did Ms. Slater make the decision in 1980 on 

12    adjustment?

13         A.    No, she did not.  The decision in 1980 was made 

14    by the director of the Census Bureau, Vincent Barabba.

15         Q.    I see.

16               Have you finished your brief description of the 

17    relevant expertise of the panel members?

18         A.    Well, those are the areas.

19               Then, of course, there was the diversity of views 

20    on the issue of adjustment.

21         Q.    Tell the court about that.

22         A.    Well, I think it's essentially the same that the 

23    majority of the panel had never --  well, they clearly had 

24    never expressed a view on the issue and the majority were, 

25    to any extent that I could determine, essentially neutral.
                                                              1479

 1               There were individuals appointed to this panel 

 2    with the express purpose of including people who had 

 3    publicly made statements about the issue of adjustment and 

 4    its inappropriateness or were in the process of making such 

 5    statements, and it was Ansley Coale I'm thinking of in 

 6    particular who I believe was an expert witness testifying 

 7    against adjustment in the 1980 litigation.

 8               Ivan Fellegi also had expressed significant 

 9    reservations with the techniques and doubt that it was 

10    appropriate to proceed with their use in this context, and 

11    he was actually quite vocal about it at our first meeting.

12               We also had someone who testified in favor of 

13    adjustment, Joseph B. Kadane in 1980.

14         Q.    To your knowledge, Dr. Fienberg, did any other 

15    members of the panel testify against correction in any of 

16    the census litigations?

17         A.    Well, the panel continued on for a number of 

18    years, and after it submitted this particular report, the 

19    bicentennial census, a number of individuals had said that 

20    the amount of time they had to put in was too extensive and 

21    they couldn't afford to give so much of their time to this 

22    activity in the coming years, and Pastora Cafferty went off, 

23    Alberto Palloni went off, and they were replaced by others. 

24               And during that second phase we, the committee 

25    added to the panel James Trussell, a demographer at 
                                                              1480

 1    Princeton University who I believe also testified against 

 2    adjustment in the 1980 litigation.

 3         Q.    Dr. Fienberg, I am going to skip forward just a 

 4    little bit, but in 1987 did the panel issue a recommendation 

 5    on the adjustment methodology?

 6         A.    Yes, it did.

 7         Q.    And just in a nutshell, what was that 

 8    recommendation?

 9         A.    Well, the panel had reviewed the work of the 

10    Census Bureau, both the research activities and also the use 

11    of these techniques in the field in select test censuses.

12               In particular I remember the evaluation of the 

13    matching techniques, the computerized matching technique 

14    introduced.  That was evaluated in the test census of 1985 

15    in Tampa.  We actually looked at computer printouts that Mat 

16    Jaro did as part of his work in applying the new techniques 

17    in 1985.

18               We also looked at the Los Angeles test of 

19    adjustment related operations, which was an attempt to 

20    integrate all of the changes and developments that had been 

21    done up to that point in a direct field test in Los Angeles, 

22    and we looked at sources of error in the application of the 

23    post-enumeration survey methodology in the Los Angeles test 

24    census and we concluded that the post-enumeration survey was 

25    --  and this was a code word --  technically feasible, and 
                                                              1481

 1    we encouraged the Census Bureau to go forward and actually 

 2    formally request specific funds for this survey because of 

 3    the high likelihood of the success in 1990 if used to 

 4    correct the census counts.

 5         Q.    Was that recommendation you just described a 

 6    unanimous recommendation of the decennial census panel?

 7         A.    Yes, it was.

 8         Q.    So by the time that recommendation was issued, 

 9    even those individuals who had earlier expressed adverse 

10    views towards correction joined in the recommendation?

11         A.    Yes, they did.  

12               MR. SITCOV:  Objection.  That is hopelessly 

13    leading, your Honor, the question.  

14               THE COURT:  Overruled.

15               The answer is yes? 

16               THE WITNESS:  Yes, they did.

17         Q.    Dr. Fienberg, were you a regular member of the 

18    decennial panel?

19         A.    I was a member of the panel in an ex officio 

20    capacity by virtue of my position as chair of the parent 

21    committee, and because of the interest I had in the general 

22    topics that were included I became, in essence, a regular 

23    member of the panel.  I began by attending the meetings of 

24    the panel and I continued working with the panel, with the 

25    exception of that period of time when I went away on 
                                                              1482

 1    sabbatical to California, where I didn't attend meetings, 

 2    but where I reviewed panel materials and commented on draft 

 3    materials for the panel's report.

 4               The panel continued on for a number of years, and 

 5    my term as chair of the committee actually was due to expire 

 6    at the end of June 1987, and at that time I was appointed as 

 7    a regular member of the panel so that my participation could 

 8    continue, moving from ex officio to regular status.

 9         Q.    So did the panel review the Bureau's plans for 

10    correction of the 1990 census?

11         A.    The panel reviewed the Bureau's plans.  At 

12    several stages along the way, moving from the end of this 

13    big report through --

14         Q.    That is Plaintiff's Exhibit 2?

15         A.    Plaintiff's Exhibit 2, the bicentennial census.

16               --  and actually issued two different letter 

17    reports on the activities and the evaluation of the planning 

18    activity of the Census Bureau.  The first of those was 

19    submitted in November 1986, and then the second in the 

20    spring of 1987.

21         Q.    Would you turn to the document behind tab 4 in 

22    your binder number 1.  That document has been marked for 

23    identification as Plaintiff's Exhibit 4.

24               Could you please describe it?

25         A.    Yes.
                                                              1483

 1               This is the first of those letter reports.  It is 

 2    signed by the chair of the panel on behalf of the entire 

 3    panel and directed to the director of the Census Bureau, Dr. 

 4    John Keane.  And in it the panel commented on the progress 

 5    that the Census Bureau had made, especially in the area of 

 6    computer matching capability, and the planning that it was 

 7    doing on an improved post-enumeration survey, and we urged 

 8    the Bureau to go forward with its request for funding for a 

 9    full scale post-enumeration survey.

10         Q.    And this letter was issued after the results of 

11    the test censuses were reported in 86?

12         A.    This was explicitly done after we reviewed the 

13    test census of 1985 from Tampa, whose results became 

14    available in 1986 to us.

15         Q.    And this letter is dated November of 1986?

16         A.    Yes, it is.  

17               MS. BARRY:  Your Honor, I would request at this 

18    time that the document marked as Plaintiff's Exhibit 4 be 

19    received into evidence.  

20               MR. SITCOV:  No objection.  

21               THE COURT:  4 is admitted.

22               (Plaintiff's Exhibit 4 marked for identification 

23    was received in evidence.) 

24               THE COURT:  Might this be a good time to break 

25    for lunch?  
                                                              1484

 1               MS. BARRY:  It would be fine with me, your Honor.  

 2               THE COURT:  All right.

 3               Let's recess and resume at 2:15.

 4    

 5    

 6    

 7               (Luncheon recess) 

 8    
      

 9    
      

10    
      

11    
      

12    
      

13    
      

14    
      

15    
      

16    
      

17    
      

18    
      

19    
      

20    
      

21    
      

22    
      

23    
      

24    
      

25    
      
                                                              1485

 1                         AFTERNOON SESSION
                           _________________

 2                             2:15 p.m.

 3    STEPHEN E. FIENBERG      resumed

 4    DIRECT EXAMINATION (continued) 

 5    BY MS. BARRY:

 6         Q.    Dr. Fienberg, when we broke for lunch, you were 

 7    telling me about the work that the decennial census panel 

 8    did.  I am going to ask you a few more questions about that.  

 9               Did the panel review the bureau's work on 
                                                           

10    post-stratification?

11         A.    Yes, it did.

12         Q.    Did the panel review the bureau's work on 

13    smoothing?

14         A.    It reviewed the work that had been done up 

15    through 1987, yes.

16         Q.    Did it review the bureau's work on identifying 

17    the sources of nonsampling error in the PES?

18         A.    Yes, it did.

19         Q.    Did it review the bureau's work on its plans for 

20    evaluating the 1990 PES?

21         A.    Yes.  We actually saw preliminary versions of 

22    several of what came to be known later as the P studies.

23         Q.    Did it review the bureau's work on matching?

24         A.    Yes, it did.  In particular, I mentioned the work 

25    in connection with the evaluation of the 1985 tests, but 
                                                              1486

 1    also again in the 1986 Los Angeles tests.

 2         Q.    Did it review the bureau's work on missing data 

 3    and imputation?

 4         A.    Yes, it did.

 5         Q.    Did it review the bureau's work on sampling plans 

 6    for the 1990 PES?

 7         A.    Yes, it did.

 8         Q.    Did it review the bureau's work on refining the 

 9    design of the methodology that had been employed in the 1980 

10    PEP?

11         A.    Yes, it it did.

12         Q.    Did the panel come to the conclusion about the 

13    quality of the work that the the bureau had done in 

14    formalizing its plans for the 1990 PES?

15         A.    Yes.  The possible concluded that the work was of 

16    high quality.  It was on that basis that we encouraged the 

17    census bureau to seek, we actually recommended that the 

18    census bureau seek full funding for the post-enumeration 

19    survey.  At that time we expressed our belief that if the 

20    activities were to be continued, there was a very strong 

21    likelihood that the post-enumeration survey would be 

22    successful in 1990 in producing corrections to the raw 

23    enumeration.

24         Q.    Are you familiar with the bureau's work in other 

25    areas regarding sources of nonsampling error?
                                                              1487

 1         A.    Yes, I am.  Earlier on this morning I made 

 2    reference to the work that the bureau had done in evaluating 

 3    nonsampling error in the current population survey, linked 

 4    to employment and unemployment statistics.  There was also 

 5    the work that the bureau had done in evaluating nonsampling 

 6    errors in the national crime survey, in the survey of income 

 7    and program participation, but also in several other bureau 

 8    surveys.  In fact, without doubt the census bureau is the 

 9    premier statistical organization in terms of focus on issues 

10    of nonsampling.

11         Q.    Are you familiar with the bureau's work in 

12    conducting sample surveys other than relating to the 

13    post-enumeration survey?

14         A.    Yes, I am.  The census bureau actually introduced 

15    the modern methods of sample surveys into the taking of 

16    large-scale sample surveys in the 19230s and has been the 

17    world leader yet again in the area of sampling methodology 

18    for large-scale surveys throughout the intervening period.

19         Q.    Overall, do you have an opinion about the quality 

20    of the bureau's work in the areas of conducting sampling 

21    surveys and in the area of evaluating nonsampling error in 

22    sample surveys?

23         A.    In general, it is very high quality work.

24         Q.    Dr. Fienberg, did you say that after reviewing 

25    the results of TARO, the decennial panel issued a written 
                                                              1488

 1    recommendation?

 2         A.    Yes, it did.

 3         Q.    Would you turn to Tab 6.  It should be in your 

 4    volume 2.  It is a document that has been previously 

 5    admitted into evidence as Plaintiff's Exhibit 6.  Can you 

 6    tell us what that document is?

 7         A.    Yes.  This is the letter report that the panel 

 8    submitted over the signature of its chair, which had become 

 9    Benjamin King in the time between the last letter report we 

10    discussed and the present one.  

11               In it the panel took special note of the progress 

12    that the census bureau had made since the previous report, 

13    and in particular commented on its review of the 1986 test 

14    of the adjustment related operations.  It used the term that 

15    I mentioned before and tried to explain "technically 

16    feasible."  That is, the bureau actually took the methods, 

17    they went out to the field, they applied them, and they were 

18    evaluated them.  They were able to measure errors and they 

19    were able to draw conclusions from the results.  

20               We recommended that that work continue and that 

21    adequate funds be provided for the full-scale survey and to 

22    prepare for the possibility of adjustment in 1990 on the 

23    basis of that survey.

24         Q.    Did the panel reach a judgment as to whether the 

25    bureau had solved the major problems concerning the 
                                                              1489

 1    methodology that had been experienced with the 1980 PEP?

 2          A.    Yes.  The panel reviewed with the census bureau 

 3    what it believed to be the major sources of error in 1980.  

 4    We had extensive discussions about alternative ways of 

 5    looking at those sources of error.  We talked about grouping 

 6    them in other ways other than the census bureau did.  

 7               In the end, we found that the way in which the 

 8    census bureau was doing that grouping and their ability to 

 9    measure those sources of error and bring them under what 

10    statisticians usually call statistical control -- that is, 

11    they are measurable and the size of the errors are 

12    sufficiently small to make the kinds of conclusions for 

13    which the surveys were designed. 

14         Q.    Did the panel reach a judgment as to the quality 

15    of the design of the methodology to be employed in 1990?

16         A.    Yes.  The panel believed that the design of the 

17    post-enumeration survey was appropriate for measuring 

18    undercount in the 1990 census as well as for the broader 

19    task of what the census bureau usually refers to as coverage 

20    evaluation, the more expanded set of activities, trying to 

21    understand the census and ways in which it could be 

22    improved.

23         Q.    Did the panel conclude that the PES was well 

24    designed?

25         A.    Yes, it did.
                                                              1490

 1         Q.    Up until the time of the panel's May 1987 

 2    recommendation, did the panel receive the full cooperation.  

 3    Census bureau in its work?

 4         A.    Yes.  We not only received the full cooperation, 

 5    but there was a very active interchange between panel 

 6    members and members of the staff at the bureau doing 

 7    research.  People from the bureau came at our request, 

 8    presented updates on their work.  They provided all the 

 9    documents we wanted, often five or six times as many as we 

10    asked for.  But usually we got everything we asked for and 

11    more.  

12               They worked with us, responded to questions.  In 

13    fact, took suggestions back into their day-to-day 

14    activities, and we could see the impact of those 

15    suggestions.  They didn't always follow everything the panel 

16    suggested, they went in different directions.  But there was 

17    a very good exchange and full cooperation, I think is the 

18    best description.

19         Q.    Did there come a time when the census bureau's 

20    cooperation ceased?

21         A.    Yes, there did.

22         Q.    Tell us about that.

23         A.    The panel held a meeting in late 1988, at which 

24    time we were reviewing a set of experimental programs that 

25    the bureau was considering mounting as part of its 
                                                              1491

 1    activities in 1990 to learn about new things that might be 

 2    of use in future censuses.  In discussing those experimental 

 3    research programs, several members of the panel explicitly 

 4    asked for updates on what the census bureau was planning to 

 5    do for coverage evaluation.  That is, they said, look, we 

 6    can't understand how an experiment will improve upon what's 

 7    going on in the census unless we have some measuring 

 8    instrument that allows us to gauge the nature of that 

 9    improvement.  

10               The staff of the census bureau who were present 

11    at that meeting told us that they could not share that 

12    information.  

13               MR. SITCOV:  Objection, your Honor.  He is 

14    talking clearly now from hearsay.

15         Q.    Were you present at this meeting, Dr. Fienberg?

16         A.    Yes, I was.

17         Q.    These were statements made by census bureau 

18    employees?

19         A.    Yes, they were.  

20               MS. BARRY:  Your Honor, I would think that they 

21    would qualify as admissions.  

22               MR. SITCOV:  Admissions of what?  

23               MS. BARRY:  By census bureau employees.  

24               THE COURT:  By people who had authority to 

25    perform the act about which they were speaking.  
                                                              1492

 1               MR. SITCOV:  But they are not admitting anything.  

 2               THE COURT:  We haven't heard yet.  

 3               MR. SITCOV:  I think that there isn't any doubt 

 4    from the question what it is that Dr. Fienberg is supposedly 

 5    going to testify that the hearsay was about.  Certainly, 

 6    there has been no testimony that it was by people who had 

 7    authority to do anything of that type.  

 8               THE COURT:  I think you went to law school before 

 9    801(d)(3) was put in the rules.  

10               MR. SITCOV:  I'm old, but not that old.  

11               THE COURT:  You are giving me a New York 

12    argument.  Your rules are the New York rules, they are not 

13    the federal rules.  Your objection is therefore overruled.

14         Q.    Could you continue, Dr. Fienberg?

15         A.    I was trying to pay attention over here.  

16               We asked for this information about the post- 

17    enumeration survey.  The staff at the census bureau who were 

18    there to present material to us told us that they were not 

19    allowed to share any information about the post-enumeration 

20    survey with us because of the pending litigation in 

21    connection with it.

22         Q.    What happened to the panel's work after that?

23         A.    At that point the panel sent a letter that we 

24    drafted together to the chair of the parent committee, the 

25    committee on national statistics, indicating that, in our 
                                                              1493

 1    judgment, we could not be of any further value to the census 

 2    bureau in evaluating the experimental research programs or 

 3    any other aspect of the 1990 census unless we had access to 

 4    information on the post-enumeration survey and the other 

 5    coverage improvement programs that they planned to have in 

 6    place.  

 7               At that point the committee, the panel, stopped 

 8    meeting.

 9         Q.    Did there come a time when the panel would have 

10    been permitted renewed access to information relating to 

11    adjustment methodology?

12         A.    In the summer of 1989 there was the stipulation 

13    and order in the litigation.

14         Q.    In this litigation?

15         A.    In this litigation.  At which time the reason 

16    that we had been given for the lack of sharing of 

17    information would no longer have been one that would have 

18    been valid, I guess.  And at that time the committee staff 

19    began to discuss with the census bureau the renewing of the 

20    panel's activities.

21         Q.    At that time was the panel prepared to resume its 

22    activities as adviser to the census bureau on issues related 

23    to adjustment in 1990?

24         A.    Yes, it was.

25         Q.    What happened?
                                                              1494

 1         A.    At a meeting of the Committee on National 

 2    Statistics in October of 1989, I have to remember his title, 

 3    it is Deputy Assistant Under Secretary or is it Assistant 

 4    Under Secretary Mark Plant came before the Committee on 

 5    National Statistics at its regular fall meeting to report on 

 6    census activities.  

 7               He discussed with the committee the formation of 

 8    the secretarial advisory committee, the group of eight 

 9    advisers.  When asked by several members of the committee 

10    about the resumption of the panel's activities, he indicated 

11    explicitly that the commerce department would no longer fund 

12    any activities of the panel that dealt with coverage 

13    evaluation or the post-enumeration survey because he had 

14    enough other advisers to turn to on that issue.

15         Q.    Did there come a time when the committee 

16    responded to that statement by Under Secretary Plant?

17         A.    Yes, there did.  In a letter conveying the 

18    committee's reactions, the chair, Burton Singer, sent this 

19    letter to Under Secretary Michael Darvey.  He in that letter 

20    reviewed very briefly the history of the panel.  He reviewed 

21    the comment that Mr. Plant made at the occasion of the 

22    committee's meeting, and he indicated that the panel 

23    believed it could no longer function advising the census 

24    bureau.  Therefore, he conveyed the information to Dr. 

25    Darvey that the panel was being disbanded.
                                                              1495

 1         Q.    So the letter was sent to whom?

 2         A.    The letter was sent to Under Secretary of 

 3    commerce Michael Darvey.

 4         Q.    Could you turn to Tab 8 in binder 2.   It is a 

 5    document that has been marked for identification as 

 6    Plaintiff's Exhibit 8.  Could you tell the Court what that 

 7    document is?

 8         A.    Yes.  This is the letter from Burton Singer, the 

 9    chair of the Committee on National Statistics, to Mr. 

10    Darvey, conveying the information I just described.  

11               MS. BARRY:  Plaintiffs would request that the 

12    document marked as Plaintiff's Exhibit 8 be admitted into 

13    evidence at this time, your Honor.  

14               MR. SITCOV:  I am going to have to object, your 

15    Honor.  It is unsigned.  It is not by or to the witness.  It 

16    is not a census bureau document.  I have no way of 

17    determining whether this was sent or if it is the one that 

18    was sent or some other version of it.

19         Q.    Dr. Fienberg, do you know if this document was 

20    sent, if this letter was sent to Michael Darvey?

21         A.    I was present during the drafting of this letter 

22    by the committee.  It is my understanding that it was sent.  

23    I wasn't in Mr. Darvey's office when the mail was delivered.  

24    I do know that he was fully aware of the reasons for the 

25    disbanding of the panel, since I had a conversation with him 
                                                              1496

 1    subsequently about it.  

 2               MS. BARRY:  Your Honor, as this was addressed to 

 3    one of the defendants in the case, I would suggest that it 

 4    is clearly admissible, at least with respect to the state of 

 5    mind of the commerce department.  

 6               THE COURT:  We have no evidence, though, that it 

 7    was ever mailed.  We have got evidence that the recipient 

 8    knew something about its contents, but no evidence that this 

 9    actual letter was actually posted to Mr. Darvey.  It is an 

10    authentication problem too.  There is no signature on here.  

11    BY MS. BARRY:

12         Q.    Dr. Fienberg, are you aware that Dr. Darvey knew 

13    about the contents of this letter?

14         A.    Yes, I am.

15         Q.    How are you aware of?

16         A.    Because there were subsequent meetings between 

17    Mr. Singer, Dr. Singer, and Mr. Darvey pursuant to the rest 

18    of the document, in which they discussed not only the 

19    disbanding of the current panel but how the committee would 

20    proceed to continue and constitute a new panel.  This is in 

21    the second paragraph on the second page, which would look at 

22    the range of issues that Mr. Plant had raised and that the 

23    commerce department had come to the committee and asked them 

24    to address.  

25               MR. SITCOV:  Whether or not Dr. Darvey was aware 
                                                              1497

 1    of the documents that may also have been discussed in this 

 2    letter doesn't authenticate this letter.  

 3               THE COURT:  Thank you for sharing that with me.  

 4               I'm afraid, Ms. Barry, you don't have enough to 

 5    authenticate the letter.  I am going to sustain the 

 6    objection.  

 7               MS. BARRY:  We will move on, your Honor.

 8         Q.    Dr. Fienberg, does the PES methodology make 

 9    assumptions?

10         A.    Of course.  Any statistical method, all 

11    statistical methodology makes assumptions of one sort or 

12    another.  That is part and parcel of the tools of the trade.

13         Q.    Do you believe those assumptions are all strictly 

14    true?

15         A.    Strictly true?  

16               THE COURT:  Relatively?

17         A.    No assumption is strictly true.  The role of 

18    assumptions in statistical methods and models doesn't have 

19    anything to do with being strictly true.  Assumptions are 

20    ways that statisticians have of shaping models and 

21    attempting to provide somewhat simplified descriptions of 

22    the world.  And in that sense they couldn't be true.  

23               The big question is not whether they are true but 

24    the effect of departures from them on the things that you 

25    are interested in describing or predicting with the models.
                                                              1498

 1         Q.    In your opinion, if one were to require the 

 2    perfect validity of assumptions, what would that be?

 3         A.    It would be a bad day for statistics.  Everything 

 4    that statisticians do, all of the methodology that we apply 

 5    day after day, that the federal government applies, even in 

 6    the conduct of the census, would all be forced to be set 

 7    aside because we knew the things were not strictly true that 

 8    you assume in order to go out and collect data and measure 

 9    things. 

10         Q.    In your opinion, what is the appropriate way to 

11    evaluate assumptions?

12         A.    As I indicated before, what the statisticians do 

13    is create models.  They have some assumptions with them.  

14    The big question is how these models are used, the impact 

15    that departures from those assumptions might have on the 

16    quantities of interest.  If they are concerned, which 

17    sometimes -- 

18               Look, some assumptions we know don't matter very 

19    much.  They are there for technical convenience.  There are 

20    other ones that sometimes you think may matter more, and 

21    those you look at and you check some sensitivity of the 

22    results to some clearly sensible alternatives in the context 

23    of what you know about the problem that you are dealing with 

24    and what it is you want to measure and how you can gauge 

25    that impact.
                                                              1499

 1         Q.    Did the bureau conduct that kind of evaluation in 

 2    connection with the 1990 PES?

 3         A.    First of all, that is what the post-enumeration 

 4    survey is all about.  It is a survey that is there to detect 

 5    departures from the kinds of assumptions that go with the 

 6    notion that you can go out and take a complete census 

 7    enumeration, something that we have demonstrable evidence to 

 8    show that that assumption is false.  

 9               The post-enumeration survey is exactly the tool 

10    that was designed to measure specific kinds of departures 

11    from the notion of a complete census.  

12               Then the census bureau went one step deeper.  

13    What it did was it created a set of studies and a set of 

14    explorations of what was involved in the post-enumeration 

15    survey.  They were designed to see if the post-enumeration 

16    survey was truly sensitive to the kinds of sensible 

17    departures that one would expect to occur on the basis of 

18    knowledge of that type of survey, the kind of thing it was 

19    designed to measure, and what you knew about the components 

20    and how they had been used in other settings.  

21               For example, in test censuses and the dress 

22    rehearsal and the like.  That is what the census bureau did.  

23    They did exactly that kind of checking on assumptions in a 

24    systematic way to see if it had the kind of effect on 

25    conclusions that would make one have some pause.
                                                              1500

 1         Q.    As an overall matter, what is your evaluation of 

 2    the bureau's work in conducting the PES and in evaluating 

 3    the PES?

 4         A.    In 1987, when I looked really carefully at the 

 5    methodology being used in the Los Angeles test, I was 

 6    impressed, because the bureau had done a lot of good work in 

 7    bringing under control the measures of major sources of 

 8    nonsampling error.  But there hadn't been the full-scale 

 9    tests, the post-enumeration survey hadn't gone into the 

10    field.  The census, the full census, wasn't going on.  And 

11    when you go from that to the large, at each step along the 

12    way it is possible that there could be some degradation.  

13               When I saw the results of the 1990 

14    post-enumeration survey, I was actually astounded, because 

15    the quality of what I saw exceeded my expectations and the 

16    expectations of others with whom I discussed the survey back 

17    in 1987.  It was very, very high quality.

18         Q.    Dr. Fienberg, did the census bureau reach a 

19    conclusion about the relative accuracy of uncorrected census 

20    counts compared to census counts supplemented with the 

21    results of the 1990 PES?

22         A.    Yes, it did.

23         Q.    What was that conclusion?

24         A.    The census bureau concluded that the corrected 

25    census counts based on the post-enumeration survey were more 
                                                              1501

 1    accurate than the raw census enumeration counts.  

 2         Q.    Do you agree with the bureau's conclusion?

 3         A.    Yes, I do.

 4         Q.    Did the bureau reach a judgment as to whether or 

 5    not supplementation of the 1990 census counts with the 

 6    results of the 1990 PES would reduce the differential 

 7    undercount?  

 8               MR. SITCOV:  Again, your Honor, absent a showing 

 9    of the Rule 801(d)(2), the witness is clearly speaking about 

10    things he ought not to speak with.  There has been no such 

11    showing.  

12               THE COURT:  The showing is that he got this 

13    information from people at the census bureau?  

14               MR. SITCOV:  That's right, that were authorized 

15    to make those statements.  

16               MS. BARRY:  If necessary, we could turn, if you 

17    want, to the recommendations of Dr. Bryant or to the 

18    Undercount Steering Committee recommendations.

19         Q.    Let me ask you, have you read those documents, 

20    Dr. Fienberg?

21         A.    Yes, I have.

22         Q.    In those documents did the bureau reach a 

23    conclusion about whether or not correction of the 1990 PES 

24    would reduce the differential undercount?

25         A.    Yes, it did.  
                                                              1502

 1               MR. SITCOV:  Is the question did the bureau or 

 2    did Dr. Bryant?  

 3               THE COURT:  Dr. Bryant.  

 4               THE WITNESS:  Both the Undercount Steering 

 5    Committee and Dr. Bryant reached the conclusion that the 

 6    corrected census counts are superior to the raw census 

 7    enumeration.

 8         Q.    Director Bryant is the director of the sends 

 9    bureau, is that correct?

10         A.    That is correct.  

11               THE COURT:  Still is?  

12               THE WITNESS:  Yes, she is, sir.  

13         Q.    Back to our last question.  Did the bureau also 

14    reach a conclusion about whether or not correction could 

15    reduce the differential undercount?

16         A.    Yes, it did.

17         Q.    What was that conclusion?

18         A.    Both the Undercount Steering Committee and the 

19    director concluded that the differential undercount between 

20    minorities and nonminorities which had plagued the census 

21    for decades would be reduced in an accurate fashion by the 

22    use of the corrected census counts.

23         Q.    Do you agree with that conclusion?

24         A.    Yes, I do.

25         Q.    Dr. Fienberg, how long has there been a 
                                                              1503

 1    differential undercount?

 2         A.    I guess as long as we have been taking censuses.  

 3    If we go back to the first census of 1790, in a comment on 

 4    it Thomas Jefferson explicitly made reference to the 

 5    underenumeration and in particular to that associated with 

 6    slaves.  Then the first explicit quantitative information 

 7    that I am aware of -- there were some debates in the 1940s 

 8    about related issues.  

 9               But the first quantitative information I am aware 

10    of was a comparison using data from the 1860, 1870, and 

11    1880' censuses done by Francis Walker and published in an 

12    article of the journal of the American Statistical 

13    Association in 1890.  

14               In that article Francis Walker showed that, in 

15    fact, using the information from the multiple censuses, that 

16    there was a differential undercount between blacks -- I 

17    think it said coloreds and whites in the article itself -- 

18    in fact, an undercount roughly of the same magnitude that we 

19    have come to expect in more recent years.

20         Q.    Who was Francis Walker Francis Walker was the 

21    superintendent of the census for both the 1870 and 1880 

22    censuses, the equivalent of the director of the bureau.  But 

23    there wasn't a bureau back then.  And he continued in that 

24    capacity throughout the 1890 -- I'm sorry -- 1880s.  He was 

25    also president of the American Statistical Association, I 
                                                              1504

 1    think about the same time also president of MIT.

 2         Q.    Dr. Fienberg, could you turn to Tab 11 in your 

 3    binder number 3.  Could you tell the Court what the document 

 4    is behind Tab 11, which has been marked for identification 

 5    as Plaintiff's Exhibit 11.

 6         A.    This is the article by Francis Walker describing 

 7    the undercount associated -- differential undercount 

 8    associated with the census of 1870.

 9         Q.    Dr. Fienberg, do you recognize this as a copy of 

10    the article contained in the American Statistical 

11    Association journal series numbers 1112 in 1890?

12         A.    Yes.  I actually had occasion to go back an 

13    examine this in some earlier forms when I was doing a 

14    history of the journal in its early years.  

15               MS. BARRY:  Your Honor, plaintiffs request at 

16    this time that Plaintiff's Exhibit 11 be admitted into 

17    evidence.  

18               MR. SITCOV:  We have no objection to admitting 

19    this to prove that there was, in fact, an undercount in the 

20    1870 census.  

21               THE COURT:  Without objection, I will take it in. 

22               (Plaintiffs' Exhibit 11 for identification was 

23    received in evidence)

24         Q.    Dr. Fienberg, I forgot to ask you, after the 

25    meeting that the Committee on Statistics had with Mark Plant 
                                                              1505

 1    that you told us about a little bit earlier, did the panel 

 2    cease meeting?

 3         A.    Yes, it did.  It was disbanded.

 4         Q.    Why was that, again?

 5         A.    The panel was disbanded because it says that 

 6    there was no purpose in its continuing if it was not allowed 

 7    to look at the coverage evaluation programs of the 1990 

 8    census.

 9         Q.    Thank you, Doctor.  

10               MS. BARRY:  Your Honor, I am going to start 

11    asking some questions about the Secretary's decision dated 

12    July 15, 1991.  For the Court's convenience, I provided a 

13    loose copy of what has already been admitted into evidence 

14    as Plaintiff's Exhibit 9 and is a part of the so-called 

15    administrative record.

16         Q.    Dr. Fienberg, you have a version there too.  Have 

17    you read the Secretary's decision rejecting the bureau's 

18    conclusion about correction in 1990?

19         A.    Yes, I have.

20         Q.    Have you familiarized yourself with the basis of 

21    the Secretary's decision to overrule the census bureau's 

22    conclusion that correction of the 1990 census counts with 

23    the results of the 1990 PES would result in more accurate 

24    census counts?

25         A.    Yes, I am.
                                                              1506

 1         Q.    Have you reviewed the reasons set forth by the 

 2    Secretary for rejecting the bureau's conclusions?

 3         A.    Yes, I have.

 4         Q.    Do you have a general opinion about the 

 5    Secretary's decision?

 6         A.    Yes, I do.

 7         Q.    Would you share it with us, please.

 8         A.    I believe that the reasons that the Secretary 

 9    gave and the technical bases on which he relied in those 

10    reasons are insufficient to reject the conclusion set 

11    forward by the census bureau that the corrected census 

12    counts are superior to the raw enumeration.

13         Q.    If you could turn to page 1-1 of the decision.  

14    Let me read the first sentence of the second paragraph.  "If 

15    we change the counts by a computerized statistical process, 

16    we abandon a 200-year tradition of how we actually count 

17    people."  

18               In your opinion, Dr. Fienberg, is that an 

19    accurate statement?

20         A.    No, it is not.  There are three parts to this.  I 

21    am going to combine two of them.  

22               First of all, although there is a 200-year 

23    tradition in the census, it certainly isn't a tradition of 

24    how we actually count people.  Then, as I will indicate, as 

25    part of that tradition the computerization is deeply 
                                                              1507

 1    embedded already and has been for literally over a hundred 

 2    years.  

 3               If we go back to the beginning of the census, the 

 4    first census ever taken, this was a census that was taken by 

 5    federal marshals.  They went out and they made lists of the 

 6    numbers of people in different places that is what census 

 7    taking was all about in 1790 and, as Thomas Jefferson 

 8    indicated in his letter, there were errors.  

 9               That kind of procedure continued for the next 

10    several censuses, with marshals continuing to collect 

11    information and the lists continuing to be compiled, until 

12    the disputes that broke out during the 1840s over the 

13    accuracy of the census.  As a consequence of a 

14    recommendation of a committee of the American Statistical 

15    Association, chaired by Lemuel Shadduck, Congress agreed to 

16    change the nature of the census.  

17               Instead of being a count of households and a 

18    simple list that it was before, the census began to collect 

19    information on individuals, and a centralized activity was 

20    set up in Washington in order to introduce some quality 

21    checks into the data that were being gathered from the 

22    field.  Up to that time there were no such checks.  

23               There was a continuation for the next several 

24    censuses with this inclusion of some professional activity, 

25    especially under the leadership of Francis Walker.  In the 
                                                              1508

 1    1880s he actually got a colleague from MIT to develop 

 2    something for use directly in the census, that is, the punch 

 3    card.  

 4               Herman Hollerith developed the punch card 

 5    explicitly so that the census bureau could do its 

 6    tabulations for the 1890 census in a quick and timely 

 7    fashion.  That is the Hollerith card that was the basis for 

 8    the creation of IBM.  There we have the beginning of 

 9    computerization.  

10               The census bureau was created in 1902.  We skip 

11    ahead to the 1930s.  

12               THE WITNESS:  This won't take all that long.

13         A.    In the 1930s the census bureau introduced the 

14    idea of sampling into its activities.  

15               THE COURT:  19?

16         A.    In the 1930s.  And in the 1940 census, as a 

17    result of those activities, it introduced the first use, 

18    major use, of sampling in census takings with the 

19    introduction of the long form sampling a fraction of the 

20    population of the United States.  That same census saw the 

21    introduction of special quality control checks that were 

22    linked to the new statistical methods being developed by the 

23    bureau.  

24               In the 1950s we saw the introduction of the first 

25    coverage improvement program directly attempting to evaluate 
                                                              1509

 1    the census.  In 1960 the census changed how it did 

 2    enumeration yet again and experimented in a partial way and 

 3    then in a full way in 1970 when it introduced mail out-mail 

 4    back.  So census enumerators stopped going to the door of 

 5    every household in the United States.  

 6               In that same census of 1970 the census bureau 

 7    used sampling in the national vacancy check and the post 

 8    office check, and introduced imputation in a formal way into 

 9    its activities.  

10               This record of tradition of methodological 

11    improvement continued through the next two decades, and as 

12    it planned for the 1990 sends the census bureau planned to 

13    introduce a major new geographic coding scheme which it did 

14    successfully introduce.  This may have been mentioned 

15    earlier.  It is called TIGER, not the animal but it's a 

16    topographical integrated something something computer 

17    mapping system.  

18               It used distributive computing, putting computers 

19    in the district offices so that it could better process the 

20    census, and it planned for the use of a post-enumeration 

21    survey for correcting the census counts in 1990.  That was 

22    the program that the census bureau put in place early on in 

23    the second, that program of improvement looking ahead to 

24    1990.

25         Q.    Are you done, Doctor?
                                                              1510

 1         A.    Yes.

 2         Q.    In your opinion, is the 1990 PES part of that 

 3    tradition of improvement you have described?

 4         A.    Most certainly, yes.

 5         Q.    Why is that?

 6         A.    Because it is part of the census bureau's 

 7    approach.  You try things out.  You explore them.  You check 

 8    to see that they work.  You build up a record of showing 

 9    that they work in test censuses.  Then you make changes that 

10    you know correct for known flaws in the procedure, in this 

11    case correcting for the differential undercount between 

12    minorities and nonminorities, the explicit reason for 

13    correction in 1990.

14         Q.    Dr. Fienberg, is there any other part of that 

15    sentence on page 1-1 that I read to you that you disagree 

16    with?

17         A.    In 1990 we certainly never counted everyone, nor 

18    in any census going back to 1790 have we ever counted 

19    everyone.  The census as an activity never counts the people 

20    in the sense that people think of going out and saying one, 

21    two, three, four, five, and so on.  Not even in the biblical 

22    sense of doing counts in the census.  

23               What the census bureau does is estimate the 

24    number of people in various geographical regions.  It has 

25    done this for years and years and years.
                                                              1511

 1         Q.    Do you know what the census bureau's position is 

 2    on whether or not the census is a head count of the 

 3    population?

 4         A.    Yes.  Over the past decade, in setting after 

 5    setting, individuals, the director of the bureau, the 

 6    associate director of the bureau, others at that level have 

 7    come into public forums and made speeches which I have 

 8    listened to and I have read and examined.  In virtually all 

 9    of those, they have explicitly acknowledged that the census 

10    is an estimate, it is not a count of the people in the 

11    United States.  

12               I recall, for example, the statement in the book 

13    by Vincent Barabba, the director of the 1980 census.  He 

14    wrote a book with two coauthors summarizing his experiences 

15    from 1980.  In that book he said that the census is an 

16    estimate.  

17               I was present as part of a seminar sponsored by 

18    the Senate Oversight Committee looking ahead to plans for 

19    1990.  In that presentation officials from the census bureau 

20    came, presented a paper, and the paper explicitly focused on 

21    all of the different aspects of estimation in the 1980 

22    census and in the 1970 census.  In fact, the title, as I 

23    recall, of several sections had that in it.  

24               Then again at the summer meeting of the annual 

25    meeting of the American Statistical Association, Barbara 
                                                              1512

 1    Bailar, the associate director of the census bureau, gave a 

 2    talk at which I was present, which was focused on the census 

 3    as an estimate.

 4         Q.    Dr. Fienberg, could you turn to Tab 13.  It is in 

 5    volume 3 of your binders.  I would ask that you identify 

 6    what has been marked for identification as Plaintiff's 

 7    Exhibit 13.

 8         A.    This is a Xerox copy of the book by Vincent 

 9    Barabba and his two coauthors about the 1980 census.

10         Q.    What is the title of that book?

11         A.    The title is The 1980 Census: Policy Making Amid 

12    Turbulence. 

13         Q.    Would you turn to page 5 of that document.

14         A.    In the chapter called "The Counting"?

15         Q.    That is correct.  

16               MS. BARRY:  Are you with us, your Honor?  

17               THE COURT:  I'm on the same page.  I dare not say 

18    I'm with you.  

19               MS. BARRY:  It depends on what standard we are 

20    applying.

21         Q.    Dr. Fienberg, I am going to read to you a section 

22    from page 5.  It is the first full paragraph on page 5.

23         A.    Yes.

24         Q.    "The number that is accepted as a true population 

25    of the United States is merely a scientific estimate.  This 
                                                              1513

 1    estimate is not a simple addition of all persons and only 

 2    those persons directly observed by census enumerators.  

 3    Indeed, the number of people in the United States cannot be 

 4    confirmed by immediate experience.  It is dependent on 

 5    complex scientific and management models and processes that 

 6    are designed to ensure the taking of an accurate census 

 7    insofar as is humanly possible."  

 8               Is that the statement by Vincent Barabba that you 

 9    referred to earlier?

10         A.    Yes, it is.

11         Q.    Do you agree with that statement?

12         A.    Yes, I do.  

13               MS. BARRY:  We are done with that one, your 

14    Honor.  

15               THE COURT:  "That one" meaning the book?  

16               MS. BARRY:  We are done.  Unfortunately, 

17    sometimes we have a lot of extra paper.

18         Q.    If you could turn to Tab 14, Dr. Fienberg,.  That 

19    should be in volume 4.  Would you please tell the Court what 

20    the document that has been marked for identification as 

21    Plaintiff's Exhibit 14 is?

22         A.    Yes.  This is a paper entitled "Counting Your 

23    Estimation In the Census, a Difficult Decision," by Barbara 

24    Bailar.  This is the manuscript version of the talk that Dr. 

25    Bailar gave at the meeting of the American Statistical 
                                                              1514

 1    Association in 1983, which I attended.

 2         Q.    Could you turn to page 3 of that document.  Let 

 3    me read a couple of lines from the first full paragraph.  

 4               "The issue is not really whether there should be 

 5    counting or estimation.  The real issue is how much 

 6    estimation there should be.  There has probably always been 

 7    some estimation in the census."  

 8               Do you agree with Dr. Bailar that there has 

 9    probably always been some estimation in the census?

10         A.    Yes, I do.  Indeed, the entire paper that Dr. 

11    Bailar delivered went on to explain the different ways in 

12    which you could see estimation in the census.  I note that 

13    even the next paragraph talks about the methods of sampling 

14    that were used in 1970 and the modeling and the estimation 

15    that were part of the taking of the 1970 census.  

16               MS. BARRY:  Thank you, Doctor.  We are done with 

17    this one as well.

18         Q.    Dr. Fienberg, could you look at the next 

19    paragraph on page 1-1 of the Secretary's decision.  Let me 

20    read the first line.  

21               MS. BARRY:  Your Honor, we probably will be 

22    asking a number of questions about the Secretary's decision, 

23    so key keep that one out.  

24               (Continued on next page) 

25    
      
                                                              1515

 1    BY MS. BARRY:
                   

 2         Q.    If you look at the second paragraph, the first 

 3    line, I will read it to you, Dr. Fienberg.

 4               "The 1990 census is one of the two best censuses 

 5    ever taken in this country." 

 6               Do you know of any standard, Dr. Fienberg, under 

 7    which the 1990 census could be described as one of the two 

 8    best censuses ever taken in this country?

 9         A.    The only standard by which I know one could make 

10    a statement like this would be using the indicator of the 

11    net national undercount which we have had estimates of since 

12    1940.

13         Q.    The national undercount is estimated by 

14    demographic analysis?

15         A.    Yes, the national undercount is estimated by 
                                                              

16    demographic analysis.  We have that from 1940 through to the 

17    present, and if you use that criterion, then, in fact, the 

18    1990 census has the second smallest such net national 

19    undercount.

20         Q.    In your opinion, Dr. Fienberg, is that an 

21    appropriate standard by which to judge the success of the 

22    1990 census?

23         A.    No, it is not.

24               The problem with using the national undercount is 

25    that it totally disguises all of the error in the census.
                                                              1516

 1               What we have from that same source of information 

 2    is knowledge that blacks have been systematically 

 3    undercounted at higher rates than nonblacks and the errors 

 4    in the census, this key error in the census which the 

 5    demographic analysis has pointed to in census after census, 

 6    if we look at it, that is the worse differential between 

 7    blacks and others in 1990 than in any of the other years for 

 8    which we have measurements.

 9         Q.    Does the Secretary mention that fact, the fact 

10    that this is the worst census in terms of differential 

11    undercount in his decision?

12         A.    No, he does not.  In fact, throughout the 

13    document --  well, the Secretary really doesn't mention the 

14    differential undercount in the document in any major way.

15               He does refer to, at the outset, undercount of 

16    different groups, but there is no reference to the history 

17    and historical trend and systematic undercount and clearly 

18    no reference to the fact that this is the worst ever in 

19    terms of the very indicator that he used to describe as the 

20    best.

21         Q.    What about the level of gross error in the 1990 

22    census?

23         A.    The level of gross errors is also disguised by 

24    the net national undercount.

25               If you had a large number of errors of people 
                                                              1517

 1    being left out and a large number of people erroneously 

 2    included in the census, then taken together they may balance 

 3    out and you could get a net national undercount that was 

 4    small, perhaps negligible.

 5               But the big question is, how are those errors 

 6    distributed across the country? 

 7               And if you look at the distribution of those 

 8    errors across the country, then the pattern that comes out 

 9    repeatedly is the difference between minorities and 

10    nonminorities and that difference is indicative of an 

11    undercount of a substantial order of magnitude.

12               And so looking at gross errors in the 1990 census 

13    is important.  In fact, in the 1990 census it's been 

14    estimated that the percentage of gross errors is --  well, 

15    there are approximately 25 million gross errors.  That 

16    amounts to over ten percent of the number of people that 

17    were claimed in the original enumeration.

18         Q.    Does the Secretary discuss that fact in his 

19    decision?

20         A.    No, he does not.

21         Q.    Dr. Feinberg, would you turn to tab 15, which is 

22    in binder 4, and could you identify the document that has 

23    been marked for identification as Plaintiff's Exhibit 15?

24         A.    Yes.  This reads appendix 17.  I believe it was 

25    included as an appendix to the Secretary's report.  It's a 
                                                              1518

 1    report on testimony delivered by L. Nye Stevens, who is an 

 2    official at the general accounting office, and it is dated 

 3    June 27, 1991.

 4         Q.    And is this a document from which one could 

 5    obtain the figures on the number of gross errors in the 1990 

 6    census?

 7         A.    Yes, it is.

 8         Q.    And can you tell us what paragraph those figures 

 9    are reported in?

10         A.    If you turn to page 4 and look at the new 

11    paragraph that begins in the middle of the page that says, 

12    "A more comprehensive approach," it ends by saying, "Using 

13    this method, on the basis of preliminary post-enumeration 

14    survey data, there were about 25.9 million gross errors in 

15    the 1990 census or about 10.4 percent of the resident 

16    population."  

17               MS. BARRY:  Your Honor, plaintiffs would request 

18    that Plaintiff's Exhibit 15 be received into evidence at 

19    this time.

20               I will note for the record that it is also part 

21    of the so-called administrative record.  

22               MR. MILLET:  Since it is so-called so-called 

23    administrative record, how can I object?  

24               THE COURT:  Plaintiff's Exhibit 15 is admitted. 

25               (Plaintiff's Exhibit 15 marked for identification 
                                                              1519

 1    was received in evidence.) 

 2    BY MS. BARRY:

 3         Q.    Dr. Feinberg, have you read the report submitted 

 4    by Ericksen, Estrada, Tukey and Wolter to the Secretary?

 5         A.    Yes, I have.

 6         Q.    For the record, that document has previously been 

 7    admitted in evidence as Plaintiff's Exhibit 195.

 8               Have you read the discussion in their report 

 9    relating to the quality of the 1990 census?

10         A.    Yes, I have.

11         Q.    Could you turn to page 3-36 of the Secretary's 

12    decision.

13         A.    I'm sorry, 3-what?

14         Q.    3-36.

15               Is this the section of the Secretary's decision 

16    where he discusses the report submitted by the members of 

17    the special advisory panel?

18         A.    Yes, it is.

19         Q.    And is this the part where he discusses the 

20    report by Ericksen, Estrada, Tukey and Wolter?

21         A.    It is.

22         Q.    Would you turn to page 3-40, and if you would 

23    look under the heading that is entitled, "Evaluation of the 

24    report on the 1990 decennial census and the post-enumeration 

25    survey." 
                                                              1520

 1               Is this the section of the Secretary's discussion 

 2    of the Ericksen-Estrada-Tukey and Wolter report where he 

 3    discusses their discussion of the quality of 1990 census?

 4         A.    Yes, it is.

 5         Q.    Do you see the first sentence, it reads, "I do 

 6    not find the discussion of the quality of the census 

 7    relevant." 

 8               Do you agree?

 9         A.    Absolutely not.

10         Q.    Would you look a little further to the immediate 

11    following sentence where the Secretary says, "Irrespective 

12    of the flaws in the census, adjustment is precluded unless 

13    the adjustment is shown to be better than the census." 

14               Do you see that, Dr. Fienberg?

15         A.    Yes, I do.

16         Q.    Are there flaws in the 1990 PES, Dr. Fienberg?

17         A.    Yes, there are.

18         Q.    Are there flaws in the 1990 census, Dr. Fienberg?

19         A.    A whole load of them.

20         Q.    In your opinion, is it possible to conclude which 

21    is more accurate, the uncorrected census or the census 

22    supplemented with the results of the 1990 PES without having 

23    an understanding of the errors in both the PES and the 

24    census?

25         A.    No, it is not.
                                                              1521

 1               In fact --  I mean, I take that as a starting 

 2    point.

 3               Even if it were the case that the 

 4    post-enumeration survey somehow was sitting off to the side 

 5    and not tied in to the census but, in fact, the corrected 

 6    counts build directly on the flaws of the census and 

 7    interweave census data and census information with 

 8    information from the post-enumeration survey so it's 

 9    absolutely possible to ignore the flaws of the census and, 

10    indeed, failing to pay attention to them and the way that 

11    they are distributed is to miss the whole point of why it is 

12    that we are taking the post-enumeration survey.

13         Q.    So then do you disagree with the Secretary that 

14    the issue of census error is not relevant to the question of 

15    whether or not to adjust the census in 1990?

16         A.    Yes, I do.

17         Q.    Would you turn back to page 1-7 of the 

18    Secretary's decision.  If you would take a look at the last 

19    paragraph on page 1-7.

20               Are you there?

21         A.    Yes.

22         Q.    Let me read to you the first two lines.

23               "The procedures that would be used to adjust the 

24    census are at the forefront of statistical methodology.  

25    Such research deserves and requires careful professional 
                                                              1522

 1    scrutiny before it is used to affect the allocation of 

 2    political representation." 

 3               Do you agree that the procedures used in the 1990 

 4    PES are at the forefront of statistical methodology and 

 5    require careful professional scrutiny before use?

 6         A.    Well, it's true that the post-enumeration survey 

 7    refined some statistical methodology, but the core of the 

 8    methods that are used in carrying out the post-enumeration 

 9    survey really is this method of capture-recapture that I 

10    mentioned before.

11               The first use of it that I know of goes back to 

12    that of I guess it's Petersen, 1896, in which the method was 

13    used to actually estimate population of fish.

14               So the methodology has been around.  I told you I 

15    worked on it in the sixties.  Lots of people have worked on 

16    it.

17         Q.    When did the Bureau first use the PES-like 

18    methodology?

19         A.    The Bureau first introduced a post-enumeration 

20    survey linked to the census of 1950, and in that survey it 

21    actually went out and measured a group of people, not unlike 

22    the way we did in 1990, and it carried out components not 

23    unlike those that form the core of the methods used in the 

24    current post-enumeration survey.  

25               In fact, the Bureau has been using variations on 
                                                              1523

 1    that same method over and over again in every census 

 2    beginning in 1950 and working its way through up through the 

 3    1990 census.

 4         Q.    Did it use that type of methodology in 1960?

 5         A.    Yes, it did.  In 1960 it used --  in 1960 it 

 6    carried out a match between census data and data in the 

 7    current population survey and those numbers ultimately were 

 8    used to estimate coverage and undercount in 1960.

 9               In 1970 it again used a match between the current 

10    population survey and the census data and, again, those 

11    numbers were used to estimate coverage error undercount in 

12    the 1970 census.

13               In 1980 those same methods were used again, again 

14    with the current population survey, with two different 

15    versions, the April one and the August one, and, again, the 

16    Census Bureau used those numbers to produce estimates of 

17    undercount in the United States associated with the 1980 

18    census.

19         Q.    Do you have an opinion about the amount of 

20    scrutiny that the PES methodology has received?

21         A.    Well, that general class of techniques has 

22    received great, great scrutiny.  But I think to understand 

23    the PES in the 1990 context, you have to look at the decade 

24    long program of research that the Census Bureau did, the 

25    extent to which that research was presented to outside 
                                                              1524

 1    groups, the extent to which it was presented in professional 

 2    meetings, the extent to which it appeared in professional 

 3    journals and was subjected to scrutiny by other 

 4    statisticians.  

 5               I think taken as a whole, the post-enumeration 

 6    survey of 1990 may well be the most studied survey that the 

 7    Census Bureau has ever done and certainly has been subject 

 8    to the greatest amount of scrutiny.

 9         Q.    Dr. Feinberg, can you take a look at the 

10    documents behind tabs 19, 20, 21 and 3, and please tell the 

11    court what they are.

12               For the record, they have been marked for 

13    identification as PX 19 behind tab 19, marked for 

14    identification as PX 20 behind tab 20, and PX 21 has already 

15    been moved into evidence.

16         A.    Yes.

17               Exhibit 19, that is a publication of the Census 

18    Bureau entitled, "The post-enumeration survey: 1950," and it 

19    is in this document that one can find a description of the 

20    PES methods and their use to estimate undercount in the 1950 

21    census.

22               In tab 20 --  

23               THE COURT:  Before you leave, they conduct a 

24    post-enumeration survey for the 1950 census? 

25               THE WITNESS:  Yes, they did.  
                                                              1525

 1               THE COURT:  I take it no adjustment was made? 

 2               THE WITNESS:  No, your Honor.  

 3               THE COURT:  And that's true of 60, 70 and 80 as 

 4    well? 

 5               THE WITNESS:  In 1960, and this is in tab 20, 

 6    what the Census Bureau reports on is --  

 7               THE COURT:  Pardon me, I'm sorry for interrupting 

 8    you.

 9               Was the 1950 the first post-enumeration survey? 

10               THE WITNESS:  The first one done by the Census 

11    Bureau.  

12               THE COURT:  Who else would have done it? 

13               THE WITNESS:  In 1930, 1931, I believe, the 

14    census of Canada, there was a survey done and it was used --  

15               THE COURT:  A Canadian census? 

16               THE WITNESS:  Yes, a Canadian census.  This is 

17    the first one done --  

18               THE COURT:  I'm not talking about the provinces. 

19               THE WITNESS:  I'm sorry, your Honor, but I'm from 

20    there and I'm back there now.  

21               THE COURT:  I know. 

22               THE WITNESS:  This 1950 was the first one done in 

23    the United States.  

24               THE COURT:  All right.

25               Now you are into 1960.
                                                              1526

 1         A.    Now I am back to 60.

 2               In 1960 you will see it refers to the CES census 

 3    match.  This wasn't a separate survey, this was a survey --  

 4    this was the current population survey that we talked about 

 5    a few times already today that is used to estimate the 

 6    unemployment rate.   The Bureau also used it here to match 

 7    into the census.

 8               They report on a summary of that in this document 

 9    and, again, that was used for coverage evaluation and 

10    subsequently used to estimate undercount.

11               We turn to 1970, that's in tab 21, and here's a 

12    report, again, on the coverage, this one is entitled, The 

13    Coverage of Housing, but housing and people are put together 

14    in the census, and this report details a similar match that 

15    was done in 1970 and indicates the extent to which the 

16    various kinds of errors occurred and the way in which they 

17    were used to gauge coverage.

18               And then, and I think I have now lost --

19         Q.    I am sorry, we have to go back to tab 3 in 

20    binder 1.  

21               THE COURT:  That is for 1980?  

22               MS. BARRY:  For 1980.

23         A.    That is for 1980.  And this is the report 

24    entitled, "Coverage of population in the 1980 census," and 

25    in it there is extensive discussion -- this is probably the 
                                                              1527

 1    thickest of the bunch --  extensive discussion of the use of 

 2    survey to carry out coverage improvement and to measure 

 3    undercount in the 1980 census, chapter after chapter deals 

 4    with this issue.  

 5               THE COURT:  So there were only two PES, one in 

 6    1950 and one in 1990? 

 7               THE WITNESS:  That's correct.

 8               But in 1960, 70 and 80, the Bureau used the 

 9    current population survey in lieu of a separate survey.  

10               THE COURT:  Got you. 

11               THE WITNESS:  In order to do the same kind of 

12    approach.

13         Q.    By that you mean in order to do the same kind of 

14    matching approach?

15         A.    Yes.  They did the same kind of matching, the 

16    same kind of correction for erroneous enumerations, for 

17    omissions and the same kind of netting out to get an 

18    estimate of coverage error in the census and the undercount.  

19               MS. BARRY:  At this time, your Honor, plaintiffs 

20    would request that Plaintiff's Exhibit 19, Plaintiff's 

21    Exhibit 20 and Plaintiff's Exhibit 3 be received into 

22    evidence.  

23               MR. SITCOV:  I don't think I have an objection.

24               Just let me take a look at the three of them.

25               (Pause)
                                                              1528

 1               No objection.  

 2               THE COURT:  All of them are admitted with the 

 3    numbers assigned.

 4               (Plaintiff's Exhibit 3, 19 and 20, respectively, 

 5    marked for identification were received in evidence.) 

 6    BY MS. BARRY:

 7         Q.    Dr. Feinberg, I believe you said that another 

 8    name for the capture or for the dual systems methodology was 

 9    capture-recapture?

10         A.    Yes, I did.

11         Q.    And how did that methodology originate?

12         A.    As I indicated --  

13               THE COURT:  It was fish.

14         A.    Fish, 1896.

15         Q.    Has it only been used for counting fish?

16         A.    No, it has not.

17         Q.    If you could turn to page 2-21, Dr. Fienberg.  I 

18    am going to read you another quote, Dr. Fienberg.  It's in 

19    the second paragraph on 2-21.

20               "The PES is based on a statistical technique 

21    called capture-recapture, which is often applied to 

22    estimating wildlife, particularly the number of fish in a 

23    pond.  Fish are caught, tagged, thrown back and some are 

24    recaught in a second catch.  An estimate of the population 

25    of fish can be made from the number of fish who are tagged 
                                                              1529

 1    on the second catch.  The analogy made for the adjustment 

 2    mechanism is that the census is the first catch and the PES 

 3    the second.  The analogy is not close and it is not routine 

 4    to adapt the wildlife model to counting the population." 

 5               Dr. Feinberg, is it correct that the analogy is 

 6    not close and that it is not routine to adapt the wildlife 

 7    model to counting the population?

 8         A.    No, that isn't true.  In fact, the analogy is one 

 9    that is pursued in paper after paper in the literature and, 

10    indeed, the adaptation of the capture-recapture technique, 

11    as I was indicating, occurred as early to human populations, 

12    occurred as early as its use linked to the 1931 Canadian 

13    census.  It is being used in vital statistics populations, 

14    it's been used in a host of different human populations.

15               The Census Bureau pioneered its use in the 1950 

16    census.  The 1950 census, it adapted the model to remove 

17    erroneous enumerations in order to get the net undercount 

18    and then it did similar things with data from 1960, 1970, 

19    1980 and, again, in 1990.

20               In 1980 it took those techniques that it had been 

21    using for decades and made them much more explicit and 

22    formal in the form of formulas that statisticians put into 

23    publications and papers for our specialized journals and 

24    those formulas made very clear exactly the order in which 

25    you did the various pieces so that there could be no mistake 
                                                              1530

 1    about how you do the calculation to get the undercount.

 2               But the basic ideas and the basic approach has 

 3    been there for decades and the analogy has been discussed in 

 4    the literature and is well known to statisticians, the same 

 5    techniques of dual systems estimates have been exported by 

 6    the Census Bureau to its sister agencies all over the world, 

 7    and so people are using these techniques in other countries 

 8    as a result of the Census Bureau's use going back to this 

 9    early period.  

10               THE COURT:  There are two malcontents out there 

11    named Citro and Cohen who don't think too highly of this.

12               Who are they? 

13               THE WITNESS:  That isn't true, your Honor.  

14    That's the panel report to which I referred that I was a 

15    member of, so --  

16               THE COURT:  He cites them as authority for 

17    blasting the analysis. 

18               THE WITNESS:  Well, I read the analogy that they 

19    cited and it sounded pretty good to me.

20               Indeed, I think I actually helped draft those 

21    paragraphs that helped explain where they were similar and, 

22    indeed, where the departure occurs.  

23               THE COURT:  He is just flat wrong when he says 

24    it's not routine to adapt the wildlife model to counting the 

25    population?  He is just wrong when he says that? 
                                                              1531

 1               THE WITNESS:  Yes.  

 2               THE COURT:  All right.  

 3    BY MS. BARRY:

 4         Q.    Dr. Feinberg, has the methodology been used for 

 5    counting other than human populations and fish populations?

 6         A.    Oh, it's used, not for tigers, for dogs, for 

 7    lizards --  

 8               THE COURT:  Canadians? 

 9               THE WITNESS:  For birds in Canada, birds in 

10    Canada for Canadians.

11               It's even been used to count particles in physics 

12    experiments.

13               In the 1920s, Geiger, the man who has his name 

14    associated with the Geiger counter, used it with a co-author 

15    in a study of particles emissions.  

16               And it's used by physicists at the Fermi Lab in 

17    the study of particles physics experiments.  

18               It's even been alleged that it is used to count 

19    stars, although I haven't found the papers in which they 

20    occurred.  

21               Some stars are brighter than others and so they 

22    are easier to see and easier to count than others and you 

23    have different people look at the pictures and make the 

24    counts.  But it's the same kind of technique.

25               Each field may find special adaptations that may 
                                                              1532

 1    be appropriate and they often develop those adaptations and 

 2    new techniques and extensions as appropriate.

 3               The core methods are similar and what you learn 

 4    in one area often carries over almost completely to the 

 5    others.

 6         Q.    Dr. Feinberg, would you turn to tab 18 and could 

 7    you tell us what has been marked or can you describe what 

 8    has been marked for identification as Plaintiff's Exhibit 

 9    18?

10         A.    Yes.  This is a bibliography of capture-recapture 

11    modeling that I recently prepared and I made special care to 

12    try to incorporate a number of applications linked to census 

13    undercount adjustment, and it is impressed in the 

14    statistical journal Survey Methodology, and most of those 

15    examples that I was citing of actual uses appear as 

16    references in the bibliography.

17               The bibliography is annotated so there is a 

18    sentence after each reference to give you a sense of what's 

19    included in it and what the contribution of the papers are.

20               But as the opening paragraph indicates, 

21    applications to physics, to wildlife, human populations with 

22    the Chandrasekar and Deming article in 1948, epidemiology, 

23    particles scanning, crime and criminals.  

24               THE COURT:  Are you anywhere near finishing?  

25               MS. BARRY:  Your Honor, I am afraid I don't feel 
                                                              1533

 1    very confident I will be able to finish very soon.  

 2               THE COURT:  Let's take a break.

 3               We will take a recess and resume at four.

 4               (Recess) 

 5               THE COURT:  Ms. Barry, you may resume.  

 6               MS. BARRY:  Thank you, your Honor.  

 7    BY MS. BARRY:

 8         Q.    Dr. Feinberg, the judge asked you a little while 

 9    ago whether or not there had been a correction in 1970, and 

10    I believe you replied that there had not been one based on 

11    the PES methodology.  Is that correct?

12         A.    That's correct.

13         Q.    Was there any other type of correction to the 

14    census in 1970 based on the results of sample surveys?

15         A.    Yes, there was.  In fact, these were the surveys 

16    I mentioned earlier --  

17               THE COURT:  CES? 

18               THE WITNESS:  No, the national vacancy check and 

19    the post office check added something like a million and 

20    then another half million to the population as a correction 

21    in the 1970 census, and those numbers came directly from 

22    sample surveys.

23         Q.    Do you still have PX 18 in front of you, Dr. 

24    Fienberg?

25         A.    Yes, I do.
                                                              1534

 1         Q.    And as I recall, you finished describing the 

 2    subject matter of that bibliography?

 3         A.    Yes, I have.  

 4               MS. BARRY:  I would move PX 18 into evidence at 

 5    this time, your Honor.  

 6               MR. SITCOV:  No objection.  

 7               THE COURT:  Exhibit 18 is admitted. 

 8               (Plaintiff's Exhibit 18 marked for identification 

 9    was received in evidence.) 

10    BY MS. BARRY:

11         Q.    Dr. Feinberg, would you turn to page 2-8 of the 

12    Secretary's decision.

13               If you look at the top of the page of 2-8, the 

14    last paragraph before "discussion -- " 

15               THE COURT:  "These and other components"?  

16               MS. BARRY:  It starts, "The current plans of the 

17    Bureau include."  

18               THE COURT:  Top of the page, then.  

19               MS. BARRY:  Right.  You were a little bit ahead 

20    of me there.

21         Q.    Dr. Feinberg, is that paragraph part of the 

22    explanation of Guideline One?

23         A.    Yes.  It begins on 2-5 and continues up to that 

24    point.

25         Q.    And is that list in the middle of the paragraph a 
                                                              1535

 1    list of the sources of error that were to be investigated by 

 2    the Bureau in connection with its evaluation of the 1990 

 3    PES?

 4         A.    Yes, it is.

 5         Q.    And is the measurement of erroneous enumerations 

 6    included in in a list?

 7         A.    Yes, it is.

 8         Q.    Would you turn to page 2-18 of the Secretary's 

 9    decision.  Is that page in the section where the Secretary 

10    is discussing Guideline One?

11         A.    Yes, it is.

12         Q.    And if you would take a look at the paragraph 

13    that picks up with the block quote after footnote called 36, 

14    it begins with, "Erroneous enumerations in cases with 

15    insufficient information are not part of the usual 

16    statistical framework for dual system estimation," and if 

17    you would read from there over until the top of 2-19, the 

18    end of the sentence that carries over to the bottom of 2-18 

19    to 2-19 and ends with footnote called 38.

20               (Pause)

21         A.    Yes.

22         Q.    Would you tell us, what is the Secretary saying 

23    about erroneous enumerations in that passage?  

24               MR. SITCOV:  I object.  He can say what he thinks 

25    the Secretary --  what his statement means, but he certainly 
                                                              1536

 1    can't say what the Secretary is saying.  

 2               THE COURT:  Thank you.  

 3               MS. BARRY:  Fair enough.  

 4               THE COURT:  Thank you for sharing that with us.

 5               Can you tell us what you think the Secretary is 

 6    saying? 

 7               THE WITNESS:  Well, I think the Secretary is 

 8    making two different points and they are separated by the 

 9    three dots in the third line.

10               The first point, he says that erroneous 

11    enumerations in cases with insufficient information aren't 

12    part of the usual framework for dual system estimation and, 

13    therefore, their modeling has received less attention than 

14    omissions, and then the second part he goes on and talks 

15    about the implications of that erroneous enumerations are 

16    what is really driving the adjustment, and that is his words 

17    in the third last line.

18         Q.    Did you have an opinion about those criticisms?

19         A.    Well, let's deal with them separately.

20               As I just pointed out, beginning with the 1950 

21    census, the Census Bureau, in carrying out a dual systems 

22    approach, corrected for erroneous enumeration in its 

23    estimate of undercount.

24               Now, I haven't studied in great detail the 

25    category cases for insufficient information, but I do know 
                                                              1537

 1    that they have been captured in different ways in the 

 2    different censuses and they are explicitly part of those 

 3    formulas linked to 1980 that I was mentioning before.

 4               So if it's referring to the usual statistical 

 5    framework from 1896, maybe not.  But as far as the 

 6    historical uses of post-enumeration surveys and dual systems 

 7    estimation within them, I think the statement is just 

 8    incorrect.  The Census Bureau has been paying attention to 

 9    them for decades.

10         Q.    You said earlier that they measured them in 

11    connection with the 1950 post-enumeration survey?

12         A.    In the 1950 post-enumeration survey they measured 

13    erroneous enumerations, they measured omissions and they 

14    netted them out in order to get the net undercount.

15         Q.    Were they measured in connection with the 1980 

16    post-enumeration program?

17         A.    Yes, they were, and explicitly those cases with 

18    insufficient information were identified and netted out at 

19    the same time.

20         Q.    And have erroneous enumerations been the subject 

21    of papers regarding the adjustment methodology during the 

22    past decade?

23         A.    They have been incorporated into paper after 

24    paper throughout the past decade.

25         Q.    What about the Secretary's second statement, the 
                                                              1538

 1    one where he says that erroneous enumerations are "driving" 

 2    the adjustment, do you have an opinion about that statement?

 3         A.    I do, and given the evidence that I have 

 4    examined, it's my opinion that it just isn't true, that, in 

 5    fact, it's omissions that are driving, as best one can say 

 6    the word "driving," the adjustment.

 7               In particular, if you looked at the relationship 

 8    between erroneous enumerations and the adjustment factors, 

 9    the things that we use to produce the undercount rate, you 

10    will find that the relationship between erroneous 

11    enumerations and adjustment factors is much, much smaller 

12    than that between omissions and adjustment factors.

13               In statistical language, we would talk about 

14    accounting for the variability in the adjustment factor and 

15    it's the case that erroneous enumerations count for a very 

16    small fraction of the variability in the adjustment factors.  

17    Omissions, on the other hand, account for well over half.

18         Q.    Would you take a look at the document behind tab 

19    29 which has been marked for identification as Plaintiff's 

20    Exhibit 29.  That is volume 5.

21               Dr. Feinberg, can you tell the court what that 

22    table is?

23         A.    This is a table of correlation coefficients.  

24    They measure the relationship among the three quantities we 

25    were just talking about.
                                                              1539

 1               Let's read through it together.

 2               In the first row, first column, we find the value 

 3    1.  That says that erroneous enumerations are perfectly 

 4    associated with themselves, which is what we would expect. 

 5    The quantity we are looking at here takes a value 1 if there 

 6    is a perfect relationship and they are perfectly related to 

 7    themselves, takes a value minus 1 if they are negatively 

 8    related to one another and zero if there is no relationship 

 9    in the usual sense here.

10               So we see the value 1.

11               Now let's go down that column to the next value.

12               That is .4868.  That says that the relationship 

13    association between omissions and erroneous enumeration has 

14    a value .5.

15               They do covary together across the 1,392 

16    post-strata for which these calculations were made.

17               The way statisticians make these tables, they 

18    duplicate numbers, and so if you look in the first row in 

19    the second column, you will see the same number.  That shows 

20    the relationship between erroneous enumerations and 

21    omissions yet again flipped over.

22               Now if we continue to the third value there, 

23    that's the correlation or association between erroneous 

24    enumerations and the adjustment factors.

25               If the erroneous enumerations go up, the 
                                                              1540

 1    adjustment factors go down, but the relationship is very 

 2    small.  It's close to zero.

 3               If we go to the next number in that column, we 

 4    see the correlation between omissions and adjustment 

 5    factors, and that is .7512.  That says as omissions go up, 

 6    adjustment factors go up and the relationship is fairly 

 7    strong.

 8               In more technical terms as I said, we normally 

 9    square those numbers and you would note then that erroneous 

10    enumerations account for only four percent, it's .2 --  

11    minus .2 and minus .2, which is four percent, four percent 

12    of variation, and omissions account for more than half.

13         Q.    So in your opinion, what conclusions do you draw 

14    from that table?

15         A.    Well, if anything is driving the adjustment 

16    factors in these calculations across the 1,392 post-strata, 

17    the correlation suggests that it's omissions and not 

18    erroneous enumerations.

19         Q.    And would your opinion about that change if the 

20    numbers shown on the adjustment factor lines up with 

21    erroneous enumerations, which is negative .1897, if that 

22    number became a positive?

23         A.    No, it wouldn't change it at all, it's still a 

24    small number and, in fact, when you square plus .1897, that 

25    is, multiply it by itself, you still get .04.  So even if it 
                                                              1541

 1    was positive, it would say that erroneous enumerations still 

 2    only account for about four percent of the variation in the 

 3    adjustment factor.

 4         Q.    Was this table run on the raw adjustment factors?

 5         A.    Yes, it was.  We are dealing with erroneous 

 6    enumeration rates, omission rates and the raw adjustment 

 7    factors, and doing the calculation for the 1,392 

 8    post-strata.  

 9               MS. BARRY:  Plaintiffs would move PX 29 into 

10    evidence at this time.  

11               MR. SITCOV:  Well, your Honor, we have a couple 

12    of objections.

13               Number one is I haven't heard from the witness 

14    that he ran these calculations or had them run or even knows 

15    what the data source is so I don't mind --  

16               THE COURT:  Can you take care of that? 

17               THE WITNESS:  The calculations were run under the 

18    supervision of Eugene Ericksen by an employee at NARA, that 

19    is the line that you see at the bottom, on April 29 in part 

20    under my direction.  I have reviewed the output of those 

21    calculations.  I have had a discussion with the individual 

22    who ran the analyses and confirm what the input was.  

23               MR. SITCOV:  Does he know what the data file is?  

24               THE COURT:  Date of --  

25               MR. SITCOV:  Data file.  
                                                              1542

 1               THE COURT:  Data time. 

 2               THE WITNESS:  Yes.  It is the data file of the 

 3    1,392 --  1,392 post-strata and the adjustment factors used 

 4    are the raw adjustment factors.  

 5               MR. SITCOV:  The other thing, your Honor, is I'm 

 6    sure it is unintentional, but this was about three weeks 

 7    after I deposed this according to the date.

 8               It would have been impossible for me to inquire 

 9    into anything about this chart at the time of his 

10    deposition, so we oppose the introduction of the chart.  

11               THE COURT:  The objection is overruled.

12               Plaintiff's Exhibit 29 is admitted.

13               (Plaintiff's Exhibit 29 marked for identification 

14    was received in evidence.) 

15    BY MS. BARRY:

16         Q.    Dr. Feinberg, even if it were true that erroneous 

17    enumerations were "driving" the adjustment, would you find 

18    that to be "disturbing" as the Secretary does?  

19               MR. SITCOV:  If he knows how the Secretary finds 

20    it disturbing.  

21               MS. BARRY:  He says at the bottom --  

22               THE COURT:  Would it disturb you? 

23               THE WITNESS:  No, it would not, because, as I 

24    mentioned, the basic techniques have been tried repeatedly 

25    carrying out clear inclusion of erroneous enumerations and I 
                                                              1543

 1    have examined what the Census Bureau did to study the 

 2    calculation of the erroneous enumerations and I am quite 

 3    satisfied that they were properly incorporated into a well 

 4    tested statistical methodology.

 5         Q.    Could you turn to page 2-31 of the Secretary's 

 6    decision.

 7               Is this part of the Secretary's discussion of 

 8    face validity tests?

 9         A.    Yes, it is.

10         Q.    And is this also part of the Secretary's 

11    discussion under Guideline One of the relative accuracy of 

12    the uncorrected versus the corrected census counts?

13         A.    Yes.

14         Q.    In your opinion, what is the purpose of that 

15    analysis of face validity in the discussion of the relative 

16    accuracy of the corrected versus the uncorrected census 

17    counts?  

18               MR. SITCOV:  Your Honor, this can only be 

19    speculation on his part.  She is asking him his opinion for 

20    what the purpose was for including it.  

21               THE COURT:  I suppose he can express an expert 

22    opinion on what the purpose is.

23               I will permit it.

24         A.    When statisticians and others run, carry out 

25    studies, surveys like the post-enumeration survey and they 
                                                              1544

 1    produce the results of those surveys, it is customary 

 2    whenever possible to compare the results to independent 

 3    sources for similar information, and especially because the 

 4    similar information will have been gathered in different 

 5    ways with different tools perhaps at different times to look 

 6    at the broad trends that are shown in the data from the 

 7    other sources by comparison with what you find in the source 

 8    that you have just produced, and when you see those trends 

 9    reproduced in other sources and you see these comparison 

10    show up again and again, then, in fact, you get comfort from 

11    the independent checks, and that is what is known as face 

12    validity.

13         Q.    Are face validity tests designed to be fine 

14    grained?

15         A.    No, they are not.  Indeed, they are used in the 

16    way I think I just suggested.  They are used to check with 

17    other sources for trends, for shapes of distributions in 

18    rough ways, sometimes historical data to see if what you see 

19    in a current survey is in line with what has occurred 

20    historically.  

21               Indeed, it's that kind of face validity check 

22    that first led Francis Walter to look at the 1870 data in 

23    the way that he did in order to look at the differences that 

24    showed the differential undercount back in 1870.

25         Q.    Did the Bureau do a face validity test of the 
                                                              1545

 1    1990 PES?

 2         A.    Yes, it did.  It did a number of them in ways 

 3    that might have been labeled face validity and others in 

 4    ways that I guess I would label face validity checks.  Some 

 5    of them are in the P studies, some of them exist elsewhere.  

 6    And in particular they made comparison with the demographic 

 7    analysis, they made direct comparison with demographic 

 8    analysis, they did it looking at age distributions, they did 

 9    it taking those data, and because they are measured only at 

10    the national level running simulations.  

11               They also did comparison, as I said, with the 

12    historical figures to see if, in fact, we have a 

13    continuation of patterns that may have been evinced in the 

14    past and would, if they showed up again, give some comfort 

15    and confidence that the results that were being shown in the 

16    1990 survey were consistent with other information.

17         Q.    And what did the Bureau conclude on the basis of 

18    those face validity tests?

19         A.    From all those I have examined, I think that my 

20    assessment of what the Bureau conclusions were was that they 

21    supported the quality of the PES in 1990.

22         Q.    And do you agree with those conclusions by the 

23    Bureau?

24         A.    Yes, I do.

25         Q.    At page 2-31 of the Secretary's decision, is the 
                                                              1546

 1    Secretary disagreeing with the approach that the Bureau took 

 2    to face validity tests?  

 3               MR. SITCOV:  I object.  

 4               THE COURT:  Overruled.

 5         A.    Yes, he is.  In fact, at the end of that first 

 6    paragraph, he says, after discussing the kinds of things 

 7    that the Census Bureau did, that face validity can show the 

 8    opposite, and he goes on in the next paragraph to cite a 

 9    series of examples which he uses to make this point that, in 

10    fact, the PES did not show face validity.

11         Q.    In your opinion, does the Secretary use face 

12    validity in a an appropriate way?

13         A.    No, he does not.

14         Q.    Why not?

15         A.    Well, there are a large number of things that you 

16    can look at in any survey, little details; the number of 

17    people who lived in Wilkinsburg, Pennsylvania and to see if 

18    that changed in some dramatic way as a result of the 

19    measurement of the undercount, and there are hundreds, 

20    indeed, close to --  there are 39,000 small areas that 

21    people look at in the United States.

22               In 39,000, I'm sure we could find lots of places 

23    where you couldn't make a natural comparison and where you 

24    might be surprised that the number was higher, the number 

25    was low or it appeared to be high when, in fact, it may not 
                                                              1547

 1    even have been high relative to what you would have expected 

 2    if only you had the right numbers.

 3               And, therefore, it's very misleading if one goes 

 4    into a data set or a survey like the post-enumeration survey 

 5    and plugs out a few numbers and says, "My goodness, why is 

 6    this number so big, or why do these numbers look so 

 7    different?" 

 8               That isn't what face validity is all about.

 9    

10    

11    

12               (Continued on the next page) 

13    
      

14    
      

15    
      

16    
      

17    
      

18    
      

19    
      

20    
      

21    
      

22    
      

23    
      

24    
      

25    
      
                                                              1548

 1         Q.    Do you know where the Secretary got the examples 
                                                                  

 2    that he cites at the bottom of page 2-31?

 3         A.    I have seen that list in that paragraph from 
                                                              

 4    beginning to end in a memorandum from Peter Bounpane written 

 5    and submitted to the Secretary.

 6         Q.    Would you turn to Tab 673, volume 6.  Dr. 

 7    Fienberg, would you tell us what PX-673 is?

 8         A.    This is a memorandum dated July 10th from Peter 

 9    Bounpane to Robert Mosbacher, Secretary of Commerce.  It has 

10    Bates stamp number 3220 at the front, on the first page.  It 

11    is the document to which I referred in which I have seen 

12    these same examples.

13         Q.    Could you show us where you saw those same 

14    examples?

15         A.    Yes.  If you turn to the second to last page, 

16    page 7, or if you like the numbers at the bottom, 3227,, 

17    there is a paragraph at the bottom of the page in a section 

18    labeled "Face Validity," and it begins with the sentence, "A 

19    few examples will illustrate the point."

20         Q.    This is the paragraph that you said provided the 

21    examples set forth at 2-31?

22         A.    Yes, sir. 

23         Q.    Is it an identical verson of that paragraph?

24         A.    There are a few words that are different, but 

25    each example and almost every sentence is virtually the 
                                                              1549

 1    same.

 2         Q.    Let's check.  I am going to read the sentences 

 3    from -- 

 4               THE COURT:  Let's not do the details.  I am 

 5    comfortable with what he just said.

 6         Q.    What is missing, Dr. Fienberg?

 7         A.    If you turn to page 8, after he gets to the end 

 8    of that paragraph, where it says, "Cleveland has such a low 

 9    estimated undercount rate, 1.4 percent," that is exactly the 

10    same as what is in the Secretary's report.  

11               The next paragraph -- 

12               THE COURT:  That is good sampling.

13         A.    Well, now, the next sentence says, "I don't think 

14    that it's reasonable to get into this kind of argument."

15         Q.    That is the next sentence in the Bounpane memo?

16         A.    Yes, it is.

17         Q.    Does that sentence appear in the Secretary's 

18    discussion of face validity?

19         A.    No, it does not.

20         Q.    In your opinion, was Bounpane right, that it is 

21    not reasonable to get into this kind of argument?  

22               MR. SITCOV:  I object, your Honor.  

23               THE COURT:  Sustained.

24         Q.    In your opinion, Dr. Fienberg, is it reasonable 

25    to get into the kind of argument that the Secretary engages 
                                                              1550

 1    in at pages 2-31, the top of 2-32 of his decision concerning 

 2    face validity?

 3         A.    No, it is not.  

 4               MS. BARRY:  Plaintiffs would move into evidence 

 5    PX-673 at this time, the Bounpane memo, which is in the 

 6    so-called administrative record.  

 7               MR. SITCOV:  No so-called objection.  

 8               THE COURT:  Plaintiffs' Exhibit 673 is admitted.  

 9               (Plaintiffs' Exhibit 673 for identification was 

10    received in evidence)

11         Q.    Dr. Fienberg, just to make sure that I made the 

12    record clear, all the examples set forth in 2-31 of the 

13    Secretary's decision and the top of 2-32 of his decision are 

14    the same examples that are set forth in the Bounpane memo, 

15    is that correct?  

16               THE COURT:  Yes, they are.

17         A.    Yes.

18         Q.    What is missing is Bounpane's statement about it 

19    not being reasonable to get into this kind of argument?  

20               THE COURT:  Yes, that is what is missing.

21         A.    Yes.

22         Q.    Dr. Fienberg, do you have any opinion about any 

23    of the examples that the Secretary cites at page 2-31 of his 

24    decision?

25         A.    I guess I could take the first one.  It says, "Is 
                                                              1551

 1    it reasonable that New Mexico has the highest undercount 

 2    rate of any state?"  My answer to that one is yes, to me it 

 3    is very reasonable.  

 4               Why?  Because I know that undercount is related 

 5    to minority status, and New Mexico has the highest rate of 

 6    Hispanics of any state in the nation.  If I thought there 

 7    was a relationship between the two, I might well expect that 

 8    New Mexico to have the highest or at least close to the 

 9    highest rate in the nation.  So it is reasonable to me.  

10               On the other hand, I don't think it is 

11    appropriate to get into these kinds of comparisons.

12         Q.    Would you turn to page 2-45 of the Secretary's 

13    decision.  In that paragraph under "Conclusions," I am going 

14    to read a sentence to you which is in the middle of that 

15    paragraph.  "Furthermore, substantial evidence casts doubt 

16    on the homogeneity assumption underlying the entire 

17    synthetic adjustment methodology."  

18               Have you reviewed the Secretary's discussion of 

19    homogeneity, Dr. Fienberg?

20         A.    Yes, I have.

21         Q.    In a nutshell, what is a homogeneity assumption?

22         A.    Because for the evaluation we grouped people into 

23    these 1392 poststrata, we are essentially using a simplified 

24    model which says that the probability of being counted in 

25    the census or the probability of being counted in the PES, 
                                                              1552

 1    or the converse, the probability of being missed, is the 

 2    same for everybody in a single poststratum.

 3         Q.    Do most statistical methods involve some kind of 

 4    homogeneity assumption?

 5         A.    Yes, they do.  Virtually every statistical 

 6    technique or method in one form or another has some kind of 

 7    homogeneity-like assumption in it.

 8         Q.    In your opinion, is there ever perfect 

 9    homogeneity?

10         A.    In my opinion, in the real world, we never get 

11    nor do we expect to get perfect homogeneity.

12         Q.    In your opinion, if one were to require perfect 

13    homogeneity, what would the impact be? 

14         A.    Well, there are 15,000 members of the American 

15    Statistical Association, and a lot of us would be looking 

16    for new jobs.  There are many federal agencies, and they 

17    would probably stop doing the bulk of their work.  Because, 

18    in one way or another, there is an assumption, either that 

19    one or one much like it, in their efforts.

20         Q.    In your opinion, what is the appropriate way to 

21    evaluate the homogeneity assumption in connection with the 

22    1990 PES?

23         A.    You would like to know the extent to which the 

24    assumption really impacts on the thing you are trying to 

25    measure.  You want to know what departures from it are 
                                                              1553

 1    typically expected to produce.  

 2               In the case of the post-enumeration survey, it is 

 3    generally accepted that heterogeneity, that is, the absence 

 4    of pure homogeneity, will produce an undercount estimate 

 5    that is too small.  That is, heterogeneity induces an 

 6    understatement in undercount.  

 7               But I wouldn't stop there.  Indeed, I would look 

 8    at the extent to which heterogeneity was achieved in the 

 9    post-ratification scheme that is used.

10         Q.    You mean, homogeneity, Dr. Fienberg?

11         A.    I'm sorry, homogeneity was achieved in the 

12    post-ratification scheme that was used.  Indeed, that was 

13    the recommendation that can be found in the original paper 

14    by Chandrasekar and Demming which discusses the formalities 

15    of using the dual systems estimates in heterogeneous 

16    populations.  They said stratify, and stratification will 

17    reduce a lot of the problem.  They also point to the 

18    residual problem that will exist, and the question is then, 

19    in what's left, is it having a big impact and in what 

20    direction is it likely to go?

21         Q.    That paper you just referred to, when was that 

22    published?

23         A.    That paper was published in 1949.  

24               The Demming that I referred to is W. Edwards 

25    Demming, who had been the associate director of the census 
                                                              1554

 1    bureau during the 1940 census and is the quality guru who 

 2    does statistical quality improvement around the globe.

 3         Q.    Did the bureau evaluate homogeneity in connection 

 4    with the 1990 PES?

 5         A.    Yes, it did.

 6         Q.    What did it conclude?

 7         A.    The bureau concluded that there was a sufficient 

 8    degree of homogeneity to allow it to determine that the 

 9    corrected census counts were more accurate than the raw 

10    census enumeration.

11         Q.    Did the Secretary disagree with the bureau's 

12    conclusions?

13         A.    Yes, he did.

14         Q.    In your opinion, was he justified in doing so?

15         A.    In my opinion, he was not.

16         Q.    Have you looked at the allegedly, quote, 

17    substantial evidence, closed quote, that the second cites at 

18    page 2-45 of his decision?

19         A.    Yes, I have. 

20         Q.    Do you have an opinion about what that evidence 

21    shows? 

22         A.    In my opinion, the evidence either has nothing to 

23    do with homogeneity at all or, at best, shows just modest 

24    levels of it and is quite consistent with the conclusion 

25    that the use of corrected counts is, in fact, an improvement 
                                                              1555

 1    upon the raw census enumeration?  What specific pieces of 

 2    evidence are you referring to?

 3         A.    I will have to turn back here.

 4         Q.    All right.

 5         A.    On pages 2-39 and 2-40 the Secretary reviews 

 6    P-12, in which the census bureau presented some of its 

 7    evidence.  Then he turns at the bottom of page 2-40 -- and 

 8    he dismisses it.  He says essentially we haven't shown that 

 9    there isn't any residual heterogeneity as a result of that.  

10    Then he says, let's look at project P-15.  So I did.  I read 

11    it from cover to cover.  

12               Nowhere in P-15 is the word "homogeneity" or 

13    "heterogeneity" used.  I read it a second time and I 

14    couldn't find it.  If it is there, it must be in a footnote 

15    or hiding maybe at the back of a table.  

16               That is a document that looks at something else.  

17    It simply provides no evidence whatsoever about the 

18    homogeneity problem that the Secretary suggests it does in 

19    the opening sentence.

20         Q.    Is there any other evidence that the Secretary 

21    cites?

22         A.    After referring to the various panel members in 

23    different ways, the Secretary on page 2-42 cites Wachter's 

24    evidence on heterogeneity, which he says is the only 

25    evidence that looks at the actual behavior of the 1990 
                                                              1556

 1    census and PES below the state level.  I went and I looked 

 2    at that evidence also.  

 3               It is true, Wachter did do a study.  He did a 

 4    study which used data for an area in Texas known as 

 5    Arlington.  He had extracted other data.  But the data were 

 6    only the beginning.  Then he did a fairly complicated 

 7    simulation, which I have tried to understand.  But the key 

 8    part is the conclusions.  

 9               The simulation purported to insert heterogeneity 

10    and show the impact of that insertion on the outcome.  What 

11    it showed in the end was that in 93 percent of the cases the 

12    majority of the district level offices, the smallest level 

13    that he chose to look at, were improved proportionately, 

14    that is, in terms of shares, in 93 percent of the cases the 

15    majority were improved.  

16               To me that is evidence that despite perhaps the 

17    evidence of some residual heterogeneity, it doesn't impact 

18    on the conclusion that the corrected census counts are 

19    superior to the raw counts.

20         Q.    In your opinion, does any of that evidence you 

21    just mentioned justify the Secretary in rejecting the 

22    bureau's conclusions about homogeneity?

23         A.    No, it does not.

24         Q.    Turn to page the 2-11 of the Secretary's 

25    decision.  I am going to read another passage to you, Dr. 
                                                              1557

 1    Fienberg.  

 2               THE COURT:  How many more of these passages do we 

 3    have?  

 4               MS. BARRY:  I have several more, your Honor.  

 5               THE COURT:  How many more?  

 6               MS. BARRY:  Maybe six or seven.  

 7               THE COURT:  Can you do them by five o'clock?  

 8               MS. BARRY:  No.  

 9               THE COURT:  This is getting painful.  You are 

10    going to be finished at five.  Get all of those six in 

11    between now and five o'clock.  

12               MR. ZIMROTH:  Your Honor, may I suggest that we 

13    take a break now and start in the morning?  

14               THE COURT:  No.  I want this done today at five.  

15    This is painful.  

16               MR. ZIMROTH:  Your Honor, this is the Secretary's 

17    decision.  

18               THE COURT:  You are wasting time.  It can be 

19    done.  

20               MR. ZIMROTH:  I don't think it can be, your 

21    Honor, with respect.  

22               THE COURT:  It can be.  Let's go.  

23               MS. BARRY:  Your Honor, if I might just speak for 

24    a second.  We do have a number of important points to make.  

25    I believe it would take us about another hour and a half to 
                                                              1558

 1    do them.  If you would just forbear with us -- 

 2               THE COURT:  You have 15 minutes and you are 

 3    losing them rapidly.  

 4               MS. BARRY:  Yes, your Honor.  

 5    BY MS. BARRY:

 6         Q.    Do you see at page 2-11, Dr. Fienberg, the 

 7    statement regarding special advisory panel member Wachter 

 8    estimating that the number of people missed by the census -- 

 9               THE COURT:  You don't have to read the whole 

10    sentence.  Tell him to read the first two lines of the 

11    second paragraph there.

12         Q.    Could you read those first two lines.  It refers 

13    to the argument that the PES miss as many as half a million 

14    people and that that is a reason not to adjust?

15         A.    Yes.

16         Q.    In your opinion, is the fact that the PES may 

17    miss 500,000 people also missed by the census a legitimate 

18    reason not to correct?

19         A.    No, it is not.  What that paragraph is saying is 

20    that the PES may miss as many as a half million people, and 

21    implicit in that is that they are distributed similarly to 

22    those already distributed in the post-adjustment population, 

23    and that there is no adjustment for that.  What they are 

24    suggesting is because we don't know exactly how those half 

25    million are distributed, then what we should do is throw 
                                                              1559

 1    away the millions, the approximately five million people 

 2    that the PES has accurately distributed across the country, 

 3    and that the only way in which that makes sense, that is, 

 4    the only way in which it would really matter would be if 

 5    that half million people were distributed in such a fashion 

 6    over the population to vitiate the gains that were realized 

 7    throughout the country as a result of the distribution of 

 8    the rest.  

 9               Now, how could that happen?  It could happen if 

10    the numbers were perversely distributed in some bizarre way 

11    that we don't expect.  Let me make a perverse example.  

12    Suppose they were all hiding in the hills of South Dakota.  

13    That would change the shares, and that would be enough, 

14    perhaps, to shift the distribution of the census.  

15               On the other hand, ask yourself what do we know 

16    about the people in this approximately half million that we 

17    speculate are not included?  They are hard to count.  They 

18    were missed by the census and they were missed by the PES, 

19    and we didn't even get them in the estimate.  If they are 

20    hard to count, then we certainly expect them to look more 

21    like the post-adjustment population than we do like the raw 

22    census distribution.  

23               So on this count it seems to me the Secretary is 

24    willing, on the basis of speculation that I clearly wouldn't 

25    accept, that that half million looks more like the raw 
                                                              1560

 1    counts or that there is some perverse distribution, we 

 2    should ignore the information, the accurate information 

 3    provided by the post-enumeration survey in a clear and 

 4    convincing way.

 5         Q.    Thank you, Dr. Fienberg.  

 6               Did the census bureau study the issue of missing 

 7    data in the 1990 PES?

 8         A.    Yes, it did.

 9         Q.    What did it conclude?

10         A.    It concluded the missing data in the 1990 PES was 

11    quite low, indeed remarkably low, by all the standards that 

12    the census bureau has to assess missing data, and in 

13    particular much, much lower than it was in 1980.

14         Q.    How does 1990 compare to 1980 in terms of missing 

15    data?

16         A.    Orders of magnitude.  In 1980, in the April 

17    current population survey, there was about 8-1/2 percent 

18    missing data.  In the August one there was 9-1/2 percent 

19    missing data, in rough terms.  In the 1990 P sample of the 

20    post-enumeration survey, missing data was approximately 2 

21    percent, much, much lower than that of any other survey 

22    carried out by the census bureau.

23         Q.    Would you turn to page 2-16 of the Secretary's 

24    decision.  Could you take a look at the paragraph under 

25    "Missing data" where he refers to what he calls two 
                                                              1561

 1    troubling findings regarding missing data.

 2         A.    Yes.

 3         Q.    Could you describe briefly what the first 

 4    troubling finding is that the Secretary makes regarding 

 5    missing data?

 6         A.    In the first, the Secretary notes that the 

 7    proportions of missing data in the P and E samples are 

 8    approximately 2 percent, which is the same proportion as is 

 9    represented by the undercount, thereby suggesting that 

10    perhaps the undercount is explained by the missing data.  

11    But, in fact, nothing can be further from the truth.  The 

12    level of missing data and its impact on the 1990 

13    post-enumeration survey represents just a small, small 

14    fraction of that total amount.  

15               What the census bureau does is, when there is 

16    missing data, it has to impute a match.  It has to put 

17    together a record from the sample with a record from the 

18    census.  It does it using a set of models that it developed 

19    through its program of research throughout the 1980s.  

20               Instead of actually saying match or no match, the 

21    model says the probability of a match.  That is what it 

22    produces.  So it produces a number between zero and 1.  

23               In evaluating the quality of missing data and in 

24    looking at alternative ways to do the imputation, reasonable 

25    alternatives on the basis of the PES methodology and how it 
                                                              1562

 1    had been used in the past, the bureau concluded in its P 

 2    study reports that the impact of looking at that time those 

 3    alternatives showed basically no difference in the 

 4    undercount rates that the PES produce.  

 5               So by looking at some variations and by using 

 6    imputation, a method that was tested in the test censuses 

 7    and in the dress rehearsal, the census bureau was able to 

 8    take a relatively low level of missing data and make this 

 9    not be a serious problem in the 1990 PES.

10         Q.    What about the second observation the Secretary 

11    makes?

12         A.    In the second observation, the Secretary says 

13    that if you look from stratum to stratum and you look at the 

14    ones where adjustment rates are high, imputation rates are 

15    high, and that is a problem.  But to me that is exactly what 

16    we would expect.  That is, what is the undercount, high 

17    undercount?  That is a result of not being able to find 

18    people.  These are hard to count people.  

19               Imputation occurs when, in fact, we have missing 

20    data, and those people are hard to count.  So we would 

21    expect imputation to go hand in hand with high undercount 

22    rates.  

23               The big question is, what is the impact of those 

24    higher levels of imputation on the numbers that we are 

25    concerned with, the numbers that address the issue of 
                                                              1563

 1    accuracy?  The best indicator of that in my mind is to look 

 2    at the nonmatches.  Those are the things that are the key 

 3    determinant of whether or not the undercount rate will be 

 4    overstated.  If we have too many false nonmatches, then in 

 5    fact we have error in our undercount estimates.  

 6               What we would like to know is the proportion of 

 7    false nonmatches due to imputation.

 8         Q.    Would you turn to Tab 38.  Does that refer to the 

 9    proportion of nonmatches due to imputation?

10         A.    Yes, it does.

11         Q.    What conclusions do you draw from that table?

12         A.    This is a table extracted from an appendix to the 

13    report of Ericksen, Estrada, Tukey, and Wolter.  It lists 

14    the 13 evaluation poststrata, and it organizes them into two 

15    groups, the minority group and the nonminority group.  What 

16    we know from other information directly from the tables that 

17    the census bureau produce is that the minority group has the 

18    higher level of undercount.  Those five have the highest 

19    levels of undercount among the 13.  

20               Now we look at the percent of nonmatch due to 

21    imputation, this criterion I suggested before.  It varies 

22    from as low as 3.3 up to 12, and it averages about .7 -- I'm 

23    sorry -- 7 percent, 7, 8 percent, roughly similar.  

24               Now let's look at the nonminority.  These are the 

25    ones that have either no undercount or essentially small 
                                                              1564

 1    undercounts.  There are eight of them.  We have numbers.  

 2    There is a 4.99 there, 11.13, some 6s, some 9s, a few 8s.  

 3    You compute the average there, and it is 7 to 8.  

 4               What we have here is exactly the comparison that 

 5    you would want to focus on to address that issue.  In those 

 6    strata that have high undercounts, what we have is exactly 

 7    the same relative impact of imputations on the key area, the 

 8    percent of nonmatches.  What we see is that they are 

 9    virtually the same when we look at minority and nonminority 

10    areas.

11         Q.    Dr. Fienberg, do you know how those percentages 

12    compare to the percentages of nonmatch due to imputation in 

13    the 1990 PEP?

14         A.    Yes.  In the 1980 --

15         Q.    I'm sorry.  1980 PEP.

16         A.    In the 1980 PEP those numbers, first of all, were 

17    almost an order of magnitude higher.  They were like 30 

18    percent, 40 percent.  In those numbers there is a consistent 

19    pattern for the April CPS and then for the August CPS, which 

20    show that the minority areas, in fact, had higher percent of 

21    nonmatch due to imputation relative to the nonminorities.  

22               So what we had was a problem in 1980 and very 

23    high levels of nonmatches, which are indicative of possible 

24    problems with false nonmatches.  What we see here in 1990 is 

25    that this problem has really been brought under control and 
                                                              1565

 1    that the differences between the two groups are essentially 

 2    nonexistent.  There is variability there, but there is no 

 3    real differences.  And in a relative sense there is no 

 4    impact of imputation that we can discern in these numbers.

 5         Q.    Could you turn to Tab 39.  Does that document 

 6    report the rates of imputation in the 1980 PEP?

 7         A.    Yes, it does.  They occur somewhere in the middle 

 8    of the document, on the 25th page.

 9         Q.    We won't bother going into these right now, Dr. 

10    Fienberg.  

11               MS. BARRY:  Plaintiffs would move into evidence 

12    at this time PX-39.

13         Q.    It is 39?

14         A.    Yes.  

15               MS. BARRY:  PX-39 and PX-38.  

16               MR. SITCOV:  No objection.  

17               THE COURT:  38 and 39 are admitted.  

18               (Plaintiffs' Exhibits 38 and 39 for 

19    identification were received in evidence)

20         Q.    Dr. Fienberg, do you know what other evidence the 

21    Secretary relied on in rejecting the bureau's conclusions 

22    that missing data was not a problem in the 1990 PES?

23         A.    Yes.  If you turn the page from 2-16 to 2-17, it 

24    refers to Professor Wachter, first notes that he finds the 

25    work of the bureau of high quality but that he is hesitant 
                                                              1566

 1    to believe the assumptions.  So what he did was his own 

 2    calculation.  I have to explain the calculation so that you 

 3    can understand how to consider it.  

 4               Remember that if there isn't a match, the census 

 5    bureau did an imputation, and it produced a probability of a 

 6    match, a number between zero and 1.  What Wachter did in 

 7    this study is he said, look, we have to check the 

 8    variability, and so what I am going to first do is assume 

 9    that everyone in the P sample corresponds to a match.  That 

10    says that everybody gets a 1 instead of a number between 

11    zero and 1.  Then in the E sample he made every one of these 

12    missing data cases an erroneous enumeration.  

13               So here is a double extremity.  First he moves 

14    everything from zero to 1 up to 1.  Then in the erroneous 

15    enumerations he does exactly the same thing.  And you have 

16    to match them in order to get the result.  If you make them 

17    go the other way, then a lot of things cancel out.  

18               By making this extreme assumption that absolutely 

19    everything was wrong, that is, you got them all one way and 

20    all one way in the other sample, you put it together and you 

21    get what he calls an upper bound.  And it is strict, because 

22    you can't go anywhere beyond 1 in order to deal with 

23    probabilities.  

24               Then he turned the tables.  He said, now suppose 

25    I go the other way and I take everything that is between 
                                                              1567

 1    zero and 1 and I make it zero, and everything that was 

 2    between zero and 1 indicating an enumeration, I will now 

 3    call that an enumeration.  So that pushes them in the other 

 4    direction.  Then he put all of that together and they 

 5    creates another strict bound, called the lower bound.  

 6               There isn't any evidence for any of these errors 

 7    in the census explicit.  So we have an upper bound and we 

 8    have a lower bound.  

 9               Then what he said is suppose we mix the two kinds 

10    of errors and we will concede for the moment that the census 

11    bureau got the proportion right, so we will do half high and 

12    half low.  What that does is it pushes all the numbers out 

13    to the extreme.  If you had asked me before I saw any part 

14    of this calculation what its impact would be, I and 

15    virtually every other statistician would have told you that 

16    will increase the variance.  That is what it is designed to 

17    do.  Sure enough, it did.

18         Q.    So in your opinion, Dr. Fienberg, does that study 

19    justify the Secretary's rejections of the bureau's 

20    conclusions regarding missing data in the 1990 PES?

21         A.    Well, it's not based on any data.  It is 

22    certainly not based on any plausible assumptions.  I don't 

23    see it as a valid reason for rejecting the high-quality work 

24    that the census bureau had done on imputation over the 

25    decade and the work that it did explicitly in the 1990 
                                                              1568

 1    census, work which the census bureau itself concluded was of 

 2    high quality and had a relatively small impact on the 

 3    estimates of undercount and the estimates of differential 

 4    undercount.

 5         Q.    Dr. Fienberg, what is did the census bureau 

 6    evaluate the issue of matching error in the 1990 PES?

 7         A.    Yes, it did.

 8         Q.    What did it conclude?

 9         A.    It concluded that matching error, which had been 

10    such a big problem in the 1980 sends, was brought under 

11    control and was of sufficiently small magnitude to allow the 

12    determination that the corrected census counts were superior 

13    to the raw census counts.

14         Q.    Have you reviewed the work the bureau did in 

15    evaluating the matching error in the 1990 PES?

16         A.    Yes, I have.

17         Q.    Do you agree with its conclusions?

18         A.    Yes, I do.

19         Q.    If you would look at page 2-17 under the heading 

20    "Matching Error" in the Secretary's decision, does the 

21    Secretary disagree with the bureau's conclusions that 

22    matching was of high quality in the 1990 PES?

23         A.    He says that despite that, there is evidence that 

24    gives him pause.

25         Q.    What is that evidence?
                                                              1569

 1         A.    It is evidence that Professor Wachter suggested.

 2         Q.    Could you briefly describe what that evidence is 

 3    to the Court?

 4         A.    This was a study that Professor Wachter did on 

 5    the issue of matching.  He went into the P sample, and he 

 6    singled out all of the households that were nonmatches.  So 

 7    where the interview in the P sample didn't match with the 

 8    corresponding census form.  

 9               In the P sample there is a household respondent, 

10    somebody who fills out the form, essentially answers on 

11    behalf of the household.  What he did is he looked for all 

12    those people who were in that group whose designation was 

13    stepchild.  That is the relationship with the person who 

14    answered the questions.  

15               So here we have stepchildren linked to the 

16    respondents for whom there was no match, nonmatches.  What 

17    he said is, let's assume half of those are mistakes.  So he 

18    took this category and he assumed half of them were wrong.  

19    Then he tracked through all of this and looked at the impact 

20    on the error associated with matching that the bureau 

21    estimated.  He notes that 10 percent of that error might 

22    have been accounted for by erroneously labeled stepchildren.  

23               Now, there isn't any evidence that this error 

24    occurred at all.  I haven't seen any evidence, either in 

25    Professor Wachter's report or in any other document, that 
                                                              1570

 1    the people who were stepchildren of the household 

 2    respondents for whom there were nonmatches actually were 

 3    errors at all.  

 4               So by assuming yet again that big errors 

 5    occurred, he has shown that he can find big errors in a 

 6    relative proportion and, as a consequence, suggests that 

 7    matching error may be too small.

 8         Q.    Dr. Fienberg, do you know what Dr. Wachter wrote 

 9    in his own report about the results of his matching study 

10    involving stepchildren?

11         A.    He said this study doesn't show anything at all.  

12    I would have to read it exactly.

13         Q.    Tab 46, Dr. Fienberg.  I believe he discusss the 

14    results of this test at page 21.

15         A.    I am going to read from the first full paragraph 

16    there, third to last line.  It says, "This small error 

17    proves nothing, but it suggests to me that errors in the 

18    allocation of matching error bias could readily amount to 

19    tenths or so of the estimated biases."

20         Q.    I think you made a mistake in reading, Dr. 

21    Fienberg.  You said this small test proves nothing, is that 

22    correct?

23         A.    Yes.

24         Q.    Does the Secretary report what Wachter said about 

25    the results of his test in his discussion of matching error 
                                                              1571

 1    at page 2-17 of his decision?

 2         A.    No, he does not.

 3         Q.    Does the Secretary's reliance on the evidence 

 4    provided in Wachter's test justify the Secretary, in your 

 5    opinion, in rejecting the bureau's conclusion that matching 

 6    was of high quality in the 1990 PES?

 7         A.    No, it does not justify it.

 8         Q.    Dr. Fienberg, do you discern any pattern in the 

 9    Secretary's discussion of errors in the PES?

10         A.    As I think back over the three points we have 

11    just gone through, we have careful studies done by the 

12    census bureau.  We have evaluations of them by the census 

13    bureau.  The Secretary essentially in each case rejects the 

14    census bureau conclusions.  

15               In the first instance he was willing to throw out 

16    the approximately five million people who were included in 

17    the post-enumeration, as a result of the post-enumeration 

18    survey.  In the second instance he was willing to say that 

19    all of those studies on missing data that the census bureau 

20    had done were not worth anything.  And in the third instance 

21    he dismisses the matching studies that the census bureau had 

22    done and all the research that had led up to it, the work on 

23    computer matching in the case of missing data the work on 

24    imputation, and so on.  

25               What he relies on is hypothetical examples, 
                                                              1572

 1    assumptions, and assumption of a really gross sort, like 

 2    half the errors of a certain category are wrong, or let's 

 3    try to push things out in as extreme a fashion as we can.  

 4    Or in the case of the 500,000 people who we are not sure how 

 5    we can allocate, let's throw away all of those people 

 6    because it may be the case that some perverse distribution 

 7    like those people hiding in the hills of South Dakota, could 

 8    destroy the distribution that the PES accurately captures 

 9    and is produced in the corrected census counts.  

10               Essentially, he is discarding high-quality 

11    studies done with care based on years of research in favor 

12    of hypothetical, anecdotal, or extreme illustrations.

13         Q.    Thank you, Doctor. 

14               There is a section in the Secretary's decision at 

15    page 2-34 where he states that only 18 of the 51 states half 

16    an undercount rate that is significantly different from the 

17    national average, and that means in 33 states we do not know 

18    if the undercount rate is higher, lower, or the same as the 

19    national average.  Put another way, we do not know if those 

20    33 states deserve more or less political representation and 

21    federal funding than they are receiving."  

22               In your opinion, is that an appropriate 

23    comparison that the Secretary is making there?

24         A.    No, it is not.  First of all, we know a lot about 

25    those 18 states.  We actually know a lot about the other 33 
                                                              1573

 1    as well.  It is easiest to illustrate if we look at the 

 2    chart from which this is extracted.

 3         Q.    It is behind Tab 28.  The document is the June 

 4    13th press release which has been previously admitted into 

 5    evidence as Plaintiff's Exhibit 478.

 6         A.    Can I suggest, your Honor, you turn to the last 

 7    page of that.  

 8               THE COURT:  Of 28?

 9         A.    Of 28.  You can turn it sideways so we can read 

10    it together.  

11               What this chart shows is the actual undercount 

12    rates for each of the 50 states plus the District of 

13    Columbia, and it shows the national average, which is 2.1, 

14    what the Secretary was referring to.  And it gives these 

15    black bars, which show the variability.  

16               Let's read from the top down quickly.  We read 

17    from the top down, and the first one that we hit over on the 

18    lefthand side is the District of Columbia.  You can read it 

19    by looking at the bottom of the chart and noticing that that 

20    bar is the second one in in the South Atlantic area.  

21               The District of Columbia has the highest 

22    undercount rate.  Its bar doesn't come anywhere close to the 

23    national average 2.1.  And we know a lot about the District 

24    of Columbia.  It has the highest rate proportion of blacks 

25    in the nation.  So it is not surprising that it is high, and 
                                                              1574

 1    the variability doesn't obscure anything at all for it.  

 2               The next one over on the right that is just above 

 3    4.0 is New Mexico.  We have already talked about New Mexico, 

 4    which has the highest rate of Hispanics.  Again we have the 

 5    bar not touching the line 2.10.  And if we continue down the 

 6    next ones you actually hit are California.  It is the fourth 

 7    one in from the right, and Arizona, which is on the extreme 

 8    lefthand side of the mountain group.  Those are indicative 

 9    of -- there are a couple of more that are above the national 

10    average and for which the bars don't overlap 2.1.  

11               It turns out that they include all the states 

12    that gain from an apportionment as a consequence of the use 

13    of corrected counts.  That is, California and Arizona.  

14               Now let's look at the bottom up.  These are 

15    easier to get now because they come in bunches.  On the 

16    lefthand side is a bunch that come from New England and the 

17    mid Atlantic reJohn.  All but four of those bars don't 

18    overlap 2.10.  So those are the states where although there 

19    is an overcount -- that is what the dots show, those dots 

20    are above zero -- but the result is less than the national 

21    average, so in relative terms those states lose.  But all 

22    the bars except for four are distinct from, they don't 

23    overlap that line.  

24               If we look over on the right, there is 

25    Pennsylvania, my former home is that it.  It has a value 
                                                              1575

 1    that is just above zero, and its bar is below 2.10.  

 2               The other group, and I think we catch them all 

 3    this way, is over in the East North Central and West North 

 4    Central.  There there are two that overlap the line in one, 

 5    where it is hard to know if it is touching or it is actually 

 6    just below.  All the rest have values that are different 

 7    from 2.10.  These are states that lie below.  That group 

 8    includes, for example, in the East North Central case, 

 9    Wisconsin on the righthand side.  Wisconsin and Pennsylvania 

10    are the two states that actually lose seats from an 

11    apportionment linked to the use of the corrected counts.  

12               What about the 33, because that is how we got 

13    into all of this?  The 33 are all pretty close to the line 

14    2.10.  So what we know is that relative to the other ones, 

15    they don't lose very much or gain very much by comparison 

16    with those that win a lot as a result or lose a lot as a 

17    result of the use of corrected counts.  Their bars are the 

18    ones that overlap zero and for which we are not really sure 

19    whether those dots belong exactly on the line, a little bit 

20    above the line, or a little bit below the line.  

21               What we certainly know is they don't belong up 

22    there with California, the District of Columbia, New Mexico, 

23    or down below with Pennsylvania and Wisconsin.  

24               So I just think that that comment misses 

25    completely what the data show.  What the data show is that 
                                                              1576

 1    adjustment clearly -- 

 2               THE COURT:  I have the points.

 3         Q.    Dr. Fienberg, the Secretary states at page 2-25 

 4    that evidence provided by the census bureau tends to support 

 5    the superior distributive accuracy of the actual 

 6    enumeration.  Is that true?

 7         A.    I'm not there yet.  2-25?

 8         Q.    2 dash 25.

 9         A.    Where are we on the page?

10         Q.    It is the last sentence.  It begins, "However, 

11    the evidence provided by."

12         A.    Got you.  It goes over to the next page, yes.

13         Q.    In your opinion, is that correct?

14         A.    No, it is not.  The census bureau did a number of 

15    comparisons.  It reported on those comparisons in one of the 

16    P studies.  They are actually at the back of P-16.  There 

17    are two parts of P-16.  In it they looked at distributive 

18    accuracy, that is, the relative shares for states and the 

19    relative shares for other areas, for places, I think is the 

20    term.  They did that using squared error loss.  They did 

21    that using absolute loss.  They did it using something that 

22    they called an apportionment loss function.  Then they did 

23    it in variations.  For each one of those, they did it over 

24    again three or four different ways.  

25               In every single comparison, every single 
                                                              1577

 1    comparison involving loss functions, the corrected census 

 2    counts are shown to be superior in terms of accuracy 

 3    relative to the raw census enumeration.  Further, in other 

 4    analyses reported in various documents leading up to the 

 5    decision by the Secretary, there were alternatives.  And in 

 6    test after test, when alternatives were considered, tests of 

 7    hypotheses, increased variance, you added 50 percent to the 

 8    variance, you doubled the variance, even those tests showed 

 9    the superiority of the corrected census counts to the raw 

10    census enumeration.  

11               So I haven't seen any evidence in any of those 

12    documents that tends to support the superiority of the raw 

13    counts, the raw census enumeration.

14         Q.    Dr. Fienberg, what is your overall assessment of 

15    the Secretary's decision?

16         A.    I come back to the statement that I made at the 

17    outset bought it provides the context.  We came into the 

18    census knowing that decade after decade the census bureau 

19    had documented the differential undercount of blacks and 

20    other minorities.  We knew that that was likely to be a 

21    problem in 1990.  Sure enough, it showed up in magnitudes 

22    that were larger than we have seen in any other census.  

23    That is what the demographic analysis said.  

24               The census bureau carried out a post-enumeration 

25    survey.  It was the most studied survey that the bureau has 
                                                              1578

 1    ever done.  It carried out a set of evaluations of these.  

 2    It looked at the various sources of error that have been 

 3    carefully identified over the decade, and it showed that 

 4    these sources of error were small and that their cumulative 

 5    impact did not prevent in any way the conclusion that the 

 6    raw census counts were clearly inferior in terms of 

 7    accuracy.  

 8               The Secretary looked at that evidence, and in 

 9    place after place he dismissed it.  He dismissed it usually 

10    making reference to hypothetical possibilities, to anecdotal 

11    information, to assumptions that he wasn't sure were fully 

12    true, when nobody believed that they were fully true, and he 

13    ignored the very evidence that the census brought to bear 

14    that showed that the impact of carefully chosen assumptions 

15    and a careful assessment of the error showed were not a 

16    serious problem in the 1990 census.  

17               He ignored the evidence adduced by the census 

18    bureau and instead relied upon alternative after alternative 

19    that I find implausible, and I think unfairly rejected the 

20    evidence presented by the bureau, which to me strongly 

21    supported the use of the corrected census counts based on 

22    the post-enumeration survey.  

23               MS. BARRY:  Your Honor, I have no further 

24    questions for the witness at this time.  

25               THE COURT:  We will resume tomorrow morning at 
                                                              1579

 1    9:30 with cross-examination.  

 2               (Adjourned to 9:30 a.m., May 21, 1992.  

 3    
      

 4

 5

 6

 7

 8

 9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25
