COUNTING PEOPLE WITH DISABILITIES: HOW SURVEY METHODOLOGY INFLUENCES ESTIMATES IN

KEY WORDS: Disability, Data Collection Mode,

Questionnaire Design, American Community Survey

1. Introduction (footnote 1)

According to Census 2000, 48.9 million people 5 years old and over living in housing units had a disability. (footnote 2) This represents 19.2 percent of that population. The Census 2000 Supplementary Survey (C2SS), a national sample that used the American Community Survey (ACS) design, estimated that 39.7 million people aged 5 and over living in housing units (15.6 percent) had a disability.

This research examines elements which cause the difference in the disability estimates between two sources.

First, this research examines the six items on disability from the questionnaire which are combined to create the overall disability rate. This step reveals that the magnitude of difference in employment

disability rates between the two surveys, C2SS and Census 2000, is greater than the differences in the other five disability items.

In the next stage of analysis, the six disability items are compared by the mode of data collection. Results indicate similarities in the rates at which some specific types of disability are reported by mail respondents in both surveys. They show that for two types of disability—difficulty going outside the home to shop or visit a doctor’s office and difficulty working at a job or business—the larger differences between the estimates of the two surveys occur in the people counted in non-response follow-up operations.

2. Terminology and the disability items.

This report uses specific terms to refer to the items on the questionnaire. The first question, number 16 on Census 2000 and number 15 on C2SS, asked about long-lasting conditions. Sensory disability, part a, includes blindness, deafness, or a severe vision or hearing impairment. Physical disability, part b, refers to a long-lasting condition which substantially limits one or more basic physical activities. The second question, number 17 on Census 2000 and number 16 on C2SS, asked about a physical, mental or emotional condition lasting six months or longer which makes certain activities difficult. Mental disability, part a, asked about learning, remembering, or concentrating. Self-care disability, part b, asked about difficulty in dressing and bathing. Go-outside-home disability, part c,

asked about difficulty going outside the home alone to shop or visit the doctor. Finally, employment disability, part d, asked if the person had difficulty working at a job or business. People 5 years and older

were eligible respondents for the first four disability types, but only people 16 years and over were asked about go-outside-home and employment disability. (Figure 1 shows the complete wording and the approximate layout of the questions from the mail-back questionnaires.)

At times, this report uses the term “with a disability.” Census 2000 and C2SS use the same process to determine a person’s disability status.

People were defined as having a disability if one or more of the following conditions were true:

• They were 5 years old or over and responded “yes” to a sensory, p physical, mental, or self-care disability;

• They were 16 years old or over and responded “yes” to a disability affecting going outside the home; or

• They were 16 to 64 years old and responded “yes” to an employment disability. (footnote 3)

3. Background

The ACS has been under development since the mid 1990s and began collecting data in four test sites in 1996. The Census Bureau designed the continuous measurement approach of the ACS to collect and disseminate demographic and socioeconomic information in a manner more timely than the decennial census sample products based on the long form. The ACS is one of the key elements in reengineering the census. If long form type

information is produced successfully using an ongoing survey, the decennial censuses of the future will be limited to short form data only.

As part of the ongoing research into the feasibility of performing a survey of this proposed magnitude during a decennial census, C2SS was

conducted in over 1,200 counties using ACS methods. The sample was sufficient in size to produce national- and state-level estimates.(footnote 4) At this time, the Census Bureau has a report already available which describes the results of C2SS as a test of operational feasibility. (footnote 5) The Census Bureau also has analysis and evaluation projects in progress that demonstrate the viability of continuous measurement as an alternative to including a long form in the decennial census.

3a. Overview: General Data Collection Operations

and Statistical Comparisons Census 2000 and the C2SS have vast and

complicated operations whose details are explained in several publications. For instance, The Census2000 Operational Plan provides details on the entire

Census 2000 operation. (footnote 6) The ACS has extensive online documentation of the procedures and explanations of the C2SS. (footnote 7)

Census 2000 and the C2SS operations both included two main components: mail-back and non-response follow-up (NRFU). During the initial mail-back, respondents were asked to fill in a questionnaire and return it to the Census Bureau for processing. During NRFU, people who did not mail back a questionnaire were contacted by Census Bureau personnel. Census 2000 employed many temporary enumerators to collect the data in person with a paper questionnaire. C2SS used permanent field representatives to interview the non-respondents either over the telephone or in person using a computer automated instrument.

For the purpose of this research, each survey has responses categorized into one of two modes, self-response (households who mailed a questionnaire, called “mail-back”) or interviewer-assisted response (households who did not respond by mail, but were captured in follow-up operations, called “NRFU”). C2SS had one mail-out questionnaire (self-response) and one automated instrument (interview-assisted response), which was used in the two main follow-up operations, computer assisted telephone interviews (CATI) and computer assisted personal interviews (CAPI). Census 2000 had many different paper questionnaires for their vast array of programs designed to improve coverage. In this report, the various Census 2000 forms also fall into two categories. This report treats two types of mail-back questionnaires as self-response. These are the mail-out questionnaire and the “update/leave” questionnaire, delivered by Census enumerators to housing units without standard addressing and therefore ineligible for mail-out. Census 2000 enumerator forms comprise the interviewer-assisted responses.

Estimates are weighted to represent the population. As with all surveys, estimates may vary from the actual values because of sampling variation or other factors. Explicit comparisons in this paper have undergone statistical testing and are significant at the 90 percent confidence level. Due to the large sample sizes in Census 2000 and C2SS, small standard errors of the estimates increase the chance of rejecting the hypothesis that two estimates are the same. When the scale of these differences is small, it may make no practical difference to users of the data. As a result, this paper focuses on differences that are more meaningful. Differences which are large in magnitude between Census 2000 and C2SS are the central theme of this research as they could result in different conclusions being reached by data users.

3b. Overview: Questionnaire Items on Disability

Prior to Census 2000, the Census Bureau, with the assistance of several federal interagency groups and commissions, developed a set of questions for inclusion on the Census Dress Rehearsal planned for 1998. A full discussion of this process is available in a Social Security Bulletin article. (footnote 8) In that process, the group developed and tested several different sets of questions for possible inclusion on the long form.

Ultimately, the questions included in the final set were chosen based on performance of the individual questions in the different sets used in the testing process. As a result, the final set was never tested as a unit before being included in the 1998 Dress Rehearsal.

For the 1999-2002 ACS, the disability questionnaire items were changed from their original (1996-1998) wording to match the Census 2000 long form. Figure 1 has an image of the disability items as they appeared on the Census 2000 long form mail-back questionnaire. The C2SS mail-back form was similar to the Census 2000 questionnaire in general; and the layout of the specific disability items on the C2SS questionnaire looked almost identical to the Census 2000 long form mail-back questionnaire.

Only the question numbers, the background color, and the size of the paper differed.

Although the wording was the same, the layout on the Census 2000 enumerator form was substantially different from the mail-back forms. (See

Figure 2.) For instance, on the mail-back questionnaire the main lead-in text is bolded and the subparts a through d are indented with the yes/no check boxes to the right of the question. The Census 2000 enumerator questionnaire has all the text of the question in bold, except the “Yes” and “No” next to the check boxes. The location of the check boxes is below each part of the questions rather than to the right. For person 1 on the enumerator form, the check boxes are on the same line, side-by-side. For the other people on the enumerator form, the check boxes are aligned vertically, one on top of the other. Finally, both the mail-back forms and the enumerator form include a skip instruction in the text of parts c and d of the second question. For the mail-back form it is in parentheses. For the enumerator form it is in italics.

In addition, the enumerator paper form has a column break in the middle of the disability questions. Figure 2 shows the layout of the disability

items for person 1 on the Census 2000 enumerator questionnaire. The dark line between items 17a and 17b indicates the location of the column break. For the other people on the enumerator form, the column break was between items 18b and 18c.

Finally, the C2SS CATI/CAPI instrument layout is different from the paper questionnaires due to its use of computer technology. The actual screens seen by the interviewer during the interview are very different. The question wording is almost exactly the same as the paper questionnaires. The computer automated instrument uses only small differences in wording to take advantage of the mode. For instance, the lead-in for the first item reads “I am now going to ask some questions about some long-lasting

conditions… Do you have any blindness, deafness, or a severe vision or hearing impairment?”

The key advantage of the computer assisted interview is that skip patterns are programmed into the computer. The interviewer is not responsible for knowing or confirming the respondents characteristics before proceeding to ask the question. Specifically, parts c and d of the second questions include the instruction to “Answer if this person is 16 years old or over.” A field representative with a computer assisted interview need not worry about this – the computer only shows these questions if the person is 16 years or older.

The wording of the disability questions on the paper questionnaires in Census 2000 and C2SS was identical, but the wording in the C2SS computer

automated instrument was slightly different from those since the skip instructions are part of the instrument. Consequently, variation in these and in related aspects of survey administration may explain differences in the resulting estimates. Specific types of errors may be related to the wording, layout, or other presentation aspects of the questions. These potential complicating factors include the following:

• Respondents may forget the context of the questions by the time they get to b, c, and d.

• The long lead-ins include several elements which respondents may not understand. For instance, “a physical, mental, or emotional condition”,

“lasting six months or more”, “any difficulty in any of the following activities.”

• On parts c and d of the second question in the mail return, respondents may have thought they were being asked if they were 16 years old or over.

Some of these factors might have been mitigated by trained interviewers who understood the questions well. While the field representatives (FRs) do not carry the manual with them, those familiar with it would know the meaning of the questions. For instance, in the ACS FR Manual under “difficulty working” is the explanation “If a physical, mental, or

emotional condition prevents or makes it difficult for the person to hold any significant employment, enter <1>”.

On the other hand, a “misunderstanding” might have been compounded by inexperienced or poorly trained interviewers or enumerators who themselves did not understand the questions and therefore placed

the emphasis in the wrong place. For instance, the Census 2000 enumerator long form has the last two items (18c and 18d, numbered 17c and 17d on the Census 2000 mail-back long form) at the top of a new column for person 2. Since the text of the questions is “working at a job or business?”, this could have been understood by the respondent as a question about their labor force status.

All these differences suggest that it is crucial to understand the impact of different modes of data collection on the disability estimates. Examining data by mode within one survey may suggest mode effects. This type of analysis might indicate that people using the mail-back form consistently respond differently to a given item than people interviewed in person. Hopefully an examination across surveys using some similar and some distinct modes explains survey-specific effects, as well. A remaining question

is the reliability of the assumption that the people who respond by mail in Census 2000 are similar to people who respond by mail to C2SS. Both were

required by law and administered in the same year. By the same token, it is reasonable to assume that people who were enumerated in Census 2000 NRFU would share characteristics with people in the CATI or CAPI universe in C2SS. These assumptions are examined for the general demographic characteristics of the population as well as for the disability status of

those people.

4. Population Characteristics

Census 2000 and C2SS had about 254.6 million people 5 years and older in housing units. (footnote 9) These two surveys produced similar estimates of some demographic characteristics.

• Approximately 12 percent of the people were Hispanic.

• Approximately 12 percent of the people reported the single race of Black or African American.

• The median age was 37.

Estimates based on C2SS and Census 2000 do differ on some basic characteristics, but while they are significantly different due to small standard errors of the estimates, these differences are small in magnitude.

• C2SS has 83 percent of the population living in families, compared to 84 percent in Census 2000.

• C2SS has 63 percent of the population living in married-couple families, compared to 65 percent in Census 2000.

The final weighted population estimates from the two surveys are not represented by the two data collection modes in the same proportion. In the

Census 2000 sample, respondents who sent in a mail-back questionnaire represented 71 percent of the housing unit population 5 and older. Respondents enumerated in NRFU represented 29 percent of that group. C2SS respondents using mail-back questionnaires represented 59 percent of the housing unit population. The remaining people responded to a telephone or personal interview and represented 41 percent of the housing unit population.

The demographic characteristics of self-respondents were different from the interviewer respondents in both surveys. For example, 10 percent

of Census 2000 mail respondents were Hispanic, compared to 17 percent of NRFU respondents. People who reported being White comprised a larger

percentage of the mil return universe. Specifically, White people represent 80 percent of the mail return and 66 percent of the NRFU. The reverse is true for people who reported being Black. The Black population was 9.7 percent of mail returns and 17 percent of the NRFU.

C2SS respondents also differed by mode. The prevalence of Hispanic origin among C2SS mail respondents was 7.3 percent, compared to almost 19 percent of respondents interviewed by a field representative with a computer automated instrument. It is important to note that the characteristics of the CATI group do differ from the characteristics of the

CAPI group. For example, 13 percent of CATI respondents reported Hispanic origin, compared to about 21 percent of CAPI respondents. However,

since both types of interviews—telephone and personal—are used to capture people who did not mail back a questionnaire, they are treated together in

this paper as a group comparable to those identified in Census 2000 NRFU.

The median age of self-respondents in the Census 2000 sample was 40 years old while the median age of people in the NRFU was 31 years old.

People who were 65 and over were more likely to mail back a questionnaire —84 percent versus 69 percent of people under 65. As a result, 15 percent of the mail-back population was people 65 and over, but only 7 percent of the enumerator universe was people 65 and over.

Similarly, the follow-up population in C2SS also differed from the mail-back population. The median age of self-respondents in C2SS was 41 years old while the median age of people in computer assisted interviews was 32 years old. (footnote 10) People who were 65 and over were more likely to mail back a questionnaire, 75 percent versus 57 percent of people 16 to 64 years old. As a result, about 17 percent of the mail-back population was people 65 and over, but only 8 percent of the interviewer universe was people 65 and over.

5. Disability Rates

The disability rates for people living in housing units are included in Figure 3 for each individual item. The first four items, sensory, physical, mental, and self-care disability, are based on people 5 years old and over. The last two items, go-outside-home and employment disability, include only people 16 years and over.

Although the differences are small in magnitude, the estimates from C2SS are slightly higher than the estimates from Census 2000 for each of the first four items. (This includes the rate for self-care disability which is actually different when taken to additional decimal places.) In contrast, the rates for the last two items are lower when estimated using C2SS than

when using Census 2000. Specifically, the prevalence of go-outside-home disability was 6.2 percent among people 16 years and over in C2SS and 8.5 percent in Census 2000. Similarly, the prevalence of employment disability among people 16 years and over was 9.4 percent in C2SS and 12.9 percent in Census 2000.

Figure 4 includes the disability rates for each item by survey and mode. Among the first four items the differences in the estimates by survey and mode are minimal and in some cases not statistically significant. For example, the sensory disability rates from C2SS, 4.1 percent among mail-back respondents and 4.2 percent among NRFU respondents, are not statistically different. The sensory disability rate for mail-back respondents in Census 2000 was 3.7 percent. Among the NRFU respondents, the rate was

3.2 percent. Although the rate for self-care disability does differ between the modes, the Census 2000 and C2SS rates are not different for a specific mode. The estimate for prevalence among mail-back respondents in both surveys is 2.6 percent and the estimate for the prevalence among NRFU respondents in both surveys is 2.3 percent.

Figure 4 also shows significant differences in the estimates for the last two items across surveys in the NRFU mode and within surveys between the modes. C2SS found a go-outside-home disability in 7.5 percent of the people in the mail-back universe, but only 4.1 percent of the people in the NRFU group. Similarly, C2SS found more people with an employment disability in the mail-back universe than in the NRFU group, 10.9 percent versus 7.2 percent.

By contrast, people 16 years and older in Census 2000 were more likely to report an employment disability if they were counted by an enumerator in NRFU rather than self-responding with a mail-back

questionnaire. Census 2000 found 10.9 percent of mail respondents reported an employment disability, compared to 17.7 percent of people captured with enumerators.

The comparison across surveys is important in this case. For employment disability, which had a clear within-survey and across-mode difference, comparing across-surveys and within-mode shows the full picture. Both Census 2000 and C2SS mail respondents reported an employment disability rate of 10.9 percent. But while the C2SS interviewers

found less employment disability, 7.2 percent, the Census 2000 enumerators found more employment disability, 17.7 percent. The two non-response follow-up operations resulted in a completely different estimate.

The results on the last two disability items by mode suggests that the mail respondents may be confused about what those items are asking. Since the people interviewed in the C2SS NRFU report lower rates of disability in these two categories, a plausible conclusion based on the design of the mail-back form is that some mail respondents – in both Census 2000 and C2SS - may have been telling us that “yes, they are 16 years old and over”. Since the people who were interviewed by a Census 2000 enumerator were far more likely to report an employment disability, they may have been saying “yes, they are employed”.

Figure 5 focuses on one final aspect of the differences between the people with disabilities as measured by item, survey, and mode. This figure

shows the employment rates for people 21-64 years old with disabilities. In this case people with one of the first three types of disability have employment rates which are close whether measured by Census 2000 or C2SS, whether measured by mail-back questionnaire or an enumerator/FR interview. For example, C2SS found an employment rate of 51 percent among people with a sensory disability captured on a mail-back form or in NRFU interview. Census 2000 found an employment rate for the people with a sensory disability in the mail-back group which was the same as the mail-back C2SS group and only slightly lower in the enumerator group at 49 percent. (footnote 11)

For the last two items, the employment rates differ across survey and mode in important ways. Forty-two percent of the people with a go-outside-home disability in C2SS captured on a mail return reported being employed. For people with the same disability but responding in a NRFU interview, the

employment rate was 19 percent. Conversely, although 45 percent of people with a go-outside-home disability in Census 2000 mail response reported working, 48 percent of the people with the same disability but captured by a NRFU interview were employed.

The results are even more dramatic for the final question, employment disability. In C2SS, 50 percent of the mail respondents who reported employment disability also reported being employed. In the NRFU group, only 21 percent of people with employment disability reported being employed. Census 2000 found a very different result. The mail respondents

with an employment disability were employed at a rate not too much higher than the C2SS mail respondents, 55 percent. However, among the people

with an employment disability captured in a NRFU interview, 75 percent were employed. This is a higher employment rate than people without an employment disability who were captured in a similar NRFU interview. (footnote 12)

The C2SS employment rates are consistent with the disability rates by mode in Figure 4. If we suspect that the NRFU interviewers found fewer people with these last two types of disabilities because some mail respondents were confused by question wording, then it makes sense that their employment rate would be lower than the people in the mail universe who also answered “yes” to the disability items. Along the same lines, if we suspect that the Census 2000 enumerator interviews had an added element of confusion about whether the person should report employment status or difficulty with employment status, then the higher employment rate for people in that mode make sense.

6. Conclusions

Two data sources—C2SS and Census 2000— administered in the same year using similar questions found divergent disability rates for the same

population. Other research has shown that disability is hard to measure consistently, but this report shows an example in which disability rates are sensitive to relatively minor differences both within surveys across modes and across two surveys.

The disability rates for sensory, physical, mental, and self-care disability are generally fairly close between the two surveys for the mail return population. In other words, whatever differences in those two forms – color and location on the page – the resulting disability rates were clearly describing the same population. Although the NRFU (interviewer/enumerator) mode disability estimates for the first two items were not as close as the mail returns, in both cases the Census 2000 rate is lower. An array of possible untested causes includes:

• different rates of item imputation and the different characteristics of those with item non-response, and/or

• C2SS adjusts for non-interview using weighting, but Census 2000 does not.

Since C2SS found a smaller percentage of people with a go-outside-home disability in the CATI/CAPI than in the mail, it is possible that mail respondents truly misunderstood the questions. It is a consistent theory across surveys, since even more mail respondents in Census 2000 responded positively to this item. However, since the Census 2000 enumerator respondents also had a high rate of go-outside-home disability, this possibility is only supported if Census 2000 enumerators were likely to

repeat the same skip pattern “mistake” as the mail respondents. It is unclear what the confounding factor in the enumerator interview is.

The Census 2000 enumerators found a questionable number of people with an employment disability, especially since both C2SS and Census

2000 mail respondents reported this type of disability at the same rate. This combined with the fact that the employment rates for people with this type of disability are much higher for the Census 2000 enumerator respondents, the evidence suggests that there may have been a problem with the Census 2000 enumerator interview and instrument.

7. Future Research

This paper has only scratched the surface of this difficult problem. The data here indicate that question design, survey administration, and interviewer training are all important elements in proper measurement of disability. Future research should follow three main paths: analysis of data available now, analysis of data we are collecting now, and testing of new questions, questionnaires, and administration methods.

Specifically, more work needs to be done analyzing the data collected so far. The ACS and the Supplementary Surveys used these questions in 2001 and 2002 as well.

The 2003 ACS instrument has made one minor change in the disability questions. Items 16c and 16d are now part of a separate question on a new page. Hopefully, the page turn, some new skip instructions, and a repeat of the lead-in will eliminate at least part of what we suspect may be a common respondent misunderstanding. Analysis of this data is crucial to further understanding this issue.

Finally, the disability research community should continue its good work in developing questions for surveys. New questions which measure

the conditions of interest should be designed which are optimized for the questionnaire and data collection mode. Testing is needed which shows how

carefully crafted questions can be used appropriately based on the type and style of the survey.

A comprehensive version of the paper is available upon request to sharon.m.stern@census.gov.

References

Adler, Michele C., Robert F. Clark, Theresa J.DeMaio, Louisa F. Miller, and Arlene F. Saluter. 1999. “Collecting Information on Disability in the 2000 Census: An Example of Interagency Cooperation.” Social Security Bulletin, Vol. 62, No. 4, pp 21-30.

Griffin, Deborah H. and Sally M. Obenski. 2001. “A Demonstration of the Operational Feasibility of the American Community Survey”, Census 2000 Testing, Experimentation, and Evaluation Program, U.S. Census Bureau (September 28).

U.S. Census Bureau, Census 2000 Operational Plan, U.S. Department of Commerce, December, 2000.

U.S. Census Bureau. 2001. Meeting 21st Century Data Needs–Implementing the American Community Survey: July 2001, Report 1: Demonstrating Operational Feasibility.

U.S. Census Bureau. 2002. Meeting 21st Century Data Needs–Implementing the American Community Survey: May 2002, Report 2: Demonstrating Survey Quality

FOOTNOTES

1 This paper reports the results of research and analysis undertaken by Census Bureau staff. It has undergone a Census Bureau review more limited in scope than that given to official Census Bureau publications. This report is released to inform interested parties of ongoing research and to

encourage discussion of work in progress.

2 All Census 2000 estimates in this report are based exclusively on the sample data. Although some of the reported characteristics are also available from the 100% data, disability data - the essential focus of this

report - are only available for the people in the sample. As a result, sample data is used for all characteristic estimates for consistency.

3 It is only in the disability status indicator that the employment disability status for people over 65 years old is excluded. In the rest of the analysis they are included.

4 C2SS produced estimates of population characteristics for all geographic areas with population of 250,000 or more. In 2001-2003, the

Census Bureau continued a national sample, which is generally called the Supplementary Survey.

5 Meeting 21st Century Data Needs – Implementing the American Community Survey: July 2001, Report 1: Demonstrating Operational Feasibility is available at http://www.census.gov/acs/www/Downloads/Report01.pdf.

6 This report is available on the Internet at

http://www.census.gov/dmd/www/pdf/Operational2000.pdf.

7 For more information see

http://www.census.gov/acs/www/SBasics/index.htm.

The ACS also has an operations plan available for 2003 and beyond at

http://www.census.gov/acs/www/Downloads/OpsPlanfinal.pdf.

8 The article, “Collecting Information on Disability in the 2000 Census: An Example of Interagency Cooperation”, is available online at

http://www.ssa.gov/policy/docs/ssb/v62n4/v62n4p21.pdf.

9 This is an example of estimates which are close in a general sense, but are statistically different when tested at the 90 percent significance level.

Specifically, the exact estimate for the number of people 5 years and over in housing units in the Census 2000 sample is 254,620,291. The confidence

interval for this estimate has a lower bound of 254,603,989 and an upper bound of 254,636,593. At the same time, the C2SS estimate of the number of people in housing units is 254,571,610. Since the standard error for this estimate is 23,130, in a test of whether the Census 2000 and C2SS estimates are the same, the null hypothesis would be rejected.

10 The median age for CATI respondents was 36 and for CAPI respondents it was 31.

11 A C2SS estimate of 51.0 percent for the mail-back group is not statistically different from 50.9 percent for the NRFU group. However, a Census 2000 estimate of 51.2 percent for the mail-back group is statistically different from the NRFU group.

The Census 2000 estimate of 34.5 percent employment among people with a physical disability in the NRFU universe is not different from the employment rate among people with a physical disability when measured in C2SS whether mail-back or NRFU. However, it is difference from the rate (34.1 percent) for the mail-back group in Census 2000 and the rate for the mail-back group in C2SS (33.8 percent) is different from the NRFU group, 35.8 percent.

For employment rates among people with a mental disability, the rate for people in the Census 2000 mail universe, 31.6 percent, is different from

the rate for people in the Census 2000 NRFU universe, 29.7 percent. The other employment rates do not differ.

12 People captured in NRFU without an employment disability had an employment rate of 72 percent.