Disability Data From the American Community Survey: A Brief Examination of the Effects of a Question Redesign in 2003

Sharon Stern and Matthew Brault

U.S. Census Bureau, Housing and Household Economic Statistics Division January 28, 2005

1. Introduction (Footnote 1)

In 2003, the Census Bureau introduced a change to the disability items on the American Community Survey (ACS) mail-back paper questionnaire. The intent of the new layout and skip instructions was to reduce confusion among respondents and thereby reduce potential erroneous reporting. This report contains the initial results of an analysis describing the effects of the change. The evidence presented suggests that it is not possible to discern which portion, if any, of the difference between 2002 and 2003 disability estimates can be attributed to actual change in the number or percentage of people with disabilities. As a result, the Census Bureau will not present any time-series data showing estimates from 2002 and earlier with data from 2003 and later for the affected items.

The objective of this short paper is to describe the changes in the questionnaire and the effect on the published disability data. Section 2 provides some background useful for understanding the sensitivity of questions on disability. The section reviews important elements of the ACS data collection process involving differences between mail-back paper questionnaires and computer-assisted interviews (CAI). The section also provides details on previous research suggesting that the paper questionnaire may be confusing to respondents and result in erroneous reports of disability.

Section 3 presents the data from the 2003 ACS compared with previous years in the context of the question change. The section shows data on the percentage of people with disabilities and their employment rates, focusing on the differences in rates by the mode of data collection. Section 4 introduces topics for future research that, in addition to this paper, will add to the understanding of the effects of the questionnaire change. Section 5 presents a summary of the key findings.

2. Background

The ACS is a critical piece of Census Bureau’s plans to reengineer the decennial census. The plan is to replace data collected from a once every 10 years sample of the population on what is widely known as “the long form” with the data every year from the ACS. Toward that end, the Census Bureau increased the ACS sample to approximately 3 million addresses per year in 2005 in order to provide aggregated 5-year rolling-average estimates for all areas of the country as small as census tracts by 2010. In the years covered by this report, the ACS was still in a demonstration phase. In 2002, the ACS sample was about 800,000 addresses, which included interviews with approximately 1.2 million individuals in about 513,000 households; in 2003, the ACS interviewed approximately 1.3 million people in 572,000 household. (Footnote 2)

In the years covered by this analysis, the ACS included only the household population. People living in non-institutionalized group quarters such as college dormitories were not yet included in the ACS. Because the standard ACS data products further restrict the population to civilians, this working paper uses the same population for consistency. In contrast, the previous research described in Section 2b used the entire household population as that earlier paper compared ACS to Census 2000 and the household population was a group common to both surveys. Because neither ACS nor Census 2000 published data for disability in the full household population, that paper included several special tabulations. This brief working paper does not include any supplemental tables, so using a population consistent with other published data is essential. The only difference between the household population used in the previous research and the civilian household population used here is the exclusion of active military personnel identified through their employment status.

2a. ACS Data Collection

The ACS is conducted using three methods of data collection, mail, telephone, and personal visits. Every address in sample gets a pre-notice letter, the ACS questionnaire package, and a reminder card. A replacement questionnaire package is sent if no response is received from the address. Attempts are made to contact people at addresses who do not respond by mail using the phone. The remaining addresses — those not responding by mail or not contacted by phone — are sub-sampled for a personal visit. Trained, permanent Census Bureau representatives recording responses on a computer rather than paper conduct telephone and personal visit interviews.

About half the weighted data in 2003 came from respondents who filled out and mailed back a paper questionnaire. The remaining households were enumerated by telephone or personal visit from a trained Census Bureau representative. The weighted response rates in 2002 and 2003 were 97.7 percent and 96.7 percent, respectively. (Footnote 3)

This report examines differences in reporting of disability by data collection mode. Because the respondents are not randomly assigned to a mode (as described above, mail non-respondents get telephone or personal interviews), differences observed in disability estimates between the modes have two sources: actual differences in the characteristics of people interviewed in each mode and the differences resulting from the environment in which the people responded.

This report also compares 2003 ACS data to previous years within data collection mode to identify the effect of the questionnaire change. This element of the analysis is particularly important since the changes made in 2003 only involved the paper questionnaire. The computer-assisted version of the disability items used by the telephone and field representatives remained unchanged in 2003. The underlying assumption is that if the differences in estimates of disability reflected actual change in the population, it would affect both mail respondents and CAI respondents in a similar fashion.

The analysis of disability rates by mode and year suggests that observed changes between 2003 and previous years can be attributed to the question change. This analysis cannot rule out the possibility that there was a shift in society and perception about disability that coincided with the question change, but which only affected the way people filled out the paper questionnaire, because no sample was given the old (2002) paper questionnaire in 2003.

2b. Previous Research on ACS disability data

Evidence presented at the 2003 Joint Statistical Meetings suggested that Census 2000 and the ACS in 2000 included a confusing skip pattern and item wording resulting in over-reporting, specifically in the last two parts of the disability questions. (Footnote 4) Because Census 2000 and ACS used a similar paper questionnaire for mailout/mailback operations, but different means of collecting data in non-response follow-up operations, investigation of mode-effects was central to that research. The evidence suggested that people may have over-reported disability on the Census 2000 and ACS mailback paper questionnaires and on the Census 2000 paper questionnaire used by the non-response follow-up enumerators. Since the ACS continued to use the same disability items in 2001 and 2002, it seemed likely that the trend of over-reporting continued in those years.

Intending to clarify the intent of the question, the 2003 ACS questionnaire addressed the issue of over-reporting in those items by changing the location of the items and redesigning the instructions explaining which people were to answer which questions. Items 16c and 16d from the 2002 questionnaire were moved to the next page and were numbered 17a and 17b on the 2003 questionnaire. Question 17 included the same introductory text as item 16 for consistency, but the embedded skip instruction that had appeared parenthetically in 16c and 16d was placed in an instruction box preceding the newly numbered questions 17a and 17b. Figure 1 shows the disability items as they appeared on the 2002 ACS questionnaire, and Figure 2 shows the items as they appeared on the 2003 ACS questionnaire.

2c. The ACS Questionnaire and definition of terms.

As shown in Figures 1 and 2 and described in section 2b above, the concepts being covered have not changed in the 2003 questionnaire. The first question, number 15 on the 2002 and 2003 questionnaires, asked about long-lasting conditions. Part a referred to sensory disability which includes blindness, deafness, or a severe vision or hearing impairment. Physical disability, part b, referred to a long-lasting condition that substantially limits one or more basic physical activities. The second question asked about a physical, mental or emotional condition lasting six months or longer which makes certain activities difficult. Mental disability, part a, asked about learning, remembering, or concentrating. Self-care disability, part b, asked about difficulty in dressing and bathing.

Despite the layout changes in the 2003 questionnaire, the last two concepts are essentially the same as they were in earlier years. Go-outside-home disability (16c in 2002 and 17a in 2003) referred to difficulty going outside the home alone to shop or visit the doctor. Employment disability (16d in 2002 and 17b in 2003) referred to people that had difficulty working at a job or business. People 5 years and older were eligible respondents for the first four disability types, but only people 15 years and over were asked about go-outside-home and employment disability.

In summary, the change to the questions implemented in the 2003 questionnaire basically separated the last two parts of question 16 into their own new question. This was meant to address two suspected misinterpretations possible with the old layout. First, some evidence suggested that people were actually answering “yes or “no” to the skip instruction itself. This might mean a “yes” response to the item “c. (Answer if this person is 16 YEARS OLD OR OVER) Going outside to shop or visit a doctor’s office” really meant, “Yes, I am 16 years or older.” Second, some evidence suggests that certain respondents forgot the lead-in asking about difficulty with activities. Those people might have meant, “Yes, I do go outside to shop or visit a doctor’s office.”

3. Prevalence of disability and employment of people with disabilities.

Although the ACS collects disability data for people 5 years and over, this report only presents data for people 16 to 64 years old. Since this analysis focuses on changes to the questionnaire aimed at people responding to all of the disability items, it was essential to exclude people 5 to 15 years old. Further, since published data from the ACS have historically omitted information on employment disability for people 65 and over, this group was also excluded from the analysis.

3a. Disability by item by mode.

In 2003, 11.5 percent of the civilian household population 16 to 64 years old reported having one or more of the six types of disabilities. By displaying disability by data collection mode, Table 1 shows that the estimates of the percentage of people reporting at least one of the six types of disability differs by the mode of interview. (Footnote 5) In 2003, 11.0 percent of people responding by mail reported a disability, whereas the disability rate was 12.0 percent for people who had a Census Bureau representative call or visit them for a computer-assisted interview. (Footnote 6) Figure 3 shows that this represents a departure from previous years. In each of the 3 previous years of ACS data collection, the mail respondents were more likely than CAI respondents to report having one or more disabilities – in those years the disability rate for mail respondents was about 2.8 percentage points higher than the rate for people in the non-response follow-up group.

The comparison across years within data collection mode is equally instructive. The percentage of people reporting any disability was relatively stable over the 4-year period (2000 to 2003) for people interviewed by a Census Bureau representative, ranging from 11.5 percent to 12.0 percent. (Footnote 7) In contrast, the disability rate for mail respondents showed a clear break in the series in 2003. For 2000 to 2002, the disability rate showed no statistically significant differences, with rates ranging from 14.3 and 14.5 percent, whereas in 2003 the rate was lower at 11.0 percent.

While examination of the reporting of overall disability by data collection mode suggests that 2003 may be the result of the redesign of the disability questions, examining each contributing item individually further illuminates the picture. In addition to the main chart showing people with any disability, Figure 3 also includes separate charts for the percentages of people reporting each type of disability included on the questionnaire. ACS estimates based on the first four items from the questionnaire – sensory disability, physical disability, mental disability, and self-care disability – are consistent both across years within mode and in the relationship between modes across years. On the contrary, estimates based on the last two items from the questionnaire – go-outside-home disability and employment disability –have clearly changed in the 2003 ACS.

For an example of consistent reporting across years, consider physical disability. The difference between mail respondents and CAI respondents in the percentage of people reporting a physical disability was less than 1 percentage point in each of the four years, ranging from 0.6 percentage points in 2001 to 0.9 percentage points in 2002 and 2003. In all cases, the mail respondents were less likely to report a physical disability than the CAI respondents.

For an example of consistent reporting across years and modes, consider the estimates of self-care disability. In each year’s data from 2000 to 2003, the prevalence of self-care disability was the same across modes — 1.7 percent for mail respondents is not different from 1.8 percent for CAI respondents in 2000 and 1.8 percent for mail respondents is not different from 1.9 percent for CAI respondents in 2003.

In contrast to the findings of stability across the years for the first four items, go-outside-home disability and employment disability both experienced a noticeable change in pattern in 2003. For both of these items, the pattern in the years 2000 though 2002 was that of a higher disability rate for mail respondents than CAI respondents. The difference between mail respondents and CAI respondents in the percentage of people reporting a go-outside-home disability was about 2.5 percentage points in 2000 through 2002. Conversely, the mail respondents (2.9 percent) and CAI respondents (2.7 percent) reported go-outside-home disability at about the same rate in 2003. The same change in response pattern holds true for employment disability. The difference between mail and CAI respondents in the percentage of people reporting an employment disability was about 3.4 percentage points in 2000 through 2002. In contrast, the percentages of mail and CAI respondents reporting employment disability were closer to one another in 2003 -- 6.7 percent of mail respondents and 6.2 percent of CAI respondents reported an employment disability.

3b. Employment among people with disabilities. (Footnote 8)

Examination of the percentage of people employed for people reporting a disability brought to light one of two theories for how respondents might have misunderstood the disability questions. The rates of employment disability, the high rate of employment among people reporting employment disability, and other evidence suggested that some people were reporting their employment status rather than their disability status when faced with the pre-2003 question 16d “(answer if 16 years or older) Working at a job or business?” Table 2 presents data on employment for people with any disability by mode for 2000 to 2003.

Between 2000 and 2002, the percentage of people employed among mail respondents who reported at least one disability ranged from 47.0 to 49.3 percent. That rate fell to 35.8 percent in 2003. Figure 4 demonstrates that the difference between employment rates of people with disabilities reporting by mail versus those in the non-response follow-up group also changed. In 2002, the difference was 7.0 percentage points – with mail respondents with disabilities more likely to be employed. In 2003, the difference was 3.0 percentage points with the employment rate for mail respondents below the CAI respondent’s employment rate.

Comparing employment rates across the 4 years of data, people reporting sensory disability had employment rates that were the same or lower than the preceding year. This was true for mail respondents and CAI respondents. For example, about 50.2 percent of people reporting a sensory disability were employed according to their 2000 survey responses. In 2003, mail respondents reported employment at 45.7 percent and CAI respondents reported employment at 48.2 percent. This same pattern can be seen for people who reported a physical, mental, or self-care disability. The employment rates are generally consistent across years within modes.

People reporting “yes” to the last two disability items, go-outside-home disability and employment disability, had somewhat consistent employment rates when comparing 2000 through 2002, but showed a steep decline in employment rates in 2003 for mail-back respondents only. This decline was associated with a decrease in the difference between mail and CAI respondents. For people who reported a going-outside-home disability, the mail responders were more than twice as likely as CAI responders to report being employed in each year from 2000 to 2002. In 2003, the employment rates for these two groups were closer, 19.0 percent for mail respondents and 14.8 percent for CAI respondents. For people who reported an employment disability, the mail responders were more than twice as likely as CAI responders to report being employed in each year from 2000 to 2002. In 2003, the employment rates for these two groups were closer, 19.1 percent for mail respondents and 16.8 percent for CAI respondents. These changes indicate that the question redesign reduced over-reporting of these two disability types by mail respondents, the intended effect.

4. Future research

This preliminary analysis leaves many questions unanswered. Additional analysis can provide a more complete picture by answering the following questions:

• How did this questionnaire change affect the reports of disability from people 65 years and over?

• Did the questionnaire change affect demographic groups in a similar fashion? Are men and women equally likely to report differently in 2003 than in previous years?

• How has the change in reporting of disability affected estimates of the social and economic characteristics of people with disabilities?

• Did this change impact disability estimates for children?

5. Summary

In response to evidence suggesting that mail respondents were reporting certain disabilities in error, the Census Bureau introduced changes to the ACS questionnaire in 2003. The initial review of the data shows that fewer people reported go-outside-home disability and employment disability. Since the Census Bureau has no independent evidence of a national trend with regard to these types of conditions, and the apparent change coincided with the question redesign, additional analysis was essential. Since the changes only involved the ACS mail questionnaire, the focus of this preliminary review was differences between people reporting by mail and people reporting by telephone or personal interview. Comparison of these groups to each other and across time shows that the question redesign had the expected effect on the mail respondents, lowering the rates of disability overall by reducing over-reporting in the go-outside-home disability and employment disability.

Footnotes:

1 This report is released to inform interested parties of research and to encourage discussion. The estimates in this report are based on data collected in the American Community Survey. As with all surveys, the estimates may differ from the actual values because of sampling variation of other factors. All statements in this report have undergone statistical testing, and all comparisons are significant at the 90-percent confidence level, unless otherwise noted.

2 For more information, see <http://www.census.gov/acs/www/index.html>. The ACS Internet site includes details on methodology, including sample design, data collection, and data dissemination. For links to the American Community Survey Operations Plan and other related documents, see <http://www.census.gov/acs/www/SBasics/index.htm>.

3 For definitions or survey response rates, see

<http://www.census.gov/acs/www/UseData/sse/res/res_def.htm >.

4 Stern, Sharon, (2003), “Counting People with Disabilities,” Proceedings from 2003 Joint Statistical Meetings, (May): 4064-4071, see <http://www.census.gov/acs/www/Downloads/ACS/finalstern.pdf>.

5 In the ACS operations, households are not randomly assigned to mail, telephone, or personal interview. Rather, all addresses in sample receive a mail questionnaire first. It is only households that do not respond to the mail questionnaire who become eligible for a telephone or personal interview. Therefore, differences attributed to mode in this paper include both the identifiable differences in operations that may affect how people respond and the unidentifiable differences associated with the type of people who respond by mail and those who do not. For links to the American Community Survey Operations Plan and other related documents, see <http://www.census.gov/acs/www/SBasics/index.htm>.

6 The term “CAI respondents” is used in this analysis to indicate people who had a telephone or personal interview with a Census Bureau representative. This group is also called the non-response follow-up group.

7 This is an example of estimates that are close in a general sense, but some are statistically different when tested at the 90 percent significance level. In 2003 12.0 percent (+/- 0.1) of 16-to-64 year olds responding in the CAI group reported having at least one type of disability. This estimate is not different from that of 2002, whereas in 2001, the estimate for the corresponding group was 11.6 percent (+/-0.2). While the statistical test at the 90-percent confidence level shows the 2001 estimate to be different from the 2003 estimate, the differences are not considered meaningful. These differences are small as a percentage of the estimate and small in comparison to the differences observed in the reporting of go-outside-home and employment disability by mail respondents. Table 1 gives both the estimates and the confidence intervals

8 Because people with disabilities are more likely than people without disabilities to be out of the labor force, this paper examines the percentage of people employed. That means that the base of the percentage is not limited to people in the labor force. This differs from the way unemployment is generally reported, as the share of the labor force made up of people who did not work.

FIGURES

1 Image of the disability items on the 2002 American Community Survey.

2. Image of the disability items on the 2003 American Community Survey.

3. Percentage of People Age 16 to 64 with Disability

4. Employment Rate of People Age 16 to 64 with a Disability by Mode by Year

TABLES

1. Percentage with Specified Disability Type by Mode: 2000-2003

2. Percentage Employed of People 16 to 64 years old with Specified Disability Type by Mode: 2000-2003

3 Percentage of People Age 16 to 64 with Disability

4 Employment Rate of People Age 16 to 64 with a Disability by mode by Year