American Community Survey (ACS)

The American Community Survey: Nonresponse Follow-Up in the Rockland County Test Site

Joseph J. Salvo and Arun Peter Lobo

Population Division
New York City Department of City Planning

Presented to the Annual Meeting of the American Statistical Association, Anaheim, California, August 1997.

The opinions expressed in this paper are those of the authors and do not necessarily reflect the views of the Department of City Planning or the City of New York nor the Census Bureau. The authors would like to thank Charles Alexander, Susan Love, and Gregg Diffendal of the Continuous Measurement Office of the Census Bureau, and Douglas Schuetz of the Rockland County Department of Planning, for their assistance.

Introduction

The American Community Survey (ACS) is the major vehicle in the Census Bureau's proposed Continuous Measurement System. While the decennial census has provided detailed socioeconomic data, the ACS is slated to provide these data throughout the decade (Alexander and Davis, 1997). The ACS is presently being tested in a monthly self-administered survey at eight sites across the U.S., and current plans call for a survey of approximately 40 counties beginning in 1999. Coverage would extend to most of the U.S. by 2001 and to every county in the U.S. by 2003, by which time the monthly sample size would total 250,000 households. Reliable estimates for political and statistical areas, down to the block group level, would be derived through the aggregation of data from successive samples. The ACS is intended to replace the decennial census long-form in 2010, but will not replace the decennial census short-form, which will remain the sole source of data for census blocks.

In its simplest form, the ACS, like the census, can be viewed as an operation involving the following: the compilation of a detailed list of addresses¹ where questionnaires can be delivered; mailing forms to each address with a request that they be mailed back; and follow-up of non-respondents to collect data that are missing (Ericksen and DeFonso, 1993). Much of the ACS evaluation, thus far, has focused on estimates of sampling variability. This is understandable, because of concerns about the reliability of small area (e.g., census tract) estimates, and since variability due to sampling is relatively easy to quantify. Yet, there are major non-sampling issues — such as the ability of the ACS to achieve high levels of response — that must be addressed if its promise is to be realized. A high level of nonresponse, especially at a small area level, can compromise the quality of the estimates. Effective nonresponse follow-up is a key to achieving a high response rate. The purpose of this analysis, which focuses on the ACS test site in Rockland County, is three-fold:

1. to compare the mail return rates for the ACS and the 1990 decennial census by census tract;
2. to examine how the ACS and census mail return rates are related to sociodemographic characteristics at the tract level; and
3. to assess the effectiveness of nonresponse follow-up. This involves comparing allocation rates at the tract level, in the ACS and census, on major socioeconomic variables.

The Rockland County Test Site

One of several counties in New York City's suburban ring (Figure 1), Rockland had a population of approximately 265,000 enumerated in 1990, with nearly 85,000 occupied housing units. On average, Rockland’s tracts were nine percent nonhispanic black and six percent Hispanic (Table 1).² However, residential patterns were marked by a high degree of racial, ethnic, and socioeconomic segregation. For example, Spring Valley (0122, 0123, and 0124) and Haverstraw (0107), which accounted for 12 percent of Rockland’s population settled 42 percent of all blacks and 36 percent of Hispanics in the county. These two areas were also home to about one-quarter of the poor and the foreign-born. Similarly, New Square (011503) and Kaser/Monsey (0121), which accounted for eight percent of the county’s population contained over 80 percent of all Yiddish-speakers, most of whom were Orthodox Jews. As a result of these concentrations, many tracts in Rockland resemble those in New York City, an area with a much higher degree of racial and ethnic diversity.

The ACS data used in this analysis were collected over a 13 month period, from December 1995 to December 1996.³ The primary means of collecting data was through a self-administered questionnaire mailed to an approximately 15 percent sample of households in the county.⁴ Persons in group quarters were not enumerated.

Mail Return Rates for the ACS and the 1990 Census

The mail return rate refers to the percentage of occupied housing units that return a questionnaire.⁵ Table 2 presents Rockland tracts grouped into those with low, mid-range, and high mail return rates. While the overall mean ACS mail return rate was 61 percent, rates varied dramatically by tract, ranging from a low of 26 percent to a high of 77 percent. Correlations between mail return rates and tract characteristics in 1990 are presented in Table 3. Mail return rates were strongly and positively correlated with home ownership (r=.80), with household income (r=.69), and with a college degree (r=.68); there was a strong negative correlation with the percentage of a tract that was foreign-born and with poor English proficiency (each with r=-.78), with the percentage of a tract that was black (r=-.71), and with poverty (r=-.57). Not surprisingly, Spring Valley (0122, 0123, and 0124) and Haverstraw (0107) that had among the highest proportions of minorities, the poor, and the foreign-born had among the lowest mail return rates. Low mail return rates were also evidenced in New Square (011503) and Kaser/Monsey (0121), which had high concentrations of Yiddish-speakers. The concentration of blacks and the poor in tracts with low mail return rates can be gauged from the fact that the 10 tracts with the lowest mail return rates were 23 percent black (compared to the mean of nine percent for the county) and had a mean poverty rate (15 percent) that was more than twice that of the county average. Residents of tracts with the highest ACS mail return were primarily native-born, owned their home, and had high levels of education and income. While zero-order correlations can pinpoint the importance of specific demographic and socioeconomic variables, these variables also interact with each other in complex ways to determine mail return rates (an area that is beyond the primary scope of this paper).

Mail return rates by tract for 1990 census long forms ranged from 34 percent to 85 percent, with an overall rate of 72 percent. Sociodemographic factors that were highly correlated with return rates in the ACS were also correlated with census mail return rates. Nevertheless, the census had a higher mail return rate than the ACS in 37 of 38 tracts, including tracts that comprise Spring Valley and Haverstraw. Only in New Square (011503), did the ACS (44 percent) have a better mail return rate than the census (34 percent). This tract, in which 74 percent of the residents spoke Yiddish, also had the highest poverty rate in the county.⁶ Nevertheless, despite its relatively low ACS mail return rate, this rate was still 10 points higher than the census mail return rate, making it distinct among tracts with low socioeconomic status.

The mean census mail return rate for long forms (72 percent) in Rockland County was 11 points higher than that of the ACS. The higher census mail return rate may be partly a result of the immense publicity that accompanies a decennial census, what Love and Diffendal (1997) call the "census environment factor." This translates into great public awareness and may partly account for the census mail return advantage (Steffey and Bradburn, 1994:189-90). The lower ACS rate may also be a reflection of the continued drop in the level of public cooperation over time — the U.S. mail return rate for long forms declined from 82 percent in 1980 to 70 percent in 1990 (U.S. Bureau of the Census, 1993:6-29). Moreover, since the 1990 decennial census, the level of racial, ethnic, and linguistic diversity in Rockland County has increased, primarily as a result of the growth in its immigrant population. The growth in these hard-to-enumerate populations may also partly account for the lower overall mail return rate in the ACS.⁷

Nonresponse Follow-up for the ACS and the 1990 Census

The methods employed in the ACS field operation attempt to emulate most of the procedures of the long-form decennial census operation. For both the ACS and the decennial census, households can be divided into two groups: those that mail back questionnaires and those that do not. Only those that do not mail back questionnaires are subject to nonresponse follow-up.⁸ In the last census, if a household did not respond via mailback, a "field follow-up" ensued with the household being visited by a field interviewer. In the ACS, households that did not mail back questionnaires were first followed-up by telephone, using CATI methods. If contact via CATI proved unsuccessful, these households were eligible for CAPI follow-up. CAPI field visits were conducted for a one-in-three sample of these remaining households.

A potential benefit of the ACS is its use of experienced interviewers in field follow-up, relative to the decennial census. The ACS field follow-up is conducted by interviewers who are likely to be long-term employees of the Bureau who have received substantial training and have considerable field experience. In comparison, in the last decennial census, follow-up involved the use of large numbers of temporary workers with varying levels of experience and training.

As the percentage of housing units that mail back forms declines, the burden increases on field workers to complete the enumeration.⁹ Since the ACS mail return rate was 11 points lower than that of the census, nonresponse follow-up increased by a similar margin. Table 2 shows that the mean follow-up required by the ACS was 39 percent compared to 29 percent for the 1990 census.¹⁰ The amount of follow-up required, especially in the ACS, was substantially higher in many census tracts. In six tracts, over one-half of the ACS questionnaires were captured via field follow-up, but this was true for only two tracts for the decennial census.

Unlike the last census, where all nonresponding households were followed-up, the ACS field follow-up occurs for a one-in-three sample of nonresponding households, which helps keep costs down. However, failure to obtain data from this sample of nonrespondents — which can be exacerbated in a low-publicity operation such as the ACS — could result in estimates based only on a small number of actual responses. Steffey and Bradburn (1994: 193) and Edmonston and Schultze (1995:132-34) argue that this could potentially compromise data quality, especially in small geographic areas. On the other hand, these same researchers have argued that the continuing presence of an experienced staff and ensuing operational efficiencies could actually serve to enhance data quality (Steffey and Bradburn, 1994:192; Edmonston and Schultze, 1995:131).

The targeted nature of nonresponse follow-up in the ACS could also result in better data compared to the 1990 census. During the last weeks of the 1990 enumeration, when repeated attempts to contact nonrespondents failed, field workers sometimes employed "last resort" enumeration, where a few basic short-form census items about a household were obtained from neighbors (U.S. Bureau of the Census, 1993). Twenty percent of all the field follow-up in central cities yielded three or fewer basic census variables, largely as a result of "last resort" methods employed in 1990 (Ericksen and DeFonso, 1993:39). These methods often result in high levels of "counting error" — erroneous enumeration that occurs because of poor field work (Ericksen and DeFonso, 1993:39; Navarro, 1990). Such erroneous enumeration occurs because of fabricated questionnaire responses (i.e. "curbstoning"), duplicate responses, and respondents counted in the wrong geographic location. Since every last nonresponding household was to be followed-up, endless cycles of visits to nonresponding households took place in many census district offices. These problems were exacerbated by the one-shot nature of the census and the ensuing shortage of trained field staff, and with the time pressures inherent at the advanced stages of a census enumeration. The nature of operations in the ACS makes last resort techniques unnecessary; in fact, interviewers are usually not permitted to obtain questionnaire responses from neighbors. This illustrates how the continuous nature of the program allows for the interactive identification of problems and solutions that affect census coverage, thereby improving coverage of the survey over time. Thus, it has been argued that a more focused ACS follow-up on a sample of nonresponding households should result in more concentrated and effective data collection, curbing the level of nonsampling error.

Allocation in the ACS and the Census

Despite intensive nonresponse follow-up efforts, interviewers are oftentimes unable to garner complete information on all households.¹¹ Missing information is then allocated — missing values for individual items are filled in on the basis of other reported information for the person or household, or in some cases, for persons or households nearby with similar characteristics (Edmonston and Schultze, 1995: Appendix G; U.S. Bureau of the Census, 1993). Values that are produced as a function of allocation are referred to as being imputed, a reference to the output of algorithms that are used to match respondents with similar characteristics and assign values based on such links.

The level of allocation can be a good indicator of data quality. High levels of allocation could result in biased estimates as the characteristics of those missed may differ from those who respond in the survey (Groves, 1989). As discussed earlier, it is virtually impossible to quantify the effects of this bias, as with most forms of non-sampling error. However, it is possible to examine allocation levels and to utilize such analyses as a measure of how much credence to place in the data.

In this analysis, the percentage of values allocated, (hereafter referred to as the allocation rate), was examined for six variables included on both the 1990 Census and the 1996 ACS questionnaires. The choice of variables was guided by attributes that help define hard-to-enumerate groups. These variables were: place of birth; language (whether a language other than English was spoken at home); mobility (whether a respondent lived in the same house or apartment five years earlier); occupation; person income; and household income. Figure 2(ab) depicts allocation rates in the ACS and the 1990 census for each variable, across 38 tracts. Tracts are in ascending order of mail return rates, going left to right on the X-axis. Table 4 presents differences between the allocation rates in the ACS and the census, for each tract and tests for significance. Tracts are sorted by the ACS mail return rate, in ascending order. A negative difference in Table 4 indicates that the ACS had a lower allocation rate, compared to the census. All means presented are unweighted for population size. It is important to remember that allocation rates presented in this paper represent allocation associated only with nonresponse follow-up; they do not include any mail return cases that received allocated values.

The ACS had significantly lower levels of allocation than the 1990 Census on four of the six variables and the dispersion of rates was noticeably lower on the ACS. For place of birth, two percent of the values in the ACS were allocated, significantly lower than the six percent for the 1990 census. Moreover, the distribution of scores in the ACS was more tightly concentrated around the mean, as reflected in the lower standard deviation (2.3 versus 3.6 for the census). The lower ACS rates across tracts can be clearly seen in Figure 2(cd). Differences in allocation rates between the ACS and census were highest in tracts with the lowest mail return rates, indicating that the ACS does a much better job in these hard-to-enumerate tracts.

There was greater allocation on language, but the ACS did significantly better than the census, especially in tracts with the lowest mail return. The mean allocation rate for the ACS (five percent) was one-half that of the census; standard deviations of 10.0 and 8.7, respectively, indicate less uniformity from tract to tract. The ACS distribution is skewed by the presence of an outlier (tract 010802) where over one-half the responses were allocated, compared to zero allocation for the census. The smaller dispersion in the ACS allocation rates is also reflected in the medians of the two distributions: just over one percent for the ACS compared to eight percent for the census. The standard deviation of the ACS distribution is less than three-quarters that of the census distribution when tract 010802 is deleted

Like place of birth, allocation rates for mobility were very low in the ACS, about two percent compared to six percent in the census, the difference being significant.¹² The uniformity of ACS scores, which can be clearly seen in Figure 2(ef), is striking and is reflected in a standard deviation of just 1.3 versus 3.7 for the census distribution. As with place of birth, differences in allocation rates between the ACS and census were highest in tracts with the lowest mail return rates.

Thus, on all three social characteristics, mobility, language, and place of birth, the ACS allocation rates were not only significantly lower than that of the census, but they did not vary greatly by tract. In contrast, the census allocation rates were sensitive to mail return rates, with the highest allocation rates in tracts with the lowest mail return. As a result, differences between the ACS and census allocation rates on the three social characteristics were significantly higher in tracts with low rates of mail return. With respect to place of birth, for example, on average, the ACS allocation for Rockland was four points lower than that of the census; however, in the ten tracts with the lowest mail return, it was over six points lower. Similarly, with respect to language, while the ACS allocation was five points lower on average, it was eight points lower in tracts with low mail return and nearly 10 points lower in tracts with average mail return rates. This pattern is indicative of more effective ACS field follow-up, relative to the 1990 Census in areas that have traditionally been difficult to enumerate. However, census and ACS allocation rates converge in tracts with high mail return rates. Thus, on mobility, the ACS allocation rate was seven points lower in tracts with low mail return, but only two points lower in tracts with high mail return rates. On birth place and language, there were no significant differences in allocation rates in the tracts with high mail return. The census has usually done well in these middle-class tracts and the ACS is unable to consistently do better on social characteristics.

The picture was mixed with respect to economic variables. The mean ACS allocation rate on occupation (7.3 percent) was not significantly different from that of the census (8.6 percent). However, the ACS had significantly lower allocation rates in 9 of 10 tracts with low mail return. While the ACS also fared better than the census in a few tracts with high mail return, the census did significantly better in two of these tracts.

The census did even better than the ACS in these high mail return tracts on person income, with significantly lower allocation in 5 out of 12 tracts. But once again, the ACS had significantly lower allocation rates in tracts with low mail return. For the county as a whole, the ACS allocation rate (23 percent) was significantly lower than that of the census (26 percent). Allocation rates were generally high on both the ACS and the census on person income. This is not only related to a reluctance on the part of respondents to answer questions pertaining to income, but to the complex manner in which the data were aggregated. On both the ACS and census, person income was an aggregation of seven separate kinds of income; missing information on even one kind of income resulted in an imputation for person income. Obtaining information on household income is even more problematic as this involves totaling person income for every member of the household. As a result, mean allocation rates were even higher, at 34 percent on both the ACS and the census. Where significant tract differences between the ACS and census exist, they tend to favor the ACS in low mail return tracts and the census in in high mail return areas. Again, this supports the contention that there may be some benefit to the census environment as a factor in eliciting response to income questions.

Thus, as with social characteristics, the ACS had significantly lower allocation rates on economic variables in tracts with low mail return. The ACS’s strength lies in its capacity to obtain responses in census tracts that possess substantial populations that are hard-to-enumerate. However, in tracts with high mail return, the census tends to be more effective on economic variables.

Discussion and Conclusion

The mean mail return rate of 61 percent in the Rockland County ACS was 11 points lower than that of the decennial census. The lower ACS mail return may be due to the absence of the immense publicity that accompanies the decennial census (the "census environment factor"), to a continued drop in public cooperation over time, and to the growth in hard-to-enumerate groups since 1990. Despite the lower ACS mail return rate, and the ensuing higher level of nonresponse follow-up, mean allocation rates for households that did not mail back questionnaires were significantly lower for the ACS on four of the six items examined — place of birth, language spoken at home, mobility (same house five year ago), and person income. For occupation and household income, mean allocation rates in the ACS were not significantly different from those in the census.

The examination of allocation rates at a census tract level, however, reveals a far more complex picture. The lower mean allocation rate in the ACS can be traced to lower levels of allocation across most tracts, but especially in tracts with low mail return. Moreover, the degree of dispersion in allocation rates is also lower than that of the census. In contrast, census allocation rates are far more sensitive to the mail return rate — the census's reliance on allocation increases substantially in harder to enumerate areas. Thus, it is in areas with the lowest mail return rates where the ACS does much better, relative to the census. At the other end of the spectrum, in tracts where mail return rates are highest, ACS and census allocation rates tend to converge on social characteristics, but the ACS has higher allocation rates on economic variables.

The ability of the ACS to elicit higher response in field follow-up in tracts with low mail return rates leads one to conclude that ACS field follow-up is a powerful and versatile tool. This may be related to the use of experienced field workers and the continuing nature of the survey. In Rockland, 40 percent of the field staff had previously worked on other Census Bureau surveys. Tracts with low mail return have a high proportion of hard-to-enumerate groups, many of whom are willing to cooperate, but need outreach that the ACS was able to provide. However, in high mail return tracts, the ACS was less successful than the census on economic variables. This may be due to the increasing reluctance of many to divulge information on sensitive items such as income, which may be exacerbated in the absence of a "census environment." Thus more effort is needed to publicize the ACS at a local level, so that more people are aware of it and see how ACS data may be used to apportion public and private resources in the county. Given the continuous nature of this program, it should be possible to coordinate some of this outreach with an annual release of data. Such an effort is currently underway in Rockland.

The lower allocation in the ACS may also be due to the follow-up of only a one-in-three sample of nonresponding households. While there were concerns that failure to obtain data from the sample of nonrespondents could mar data quality, the more focused follow-up may have resulted in more effective data collection. The merits of a follow-up on only a sample of nonresponding households have led the National Research Council to recommend such a procedure for the 2000 census (White and Rust, 1997).

With increasing racial and ethnic heterogeneity in urban areas, the problem of nonresponse is likely to increase in the coming years. Lower mail return rates translate into a more labor-intensive census because of the follow-up required on those who fail to respond. Large numbers of recent immigrants, many of whom do not speak English and/or come from nations where oppression makes them fearful of government-sponsored activities, will be a challenge to properly enumerate, especially in the big cities. Other Americans who have traditionally been hard to enumerate will continue to make the census counts problematic — these include persons without a usual residence, the urban and rural poor, and persons who are alienated from mainstream society (Steffey and Bradburn, 1994). At the same time, skilled personnel required to conduct the enumeration are becoming more difficult to recruit. The Continuous Measurement Program, through its main vehicle the ACS, was designed to be an effective alternative to the collection of census long-form information. It is thus essential that more attention be paid to the harder-to-quantify, but nonetheless important elements related to the procedures and operations of the ACS. In the final evaluation, such non-sampling considerations may prove to be as important as sampling issues in deciding the quality of small area data. Continuing to test the integrity of these operations is essential toward a refinement of methods that may ultimately determine the course of survey data collection for the nation in the next century.