American Community Survey (ACS)

The 1996 American Community Survey Monthly Response Rates, by Mode

Susan Love and Gregg Diffendal

U.S. Census Bureau
Washington, DC 20233

Presented to the American Community Survey Symposium, March 1998.

This paper reports the results of research and analysis undertaken by Census Bureau staff. It has undergone a more limited review than official Census Bureau publications. This report is released to inform interested parties of research and to encourage discussion.

The American Community Survey (ACS) has been designed and is being tested as a replacement for the decennial census long form. The 1996 ACS was the first full test of the operations and systems for the survey. It was conducted in only four sites, three of them urban (Portland City and Multnomah County, Oregon, Brevard County, Florida, and Rockland County, New York), and one rural (Fulton County, Pennsylvania). The data collected in these four sites were used to estimate their 1996 calendar year demographic and socioeconomic characteristics.

The ACS samples for the three urban sites were drawn from the Master Address File (MAF). The MAF, produced by the Census Bureau’s Geography division, is a merging of housing unit addresses from the 1990 census Address Control File (ACF) and the current United States Postal Service’s (USPS) Delivery Sequence File (DSF) of residential addresses. A MAF could not be produced for Fulton County because the county lacks a house number/street name addressing system. The address list used to sample units in Fulton was created by field personnel hired by the Census Bureau.

For most areas a 15 percent systematic sample of housing units was selected for the 1996 test. For small governmental units (places with less that 1000 housing units) a 30 percent sample was selected, corresponding to the oversampling done in the decennial census to increase the reliability of the census long form data estimates for small places. The 1996 ACS sampling rates were higher than what is planned when ACS is in full production in 2003, but it allowed us to produce estimates of characteristics for all tracts and block groups after a single year of data collection.

Each ACS sample unit was systematically assigned to a test month, or panel, and the data is collected over a three month data collection period. The designated panel dictated when the mail phase of data collection began for sample units. Urban site units not returning an ACS questionnaire within their panel month and for which a telephone number had been obtained from a commercial vendor were included in a Computer-Assisted Telephone Interview (CATI) nonresponse follow-up operation during the second month of data collection. All sample units--urban and rural--remaining uninterviewed after two months were subsampled at a rate of one-in-three and sent to a field Computer-Assisted Personal Interview (CAPI) nonresponse follow-up operation. Because of this unique survey data collection design, ACS has provided considerable insight into response patterns and characteristics of the test site populations for each of the three data collection methods. This paper attempts to describe some of them, and is based on results from the first 12 months of ACS data collection.

The Mail Phase

During the 1996 survey, every unit selected received a self-response questionnaire, delivered by the USPS in the urban sites or by an ACS field representative in the rural site. Urban sample units could receive up to four first class mailing pieces. A pre-notice letter was sent during the third week of the month prior to a unit’s panel month, informing the selected households about the ACS and that they would be receiving a questionnaire in the mail shortly. The initial questionnaire package was mailed about a week later. A reminder card was mailed to each unit about a week after the initial package, requesting that the household return the completed form if it had not already done so, and thanking the household if it had. About two weeks after the reminder card mailing, a second questionnaire package was sent only to those units that had not returned a completed questionnaire up to that point. The mail out procedure for urban sample units could not be used in Fulton County because no stable mailing system existed. Questionnaire packages were delivered to the sample units by Census Bureau field representatives using location descriptions and map spots noted during the earlier address listing of county addresses. No prenotices, reminder cards, or second questionnaire packages were delivered.

The content of the 1996 ACS questionnaire was very similar to that of the 1990 census long form. The design of the form, however, was changed to make it shorter and easier for respondents to complete. Complete survey data could be collected for only five persons on the ACS questionnaire, instead of the space for seven on the 1990 census long form. However, a concerted attempt was made to contact every household whose mail return indicated more than five members through a failed edit follow-up telephone operation, and continuation forms were completed for the remaining household members. The ACS form was also simplified by combining the 1990 census 100% and sample housing questions and reordering them more logically, reducing the number of skip patterns. Instructional banners were also added, particularly in the detailed population section of the form, to make it easier for respondents to follow.

Table 1 summarizes mail response over the first year of the survey for each site and overall.

Table 1. Summary of Mail Response for the November 1995 through October 1996 Samples

1/ Total Sample is the denominator for the Initial Q check-in rate and the Total Mail Response rate.

2/ Second Mailing is the denominator for the Replacement Q check-in rate.

3/ Total Mail Response is less than the sum of Initial Qs Checked in and Replacement Qs Checked in since some households returned both questionnaires.

The ACS staff watched with great interest the mail response rates month after month, since the success or failure of the mail data collection phase would probably decide the fate of the continuous measurement concept. The survey had to elicit a reasonably consistent response rate, and we were looking for seasonal fluctuations that might seriously impact the survey design.

The mail response rates for each sample in each site over the first twelve panel mail outs are shown in Figure 1. These rates are calculated by dividing the number of units returning a questionnaire or providing the information over the phone to Telephone Questionnaire Assistance (TQA) by the panel’s total sample size. ACS mail response rates provide the proportion of the sample interviewed by mail and TQA, and measure the size of the interviewing workload remaining.

Since mail response rates are based on the total number of addresses selected for the survey they can be greatly influenced by the number of vacant units in sample, as well as addresses that are nonexistent. Mail returns are accepted over a three month period, until the panel is closed out after the completion of the CAPI data collection phase.

The mail response rates month after month for Multnomah and Rockland were surprisingly stable. Mail response in Fulton County, with its small sample sizes, were highly susceptible to variations in the number of vacant and nonexistent units each month. Brevard County definitely provided the most interesting changes in mail response over the year. Its gradual increase in the response rate during the late fall and into the winter months, and continuous drop in rate from April through the summer months was just what we thought we might see.

The ACS adopted a rather unique concept of residency that considers anyone staying at a sample address for more than two months to be a current resident of the unit, and therefore eligible to be interviewed for the survey. This concept of residency is quite different from the usual residence concept used by the decennial census and most other demographic surveys. It changes the definitions of occupied and vacant units, and allows a unit’s status to change more often since it is not dictated by occupants’ one and only one "usual" residence. Areas where large numbers of people stay for extended periods of time but are not their "usual" residence would be most effected by this new concept. Of the sites in the 1996 test, Brevard was expected to be the one most affected. And it does seem that we are watching the movement of "snowbirds" in and out of the county, the subsequent change in the vacancy rate reflected in the rising and falling of the mail response rates. When units are sitting empty, or are occupied by short term vacationers, fewer questionnaires are returned by mail!

The ACS staff has been very pleased with the mail response that the survey has enjoyed, having concentrated on adopting methodologies that had been shown to impact self-response. Shortening and simplifying the questionnaire was foremost, and being the first survey considered mandatory by OMB helped, too. But we also implemented the multiple mail contacts found to have a positive effect on mail response. The most important component is the mailing of a second questionnaire package to addresses not returning the initial questionnaire by a certain date. Figure 2 shows the overall picture for mail response rates for the first year, and illustrates the importance of the replacement mailing on the success of the mail data collection phase. It plots the total mail response rate for each site against the response rate to just the initial questionnaire. The difference in the two rates shows the impact of the replacement mailing on total ACS mail response. It is remarkably similar in the three urban sites, adding 10 percentage points to the initial response in each site. The fact that some respondents returned the initial form after receiving the replacement only serves to understate the actual impact of the replacement mailing shown in our data. Fulton County’s mail response rate did not have the benefit of the delivery of a replacement questionnaire package. The county’s initial and total rates are the same. But the rate is comparable to the initial rate for Multnomah and Brevard, and we would expect a replacement form to have increased Fulton’s mail response by 10 percentage points, too.

Mail response rates determine the amount of interviewing work left to do after the self-response data collection phase. These rates can be calculated at any point in the process because they do not require that the status of all the sample units be known. Calculating mail return rates--the true measure of public cooperation with the survey--must wait until all phases of data collection on a sample panel have been completed since they are based only on those units that are occupied. Mail return rates are not influenced by vacant or nonexistent units.

Figure 3 shows the mail return rates for each sample panel in each site. These rates certainly proved to be more stable within and between sites than the mail response rates. Fulton is a good example. Although the mail response rate in Fulton County was consistently the lowest of the test sites, this is not true for Fulton’s mail return rate. Another welcome result can be seen in the Brevard mail return pattern. Removing the vacant and nonexistent units from the Brevard panels shows that public cooperation from month to month in this site is remarkably high and stable, regardless of the season or the population migration. Although Multnomah showed the highest mail response for the year, Brevard came in first in cooperation with the highest mail return rate--70 percent!

The Telephone Phase

The telephone data collection phase is aimed at those ACS addresses in the three urban sites for which we were provided a phone number from one of our commercial sources, and from which we had not received a completed questionnaire in the mail by a scheduled date. The telephone nonresponse follow-up operation is conducted over a three and a half week period, and data are collected using a Computer-Assisted Telephone Interviewing (CATI) instrument that builds in the required skip patterns and triggers interviewer checks on respondent answers. Because Fulton County lacks an addressing system, no telephone numbers could be acquired for sample addresses in this county and so no CATI data collection could be conducted.

Table 2 summarizes response to the first year of telephone nonresponse followup for each site and overall.

Table 2. Summary of CATI Response for the November 1995 through October 1996 Samples

1/ CATI Universe is the denominator for Late Mail Returns Removed and Ineligible Phone Number rates.

2/ Eligible for Interview is the denominator for Interview, Refusal, and Other Noninterview rates.

Over the course of the year our commercial vendors were able to provide telephone numbers for only about 50 percent of all the urban mail nonresponse addresses. Nearly one-fifth of these addresses returned a questionnaire in the mail after the CATI universe was defined, and these cases were removed from the telephone workload. Many of these "late" mail returns were probably prompted by the initial CATI telephone call.

Over one fourth of the phone numbers sent to CATI proved to be of no value to the data collection task. The numbers either did not connect the interviewer with the targeted address or were not working numbers. The removal of the late mail return and deficient phone number cases left only 55 percent of the original CATI workload eligible to be interviewed for the survey.

The interview rates for the telephone phase in the individual urban sites are shown in Figure 4, by the month in which the interview was conducted. Since the Census Bureau had very little experience with attempting a telephone follow-up on a mail nonresponse universe, we were unsure of what to expect from this collection phase, or how we would judge its success. Would we see seasonal response patterns, and would response rates differ noticeably by site? The monthly interview rates shown express the number of completed telephone interviews as a percent of the sum of cases for which the phone number reached the sample address and for which the phone number-address combination was verified. This measure is a combination of the telephone cooperation level of households with listed phone numbers that had chosen not to complete an ACS questionnaire and the probability that someone will answer the phone.

An area’s CATI interview rate appears to be less stable than in its mail return rate. The month-to-month response pattern differs considerably between the sites, but it tends to be highest in the winter and lowest in the spring and summer. Much of this observed response instability may not be attributable to the respondents themselves but to the changes made over the months to the operation. The ACS CATI operation has been fertile ground for experimentation and research, attempting to increase the number of interviews collected. For half the year, specific cases were included in the CATI workload with only a name associated with a sample address, to see if directory assistance could provide a phone number for the name-address combination. The effort gained little. The extra telephone number lookup effort did not produce interviews in sufficient number to be cost-effective, and the experiment was halted.

It is especially difficult to get phone numbers for units in multi-unit structures. To address this problem we tested the use of a business database to obtain telephone numbers for management or association offices associated with ACS addresses in multi-unit structures. It was felt that if we could identify and reach rental or management offices, they may be able to provide help in contacting the occupants of the nonresponding apartment units. Unfortunately, this also proved not to be cost-effective.

Having been unsuccessful in our experiments aimed at increasing CATI productivity by increasing the workload, we turned to researching ways to make this collection phase more efficient without negatively impacting the very commendable interview rates. We analyzed thousands and thousands of call records, paying particular attention to the number of call attempts being made and the case outcomes, and looking for the most productive days of the week and times of the day for calling. The results of this analysis was shared and discussed with the Jeffersonville Telephone Center, and led to the placing of a cap on the total number of ACS follow-up hours that could be charged each month. The Telephone Center refocused its efforts on interviewer staffing and telephoning during the times found to be the most productive, and has been very successful in completed the ACS data collection operation on schedule while maintaining its high interview rate and decreasing interviewing costs.

The Personal Visit Phase

The final data collection phase of the three-phase ACS is a personal visit by a field representative (FR). Since personal visits are the most expensive method of data collection, only a one-third subsample of the remaining nonresponse addresses is selected and sent to the field. Not surprisingly, most of the units successfully surveyed by the mail and telephone collection operations are occupied. The unresolved addresses remaining to be interviewed in the personal visit phase are disproportionately vacant housing units, and households for which we were unable to obtain telephone numbers. In this stage also are addresses that either do not exist or do not identify housing units. The rest of the field interviewing workload are households that tend to be difficult to interview, including a sample of those that refused not only the mail questionnaire but the CATI interview.

Table 3 summarizes the survey response to the personal visit interviewing during the first year in each site and overall.

Table 3. Summary of CAPI Response for the November 1995 through October 1996 Samples

1/ CAPI Subsample is the denominator for the Late Mail Returns Removed and Delete rates.

2/ Eligible for Interview is the denominator for the Interview, Refusal, and Other Noninterview rates.

Field interviewing is conducted using a Computer-Assisted Personal Interview (CAPI), and is conducted over a three or four week period. Mail returns continue to be accepted for a sample panel throughout the panel’s entire interviewing period, and they trickle in throughout the CAPI data collection phase. Over the course of the year about 3 percent of the personal visit subsample produce mail returns, many of them handed to field representatives by respondents when they are contacted to be interviewed. Deleted addresses--those that do not exist, or that do not identify housing units--account for another 8 percent of the workload. The remaining addresses require an interview.

Figure 5 shows the CAPI interview rates for each site by the month in which the interview was conducted. The rates are based on units eligible to be interviewed--those remaining after mail returns and deletes have been removed from the workload. Nearly 21 percent of these units were vacant. The ACS collects information on the characteristics of vacant as well as occupied units. Vacant interviews are conducted with knowledgeable respondents--owners, rental or real estate agents, or neighbors.

The CAPI interview rates are remarkably high considering the nature of the universe and the type of information being collected by the survey, i.e., the census long form sample data. We have watched these rates closely. The lower interview rates for the first month, besides being due in part to "first month jitters" (all ACS interviewers were new field interviewers), also resulted from a misunderstanding about the use of proxy respondents. Some interviews for occupied units were discovered to have been conducted with nonhousehold members such as neighbors. These were converted, after the fact, to noninterviews, and the FRs were retrained on this important aspect of the survey before ACS interviewing the next month.

Not only have the field interview rates been excellent, but the workloads within each site have not varied greatly from month to month, thanks to a steady combined mail and telephone response rate. This stable response from the first two data collection phases is an extremely important observation for a survey intended to collect information from independent national samples month after month, and designed to rely on field visits in every county every month. If the nature of response to the first two phases was such that the final field workloads could not be predicted within reason, the use of a uniform subsampling rate each month would have to be reconsidered. The number of field representatives required to conduct the ACS interviews should be the same each month for the survey to be successfully managed. As the survey expands into more and more sites with widely varying characteristics, the size of the personal visit workloads from month to month in each site will be closely watched by the ACS staff.

Summary of the First Year

When the results of all three data collection phases are combined the results are rather remarkable. Over the first year of testing, the ACS noninterview rate averaged about 2 percent in each of the three urban sites and 1 percent in the rural site. Each phase contributed important information on the survey’s design as well as the on types of households it tends to represent. Figure 6 illustrates how the collected data was distributed by phase.

The mail phase produced over 78 percent of the survey records. This is important for at least two reasons--collecting the survey information by mail is inexpensive when compared to the other two options, and the information is most accurate when it comes directly from the respondents themselves.

The CATI phase produced 10 percent of the survey data records--a larger percentage than some of us had originally hypothesized. As a result of the sustained success of the first two collection phases, CAPI was left with collecting less than 12 percent of the data records--those from the one-third subsample of the remaining nonresponse addresses.

Current Research and Future Work

There is still a lot to investigate and learn about designing and implementing the best tri-modal survey. We have only touched on some of the operational analysis of and research on the CATI phase that was initiated during the 1996 test. Experimentation has continued on other aspects of the design during the 1997 ACS test. For example, the schedule for mailing the prenotice, initial questionnaire package, the reminder card, and the replacement questionnaire package, if needed, has been changed. The time period over which the two questionnaires and the reminder are mailed has been decreased to from two weeks to about 10 days to see if this would increase their reinforcement of each other. In an attempt to reduce costs, we are currently testing the use of a lower class mailing rate, and will compare its response results against that realized with the first class rate to determine its possible impact. If there is no decrease in mail response nor increase in calls to Telephone Questionnaire Assistance reporting the misdelivery of mailing packages, the lower rate may be adopted, especially since it is quite likely that the length of the ACS questionnaire will increase next year, forecasting higher printing and postage costs.

We have completed a test of a new ACS computer-assisted instrument design against the instrument that is currently being used. The experimental instrument structures the ACS interview in a topic- or question-driven manner, asking for responses to a specific question or topic for everyone in the household for which the information is required before going on to the next question or topic. The current ACS instrument, once the basic population information for all household members has been obtained and the housing questions asked, requires that all the detailed population questions for the first person in the household be asked, and then repeats the same series of questions for each individual household member. Results of the controlled test of these two CATI instruments on a national sample should be available soon.

The 1997 ACS expanded into four more test sites, including Houston, Texas--the first major city site. The other three new sites are Douglas County, Nebraska (Omaha), Franklin County, Ohio (Columbus), and Otero County, New Mexico. In 1998, ACS added Broward County, Florida (Fort Lauderdale) and Richland and Kershaw Counties in South Carolina. These South Carolina counties are also a part of the Census 2000 Dress Rehearsal. By conducting the ACS in these counties we hope to be able to judge the possible impact of the ACS on Census 2000, as well as the census’s impact on the ACS. In 1999, the ACS plans to expand into 40 more sites around the country. The ACS data collected from 1999 through 2001 will be compared with the corresponding census long form information for tracts and block groups. If all goes well, and funding permits, the ACS will become a nationwide survey starting in 2003, and continuing indefinitely.