While at RAND, I developed familiarity with and expertise using survey panels. Much of the previously-mentioned work invokes data collected using a probability-based panel (Robbins et al., 2021), and I have recently authored a manuscript that illustrates the pitfalls of using volunteer opt-in survey panels in lieu of probability-based data. Furthermore, I served as co-PI for the projects that were involved in the founding of RAND’s American Educator Panels; this work was funded by multi-million dollar grants from the Bill and Melinda Gates Foundation. My contributions involved the sample design for the successful recruitment of 30,000 teachers and 13,000 school leaders into the panels, as well as the design of the processes for weighting the panels. Articles were published on the methods used to recruit educators (Robbins et al., 2018; Robbins & Hawes-Dawson, 2020), and a technical document was published which outlines the work (Robbins & Grant, 2020).
Collaborators:
References
Journal Articles
-
Blending of probability and convenience samples as applied to a survey of military caregivers
M. W. Robbins, B. Ghosh-Dastidar, and R. Ramchand
Journal of Survey Statistics and Methodology, 2021
Probability samples are the preferred method for providing inferences that are generalizable to a larger population. However, in many cases, this approach is unlikely to yield a sample size large enough to produce precise inferences. Our goal here is to improve the efficiency of inferences from a probability sample by combining (or blending) it with a nonprobability sample, which is (by itself) potentially fraught with selection biases that would compromise the generalizability of results. We develop novel methods of statistical weighting that may be used for this purpose. Specifically, we make a distinction between weights that can be used to make the two samples representative of the population individually (disjoint blending) and those that make only the combined sample representative (simultaneous blending). Our focus is on weights constructed using propensity scores, but consideration is also given to calibration weighting. We include simulation studies that, among other illustrations, show the gain in precision provided by the convenience sample is lower in circumstances where the outcome is strongly related to the auxiliary variables used to align the samples. Motivating the exposition is a survey of military caregivers; our interest is focused on unpaid caregivers of wounded, ill, or injured US servicemembers and veterans who served following September 11, 2001. Our work serves not only to illustrate the proper execution of blending but also to caution the reader with respect to its dangers, as invoking a nonprobability sample may not yield substantial improvements in precision when assumptions are valid and may induce biases in the event that they are not.
-
A comparison of strategies for recruiting teachers into survey panels
M. W. Robbins, G. Grimm, B. Stecher, and V. D. Opfer
SAGE Open, 2018
We examine a range of options for recruiting teachers into a nationally representative survey panel. Recruitment strategies considered include a telephone-based approach and the use of promised incentives and pre-incentives of varying amounts and forms. Using a randomized experiment, we evaluate the effectiveness of five separate recruitment strategies and conduct a cost-benefit analysis. Our preferred strategy is one that uses a US$10 gift card as pre-incentive (it yielded a 27% rate of successful recruitment at a cost of US$78 per recruited teacher). Statistical comparisons indicate that no other technique was superior to this strategy in terms of recruitment rate or cost-effectiveness. Efforts at refusal conversion after the initial approach were mostly ineffective. A comparison across demographic type characteristics of enrolled panelists and nonrespondents shows no substantial differences for any recruitment strategy considered. Hence, the potential for recruitment-level nonresponse to induce large bias into findings from surveys administered to the panel is minimal.
-
The effect of incentives and mode of contact on the successful recruitment of teachers into survey panels
M. W. Robbins and J. Hawes-Dawson
Survey Practice, 2020
This article discusses an experiment run at the beginning of an effort to substantially enlarge the RAND American Teacher Panel, a nationally representative sample of K-12 public school teachers. Ten strategies were evaluated. We considered different modes of contact (FedEx vs. US Postal Service [USPS]), contingency status of incentives (pre- vs. promised incentive), amounts of incentive ($2, $10, $40, or $60), and types of incentive (cash, check, gift card, electronic). Strategies were compared in terms of response rate and cost effectiveness. Our study yielded several findings that should advance the literature: The use of FedEx clearly outperforms USPS, and the use of a moderate preincentive ($10) outperforms much larger promised incentives (up to $60) with respect to both responsiveness and cost-effectiveness. In addition, cash and check seem preferable to gift cards. Finally, we assess the potential for nonresponse bias by comparing enrollees to nonresponders across a variety of demographic characteristics for each strategy.
Technical Reports
-
RAND American Educator Panels (AEP) Technical Description
M. Robbins and D. Grant
Mar 2020
This report describes the methodology behind the RAND American Educator Panels, which consist of the American School Leader Panel and the American Teacher Panel. It provides detailed information on the panels, the multiple phases and waves of recruitment used to create the panels, weighting, and variance estimation.