Managers of workforce training programs are often unable to afford costly evaluations to determine the effectiveness of their programs and many rely on the survey responses of participants. How successful this approach is and whether it can be improved are addressed in a new book from the Upjohn Press.
In "Are Participants Good Evaluators?" Jeffrey Smith, Alexander Whalley, and Nathaniel Wilcox are the first to attempt to assess such measures, despite their already widespread use in program evaluations. They develop a multidisciplinary framework and apply it to three case studies: the National Job Training Partnership Act Study, the U.S. National Supported Work Demonstration, and the Connecticut Jobs First Program.
These studies were subjected to experimental evaluations that included a survey-based participant evaluation measure. The authors apply econometric methods specifically developed to obtain estimates of program effects among individuals in the studies and then compare these estimates with survey-based participant evaluation measures to obtain an assessment of the surveys’ efficacy.
The authors dedicate a single chapter to each study and include the following:
- a discussion of the program or policy and the population it served;
- a description of the design and implementation of the experimental evaluation of each program or policy, with emphasis on the participant evaluation measure;
- an examination of the correlation between the participant evaluation measure and the experimental and econometric estimates of program impacts; and
- an examination of the relationship between the participant evaluations and other factors, including respondent and program characteristics and other proxies for program impact.
Smith, Whalley, and Wilcox point to six contributions that arise from their groundbreaking work. These include 1) developing a theoretical framework for using data from evaluations that include participant evaluation responses, 2) devising an econometric framework for studying data from evaluations that contain participant evaluation responses, 3) examining the correlation between participant evaluations and estimates of program impacts, 4) investigating the determinants of positive participant evaluations, 5) documenting the range of participant evaluation measures available in the literature that addresses active labor market programs, and 6) proposing alternative participant-evaluation question formats aimed at better capturing causal impacts of program findings.
The authors also discuss how their findings fit into the broader literatures in economics, psychology, and survey research.
"Are Participants Good Evaluators?" by Jeffrey Smith, Alexander Whalley, and Nathaniel Wilcox, is published by the W.E. Upjohn Institute for Employment Research. Click “learn more” to order a hardcover or paperback copy of the book, or download the first chapter.