By Saville Kushner
Sage Publications, London. 2000
ISBN 0-7619-6362-6
By Saville Kushner
Sage Publications, London. 2000
ISBN 0-7619-6362-6
Saville Kushner's new book, Personalizing Evaluation (Sage, 2000), is about how one behaves as an evaluator and as a researcher in the real world of naturalistic inquiry. I recommend it for students of program evaluation and for researchers looking for ways to understand activities taking place in complex social contexts. It is about the evaluation process. It is about evaluation as an agent of change, it is about the levels of accountability in the evaluators role, recounted through Kushners own experiences in Great Britain. There is a great deal in this slim volume. I can only touch on some of the thought-provoking strands that emerged for me.The book contains a major case study in a complex social context. He tells of his work with students in a music conservatory program. His retelling is an excellent way for anyone new to the field of evaluation to understand how intricate negotiations about information and access to information can be. This innovation brought the students into contact with the community, and one group was studied interacting with a hospice. His descriptions of interviews and excerpts show his technique for dealing with participants as he constructs a version of the activity. At the same time he reveals the delicate exchanges with the people "at the margins of the program." (Those workers at the hospice who have an interest in the information shared by music students, and how it is used.)
This is a book about democracy and social justice. The evaluators role is to act as an advocate for social reform, giving voice to the programs participants. The role may cause her to filter her interactions with administration, to keep confidentiality, for example. For, in Kushners view, evaluation is really assessment of the effectiveness of policy. Programs, human service, educational, and so forth, are policy operationalized. Participants are the objects of the policy at the societal level. Evaluators stand at the cusp of contending interests and will have to make decisions based on their social conscience. Kushner searches for " approaches that seek to generate and publicize as many program perspectives as possible " (p.4).
In the chapter entitled "Ethics, Mortality and the Conduct of Evaluation," he lays out three areas in which questions arise in the work of evaluation. Ethics of role (the legitimacy and status of the evaluator); ethics of evaluation agreements (here he talks about consensus as well as contract); and ethics of conduct (including publication). "Keep talking," he advises as the first of five very thoughtful and useful procedural principles for doing evaluation. (p.180)
Kushner, a well-regarded and very experienced theorist and practitioner of program evaluation, is Chair of Applied Research at the University of the West of England. Prior to that he worked at the Centre of Applied Research in Education (CARE) at the University of Norwich. He draws upon this experience to illuminate the nature of program evaluation. He includes a valuable treatment of the history of program evaluation in Britain and in the United States.
Personalizing Evaluation offers a perspective on what is known as Democratic, or participatory, evaluation, where the process of the evaluation and the results are collaborative and shared before publication. Kushner advocates conducting program evaluation person-to-person with program participants. This is the opposite of top-down approaches that usually begin by describing the program that won that grant, not the program that is in place. Personalizing evaluation involves describing a program that exists through the experiences of the clientele. Kushner demonstrates his technique for eliciting personal narratives from the music students about their experiences.
Kushner writes with insight about the political nature of working with publicly funded programs in the private sector. He makes clear that, beyond issues of objectivity and subjectivity, personal expression is very much a part of program evaluation. These are programs that have been "Thatcherized," that is, left to the free enterprise system, and now monitored for a variety of criteria established by a government budget office. Here, he suggests, the advocate may take sides, shielding the service provider from the irrelevant measurement by official inspectors. He shares examples of his own work that show the complexity, or "messiness," of program evaluation in ways that a student of the field or a novice might understand the true nature of naturalistic enquiry. And finally, he makes the point that programs of social change, messy and political as they are, can best be measured against their effect on individuals.
Kushner shares a vision of evaluation that has as a central purpose the strengthening democratic processes. Program participants see the role of the evaluator as an independent and non-judgmental observer. That is, the very way in which one engages in and reports evaluation can support democratic ideals and help build sustainable democratic institutions, quite apart from the content of the findings.
In my own work, I move between the roles of program evaluator and research. Kushner helps to clarify the distinctions between these activities. What distinguishes evaluation from research, he writes, is a reduced freedom for independent scholarly inquiry and greater political and ethical consequences. Kushner cites MacDonald (1987), who believes that the researcher has freedom to construct an agenda to match the "problem solving techniques of his craft," while the evaluator has those techniques as, "a repertoire to respond to problems conjointly discovered with program participants" (p. 37).
What distinguishes personalized evaluation from ethnography is an interest in outcomes, a policy consideration that is expressed in the evaluation agreement. Kushner makes the point that evaluation itself must be subject to evaluation. He recounts the story of Summerhill School and its difficulties with ideologically opposed government inspectors. The inspectors were, in turn, evaluated and found wanting and their denial of support for Summerhill was reversed. A happy outcome. Evaluation as detective work.
This book made me think about the variety of relationships that are involved between the evaluator and the program participants, the program staff and administration, and those peripherally associated with the program who will be part of the report. For example, in his case study, what the students said about their experience at the hospice was authentic and frank and not what hospice workers usually want the public to hear about working with the dying. The difficulty is to maintain accountability to both the client with whom one has a contract for an evaluation and, at the same time, ethically manage to be allowed access to the program site.
Kushner writes clearly from long experience, and I think one would come away from the book with a sense for the messiness of fieldwork and the pressures that the work carries. The desire to find concrete data to show program effects is a chimera and one looks for ways to take a "big picture" look at an activity by letting the participants tell what it is that they are doing. Thereby amassing a great deal of material through which one has to sort in order to find salient strands of concerns or successes among the experiences, searching for and understanding the metaphors that respondents use in talking about those experiences.
Table Of Contents:
Overture
1. Program Evaluation: The Heart of the Matter
2. Personalizing Evaluative Enquiry
3. Love and Death and Program Evaluation
4. Evaluation and a Philosophy of Individualism
5. Knowing Me, Knowing You: Evaluation Interviewing
6. To Have and Have Not: Critical Distance and Emotional Proximity
7. Essences, Contexts and Transition: The Individual at the Margins of the Program
8. Ethics, Mortality and the Conduct of Evaluation
9. People in Change
Coda
10. Robert Campbell and Cultural Standards in Curriculum Evaluation
MacDonald, B. & Kushner, S. (eds.) (1987) Bread and Dreams: a case study of Bilingual Schooling in the U.S.A., CARE Occasional Publications No.12, Norwich UK:CARE, University of East Anglia.
J. David Betts
