Lessons Learned from Evaluating a Case Management Program


Jessica Broome

A client of mine is a social service agency. One of their smallest programs (serving about 40 clients at any given time) offers case management for individuals involved with the criminal justice system. The typical program participant has recently been released from a long prison sentence, is drug-addicted, and suffers from a chronic health condition (most often HIV, but also hepatitis, diabetes, and cancer). The audience is hard to engage, service staff are chronically overworked, and, largely because the number of participants is so small, the quarterly evaluation has sometimes not shown much progress. After almost four years of evaluating this program, I’ve learned a few lessons that I hope can help other evaluators assess their own approaches:

  1. Use clear and relevant objectives to track progress: This may seem obvious, but it has been a bumpy road! The objectives put forth by the program’s funder were broad and, initially, impossible to track. (“Reduce the spread of HIV in served communities” was one request that we simply could not collect data to measure.) Instead of trying to force our program into this rubric, we came up with a few relevant sub- objectives that we COULD track, like education of community members and decreased viral loads among program participants.
  1. Involve service providers in selecting evaluation techniques: The counselors and case managers who are on the “front lines” of client service have been my best resource when it comes to designing data collection mechanisms and deciding which outcomes should be tracked. For example, a counselor pointed out that program participants often have limited cognitive abilities and are just not able to reliably answer questions rating their health. Further, requiring this was posing a large burden on already-overworked staff, who had to collect the data from participants at regular intervals. Instead of a self-assessment of health, we started collecting lab results directly from medical providers, which provided a more objective measure, and reduced the burden on both participants and line staff.
  2. Keep evolving data needs and sources: For several years, I conducted focus groups and client satisfaction surveys with current clients, who consistently gave the program rave reviews. Ultimately, I realized that we weren’t learning anything new from these evaluative approaches; participants who responded to the survey and attended the focus groups were those who came to the program every day. Those who were unhappy with the services they received had stopped coming, or came only rarely. This year, we’re implementing a “track down” program, where all the resource that were used for the focus groups are diverted to locate and interview individuals who have stopped attending the program.

It’s important to point out that these lessons can be applied to almost any program evaluation! This program works with a very specific target audience, but I will take these lessons with me when I work on other evaluations.

Author: Jessica Broome
Jessica’s question to you
What other lessons have you learned from program evaluations?

Please click the link above to view responses or comment.


IYE Operationalize and Execute Strategy Storython



Click the table to enlarge




What and How?   | Privacy Statement  | Join us

 tellyourstoryhere   askquestions storytelling whatsyourstory

videosign  lessonslearned keeneye

Yourstorycouldhelp   shareyourtoons


Question and Answers

Please click the link (s) to view or respond to questions.

How to operationalize an Effective Culture of Evaluation at the National and Organizational Levels?

Strategy translation and elaboration questions and answers for non-profits and community based organizations

How can home based consultants and small businesses execute their planned strategies in a timely manner?

Critical factors, behaviours and processes that worked – operationalizing a culture of evaluation in organizations.

EVAL Year Logo Evaluation_Tourch_2015_r1 softenedmagatelogogreenstool.v

All rights reserved ─ Copyright © 2015 by Magate Wildhorse ℠

IYE Evaluation Methods & Design Blogathon

Prospect%20Plantation%20Or_at_Prospectp103 Blogathondate comeshareblogth  Click on the table to engage





Jessica Broome

Lessons Learned from
Evaluating a Case Management Program

 What and How?   | Privacy Statement  | Join us





Please click the link (s) below to view questions, post answers or comments.

Best Lines of Evidence for Evaluating Strategic Plans

Impact of Evaluation Belief vs Actual Practice on Choice of Design

General comments and feedback

EVAL Year LogoEvaluation_Tourch_2015_r1softenedmagatelogogreenstool.v

All rights reserved ─ Copyright © 2015 by Magate Wildhorse ℠

Research Methodology or Method━ How to Tell the Difference

A keen ear for topics related to research may often hear examples of a research method being described as a research methodology; or of a research methodology described as a research method.

In this post we hope to help the mind to believe what the eyes see and what the ears hear by presenting what I will call classic or strict definitions and examples for accurately distinguishing between the two terms.

Thanks to Google I will not have to re-invent the wheel; instead I will refer to the article “Research dilemmas: Paradigms, methods and methodology”, by Noella Mackenzie and Sally Knipe of Charles Sturt University.   In the article Mackenzie and Knipe cited the definition of research methods offered by McMillan & Schumacher in Research in Education, it reads as follows ━   Research methods – how data are collected and analysed – and the types of generalizations and representations derived from the data (McMillan & Schumacher, 2006, p. 12)”.

By explaining that method consists of “systematic modes, procedures or tools used for collection and analysis of data”, the authors make it easier for readers to understand what may be described as research method.  Among the variety of data collection tools which make up methods are: survey questionnaires, interviews, focus groups, and photographs.

Methods Magate Wildhorse

Magate Wildhorse

Framework for researchmethodology

James R. Martin, Ph.D., CMA Professor Emeritus, University of Sourth Florida (Management and Accounting Web) http://maaw.info/ArticleSummaries/FrameworkForResearchMethodology.gif

With our memories refreshed as it relates to methods, let us now turn our attention to the term methodology which covers much more than methods do. Mackenzie and Knipe describe research methodology as “the overall approach to research linked to the paradigm or theoretical framework”.  In other words methodology explains how the researcher will solve the problem that is being addressed by the research and include among other components, the methods, frameworks, and indicators of success that will be applied to the study.

Now that we have focused our eyes on what is covered by the term research method, namely: tools, modes and procedures for data collection and analysis; versus methodology which addresses theoretical frameworks, methods and other components described above we hope it will be easier for the mind to distinguish between often heard misuse of the terms and their more accurate, accepted, and formal meaning.

You might find it useful to visit the article and to take a quick review of the two tables listed below, which were used by the authors to illustrate the differences in meaning between research method and research methodology.

  • Table 1: Paradigms: Language commonly associated with major research paradigms
  •  Table 2: Paradigms, methods and tools  

“Research dilemmas: Paradigms, methods and methodology”, (Issues in Educational Research, Volume, 16, 2006), is available [Online] at: http://www.iier.org.au/iier16/mackenzie.html.

Scott, M. E., Copyright © 2015 Magate Wildhorse. All rights reserved.

Thank you for visiting Magate Wildhorse and this post, we look forward to your comments and future visits.