Meegan Scott:  The task at hand in post-implementation is to make a judgement about the strength of the organization at a milestone review period (when we’re asked to do evaluation).

It could be a Mid-term Review of a Plan or at the end of a Plan Period.

We ask if the organization is stronger at the end of that milestone period or the planned period than when the plan was created, and at the start of execution.  Was the strategy executed successfully?  This is an attempt to assess the effectiveness of a plan in guiding the organization towards achieving improved performance.  We look at that in terms of effectiveness, efficiency, relevance, financial viability, cost effectiveness, and for some type entities, we would go deeper into looking at quality aspects. 

For that type of plan, let’s say for a manufacturer of clothing, we may use the Hoshin Planning Model to add those related lines of questions to the evaluation.  In general, we also look at how the plan helps the organization with adjusting to changes in the environment.  These include political factors, social factors, competitor inflation, interest rates, legislation and even ecological factors.  Sometimes, we’ll find that entities do not know all the governing legislation affecting them.  So, we normally place a table up at the front of a plan that includes governing legislations.

We also look at the plan logic and the premises and predictions.  How did those work according to plan?  Were they accurate, to what extent, and what needs to be adjusted?  We look at whether or not the plan helped to improve motivation in a desired culture and advance the mission of the organization.  We also look at its impact on the organization, on its history.

Did it help to create new history?  Did the plan carry out what was meant to be done?  Did it help to strengthen and improve financial management partnerships, program management, leadership at different levels, and HR capacity to support both the present and the desired future?  Analyzing HR capacity is in part to help management retain tacit knowledge in the organization rather than just staying put and waiting to hire staff.

Typically, the evaluation will take the form of a self-assessment.  Even though it may be a request for an evaluation by a donor, we try to make it into a self-assessment so that the client can benefit from owning and growing that culture of performance and measurement and improvement.

The client also benefits from receiving information for decision-making related to their strategic choices for strategy updates and reformulation for the next plan and milestone period. That adds much more benefit than if you did the evaluation with merely accountability in mind.  We therefore approach the evaluation with a view to gathering performance information, meeting accountability requirements, and to guide resource allocation.  Another important thing we look at is the infrastructure for delivering strategy.

If the organization is implementing multi programs as would be in the case of a government department, and many NGOs, the approach would be heavily influenced by the terms of reference.  That comes with a call for proposal versus if it was just the organization that came up with the idea and asked for a proposal.  In the latter case we are left with greater leverage in designing what it is that we will be doing.

Our approach includes a blend of evaluation approaches.  This depends on the competence of the organization in collecting and using performance information and the information needs outlined or that we glean from the call for the evaluation and the intended users.  That blend would involve components of the utilization focus evaluation approach.

It would include consultations for ensuring that the information collected will be of benefit and is what is desired by the organization and its stakeholders.  We may even use a theory-based evaluation approach for assessing the logic for addressing a particular problem, the effectiveness, and the context.

We would want to look at the theory of change, how it’s holding up against what was expected, the participants and their attitude, and how their participation impacts the outcomes for them.  We could also use a more all-inclusive strategic evaluation approach.  That would be a strategic evaluation into the outcomes and the impact of the target population.

Irrespective of the evaluation approach or blend thereof, we would consider planned results against actual results and unintended results.

So, going back to the strategic evaluation— we’d look at the results and service levels as well whatever they are creating, selling or giving away.  For outcomes, our examination would be in terms of their relevance and effectiveness.  For outputs, the focus would be the products or services and how efficient the organization was in producing them (the outputs). Other output related questions to answer would be— along the lines of how cost-effective it was to deliver the solutions, and the quality of those outputs that were delivered.

You also want to analyze the internal management and leadership as it relates to output processes that are involved and for developing them.

So you perform an individual level assessment and review of measures.  This can be an area of challenge or resistance.  The moment you begin to ask for job descriptions and such, expect a break or stop in information flow.  At the plan level, we review measures indicators, strategy identity, et cetera.

A key component of the exercise is the management response session that we’ll lead for discussing the findings, the recommendations, and the judgment.  From that, you’ll get feedback into how management feel about the judgment and the findings.  This may result in some insight about the context and maybe some adjustment.  You will also draw out of that process actions for improvement and try to get some calendar and resource commitments towards that.

A review of external literature and internal organizational documents is part of the process.  External literature includes literature of the external environment.  When looking at internal literature, we examine their reporting and administrative documents, the operations plan, the corporate strategy plan, performance reports, and minutes from board meetings.  Other methods or lines of evidence for data analysis could include conducting surveys, interviews, and consultations.  If some instances we must calculate, develop estimates, or undertake social media searches. At times we even have to look at or conduct lab research or participatory.

The output would be typically an organizational assessment and development report.  This would include the proposed strategic options and choices for informing the development of a new strategy plan, strategy update, or a plan for the next plan period.

Let’s turn to the kinds of questions that we would ask. Different evaluations may have their own unique questions.  But in general, we’d ask questions such as, what were the goals and objectives in the plan?  How did the organization perform based on the strategic intent stated in the plan and its related goals?  We’d also ask how effective did the organization use the plan to manage the delivery of its results?  That is, the priority, the focus areas, the approaches, and accountability.  Another question is: how easily or difficult did the plan make the performance management and measurement process?  Because if it is just a summary of a plan and not fully elaborated and support by measurable indicators, there’s going to be trouble at the execution stage.

An important question to ask is: does the plan include an alignment mechanism for cascading and aligning?  We also look at whether or not the major initiatives and commitments were delivered on time and in budget.  If there were deviations, how wide was the spread and what needs to be changed.  Therefore, if they finished before schedule, were late or on time, we want to understand the reasons.  We also look at the overall workings of the plan logic based on the theory or theories of change or the strategy maps and strategy framework or income output map, or any combination of them.  We also look at whether or not the scope of operations is made clear by the plan.  How suitable was the initiatives or initiatives for building capacity and advancing the strategic direction articulated in the plan— is another area which we examine.

So, to arrive at a judgment we’ll use multiple sources of qualitative and quantitative evidence.  We look at the use of the plan and the process and the annual and periodic review and strategy update.  Are they following that guide and are they updating the plan?  We ask if a priority trade off happened and, if so, why?  We also look at the effectiveness of the plan in communicating to the board, to management, to partners, to funders, and staff.  Do they understand the plan or do they find it to be a burdensome document?  Does it address the value chain and how they’re going to make product relationships and leverage them in terms of the products or services or the supply chain? 

Moreover, we analyze how effective the plan is in articulating the strategic identity.  So that’s how we do that post-implementation evaluation.

See also:

Interview Transcript on IAF

Strategy Execution Challenges

Pre-implementation evaluation of the corporate strategy plan

Monitoring of the corporate strategy plan