Sheila B Robinson

Reflections of an everyday educator/program evaluator/professional developer…LEARNER

What Is Different about K-12 Public Education Evaluations? Guest post by Rakesh Mohan and Lance McCleve

4 Comments

I’m pleased to publish this guest post by my evaluator colleagues in the Idaho State Legislature’s Office of Performance Evaluations. Working in a governmental context gives them a unique perspective on education evaluation, as their work encompasses numerous fields and they experience evaluation in many different contexts. I’ve also long been a fan of their evaluation reports which exemplify a modern approach to reporting and dissemination: the report formats are visually appealing, and therefore easy to navigate, read, and comprehend, and they use well-designed visualizations to communicate key data points. 

Rakesh Mohan, Idaho Legislature Office of Performance Evaluation

Rakesh Mohan

Rakesh Mohan is currently a candidate for the president of the American Evaluation Association. Since 2002, he has been the director of the Office of Performance Evaluations, an independent agency of the Idaho State Legislature. Under Rakesh’s leadership, the office received the 2011 Alva and Gunnar Myrdal Government Evaluation Award.

Lance McCleve, Idaho Legislature Office of Performance Evaluation

Lance McCLeve

Lance McCleve is a principal evaluator with the Office of Performance Evaluations. He was the lead evaluator on the three evaluation examples mentioned in this post.

 

At least in the USA, K-12 public education evaluations seem to be curried with a bit of extra politics, so much so that the politics is too spicy for many evaluators to handle.[i] Like a curry, the politics of education evaluations is regional/local, peopPicture of curryle centered, and contains one or two secret political ingredients that evaluators are last to know, if ever.

All evaluations are inherently political, but evaluators and the evaluation process are expected to be non-political.[ii] Usually the politics of evaluation is seen at two stages: (1) when the decision is made to ask for an evaluation and (2) how the results of an evaluation are used or not used.

Here we use three examples of recently conducted education evaluations that came with especially spicy political curry:

Teachers Front CoverWorkforce Issues Affecting Public School Teachers

ISEE Front CoverThe K-12 Longitudinal Data System (ISEE)

Schoolnet Front CoverIdaho’s Instructional Management System (Schoolnet) Offers Lessons for Future IT Projects

 

These three reports give the reader a clear sense of the politics that permeated the environment in which these three evaluations were assigned, conducted, and reported. For additional details reflecting the politics, see the related media stories:

Idaho teacher workforce OPE report: ‘Strong undercurrent of despair’

Goedde: Pay cuts beat teacher layoffs, focus on successes to build teacher morale

Lawmakers receive stinging report on student data system

After the $61 million Schoolnet fiasco, a scramble begins

The purpose of using these stories in this post is to illustrate the political nature of education evaluations. We do not intend to take a position on any of the political issues discussed in these stories.

Reasons for the Extra Politics

We think at least five factors contribute to the extra politics in education evaluations:

  1. Nearly everyone went to some school and has assumptions, beliefs, or reactions based on individual experiences. Unlike nuclear physics, nanotechnology, and neurosurgery, people from all walks of life act as though they are experts in education.
  2. Education policies affect most people either because they have children in school or they are taxpayers, or both.
  3. Providing adequate and equitable educational opportunities to all children is the responsibility of each individual state. Generally, one of the biggest parts of a state’s annual budget is for public education. For example, Idaho spends 48 percent of its general fund on public education.
  4. Among various stakeholders, teachers’ unions play a significant role in influencing education policies.
  5. Education policies have consequences for the future of our children and the nation.

So what should we evaluators do about the extra politics?

Suggestions for Managing the Extra Politics

The following four suggestions will apply to any type of evaluation, but they should be treated like edicts when it comes to conducting education evaluations.

Listen to all stakeholders with respect, regardless of how much you disagree with their views. This will help you fully understand the context of the evaluation, that is, who the key stakeholders are and what their competing, conflicting, and complementing interests are. Furthermore, this is the first step in establishing trust with all stakeholders. If you start on the right foot with stakeholders, your life as an evaluator will be pleasant and productive. The absence of trust with stakeholders could at best make the evaluation report useless from the get go and at worst, make conducting the evaluation a hellish experience. The methodology sections of our evaluation reports list the large number of stakeholders we needed to work with.

Don’t take sides, or you will get swallowed up in the politics. In all three evaluations, stakeholders were strongly polarized in their views about the condition of the programs we were evaluating and on who was to blame. We were able to avoid drawing conclusions based on stakeholders’ opinions, as persuasive or forceful as they might have been, by sticking to defensible methods. However, the bigger challenge was carefully conducting the evaluation and developing findings in a way that avoided the appearance of us having taken sides.

Stand up and own your evaluation approach and findings or you will get trampled. We were happy to see our report about workforce issues affecting teachers make headlines, but it came at a cost. Because of the intensely political nature of K-12 education issues, we immediately saw political backlash. Those who were not happy with our evaluation headed straight for our evaluation approach and methods to start criticizing. We easily could have stepped aside and let people do as they will with our evaluation but instead we took a stand and defended our approach and methods.

Idaho Statesman article: Survey Shows Teachers' Worries

The disagreement between our office and those who were unhappy with the evaluation was significant enough to become the subject of several additional news stories. Although the disagreement was uncomfortable to say the least, it only served to strengthen our credibility. 

Idaho Statesman article: Legislature's Auditor Defends Teacher Report

Go out of your way to show respect for those who you’ve had disagreements with. You never know when they might champion your evaluation message in the future. Over the years, every policymaker mentioned in the news stories has agreed more than disagreed with our findings and recommendations. Remember—those who you had disagreements with are also doing their job, likely with the same passion as you do, and addressing the needs and demands of their constituents and stakeholders. Most of the time, disagreements are not personal – and even if they may seem personal, you should treat the people involved with total respect. This goes a long way toward building relationships and establishing credibility as independent evaluators who do not have a dog in the fight.

Final Thoughts

Acting on political knowledge is complex and potentially dangerous to evaluators’ credibility. However, by acknowledging the fact that all evaluations are inherently political and adhering to the four suggestions mentioned above, you will be able to manage the extra politics that comes with almost every education evaluation.

 

 

[i] K-12 public education evaluations are henceforth referred to as education evaluations.

[ii] Evaluator Advocacy: It Is All in a Day’s Work. American Journal of Evaluation, September 2014, 397-403.

Advertisements

Author: Sheila B Robinson, Ed. D

Voracious learner, career educator... Evaluation, Professional Development, Special Education.

4 thoughts on “What Is Different about K-12 Public Education Evaluations? Guest post by Rakesh Mohan and Lance McCleve

  1. Great stuff.

    I’d suggest a sixth driver of politics in education evaluations, which I increasingly think is at the real center of this dynamic – a fundamental disagreement about the nature of public education; whether it serves a sorting purpose or a human capital building purpose. Call it “competition” versus “cooperation” if it’s useful to focus on observable processes and cultural attributes rather than the bigger concept.

    Some people (in whatever stakeholder group) believe that, broadly speaking, education is about sorting kids with more talent (or intelligence, desire, etc.) from kids with less, by giving all of them the same opportunity to succeed. Others believe that education should be aim to improve all kids’ situations, regardless of where they start and where they might end up.

    These two groups are going to have completely different definitions of success for any education endeavor, will value different kinds of programs differently, and will have corresponding opinions about funding and evaluating them.

    The hints above apply here but I find that keeping this dichotomy in mind helps me attend better to context (e.g., knowing how “sides” might be defined as I’m trying not to take one).

    Kirk Knestis PhD
    Hezel Associates
    Syracuse, NY

    • Thank you, Kirk. You raise an excellent point. I hope readers see your comment. Having lived in this country for 39 years, I totally understand the importance of this dichotomy. I wish this dichotomy existed in India when I went to school and college there in the sixties and the seventies. Unfortunately, the attitudes of the public, policymakers, and educators in India were heavily tilted toward the “competition” model.

      We certainly have seen this dichotomy (although not explicitly) play out as we conduct our work in Idaho.

  2. Pingback: Evaluating the job of K-12 evaluations - Idaho Ed News

  3. Pingback: Evaluating the job of K-12 evaluations

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s