Evaluspheric Perceptions

Reflections of an everyday evaluator/learner/educator exploring the evalusphere


4 Comments

Presentation Principles: Right-sizing the Room to Maximize Audience Engagement

Presentation Principle:

The right setting can impact how your audience experiences a presentation.

It’s the Goldilocks conundrum: You’re scheduled for a conference presentation (i.e. a non-ticketed event were people do not pre-register) and you get to the room, only to discover that it is either too small or too big for your anticipated audience size… definitely not just right!

I recently presented with a colleague at a multi-day conference with hundreds of concurrent sessions where they assigned our session to a room that seats about 100 people. Our topic was a popular one, and the time slot was a preferred one for that conference (i.e. not during a time that people would naturally take a break and choose not attend a session). Needless to say, the room filled up and people just kept coming in long after all chairs were taken.

At another conference, one of my presentations was assigned to a gigantic ballroom that seated over 1000 people! My co-presenters and I expected that our session, given the topic and time slot, would more likely draw an audience of perhaps 20- 50 people.

What do you do when the room isn’t the right size for your presentation?

First course of action: If time permits, ask to be reassigned to a different size room. Unfortunately, this is often impossible for the conference organizers to do, but hey, it never hurts to ask, right? If that doesn’t work, try some of these strategies:

If the room is too small:

1.) When the room starts looking full enough that people are scanning for a seat before they even get too far in, ask those seated to raise their hands to signal an empty seat next to them.

2.) Often the hosts will put one or two chairs by your presenter table. If you won’t be using these, offer them to audience members. If your conference organizers or hosts are nearby of course, you can ask for extra chairs, but often, this option won’t be available.

3.) When all seats are taken, invite people to sit on the floor in the front of the room. Whaaaaaat? Adults sitting on the floor? Yes! You may be surprised, but many people will choose this as an alternative to being locked out an enticing presentation.

4.) If the inevitable happens, and people are locked out, here’s a way to acknowledge their attempt to make it in, and give them a little something for their trouble. Put a sign on the outside door with something to the effect of, “If this room is full and you can’t get in, here’s the URL where you can download presentation materials.”*

If the room is too big:

1.) If you get to the room early enough, tape off or otherwise block off the seats near the back or on one side of the room. This is one of the many reasons you may want to carry a roll of painter’s tape (which generally won’t damage surfaces) with you to presentations.

2.) Greet people as they enter and ask them to choose seats in a particular section or near the front.

3.) Quickly create a PowerPoint slide that asks people to sit near the front and have this on the screen prior to starting.

4.) If people do end up scattering and spread out in a large room, be sure to walk around during your presentation using proximity to engage them. You can also offer them the opportunity to move closer during a break.

Any other ideas to share on right-sizing your presentation space? Please add them in the comments!

*Thanks to Chris Lysy for this strategy he used at a recent conference.


Leave a comment

Presentation Principles: Foundations of Effective Presentations

There are multiple pathways to great presentations.

Some presenters are fortunate enough to be naturally possessed of  stage presence, charisma, or a je ne sais quois that keeps their audience hanging on their every word. After all, Martin Luther King, Jr. didn’t use PowerPoint slides.* Steve Jobs never asked his audience to turn and talk to an elbow partner. The rest of us, though, are just not there. We’re presenting to the board, our colleagues, university students, fellow conference-goers, or some other audience who may or may not know us or be familiar with our work. “Star quality” is not something we can rely on for a successful presentation.

What is an everyday presenter to do?

Leverage the power of principles. You can find hundreds of books, articles, websites, and even courses on effective presentations with any number of strategies, recipes, steps, tips, tricks, hacks, or rules devoted to helping you design and deliver effective presentations. How do you choose what’s right for you? Rely on your principles.

Why principles?

There are certainly any number of definitions for the word “principles,” but I’m framing them here as the synthesis of fundamental rules and values that guide our decisions and actions. As such, principles are descriptive, rather than prescriptive. Principles can be shared in common with others, but in practice, may look quite different from presenter to presenter when applied. You and I could each be guided by the principle that audience engagement is necessary for effective presentations, but we will likely go about applying this principle in very different ways.

Under principles we can certainly suggest strategies, models, formulas, or step-by-step plans for adhering to them, but these do not need to be applied assiduously in a “take as directed” sort of way. The broad principles we choose guide our work and our decisions about how to use suggested strategies.

What principles might guide presentation work?

The American Evaluation Association’s Potent Presentations (p2i) Initiative rests on the principle that the way in which you structure your message, design your presentation, and deliver it are key elements to success.

Here are others from presentation experts:

Every presentation worth doing has just one purpose: To make a change happen. – Seth Godin

Great content is a necessary condition, but not a sufficient one.  – Garr Reynolds

Make it Understandable; Make it Logical; Make it Real…to be more persuasive.Andrew Dlugan

Conceptualizing and framing what you want to say is the most vital part of preparation.Chris Anderson

What key presentation principle guides my work?

I’m a career educator who has taught everything from algebra to piano lessons, and my number one guiding principle is this: All presentations are lessons and all presenters are teachers. This principle guides my thinking as I develop presentations with participants’ needs in mind: what do they need to know, understand, or be able to do as a result of attending my presentation?

Look for upcoming blog posts on presentation principles, along with ideas, strategies, and steps you can apply in your own ways as you are guided by your presentation principles.

good presentation by C Lysy

Image Credit: Original cartoon by Chris Lysy

*Thanks to Jon Schwabish for that gem!


2 Comments

Presentation Principles: A Q&A on the Q&A

You’re ready for the big day. You have your best content all set to go, well-designed visuals, and a plan for successful delivery including how to engage your audience. You’ve practiced in the mirror, on your family, and on your pets, and they have all given you the go-ahead for your presentation. The night before, however, you wake up at 3:17am in a cold sweat thinking, “How will I handle the Q&A?”

Navigating questions and answers during your presentation can be daunting if you’re not prepared for them. But how can you prepare for the unknown? Go back to sleep…you CAN be ready for this key component of your presentation.

Image credit: opensource.com via Flickr

Image credit: opensource.com via Flickr

Presentations come in all shapes and sizes, from a 5-6 minute Ignite or Pecha Kucha, to a 15-20 minute TED talk, an hour long keynote, a sales pitch, market report, research presentation, webinar, professional development workshop, or a college course. And there are certainly a lot more. WIth that in mind, you’ll have to tailor these answers to the specific context and characteristics of your presentation.

These may be some of YOUR questions to which I offer some answers:

How much time should I leave for Q&A?

You may not like this answer, but it depends. Here are some questions for consideration:

How much time are you comfortable leaving? Are you prepared with additional material should there be few or no questions? Are you prepared to cut off questions with a plan to answer them later (e.g. by offering to stay after the presentation, or be contacted via email) if time runs out?

Image credit: Rabih via Flickr

Image credit: Rabih via Flickr

What if too many people ask questions and time runs out?

If time runs out and there are more questions, that’s often a good thing! It can mean people are engaged and interested in your topic. The bad news is it can also mean that you didn’t explain something well, or that you didn’t cover the content they were looking for. Of course, if you’re an optimist, you may see this as an opportunity to hone your craft and refine your content for future iterations of your presentation.

What if no one asks a question?

Image credit: Oliver Tacke via Flickr

Image credit: Oliver Tacke via Flickr

You can use one of three strategies here:

1.) Say “thank you” and close your presentation early. For some audiences, this will be a welcome little gift of time. Be cautious, however, that you won’t disappoint your host or client or be in breach of contract by ending early. Best to check out this possibility ahead of time. If you do end early, offer to stay in the room at least until the end of the original time. This will allow individuals the opportunity to ask questions of you in a less threatening and more private environment.

2.) Be prepared with additional content to offer for the remaining time. This might include sharing and answering questions previous audiences have asked – Ex. “I’m often asked about…and this is what I have to say about that…”

3.) Pose one or more questions to the audience to stimulate discussion or even catalyze questions from them.

What if someone asks a question I can’t answer?

Be honest! Acknowledge that a question may be outside of your scope of work or specific area of expertise. Audiences appreciate humility and it while it may seem counterintuitive, it can actually boost credibility. Offer resources for the questioner  – books, websites, or people – to find out the answer, or better yet, offer to research the answer yourself and contact the questioner at a later time.

Image credit: cielleandlacey charron and… via Flickr

Image credit: cielleandlacey charron and… via Flickr

What if I don’t understand the question?

Again, be honest! Ask the questioner to explain, rephrase, or provide an example to clarify. If that doesn’t work, take responsibility for not understanding (i.e. “Maybe it’s me and I’m just not getting it…”) and ask if anyone in the audience can answer. Of course, this takes a great deal of confidence and poise. In many cases, even before you do this, another audience member may attempt to rephrase for the original questioner.

Should I ask people to save their questions for the end?

Image credit: Nadi0 via Flickr

Image credit: Nadi0 via Flickr

Unless you are giving a TED Talk, keynote, or timed presentation where it would be inappropriate to stop and answer questions, let people ask questions when they have them to the extent that you can. Often, and especially in a professional development or workshop environment, people will have questions about something you just said. They may not remember these questions at the end of the session, or, more likely, they need to know the answer to the question so that they can continue to learn with with you. If you don’t take the time to answer clarifying questions, you risk losing some audience members who may become frustrated and shut down if they are not understanding the content.

Do you have any other advice, Sheila?

OK, you probably didn’t wake up with this question, but it’s my blog and I’ll share if I want to.

Look at questions as a gift – an opportunity to improve future presentations. After all, it’s a free needs assessment of your audience! They’re telling you exactly what they need to learn from you, what wasn’t clear, or what wasn’t covered. For this reason, ask someone ahead of time (I usually seek out the person who is hosting me) to write down audience questions and give them to you at the end. I typically walk away from presentations not remembering all the questions I was asked. Use these written questions to refine your presentation for next time.

What other questions might a presenter have about the Q&A? Please add them in the comments.


Leave a comment

Data Visualization: Sail Forth – Steer for the Deep Waters Only (Part II)

Either you decide to stay in the shallow end of the pool, or you go out into the ocean.

-Christopher Reeve

In an ongoing quest to improve my data visualization skills, I recently ventured out from the security of the shallow end and took a stab at the next level of sophistication with some basic charts. In Part I of this series I describe the process I used to create my first back-to-back bar chart. Once again, I learned most of the skills I applied for these from Stephanie Evergreen and Ann K Emery, both wonderful dataviz artists and great teachers.

Here is my first stab at small multiples. I evaluated a summer Leadership Academy for school leaders where participants were asked to name their top “take aways” from the Academy.

small multiples

In this way, the reader can see not only the top 9 topics cited, but can also determine what appeared to have more of an impact on participants who work at different levels. It’s clear that topics A and H are important to the secondary folks, while elementary folks seemed more interested in topics C, B, G and H. There were many fewer K-12 people at the Academy, but for that group, Topics A and C appear to stand out. Knowing this can help inform those who need to plan future academies and other professional learning opportunities.

Here is how to do this:

  1. Create a simple bar graph for one dataset (Topic A).
  2. Make these basic changes:
    • DELETE gridlines, x-axis and its tick marks, chart border. (NOTE: I could have deleted the x-axis line as well, but I left it in this chart for the purpose of visually “anchoring” each small bar chart)
    • ADD data labels.
    • FORMAT fonts (larger, bold).
    • ADJUST colors (I used colors associated with the organization).
  3. When you have everything looking the way you want it to on the first graph, create the remaining graphs by copying the first and editing the data.
    • Click “select data” and edit to get the correct data on the chart.
  4. ALIGN all the small charts.
    1. DELETE the y-axis on all but the leftmost charts. (Again, this could also be deleted, but I liked the way it “anchors” the the group of charts.)
  5. INSERT text boxes to identify the topics (this eliminates the need for the legend). (I created one – Topic A, then copied, pasted, and aligned it, and then replaced the text for the other topics).
  6. INSERT text boxes at the bottom for the shared x-axis (the categories secondary, elementary, and K-12 need only appear at the bottom of each column of small graphs).
  7. ADD a title and subtitle.

For more on small multiples and different ways of creating them, see Stephanie’s post and Ann K Emery’s blogs on small multiples.

Let’s be honest. This graph took some time and some fiddling to get it right. But, the investment will most certainly pay off in the future. Each time I clicked, I learned something new, and my next chart will be even better in less time! Onward!

 


Leave a comment

Data Visualization: Sail Forth – Steer for the Deep Waters Only (Part I)

Sail Forth- Steer for the deep waters only. Reckless O soul, exploring. I with thee and thou with me. For we are bound where mariner has not yet dared go. And we will risk the ship, ourselves, and all.

-Walt Whitman

I consider myself a novice, for now, staying safe in the shallow waters of data visualization. I’ve learned to create clean and modern-looking bar, column, and {gulp!} the occasional pie charts. I follow basic safety rules, dispensing with the unnecessary – gridlines, tick marks, superfluous axis labels and legends. I avoid default colors, and proudly leave 3D charts to the real amateurs. To venture beyond that, into the deep current of dashboards and interactivity, I would need a life jacket and tow rope. Recently, though, I waded a bit deeper, and experimented with two variations – back-to-back bar charts and small multiples. In this post, I share how I created my first back-to-back bar chart. Part II will tackle the small multiples.

I created a back-to-back bar chart to display some common public school data. Much of this process I learned from Stephanie Evergreen’s blogs on dataviz. I chose a back-to-back bar chart because I have two datasets that people need to see in one chart, yet they don’t necessarily need to compare one dataset (Math) to the other (ELA). What they do want to compare is one year to the next for each grade level.

Back to back chart

Here is how to do it:

  1. Create one horizontal bar chart.
  2. Make these basic changes:
    • DELETE gridlines, x-axis tick marks and line, y-axis, chart border. (NOTE: I could have deleted the x-axis as well, but I left it in this chart for the purpose of drawing the reader not just to the absolute scores, but to the scores as compared to 100%)
    • ADD data labels and title.
    • FORMAT fonts (larger, bold).
    • ADJUST colors (I used colors associated with the organization)
    • REDUCE gap width (I chose 40% for this graph).
  3. When you have everything looking the way you want it to on the first graph, create the second by copying the first and editing the data.
    • Click “select data” and edit to get the correct data on the chart.
    • FLIP the graph on the left (my Math graph) by right clicking on the x-axis, and checking “values in reverse order.”
  4. INSERT text boxes in the top bars of each graph to identify the years (this eliminates the need for the legend).
  5. INSERT text boxes in the center (between the two graphs) for the shared y-axis (I created one – Grade 3), then copied, pasted, and aligned it, and then replaced the text for the other grade levels).
  6. OUTLINE the bars in white, (a cool trick I learned from Ann K Emery’s blogs about dataviz) and increase the line weight bit (I put it up to 1.25).
  7. GROUP graphs by selecting both along with all text boxes and right click to “group” them. Grouping can be tricky in Excel and when you go to copy and paste, be sure to grab the grouped object, rather than just the graph.

 

SUCCESS! Go on, try it yourself! You know you want to…

Look for Part II on small multiples…coming soon.


2 Comments

Why Are Evaluators So Tentative about the Advocacy Aspect of Our Profession? (Guest Post by Rakesh Mohan)

I’ve recently had the pleasure of meeting an evaluator whose work I’ve followed, and I invited him to write for Evaluspheric Perceptions. Rakesh Mohan has been a regular on EvalTalk, and I’ve admired him for putting his work out there and asking for feedback. In early 2014, his office released a report, Confinement of Juvenile Offenders, and I found myself curious enough to read it. Quite honestly, it wasn’t the topic that interested me, but rather, it was the idea of reading a governmental evaluation report produced by someone who is a great fan of and frequent presenter at the American Evaluation Association. Needless to say, the report impressed me to no end. Rakesh put into place nearly every principle of good evaluation reporting and data visualization that I have been learning and studying myself. I’m not the only one impressed by the work coming out of his office. In 2011, they received AEA’s Alva and Gunnar Myrdal Government Evaluation Award. Recent posts on two of my favorite blogs highlight the work of his office as well. One can be found at Better EvaluationWeek 15: Fitting reporting methods to evaluation findings – and audiences and the other at AEA365Sankey diagrams: A cool tool for explaining the complex flow of resources in large organizations.Mohan, 2014

I’m also happy to learn that Rakesh is on the 2014 ballot for the AEA presidency.

Today, Rakesh presents another topic that often confounds me, but he demystifies it with ease. So, please join me in learning about Rakesh Mohan, and evaluator advocacy!

Why Are Evaluators So Tentative about the Advocacy Aspect of Our Profession?

My mother used to say that where there are two or more people, there will always be politics over resources. Because evaluations involve making judgments about prioritization, distribution, and use of resources, evaluations will always be inherently political.

Greetings! I am Rakesh Mohan, director of the Office of Performance Evaluations (OPE), an independent agency of the Idaho Legislature. This year our office is celebrating 20 years of promoting confidence and accountability in state government.

OPE logo

At OPE, there is nothing tentative about advocating for our work—i.e., promoting the use of our evaluations, defending our evaluation approaches and methodologies, and educating people about evaluation. For us, evaluator advocacy is all in a day’s work.

I believe it is the fear of politics that makes many evaluators tentative about advocacy. Some evaluators say that it is not their job to mess with the politics lest they be perceived as taking sides, while others do not even acknowledge the fact that evaluation and politics are intertwined. The option for evaluators is not to ignore the political context of evaluation, but to understand and manage it without taking sides.

The following advocacy activities of my office are grounded in professional evaluation and auditing standards and are guided by our personal ethics:

  1. Conduct my “daily sojourn.” Each year during the legislative session, I visit the capitol every day even if I do not have a scheduled meeting. These visits help me to inform others about the work of my office and be informed about the political context in which we conduct evaluations.
  2. Keep legislative leadership informed. This is the first step in building relationships with policymakers and gaining the confidence and support of the leadership.
  3. Keep stakeholders informed. This is imperative if we want to have the buy-in from key stakeholders.
  4. Assist policymakers with evaluation requests and legislation. This helps with getting good evaluation assignments and subsequently facilitates the implementation of our evaluation recommendations.
  5. Educate policymakers and others about evaluation. We should let policymakers and those who influence policymaking know who we are, what we do, and why we do it.
  6. Work with the news media effectively. If betterment of the society is one of the purposes of evaluation, we need to reach out to the people. The press can serve as a bridge between evaluators and the public. Here are three examples of how the media can help evaluators and evaluation offices:

Mohan 2

 

Details about these strategies and other thoughts on evaluator advocacy are discussed in my recent article, Evaluator Advocacy: It Is All in a Day’s Work (April 25, 2014).

This article was published along with two related articles in the Forum section of the American Journal of Evaluation:

How to Become an Effective Advocate without Selling Your Soul (George Grob, April 22, 2014)

Broadening the Discussion about Evaluator Advocacy (Michael Hendricks, April 17, 2014)

All three articles are available from OnlineFirst of the American Journal of Evaluation.


14 Comments

When a Direct Question is NOT the Right Question

Who hasn’t answered the question, “What did you learn?” after attending a professional development session? As a PD facilitator and evaluator, I’ve certainly used feedback forms with this very question. After all, measuring participant learning is fundamental to PD evaluation.

In this post, I’ll share examples of actual data from PD evaluation in which we asked the direct question, “What did you learn?” I’ll then explain why this is a difficult question for PD participants to answer, resulting in unhelpful data. Next, I’ll offer a potential solution in the form of a different set of questions for PD evaluators to use in exploring the construct of participant learning. Finally, I’ll show where participant learning fits into the bigger picture of PD evaluation.

What happens when we ask “What did you learn?” 

Here are examples of actual participant responses to that question:

  • After a session on collaborative problem solving with students with behavioral difficulties: How to more effectively problem solve with students
  • After a session on co-teaching: Ways to divide up classroom responsibilities
  • After a session on teaching struggling learners: Some new strategies to work with struggling students

In my experience, about one-third to one-half of participant responses to that ubiquitous question are nothing more than restatements of the course title and thus similarly uninformative to an evaluator.

On the futility of asking “What did you learn?”

It’s challenging to get people to clearly articulate what they have learned on a feedback form distributed after a professional development session. Whether the question is asked immediately after the learning has taken place, or after some time has passed and the participant (in theory) has had time to process and apply the learning, the outcome (in terms of the data collected) is the same. People don’t seem to be able (I’m working under the assumption that they are indeed willing) to answer “What did you learn?” with the depth and richness of written language that would help inform professional learning planners make effective decisions about future programming. They’re not to blame, of course. It’s just as difficult for me to answer that question when I’m a participant.

Parents and teachers know this: When you ask a child a question and he or she answers with “I don’t know,” that response can have a whole range of meanings from “I can’t quite articulate the answer you’re looking for” to “I’m not certain I know the answer” to “I need more time to process the question” to “I don’t understand the question” to “I really don’t know the answer” to “I don’t want to tell you!” It’s no different for adults. Someone who answers “What did you learn?” by essentially restating the title of the PD session is in effect saying, “I don’t know.” As a PD evaluator, it is my job to figure out exactly what that means.

How else can we know what participants learned?

Of course, we’re talking about surveys – self-reported perceptions of learning. There are certainly other ways for evaluators to gain an understanding of what participants learned.

We can interview them, crafting probes that might help them more clearly articulate what they learned. Interviews include dedicated time and the opportunity for participants to give full attention to the question. In contrast, surveys are often completed when participants feel rushed at the end of a PD session, or at a later time, when they are fitting survey completion in with a myriad of other job duties.

We can observe participants at work, looking for evidence that they are applying what they learned in practice, thus getting at not only what they may have learned, but also what Kirkpatrick called “behavior” and Guskey calls “participant use of new knowledge and skills” (see below for more on these evaluators and their prescribed levels of PD evaluation).

Both interviews and observations, however, are considerably more time consuming and thus less feasible for an individual evaluator.

As an alternative, I wondered what might happen if rather than asking, “What did you learn?” we asked, “How did you feel?” Learning has long been highly associated with emotions. (For more on learning and emotions, check out this article, this one, and this one, and look at the work of John M. Dirkx and Antonio Damasio, among many others.) Would PD participants be better able to articulate how they felt during PD, and would their learning then become evident in their writing?

What happens when we ask a different question?

Well, a different set of questions, really. A colleague and I created a new feedback form to pilot with PD participants in which we seek to understand their learning through a series of five questions. We discussed at length what it is we want to know about participants’ learning to inform our programmatic decisions. We concluded that it is not necessarily the content  (i.e. if participants attend a course on instructional planning, then we expect they will learn something about instructional planning), but whether participants experience a change in thinking, feel they have learned a great deal, and whether or not the content is new for them.

We begin with these three questions using 5-point standard Likert response options (Strongly disagree, disagree, neither agree nor disagree, agree, strongly agree):

  1. This professional learning opportunity changed the way I think about this topic.
  2. I feel as if I have learned a great deal from participating in this professional learning opportunity.
  3. Most or all of the content was a review or refresher for me (this question is reverse-coded, of course).

We then ask participants about their emotions during the session with a set of “check all that apply” responses:

During this session I felt:

  • Energized
  • Renewed
  • Bored
  • Inspired
  • Overwhelmed
  • Angry
  • In agreement with the presenter
  • In disagreement with the presenter
  • Other

Finally, we ask participants to “Please explain why you checked the boxes you did,” and include an open essay box for narrative responses.

I’ve only seen data from one course thus far, but it is quite promising in that participants were very forthcoming in their descriptions of how they felt. Through their descriptions we were able to discern the degree of learning and from many responses, how participants plan to apply that learning. We received far fewer uninformative responses than in previous attempts to measure learning with the one direct question. As we continue to use this new set of questions, I hope to share response examples in a future post.

Image Credit: Collette Cassinelli

Image Credit: Collette Cassinelli via Flickr

Where does participant learning fit into the PD evaluation picture?

Donald Kirkpatrick famously proposed four levels of evaluation of training or professional development – essentially measuring participants’ 1.) reactions, 2.) learning, 3.) behavior, and 4.) results –  for training programs in the 1950s. Thomas Guskey later built upon Kirkpatrick’s model, adding a fifth level – organizational support and learning (Guskey actually identifies this as level 3; For more on this topic, see this aea365 post I wrote with a colleague during a week devoted to Guskey’s levels of PD evaluation sponsored by a professional development community of practice).

For hardcore evaluation enthusiasts, I suggest Michael Scriven’s The Evaluation of Training: A Checklist Approach

What other questions could we ask to understand PD participant’s learning?

I welcome your suggestions, so please add them to the comments!

 

Follow

Get every new post delivered to your Inbox.

Join 780 other followers