Evaluspheric Perceptions

Reflections of an everyday evaluator/learner/educator exploring the evalusphere

Leave a comment

It’s All in How You Ask: The Nuances of Survey Question Design (cross post with Actionable Data blog)

If you do not ask the right questions, you do not get the right answers.

– Edward Hodnett, 20th century poet and writer

At Evaluation 2014, the American Evaluation Association’s annual conference, the incredible Kim Firth Leonard (find her over at Actionable Data blog) and I facilitated a 90-minute skill building workshop on survey question design. Kim and I have co-authored several posts on our shared passion for survey design. You can find these posts here, here, and here. We were thrilled to geek out with such a great group ready to humor us by taking our pop quiz, listening intently as we shared the science behind answering questions about behavior, nodding as we reviewed fundamental principles of survey design, and genuinely engaging with us in exploring better ways to approach surveys.

As we shared during the workshop — we have both been in the business of collecting data long enough to be pretty annoyed by poorly written surveys, and see a real need for better, more specific advice for question design than we’ve been able to find to date. Our main goal has always been to learn together (and with anyone else interested in joining in the fun) to improve our own survey design, and to encourage others to take survey design more seriously. And we’re so grateful for the many attendees who were willing to learn with us in Denver in October – thank you! For those who did attend, or who didn’t but wish they were there (we wish you were there too), here are our slides:

Slides: It’s All in How You Ask: The Nuances of Survey Question Design – presentation at Evaluation 2014, Denver, CO

Favorite tips generated during the session:

Our participants offered a wealth of tips and advice for survey researchers. We don’t necessarily agree with all the tips that were shared, but think that most of them have something to offer in terms of food for thought and consideration.

  • We received quite a few tips related to planning surveys
    • Be sure to identify and clearly outline your needs as well as how you intend to use the results
    • Start with your data quality objectives — what you want to report out on. Then you know you have what you need without extras
    • Identify what you “need to know” and what would be “nice to know” in order to prioritize what questions to include, and limit burden on respondents.
    • Test survey instrument with the client (e.g donor), walk through each question and say – if the results of this question are positive, what will you do with that information? If negative – what then? What will be helpful?
  • Some tips related to how to proofread or even roughly validate your survey questions
    • Even if you’re using a mail/web survey, read it out loud – you may hear things you don’t notice when reading.
    • Work closely with subject matter experts to ensure language, etc. is correct.
    • Cognitive interviews/ think aloud strategies are especially helpful when surveying communities of which you are not a member (e.g. kids think about things in different ways, and your questions might not mean the same thing to them as to you).
    • In some situations it may make sense to start with open-ended questions with a small group of future respondents (or those in like positions) and use their responses to generate the response options, rather than starting from your own assumptions. 
  • And many were even more specific!
    • For questions with scalar response options, between 4 and 7 is usually appropriate total number of response options.
    • Remember that neutral is different than no opinion. Both options could even be used together in the same question.
    • Limit questions asking what the respondent thinks about OTHER’S motivations (e.g. why did they do x) for those situations where this is truly appropriate
    • Be mindful of the AGE of respondents. Seniors will interpret words/tone very differently from those in their 20s.
    • Beware of “yes/no” questions — won’t be as useful if you intended to look for change. And often too binary — life is often more complex than yes/no.

Next Steps:questions by Marcus Ramberg via Flickr

Judging by the session turn-out and follow-up emails we’ve received, the conversation we started is well worth continuing! One way we hope to do so is through continued crowdsourcing of survey design tips and challenges. To that end, we’ve started a GoogleDoc with some of our favorite tips and many of those gathered through our workshop. We’ve made this available to everyone and anyone to add to (and hopefully learn from).

Our intention is to continue to build on this, and to develop a checklist or similar tool to help support better survey design. We look forward to continuing to learn with you! Stay in touch if you’re interested in continuing to geek out with us about this.

Please add tips and include your name (and sources as appropriate). Use THIS LINK to add to the GoogleDoc (and see the live doc below)!


Leave a comment

Presentation Principles: The Audience Engagement Strategy Book

Presentation Principle:

Audience engagement through interactive strategies is key to a successful presentation.

discussion iconA presentation is a precious opportunity – Seth Godin

In summer 2014, I had the opportunity to write the Audience Engagement Strategy Book for the American Evaluation Association’s Potent Presentations (p2i) Initiative (available for free download on the p2i website). I’d written about audience engagement before (see here and here), but this time wanted to share specific interactive strategies that come with sets of directions and steps, and that can be customized to fit a presenter’s specific context.

This workbook describes more than 20 strategies for audience engagement drawn from literature on effective presentations, and on instructional strategies for engaging learners. It begins with some background – the WHAT, WHY and HOW of audience engagement. Each strategy is then rated on a number of dimensions, so you know what you’re getting into. Audience engagement can be as simple as using eye contact strategically, to preparing specific materials and actually teaching a lesson where participants engage in discussion, reading, writing, sharing, or creating.

Audience engagement strategy book

Being a career educator, I simply took student and participant engagement strategies I’ve learned from years of public school and university teaching along with facilitating professional development sessions, and applied them to conference and other types of presentations. In fact, most K-12 teachers would recognize a majority of the strategies.

However, for those who haven’t had the unique pleasure of teaching groups of active, energetic preadolescents/adolescents whose brains are not wired for dry-as-dust lectures, and having to creatively engage them in learning, these strategies may come as welcome additions to your toolbox of effective presentation skills.

As with any publication, the second it went out, I thought of more to add. And that brings me to the blog! Look for these additions in future posts.

Do you have favorite strategies for audience engagement or interaction? I’d love to hear about them in the comments!


Presentation Principles: Right-sizing the Room to Maximize Audience Engagement

Presentation Principle:

The right setting can impact how your audience experiences a presentation.

It’s the Goldilocks conundrum: You’re scheduled for a conference presentation (i.e. a non-ticketed event were people do not pre-register) and you get to the room, only to discover that it is either too small or too big for your anticipated audience size… definitely not just right!

I recently presented with a colleague at a multi-day conference with hundreds of concurrent sessions where they assigned our session to a room that seats about 100 people. Our topic was a popular one, and the time slot was a preferred one for that conference (i.e. not during a time that people would naturally take a break and choose not attend a session). Needless to say, the room filled up and people just kept coming in long after all chairs were taken.

At another conference, one of my presentations was assigned to a gigantic ballroom that seated over 1000 people! My co-presenters and I expected that our session, given the topic and time slot, would more likely draw an audience of perhaps 20- 50 people.

What do you do when the room isn’t the right size for your presentation?

First course of action: If time permits, ask to be reassigned to a different size room. Unfortunately, this is often impossible for the conference organizers to do, but hey, it never hurts to ask, right? If that doesn’t work, try some of these strategies:

If the room is too small:

1.) When the room starts looking full enough that people are scanning for a seat before they even get too far in, ask those seated to raise their hands to signal an empty seat next to them.

2.) Often the hosts will put one or two chairs by your presenter table. If you won’t be using these, offer them to audience members. If your conference organizers or hosts are nearby of course, you can ask for extra chairs, but often, this option won’t be available.

3.) When all seats are taken, invite people to sit on the floor in the front of the room. Whaaaaaat? Adults sitting on the floor? Yes! You may be surprised, but many people will choose this as an alternative to being locked out an enticing presentation.

4.) If the inevitable happens, and people are locked out, here’s a way to acknowledge their attempt to make it in, and give them a little something for their trouble. Put a sign on the outside door with something to the effect of, “If this room is full and you can’t get in, here’s the URL where you can download presentation materials.”*

If the room is too big:

1.) If you get to the room early enough, tape off or otherwise block off the seats near the back or on one side of the room. This is one of the many reasons you may want to carry a roll of painter’s tape (which generally won’t damage surfaces) with you to presentations.

2.) Greet people as they enter and ask them to choose seats in a particular section or near the front.

3.) Quickly create a PowerPoint slide that asks people to sit near the front and have this on the screen prior to starting.

4.) If people do end up scattering and spread out in a large room, be sure to walk around during your presentation using proximity to engage them. You can also offer them the opportunity to move closer during a break.

Any other ideas to share on right-sizing your presentation space? Please add them in the comments!

*Thanks to Chris Lysy for this strategy he used at a recent conference.

Leave a comment

Presentation Principles: Foundations of Effective Presentations

There are multiple pathways to great presentations.

Some presenters are fortunate enough to be naturally possessed of  stage presence, charisma, or a je ne sais quois that keeps their audience hanging on their every word. After all, Martin Luther King, Jr. didn’t use PowerPoint slides.* Steve Jobs never asked his audience to turn and talk to an elbow partner. The rest of us, though, are just not there. We’re presenting to the board, our colleagues, university students, fellow conference-goers, or some other audience who may or may not know us or be familiar with our work. “Star quality” is not something we can rely on for a successful presentation.

What is an everyday presenter to do?

Leverage the power of principles. You can find hundreds of books, articles, websites, and even courses on effective presentations with any number of strategies, recipes, steps, tips, tricks, hacks, or rules devoted to helping you design and deliver effective presentations. How do you choose what’s right for you? Rely on your principles.

Why principles?

There are certainly any number of definitions for the word “principles,” but I’m framing them here as the synthesis of fundamental rules and values that guide our decisions and actions. As such, principles are descriptive, rather than prescriptive. Principles can be shared in common with others, but in practice, may look quite different from presenter to presenter when applied. You and I could each be guided by the principle that audience engagement is necessary for effective presentations, but we will likely go about applying this principle in very different ways.

Under principles we can certainly suggest strategies, models, formulas, or step-by-step plans for adhering to them, but these do not need to be applied assiduously in a “take as directed” sort of way. The broad principles we choose guide our work and our decisions about how to use suggested strategies.

What principles might guide presentation work?

The American Evaluation Association’s Potent Presentations (p2i) Initiative rests on the principle that the way in which you structure your message, design your presentation, and deliver it are key elements to success.

Here are others from presentation experts:

Every presentation worth doing has just one purpose: To make a change happen. – Seth Godin

Great content is a necessary condition, but not a sufficient one.  – Garr Reynolds

Make it Understandable; Make it Logical; Make it Real…to be more persuasive.Andrew Dlugan

Conceptualizing and framing what you want to say is the most vital part of preparation.Chris Anderson

What key presentation principle guides my work?

I’m a career educator who has taught everything from algebra to piano lessons, and my number one guiding principle is this: All presentations are lessons and all presenters are teachers. This principle guides my thinking as I develop presentations with participants’ needs in mind: what do they need to know, understand, or be able to do as a result of attending my presentation?

Look for upcoming blog posts on presentation principles, along with ideas, strategies, and steps you can apply in your own ways as you are guided by your presentation principles.

good presentation by C Lysy

Image Credit: Original cartoon by Chris Lysy

*Thanks to Jon Schwabish for that gem!


Presentation Principles: A Q&A on the Q&A

You’re ready for the big day. You have your best content all set to go, well-designed visuals, and a plan for successful delivery including how to engage your audience. You’ve practiced in the mirror, on your family, and on your pets, and they have all given you the go-ahead for your presentation. The night before, however, you wake up at 3:17am in a cold sweat thinking, “How will I handle the Q&A?”

Navigating questions and answers during your presentation can be daunting if you’re not prepared for them. But how can you prepare for the unknown? Go back to sleep…you CAN be ready for this key component of your presentation.

Image credit: opensource.com via Flickr

Image credit: opensource.com via Flickr

Presentations come in all shapes and sizes, from a 5-6 minute Ignite or Pecha Kucha, to a 15-20 minute TED talk, an hour long keynote, a sales pitch, market report, research presentation, webinar, professional development workshop, or a college course. And there are certainly a lot more. WIth that in mind, you’ll have to tailor these answers to the specific context and characteristics of your presentation.

These may be some of YOUR questions to which I offer some answers:

How much time should I leave for Q&A?

You may not like this answer, but it depends. Here are some questions for consideration:

How much time are you comfortable leaving? Are you prepared with additional material should there be few or no questions? Are you prepared to cut off questions with a plan to answer them later (e.g. by offering to stay after the presentation, or be contacted via email) if time runs out?

Image credit: Rabih via Flickr

Image credit: Rabih via Flickr

What if too many people ask questions and time runs out?

If time runs out and there are more questions, that’s often a good thing! It can mean people are engaged and interested in your topic. The bad news is it can also mean that you didn’t explain something well, or that you didn’t cover the content they were looking for. Of course, if you’re an optimist, you may see this as an opportunity to hone your craft and refine your content for future iterations of your presentation.

What if no one asks a question?

Image credit: Oliver Tacke via Flickr

Image credit: Oliver Tacke via Flickr

You can use one of three strategies here:

1.) Say “thank you” and close your presentation early. For some audiences, this will be a welcome little gift of time. Be cautious, however, that you won’t disappoint your host or client or be in breach of contract by ending early. Best to check out this possibility ahead of time. If you do end early, offer to stay in the room at least until the end of the original time. This will allow individuals the opportunity to ask questions of you in a less threatening and more private environment.

2.) Be prepared with additional content to offer for the remaining time. This might include sharing and answering questions previous audiences have asked – Ex. “I’m often asked about…and this is what I have to say about that…”

3.) Pose one or more questions to the audience to stimulate discussion or even catalyze questions from them.

What if someone asks a question I can’t answer?

Be honest! Acknowledge that a question may be outside of your scope of work or specific area of expertise. Audiences appreciate humility and it while it may seem counterintuitive, it can actually boost credibility. Offer resources for the questioner  – books, websites, or people – to find out the answer, or better yet, offer to research the answer yourself and contact the questioner at a later time.

Image credit: cielleandlacey charron and… via Flickr

Image credit: cielleandlacey charron and… via Flickr

What if I don’t understand the question?

Again, be honest! Ask the questioner to explain, rephrase, or provide an example to clarify. If that doesn’t work, take responsibility for not understanding (i.e. “Maybe it’s me and I’m just not getting it…”) and ask if anyone in the audience can answer. Of course, this takes a great deal of confidence and poise. In many cases, even before you do this, another audience member may attempt to rephrase for the original questioner.

Should I ask people to save their questions for the end?

Image credit: Nadi0 via Flickr

Image credit: Nadi0 via Flickr

Unless you are giving a TED Talk, keynote, or timed presentation where it would be inappropriate to stop and answer questions, let people ask questions when they have them to the extent that you can. Often, and especially in a professional development or workshop environment, people will have questions about something you just said. They may not remember these questions at the end of the session, or, more likely, they need to know the answer to the question so that they can continue to learn with with you. If you don’t take the time to answer clarifying questions, you risk losing some audience members who may become frustrated and shut down if they are not understanding the content.

Do you have any other advice, Sheila?

OK, you probably didn’t wake up with this question, but it’s my blog and I’ll share if I want to.

Look at questions as a gift – an opportunity to improve future presentations. After all, it’s a free needs assessment of your audience! They’re telling you exactly what they need to learn from you, what wasn’t clear, or what wasn’t covered. For this reason, ask someone ahead of time (I usually seek out the person who is hosting me) to write down audience questions and give them to you at the end. I typically walk away from presentations not remembering all the questions I was asked. Use these written questions to refine your presentation for next time.

What other questions might a presenter have about the Q&A? Please add them in the comments.

Leave a comment

Data Visualization: Sail Forth – Steer for the Deep Waters Only (Part II)

Either you decide to stay in the shallow end of the pool, or you go out into the ocean.

-Christopher Reeve

In an ongoing quest to improve my data visualization skills, I recently ventured out from the security of the shallow end and took a stab at the next level of sophistication with some basic charts. In Part I of this series I describe the process I used to create my first back-to-back bar chart. Once again, I learned most of the skills I applied for these from Stephanie Evergreen and Ann K Emery, both wonderful dataviz artists and great teachers.

Here is my first stab at small multiples. I evaluated a summer Leadership Academy for school leaders where participants were asked to name their top “take aways” from the Academy.

small multiples

In this way, the reader can see not only the top 9 topics cited, but can also determine what appeared to have more of an impact on participants who work at different levels. It’s clear that topics A and H are important to the secondary folks, while elementary folks seemed more interested in topics C, B, G and H. There were many fewer K-12 people at the Academy, but for that group, Topics A and C appear to stand out. Knowing this can help inform those who need to plan future academies and other professional learning opportunities.

Here is how to do this:

  1. Create a simple bar graph for one dataset (Topic A).
  2. Make these basic changes:
    • DELETE gridlines, x-axis and its tick marks, chart border. (NOTE: I could have deleted the x-axis line as well, but I left it in this chart for the purpose of visually “anchoring” each small bar chart)
    • ADD data labels.
    • FORMAT fonts (larger, bold).
    • ADJUST colors (I used colors associated with the organization).
  3. When you have everything looking the way you want it to on the first graph, create the remaining graphs by copying the first and editing the data.
    • Click “select data” and edit to get the correct data on the chart.
  4. ALIGN all the small charts.
    1. DELETE the y-axis on all but the leftmost charts. (Again, this could also be deleted, but I liked the way it “anchors” the the group of charts.)
  5. INSERT text boxes to identify the topics (this eliminates the need for the legend). (I created one – Topic A, then copied, pasted, and aligned it, and then replaced the text for the other topics).
  6. INSERT text boxes at the bottom for the shared x-axis (the categories secondary, elementary, and K-12 need only appear at the bottom of each column of small graphs).
  7. ADD a title and subtitle.

For more on small multiples and different ways of creating them, see Stephanie’s post and Ann K Emery’s blogs on small multiples.

Let’s be honest. This graph took some time and some fiddling to get it right. But, the investment will most certainly pay off in the future. Each time I clicked, I learned something new, and my next chart will be even better in less time! Onward!


Leave a comment

Data Visualization: Sail Forth – Steer for the Deep Waters Only (Part I)

Sail Forth- Steer for the deep waters only. Reckless O soul, exploring. I with thee and thou with me. For we are bound where mariner has not yet dared go. And we will risk the ship, ourselves, and all.

-Walt Whitman

I consider myself a novice, for now, staying safe in the shallow waters of data visualization. I’ve learned to create clean and modern-looking bar, column, and {gulp!} the occasional pie charts. I follow basic safety rules, dispensing with the unnecessary – gridlines, tick marks, superfluous axis labels and legends. I avoid default colors, and proudly leave 3D charts to the real amateurs. To venture beyond that, into the deep current of dashboards and interactivity, I would need a life jacket and tow rope. Recently, though, I waded a bit deeper, and experimented with two variations – back-to-back bar charts and small multiples. In this post, I share how I created my first back-to-back bar chart. Part II will tackle the small multiples.

I created a back-to-back bar chart to display some common public school data. Much of this process I learned from Stephanie Evergreen’s blogs on dataviz. I chose a back-to-back bar chart because I have two datasets that people need to see in one chart, yet they don’t necessarily need to compare one dataset (Math) to the other (ELA). What they do want to compare is one year to the next for each grade level.

Back to back chart

Here is how to do it:

  1. Create one horizontal bar chart.
  2. Make these basic changes:
    • DELETE gridlines, x-axis tick marks and line, y-axis, chart border. (NOTE: I could have deleted the x-axis as well, but I left it in this chart for the purpose of drawing the reader not just to the absolute scores, but to the scores as compared to 100%)
    • ADD data labels and title.
    • FORMAT fonts (larger, bold).
    • ADJUST colors (I used colors associated with the organization)
    • REDUCE gap width (I chose 40% for this graph).
  3. When you have everything looking the way you want it to on the first graph, create the second by copying the first and editing the data.
    • Click “select data” and edit to get the correct data on the chart.
    • FLIP the graph on the left (my Math graph) by right clicking on the x-axis, and checking “values in reverse order.”
  4. INSERT text boxes in the top bars of each graph to identify the years (this eliminates the need for the legend).
  5. INSERT text boxes in the center (between the two graphs) for the shared y-axis (I created one – Grade 3), then copied, pasted, and aligned it, and then replaced the text for the other grade levels).
  6. OUTLINE the bars in white, (a cool trick I learned from Ann K Emery’s blogs about dataviz) and increase the line weight bit (I put it up to 1.25).
  7. GROUP graphs by selecting both along with all text boxes and right click to “group” them. Grouping can be tricky in Excel and when you go to copy and paste, be sure to grab the grouped object, rather than just the graph.


SUCCESS! Go on, try it yourself! You know you want to…

Look for Part II on small multiples…coming soon.


Get every new post delivered to your Inbox.

Join 818 other followers