Understanding the Data

“We do a good job of asking students what they know in school; we need to do much better asking students what they think and feel about their school.”  

– Dr. Russell Quaglia

All My Voice reports categorize answers to the My Voice Survey in tables organized by the 8 Conditions that Make a Difference®: Belonging, Heroes, Sense of Accomplishment, Fun & Excitement, Curiosity & Creativity, Spirit of Adventure, Leadership & Responsibility, and Confidence to Take Action.

My Voice Student Report (U.S. grades 6-12; U.K. years 7-13): For each indicator, the tables in this report display the “total in agreement” (the sum of responses marked “strongly agree” and “agree” divided by all those who responded to that question).  In addition, the "total in agreement" is broken down by gender and grade level.

My Voice Student Report (U.S. grades 3-5; U.K. Key Stage 2): For each indicator, the tables in this report display percentages for all "yes," "no," and "sometimes" responses. In addition, the "yes" responses are broken down by gender and grade level.

My Voice Staff Report: For each indicator, the tables in this report display the "total in agreement" (the sum of responses marked “strongly agree” and “agree” divided by all those who responded to that question).

My Voice Parent Report: For each indicator, the tables in this report display the "total in agreement" (the sum of responses marked “strongly agree” and “agree” divided by all those who responded to that question).

Reporting “total in agreement” provides schools with the most reliable information available from the survey and helps those reading the results to stay action oriented. Try not to get bogged down in a discussion about participants who marked “undecided” or “sometimes.” They cannot be counted among those who agree, and agreement is ultimately the goal for each indicator.

Each school is asked to survey every individual in a designated group – students, staff, parents­; as a result, there is no “sampling error” in the report. This provides schools with a solid foundation for interpretation and closes the door on wondering whether those who responded happened to be “axe-grinders” or, conversely, were those who “see the world through rose-colored glasses.” Your school can be confident that the results reflect the general sentiments of the entire student body, staff, or parents.

There are three important concepts to keep in mind when analyzing your survey results:

  1. The results reflect respondents’ perceptions of reality.
  2. The data are merely numbers.
  3. The most helpful comparisons are internal.

PERCEPTION IS DIFFERENT THAN REALITY

My Voice Surveys are opinion surveys. As such, the results reflect the respondents’ perceptions of reality, not necessarily reality. That is not to dismiss the results (as some are inclined to do) as merely respondents’ perceptions or opinions: “That’s just what the students think, we know what the truth is.”

On the contrary, your students’ perceptions shape what they think and how they act in your school and so provide an accurate picture of your school’s climate and culture. The fact of the matter is that reality is complex; to truly understand it, we need to hear various points of view on the same realities.

When students’ perceptions are put in dialogue with the staff and parents’ perceptions, insights emerge and can form a basis for sound decision-making and action.

DATA ARE MERELY NUMBERS

Numbers contain no insights, judgments of fact or value, decisions, or action plans. Mistakes sometimes made in reacting to data include:

  • To move too quickly from data to judgment of fact: “We correctly understand what students/staff/parents are saying.”
  • To jump from data to judgment of value: “This is bad.”
  • To leap from data to action planning: “We have to do something.”

Rather, interpreting results is a process of moving from data to insights (What do these numbers mean?) to judgments of fact (Have we understood the data correctly?), then value (Is this important to us?) and from there to action (What should we do?).

This process begins with inquiry and dialogue among colleagues, students, and parents. In these conversations, staff can question the data and form hypotheses that might explain the results. For example, if only 35% of your students report feeling proud of their school, what might be the source of that diminished pride? A lack of success in sports? An aging physical plant? Failing to meet AYP? If 93% of staff report that your school is a dynamic and creative learning environment, what contributes to this? Collaboration among teachers? A respectful atmosphere that encourages risk-taking?

While the process begins with collegial conversation, it must not end there. "Analysis paralysis" is also a danger. QISA and the Pearson Foundation recommend conducting focus groups to learn whether adults have correctly understood what students, staff, and parents are saying. (See Appendix 2 of this Administrator's Guidebook for instructions on conducting focus groups.) Only when you verify that you have understood correctly can you decide what actions to take to improve the Conditions that affect students' aspirations.

COMPARE YOURSELF TO YOURSELF

Many schools want to interpret their results in comparison to a national or state aggregate. While this can be interesting, it is not always helpful. The 8 Conditions are not norms, they are ideals. We want every member of the school to feel like he or she belongs. Every student should feel a sense of accomplishment. We should not be satisfied that a few percent less of our students are bored in school than the national average. As ideals, the “normative” level for each Condition is 100%. Likewise, we want every staff member to know the school’s mission statement and every parent to feel welcome in our school.

The most illuminating and helpful comparisons are internal. What do boys in your school say as compared to girls? If there is a discrepancy, what accounts for it? Is the gap acceptable? What is happening from grade level to grade level? You might also compare your survey results to your mission statement or core beliefs. Does your school profess the centrality of community? How does that compare with what your staff say about Belonging? Does your mission statement espouse the importance of actively engaging students in the learning process? If so, is this reflected in the tables having to do with Fun & Excitement, Curiosity & Creativity, and Spirit of Adventure? Do your school’s goals include fostering a sense of purpose among students and staff? Do your results for Leadership & Responsibility and Confidence to Take Action bear out this emphasis?

CONCLUSION

The process described above is critical to improving aspirations at your school. You can make a difference when you take the time to listen, take what you have heard seriously, and take on the challenge of improving the teaching and learning environment for you and your students.

Summary:

  1. Data in and of itself is merely data.
  2. This data represents respondents’ perceptions of reality.
  3. These are baseline numbers – a starting place.
  4. The survey measures the 8 Conditions, which are ideals, not norms.