In This Issue:

1. Will multiple observers improve the quality of teacher evaluations?

2. The role of classroom discussion in high-level learning

3. Making later high-school start times work

4. Generating truly useful data in a school library

5. Short items: (a) Measures of America; (b) Monitoring student participation in discussions

 

Quotes of the Week

“[T]eachers who believed that administrators did not spend enough time in their classrooms questioned the validity of their evaluators’ assessment, doubted that the appraisal reflected an understanding of their daily work, and complained that the evaluation process lacked credibility.”

            Stefanie Reinhorn (quoted in item #1)

 

“We have learned that when you enact change, you need to have a powerful ‘why’ first.”

Teton County Superintendent Pamela Shea (quoted in item #3)

 

“Student work talks. It may talk louder, more eloquently, and more authentically than test results, and it could be our most important and most overlooked data.”

            Joyce Kasman Valenza (see item #4)

 

“Anyone who’s ever tried to rouse a high-school student from bed to catch that 6:55 a.m. bus to arrive at school on time knows how tenaciously that teen will cling to the bedcovers.”

Merri Rosenberg (see item #3)

 

“With varying degrees of frequency, all higher-achieving countries slowed down instruction at some point in some lessons to ensure that students had rich opportunities to learn – time to grapple with the key mathematics ideas and connect them.”

            Ronald Gallimore, James Hiebert, and Bradley Ermeling (see item #2)

 

“The pathway to improvement lies not in the increased use of a single compelling instructional method, but in building of a full repertoire of effective methods, and a nuanced understanding of how and when each will propel a teacher’s students toward ambitious learning goals.”

            Ronald Gallimore, James Hiebert, and Bradley Ermeling (ibid.)

 

 

 

 

 

 

 

 

1. Will Multiple Observers Improve the Quality of Teacher Evaluations?

            In this paper from the Carnegie Foundation for the Advancement of Learning, researcher Taylor White cites three reasons for the increase in districts having more than one observer visiting each teacher. First, a number of states now require that teachers be formally observed more than once a year, which increases the workload on principals beyond what many can manage in their busy schedules. Second, expectations have increased for using teacher evaluation as a tool to improve teaching and learning, raising the bar for administrators who were able to get by with superficial evaluations in the past. And third, there’s the perennial challenge of principals with relatively narrow teaching and subject-area expertise giving effective feedback to teachers in a number of grades and content areas. “By adding more eyes to these evaluations,” says White, “districts aim not only to relieve principals but, more important, to lend new perspectives, deeper expertise, and greater objectivity to the evaluation process.”

Using multiple observers seems like a logical solution to these challenges – more classroom observers, different areas of expertise, and a reduction in the demands being placed on principals. And indeed, the recent Gates-funded Measures of Effective Teaching (MET) study endorsed the notion that teacher evaluations are more reliable with multiple observers.

            Principals’ very challenging workload is a key issue, and there is a tendency for corners to be cut when requirements escalate – increased frequency and length of classroom visits and more-elaborate, high-quality feedback to teachers. In a recent study, Stefanie Reinhorn of Harvard University found that “teachers who believed that administrators did not spend enough time in their classrooms questioned the validity of their evaluators’ assessment, doubted that the appraisal reflected an understanding of their daily work, and complained that the evaluation process lacked credibility.”

White studied 16 districts of varying size and composition – from New York City to Transylvania County, North Carolina – and found that the use of multiple classroom observers varied along three dimensions:

• Which teachers get multiple observers – Given limited resources, districts decided which teachers would have more than one rater based on what policymakers hoped to accomplish and what resources were available. Some districts focused on teachers who were on track to get the highest or lowest ratings – those deserving recognition or sanction. Some districts focused on teachers with less-effective track records. For example, Washington, DC decided that teachers with five consecutive years of Highly Effective or Effective ratings would be observed only by their school administrators. Other districts used multiple observations only with novice teachers or those with problematic performance.

• Who does the observing – Some districts used only principals, assistant principals, and central-office staff (visiting classrooms together or at different times); some used other expert observers, including master and mentor teachers. Collective bargaining agreements forbidding teachers to evaluate union colleagues played a part in these decisions. Some districts used “third-party validators” from outside the district to monitor the validity of observers’ impressions.

• How raters are deployed – Districts used a variety of strategies here, including having only principals and assistant principals be multiple observers, having an assistant superintendent come in to rate marginal teachers, and using master and mentor teachers who were released from some portion of their classroom responsibilities. The challenge was getting a second (or third) opinion and yet having the primary evaluator see enough of teachers’ work to give meaningful feedback and a fair evaluation. Only one district (Greene County, Tennessee) had observers regularly visiting classrooms in tandem and discussing their observations afterward. This district reported greater objectivity in scoring, a broadening of expertise in giving feedback, and helpful professional development for principals.

White goes on to list several of the challenges districts encountered using multiple observers:

-   Inter-rater reliability – Training observers is the challenge here, since high-stakes ratings need to be consistent, common, precise, and reliable. “Teachers lose faith in the system when scores are obviously out of sync,” says White.

-   Cost – This depends on the district’s design: Hillsborough County, Florida spent $11.9 million on its evaluation system in 2011-12, with more than 85 percent for its multi-rater system. Other districts spent less – for example, Santa Fe, New Mexico used existing administrators at no additional cost.

-   Staffing considerations – Contractual and state policies limited some districts in who could be involved in classroom visits. Chicago was unable to use department heads and master teachers because of union push-back, and Washington, DC’s “master educators” and instructional coaches are not permitted to freely share any details about their interactions with specific teachers.

-   Human complexities – Many principals were uncomfortable when outside “experts” came onto their turf. “But hiring raters from within may also introduce complex interpersonal dynamics,” says White, “as teacher leaders adjust to their new authority and redefine relationships with former colleagues.”

-   Logistics and coherence – Additional raters require careful coordination – scheduling, monitoring, collecting and storing observations, and time for raters to calibrate their scoring practices to provide consistent, coherent feedback. “A small number of districts mentioned deliberate efforts to track and coordinate the feedback teachers receive from various raters,” says White, “but few had clear, established systems to ensure that feedback was consistent and coherent.”

In short, concludes White, implementing multiple raters “is often easier said than done.” Some districts mitigated these challenges by using online applications to aggregate and store data from raters and allowing teachers access to the information. Some districts hired full-time staff to manage the process. Others counted on the new wave of classroom observers to improve principals’ “eye” for instruction and skills at giving feedback (although some principals felt “policed” by their new colleagues).

            [This paper provides a helpful analysis of the possibilities and challenges of multiple classroom observers. However, I have four concerns: (a) The main problem with traditional teacher evaluations is not that they are “drive-bys” but that they are infrequent, announced in advance (the “dog-and-pony show”), and usually require a detailed, formal report that takes several hours to complete (making frequent visits and informal coaching virtually impossible).

(b) State and district policies increasing the number of required annual observations do indeed create an impossible workload for principals – but only if the length of observations and the amount of administrative paperwork remains the same. Frequent, short, unannounced visits with prompt face-to-face feedback to each teacher take less time and are likelier to improve teaching and learning, which is the core mission of teacher evaluation.

(c) If observers are required to score (that is, numerically evaluate) each classroom observation, inter-rater reliability becomes the main issue and helpful coaching of teachers won’t happen. In addition, monitoring quality and reliability becomes much more difficult since observers are mostly seeing different classes at different times, impeding comparisons of the data. The solution is to use evaluation rubrics the way they were designed to be used – as an end-of-year summation of multiple inputs during the year, including at least ten brief classroom visits per teacher, brief coaching conversations each time, observing teacher teams as they analyze student work, and looking at unit and lesson plans and perhaps student perception surveys. The only exception is chronically ineffective teachers, who get a more formal process, an improvement plan, lots of support, and a tight timeline for progress.

(d) As White documents, union, logistical, and human problems pose major challenges for multiple-observer systems as part of the official evaluation process. While it’s often helpful for teachers to get feedback from more than one person, and for principals to get the perspective of those who have more specific expertise, the rating officer in virtually all schools will still be the principal, assistant principal, or department head. Superintendents need to focus less on the elusive goal of inter-rater reliability and more on the quality of their front-line administrators’ evaluations. The best way to do that is for district leaders to make brief, unannounced classroom visits with their school administrators and check on the quality of their instructional “eye,” their judgment deciding what’s most important in each classroom, and their ability to deliver helpful feedback to teachers.         K.M.]

 

“Adding Eyes: The Rise, Rewards, and Risks of Multi-Rater Teacher Observation Systems” by Taylor White, Carnegie Foundation for the Advancement of Learning, December 2014; http://bit.ly/1yBOUW6

Back to page one

 

2. The Role of Classroom Discussions in High-Level Learning

            In this Teachers College Record article, Ronald Gallimore (UCLA), James Hiebert (University of Delaware), and Bradley Ermeling (Pearson Research and Innovation Network) question the “clarion call” of many education reformers for teachers to increase the amount of “rich classroom discussion.” A key assumption behind this goal is that good discussions produce higher-quality student thinking, expression, and achievement. Reformers’ aim is to replace traditional teacher-dominated discussions with “students talking in class about their ideas, asking questions of peers, engaging in debates with peers, explaining their reasoning, and sharing some roles traditionally assumed by teachers.”

            This push seems justified in light of sobering statistics about classroom interactions over the years. A 1912 study found that teachers across all grades and subjects talked an average of 64 percent of the time, and nearly 80 percent of classroom discourse involved rote memory or superficial comprehension. And apparently things have not changed much since then: A 1997 study of 9^th-grade ELA classes found that 85 percent of classroom time was lecture, recitation, and seatwork.

The curious thing is that, according to the 1999 TIMSS Video Study of Mathematics, countries with better student achievement than the U.S. are not better when it comes to the amount of rich classroom discussions. If anything, American classrooms have more opportunities for student talk than those in higher-achieving countries. So are rich classroom discussions not the key variable in classrooms that foster truly impressive learning? Gallimore, Hiebert, and Ermeling believe this is a good question and suspect that something deeper is involved.

            “What distinguished higher-achieving countries from the U.S. was the nature of learning opportunities provided to students,” they say. “With varying degrees of frequency, all higher-achieving countries slowed down instruction at some point in some lessons to ensure that students had rich opportunities to learn – time to grapple with the key mathematics ideas and connect them.” These moments seemed to be the “critical and significant distinction between the U.S. and higher-achieving countries.”

The authors have come to believe that these slowed-down, concentrated, students-grappling moments are the goal, and rich classroom discussion is one tool among many others for making them happen. In other words, principals and other supervisors shouldn’t be looking so much for rich classroom discussions in classrooms as for those rich learning opportunities that “nurture the advanced competencies and more-ambitious achievements U.S. reformers seek.”

John Dewey was not against teachers “telling” students important information, say the authors. But Dewey warned that students are unlikely to hear things they are unprepared to hear. How do we get students to the point where they hear what we’re telling them? This question need lots of study, say Gallimore, Hiebert, and Ermeling, and it’s closely tied to another one: How do we create rich opportunities to learn advanced concepts? They identify three key elements:

-   High-quality, sustained collaboration between teachers and researchers;

-   Studying curriculum and classroom practice to pinpoint the key learning opportunities in each unit that might benefit from rich classroom discussions;

-   Helping teachers as they get better and better at managing complex decisions and making the best instructional choices.

The authors believe rich classroom discussion is much more likely to be productive if it’s seen as a means to that end, rather than an end in itself. This, they say, “seems a more realistic vision than advocating for its greater use without respect for why, when, and for whom.”

“The pathway to improvement,” conclude Gallimore, Hiebert, and Ermeling, “lies not in the increased use of a single compelling instructional method, but in building of a full repertoire of effective methods, and a nuanced understanding of how and when each will propel a teacher’s students toward ambitious learning goals.”

 

“Rich Classroom Discussion: One Way to Get Rich Learning” by Ronald Gallimore, James Hiebert, and Bradley Ermeling in Teachers College Record, October 9, 2014; to purchase this article, go to http://www.tcrecord.org/content.asp?contentid=17714; Gallimore can be reached at [email protected].

Back to page one

 

3. Making Later High-School Start Times Work

            “Anyone who’s ever tried to rouse a high-school student from bed to catch that 6:55 a.m. bus to arrive at school on time knows how tenaciously that teen will cling to the bedcovers,” says Merri Rosenberg in this School Administrator article. “Pity the teacher who has to instill complicated algebraic concepts at 7:30 in the morning or discuss the subtler points of the American Revolution during that groggy first-period class.” Influenced by an ongoing stream of research, the American Academy of Pediatrics said in August 2014 that school shouldn’t start earlier than 8:30 for adolescents. And indeed, about 1,000 of the nation’s 18,000 high schools have moved their start times forward or are planning to do so, including the 187,000-student Fairfax County district in Virginia.

            What exactly does the research tell us? In a sidebar article, Kyla Wahlstrom, director of the Center for Applied Research and Educational Improvement at the University of Minnesota/ Minneapolis, says the most important finding is the “sleep phase shift” that takes place during adolescence, making it difficult for teens to fall asleep before 11:00 (this is aside from their fascination with their peers and electronics). Despite a strong body of research pointing to the wisdom of later school start-times, the issue has been contentious in many districts, with concerns about bus schedules, increased costs, after-school athletics, after-school jobs, and teens having less time to care for younger siblings after school. Some districts have tried to win over opponents by making incremental changes in start times – for example, moving from 7:30 to 7:50, or phasing in later times over several years. But Wahlstrom has found that “every change, no matter how small in scale or limited in the amount of the time shift, caused the same amount of community disruption.” Better to go for broke, she says.

“The districts making the most significant moves to later high-school start times tend to be the places that carefully and completely gathered and discussed all the factual information now available,” reports Wahlstrom. “The medical research about the development of the teen brain and the role of sleep in academic learning, healthy choices, and emotional well-being is so strong that it is difficult to ignore those facts… Individuals still skeptical are generally those who are unaware of the strength of research or those who choose not to believe the facts.” Wahlstrom says research has shown that later dismissal times (say 3:30) have had no negative impact on the level of participation in sports and extracurricular activities. Some coaches, after initially objecting to later times, have found that students are more mentally alert with more sleep. Nor do most teens stay up later because they don’t have to get up so early in the morning.

From districts that have made a successful shift to later secondary-school hours, here are some problem-solving suggestions:

-   To address transportation challenges, some districts have flipped secondary and elementary bus tiers, resulting in little or no budgetary impact. This does necessitate additional parental supervision of younger students waiting for buses in the dark during the winter months.

-   For some parents, earlier elementary start times mean they can drop kids at school on the way to work rather than first taking them to day care.

-   Some districts have increased the number of buses, which is costly.

-   Another approach is creatively rethinking bus routes (hub-and-spoke or combining multiple ages and schools on the same bus) and coordinating with the city transit system.

-   To accommodate parent work schedules, some districts have added or expanded after-school programs and partnered with day-care providers.

Rosenberg goes on to profile three districts that have successfully shifted secondary-school start times forward:

• Edina, Minnesota – Nineteen years ago, Edina became the first U.S. district to move its secondary-school hours. After a six-month study of research and ramifications (the strongest pushback came from athletic coaches and local employers who had students working part-time right after school), the district changed the middle school opening time to 7:50 and high school to 8:30. When there are out-of-town games with districts with earlier dismissal times, teachers allow student athletes to leave early. “The kids were on board right away,” says then-superintendent Kenneth Dragseth, and parents saw the benefit of having older students around the house to watch younger children in the morning. The district has maintained the later hours ever since and is proud of having made a decision based on what was best for student learning.

• Columbia, Missouri – The opening of a new high school in this spread-out 18,000-student district prompted a discussion of start times in 2012-13. A committee consisting of the school-board president, deputy superintendent for transportation, principals, teachers, and parents made its recommendations and the board decided on an 8:50 opening time for the high school and earlier hours for younger students, with some having bus pick-up times as early as 6:30. Because of extensive outreach and communication with parents, resistance has eased, and student attendance K-12 has increased from 85 to 90 percent among all demographics.

• Jackson, Wyoming – The district explored later hours in 2006-7 but encountered such strong resistance that the idea was tabled. But chronic student tardiness and parent complaints about the struggle to get kids in gear in the morning resurfaced the issue. “We have learned that when you enact change, you need to have a powerful ‘why’ first,” says Teton County Superintendent Pamela Shea. “It has to be very purposeful and address a need. You have to engage communities. People want to know and be involved.” The district ended up moving the high-school kick-off to 8:55, one of the latest in the country, mostly to ensure that kindergarten students weren’t waiting for buses too early. The initial cost was $250,000, including new buses and drivers and revamping bus routes. The result: 220 fewer tardy students and a 70 percent reduction in car accidents involving teens. In three years, the district will review the results and decide if changes are needed.

 

“Clearing the Snooze Hurdles” by Merri Rosenberg and “Creative Solutions for the ‘Sleep Phase Shift’” by Kyla Wahlstrom in School Administrator, January 2015 (Vol. 72, #1, p. 14-20), www.aasa.org; Rosenberg and Wahlstrom can be reached at [email protected] and [email protected].

Back to page one

 

4. Generating Truly Useful Data in a School Library

            In this article in Knowledge Quest, Joyce Kasman Valenza (Rutgers University) describes the “output measures” she once collected as a school librarian: books checked out, student visits to the library, reference questions answered, classes taught, and teacher collaboration. Valenza compiled the data in colorful charts and professional-looking reports to justify her budget, maintain staffing, and advocate for the program. But the process troubled her. Not only did it encourage “systemic competitive cheating” – it didn’t tell about the most important events in her library for students and colleagues:

-   Selecting quality sources;

-   Asking good questions;

-   Synthesizing information;

-   Ethically and creatively constructing and communicating new knowledge;

-   Addressing administrators’ achievement concerns;

-   How engaged teachers were in the library’s resources.

“I missed the connection between data and results and lost sight of essential questions,” says Valenza. “How does my work make a difference in improving teaching and learning? What is my value to the learning culture? How might I use evidence to improve my practice and enhance learning?”

            To get feedback on how she was doing on these core outcomes, Valenza began to conduct focus groups of 6-10 seniors. Among the questions she asked: What have you learned about finding information? Which databases have been most useful? How do you know when you’ve found a quality source? What have you learned about communicating what you learned during your research? What did you use on our library website and what improvements would you suggest? What parts of the research process do you feel are the most challenging? Do you feel ready for university research? What will you miss most about our library? What is the one thing you most wish you could have changed about our library?

            End-of-year interviews are useful, but Valenza has found that ongoing, formative evidence-gathering is better for making immediate improvements in a library program. Here are her suggested approaches, with tech tools for each (her article has a much longer list of tools):

            • Exit slips – Here are some prompts to get quick reactions from students on index cards or sticky notes (or using clickers) at the end of a class or project:

-   Share three takeaways from today’s class.

-   What did you or your group accomplish this period?

-   What were the best sources you discovered today?

-   Which criteria did you use to evaluate the resources you selected today?

-   How might you apply what you learned today to a situation outside school?

-   I didn’t understand…

-   I would like to learn more about…

Sorting through exit slips can provide immediate insights about challenges faced by an entire class or by specific students – and ways to fix them in subsequent classes. Valenza recommends several paper-free exit ticket products, including Exitticket http://exitticket.org, allowing real-time feedback and performance metrics, and responses can be shared with multiple teachers.

            • In-depth surveys – For some issues (plagiarism, for example), an anonymous online survey is the best way to gather information and decide what action to take. Google Forms https://docs.google.com/forms and Survey Monkey www.surveymonkey.com work well.

            • Examining student work – “Student work talks,” says Valenza. “It may talk louder, more eloquently, and more authentically than test results, and it could be our most important and most overlooked data.” It tells about the performance of individual students and groups of students, the clarity of instructions and rubrics, and the effectiveness of instruction – and students’ prospects on high-stakes assessments. And if 20 percent of students didn’t meet our objectives, it gives us clear direction on what to do next. Some possible focus areas: students’ ability to discern point of view, identify compelling evidence, find relevant high-quality sources, construct an argument, introduce and embed quotes, synthesize resources, and organize information. Google Classroom www.google.com/edu/classroom simplifies creating, collecting, and sharing student work.

            • Transparent instruction and planning – Google Docs and Google Sheets allow librarians and teachers to collaborate on units and lessons in remote locations and then, after trying them out with students, ask What worked? and What can we do better next time? and make edits. Many librarians are also using blogs to share their experiences, student projects, and reflections on their practice. This Pinterest board gives access to several library bloggers:

www.pinterest.com/joycevalenza/teacher-librarian-bloggers-and-other-blogging-frie.

            • Analytics – Tools like LibGuides http://help.springshare.com/lgstats can help

librarians gather data and identify strengths and areas for growth – and which investments are

paying off in measurable student learning.

            • Using a camera – Still and video cameras can capture images of library programs, displays, author visits, other events, and how the space is being used. Student photographers can help, and images can be shared via Flickr, Pinterest, and YouTube. The app Hyperlapse http://hyperlapse.instagram.com allows time-lapse filming of activity in a library.

            • Displaying and sharing evidence of practice – Digital storytelling, an interactive poster, or an infographic can get the word out. Valenza recommends several platforms, including Thinklink www.thinklink.com, Infogr.am http://infogr.am, and Piktochart http://piktochart.com.

            • Bad news is not bad – “Not all evidence is going to be pretty,” says Valenza. “Exit interviews, for me, were sometimes profoundly disappointing… Bad news presents a baseline, a realistic situation upon which you build a plan of action and grow as a professional.”

 

“Evolving with Evidence: Leveraging New Tools for EBP” by Joyce Kasman Valenza in Knowledge Quest, January/February 2015 (Vol. 43, #3, p. 36-43), www.ala.org/aasl; Valenza can be reached at [email protected].

Back to page one

 

5. Short Items:

            a. Measure of America – Check out this link for a wide range of maps, charts, and data on the U.S.: http://www.measureofamerica.org/

Back to page one

 

            b. Monitoring student participation – This app allows a teacher to quantify discussion-hogging and under-participation and do something about it:

https://itunes.apple.com/us/app/meeting-monopolizer-monitor/id463775303?mt=8

Back to page one

 

           

 

 

 

 

 

© Copyright 2015 Marshall Memo LLC

 

Do you have feedback? Is anything missing?

If you have comments or suggestions, if you saw an article or web item in the last week

 that you think should have been summarized, or if you would like to suggest

additional publications that should be covered by the Marshall Memo,

please e-mail: [email protected]

 

 

About the Marshall Memo