By Stuart Singer, The Teacher Leader

The key to creating an illusion is to distract the viewer.  Draw the attention away from the sleight of hand and the audience will believe they have seen magic.  Apparently this technique is now a key element when dealing with the analysis of standardized test results.  The plan appears to be to use some new and often outrageous assertion to distract the public and “abracadabra” many of the problems inherent in the No Child Left Behind (NCLB) and end-of-course standardized testing magically disappear. 

The latest form of deception is an idea being floated by educational leaders in Virginia.  They are considering a proclamation announcing that a “pass advanced” on the state’s Standards of Learning (SOL) exams is an indication of college preparation.  The word “advanced” in most contexts indicates a significant level of accomplishment; in this case, however, it should not be confused with readiness for post high school study. 

It is actually more of a numbers game

At first glance a “pass advanced” might appear to be a significant achievement.  It requires a score of 500 or more on a test scaled to go from 200-600 (400 is required for passing).  But as in other forms of magic, these numbers are an illusion.  Like many other end-of-course exams being used for NCLB, the SOL is a four-option multiple-choice test with no penalties for wrong answers.  Consequently the actual range of scores is not nearly as large.  The laws of probability decree anyone answering 50 such questions would start with 12 or 13 correct responses simply by random guessing.  In 2011 a passing score of 400 on the Algebra 1 exam required 23 correct answers.   As a result of that scale, every student begins with a score of at least 340.  Thus the real possible range is 340-600.  Suddenly a tally of 500 does not seem quite so “advanced”. 

Forty-four correct answers will earn a student a 500.   Even if this were an exam with open-ended questions and penalties for wrong responses, mastery of only 88% of the curriculum is hardly college-level work.   But with a multiple-choice, no penalty format, 44 accurate responses represent much less.  A few quick calculations reveal that if a student can answer 42 questions, probability will produce the missing two from the remaining eight.  Now the mastery level is down to 84%--a “B-“on most grading scales.  Even those numbers are a bit skewed.  If an individual can eliminate one or two potential answers in a question the likelihood of a successful “guess” increases exponentially. 

Adding to these misperceptions is the limited nature of such forms of questions.  They cannot require multiple-step responses or demand a true demonstration of mastery of the most complex or intricate aspects of a subject.  They can only ask questions that have reasonably simple answers.  It quickly becomes clear that based on almost any analysis, a pass “advanced” on these tests is not a predictor of college success. 

To get quality, you need quality

As Mel Riddile discussed in a previous post, tests made on the cheap are susceptible to both cheating and inflated results.   If Virginia and other states want to administer tests that are indicators of future educational success, they will need to move away from the current easy-to-grade and inexpensive formats and invest in exams that will accurately measure a student’s mastery of a class.  Until then, any claim of academic prowess based on the results is nothing more than an illusion and distraction.

Posted by Mel Riddile at 06:41 AM | Permalink

by Stuart Singer, The Teacher Leader

In a recent post, Mel Riddile highlighted several of the shortcomings associated with the latest method for determining graduation rates mandated by the U.S. Department of Education.  While the goal of this policy is worthy—standardizing the calculations throughout the nation—the resulting process is flawed, unfair and ultimately inaccurate.  The two key problems are an arbitrary time frame and an unrealistic as well as potentially punitive accounting for transfer students. 

Simple does not equal accurate

The ultimate goal of a graduation rate should be to measure a school’s effectiveness in producing successful students.  One of the reasons why the accuracy of this statistic has been so elusive is that it depends on so many variables.   In attempting to simplify the process, the Department of Education has built a method of measurement that will make for easy computations but misleading results.  What this overhaul ignores is that schools with high rates of poverty, mobility and ELL populations will be evaluated on information that does not truly reflect the academic performance occurring in their building. Dr. Riddile explained the problems with this new system from the perspective of a principal.  Here are three stories to illustrate the view from inside the classroom.

Easy come, easy go

In schools with high mobility the percentage of students who spend their entire academic careers at the same location can be surprisingly small.  My former school had an annual mobility rate of about 30% meaning that approximately one of every three students left during a typical year.  But during that time the size of the student body was extremely stable.   With remarkable consistency virtually every individual who moved to a new district or returned to their home country would be offset by someone coming in the opposite direction.  I once taught an introductory ELL course, Individual Math 1, which began the year with 24 students and finished with the same number.  Only one-half of those students were enrolled for the entire course.  This level of movement is not a reflection of the quality of education being provided; rather, it reveals the nature of the student body.  Consequently, under the new Department of Education guidelines while the actual instruction in that classroom may have been excellent, the dropout rate of these ninth-graders bordered on 50%.   Clearly labeling unverified transfer students as dropouts places an unfair burden on such schools.   The alternative, precise tracking of all of these individuals as mandated by this plan, would be far too costly in terms of time and energy. 

A tale of two graduates

Maria was in my Algebra 1 class.  She was smart, hardworking, reliable and the mother of a one-year old son.  Her boyfriend (soon to be her husband) and the baby lived with her family while she completed high school.  During her freshman year she made a detour to give birth but when she returned the next year there was no request for special treatment.  She passed my class and all of her other classes.   After five years at the school marched across the stage at graduation.  When I last heard from her she was an assistant manager at a bank.  While the data may label her as a drop-out, I can promise you she was not.

Michelle, my second example, was a quiet and petite member of my Pre-calculus class.   A refugee from Viet Nam she did not look like a 22-year-old.   But the path to her senior year of high school was very different than most.  What should have been her middle school years were spent either on a boat or in a prisoner of war camp.  When she finally arrived in this country she had the life experiences of an adult and the education of a child.  As a 17-year-old “freshman” she had to overcome language and emotional difficulties few others could even imagine.  Despite these immense challenges she succeeded in a college preparatory curriculum to earn a diploma as she approached her 23^rd birthday.  Though her story was far from traditional it deserved positive recognition by any legitimate measure.

Numbers cannot measure everything

As a mathematician, I would love to be able to state that every human endeavor can be measured quantitatively.  Unfortunately such precision is not always possible.  Even baseball, the most data driven sport on the planet, cannot evaluate simply on numbers. Every pitch, every swing of the bat, every spit is recorded, calculated and documented.  But despite this wealth of statistical data there are players in baseball who need to be seen to be appreciated.  There are no precise metrics to measure breaking up a double play, diving to catch balls, sprinting on every play or simply being unselfish. 

Formulating a simple yet accurate device for determining high school graduation rates is equally difficult.  What many of the educational policy makers may not understand is that the diversity of a student body is not just about ethnicity, religion or socioeconomic status.  It also includes both the time required to graduate and the stability of their residency.  This most recent attempt by the Department Education is clearly not the answer. 

Posted by Mel Riddile at 11:25 AM | Permalink

by Stuart Singer, The Teacher Leader

Evaluations that have value

Academic success has been directly linked to the quality of a school’s teaching and administrative staffs.  Ideas are constantly being presented for improving educational personnel.  But singular approaches such as merit pay, improved evaluations, and easier termination policies will not individually accomplish that goal.  What is needed is a multi-faceted program that will address all of the shortcomings in the current system. Previously, eight fundamental areas were presented that need to be addressed in order for districts to be able to hire, improve and retain the most talented educators.    

This is the fourth in a series of articles that will detail the steps needed to implement those improvements.  In earlier posts hiring practices and effective interviewing techniques have been discussed. Part 4 will focus on the evaluation process.

Changing more than the format

Nearly every proposal for improving teacher evaluation revolves around the use of data particularly standardized tests.  While that is an important item for consideration, the most pressing reform may well involve redefining who is best qualified to actually perform those evaluations.  The first step in true evaluation reform should be to create a new set of individuals who are responsible for this portion of the process. 

Not the best choice for the job

In most school districts the primary responsibility for this process falls to the local administrative team and there are many assistant principals who are well equipped to review the work of a staff member.  But even though the vast majority undertake this time-consuming task with a high level of commitment, they have an overwhelming array of day-to-day responsibilities. For example, the AP who evaluated the math teachers at my former school was tasked with the discipline of more than three hundred freshmen, administering (and evaluating) three different departments, implementing the testing plan for state barrier exams, hall duty between change of classes, supervising several extra-curricular and athletic events and interviewing candidates for vacancies.  In addition there were the almost daily emergencies that always arise in a high school.  A majority of administrators have similar job descriptions.  It is not surprising to find the time to evaluate teachers falling toward the bottom of this lengthy “to-do” list. 

Creating professional evaluators

A better option for school districts is to train a group of master teachers to become full-time, system-wide professional evaluators.  Such individuals would be significantly better equipped to accurately assess the skill of an instructor than administrators who in all likelihood were not hired primarily for their evaluation skills.  The money required for forming such a cadre could be offset by a reduction in administrative staff and an improved evaluation process.

An even more important argument for this innovation revolves around fairness and consistency.  School-based evaluators cannot help but be somewhat biased by their daily interaction with the staff.  Within a building there is an intuitive sense of which individuals perform well and which perform poorly in the classroom.  These reputations are rarely based on quantitative data; they are the result of comments by students, parents, faculty and other subjective experiences.  Minimally these unsubstantiated ideas can influence the amount of effort spent on an evaluation.  If pressed for time, an argument could be made to shorten the observation of a teacher who is widely “recognized” as being outstanding.  Conversely, negative sentiments from the school community can result in closer scrutiny of less well respected educators.  In either case a measure of fairness is compromised. 

A team of district-based evaluators would eliminate this problem and would also create consistency throughout the system.  The evaluations at school A could be compared with confidence to those at school B.  A number of issues ranging from merit pay, transfers and the termination of contracts could be resolved more reliably.      

What would the process contain?

Here is one hypothetical fix for the ineffective and unproductive teacher evaluations that are sadly typical.  Three professionals would form a teacher’s evaluation team.  One (generalist) would be a highly trained observer who is thoroughly versed in the fundamentals of good teaching.  Another (curriculum specialist) would have similar training but would have taught in the subject field being observed.  The third would be an assistant principal at the school. The generalists and curriculum specialists would be required to have at least ten years of successful teaching experience in addition to extensive training in observing and interpreting classroom activities.  Successful retired teachers could be an outstanding and economical talent pool for these positions. 

The actual evaluation process would be intense.  At least five formal observations would be required.  The generalist and specialist would have two announced and two unannounced.  Additional observations can be done when necessary or desired.  A local assistant principal would perform one unannounced visit.  This experience would familiarize the AP with the teacher being evaluated.  The results of this observation would not be included in the final document but should allow the administrator to better understand and interpret the input from the other team members.  All observations would encompass an entire class period.  The two announced would be videotaped which would become a central component in post-observation conferences.  The videos would also be available to the other members of the team.  When appropriate, standardized test scores and other pertinent data would be included in the overall assessment.  Evaluators will, of course, be carefully trained in analyzing such data and how to utilize it in a fair, accurate manner.  At the end of the process, the three observers would meet and create the overall rating that would then be shared with the teacher.   The primary source of the final conclusions would be the two observers; the role of the AP would be to coordinate the process and provide further input if needed.

The next steps

The purpose of an evaluation should be to both determine the quality of one’s performance and to construct approaches to improve and enhance skills.  Ultimately it should also be a tool in determining pay, advancement and termination.  Those will be the focus of the next part of this series.  

Posted by Mel Riddile at 01:09 PM | Permalink

As measured by gains in student achievement, teachers, even experienced, mid-career classroom instructors, improved as a result of their participation in a formal evaluation process. An Education Week report cites two studies conducted in Cincinnati, which has an extended history using a formal evaluation system consisting of four formal observations. In Cincinnati, teachers are given a rating relating three standards: classroom practices, classroom management, and questioning and discussion techniques.

In addition to discovering that 1. Teacher participation in a formal evaluation process improved student achievement, the first study concluded the following:

2. Classroom Management Improves Math Performance

The study found that "while overall teaching practice was the best predictor of student achievement, classroom management was more highly correlated with better math performance."

3. Questioning Improves Reading

Teacher use of open-ended questions was more highly correlated with student performance than classroom management.

In a second study, also conducted in Cincinnati, student performance not only improved in the year that the mid-career teachers were being evaluated, but the improvement in student performance continued and even increased in the years following the evaluation.

It is important to note that in the Cincinnati evaluation system teachers are not evaluated annually and that the evaluation process does not use a value-added component. In addition, the Cincinnati teacher evaluation system connects to a career ladder for teachers, which may be a motivating factor.

Implications for School Leaders

- Teacher evaluation is not the most enjoyable part of a school leader's job. However, knowing that teacher evaluation improves student achievement makes the process more meaningful.

- If school leaders want to make a difference in student achievement, teacher evaluation is a no-cost way to do so.

- Teacher evaluation is most effective when the teachers are clear on what behaviors will be evaluated, and if both teachers and principals have had extensive, multi-year professional development. Cincinnati was chosen precisely because both factors were present. In addition, many states and districts are proposing similar evaluation models and Cincinnati's decade long experience should help inform future practice.

- Even more encouraging is the fact that student achievement continues to improve in the years following a meaningful evaluation process of even the most experienced teachers.

- It makes sense that questioning improves reading, but I wonder why classroom management makes a bigger difference in math classes. Perhaps the sequential nature of math demands continuous attention and student engagement--students miss out if they miss a step in the process.

Posted by Mel Riddile at 11:52 AM | Permalink