Showing posts with label value-added assessment. Show all posts
Showing posts with label value-added assessment. Show all posts

Tuesday, January 24, 2012

Can a Few Years’ Data Reveal Bad Teachers?

This from the New York Times:

With years of data, it seems possible to distinguish good teachers from poor ones. Does that indicate that, after collecting two or three years’ data on each new hire, districts should be using test scores for decisions about firings, tenure and pay?

The following in an online "debate."

The Value of Test Scores

Raj Chetty Raj Chetty and John N. Friedman, economists at Harvard, and Jonah E. Rockoff of Columbia co-wrote the recent study "The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood."
John N. Friedman
We all agree that teachers can make a tremendous difference in the lives of students, and we all can remember a great teacher who was important in our own lives. The challenge is to identify more great teachers. Value-added measures, which rate teachers based on their impacts on students’ test scores, can help us do so. Our recent study shows that when a high-value-added teacher enters a school, test scores for students in the grade taught by that teacher rise immediately (as shown in the figure below). And the gains don’t stop there: the students who learn from that teacher are more likely to attend college, earn more, and are less likely to have children as teenagers. Even when new teachers are evaluated with just a few years of data, those who get high value-added ratings produce large gains for their students...

Let’s Not Rush Into Value-Added Evaluations

Jesse RothsteinJesse Rothstein is an associate professor of public policy and economics at the University of California, Berkeley. He has studied the relationship between classroom assignments and estimates of "value-added" by teachers.

The new Harvard-Columbia study provides important information about the relationship between student test scores and longer-run outcomes. But there is much that we still don’t understand. We need careful study of pilot programs, not to remake our education system.
One unheralded new result is that teachers’ effectiveness changes over time. The study shows that some teachers are ineffective at first but improve as they age, while others start strong but then burn out. Policy design must account for this. How many teachers who are fired early on for poor student achievement would improve given the chance? And how many who get early raises will continue to draw them through years of later coasting? Calculations that firing a poor teacher saves $2.5 million entirely ignore this factor...
  

Dawn ShirkDawn Shirk teaches English as a second language at Reidsville Middle School and Reidsville High School in Reidsville, N.C.

Traditionally, teachers have been observed by their principal once a year, and evaluated solely on that one encounter. Long-time teachers would often go years without having an observation, or even a casual walk-through by an administrator. Fortunately, times are changing.
In North Carolina, we are in our second year of using a new teacher evaluation system, developed by Mid-continent Research for Education and Leadership. Through a series of rubrics, teachers are evaluated in five areas — leadership, respect for diversity, understanding the content they teach, the lessons they create and execute, and their reflection on their own teaching. Sub-topics include use of technology, global awareness, relationships with parents, and the use of data from standardized tests to drive instruction. It is a yearlong process that includes several observations, by principals and peers, conferences between administrators and teachers, and a collection of artifacts that teachers present to demonstrate the many things they do that may not be seen in an observation, which includes parent conferences, teacher collaboration, working with community organizations, professional development, committee work and much more...

We Know Which Teachers to Fire

Lance T. Izumi is the senior director of education studies at the Pacific Research Institute.

Lance T. IzumiA study of 2.5 million students by Harvard and Columbia researchers strongly indicates that individual teacher quality based on student test scores significantly impacts students’ life outcomes. While that should be enough evidence for school districts to consider using testing data to help inform firing, tenure and pay policies, the key will be overcoming teacher union opposition.
Analyzing two decades of test results, the researchers found that improving teacher impact on test scores, referred to as “value added,” even in a single grade raises the probability of students attending college and increases their future earnings. The data also suggests that getting rid of the lowest-performing teachers would significantly raise students’ lifetime incomes. Yet, many teacher unions are likely to be unmoved by these findings...

Results Are In; How Will We Respond?

Arun Ramanathan is the executive director of The Education Trust—West.
Arun Ramanathan

There is plenty of strong evidence that we can use data to assess the impact of teachers on student outcomes. In a recent report, The Education Trust—West found that on average, students placed with the strongest teachers gained half a year more in English than students placed with the least effective teachers.
Differences of this magnitude are important for all students, even our highest achievers. But they can be devastating for low-income students and students of color who often enter school already behind. With a series of effective teachers, struggling students can quickly catch up to grade level. But one or two weak teachers can prevent them from reaching their academic potential.

Unfortunately, our research revealed that African-American, Latino and low-income students are far less likely to have access to the best teachers. Just as worrisome, high-need students can lose access to highly effective teachers as the result of quality-blind layoffs based solely on seniority. No wonder we have failed to close the achievement gap...

Use the Data, but Constructively

Sydney MorrisSydney Morris is a former public school teacher and the co-founder of Educators 4 Excellence, a national nonprofit that seeks to elevate the voices of teachers in education policy.

The recent study by Harvard and Columbia economists showed a link between quality teaching and higher test scores and between higher test scores and positive life outcomes. Researchers found that students with top teachers are less likely to become pregnant as teenagers, more likely to enroll in college, and more likely to earn more money as adults. These new findings highlight something that we as teachers have always intuitively known – that what we do everyday in our classrooms has far-reaching impact.  
 
Despite this knowledge, a false dichotomy exists between proponents and opponents of using student-growth data to evaluate teachers. We often hear of the “reformers” who want to use student test scores to identify and fire the lowest-performing teachers, and conversely, the teachers’ unions who are painted as defenders of the status quo.  
Lost in this back and forth are the voices of real classroom teachers who want meaningful evaluations that give them the feedback and support they need to improve their craft. In Educators 4 Excellence’s work with nearly 4,000 teachers in Los Angeles and New York City – the nation’s two largest school districts – we have consistently heard from educators that they believe their students’ academic growth should be one, among many, indicators of their performance...

Value-added evaluation of teachers - yet more problems

This from the Daily Kos:

The default assumption in the value-added literature
is that teacher effects are a fixed construct that is
independent of the context of teaching (e.g., types of
courses, student demographic compositions in a class,
and so on) and stable across time. Our empirical exploration
of teacher effectiveness rankings across different courses
and years suggested that this assumption
is not consistent with reality.
In particular, the fact that an individual student’s
learning gain is heavily dependent upon who else is
in his or her class, apart from the teacher,
raises questions about our ability to isolate a
teacher’s effect on an individual student’s learning,
no matter how sophisticated the statistical model might be.

Those words are from a new study on the stability of teachers scores using Value-Added methodologies toascertain teachers effects upon students scores on tests.

The study in question, titled Value-Added Modeling of Teacher Effectiveness: An Exploration of Stability across Models and Contexts, was released late last month at Education Policy Analysis Archives, a peer-reviewed online journal of education policy previously edited by our own SDorn, Sherman Dorn of the University of South Florida (disclosure - and like me an alumnus of Haverford College), who is still on the editorial board.  Two of the four authors of this study, Edward Haertel and Linda Darling-Hammond, are also among the authors of the EPI policy brief.

Abstract:
Recent policy interest in tying student learning to teacher evaluation has led to growing use of value-added methods for assessing student learning gains linked to individual teachers. VAM analyses rely on complex assumptions about the roles of schools, multiple teachers, student aptitudes and efforts, homes and families in producing measured student learning gains. This article reports on analyses that examine the stability of high school teacher effectiveness rankings across differing conditions. We find that judgments of teacher effectiveness for a given teacher can vary substantially across statistical models, classes taught, and years. Furthermore, student characteristics can impact teacher rankings, sometimes dramatically, even when such characteristics have been previously controlled statistically in the value-added model. A teacher who teaches less advantaged students in a given course or year typically receives lower effectiveness ratings than the same teacher teaching more advantaged students in a different course or year. Models that fail to take student demographics into account further disadvantage teachers serving large numbers of low-income, limited English proficient, or lower-tracked students. We examine a number of potential reasons for these findings, and we conclude that caution should be exercised in using student achievement gains and value-added methods to assess teachers’ effectiveness, especially when the stakes are high.

In other words: 
We find that judgments of teacher effectiveness for a given teacher can vary substantially across statistical models, classes taught, and years.   The key words are vary substantially -  if we are talking about something that is basic, it should not vary that much because of the particular value-added methodology being used.  For point of comparison, where I do measure you by yardstick, tape measure, using at one time inches and another centimeters, if I am really measuring the same thing (height) I would get consistent results (which perhaps some slight variation due to measurement error).

But the method of measuring is only one problem.  If there are different results because of classes taught, that MAY be because of different levels of effectiveness in different curricula, or it could be something else.  And if there is substantial variance from year to year, either the teacher is very inconsistent (which does not seem all that likely) or that variance is due to something not under control of the teacher.

Furthermore, student characteristics can impact teacher rankings, sometimes dramatically, even when such characteristics have been previously controlled statistically in the value-added model. -  this is CRITICAL.  People have justifiably argued that using the single score at the end of a year tells us little about what the teacher has done, and may be primarily due to knowledge with which students arrived.  Value-added analysis is supposed to provide a method that allows us to control for different characteristics among students, and thus enable use to focus in on what impact the teachers actually had.  But if despite our attempts to control for variance among students, that variance still seriously impacts the derived value-added score, then relying upon value-added approaches is dangerous, because we will be making decisions that cannot be justified by the data we are using...