Teachers and Reflection

I wrote this in response to a thoughtful posting on another blog, The Speculum, where the author, Lennie Irvin, presents a draft of a pre-dissertation proposal. It's about writing and reflections, and presents interesting definitions of reflections.

-----------------

A practitioner's response:

I'm the retired (ha!) director of the Willamette University School of Education where I gave numerous assignment for student/student teachers to 'reflect' on either what they saw in a observation of another teacher or on their own practice as a student teacher. Our program was heavy in requiring reflective tasks. From that experience comes this caution - the reflective practices of your students may not be done under the same definition you are proposing. If I use hindsight to define reflection as used by the vast majority of our students, it was to describe what they saw or remembered, with a cautious evaluative conclusion.

Again in hindsight, I don't think the faculty had a clear definition of reflection, certainly not a common one, and I don't believe, based on the outcomes, that we make the task of reflection clear to the students.

As I 'reflected' on my career in teacher education (reviewing what I had done, what was effective/not effective, identifying a central theme to my practices, and what I would change were I to start over) - my post retirement definition that I wish I had used with students - I concluded that my efforts to improve their teaching skills through praise, criticism, and offering of various solutions to their teaching problems was largely ineffective. It wasn't until I took a more objective role in observation/evaluation that I found success, both in the depth of teacher reflection and in effective change in teaching practices.

I found that when I shifted from feedback in the form of anecdotal notes to providing objective data on what was happening in the classroom, the students/student teachers shifted from a defensive/deflective or accommodative response to one of independent reflection (definition above), problem solving, and change.

In my now failed retirement, and initially as a mental exercise, I developed the Data-Based Observation Method and then wrote a software program to support it. The software is not necessary, but make it much easier.

In the Data-Base Observation Method the focus of an observation is collaboratively determined between observer and observee - "What do you want to know about your classroom?". Guiding questions can be used to focus the process on categories such as content delivery, class management, student relations, etc. Once the focus has been determined, the observer will gather objective frequency and duration data on classroom behaviors of teacher and/or students.

Here's where the reflection come in. The data is provided to the teacher with these questions "Is this what you thought was happening in your classroom? Is a change needed? If so, what will you change?" When this external, objective picture of what had happened in their classroom (part one of my definition - reviewing what I had done) is presented, the teachers make the determination of whether the teaching/learning was effective/not effective (part two of the definition).

Professional level discussion ensues, based on the objective data, to determine if and/or what changes should be made (part four of the definition - part three, identifying a central theme, comes after a number of the data-based discussions).

There now has been five years of implementation of this method by me, school administrators, and peer coaches and I have found that it builds the skill of reflection in teachers, a skill I am coming to see as a critical element in becoming a long-term successful teacher. I believe that 'reflection' is shallow and surface when the person does not have the factual basis for understanding what occurred and when the reflective task is to be either self or externally evaluative in nature. The question is not "How did I do?" (Answer: good, poor, passed, a 6 out of 10, I liked it, etc), but "What happened and was it effective?"

Teacher Evaluations From The Teacher's Viewpoint

In a May, 2008 report by the Education Sector titled Waiting To Be Won Over there is an interesting and revealing question: exactly what do teachers think of formal evaluations. The graphic here illustrates a significant problem. In this chart 41% of the teachers viewed formal observations as a 'formality'. When I look at the evaluation forms and the standards they are based on, it's quite obvious why they would think so. The language in the standards/indicators and on the forms is very frequently so vague as to be unverifiable.

Only 26% viewed the evaluations as 'useful and effective' - just 1/4 of the teachers!

But what I found most interesting was the 32% that saw formal evaluations as 'well-intentioned but not helpful'. To me that indicates that both parties - administrators and teachers - had hoped that the evaluations would serve a good purpose, but were disappointed.

The reason for these disappointing numbers is twofold: the problematic language in standards and the subjective nature of most evaluations. To illustrate, here are three standard/indicator statements taken from a large district's teacher evaluation/professional development document:

"Students initiate and take responsibility for their own learning."  Not only is this a stretch to view this as within the responsibility and power of the teacher to invoke, but this is not a behavior that is observable through the series of classroom observations nor through a paper-trail of student work.  It's a wonderful value, but unusable as a basis for teacher evaluation.

"Teachers connect new knowledge to prior learning."  This can be observed by looking for statements or assignments that directly make the connection between a students prior knowledge and the lesson at hand. Data can be collected on both the teacher's statements and the student's statements and questions.

"Teachers make the lesson objectives and learning targets clear for students in advance of the lesson."  This too can be observed by tracking statements made by the teacher; questions, statements, and errors made by the students; and the response by the teacher to student errors. However, the phrase '...in advance of the lesson' make this a bit more narrow - this is a standard to be accomplished before the lesson begins, and does not account for the changes that need to be made during the typical lesson. Of course, making the goals clear before the task begins is very important, but there will also be a need for objectives and targets to be adjusted as a result of feedback during the lesson.

The key to all this is that the language in standards/indicators were initially written as general guidelines, and are now being used as the basis for evaluations. There is a serious need to revisit the standards and rethink/rewrite them into evidence that can actually be gathered. It's not fair or reasonable for either the administrator nor the teacher when the objectives and goals of their behavior isn't clear. 

With clear standards, written in observable terms, and objective tools for collecting the data, the subjective nature of the process is diminished. When the objective data is presented to teachers, I predict the numbers in both the 'formality' and 'well-intentioned, but....' categories will significantly decrease. 

Aligning Tools To Standards

I've been doing extensive work aligning (and creating) eCOVE tools that align with various district and state standards for teacher performance. As I create more tools, needed to provide objective data in line with the language in the standard/indicator, I am continually amazed at the uselessness of many of the standards.

It's not that the standard lacks value as a broad guideline, but they are often worded so that the interpretation of the standard can be so broad to lack any form of reliability -- or that the standard/indicator points at some internal value such as 'caring' for the students or 'believing' in the students.

An example from Danielson's latest version of Framework for Teaching: "Teacher's plans and practice reflect accurate understanding of prerequisite relationships among topics and concepts." What would one observe, in print or in a live situation, that would serve as evidence that the teacher understood a prerequisite relationship? And what knowledge and skills would the observer need to adequately judge this indicator for a variety of content areas? Nonetheless, I am sure that there are many evaluation sheets being checked on the basis of  'a feeling' or a vague interpretation of this language. No wonder 40% of teachers view observations as a formality.

The solution is not to rely (solely) on eCOVE to identify ways to gather data that might be conceived as supporting this indicator, but to REVISE the standards language. In fact, I'm setting a high priority for myself and eCOVE to create a set of standards that have observable behavior indicators attached. If you'd like to help, let me know. Stop by the website - www.ecove.net - and email or call me. If you have a set of standards that have been revised and have observable indicators, PLEASE contact me -- you'll be doing the profession a big service.

If we can't identify the evidence that would support the achievement of a standard, then the standard is useless at best, and frequently dangerous and misleading. When the standard lacks the possibility of a clear line of evidence, it can be interpreted and applied by each observer - and thus is no standard at all.

Self-Directed Professional Growth

I've been in numerous discussion with Teacher Leaders recently, and come across a common frustration. It seems that no matter how hard they try to convey that they are 'just there to help', there lingers a resistance and trust issue. When I did deeper, I find that the observation process is one of identifying 'good' practices, and sometimes 'bad' practices; or it's one of taking notes on 'what occurred during the observation'.

I think that when the process has, in any way,  judgment or valuing language involved there will be a defensive resistance. Even the notes are a problem as the choice of what to record is a judgment made by the observer.

Since this doesn't occur in the Data-Based Observation Method, it's really easy to get past the initial trust concerns - you just have to follow the system and prove that you are not there to hammer them with the data.

-------

Key concepts:   Don't Praise, Don't Criticize, Don't Provide Solutions!

-----

Follow this sequence of interaction:

Pre-conference: Centered around determining what data to collect - "What do you want to know about your classroom?". This an be in light of a teacher's individual goals, some teacher-perceived problem, or building/district/state standards.

-----

During the observation, gather data without making any comments reflecting praise, criticism, or solutions.

-----

In the post conference, ask these questions when presenting the data:

• Is this what you thought was happening in the classroom? (teacher reflection and interpretation)
  
• Do you think a change is indicated? (teacher and observer professional discussion about the interpretation of the data)
  
• If so, what will you change (teacher ownership and empowerment; enhanced professional discussion)
  
• How can I support you? (professional collaboration)
  
• When should follow-up data be collected to see if the change is effective? (making the entire process not one of pleasing the observer, but in implementing effective change)
  

This approach shifts the dynamic from defensiveness to empowerment, from judge to colleague. There is no observer, Teacher Leader or Administrator, who can solve every classroom problem. It's far better to develop the teachers' skills in reflection and problem solving. This can be accomplished by basing the discussions on data rather than opinion.

-----

I welcome comments about any of my thoughts.

Teacher Pay, Portfolios, and Best Practices

I just read an article in Ed Week about the Houston and Denver districts' efforts in teacher pay for performance. Both programs are broad implementations of the pay-for-performance system and are struggling with enrollment and acceptance. The most interesting quotes were by Gayle Fallon, President of the Houston Federation of Teachers. Both quotes, "It's better than last year. Still, they are handing out money and getting nothing in return."  and "What we hear from teachers consistently is that they have no clue what they did to get the money", point to the black-box nature of using student test scores as a primary determiner in awarding pay or other rewards. While student learning is the primary goal, the connection between the teacher's direct influence on student scores (as they indicate learning) is very difficult to determine. I can understand the teachers not 'having a clue' when the results of their efforts (the test scores) are calculated and revealed sometime in the future and those results also include the influence of a large number of other variables.

The Sherwood School district in Oregon has recently received a large grant from the Chalkboard Foundation to improve student learning. Part of their efforts include a bonus pay system based on a teacher portfolio of evidence, which can include student scores as well as other strong evidence of exemplary teaching and professional conduct. They contacted me to discuss the use of data-based observation data on best practices as a part of that process. We haven't finalized anything, but it sure makes sense to me (naturally).

We have credible research about teaching practices that result in increased student learning. We have a tool (eCOVE Software) that will track the implementation of those practices in an individual classroom. Now we have the opportunity to reward teachers who are implementing those researched best practices. The process is not difficult to manage - identify the behaviors that everyone is confident in as directly influencing student learning (Class Learning Time, Time on Task, Wait Time, Level of Questions {as answered by students, not just asked by the teacher}, etc, etc), train observers (teachers, aides, paid data gatherers, administrators) to competently use the data collection tools, and determine the appropriate data collection procedures (number of data points, length of individual data collection events, etc).

The result of this, I predict, will be interesting and engaging. Not only will teachers know immediately that they are using the researched best practices in their classroom, but they will have a running record of that. It's that running record that is the greatest benefit - it can provide feedback to the teacher who has a goal of becoming an exemplary teacher in a timely and useful manner. They can immediately see if they are moving toward a higher level of proficiency instead of waiting for months to find out if they 'won'. As I've said before, teachers are deeply dedicated to effectively teaching their student in the best manner possible. Bringing the objective feedback to the classroom level in real time will build more effective teachers; then we'll know why those scores went up as well have the 'clues' we need.

As a side note, I'm a bit concerned that the student performance/more money is a strong extrinsic motivator and will shift the focus on why one becomes/continues to be a teacher. I think the immediate, objective, and over time feedback that eCOVE provides will not only reinforce the skills of teaching  but will also reinforce the teacher's perception of their skills. Since we love doing what we do well, the data and teacher reflection become the intrinsic motivator. As teachers become/continue to be successful in their craft, and are clearly aware of their successes, they will keeping doing what they love - helping kids.

Walk-Throughs and Data-based Observations

Can data-based observations be used in walk-throughs? Yes, and doing so can greatly improve the usability and acceptability of the results. But first one must determine the purpose of the walk through. 

 

In the ASCD Journal Educational Leadership, Jan, 2008 issue (Vol 65, No 4) is a section "What Research Says About..." by Jane David, director of the Bay Area Research Group, and it's topic is Classroom Walk Throughs. She makes it clear that walk throughs are "...not intended to evaluate individual teachers or principals or even to identify them by name in postobservation reports. rather, the goals of walk-throughs are to help administrators and teachers learn more about instruction and to identify what training and support teachers need."

 

With that purpose in mind, focusing on a single behavior and gathering objective data as opposed to the checklists typically used will help relieve the fears about bias of the observer. Gathering the same data in all classrooms (Class Learning Time, Wait Time, Level of Question, etc) can present a much more useful picture of the instruction in the school, and provide the basis for professional discussions.

 

Checklists, even when they are focused on observed behavior, are nearly always a record of the observer's impression of what is happening in the classroom. For example, walk through checklists often include an item on student engagement, and the choices are some form of 'poor, good, great' or 'none, few, some, most, all'. The item is checked based on the observer's impression or estimate of the level of engagement.

 

However, it's quite simple to use a data collection tool to do time sampled data collection of each student's level of engagement by doing sweeps of the classroom and recording the on/off task behavior of each student at that instance. In a classroom of 30 students, a sweep can easily be done in 90 seconds; the total time for the recommended 5 sweeps is less than 8 minutes. The results are an objective picture of the on/off task behavior of the entire class. When this is repeated for the entire school, the results can form a basis for decision making regarding goal setting and teacher support AND serve as the basis for determining actual change over time.

A series of 'impressions' is of much less value than a series of data based observations.

 

Be sure that the teachers are clear regarding the use of the data (not for evaluation, no names recorded), and that they are involved in the interpretation of the results. La Paz Middle School in Salinas, CA organized a data-collection day (it actually took a couple of days) where teachers gathered data in each other's classrooms using five laptops. Since the task is not one of judging or ranking behavior, but simply gathering data the potential for engaging the full staff will go far toward building a professional community engaged in a common endeavor.

Observation checklists versus observation data

I've looked at a couple dozen books on how to conduct a classroom/teacher observations and have downloaded another 30 or so observation forms used by various districts across the country. I'm struck by the general nature of most of them, and by the lack of specifics in the descriptors. The rating scales used run from observed/not observed to met/not met to a 5 to 7 likert scale. Some of the forms used by districts are designed to record anecdotal notes without a rating scale and/or with the evaluation of the performance on another summary page.

An example: This 'standard' came from a school district set of guidelines for conducting observations: "Establishes and maintains an orderly and supportive environment for students", and is, in one form or another, pretty common in standards.

I just can't see how a checklist or scale is helpful, let alone accurate, as a record of what happened in the class. If the observer checks 'observed', does that mean the class was at one point orderly and supportive? Or does it mean that when things started to get disorderly, the teacher responded and brought the class back in focus? Or, since the phrase 'supportive environment' is also included, does that 'observed' indicate that the teacher made positive, encouraging statements? To all the students? Could the observer be satisfied with student work being posted on the walls, and a 'student of the week' bulletin board being present -- that's certainly supportive.

And if the class was orderly some of the time and not others, and the observer checks 'not observed', wouldn't the teacher respond with "Are you saying my class was never orderly? Or that I was never supportive? Or both?"

Observed/ not observed doesn't work. It does not convey any helpful information and will lead to conflict between the observer and observee.

So what about a scale? Scales are typically designed to be either a 1 to 5 (poor to great) or a rubric of 'unsatisfactory, basic, emerging, competent, distinguished' type. Some of them will have descriptors for each of the levels, the worst being the range from 'did not observe/ observed some of the time/ observed most of the time/ observed all of the time'. These descriptors are worthless as they are nearly impossible to mark in a way that conveys what happened. For example, if the class was orderly for the first 3 minutes and then in chaos the rest of the time, the orderly standard would actually be met 'some of the time'. Actually, if the class were orderly up to 49% of the time, the same checkbox would apply; and if things were orderly 51% to 99% of the time it would be 'most of the time'.

There are other descriptors or indicators for each of the levels that seem more specific. For example, Charlotte Danielson in her Framework for Teaching has a standard for Management of Instructional Groups with a proficient level of competence described as "Tasks for group work are organized, and groups are managed so most students are engaged at all times." In the many districts who use some variation of Danielson's work as their standards, the observer would be asked to judge if the teacher was at this level during the current observation.

Skip the fact that there are two behaviors in this indicator - organizing tasks and managing groups - which also confuses the issue, and just look at the act of making that determination of worth based on the second behavior. When you watch a typical classroom, the complexity in deciding if 'most' (is that 51% or is it really a higher target than that?) 'are engaged' (physically or mentally? engaged in low level or high level work?) 'at all times' (what would be the determination if the full class went off task for 3 minutes?), has such wide variation across classrooms and observers that the validity is suspect.

In discussions with administrators, what I find very often happens is that the observer adds additional criteria to the specific situation. 'Most students' sometimes turns into almost everyone in the class (a higher standard that stated); 'engaged' equates to looking and acting busy without regard for the level or quality of engagement; and 'at all times' is ignored in lieu of an unspoken criteria of 'most of the time'. If the class is known to have kids with behavior problems or the number of students is high, the criteria is functionally lowered.

What's really happening is that the observer has internally defined a level that is satisfactory to him or her based on personal experiences, the makeup of the class, and the relationship with the teacher, and that definition is applied unevenly across classrooms. That inconsistency is confusing to everyone, and the results of classroom observations can not be compiled across the building, let alone the district, as a basis for broad decisions. The system has a built in subjectivity and personal interpretation of the standards that makes it difficult for any observer to be consistent and fair.

Rating scales don't work, especially when extremely little effort is put into rater reliability and clear statement of the objectives and indicators.

Data based observations can make a significant difference. Some of the texts on classroom observations provide steps on how to turn a judgment on a rating scale into numbers and then process those numbers as if they were data -- but given all the issues with rating scales I believe this to be a false path.

Instead, I recommend using the Data-Based Observation Method, a 5 step process that includes the actual collection of observable behavior data. The steps are:

1. Identify the standards. Be sure that they are worded so that observable behaviors demonstrating those standards can be clearly identified.
Good standard: Students will be engaged in learning activities. Bad standard: Teachers will act in an ethical and professional manner at all times (what does 'ethical' look like?).

2. Create indicators. Be sure that they describe the observable behavior identified in the standard.
Good indicator: Students will listen attentively to the teacher, be productively engaged in individual work, or contribute to the work of a small group. Bad indicator: Students will follow teacher instructions as given (too vague and general).

3. Set criteria. As a profession, we've not engaged in setting criteria for ourselves in concrete terms, so this part is a new conversation. What is the criteria for engaging students in learning? Should they be engaged 25% of the time? No, that's clearly too low. How about 50% of the time -- still sounds low. What about 95% of the time? Too high for real classes? The answer here is not to set the criteria arbitralily, but to turn to (or conduct) research to establish a criteria in which we can have confidence.

If the standard is an important one, and the indicators are valid, there will be a correlation between the behavior observed (such as student engagement) and the final desired outcome (student learning). We need to find those connections and use them as guides for improving teaching. We actually have research that identifies a significant number of them, but we're not applying that research at the classroom level.

4. Design data collection tools. I developed the eCOVE Classroom Observation Software as an easy and efficient way to collect the objective data, but you can use pencil and paper, a stop watch, the wall clock in the classroom, etc. to collect the data once you've carefully identified what data is important to collect.
Good tool: A counter tracking on/off task behavior and using the time sample data collection method to record the percent of time engaged and the percent of time not engaged for the entire class.. By using the time sample data collection approach and repeated sweeps of the class to record the on/off task behavior of each student, a quite accurate data-based, objective picture of the class behavior is produced. This becomes a factual basis for making decisions. Other useful tools include Class Learning Time, Level of Questions, Teacher Talk/Student Talk, and other tools reflecting research on best teaching practices.

5. Analyze and interpret the data. Did it meet the criteria? Is there a need or desire for a change? Given the context (number and diversity of the students, physical space, materials at hand, etc) what is most likely to bring a positive change? When and where will the new approach be initiated? When will the next set of data be collected, analyzed, and interpreted?

For the greatest success, it's critical to operate with the belief that every vested interest be involved in this process. Administrators, teachers, parents, aides, students, counselors, etc. all have an important contribution to make where the purpose of the observation is the improvement of teaching and learning.

-----------

I'm coming to realize what a big shift this is in the education field. We've tried to cite 'professional judgment' when the inconsistency in the process and the unreliability of the results support neither the process nor the conclusions. Serious collaborative discussions are needed to move to a more concrete basis for judging what we say we value, and how to use the specifics to guide the improvement of teaching.

And where should we start......? Let me think about that -- I suspect
it's with a reexamination of the standards. More to come :-)

Powered by ScribeFire.