Questions on assessment (from my class)

1. Try to recall some of your strongest memories of exams and testing from your college days. What were your most positive evaluation experiences? Describe. Conversely, what were the most negative ones? Why?

Strongest negative memory: An upper division history course taught by a professor who received two PhDs from Oxford. We were all in awe of him. I studied like crazy, took lots of notes, prepared for the final (the only test) and made a B because I did not make personal statements about the topic. (I honestly didn’t understand it well enough to be able to do that, but if the exam was supposed to be grading my learning, then it was okay there. But I still hated it.)

Other negative memory: A Texas history professor who asked such nit-picky details that we described his test questions as “things like the name of the horse that the survivor of the Alamo rode to get away.”

Strongest positive memory: This was not of tests. My strongest positive memories were of research papers. I loved the diving into a subject, studying it, and immersing myself in it. Sometimes it would take me a while to get started, but I really did well once I chose a topic. My favorites were in recent European history and Latin American history. I made B’s in the courses, but I made high A’s on the research papers and I have re-used the information I learned in those papers for decades. They were very useful.

2. Review the courses you teach (course outline, syllabus, and textbooks), as well as the exams and assignments that contribute to the final course grade. Are all major topics and the course learning outcomes thoroughly evaluated? Are the subordinate competencies (specific and discrete) adequately evaluated?

I discovered that my final was not covering what it should have been covering in Brit Lit. I am going back over it today and for the next few days to recreate it.

I also may have to go over my whole syllabus and remove some works in order to give more room to higher level learning. (We can’t read fifty things in twenty days if we are supposed to be analyzing, synthesizing, and evaluating them.)

3. Do the graded assignments and examinations reflect the domains (cognitive, psychomotor, affective) and specific taxonomy levels listed in the course outline? In other words, are high-level learning outcomes and competencies evaluated? Furthermore, are psychomotor and affective domain mastery evaluated (or just cognitive)?

See answer above. But I think that I give a lot more lower-level grading than I would like to in order to ensure that the reading is actually done.

The papers are all higher level learning.

And I have already added two affective questions to this next final. I’m asking them to identify a work for each era which they feel would be the seminal work if we could only read one for that time period and to discuss why they think that, especially in light of the learning objectives and outcomes listed for the course.

4. Are there adequate evaluation opportunities in your courses? Are a variety of evaluation processes used?

Yes. All my classes have TONS of grading because I am going overboard to do the opposite of what Dr. Spencer did… Maybe I need to stop doing quite so much grading because I am getting a little behind on that.

5. What does the phrase “teaching to the test” mean to you?

It means that you only teach what is on the test and you teach nothing else. If the test is something like the NClex, for nursing, then you must do that because the test evaluates high level learning by asking the student to choose the best answer… It means, though, that there is not a lot of leeway in what you teach.

College teachers usually create their own tests. They know what is on them and they should certainly make the information available to the students through discussions, readings, lectures, activities, and assignments.

But students often expect a review handout that covers every single question on a test, rather than types of questions or types of information. If the teacher says that is what the review handout is for, then clearly it should do that. However I often see students not studying, but memorizing the review handout. Then, when they go to a test, they are upset because the question, as presented in the review, was not on the exam. That’s ridiculous. Why should you need to come to class if the review handout is all you need to know for the exam?

In addition I have upon rare occasions had a teacher who taught only what was on the test and nothing else. Each class period consisted of covering ten or fifteen minutes’ worth of material which was boiled down to a single question on the exam. And that was all the class consisted of. I am opposed to this approach.

Thankfully I am in English and for freshman composition, my most common course, the tests of all kinds are writing. Since the focus of the course is writing and the students are writing and the tests are writing, there is a clear confluence of testing and teaching coming together. (At least that is the goal.)

In my British Literature course, I primarily use out of class papers, where the students have to integrate what they learned in class and what they have found or discovered on their own. I do have two exams, though, which cover factual materials in a paragraph form. In those exams the students must write about the work addressing a certain question.

Because the course is involved and we do lots of reading, I allow the students to use their notes. They cannot use the book, but they can use anything that they created. This does two things. It encourages reading as we go along (because there is no way they can do all the reading before the exams) and it encourages them to take extensive notes on my lectures, our discussions, and the texts themselves.

Some teachers might feel that they ought to remember the issues on their own, but I feel that the way I have created the test allows me much more flexibility and gives the students a greater likelihood of doing well.

I don’t expect them to have memorized Beowulf or Paradise Lost. I don’t expect them to remember every description of the characters in Gulliver’s Travels. But I do expect them to have a general grasp of the work (both for the test and afterwards) and to be able to find the more specific information (either in their notes for the test, or on the internet throughout their life).

In addition, for the final, I ask the students to create questions based on the information we have covered in the course. I usually use at least two of those and if they come up with a question I was already planning on, I let them know that as well. …In a class with twenty questions, I will say, “Four of these will be on the test. Two of them were already on the test and two of them have been chosen from your suggestions.” I do not, though, tell them which those are.

This was an answer I gave to a related question before we had really read about or talked about assessment.

I still like my final; I’m just no longer sure that it is testing what it ought to be testing.

6. How do you feel about “high stakes” end of course tests?

Because of my experience with the history course, I do not like them.

Because of my experience with the Regents finals in New York high schools, I love them.

I think that the concept of teaching to the test is mostly, though not entirely, a function of the accountability process put into place in our K-12 system. Most people who teach to the test in this environment are doing so with the expectation of teaching their students the fundamental knowledge which is necessary for them to pass on to their next stage of learning.

Sometimes this is essential and useful. What if there were no understanding of end outcomes for a course/class/year/school? Then each one would be different and a student from one would not have learned or even studied the same things that a student from another did. We would not have anything like a basic level that could be assumed within education.

And this can be useful for a teacher as well. I went to high school in New York, where every student in a course must take the statewide final exam in their course. It did several things for the teachers.

First, it removed the onus of “it was too hard for X” because it is a statewide requirement that you know how to prove that a line has 180 degrees (or whatever).

Second, the final was useful because it gave the teachers a clear set of objectives to be aiming for. World history wasn’t just supposed to talk about the history of a single country outside of the US, but was supposed to cover art and architecture and politics throughout world cultures. European history wasn’t just modern or early, but covered everything from the Etruscans forward.

Third, it allowed teachers a bit of flexibility in grading. This was not mandated (like it presently is in Dallas or Pittsburgh), but a student not doing well in a course, but TRYING, could be given a low passing grade with the clear understanding that the student would not pass the course if they could not pass the final. That was a statewide requirement. So a teacher could reward effort of a student without doing social promotion or effort promotion for the entire course.

7. Do you feel there is a conscious effort to tie assessments to objectives?

You know, I do. Even after having reviewed my final for Brit Lit and having found it wanting, I still think so.

I wrote the Brit Lit final based on what I wanted.

I wanted the students to read and remember the works. I wanted them to take notes. I wanted them to know the different types of literature we covered.

All that is basic learning that doesn’t always happen in a classroom. So it is what I was testing for.

However, the learning outcomes from the syllabus, as given by the department, don’t ask for knowledge level information.

8. Do you have an thoughts on course and instructor assessments?

Of course.

I like them all right, even though the people who write the most are usually the ones who are the most unhappy.

9. How can informal assessments inform our teaching?

As described by the class facilitator, not through much. I did give a quiz over a lecture to see if they were listening. (I don’t know if they were. I haven’t yet taken the time to grade it.)

I would love to hear what you would answer to some or all of these questions. Which ones struck the strongest chord with you?

Informal Assessments, and how I use them in my classroom

For the class I am taking, I had to bring in an example of informal assessments. There was no discussion of this in the reading or on the boards, so I was not sure what the facilitator was actually looking for. I went online, surfing for possible explanations and found a great introduction to informal assessments, both structured and unstructured. (Note: While I have not been able to find the original for a while, I did find it. Directions for how to find it are at the end of the post.)

Writing samples – When students write anything on specific topics, their products can be scored by using one of the techniques described in Table 3. Other creative writing samples that can be used to assess student progress include newspapers, newsletters, collages, graffiti walls, scripts for a play, and language experience stories.

Homework – Any written work students do alone, either in class or in the home, can be gathered and used to assess student progress. With teacher guidance, students can participate in diagnosing and remediating their own errors. In addition, students’ interests, abilities, and efforts can be monitored across time.

Logs or journals – An individual method of writing. Teachers can review on a daily, weekly, or quarterly basis to determine how students are perceiving their learning processes as well as shaping their ideas and strengths for more formal writing which occurs in other activities.

These are, I think, the most commonly used in English classes.

I have multiple papers, homework, and journal assignments. Most of the homework is graded from a “did all,” “did most,” “did some,” “did none” point of view. But the journal assignments range from very particular grading of grammar and content and following directions to a general “turned in” 100.

These are not, according to the class leader, informal assessments because they are graded. So I am not sure what that means for me or for my class.

The only writing samples I don’t grade in some form or fashion are for one freshman writing course at SLAC. There the first class period is a writing topic so that you can ascertain whether the student should be in that course or needs to move into remedial English.

Other than those, I grade everything because I liked having grades when I was in college.

Games – Games can provide students with a challenging method for increasing their skills in various areas such as math, spelling, naming categories of objects/people, and so on.

Debates – Students’ oral work can be evaluated informally in debates by assessing their oral presentation skills in terms of their ability to understand concepts and present them to others in an orderly fashion.

Brainstorming – This technique can be used successfully with all ages of children to determine what may already be known about a particular topic. Students often feel free to participate because there is no criticism or judgment.

These are things I use less often in class, but I don’t grade them or they are only a participation grade.

I am working on a Jeopardy game for grammar. Someone else has one and I want to work on it.

One of my sets of classes is going to have a debate over a controversial issue as part of the preparation for their writing. The good thing about that is that they will hear other people’s points of view. The bad point of that is all the writing will be on the same topic.

Brainstorming is something I use to get my students started thinking. Yesterday, for example, we spent most of the class time doing prewriting. I asked a question “list things you know about” and then I put my own on the board (to give them suggestions) while they wrote their own down. These included TV shows I watch, movies I love, books I read, and music I listen to. Then I said, “list friends, family, restaurants, meals, dates.” I was trying to get them to think of a lot of different ways to think of things which they could use for a compare/contrast paper. We spent the whole class period writing and they came up with two pages of ideas.

Story retelling – This technique can be used in either oral or written formats. It provides information on a wide range of language-based abilities. Recall is part of retelling, but teachers can use it to determine whether children understood the point of the story and what problems children have in organizing the elements of the story into a coherent whole. This also can be used to share cultural heritage when children are asked to retell a story in class that is part of their family heritage.

Anecdotal– This method can be used by teachers to record behaviors and students’ progress. These comments can include behavioral, emotional, and academic information. For instance, “Jaime sat for five minutes before beginning his assignment.” These should be written carefully, avoiding judgmental words.

Naturalistic – Related to anecdotal records, this type of observation may take the form of notes written at the end of the day by a teacher. They may record what occurred on the playground, in the classroom, among students, or may just reflect the general classroom atmosphere.

Story retelling in my classroom is the narrative paper. We do some verbal discussions, too, to get ready for the writing. (The students get in triads and tell short versions of the story they are going to write.)

The other two assessments are only used when there is a problem in the classroom or with a student.

Checklists – Checklists specify student behaviors or products expected during progression through the curriculum. The items on the checklist may be content area objectives. A checklist is considered to be a type of observational technique. Because observers check only the presence or absence of the behavior or product, checklists generally are reliable and relatively easy to use. Used over time, checklists can document students’ rate and degree of accomplishment within the curriculum.

Cloze tests – Cloze tests are composed of text from which words have been deleted randomly. Students fill in the blanks based on their comprehension of the context of the passage. The procedure is intended to provide a measure of reading comprehension.

Criterion-referenced tests – Criterion-referenced tests are sometimes included as a type of informal assessment. This type of test is tied directly to instructional objectives, measures progress through the curriculum and can be used for specific instructional planning. In order for the test to reflect a particular curriculum, criterion-referenced tests often are developed locally by teachers or a school district. Student performance is evaluated relative to mastery of the objectives, with a minimum performance level being used to define mastery.

I use checklists with my students for peer review projects.

I have used cloze tests in my high school biology classroom, but have not used it in my English classroom. However, I am considering re-writing an exam for my Brit Lit I course and this is a possibility for that.

I don’t do the criterion tests. My assignments match the criteria and so I don’t feel the need for that.

Rating scales – This is an assessment technique often associated with observation of student work or behaviors. Rather than recording the “presence” or “absence” of a behavior or skill, the observer subjectively rates each item according to some dimension of interest. For example, students might be rated on how proficient they are on different elements of an oral presentation to the class. Each element may be rated on a 1 to 5 scale, with 5 representing the highest level of proficiency.

Questionnaires – A questionnaire is a self-report assessment device on which students can provide information about areas of interest to the teacher…. For a questionnaire to provide accurate information, students must be able to read the items, have the information to respond to the items, and have the writing skills to respond.

I have occasionally used rating scales to identify potential problem areas. It’s more like a list of things they need to be doing in the class (attendance, prompt arrival, on-time homework, etc) and a rating scale there.

I don’t know that I have ever used questionnaires before, but I may do so this semester to elicit a kind of pretest body of information.

The original article, Informal Assessment in Educational Evaluation (written by Cecilia Navarete, Judith Wilde, Chris Nelson, Robert Martinez, and Gary Hargett) has a lot more information including:
Scoring Assessments for Unstructured Activities
Combining Assessments for Evaluation (with a very detailed discussion of portfolios)
and others… The article focuses on bilingual education, but it made a lot of sense to me for English education in general.

Update: The original article is no longer available on the same page. It is, however, available from NCELA as a PDF. The original article is the first link.

What IS in a grade? What do they mean?

The answer to that question these days is often, “Not much.”

What’s in a grade? (You’ll have to scroll down.) has some interesting thoughts on grade inflation and the problems with it.

They also have quite a bit on Texas in that article.

Turning all sorts of “Fs” (actually, they’re “Es” in Pittsburgh, presumably because they’re loath to further dishearten students with reality) into a standard 50 is unabashed grade inflation under the dubious guise of giving struggling students another chance. The resulting skewed metric will unduly reward them for subpar work. Dallas tried this F-is-always-a-50 scheme last year and teachers ultimately begged to have it reversed.

Also in Texas, meanwhile, the Higher Education Coordinating Board (which, although it is mainly concerned with Texan tertiary education, controls the secondary GPA question too, as related to Texas college admissions policies) is proposing to exclude from GPA calculations all subjects save English, math, science, social studies and foreign languages. What’s more, only courses that count for college credit (International Baccalaureate, Advanced Placement and dual enrollment) will be awarded extra points in GPA calculations (currently, honors classes are given an extra half point, too). The purpose of these negotiations is to standardize college admission standards across the state. Not surprisingly, the proposal, which will go up for a vote on October 23, has generated quite an uproar. Parents and teachers alike are concerned that students will opt out of electives and rarely show up or participate in required music, physical education, and art classes.

It’s not difficult to see that Texas may be going too far. Limiting which courses qualify for heavier GPA weighting is one thing but not counting music, art and PE at all is another. Teenagers will be teenagers; tell them a class doesn’t count and lower enrollment, attendance, and effort will surely follow. We’ve already sidelined these subjects with standardized testing and NCLB, and budget crunches have led some districts to cut them. Enough. Let’s not diminish them further with perverse incentives. Non-academic pursuits have been shown to lower drop-out rates, especially amongst at-risk teens. And extracurricular activities, often inspired by exposure in these non-academic classes, may be the secret to the (moderate) success of our public education system.