|
Transcript of NTA Conference Call Presentation held on 11/19/96 Developing Skills Standards, Academic Standards and Accountability Systems that Include ALL Youth, including Youth with Disabilities: THE OREGON STORYPresented by Oregon Department of Education |
![NTA [logo link]](../../images/nta.gif){kind=small} |
[The tape began in the middle of Nancy's opening. We've reconstructed the first sentence, only.]
Nancy Hargis (Oregon): We're really excited in Oregon because we anticipate that in December, our State Board of Education will adopt 12th grade benchmarks in the following areas: English, Math, Science, Social Studies, Second Language and the Arts. (Afternote: On December 19th, the State Board of Education did adopt these benchmarks.) It will include a career related learning standard that would be assessed at the 12th grade level. We're also part of a multi-state consortium looking at portable credentials across state lines in the area of business and management. So we're sort of on the brink, poised and ready to move ahead.
We had an interesting presentation this morning before our Senate Interim Education Committee. It became real clear that the questions that policy makers have--about moving to standards-based systems and assessment of students--are addressed best, I think, by having some real examples of something they know. One of the concerns that was raised was the high stakes nature of any one assessment event (i.e., if a student didn't succeed would they be flunked, could they graduate or what options would they have). By making it clear that students have multiple opportunities and multiple avenues, I think we were able to make people feel much more comfortable about this kind of really significant change.
With that as our introductory remarks, I have two colleagues with me this morning, Wayne Newberger who is our Assistant Superintendent for the Office of Assessment and Pat Almond, who is a specialist in the Assessment Office, who's recently moved from our Office of Special Education to the Assessment group. I think that that, in and of itself, demonstrates some of our state's real commitment to including All students in the assessment process and to trying to meet their individual needs. So with that, I'm going to turn this over to Wayne and let him talk about some of the details as far as where we're headed and how we hope to get there.
Wayne Newberger (Oregon): I'm going to give you an overview of our program, then I'm going to have Pat talk more specifically about what we've been doing. We are in a period of transition; we're in the process of really trying to establish our criteria for the inclusion of All students in our assessment, and for students who may be exempted, to make sure that it's for the right reasons. I think what I'm going to do is turn this over to Pat who's been working with a committee in the state here to help us identify those criteria and actually do some research to help us determine which procedures are most appropriate. So Pat, why don't you take it.
Pat Almond (Oregon): Hi, this is Pat Almond and I can tell you a little bit about the context. We have the Accommodations Panel, which has been meeting for about a year and a half. It's represented by teachers, administrators around the state, university folks, and Department of Education state people. We've been looking at a number of topics, but one of the things that we had as a context was the state assessment, the way it is proceeding. Wayne had referred to the move that is taking place to move to high stakes for an individual student. Up until recently, we've been focusing primarily on accountability in a state system, and now starting in 1998-99, decisions about whether or not individual students attain a certificate of mastery will be based on the state assessment. The other thing that is important to note about Oregon's assessment is that its Criterion Reference Test is a power test and doesn't have strictly enforced timed elements to it. So it has some flexibility for the kinds of accommodations that would allow students to really show what they know.
What I'm going to go through right now is some of the foundations that we've come to. One thing I wanted to add is that the state test, when it's in full force, will include a multiple choice component and a performance assessment component that is an on-demand performance assessment. In addition, the evidence for students will be Work Samples. So there is an array of evidence provided showing that the student has attained mastery. Within that context, we've been trying to figure out how to ensure that All students can participate. We started by talking about accommodations, knowing that some of them would not be able to take the test under the regular conditions. So we've named a set of goals about the accommodations and what their purpose is (see attachment):
So, we're walking a line between ensuring that the disability doesn't interfere with the student's performance, and not overly advantaging a student so that they don't really meet the high standards. We've been working with the Council of Chief State School Officers who has the group working on special education issues, and with the National Center for Educational Outcomes. Through that work, we got access to some definitions and we decided that really in order to carry on our conversation and to talk meaningfully, we needed to make a distinction between what we meant by an accommodation and what we meant by a modification. Again this is looking at the testing issues related to demonstrating mastery. There's a term offered by Colorado in 1995 that was reported in a terminology handout that seemed applicable to us. It's a broad umbrella term, which says,
"An adaptation is a change made to the environment, curriculum, instruction, and/or assessment practices in order for a student to be a successful learner. Adaptations include accommodations and modifications. Adaptations are based on individual student's strengths, needs and may vary in intensity and degree."
The important thing is that an adaptation became an umbrella term and then within that we wanted to frame what accommodations meant. So for an accommodation, we have (see attachment):
"...an alteration in how a test is presented to or responded to by the person tested. It includes a variety of alterations in presentation format, response format, setting in which the test is taken, timing or scheduling. The alterations do not substantially change level, content, or performance criteria. The changes are made in order to provide a student equal access to learning and equal opportunity to demonstrate what is known."
This was very helpful to us in moving toward an understanding of how to use results that include accommodations, as distinguished from modifications. Here is our working or operating definition of a modification, it's probably not a final definition (see attachment):
"Modification: Substantial changes in what the student is expected to learn and/or demonstrate. The changes are made to provide a student opportunities to participate meaningfully and productively in learning experiences and environments. They include changes in instructional level of content and performance criteria."
Early in this work the concern was that we would lower the standards for some students, in other words not maintaining high standards for All. So this definition of a modification is really clarifying that there may be significant changes in the contents itself. In our test for the multiple choice component, we have then three conditions under which the student participates in order to try to get a full level of participation accounted for. They include:
In order to make this move into 1998-99, when we're making individual decisions about students, we started looking at specific accommodations and trying to distinguish which ones would be allowable under a Standard administration, and which ones actually constitute a significant change. So I have included in the handout the preliminary draft of something that we're using for the multiple choice component of our test. It talks about the kinds of accommodations that would be allowed; the student's score would be a standard score and would be reported in the aggregate along with school, district, and state averages. Now this lists some that are currently considered standard, to give you an idea of what those might be like (see attachment):
"Extended time, frequent breaks, dividing the test over several sessions, finding the directions, clarifying directions, using a large print version of the test, putting the writing and math problems on tape, allowing the students to dictate the multiple choice responses to a scribe, testing an individual student in a separate room, allowing calculators and a spelling dictionary."
One of the things that we've been doing with our research is trying to set the criteria for what accommodations would constitute "standard" and when we have a modification that it's believed would actually alter or change the content or the difficulty level, to develop a set of research procedures around that to determine whether or not it actually does constitute that kind of change. So some of the things in one category could shift from Standard to Modified or from Modified to Standard, pending for the research. But, a Braille version of mathematics right now is thought to be a question mark; we think that it probably makes sense, in a common sense way, that a blind student could take a math test in Braille, but we need to do some content analysis around the graphics, just to make sure that we're testing the same content. The writing test, if a student cannot write, so they dictate their writing response to a scribe, right now we don't have enough information about that to call it a Standard Administration, because it is a process of writing. So these are some things that would be called Modified Administrations in order to implement immediately during the current year. One other thing that is important to note is that this is a process and as we learn more, we get more clear about the definitions and about how to use this information.
I'm sort of thinking that, before I talk anymore, it might be helpful to have some reactions and to have people talk, because I feel like we're sharing a work in progress, and certainly we're both interested in telling you how we came to our thinking on certain things and also to learn how other states are doing that.
Wayne Newberger (Oregon): Maybe one other thing that we might mention before getting into that is one of the things that Pat did last spring was to conduct a series of focus group meetings around the state that included teachers, administrators, special education directors, and so on, to find out exactly what the current practice is around the state. Much to our chagrin, I don't if it was necessarily to our surprise, we found that there was just a wide variation in how different people were handling a testing when it came to deciding on how students were going to be tested and who is going to be included and who is not going to be included. So we recognized that there had to be a lot better agents [?] and also a lot better training of teachers for this to be consistent, and particular for kids on IEP that the IEP process had to be brought into the discussion about what kind of testing procedures those students would engage in. And that had to be done well before the time of testing, because you can't decide the day before testing how you're going to administer this test to a student. There needs to be a fair amount of planning that goes into it. Probably other states are much better at this than we are, I don't know, but we knew we had a lot of work ahead of us when we did the study.
Mary Mack (NTN/NTA): This is Mary Mack. One of the difficulties, I think, is the local versus the state role. So what you're trying to do in Oregon is to develop a state-wide system that would be used in all localities?
Wayne Newberger (Oregon): That's correct.
Pat Almond (Oregon): Mary, one of the things I can say about our system is that with three levels of evidence--multiple choice test, an on-demand performance test, and work samples--that the state test itself is comprised of the multiple choice and the on-demand performance. There's a set of evidence that will be brought forward by the local district where the tasks and the scoring will be a lot more in the hands of the districts, local education agencies, and teachers themselves. So its kind of both. We've got to have a state system that everybody subscribes to, and then there's a lot of opportunities for evidence coming from the local area.
Judy (California): Hi, this is Judy from California, and I'm wondering if you've been trying this with students at this point, or if this is still just setting up the structure?
Pat Almond (Oregon): The state test, for quite some time, has had the ability to test students in smaller groups, over an extended period of time, or by breaking the testing session into multiple sessions. Also, the Standard administration for All students is that a math item can be read to a student if they are having trouble with the reading component. So some pieces of this have been in place for awhile. We've also been engaged in some research to try to determine whether or not the accommodation or modification actually affects the content or performance. For instance, last spring we administered tests to about 1,000 individual students. There were about 500 in writing (to see the effects of word processing on the writing performance) and about 500 in reading and math, multiple choice (to see the effect of reading the math items, and to see what happened if you allow students, fourth graders, to mark in their test booklet instead of on the bubble sheet). So we've been questioning specific accommodations and we're going to do more case study in single subject design work this year to try to tease out the effects of these accommodations. I would say we're doing both, some are already in place and some are being investigated, and in some cases, a student will require a modification just in order to participate. What we're hoping is that the ability to use a modification will increase participation, that more and more teachers in schools will have their students participate because they can use a modification as a part of the participation.
Margo (Ohio): This is Margo from Ohio. Pat, do you now have data that would indicate about what percentage of students with disabilities are taking the state test?
Pat Almond (Oregon): Yeah, I don't know that we have data that agrees completely. The look I've taken at it suggests that if you take the entire population of students with IEPs, around 10 or 11% of the public school student enrollment. About half the kids are taking the test. It looks like we administer state tests at 3rd, 5th, 8th and 10th [grades], though 3rd grade isn't that high.
Margo (Ohio): So, of the 50% taking it, what percentage are passing?
Pat Almond (Oregon): Well, we're right in the middle of changing our standards. So, I don't think I can tell you that for sure. It's not 100%.
Wayne Newberger (Oregon): We can tell you that for kids who are Title 1, they're portion is about half of the proportion of the regular population. So I don't know how that would compare with special education.
Pat Almond (Oregon): I think its less. I think that maybe 20 or 30% are achieving the standard. Again, we're talking about a Criterion Reference Test and a standard that was just recently adopted by the board. It's a pretty high standard.
Margo (Ohio): Okay, thanks. What percentage of kids without disabilities are achieving this standard? If 20-30% of the 50% of kids with disabilities who are taking it are achieving the standard, what percentage of kids without disabilities are achieving the standard?
Wayne Newberger (Oregon): That varies by grade level. At grade three, we add on the three subjects, reading, math and writing. It's about 50% of the kids in math, and close to 70% in reading, and in writing it's about 60%. That's probably the highest grade level. It kind of goes down from then. The tenth grade level, we have I think it's 32% who can meet our standard in math, it's a little less than 50% in reading, but it's a little higher in the writing, it goes up to around 67% who can meet our standard. So, this is the first year that we've applied that standard to our data. We anticipate that it will go up, but right now there's a lot of kids that can't meet it.
Margo (Ohio): Well, I'm just curious. Is anybody concerned, with the emphasis on testing and proficiency testing, that now we have more teachers teaching to pass the test, instead of teaching for transition and the skills needed to navigate employment and adult life?
Anonymous (Vermont): Yes, this is Vermont. It sounds too much to us like the New York Regents, where that's what they do, they prepare to test for a year, and focus is all on that. Also, I think part of what's driving at least the public discussion I hear in Vermont, is the demand of the public to compare. And it's more of the result of the comparison is more important than the outcome and the readiness of the kids to move on to their next stage in life, that's a little bothersome.
Pat Almond (Oregon): This is Pat again. If I can offer one interesting consideration.... In the focus groups that we've done, there is another side expressed, and this is by teachers, testing coordinators, and special education directors. They point out that regular classroom teachers are sometimes quick to exempt students from standards or to consider them not likely to meet them. And so, they give up and don't try. I'm thinking specifically of students who might actually be college bound, if on the runway toward achieving the standards they started a little bit earlier. For example, we might be excluding kids who are blind from high level math, because it would be quite difficult and we don't really know how to do it. So, I agree with you that there can be too much emphasis, but we do hear some concerns about people giving up on kids a little bit too early in the academic achievement area.
Wayne Newberger (Oregon): Part of the issue has to do with what you're going to assess. We've talked here primarily about our multiple choice test and that's only one leg of a three legged stool of our assessment program and the students have to be able to perform in all three. The multiple choice test is basically to get at some indication of whether or not students have some breadth of knowledge about a subject area. But, the other parts of it, particularly the part that Pat referred to as Work Samples, are pieces where the kids are going to have to demonstrate their ability to carry out some fairly complex processes that I think are more along the line of what you're talking about. We've had a lot of discussion in the state about the form of assessment that we're going to have and we started off primarily with Work Samples. But, frankly the public is a little concerned that if you're only doing Work Samples, that you don't have an indication of what kind of foundation the kids are bringing to it. So, we've been basically forced into having an assessment system that incorporates these different components so that there's some sort of balance to it. I think each state has to deal with that in its own way. I think each state has to deal with this issue of assessment in relationship to what the issues of the community are. The legislature has basically laid these pieces out in front of us, saying that these are the parts that are going to be part of it. Our chore is to make sure that they work for All students.
Almont Bassinger (New Mexico): This is Almont Basinger in New Mexico. Is there a lot of pressure or discussion going on about objective versus subjective evaluation?
Wayne Newberger (Oregon): Yeah, there's concern about the subjective part of it. What we talked about is that all of it has clear criteria as to what you look for. Some of it requires judgment, but if you talk about the workplace, most of the evaluation that goes into a lot of workplaces requires judgment, but there's clear criteria. It's when you don't have clear criteria that it's subjective, so you really don't know what you're basing the judgment on. Then I think you really run into problems. After some of the people see the criteria that we use that have been critical, they come around, I think, in terms of their understanding. We would not accept something that requires some judgment on the part of professionals.
Teri Wallace (NTN/NTA): To wrap this up, I just want to say that I'm very thankful to everyone, particularly our speakers today, and apologize very much for whatever hand we had in the communication that went down in terms of getting this thing scheduled. Additionally, I'd like to encourage you all to please use the listserv. There's a whole lot of discussion that needs to continue to happen around assessment, accountability systems and standards. The listserv is one place you can do it.
Our next call is scheduled for December 18th at 1:00 PM, and we'll look forward to talking with you then. At that time, there will be representatives from the Center for Law and Education, they will be discussing Proactive Approaches to Addressing the Rights and Needs of Youth with Disabilities in School-to-Work Systems. With that, take care and thank you very much.
National Transition Alliance for Youth with Disabilities (NTA) conference call presentations are sponsored by the NTA and coordinated by the National Transition Network. For a copy of this or other transcripts, contact us at:
URL:
http://ici1.umn.edu/ntn/audio/1996/nov.html
Posted January 16, 1997
ncset@umn.edu
![NTN [logo]](../../images/ntnThL.gif)