North Dakota: Understanding Student Achievement Within State Assessment (page 3)

Introduction

Within education, a direct relationship exists among standards, instruction, and assessment. Standards drive instruction, instruction fosters learning, and assessment measures understanding.

Standards identify what students should know and be able to do. Teachers craft instruction to facilitate student learning. Teachers assess students to determine what level of learning has occurred. Assessment results inform teachers where further instruction is required. Assessment results report student achievement in terms of expected learning goals, which in turn guide future curriculum and instruction improvement.

The practice of assessing or testing students is well established within classrooms. Teachers routinely teach and assess their students in terms of instructional goals. Assessment results become the basis for grading, promotion, and ultimately graduation.

Similarly, assessments are well established for use by the state. The North Dakota State Assessment measures all students’ achievement with a common tool using a standardized scoring scale. The State Assessment ensures that all students are included into a common accountability system and that all students have access to a comparable educational opportunity.

I. Legal Foundations for the North Dakota State Assessment

A. State Assurances for Comparable Education

The North Dakota constitution mandates that the Legislative Assembly ensure that all North Dakota citizens are literate and afforded an opportunity to receive a high-quality education. The Legislative Assembly has empowered local school boards with the duty to provide such educational opportunities to their residents. The Legislative Assembly provides foundational funding that forms the base for local and other supplemental funding sources to accomplish this educational mandate. Although school districts determine local instructional goals and implement their respective programming, the State retains an inherent constitutional responsibility to ensure comparable and equitable educational opportunities statewide.

It has been the long-standing practice of the State to administer annual assessments at specific grade levels as a means of monitoring overall student achievement levels. These student achievement results have been used to assure a school’s improvement activities required under State accreditation rules, to evidence fundamental civil rights compliance, and to identify schools for program improvement under Title I of the Elementary and Secondary Education Act.

The North Dakota State Assessment assesses all students in designated grades within a single, unified, statewide assessment that measures students’ performance in terms of the state’s challenging content and achievement standards. Such an assessment strategy offers a means to measure student achievement and to assure that all students receive a comparable educational opportunity.

B. State Standards Assure Comparability

State academic standards provide school districts with a means to ensure that all students receive a comparable quality education. Common state standards offer greater assurances of comparable opportunities. To hold schools accountable for the educational services they offer, the state has developed, with the assistance of educators statewide, an assessment system that is designed to measure student performance in terms of these state standards. To ensure that the state’s accountability system engenders confidence among all constituents, the Department has established a system of prescribed activities, or protocols, that are designed to assure procedural validity and reliability, product quality, and systemic integrity. These protocols articulate the governing rules for the development of state academic standards and assessments. A state-level advisory committee consisting of local and state representatives, titled the Standards, Assessment, Learning and Teaching (SALT) Team, assists the Department regarding all state standards and assessment development.

North Dakota state law (NDCC 15.1-02-04.3) places responsibility for the development of state academic content standards with the State Superintendent. Content standards define what a student should know or be able to do at a given grade level. The North Dakota Department of Public Instruction has developed and adopted academic content standards in mathematics (reference these standards at http://www.dpi.state.nd.us/standard/content/math/math.pdf) and English language arts see at http://www.dpi.state.nd.us/standard/content/ELA/ELA.pdf).

State achievement standards complement the state content standards. Achievement standards identify what a proficient student should be able to demonstrate in knowledge and skills at a given grade level. The North Dakota Department of Public Instruction has developed and adopted academic achievement standards in mathematics and English language arts. These achievement standards are contained within the same document as the context standards at the same address.

The state content and achievement standards have been developed for grade Kindergarten through grade 12 in accordance with the North Dakota Standards and Assessment Development Protocols (see http://www.dpi.state.nd.us/standard/protocols.pdf.)These state content and achievement standards form the foundation for the State Assessment.

C. State Law Authorizes State Assessments Aligned to Standards

North Dakota state law (NDCC 15.1-21-08) places responsibility with the State Superintendent for the administration of state assessments that are aligned to the state’s content standards in reading and mathematics. Beginning in 2001-02, state law required that the assessments be administered to at least one grade level selected within each of the following grade spans: grades three through five; grades six through nine; and grades ten through twelve. In 2001-02, 2002-03, and 2003-04, the North Dakota Department of Public Instruction developed and administered assessments at grades four, eight, and twelve to correspond with the state’s content standards.

North Dakota state law further requires that beginning no later than the 2005-06 school year and annually thereafter, the state assessments will be administered in reading and mathematics to all public school students in grades three, four, five, six, seven, eight, and eleven. The Department initiated administration to those grades in 2004-05.

State law requires that the state assessments compile both aggregated and disaggregated results. The state assessments must compile student achievement data that allow for a comparison of individual students, classrooms within a given school and school district, schools within the district, and school districts within the state. The test scores must also allow for comparisons based on students’ gender, ethnicity, economic status, service status, and assessment status, unless doing so enables the identification of any individual student.

State law requires the State Superintendent to present to the Legislative Council the test scores publicly for the first time at a meeting of a legislative committee designated by the Legislative Council. At the meeting, the State Superintendent and representatives of the testing service that created the tests are required to provide detailed testimony regarding the testing instrument, the methodology used to test and assess the students, and the significance of the test scores.

State law requires the State Superintendent to ensure that the State Assessment not include questions that might be deemed personal to a student or to the student’s family and that the assessment not include questions requiring responses that might be deemed personal to a student or to a student’s family. Before a test is finalized for use in North Dakota, the State Superintendent must require that the test be reviewed by a standards-alignment committee appointed by the State Superintendent to ensure that the test meets the requirement of privacy.

State law (15.1-21-14) requires school districts to allow any individual over the age of twenty to view any test administered under sections 15.1-21-08 as soon as the test is in the possession of the school district.

B. Test Administration

The North Dakota State Assessment is administered uniformly statewide to assure a high degree of reliability. Additionally, the scoring and analysis rules employed by CTB/McGraw-Hill offers assurances that any reported results are accurate in their content and that valid inferences can be derived from them.

1. Scoring

The State Assessment is comprised of two types of items: selected-response and constructed response. Student responses to selected-response items are machine scored, and item parameters associated with this type of item capture the difficulty, and discrimination associated with each item. Constructed-response items are scored by CTB/McGraw-Hill’s Hand-scoring Department. Each item has an associated scoring rubric, and scorers receive extensive training in the use of carefully developed scoring guides. The training and monitoring include empirical determinants of reader readiness such as check-sets, read-behinds, and double-blind reads. Most reading, language arts, and mathematics items show high inter-rater adjacent agreement.

2. Analysis

Item and test analyses are performed in two stages. In stage one, CTB/McGraw-Hill research specialists examine the raw data for reasonability using descriptive statistics, such as mean raw scores, item p-values, and point bi-serial correlations. In stage two, CTB/McGraw-Hill research specialists calibrate the data using item response theory models.

3. Reporting

All rules, requirements, and procedures for reading and compiling responses, bridging, assigning scores, and generating reports follow internal reporting protocols. These specifications were determined jointly by Department of Public Instruction personnel and the CTB/McGraw-Hill North Dakota Contract Team. Similarly, the content, format, and type of State Assessment reports were developed jointly by the Department and CTB/McGraw-Hill. Prior to generating reports for North Dakota schools, CTB/McGraw Hill conducts intensive quality assurance reviews of all technical aspects of scoring, bridging, report printing, and other features to verify accuracy and completeness of data. If problems are identified, they are solved, and the quality assurance procedures are repeated until all are verified as correct.

During school years 2001-02 and 2002-03, CTB/McGraw-Hill transmitted student demographic and special codes information electronically to the Department of Public Instruction while scoring was being conducted. This allowed the Department to conduct computer edits to identify data that were beyond accepted parameters. School district testing coordinators were contacted and asked to correct errors and provide complete data. Records were cross checked with school enrollment statistics, Child Count, surveys of schools regarding students with limited English proficiency, and the National School Lunch Program, and differences were identified and reconciled. When student test results were received from CTB/McGraw-Hill, the student score data were merged with verified demographic and student codes data. Performance level data for students using the North Dakota Alternate Assessment were entered.

Adequate Yearly Progress as well as the School Report Card and Profile were generated from the complete records Beginning in 2003-04, much of the student demographic and special codes data were entered by school personnel using the Department’s On-Line Reporting System (ORS). This replaced manual completion for most of the demographics and special codes previously entered on the front cover of each test booklet, utilized student data previously submitted on-line, and allowed for electronic edits of data, increasing efficiency of the coding and validation process. Introduction of a unique student identification number in 2004-05 further facilitated the coding and editing process.

In 2005-06, the Department initiated the use of a student bar code label that contained the student name, unique ten-digit state identification number, date of birth, gender, and grade. These data were extracted from the Pupil Membership File submitted by each school district to the Department by September 15 of each year. CTB/McGraw-Hill generated labels, and shipped them along with testing materials to each school district. Labels were adhered to test books for those students enrolled during the testing window. If a student had transferred from the school prior to testing, the label was not used. For those students who were new to the school district after September 15, identification data were handentered on the front cover of the test book. This move to the use of labels reduced the errors that had resulted from hand-entry.

In 2005-06, the Department continued to use the procedure of securing demographic and special codes for each student through the on-line reporting system (ORS). Many data elements were prepopulated by data already submitted by school districts through the Pupil Membership Report.

C. Reporting Student Achievement Results

State law requires that the state assessment system compile aggregated results and disaggregated results in terms of the State’s challenging achievement standards. The state assessment system must compile student achievement data that allow for a comparison of individual students, classrooms within a given school, schools within a district, and school districts within the State. The test reports must also allow for comparisons based on students’ gender, ethnicity, economic status, service status, and assessment status, unless doing so enables the identification of any individual student. Actual student and school reports are generated and delivered to schools by the state’s testing contractor, CTB/McGraw-Hill.

The North Dakota State Assessment uses three dedicated reports: (a) Student Report, (b) Content Standard Performance Report, and (c) Summary Report. These reports are described below.

1. Student Report

Two sets of the Student Report is generated and delivered to the school district. This report presents achievement level (i.e., novice, partially proficient, proficient, or advanced) for each of the two domains of reading/language arts and mathematics. A narrative describes what knowledge and skills define each of the four performance levels. In addition, total number of points possible and percentage of points earned for each benchmark and standard are included. The reverse side of the Student Report provides a brief description of the State Assessment for the subject area, how it was developed, how to interpret the information, how to access the standards, and how to work with the school to raise the student’s performance level.

2. Content Standard Performance Report

The Content Standard Performance report is provided for each grade, at the classroom, and school building levels. Students are listed alphabetically, with individual data displayed on percentage of possible points correct on each standard and benchmark. Additionally, each student’s scale score and level of achievement is provided.

The Content Standard Performance Report also displays summary data for each grade (average percentage correct by standard and benchmark) for each class, school building, district, and state level, as well as the number of points possible for each standard and benchmark. A school district level report provides summary information that parallels the Content Standard Performance Report. However, this report, known as the Benchmark Summary Report at the school district level, includes no individual student names or average percent correct by class.

3. Summary Report

This school and school district level report provides summary data on the number and percentage of students at each of the four levels of achievement at the state, school district, and school building level by grade. In addition, summary statistics on numbers of students assessed, having no valid attempt, and invalidated are presented. The reverse side describes North Dakota content and achievement standards.

The North Dakota State Assessment reports reference state content and achievement standards. These standards can be accessed at the following web sites: http://www.dpi.state.nd.us/standard/content.shtm and http://www.dpi.state.nd.us/standard/perform/index.shtm.

Since the 2001-02 school year, a School Report Card and Profile has been prepared annually by the Department of Public Instruction for every school, district, and the State, consistent with State and Federal reporting requirements. The School Report Card and Profile presents student achievement data based on the State’s standards, as opposed to previous reporting based on national norms. Districts and schools must inform all parents of the availability of the School Report Card and Profile.

Four distinct levels of student achievement levels are identified, as defined within the State’s achievement standards:

• Advanced: Demonstrates exemplary understanding and exceeds expected level of achievement.
• Proficient: Demonstrates understanding and meets expected level of achievement.
• Partially Proficient: Demonstrates an emerging or developing level of achievement.
• Novice Attempt made; lack of understanding evident.

III. North Dakota State Assessment Report Guide

A. Student Achievement Report

Two sets of individual student reports from the North Dakota State Assessment are provided for both mathematics and reading/language arts. Typically one set remains in the student’s cumulative folder, while the second set is provided to the student and the student’s parent(s). In addition, a label with summary information is provided for the cumulative folder.