North Dakota: Understanding Student Achievement Within State Assessment

I. Legal Foundations for the North Dakota State Assessment

A. State Assurances for Comparable Education

The North Dakota constitution mandates that the Legislative Assembly ensure that all North Dakota citizens are literate and afforded an opportunity to receive a high-quality education. The Legislative Assembly has empowered local school boards with the duty to provide such educational opportunities to their residents. The Legislative Assembly provides foundational funding that forms the base for local and other supplemental funding sources to accomplish this educational mandate. Although school districts determine local instructional goals and implement their respective programming, the State retains an inherent constitutional responsibility to ensure comparable and equitable educational opportunities statewide.

It has been the long-standing practice of the State to administer annual assessments at specific grade levels as a means of monitoring overall student achievement levels. These student achievement results have been used to assure a school’s improvement activities required under State accreditation rules, to evidence fundamental civil rights compliance, and to identify schools for program improvement under Title I of the Elementary and Secondary Education Act.

The North Dakota State Assessment assesses all students in designated grades within a single, unified, statewide assessment that measures students’ performance in terms of the state’s challenging content and achievement standards. Such an assessment strategy offers a means to measure student achievement and to assure that all students receive a comparable educational opportunity.

B. State Standards Assure Comparability

State academic standards provide school districts with a means to ensure that all students receive a comparable quality education. Common state standards offer greater assurances of comparable opportunities. To hold schools accountable for the educational services they offer, the state has developed, with the assistance of educators statewide, an assessment system that is designed to measure student performance in terms of these state standards. To ensure that the state’s accountability system engenders confidence among all constituents, the Department has established a system of prescribed activities, or protocols, that are designed to assure procedural validity and reliability, product quality, and systemic integrity. These protocols articulate the governing rules for the development of state academic standards and assessments. A state-level advisory committee consisting of local and state representatives, titled the Standards, Assessment, Learning and Teaching (SALT) Team, assists the Department regarding all state standards and assessment development.

North Dakota state law (NDCC 15.1-02-04.3) places responsibility for the development of state academic content standards with the State Superintendent. Content standards define what a student should know or be able to do at a given grade level. The North Dakota Department of Public Instruction has developed and adopted academic content standards in mathematics (reference these standards at http://www.dpi.state.nd.us/standard/content/math/math.pdf) and English language arts see at http://www.dpi.state.nd.us/standard/content/ELA/ELA.pdf).

State achievement standards complement the state content standards. Achievement standards identify what a proficient student should be able to demonstrate in knowledge and skills at a given grade level. The North Dakota Department of Public Instruction has developed and adopted academic achievement standards in mathematics and English language arts. These achievement standards are contained within the same document as the context standards at the same address.

The state content and achievement standards have been developed for grade Kindergarten through grade 12 in accordance with the North Dakota Standards and Assessment Development Protocols (see http://www.dpi.state.nd.us/standard/protocols.pdf.)These state content and achievement standards form the foundation for the State Assessment.

C. State Law Authorizes State Assessments Aligned to Standards

North Dakota state law (NDCC 15.1-21-08) places responsibility with the State Superintendent for the administration of state assessments that are aligned to the state’s content standards in reading and mathematics. Beginning in 2001-02, state law required that the assessments be administered to at least one grade level selected within each of the following grade spans: grades three through five; grades six through nine; and grades ten through twelve. In 2001-02, 2002-03, and 2003-04, the North Dakota Department of Public Instruction developed and administered assessments at grades four, eight, and twelve to correspond with the state’s content standards.

North Dakota state law further requires that beginning no later than the 2005-06 school year and annually thereafter, the state assessments will be administered in reading and mathematics to all public school students in grades three, four, five, six, seven, eight, and eleven. The Department initiated administration to those grades in 2004-05.

State law requires that the state assessments compile both aggregated and disaggregated results. The state assessments must compile student achievement data that allow for a comparison of individual students, classrooms within a given school and school district, schools within the district, and school districts within the state. The test scores must also allow for comparisons based on students’ gender, ethnicity, economic status, service status, and assessment status, unless doing so enables the identification of any individual student.

State law requires the State Superintendent to present to the Legislative Council the test scores publicly for the first time at a meeting of a legislative committee designated by the Legislative Council. At the meeting, the State Superintendent and representatives of the testing service that created the tests are required to provide detailed testimony regarding the testing instrument, the methodology used to test and assess the students, and the significance of the test scores.

State law requires the State Superintendent to ensure that the State Assessment not include questions that might be deemed personal to a student or to the student’s family and that the assessment not include questions requiring responses that might be deemed personal to a student or to a student’s family. Before a test is finalized for use in North Dakota, the State Superintendent must require that the test be reviewed by a standards-alignment committee appointed by the State Superintendent to ensure that the test meets the requirement of privacy.

State law (15.1-21-14) requires school districts to allow any individual over the age of twenty to view any test administered under sections 15.1-21-08 as soon as the test is in the possession of the school district.

D. State Assessments Fulfill Federal Accountability Requirements

North Dakota, through an agreement with the U.S. Department of Education, has established an assessment plan to bring the state into full compliance with Section 1111(b)(1) requirements of the Elementary and Secondary Education Act (ESEA). In accordance with North Dakota’s approved assessment agreement and the North Dakota Standards and Assessment Development Protocols (reference protocols at http://www.dpi.state.nd.us/standard/protocols.pdf), state assessments were developed and adopted in mathematics and reading/language arts as indicated in Table 1 below on or in advance of the schedule. In 2006-07, one year prior to that required, North Dakota will develop state assessments in science at grades 4, 8, and 11 in accordance with state protocols and ESEA, section 1111(b)(1) requirements.

North Dakota has submitted its plan to expand the development of grade specific assessments to meet the requirements of the No Child Left Behind Act (NCLBA). This plan supported the state’s Consolidated Application for ESEA funding, dated June 2002, and can be accessed at http://www.dpi.state.nd.us/grants/DOEapp.pdf. The state Consolidated Application has since been approved by the U.S. Department of Education.

The North Dakota State Assessment provides for a single, unified, statewide tool that measures the performance of all students in terms of the state’s challenging content and achievement standards. As required by state law, the North Dakota Department of Public Instruction has contracted with a single assessment vendor to develop and administer the state’s assessment tool, within the general guidance of state protocols and under the supervision of the Department of Public Instruction. The State neither provides for nor permits any assessment alternatives administered by any other local school district, school, or outside entity, aside from the statewide assessment prescribed by the State Superintendent. The state has contracted with CTB/McGraw-Hill to coordinate the development and scoring of the State Assessment. Students with significant disabilities are assessed with the North Dakota Alternate Assessment, and their scores are integrated into the schools’, districts’, and state’s overall student achievement database.

II. Technical Design and Quality Assurance Provisions for the North Dakota State Assessment

The design, administration, and reporting of the 2001-02 State Assessment marked a significant change from the past. The State moved from the former, off the-shelf, norm-referenced assessment to a criterion-referenced assessment aligned to the state’s content standards. Additionally, the State moved from referencing student achievement in terms of national norms to reporting student achievement in terms of the state’s challenging achievement standards.

A. Assessment Design

The North Dakota Assessment System uses an assessment tool that is aligned to North Dakota’s content standards. For school years 2001-02 through 2003-04, the Department of Public Instruction contracted with CTB/McGraw-Hill to develop and administer CTB/McGraw-Hill’s Terra Nova, Second Edition, Basic Multiple Assessment with a dedicated State Supplement of uniquely aligned test items. The combination of the Terra Nova and the State Supplement constituted the North Dakota State Assessment in those school years.

The assessment design for the 2004-05 instrument demonstrated marked changes. Rather than using a separate additional instrument, the North Dakota State Supplement, to ensure reliable measurement of all standards, one integrated criterion-referenced instrument was developed. The State utilized the CTB/McGraw-Hill test item bank to select items that addressed state content standards and benchmarks. This new design coincided with the revision of the content standards for English language arts and mathematics, as well as the expansion of the North Dakota State Assessment from three grades (4, 8, and 12) to seven (3, 4, 5, 6, 7, 8, 11). Further, 2004-05 was the first year of a new contract with the testing vendor, CTB/McGraw-Hill.

Since 2001-02, the North Dakota State Assessment has been aligned to State standards. The alignment process, anchored in North Dakota teacher participation, began with the development of the English language arts and mathematics state content standards. The second step was to write the State achievement standards. These achievement standards provided the basis for establishing cut-scores for the State Assessment. The State Superintendent has published the results of the alignment process and provided validation that the alignment process was conducted with professional integrity and scientific rigor.

Cut scores for school year 2001-02 through 2003-04 addressed grades 4, 8, and 12. With the assessment of more grades in 2004-05, cut scores for grades 3 – 8 and 11 were established in the spring of 2005. The publication of the State Assessment’s cut scores can be accessed at the following web sites:

http://www.dpi.state.nd.us/testing/assess/cutscores02.pdf for 2002, and http://www.dpi.state.nd.us/testing/assess/cutscores05.pdf for 2005.

By its design as a multiple-measures assessment, the North Dakota State Assessment measures higher order thinking skills and understanding. The State Assessment combines selected-response test items (e.g., multiple choice items) with constructed-response test items (e.g., answers written by the student) into a unified assessment tool. The selected-response items require the student to incorporate reasoning, analysis, and problem-solving skills. Constructed-response items require the student to demonstrate actual writing, reasoning, analysis, and problem-solving skills. It is the commitment of the Department of Public Instruction to employ an item-replacement model that steadily increases the number and quality of constructed-response test items, including greater use of extendedresponse items. The North Dakota Department of Public Instruction has raised this priority to a high level of attention. The assessment design for the North Dakota State Assessment includes a schedule for the advancement of constructedresponse test items.

B. Test Administration

The North Dakota State Assessment is administered uniformly statewide to assure a high degree of reliability. Additionally, the scoring and analysis rules employed by CTB/McGraw-Hill offers assurances that any reported results are accurate in their content and that valid inferences can be derived from them.

1. Scoring

The State Assessment is comprised of two types of items: selected-response and constructed response. Student responses to selected-response items are machine scored, and item parameters associated with this type of item capture the difficulty, and discrimination associated with each item. Constructed-response items are scored by CTB/McGraw-Hill’s Hand-scoring Department. Each item has an associated scoring rubric, and scorers receive extensive training in the use of carefully developed scoring guides. The training and monitoring include empirical determinants of reader readiness such as check-sets, read-behinds, and double-blind reads. Most reading, language arts, and mathematics items show high inter-rater adjacent agreement.

2. Analysis

Item and test analyses are performed in two stages. In stage one, CTB/McGraw-Hill research specialists examine the raw data for reasonability using descriptive statistics, such as mean raw scores, item p-values, and point bi-serial correlations. In stage two, CTB/McGraw-Hill research specialists calibrate the data using item response theory models.

3. Reporting

All rules, requirements, and procedures for reading and compiling responses, bridging, assigning scores, and generating reports follow internal reporting protocols. These specifications were determined jointly by Department of Public Instruction personnel and the CTB/McGraw-Hill North Dakota Contract Team. Similarly, the content, format, and type of State Assessment reports were developed jointly by the Department and CTB/McGraw-Hill. Prior to generating reports for North Dakota schools, CTB/McGraw Hill conducts intensive quality assurance reviews of all technical aspects of scoring, bridging, report printing, and other features to verify accuracy and completeness of data. If problems are identified, they are solved, and the quality assurance procedures are repeated until all are verified as correct.

During school years 2001-02 and 2002-03, CTB/McGraw-Hill transmitted student demographic and special codes information electronically to the Department of Public Instruction while scoring was being conducted. This allowed the Department to conduct computer edits to identify data that were beyond accepted parameters. School district testing coordinators were contacted and asked to correct errors and provide complete data. Records were cross checked with school enrollment statistics, Child Count, surveys of schools regarding students with limited English proficiency, and the National School Lunch Program, and differences were identified and reconciled. When student test results were received from CTB/McGraw-Hill, the student score data were merged with verified demographic and student codes data. Performance level data for students using the North Dakota Alternate Assessment were entered.

Adequate Yearly Progress as well as the School Report Card and Profile were generated from the complete records Beginning in 2003-04, much of the student demographic and special codes data were entered by school personnel using the Department’s On-Line Reporting System (ORS). This replaced manual completion for most of the demographics and special codes previously entered on the front cover of each test booklet, utilized student data previously submitted on-line, and allowed for electronic edits of data, increasing efficiency of the coding and validation process. Introduction of a unique student identification number in 2004-05 further facilitated the coding and editing process.

In 2005-06, the Department initiated the use of a student bar code label that contained the student name, unique ten-digit state identification number, date of birth, gender, and grade. These data were extracted from the Pupil Membership File submitted by each school district to the Department by September 15 of each year. CTB/McGraw-Hill generated labels, and shipped them along with testing materials to each school district. Labels were adhered to test books for those students enrolled during the testing window. If a student had transferred from the school prior to testing, the label was not used. For those students who were new to the school district after September 15, identification data were handentered on the front cover of the test book. This move to the use of labels reduced the errors that had resulted from hand-entry.

In 2005-06, the Department continued to use the procedure of securing demographic and special codes for each student through the on-line reporting system (ORS). Many data elements were prepopulated by data already submitted by school districts through the Pupil Membership Report.

C. Reporting Student Achievement Results

State law requires that the state assessment system compile aggregated results and disaggregated results in terms of the State’s challenging achievement standards. The state assessment system must compile student achievement data that allow for a comparison of individual students, classrooms within a given school, schools within a district, and school districts within the State. The test reports must also allow for comparisons based on students’ gender, ethnicity, economic status, service status, and assessment status, unless doing so enables the identification of any individual student. Actual student and school reports are generated and delivered to schools by the state’s testing contractor, CTB/McGraw-Hill.

The North Dakota State Assessment uses three dedicated reports: (a) Student Report, (b) Content Standard Performance Report, and (c) Summary Report. These reports are described below.

1. Student Report

Two sets of the Student Report is generated and delivered to the school district. This report presents achievement level (i.e., novice, partially proficient, proficient, or advanced) for each of the two domains of reading/language arts and mathematics. A narrative describes what knowledge and skills define each of the four performance levels. In addition, total number of points possible and percentage of points earned for each benchmark and standard are included. The reverse side of the Student Report provides a brief description of the State Assessment for the subject area, how it was developed, how to interpret the information, how to access the standards, and how to work with the school to raise the student’s performance level.

2. Content Standard Performance Report

The Content Standard Performance report is provided for each grade, at the classroom, and school building levels. Students are listed alphabetically, with individual data displayed on percentage of possible points correct on each standard and benchmark. Additionally, each student’s scale score and level of achievement is provided.

The Content Standard Performance Report also displays summary data for each grade (average percentage correct by standard and benchmark) for each class, school building, district, and state level, as well as the number of points possible for each standard and benchmark. A school district level report provides summary information that parallels the Content Standard Performance Report. However, this report, known as the Benchmark Summary Report at the school district level, includes no individual student names or average percent correct by class.

3. Summary Report

This school and school district level report provides summary data on the number and percentage of students at each of the four levels of achievement at the state, school district, and school building level by grade. In addition, summary statistics on numbers of students assessed, having no valid attempt, and invalidated are presented. The reverse side describes North Dakota content and achievement standards.

The North Dakota State Assessment reports reference state content and achievement standards. These standards can be accessed at the following web sites: http://www.dpi.state.nd.us/standard/content.shtm and http://www.dpi.state.nd.us/standard/perform/index.shtm.

Since the 2001-02 school year, a School Report Card and Profile has been prepared annually by the Department of Public Instruction for every school, district, and the State, consistent with State and Federal reporting requirements. The School Report Card and Profile presents student achievement data based on the State’s standards, as opposed to previous reporting based on national norms. Districts and schools must inform all parents of the availability of the School Report Card and Profile.

Four distinct levels of student achievement levels are identified, as defined within the State’s achievement standards:

• Advanced: Demonstrates exemplary understanding and exceeds expected level of achievement.
• Proficient: Demonstrates understanding and meets expected level of achievement.
• Partially Proficient: Demonstrates an emerging or developing level of achievement.
• Novice Attempt made; lack of understanding evident.

III. North Dakota State Assessment Report Guide

A. Student Achievement Report

Two sets of individual student reports from the North Dakota State Assessment are provided for both mathematics and reading/language arts. Typically one set remains in the student’s cumulative folder, while the second set is provided to the student and the student’s parent(s). In addition, a label with summary information is provided for the cumulative folder.