North Dakota: Understanding Student Achievement Within State Assessment (continued)

Since 2001-02, the North Dakota State Assessment has been aligned to State standards. The alignment process, anchored in North Dakota teacher participation, began with the development of the English language arts and mathematics state content standards. The second step was to write the State achievement standards. These achievement standards provided the basis for establishing cut-scores for the State Assessment. The State Superintendent has published the results of the alignment process and provided validation that the alignment process was conducted with professional integrity and scientific rigor.

Cut scores for school year 2001-02 through 2003-04 addressed grades 4, 8, and 12. With the assessment of more grades in 2004-05, cut scores for grades 3 – 8 and 11 were established in the spring of 2005. The publication of the State Assessment’s cut scores can be accessed at the following web sites:

http://www.dpi.state.nd.us/testing/assess/cutscores02.pdf for 2002, and http://www.dpi.state.nd.us/testing/assess/cutscores05.pdf for 2005.

By its design as a multiple-measures assessment, the North Dakota State Assessment measures higher order thinking skills and understanding. The State Assessment combines selected-response test items (e.g., multiple choice items) with constructed-response test items (e.g., answers written by the student) into a unified assessment tool. The selected-response items require the student to incorporate reasoning, analysis, and problem-solving skills. Constructed-response items require the student to demonstrate actual writing, reasoning, analysis, and problem-solving skills. It is the commitment of the Department of Public Instruction to employ an item-replacement model that steadily increases the number and quality of constructed-response test items, including greater use of extendedresponse items. The North Dakota Department of Public Instruction has raised this priority to a high level of attention. The assessment design for the North Dakota State Assessment includes a schedule for the advancement of constructedresponse test items.

B. Test Administration

The North Dakota State Assessment is administered uniformly statewide to assure a high degree of reliability. Additionally, the scoring and analysis rules employed by CTB/McGraw-Hill offers assurances that any reported results are accurate in their content and that valid inferences can be derived from them.

1. Scoring

The State Assessment is comprised of two types of items: selected-response and constructed response. Student responses to selected-response items are machine scored, and item parameters associated with this type of item capture the difficulty, and discrimination associated with each item. Constructed-response items are scored by CTB/McGraw-Hill’s Hand-scoring Department. Each item has an associated scoring rubric, and scorers receive extensive training in the use of carefully developed scoring guides. The training and monitoring include empirical determinants of reader readiness such as check-sets, read-behinds, and double-blind reads. Most reading, language arts, and mathematics items show high inter-rater adjacent agreement.

2. Analysis

Item and test analyses are performed in two stages. In stage one, CTB/McGraw-Hill research specialists examine the raw data for reasonability using descriptive statistics, such as mean raw scores, item p-values, and point bi-serial correlations. In stage two, CTB/McGraw-Hill research specialists calibrate the data using item response theory models.

3. Reporting

All rules, requirements, and procedures for reading and compiling responses, bridging, assigning scores, and generating reports follow internal reporting protocols. These specifications were determined jointly by Department of Public Instruction personnel and the CTB/McGraw-Hill North Dakota Contract Team. Similarly, the content, format, and type of State Assessment reports were developed jointly by the Department and CTB/McGraw-Hill. Prior to generating reports for North Dakota schools, CTB/McGraw Hill conducts intensive quality assurance reviews of all technical aspects of scoring, bridging, report printing, and other features to verify accuracy and completeness of data. If problems are identified, they are solved, and the quality assurance procedures are repeated until all are verified as correct.

During school years 2001-02 and 2002-03, CTB/McGraw-Hill transmitted student demographic and special codes information electronically to the Department of Public Instruction while scoring was being conducted. This allowed the Department to conduct computer edits to identify data that were beyond accepted parameters. School district testing coordinators were contacted and asked to correct errors and provide complete data. Records were cross checked with school enrollment statistics, Child Count, surveys of schools regarding students with limited English proficiency, and the National School Lunch Program, and differences were identified and reconciled. When student test results were received from CTB/McGraw-Hill, the student score data were merged with verified demographic and student codes data. Performance level data for students using the North Dakota Alternate Assessment were entered.

Adequate Yearly Progress as well as the School Report Card and Profile were generated from the complete records Beginning in 2003-04, much of the student demographic and special codes data were entered by school personnel using the Department’s On-Line Reporting System (ORS). This replaced manual completion for most of the demographics and special codes previously entered on the front cover of each test booklet, utilized student data previously submitted on-line, and allowed for electronic edits of data, increasing efficiency of the coding and validation process. Introduction of a unique student identification number in 2004-05 further facilitated the coding and editing process.

In 2005-06, the Department initiated the use of a student bar code label that contained the student name, unique ten-digit state identification number, date of birth, gender, and grade. These data were extracted from the Pupil Membership File submitted by each school district to the Department by September 15 of each year. CTB/McGraw-Hill generated labels, and shipped them along with testing materials to each school district. Labels were adhered to test books for those students enrolled during the testing window. If a student had transferred from the school prior to testing, the label was not used. For those students who were new to the school district after September 15, identification data were handentered on the front cover of the test book. This move to the use of labels reduced the errors that had resulted from hand-entry.