Developing Assessments: A Guide to Multiple Choice, Constructed Response,Thematic Essays, and DBQ Items

Developed by Peter Pappas
K - 12 Coordinator for Social Studies Standards

Pittsford Central Schools

Sources: NYS Social Studies Resource Guide, National Assessment of Educational Progress,
"Designing and Collecting Data for School" by Paul Brandon and Terry Higa (University of Hawai’a)

Foundations:

A. Identify the topics to be addressed in the test. Remember that assessments should always be in alignment with district standards:

Standard: What should the students understand, know and be able to do?

Assessment: How will we know if students have achieved the desired results? What will we accept as evidence of student understanding and proficiency?

B. Assess at a variety of cognitive levels. Examples:

Knowing. Tasks in this area can measure students' ability to answer questions by recalling information. "What is the world’s largest tropical rainforest?"

Understanding. In this area, students can be asked to attribute meaning to what has been observed and to explain events. "Why are tropical rainforests located near the equator?"

Applying. This area of thinking can call on students to use many tools and skills as they attempt to develop a comprehensive understanding of a problem en route to proposing viable solutions. "Identify two developments that are threatening the tropical rainforests."

C. Use a wide range of authentic materials as stimuli for assessment questions. These include an atlas, maps, charts, graphs, tables, text-based primary source documents and literary works, and various art forms, including photographs, paintings, cartoons, and posters.

D. Assess a range of skills related to these stimuli. Students can be asked to synthesize information from multiple stimuli or to use outside knowledge in order to interpret a given stimulus. They might, for example, be asked both to describe the data in a table and to draw on outside knowledge to give factually accurate explanations for the patterns revealed. Include performance tasks that, for example, require students to create maps or graphs based on narratives or tables of quantitative data.

 

Writing Multiple-choice Test Items

A. Basic principles

The question is called an item.

Each item should be should be concerned with a single problem and presented in clear and simple language. Avoid vague terms.

Extraneous material should be avoided. Students should be told as much as is needed to understand the question, but no more.

Write items at the appropriate level of difficulty.

B. Writing item stems

The beginning of the multiple-choice item is called the stem.

It should be either a direct question ending in question mark or an incomplete statement ending with a colon. It should include words that otherwise would be repeated in each response.

Items asking for definitions should put the term to be defined in the stem and alternative definitions in the responses.

It is not necessary to include the phrase "of the following" in the stem.

C. Writing item responses

Ideally, each item should have four possible responses.

The correct response should clearly be the best of all the possible responses.

The responses "all of the above" or "none of the above" cannot be used.

Make all responses plausible and attractive to examinees that lack the information or ability necessary to correctly answer the item.

Each response should be distinct from the others; avoid overlapping responses.

Arrange the responses in a logical order, if one exists. Responses consisting of numbers should be in ascending or descending order.

Insure that the correct responses are equally distributed in responses a, b. c, d.

Avoid irrelevant clues to correct answers, such as, when the correct answers are stated more precisely or with more words than the alternative responses.

D. Mechanics

Provide clear instructions.

If the stem is an incomplete statement, each response should begin with a lower case letter and end with a period.

Responses following stems that are complete sentences should begin with an uppercase letter, and either end without a period if they are phrases, or end with a period if they are complete sentences.

Responses should be grammatically consistent with the stem and parallel with one another in form. Sentence structure should be as simple as possible.

When possible, avoid negatively stated items. When writing items stated in the negative, draw the respondents’ attention to the negative form by printing negative words in italics.

 

Constructed Response Items

A. Constructed response questions:

are open ended, short answer questions that measure application-level cognitive skills as well as content knowledge. No opinion questions.

use a range of primary and secondary stimuli and authentic "real world" examples including time lines, maps, graphs, cartoons, charts, and short readings.

are graded against specific criterion (employ a scoring rubric).

B. Constructed response questions should be based on a given stimulus and include a series of questions which build from simple to complex.

First question: A general or specific question with the answer found in the stimulus. "What was the population in the year?"

Second question: Make connections between and among the different parts of the stimulus. "In which year was the immigration the greatest?"

Third question: Require the student to respond with information related to (but not included in) the stimulus. "State a historic trend that explains the data?"

C. Constructed response questions can assess higher level thinking:

make comparisons, contrasts, causes, effects, changes

identify patterns or conflicting points of view

categorize or summarize information

construct graphs or charts from data

state a generalization, conclusion, explanation or prediction

D. Scoring: Each constructed-response question should be scored according to a rubric (scoring guide) that gives varying degrees of credit for correct or partially correct answers. The rubric should include enough information or examples to allow different raters to at arrive the same score for a given student response.

1. A rubric for a short constructed-response question can be:

"Complete" score (2) represented a complete and appropriate answer.

"Partial" score (1) indicated that the response had some, but not all, of the components of an appropriate response.

"Inappropriate" score (0) represented an answer that had none of the components of an appropriate response.

2. Extended constructed-response questions are lengthier and more complex exercises that allow for a finer level of discrimination in scoring the responses. Their rubric can be:

"Complete" score (3) was assigned to a response that was complete and appropriate.

"Essential" response (2) was less complete but included the most important components of an appropriate response.

"Partial" response (1) included some appropriate components, but fewer or less central ones than those required for an "Essential" score.

"Inappropriate" (0) response included only inappropriate material.

 

Thematic Essays

Thematic essays will require students to write a coherent essay that has an introduction, supporting paragraphs and a conclusion. Note that the current SED social studies essay is divided into multiple sections, which invite independent responses. Currently, students can receive full credit without writing an introduction or conclusion.

Thematic essays:

focus on concepts and themes in the Social Studies Learning Standards.

go beyond recall and reiteration of facts.

emphasize critical thinking skills.

ask students to make connections and linkages.

are criterion referenced and employ a 5 point scoring rubric.

require students to move beyond responding to the separate tasks in the question, and consider the connections between sections of the question.

 

Sample thematic essay question – Global History:

Theme: Nationalism -  "Throughout global history, nationalism has
had positive and negative effects."

Task: Define the term nationalism.

Select one country you have studied and give specific historic examples
of nationalism within that country.

Describe a situation where nationalism was either a positive
or a negative force within the country’s history.

To receive full credit in a five point rubric scale, the essay would need to:

show a clear understanding of the theme.

address all aspects of the task.

show an ability to analyze issues and events.

write a cohesive, cogent essay that uses a rich variety of details.

summarize key arguments and points made in the essay.

include a strong introduction and conclusion.

Credit would be reduced if the essay:

failed to address all aspects of the task.

lacked depth or accuracy.

used faulty analysis of the theme or lacked any analysis of the theme.

lacked focus, concrete examples or historic details.

lacked an introduction or conclusion.

 

Document-Based Questions

Document Based Questions (DBQ) assess the ability of each student to work with historical sources in multiple forms. The secondary DBQ will have a maximum of 8 documents; at least 2 of which will be visuals. The 7/8 assessment will have a DBQ with 4 documents (including 2 visuals). The DBQ requires many of the same skills used in developing a research paper - interpreting primary and secondary sources, evaluating sources, considering multiple points of view, using historic evidence, developing and supporting a thesis.

Document-based questions:

are based on the Social Studies Learning Standards, themes and concepts.

focus on critical thinking skills and ask students to make comparisons, draw analogies, apply knowledge to the given data, and require students to apply historic analysis.

ask students to take positions on issues or problems and support their conclusions.

require students to look at issues from multiple perspectives.

require students to apply skills they use as adults

are criterion referenced and employ a scoring rubric.

The DBQ will consist of two parts. In the first part, students will be required to answer a key question on each document. This generally involves interpreting the main idea or point of view expressed in the document. This section will be "scaffolded" to the second part in which the student will write an essay using the documents to respond to a specific question. In this second part, students are asked to utilize all the sources, plus their understanding of history to respond to a question. This second part will require them to move beyond simply quoting all the documents. They will need to use the documents as evidence in support of a thesis that responds to the assigned (part two) question. The student will be required to use all documents and incorporate outside information. The student will have no choice on the DBQ.

To receive full credit in a five point rubric the essay will need to:

thoroughly address all aspects of the task by accurately interpreting the documents plus incorporate outside information related to the documents.

discuss all aspects of the task and support with accurate facts, examples and details.

weigh the importance, reliability and validity of the evidence.

analyze conflicting perspectives presented in the documents. Weave the documents into the body of the essay.

include a strong introduction and conclusion.

Credit would be reduced if the response:

did not recognize the reliability, validity, or perspectives of the documents.

reiterated the content of the documents with little or no use of outside information.

discussed the documents in a descriptive rather than analytic manner.

showed little recognition of the tasks, lacked an introduction or conclusion.

 

home  |  K-5  |  6-8  |  9-12

archived project by Peter Pappas 
Former Social Studies Coordinator Pittsford Central Schools

 Peter Pappas 

Twitter: edteck
Blog: www.peterpappas.com 

Copyright © 1999-2014, Peter Pappas

All content licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License.