Designing effective online examinations
Online open book exams
Online open book exams can take a variety of formats, for example, timed conditions with access to notes, with a summary of notes, with any resources, or pre-seen questions. This format of exam can also simulate conditions that students will face in their future professional lives, with the need to use selected resources and apply knowledge to solve a problem, discuss a topic, or test a hypothesis etc.
Online open book exams require questions which encourage greater engagement with critical and analytical thinking, requiring students to apply knowledge, rather than simply recalling it. Similarly, locating data or resources should be followed-up with the need for students to probe and apply their findings.
Where students are required to spend time searching for and interpreting information or data in an open book exam, so it is important to make allowances for the length of expected answers, as most will not be able to write as much per hour as in a traditional closed book exam.
The formation of questions for online open book exams is crucial in setting the appropriate level of challenge and in promoting academic integrity (e.g. limiting opportunities for students to cheat or collude).
Design your questions with the intended learning outcomes in mind. What skills and knowledge are you assessing? How will students provide evidence of achievement? Which skills will they need to make use of their knowledge? (e.g. to present successfully on a topic, students may need to be able to identify key messages, interpret research, communicate verbally using a variety of media, etc).
Intended learning outcome verbs should provide direction towards the choice of question type. Verbs such as identify, select, arrange, distinguish, match indicate that students need to determine the correct response. Verbs such as analyse, interpret, compare, summarise, indicate that students should construct a response.
Think about your choice of verb – if a student could simply look up the answer, you may want to find a different approach to the task that would make this more challenging.
Case-based or problem-based exam questions require students to apply critical reasoning skills in response to a given scenario. This type of scenario-based question requires a more multi-faceted, longer answer response which could relate to more than one of the intended learning outcomes. Longer questions will also test a student’s ability to reason, create, analyse, synthesise, and evaluate and demonstrate higher level skills and knowledge.
Present relevant qualitative or quantitative data and then ask interpretative and application questions: What does the data show? What relevance does this data or does the scenario have in terms of…? What other factors could potentially affect this data? How would you test for these?
Examples of questions
Adapted from: A Guide for Academics – Open book exams
|Applying||What examples can you find to…?
How would you solve X?
How would you use…?
What would happen if…?
What effect would that have?
|Analysing||What are the parts/features of..?
Compare and contrast A and B?
What is the relationship between A and B?
Why is X different to Y?
|Creating||How would you design a …?
What changes would you make…?
What alternatives are there to…?
How would you evaluate…?
|Evaluating||What does this data show?
Is anything missing? How could you provide this?
What methods would be effective?
Which method is best?
|Reasoning||What would be an example of this?
What other information do we need?
Can you explain your reasoning?
Is there reason to doubt this data/evidence?
|Implications and consequence questions
|What effect would that have?
What is an alternative?
What are you implying by that?
If that happened, what else could happen as a result? Why?
|Viewpoint questions||How would other groups of people respond to this question? Why?
How could you answer the objection that ______would make?
What might someone who believed _____ think?
What is an alternative?
How are ____ and ____’s ideas alike? Different?
Encouraging good academic practice
Recommendations for encouraging good academic practice by your students include:
- Setting questions that require students to interpret or apply subject material rather than simply locating, defining or rewriting information.
- Requiring students to submit workings, calculations, proofs or justifications for their answers.
- Requiring student responses to be contextualised to their own experience.
- Providing guidance on ethical scholarship, such as reminding students to work through the Academic Integrity training on Referencing and Avoiding Plagiarism, especially sections 9-12 of Avoiding Plagiarism which provides guidance on collaboration vs collusion.
Allow students to practice and familiarise themselves with open book exams and different styles of questions before any summative open book assessment. This could be in your unit, or in other units in the year.
Ask students to time themselves when doing a set task with particular resources on their own, then discuss their results in a class session where they can mark their own or each other’s work.
Students can reflect on their approach to an open book exam: How did they revise or prepare notes? How effective was their approach?