Designing effective online open book exams
Online open book exams can take a variety of formats, for example, timed conditions with access to notes, with a summary of notes, with any resources, or pre-seen questions. This format of exam can also simulate conditions that students will face in their future professional lives, with the need to use selected resources and apply knowledge to solve a problem, discuss a topic, or test a hypothesis.
Top tips for writing effective questions
The formation of questions for online open book exams is crucial in setting the appropriate level of challenge and in promoting academic integrity (e.g. limiting opportunities for students to cheat or collude). Use our top tips for writing effective questions:
What skills and knowledge are you assessing? How will students provide evidence of achievement? Which skills will they need to make use of their knowledge?
Intended learning outcome verbs should provide direction towards the choice of question type. Verbs such as identify, select, arrange, distinguish, match indicate that students need to determine the correct response. Verbs such as analyse, interpret, compare, summarise, indicate that students should construct a response.
Online open book exams require questions which encourage greater engagement with critical and analytical thinking, requiring students to apply knowledge, rather than simply recalling it.
What examples can you find? What does this data show? Is anything missing? How could you provide this? Multi-step questions, where one question builds on the previous one, can work well.
Think about your choice of verb – if a student could simply look up the answer, you may want to find a different approach to the task that would make this more challenging. A range of example questions built from different verbs are available at the bottom of this page.
Case-based or problem-based exam questions require students to apply critical reasoning skills in response to a given scenario. This type of scenario-based question requires a more multi-faceted, longer answer response and can help students put themselves into the role of a professional in their field. Longer questions will also test a student’s ability to reason, create, analyse, synthesise, and evaluate and demonstrate higher level skills and knowledge.
Using specific case studies will reduce the chances of an essay mill being able to provide a stock answer.
Locating data or resources should be followed-up with the need for students to probe and apply their findings, rather than just re-write information. Students can submit workings, calculations, proofs, or justifications for their answers.
Present relevant qualitative or quantitative data and then ask interpretative and application questions: What does the data show? What relevance does this data or does the scenario have in terms of…? What other factors could potentially affect this data? How would you test for these?
Give clear instructions to help students answer the question e.g. 'Provide a hand-drawn graph...'.
Asking questions which are linked to a specific element of a course requires students to apply their knowledge, demonstrate their critical thinking, and synthesise new ideas e.g. ‘Using the materials and your notes from weeks 6-9, evaluate the importance of...'.
Recommendations for encouraging good academic practice by your students include:
- Setting questions that require students to interpret or apply subject material rather than simply locating, defining or rewriting information.
- Requiring students to submit workings, calculations, proofs or justifications for their answers.
- Requiring student responses to be contextualised to their own experience.
- Providing guidance on ethical scholarship, such as reminding students to work through the Academic Integrity training on Referencing and Avoiding Plagiarism.
For some exam types, the ability to randomise questions by drawing them from a larger base, or shuffling the sequence, can be beneficial and help reduce opportunity for academic misconduct (Aparna Chirumamilla et al. 2020).
Where students are required to spend time searching for and interpreting information or data in an open book exam, it is important to make allowances for the length of expected answers as most will not be able to write as much per hour as in a traditional closed book exam.
Add a suggested word length for essay-style answers, both to guide students and to manage marking. Where relevant, you can provide an explicit limit on what is needed, such as 'Provide three bullet points to justify your answer' or 'Provide one short sentence for each issue you feel relevant'. Students should spend time searching for and interpreting information/data in an open book exam, so may not be able to write as much per hour than a closed book exam.
Allow students to practice and familiarise themselves with open book exams and different styles of questions before any summative open book assessment. This could be in your unit, or in other units in the year.
Ask students to time themselves when doing a set task with particular resources on their own, then discuss their results in a class session where they can mark their own or each other’s work.
Formative assessment should include any tools that you are expecting students to use in the exam (e.g. Matlab).
Students can reflect on their approach to an open book exam: How did they revise or prepare notes? How effective was their approach?
Adapted from: A Guide for Academics – Open book exams
|Applying||What examples can you find to…?
How would you solve X?
How would you use…?
What would happen if…?
What effect would that have?
|Analysing||What are the parts/features of..?
Compare and contrast A and B?
What is the relationship between A and B?
Why is X different to Y?
|Creating||How would you design a …?
What changes would you make…?
What alternatives are there to…?
How would you evaluate…?
|Evaluating||What does this data show?
Is anything missing? How could you provide this?
What methods would be effective?
Which method is best?
|Reasoning||What would be an example of this?
What other information do we need?
Can you explain your reasoning?
Is there reason to doubt this data/evidence?
|Implications and consequence questions
|What effect would that have?
What is an alternative?
What are you implying by that?
If that happened, what else could happen as a result? Why?
|Viewpoint questions||How would other groups of people respond to this question? Why?
How could you answer the objection that ______would make?
What might someone who believed _____ think?
What is an alternative?
How are ____ and ____’s ideas alike? Different?