We have certification courses where learners need to complete a system programming and operation scenario as part of the assessment.
Due to the nature of the content and requirements, the assessment is divided into logical sections - learner completes stage 1 programming, then completes assessment stage 1, with questions and operational tests relating to that programming. When they pass, they move on to stage 2. If they don’t have stage 1 right, then that will prevent things working correctly in later stages. They are allowed up to 3 attempts on each stage, with a 1 hour enforced review and learning between each failed attempt. We need to allow this so they have the opportunity to go back and correct their programming - the checking and correcting is as much part of the learning and assessment as the programming.
We’ve recently caught some learners sharing screenshots of their exam responses with colleagues - circumventing the programming requirement entirely. We have no doubt this will continue, and has likely occured more than we’ve identified. Being that these are certification courses, which grants access to licensing, downloads, support and installation privileges, this is a big concern for us.
I’ve looked at using the ‘show certain number of questions from each category’ option, with the idea that we would create a pool/category of questions for each programming requirement. (We can’t just use an overall category for the exam, because there are specific programming items which need to be validated at each stage, so we need a category for each one to ensure each item is covered.)
In theory this function should cover what we need. It should effectively present a different version of the exam for each learner (or attempt), and prevent the sharing,
but... when I tested this I found myself feeling like I was seeing the same questions each time. So after 5 attempts I checked the report stats and found that some questions had been asked 4 times out of the 5 attempts, while others hadn’t been asked at all. I continued to 10 attempts and found some questions asked 5 or 6 times and others only once or twice.
I understand this is a truly random function, and 10 attempts is a relatively small sample, but my concern is we go and invest a massive truckload of time creating 5 different versions of every question for every exam stage for every course, and still when somebody fails and repeats a stage, or shares a screenshot of their answers with their friend, only one or two of those questions is different. This would have very little effect on addressing our problem.
I’m not really optimistic given the research I’ve already done into all the options, but I’m hoping somebody might have found a clever solution for this.
