In my past life, I spent the better part of a decade as a college professor. Prior to that, I prepped students for standardized tests. And a few years ago, I designed questions for state proficiency tests here in Georgia. 
That’s a long-winded way to say I’ve dealt with a lot of test questions over the years. That experience has taught me a lot about what it takes to develop fair, effective questions for learners.
An abundance of resources exist on the topic, but I wanted to distill my own findings from over the years into five insights you can start using today to improve the quality of your questions.
1. Test what matters.
Imagine I’d just assigned students the novel, Moby Dick. I write the following two exam questions: 
Complete the sentence “Call me ________.”
What might Ahab’s quest for the great white whale represent?
Ask yourself: what’s being tested?
In the first case, it’s simple recall of the first line of the novel. Any reader with a good enough memory can fill in the blank.
The second question assesses the readers’ ability to evaluate the text, sift through the evidence, and construct their own interpretations.
Both kinds of questions have value. Sometimes, your learners need to know brute facts -- for instance, in compliance situations where there’s a single answer they need to have down pat. In other cases, they’ll need to know how to apply what they’ve learned in a new context.
When you’re writing knowledge tests, align your questions with what you value. Choose your subjects carefully. Construct your prompts with your learning goals in mind. Whether it’s mastery of core facts or ability to apply a practical skill, make sure you’re evaluating what matters.
2. Format your questions consistently.
See if you can select the correct answer without having read the question. 
A. who
B. what
C. when
D. where
E. the giraffe
If you’ve ever seen the Sesame Street feature “One of These Things Just Doesn’t Belong,” you should have been able to identify E. the giraffe as the answer without even reading the question stem (“Which of these is NOT a journalistic question?”).
Unless you're trying to teach your learners how to succeed in test-taking without really trying, maintain consistent formatting. It will focus attention on the evaluative content rather than the formal appearance.
3. Keep it simple and accessible.
When I was in high school, one of my instructors was notorious for devising fiendishly difficult test questions. The prompts could be as long as a paragraph, and the answer choices were just as complicated (think A and C, but not B, None of the above, Not enough information was given, etc.). 
His tests made you think -- I’ll give him that. But they privileged patience and carefulness above competence.
It was too easy for someone who had done the preparation to overlook a detail and miss the whole question.
For differently-abled persons, or learners for whom English is a second language, these tests must have felt even more nightmarish and discriminatory.
Pare down your questions to just what learners need, and follow a predictable, universal design to ensure a fair, accessible, and successful test-making experience for all your learners.
4. Make the testing part of the learning.
We tend to think of tests as discrete events, separate from learning. We take the course. Then we take the test to prove we understood the information presented in it. 
In fact, tests should function as part of the learning experience.
An opportunity to demonstrate proficiency should provide fresh challenges to solidify understanding, contextualize knowledge, and boost learners’ confidence in the material covered (provided they’ve done the work!).
One strategic way to practice this principle is to use incorrect (but plausible!) answers to teach your learners to make important distinctions. Suppose you’re teaching clients how to automate enrollment into your courses. From experience using the feature and from talking with a subject matter expert in support, you know that most problems with the feature arise from two mistakes users make. When you test learners, then, include those two errors as answers. If you’re able, provide an explanation of why the correct answer is right, and why the incorrect answers are wrong. This way, even getting a question wrong can teach learners to get it right when they use the feature.
5. Vary challenge, not difficulty.
When learners inform you a test question was hard, what do they mean? 
It’s possible they’re saying the test question was difficult -- that is, that it required a cognitive effort that few test-takers could muster to answer successfully.
Alternatively, they could be saying the test was challenging -- that is, that it was complex enough to demand a deeper engagement with the subject.
To illustrate the difference, consider two test questions (drawn from this helpful resource):
List all the US Presidents in chronological order by terms in office.
Which President of the United States had the greatest impact on the growth and development of our nation?
The first question is difficult. You have to take the time to memorize, keep straight, and reproduce information on command.
The second question is challenging. You have to engage deeply and critically with the material to respond successfully.
As a rule, don’t make questions any more difficult than your learning goals require. Instead, make them more challenging by introducing layers of complexity to deepen understanding. (For guidance on how to do this, consider reading up on Webb’s Depth of Knowledge model). 
*******
Need some enablement resources to create tests in your platform? We’ve got you covered.
Creating and Managing E-Learning Tests course in Docebo University
Creating Tests and Managing Test Questions article in Docebo Help
Managing Test Options article in Docebo Help
Feedback Management for Tests article in Docebo Help
Managing the Question Bank article in Docebo Help
*******
Did these five tips help you? What would you add to this list? 
Let us know in the comments. 
