NB This post is a work in progress.
Download example files: example_question.zip
This post documents the process I used to programmatically generate multiple variations on a quiz question for the Brightspace virtual learning environment. The quiz question displays an example C program and asks what number it will print out. In each generated variation of the question, placeholder variable names (VAR1, VAR2, VAR3) and numerical values (VAL1, VAL2, VAL3) are replaced by different strings.
The quiz feature in Brightspace provides an “Arithmetic” question type that allows certain values in the question to be randomised and the correct answer to be calculated with a user-specified formula. Unfortunately, the set of supported mathematical operators is very limited. In my code analysis questions, changing the numerical values in the code can alter the flow of control through the program. This forces students to trace the flow of control very carefully, even when attempting the quiz multiple times. Calculating the value printed by each version of the program is more complex than what Brightspace’s Arithmetic question type supports.
My alternative approach for programmatically generating many variations of a question that use the same code structure:
- Write template code (similar to that shown below) in which some variables and numerical values are represented by predefined text markers. Save this file as “code.c”.
- Programmatically generate a CSV file, “questions_list.csv”, in which each line contains a list of variable names and numerical values to be substituted into the template code to create a unique version of the program.
- Automatically process “questions_list.csv”, generating a different copy of the template code for each variation (“generated_code.c”). In each generated code file, the text markers (VAR1, VAL1, etc.) are replaced by the values specified in the corresponding line of “questions_list.csv”.
- Using ImageMagick, generate a PNG image of the code in “generated_code.c”. This not only ensures that the code will appear neatly formatted in the Brightspace quiz, but also prevents students from copying and pasting the code into a file and running it.
- Automatically compile each copy of “generated_code.c” using GCC, then automatically run the program and pipe its output into a file “answer.txt”.
- Generate another file, “brightspace_import_file.csv”, which contains a paragraph of comma-delimited data for each variation of the question (including the filename of the corresponding PNG image, the correct program output, etc.).
- Upload all PNG images of the generated code to the Brightspace module.
- Import “brightspace_import_file.csv” into the Question Library of the Brightspace module so that the questions can then be added to a quiz.
Example template code (“code.c”) for a set of variations on the same code analysis question:
#include <stdio.h>
int main()
{
int VAR1=VAL1, VAR2=VAL2, VAR3=VAL3;
VAR3 = VAL1;
if (VAL2)
{
VAR3 = VAL2;
}
printf("%d", VAR3);
return 0;
}
Shown below is an example PNG image produced from one variation on the code template above. “VAR1” has been replaced by “alice”, “VAL1” has been replaced by “-1”, and so on. The modified code has been automatically rendered as a PNG image using ImageMagick. Students are asked to identify the number that the program prints out.
By automating the process of generating variations on a single code template, it’s possible to create code analysis quizzes that remain challenging to students even when attempted multiple times.
Preparation
On my computer (running Xubuntu Linux 20.04), using ImageMagick to generate a PNG image from each file of example C code required a change to its default security policy. As root, I commented out the following line in /etc/ImageMagick-6/policy.xml
<policy domain="path" rights="none" pattern="@*"/>
Generating the question list file
The following Python program is used to generate a question list file, with a line of text describing each variation on the question.
#
# generate_list.py - written by Ted Burke - 30-Nov-2021
#
# To generate a list of question variations:
#
# python3 generate_list.py
#
from random import randrange
names = [('x','y','z'),('a','b','c'),('alice','bob','charlie')]
# This csv file will be imported into the Brightspace quiz
with open('question_list.csv', 'w') as brightspace_import_file:
# Write a line to define the question base id, question title, question text
brightspace_import_file.write('COMP2109-FlowOfControl-If1,if statements,What number will this program print?,,\n')
# Write a line to define the text markers that will be substituted in the example code
brightspace_import_file.write('VAR1,VAR2,VAR3,VAL1,VAL2,VAL3\n')
for n in range(20):
# Randomly select one of the three sets of variable names
name1,name2,name3 = names[randrange(3)]
# The following line can be edited to control the numerical range of each value generated in the code
value1,value2,value3 = str(randrange(20)-10), str(randrange(20)-10), str(randrange(20)-10)
brightspace_import_file.write(name1 + ',' + name2 + ',' + name3 + ',' + value1 + ',' + value2 + ',' + value3 + '\n')
Shown below is an example of the “question_list.csv” file produced by the above Python program. With the exception of the first two lines, each line defines a different variation on the question. The first three items on each line are the strings to substitute for VAR1, VAR2 and VAR3. The last three items on each line are the strings to substitute for VAL1, VAL2 and VAL3. Different text markers (as many or few as required) can be used by editing line 2.
COMP2109-FlowOfControl-If1,if statements,What number will this program print?,,
VAR1,VAR2,VAR3,VAL1,VAL2,VAL3
alice,bob,charlie,-5,6,0
a,b,c,1,6,6
alice,bob,charlie,-7,1,-3
a,b,c,-4,-6,4
x,y,z,0,3,-10
alice,bob,charlie,3,-9,-7
alice,bob,charlie,-10,-8,0
x,y,z,-6,-5,8
x,y,z,-3,2,2
a,b,c,-2,2,2
alice,bob,charlie,-9,-1,-9
a,b,c,4,-1,8
x,y,z,9,6,8
a,b,c,-10,4,0
x,y,z,-2,4,1
a,b,c,-3,3,-6
alice,bob,charlie,-7,9,-9
alice,bob,charlie,-9,-10,-1
x,y,z,7,-6,4
alice,bob,charlie,8,-4,2
The first line of the above file specifies three items:
- The base question ID string. Each generated variation of the question will be assigned a unique question ID by appending a three-digit number to the base ID string.
- The question title that will be displayed in the Question Library in Brightspace.
- The question text that students will see when taking the quiz.
Generating the question variations – PNG images and Brightspace CSV import file
Once the file “question_list.csv” is generated, we’re ready to generate the code PNG images and the CSV file containing the information for each question which will be used to import the question variations into Brightspace.
#
# generate_questions.py - written by Ted Burke - 30-Nov-2021
#
# To generate variations on a single question:
#
# python3 generate_questions.py
#
import os
# This csv file will be imported into the Brightspace quiz
with open('brightspace_import_file.csv', 'w') as brightspace_import_file:
# Process the question generation list
with open('question_list.csv', 'r') as list_file:
# Read all lines from file
lines = list_file.readlines()
# First line contains question id, title and text
question_info = lines[0].strip().split(',')
question_id_base = question_info[0]
question_title = question_info[1]
question_text = question_info[2]
# Second line contains the text markers to be replaced in the code
keys = lines[1].strip().split(',')
# Now generate many variations on the question
count = 0
for line in lines:
# Only begin generating question variations on the third line
count = count + 1
if count <= 2:
continue
# Generate the unique question ID
question_id = question_id_base + '-' + str(count-2).zfill(3)
print('Generating question ' + question_id)
# Parse values from current line for next variation of the question
values = line.strip().split(',')
# Read in the original code from file code.c
with open('code.c', 'r') as code_file :
code = code_file.read()
# Replace the marker strings in the code
for key,value in zip(keys,values):
code = code.replace(key, value)
# Write the updated code to generated_code.c
with open('generated_code.c', 'w') as file:
file.write(code)
# Generate a png image of the code
os.system('convert -size 1000x1000 xc:white -font "FreeMono" -pointsize 16 -fill black -annotate +15+15 @generated_code.c -trim -bordercolor "#FFF" -border 10 +repage ' + question_id + '.png')
# Compile and run the generated code to get question answer
os.system('gcc generated_code.c -o prog')
os.system('./prog > answer.txt')
with open('answer.txt', 'r') as answer_file :
question_answer = answer_file.read()
# Write the question data to brightspace_import_file.csv
brightspace_import_file.write('NewQuestion,SA,,,\n')
brightspace_import_file.write('ID,' + question_id + ',,,\n')
brightspace_import_file.write('Title,' + question_id + ',,,\n')
brightspace_import_file.write('QuestionText,' + question_text + ',,,\n')
brightspace_import_file.write('Points,1,,,\n')
brightspace_import_file.write('Difficulty,1,,,\n')
brightspace_import_file.write('Image,images/' + question_id_base + '/' + question_id + '.png,,,\n')
brightspace_import_file.write('InputBox,1,6,,\n')
brightspace_import_file.write('Answer,100,' + question_answer + ',,\n')
brightspace_import_file.write('Hint,,,,\n')
brightspace_import_file.write('Feedback,,,,\n')
brightspace_import_file.write(',,,,\n')
Shown below is an example of the Brightspace CSV import file, “brightspace_import_file.csv”. This is generated by the program above.
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-001,,,
Title,COMP2109-FlowOfControl-If1-001,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-001.png,,,
InputBox,1,6,,
Answer,100,6,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-002,,,
Title,COMP2109-FlowOfControl-If1-002,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-002.png,,,
InputBox,1,6,,
Answer,100,6,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-003,,,
Title,COMP2109-FlowOfControl-If1-003,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-003.png,,,
InputBox,1,6,,
Answer,100,1,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-004,,,
Title,COMP2109-FlowOfControl-If1-004,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-004.png,,,
InputBox,1,6,,
Answer,100,-6,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-005,,,
Title,COMP2109-FlowOfControl-If1-005,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-005.png,,,
InputBox,1,6,,
Answer,100,3,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-006,,,
Title,COMP2109-FlowOfControl-If1-006,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-006.png,,,
InputBox,1,6,,
Answer,100,-9,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-007,,,
Title,COMP2109-FlowOfControl-If1-007,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-007.png,,,
InputBox,1,6,,
Answer,100,-8,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-008,,,
Title,COMP2109-FlowOfControl-If1-008,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-008.png,,,
InputBox,1,6,,
Answer,100,-5,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-009,,,
Title,COMP2109-FlowOfControl-If1-009,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-009.png,,,
InputBox,1,6,,
Answer,100,2,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-010,,,
Title,COMP2109-FlowOfControl-If1-010,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-010.png,,,
InputBox,1,6,,
Answer,100,2,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-011,,,
Title,COMP2109-FlowOfControl-If1-011,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-011.png,,,
InputBox,1,6,,
Answer,100,-1,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-012,,,
Title,COMP2109-FlowOfControl-If1-012,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-012.png,,,
InputBox,1,6,,
Answer,100,-1,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-013,,,
Title,COMP2109-FlowOfControl-If1-013,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-013.png,,,
InputBox,1,6,,
Answer,100,6,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-014,,,
Title,COMP2109-FlowOfControl-If1-014,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-014.png,,,
InputBox,1,6,,
Answer,100,4,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-015,,,
Title,COMP2109-FlowOfControl-If1-015,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-015.png,,,
InputBox,1,6,,
Answer,100,4,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-016,,,
Title,COMP2109-FlowOfControl-If1-016,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-016.png,,,
InputBox,1,6,,
Answer,100,3,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-017,,,
Title,COMP2109-FlowOfControl-If1-017,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-017.png,,,
InputBox,1,6,,
Answer,100,9,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-018,,,
Title,COMP2109-FlowOfControl-If1-018,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-018.png,,,
InputBox,1,6,,
Answer,100,-10,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-019,,,
Title,COMP2109-FlowOfControl-If1-019,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-019.png,,,
InputBox,1,6,,
Answer,100,-6,,
Hint,,,,
Feedback,,,,
,,,,
NewQuestion,SA,,,
ID,COMP2109-FlowOfControl-If1-020,,,
Title,COMP2109-FlowOfControl-If1-020,,,
QuestionText,What number will this program print?,,,
Points,1,,,
Difficulty,1,,,
Image,images/COMP2109-FlowOfControl-If1/COMP2109-FlowOfControl-If1-020.png,,,
InputBox,1,6,,
Answer,100,-4,,
Hint,,,,
Feedback,,,,
,,,,
Uploading the question variations to Brightspace
- Go to the Brightspace module in which you want to create the quiz.
- Go to “Module Tools -> Module Admin -> Manage Files”.
- Click into the “images” folder.
- Create a new folder with the same name as the base id of the question pool, e.g. “COMP2109-FlowOfControl-If1”.
- Click into the new folder.
- Click “Upload” and select the 20 code images generated above.
- Go to “Assessment -> Quizzes -> Question Library”.
- Click “New -> Section”.
- Set the “Section Title” to the base id of the new question, tick the check boxes for “Hide Section Title from learners” and “Hide Section Text from learners”, and click “Save”.
- Click into the newly created section to navigate into it, then click on “Import -> Upload a File” and select the CSV file generated above, “brightspace_import_file.csv”.
- Go to “Assessment -> Quizzes” and click on the quiz to which you want to add the new question.
- Click “Add/Edit Questions”.
- Click “Add -> Question Pool”.
- Set the question pool title to the base id of the new question, e.g. “COMP2109-FlowOfControl-If1”.
- Click “Browse Question Library”.
- Tick the check box beside the question library section of the new question, e.g. “COMP2109-FlowOfControl-If1” and click “Import” to add all question variations to the question pool for the new question.
ad hocumentation • n. fast documentation of ideas and solutions. 