Instead of learning, some students "game" computer-based
teaching programs. New research at the USC Information
Sciences Institute is looking at ways of predicting this
behavior, and using such predictions to adapt the systems to
fit individual student needs. Early results are promising.
The use of Intelligent Tutoring Systems( ITS) and other
high-tech learning tools is increasing across the nation, but
the effects are often below expectations. "Intelligent tutoring
systems can provide effective instruction," writes ISI
researcher Carole Beal in a paper that will be presented in
Boston July 20 at the AAAI 21st National Conference on Artificial
Intelligence , "but learners do not always use such
systems effectively."
According to Beal, motivated students interested in course
material take to ITS readily, but others will improvise ways
to get through without putting in much effort: answering at
random, or, quite commonly, abusing the program's help
feature by always asking for help as a way to get the answer
without understanding the method.
Limiting access to the help function, for example, effectively
defeats this last strategy - but doing so would hinder other
students, for whom help is part of the learning
experience.
To try to find out which students were most likely to game
the system, Beal studied the behavior of a sample of 91 high
school students working with a math ITS. Her method
integrated three data sources: Students' reports on their
own motivation; teachers' reports on the same students'
motivation; and, finally machine records of how the students
in question used a web-based high school math tutoring
system.
This last consisted of records of how students attacked math
problems, and five different patterns emerged. Two of these
were clearly unproductive. In one, students clearly selected
answers at random, and kept doing so until they found the
right answer by chance. In the other, they just started
clicking on the help icon immediately after the problem was
presented and kept clicking it repeatedly, to push through to
the answer, and then repeating the process.
Matching up records with ITS behavior, some correlations
were completely unsurprising. Students whose teachers
identified them as motivated and who described themselves
as motivated to do well in math showed little or no game-
the-system behavior.
Other results were less obvious. "Proportionally speaking,"
Beal reported, "students who described themselves as not
good at math, not attracted to math, and not expecting to do
well in math were most likely to use the ITS in a way that
suggested a genuine effort to learn, by spending time
reading the problem, and looking at the help features
carefully and thoroughly".
"The relatively high rate of learning-oriented ITS use by
disengaged students suggests that technology-based
instruction has potential to reach students who are not doing
well with regular classroom instruction&hellip. The opportunity to
learn from software may offer an appealing alternative
because the student can seek help in private."
But between these poles, a large uncertain area remains.
The largest single group of students was those with average
motivation. About half of these followed learning strategies,
the other half guessed. And the guessers were just as likely
to be students whose teachers identified them as having
higher math skills.
Within this group, however, one clue emerged. In the
questionnaire used to elicit the self-descriptions, those who
believed that mathematical skill was intrinsic, something
students either had or didn't have, were more likely to
guess. Those who thought math skill was something
learnable were less likely to.
"This work is only a beginning," says Beal. Her next step will
be to use recently developed, sophisticated models of
learning based on studies of expert human tutor, who (as
Beal writes) accomplish their work "through a repertoire of
feedback messages, sophisticated problem selection, and
judicious offers of learner control when the learner appears
to be flagging."
By refining the ability to determine how a student is using
the system -- what their strategy is - Beal believes she and
her team will be able to make ITSs more useful not just for
the two categories of students using game-the-system
strategies, but also for the other three, who seem to be
trying to learn.
Beal also holds an
appointment as a research professor at Daniel J. Epstein Department of
Industrial and Systems Engineering
Her collaborators on this research included graduate
students Lei Qu and
Hyokyeong Lee, both in the USC Viterbi School of
Engineering computer science department; the work was
funded by a grant from the NSF. |
