• No class on 4/27
• Identify another research article that relates to your learning principle - create another article review blog post (due 5/4) 
• Work on your TEL toolkit. Come prepared to discuss it in class on 5/4

Check List: April 13th (Due April 20th)

To do:

• Book - Read chapter 9: What about my mind? and conclusion
• Identify another research article that relates to your learning principle - keep looking for articles that provide research-based evidence for your principle. Next week you will create another article review blog post (article review due 4/27) 
• ECD assignment - compile competencies, facets, and indicators for your topic. Follow the self-regulated learning example we worked on in class. When thinking about your indicators ground them in a learning tool and think of ways that students could demonstrate that they are performing the competencies. NOTE: if you can't think of ways to gather evidence or indicators in the LMS (Canvas) you may need to think of using another tool or a combination of tools.

Reminder: 

• We will not have class on April 27th. 

Article Review: Stupid Tutoring Systems, Intelligent Humans

Citation

Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans.International Journal of Artificial Intelligence in Education, 1-15.

Google Scholar Link

Summary

Purpose of the research

In this article Ryan Baker discusses the potential of educational data mining (EDM) for driving human decision-making surrounding learning. He suggests an alternate paradigm that focuses on intelligence amplification (amplification of the teacher or learner) rather than artificial intelligence or intelligent tutoring systems.

Research questions

Although not explicitly stated, the overarching researcher’s question is what is the future and potential for EDM and LA?   

Methods

The author uses a literature review and insight to present a alternative paradigm to EDM and LA.

Important note: I reviewed this article due to the significance of Dr. Bakers standing in the EDM and LA research community. Although this article does not provide any new data or evidence, I consider Dr. Baker’s opinion and perspective to be increasingly important. His perspective on the direction for LA research, coming from a deep understanding of EDM is significant and important. 

Subjects (describe subject – including number of)

N/A

Results

Dr. Baker suggests that despite the promise of intelligent tutoring systems to create complex learning interactions, “the learning tutoring systems used at scale today are much simpler” and are not delivering on the larger promise (p. 1). He states that, “… there is a disconnect between the vision of what intelligent tutoring systems could be, and what they are; a disconnect the most impressive examples of what intelligent tutors can do, and what current systems used at scale do” (p. 2).  To expand our understanding, he explains that most tutoring systems today rely on simple methods to extract student behavior such as having students get a concept right three times in a row.

Because the development of rich intelligent tutoring systems have been slow and less that desirable, he poses an alternative. He states that, “Perhaps we do not need intelligent tutoring systems. Perhaps instead what we need, what we are already developing, is stupid tutoring systems. Tutors that do not, themselves, behave very intelligently. But tutors that are designed intelligently, and that leverage human intelligence” (p. 3).

He goes onto site several examples of this type of ecosystem where learning tools give instructors ongoing distillation of student activities allowing the teacher take real time actions to improve the quality of the classroom experience. For example, “Reasoning Mind, uses reports in real-time, obtaining information that a student is struggling with a specific concept right now, and engaging in proactive remediation” (p. 7).

He also mentions ASSISTments stating that, “teachers often read the reports of the previous night’s homework before class, and re-work their planned lecture base on data about what questions students struggle with” (p. 7).

Discussion

Implications

I believe that this alternative paradigm is vital and important for the overall direction of Technology Enhanced Learning (TEL). I concur with Dr. Baker that hybrid systems that enhance teacher intelligence through identification and reports is important. As he states, automated interventions (e.g., ALEKS) are time-consuming and expensive to author. In addition they are brittle, for example “an encouraging message may not be encouraging the 12^th time” (p. 8). Students adapt quickly and will figure out how to defeat the system.

This is why I believe that our work on Math 160 is relevant. We are not attempting to create an automated tutoring system. In fact, we are keeping the human (Ben, Gabby, and Jessica) at the center of the intervention. We are attempting to use our technical system to identify study behaviors that are not engaged, or less than desirable. Instead of forcing students to follow a similar learning path (e.g., getting three right in a row), one that can then become burdensome or tiresome, we are detecting behaviors and allowing intelligent humans to do what they do best, adaptation.

As Dr. Baker suggested, “Humans are flexible and intelligent. Humans can’t sift through large amounts of information quickly, which is why then need data mining and reporting to inform them. But once informed, a human can respond effectively” (p. 9).  The unique ability that humans possess for adaptation and change should be leveraged to enhance learning, in my estimation.  We are looking for the sweet spot where human ingenuity and computer computation work synergistically to enhance learning and teaching.

Contribution and future research

Future research focus

Future research should focus on a learning ecosystem approach, where we leverage what humans do best (adaptation, ingenuity) and we leverage what computers do best (analysis, pattern recognition), bringing those strengths together to enhance learning.

Implications for the technology enhanced learning (TEL) environment

(see discussion above)

Check List: April 6th (Due April 13th)

To do...

• Identify another research article that relates to your learning principle. Write an article review and post it to your blog. 
• Book - read chapter 8 "How can I help slow learners?"
• Identify an area that you believe would benefit from evidence-centered design (ECD). Identify elements of the competency model (CM) - see the Shute article for a review of CM. Bring CM elements to class for discussion. 
• Here is another article that may be beneficial to your understanding of stealth assessment and ECD. Stealth Assessment: Measuring and Supporting Learning in Video Games
• ONE MORE...
• Work on your TEL toolkit here is a snapshot of mine. 

Check List - March 30th (Due April 6th)

To do...

• Book: Read chapter 7: How should I adjust my teaching for different types of learners? 

• Watch Mark Wilson's "Three Years of Logging my Inbox" from the Quantified Self Blog. 

• Read Dr. Valerie Shute's book chapter titled "Assessment and Adaptation in Games" (LINK)

• Identify another research article that relates to your learning principle - keep looking for articles that provide research-based evidence for your principle. Next week you will create another article review blog post (article review due 4/13) 

Optional class related activity -- 

• Attend: Center for the Analytics of Learning and Teaching (C-ALT) monthly meeting. 
• Time: 2:00 
• Day: Tuesday, April 5th 
• Location: TILT 104
• Topic: ASSiSTments Presentation - see agenda for more information. 

Article Review - Spaced retrieval practice increases college students’ short- and long-term retention of mathematics knowledge.

This article describes an empirical research study that applies the spaced retrieval practice method that has been recommended by cognitive psychologist to improve retention.  The article makes a significant contribution because it applies this practice within a college-level pre-calculus course and measures the impact of that practice on course level objectives and outcomes. By simply manipulating the frequency and timing of key (key objectives) practice sessions and quizzes (with hints) researchers were able to measure significant differences in performance that equated to half a letter grade between subjects.

Citation

Hopkins, R. F., Lyle, K. B., Hieb, J. L., & Ralston, P. A. (2015). Spaced Retrieval Practice Increases College Students’ Short-and Long-Term Retention of Mathematics Knowledge. Educational Psychology Review, 1-21.

Google Scholar Link

Summary

Purpose of the research

To determine the utility of “spaced retrieval practice” at a real university setting, and determining retention of commonly encountered math content across courses and semesters (long-term retention).

Their goal was to “determine whether spaced retrieval practice can increase retention of a complex body of college-level mathematical knowledge in both short- and long-term -- i.e., within the pre-calculus course and into the calculus course respectively” (Hopkins et. al, 2015, p. 15).

Research questions

Although research questions were not explicitly stated, researchers stated that they intended to test the effect of spaced retrieval practice in multiple ways including

1) within-subjects by examining whether individuals in the experiment group retained spaced objectives better than massed ones.

2) between-subjects by testing across groups to see if spaced-objectives in the experimental group was greater than retention in the control group (massed study).

3) hybrid by testing retention of massed objectives in the experimental group to retention in the control group. The researchers were interested in seeing if using spaced retrieval practice on some objectives would benefit non-spaced objectives

Methods

Manipulated spaced retrieval practice versus massed practice within a pre-calculus for engineering course. Under the experimental condition, target (key) objectives (those considered to be extremely important) were given priority and treated with a spaced retrieval practice. Those not selected (non-targeted) were treated in the traditional massed practice process.

The Quiz Me tool in MyMathLab (Pearson) was used to control the timing of the presentation of quiz questions (practice quiz questions) for each group. Quiz questions that focused on the targeted objectives were presented three times over the course of the semester. The first practice quiz was presented at the time of acquisition, the second practice quiz was presented on the next quiz, the third practice quiz was presented on the fourth quiz (increasing the time interval by skipping a presentation in the third quiz). It is interesting to note that the researchers skipped the third quiz to make the retrieval more effortful (thinking about desirable difficulties).

Subjects were randomly assigned to either the control or spaced retrieval practice.

Spaced versus massed retrieval practice was manipulated in a hybrid between- and within-subjects design. Click here to view a nice graphical explanation of experimental designs in educational research (see slide #8 for between- and within-subject design).

A unique characteristic about he methods used in this study is that they tested retention over an extended time. Tests included retention of content from the beginning of the semester (2.5 weeks) to the final exam (3 months). Previous studies had tested shorter durations. In addition, researchers followed a subset of subjects into the subsequent Engineering Analysis 1 (EA1) course (this is the first calculus course for engineering students) and tested their retention of foundational calculus material. The examined three performance measures in the EA1 course that included 1) first exam, 2) third unit exam, and 3) cumulative final exam.

Subjects

All subjects were enrolled in Introductory Calculus for Engineering (ICE) and a subset was then enrolled in Engineering Analysis 1 (EA1). 

Of those enrolled in ICE and that took all the ICE exams 46 experienced the control condition (traditional massing) and 40 the experimental condition (some objectives spaced, some massed).

Of those who continued into EA1 and took all the EA1 exams 29 had experienced the control condition (in ICE) and 25 had experienced the experimental condition (in ICE).

Results

Overall performance results were calculated based on performance on final exam questions that specifically covered target objectives (ignoring questions which covered non-target objectives). A proportion of questions answered correctly was calculated for each student.

Experimental group = 1) proportion of spaced objectives correct, and 2) proportion of massed objectives correct

Control group = 1) proportion of massed objectives correct

Within-Subjects Analysis

·      Spaced to massed objectives (Control versus Experimental)

·      Proportions correct submitted to repeated measures analysis of covariance (ANCOVA) – F(1,35)=7.96, p=.008, n²=.150

o   Proportion correct on spaced objectives (M=.71) was higher than on massed objectives (M=.68) – See figure 1 (bar chart)

Between-Subjects Analysis

·      Spaced objectives (in experiment group) to all-massed group

·      Proportions correct submitted to repeated measure analysis of covariance (ANCOVA) – F(1, 80)=9.01, p= .004, n²=.009

o   Proportion correct on spaced objectives (M=.71) was significantly higher than  massed ones (M= .63)

·      Also, calculated subjects’ proportion correct collapsed across spaced and mass objectives in the experiment group (M =.70) to target objectives in the control group (M= .62) – F(1, 80) = 6.58, p= .012, n² = .074.

Engineering Analysis 1 (EA1)

·      First Exam: Students in the experimental group (space retrieval practice) scored higher on the first unit exam (M= .62) than those in control (M= .56)

o   Only approached significance F(1, 64)=3.21, p= .078, n² = .043

·      Third Exam: Groups almost identical on target objectives

o   Experiment (M= .66) & Control (M = .67)

·      Cumulative Final: Significant advantage for students who had been in experiment group (M= .53) and control group (M= .43) F(1, 48)=4.68, p= .035, n² = .087

Other

o   Satisfactory and unsatisfactory grades in EA1

o   Also conducted satisfactory (C or better) versus unsatisfactory (C- or worse) overall grade analysis

o   Control (48.3 % = satisfactory) & (51.7% = unsatisfactory)

o   Experimental (68.0 % = satisfactory) & 32.0% = unsatisfactory)

o   Those in experiment group were more than twice as likely as those in control group to earn a satisfactory grade

Discussion

Implications

This was a well-designed study that controlled for student differences through the randomized assignment of groups. In addition, groups were statistically indistinguishable on race, gender, high school GPA, and Math ACT. Furthermore, these variables were used as covariates in all analysis.

I agree with the authors that the between-group differences in both EA1 and ICE performance differences can be attributed with some confidence to the manipulation of retrieval practice (experimental condition) in the pre-calculus course.

Contribution and future research

Future research focus

Additional studies should attempt to replicate these findings in similar classes, including similar requirements of retention of materials over longer periods of time.

This study has made a significant contribution providing evidence that the spaced retrieval practice can make be applied in real college courses and that that impact can have a significant impact on student performance over time.

This research should be expanded to include other learning resources and methods beyond Pearson’s MyMathLab, to confirm findings.

Implications for the technology-enhanced learning (TEL) environment

The main manipulation in this research was accomplished through technology, mainly Pearssn's MyMathLab. This is extremely interesting because this type of automated manipulation can be scaled and, therefore, can have a larger impact if TEL's are designed or utilized with these principles at the forefront.

This research should be expanded to include other learning resources and methods beyond Pearson’s MyMathLab, to confirm findings.

Un-Check List - Spring Break - ECD for dummies

Un-Check List (Spring Break) 

No assigned work over Spring Break. When you return Monday/Tuesday (or over break) take a look at the following

• I located a very useful online presentation titled Evidence Center Design (ECD) for Dummies. No, we are not Dummies - but this presentation is very informative and practical in orientation. Take a look at that online presentation here. 
• Think about how we can use this framework within our TEL toolkit 
• Carry over from March 9th: Locate articles on your learning principle (prepare to talk about what you found on March 23rd) 

Have a great break!