• No class on 4/27
• Identify another research article that relates to your learning principle - create another article review blog post (due 5/4) 
• Work on your TEL toolkit. Come prepared to discuss it in class on 5/4

Check List: April 13th (Due April 20th)

To do:

• Book - Read chapter 9: What about my mind? and conclusion
• Identify another research article that relates to your learning principle - keep looking for articles that provide research-based evidence for your principle. Next week you will create another article review blog post (article review due 4/27) 
• ECD assignment - compile competencies, facets, and indicators for your topic. Follow the self-regulated learning example we worked on in class. When thinking about your indicators ground them in a learning tool and think of ways that students could demonstrate that they are performing the competencies. NOTE: if you can't think of ways to gather evidence or indicators in the LMS (Canvas) you may need to think of using another tool or a combination of tools.

Reminder: 

• We will not have class on April 27th. 

Article Review: Stupid Tutoring Systems, Intelligent Humans

Citation

Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans.International Journal of Artificial Intelligence in Education, 1-15.

Google Scholar Link

Summary

Purpose of the research

In this article Ryan Baker discusses the potential of educational data mining (EDM) for driving human decision-making surrounding learning. He suggests an alternate paradigm that focuses on intelligence amplification (amplification of the teacher or learner) rather than artificial intelligence or intelligent tutoring systems.

Research questions

Although not explicitly stated, the overarching researcher’s question is what is the future and potential for EDM and LA?   

Methods

The author uses a literature review and insight to present a alternative paradigm to EDM and LA.

Important note: I reviewed this article due to the significance of Dr. Bakers standing in the EDM and LA research community. Although this article does not provide any new data or evidence, I consider Dr. Baker’s opinion and perspective to be increasingly important. His perspective on the direction for LA research, coming from a deep understanding of EDM is significant and important. 

Subjects (describe subject – including number of)

N/A

Results

Dr. Baker suggests that despite the promise of intelligent tutoring systems to create complex learning interactions, “the learning tutoring systems used at scale today are much simpler” and are not delivering on the larger promise (p. 1). He states that, “… there is a disconnect between the vision of what intelligent tutoring systems could be, and what they are; a disconnect the most impressive examples of what intelligent tutors can do, and what current systems used at scale do” (p. 2).  To expand our understanding, he explains that most tutoring systems today rely on simple methods to extract student behavior such as having students get a concept right three times in a row.

Because the development of rich intelligent tutoring systems have been slow and less that desirable, he poses an alternative. He states that, “Perhaps we do not need intelligent tutoring systems. Perhaps instead what we need, what we are already developing, is stupid tutoring systems. Tutors that do not, themselves, behave very intelligently. But tutors that are designed intelligently, and that leverage human intelligence” (p. 3).

He goes onto site several examples of this type of ecosystem where learning tools give instructors ongoing distillation of student activities allowing the teacher take real time actions to improve the quality of the classroom experience. For example, “Reasoning Mind, uses reports in real-time, obtaining information that a student is struggling with a specific concept right now, and engaging in proactive remediation” (p. 7).

He also mentions ASSISTments stating that, “teachers often read the reports of the previous night’s homework before class, and re-work their planned lecture base on data about what questions students struggle with” (p. 7).

Discussion

Implications

I believe that this alternative paradigm is vital and important for the overall direction of Technology Enhanced Learning (TEL). I concur with Dr. Baker that hybrid systems that enhance teacher intelligence through identification and reports is important. As he states, automated interventions (e.g., ALEKS) are time-consuming and expensive to author. In addition they are brittle, for example “an encouraging message may not be encouraging the 12^th time” (p. 8). Students adapt quickly and will figure out how to defeat the system.

This is why I believe that our work on Math 160 is relevant. We are not attempting to create an automated tutoring system. In fact, we are keeping the human (Ben, Gabby, and Jessica) at the center of the intervention. We are attempting to use our technical system to identify study behaviors that are not engaged, or less than desirable. Instead of forcing students to follow a similar learning path (e.g., getting three right in a row), one that can then become burdensome or tiresome, we are detecting behaviors and allowing intelligent humans to do what they do best, adaptation.

As Dr. Baker suggested, “Humans are flexible and intelligent. Humans can’t sift through large amounts of information quickly, which is why then need data mining and reporting to inform them. But once informed, a human can respond effectively” (p. 9).  The unique ability that humans possess for adaptation and change should be leveraged to enhance learning, in my estimation.  We are looking for the sweet spot where human ingenuity and computer computation work synergistically to enhance learning and teaching.

Contribution and future research

Future research focus

Future research should focus on a learning ecosystem approach, where we leverage what humans do best (adaptation, ingenuity) and we leverage what computers do best (analysis, pattern recognition), bringing those strengths together to enhance learning.

Implications for the technology enhanced learning (TEL) environment

(see discussion above)

Check List: April 6th (Due April 13th)

To do...

• Identify another research article that relates to your learning principle. Write an article review and post it to your blog. 
• Book - read chapter 8 "How can I help slow learners?"
• Identify an area that you believe would benefit from evidence-centered design (ECD). Identify elements of the competency model (CM) - see the Shute article for a review of CM. Bring CM elements to class for discussion. 
• Here is another article that may be beneficial to your understanding of stealth assessment and ECD. Stealth Assessment: Measuring and Supporting Learning in Video Games
• ONE MORE...
• Work on your TEL toolkit here is a snapshot of mine. 

Check List - March 30th (Due April 6th)

To do...

• Book: Read chapter 7: How should I adjust my teaching for different types of learners? 

• Watch Mark Wilson's "Three Years of Logging my Inbox" from the Quantified Self Blog. 

• Read Dr. Valerie Shute's book chapter titled "Assessment and Adaptation in Games" (LINK)

• Identify another research article that relates to your learning principle - keep looking for articles that provide research-based evidence for your principle. Next week you will create another article review blog post (article review due 4/13) 

Optional class related activity -- 

• Attend: Center for the Analytics of Learning and Teaching (C-ALT) monthly meeting. 
• Time: 2:00 
• Day: Tuesday, April 5th 
• Location: TILT 104
• Topic: ASSiSTments Presentation - see agenda for more information. 

Article Review - Spaced retrieval practice increases college students’ short- and long-term retention of mathematics knowledge.

This article describes an empirical research study that applies the spaced retrieval practice method that has been recommended by cognitive psychologist to improve retention.  The article makes a significant contribution because it applies this practice within a college-level pre-calculus course and measures the impact of that practice on course level objectives and outcomes. By simply manipulating the frequency and timing of key (key objectives) practice sessions and quizzes (with hints) researchers were able to measure significant differences in performance that equated to half a letter grade between subjects.

Citation

Hopkins, R. F., Lyle, K. B., Hieb, J. L., & Ralston, P. A. (2015). Spaced Retrieval Practice Increases College Students’ Short-and Long-Term Retention of Mathematics Knowledge. Educational Psychology Review, 1-21.

Google Scholar Link

Summary

Purpose of the research

To determine the utility of “spaced retrieval practice” at a real university setting, and determining retention of commonly encountered math content across courses and semesters (long-term retention).

Their goal was to “determine whether spaced retrieval practice can increase retention of a complex body of college-level mathematical knowledge in both short- and long-term -- i.e., within the pre-calculus course and into the calculus course respectively” (Hopkins et. al, 2015, p. 15).

Research questions

Although research questions were not explicitly stated, researchers stated that they intended to test the effect of spaced retrieval practice in multiple ways including

1) within-subjects by examining whether individuals in the experiment group retained spaced objectives better than massed ones.

2) between-subjects by testing across groups to see if spaced-objectives in the experimental group was greater than retention in the control group (massed study).

3) hybrid by testing retention of massed objectives in the experimental group to retention in the control group. The researchers were interested in seeing if using spaced retrieval practice on some objectives would benefit non-spaced objectives

Methods

Manipulated spaced retrieval practice versus massed practice within a pre-calculus for engineering course. Under the experimental condition, target (key) objectives (those considered to be extremely important) were given priority and treated with a spaced retrieval practice. Those not selected (non-targeted) were treated in the traditional massed practice process.

The Quiz Me tool in MyMathLab (Pearson) was used to control the timing of the presentation of quiz questions (practice quiz questions) for each group. Quiz questions that focused on the targeted objectives were presented three times over the course of the semester. The first practice quiz was presented at the time of acquisition, the second practice quiz was presented on the next quiz, the third practice quiz was presented on the fourth quiz (increasing the time interval by skipping a presentation in the third quiz). It is interesting to note that the researchers skipped the third quiz to make the retrieval more effortful (thinking about desirable difficulties).

Subjects were randomly assigned to either the control or spaced retrieval practice.

Spaced versus massed retrieval practice was manipulated in a hybrid between- and within-subjects design. Click here to view a nice graphical explanation of experimental designs in educational research (see slide #8 for between- and within-subject design).

A unique characteristic about he methods used in this study is that they tested retention over an extended time. Tests included retention of content from the beginning of the semester (2.5 weeks) to the final exam (3 months). Previous studies had tested shorter durations. In addition, researchers followed a subset of subjects into the subsequent Engineering Analysis 1 (EA1) course (this is the first calculus course for engineering students) and tested their retention of foundational calculus material. The examined three performance measures in the EA1 course that included 1) first exam, 2) third unit exam, and 3) cumulative final exam.

Subjects

All subjects were enrolled in Introductory Calculus for Engineering (ICE) and a subset was then enrolled in Engineering Analysis 1 (EA1). 

Of those enrolled in ICE and that took all the ICE exams 46 experienced the control condition (traditional massing) and 40 the experimental condition (some objectives spaced, some massed).

Of those who continued into EA1 and took all the EA1 exams 29 had experienced the control condition (in ICE) and 25 had experienced the experimental condition (in ICE).

Results

Overall performance results were calculated based on performance on final exam questions that specifically covered target objectives (ignoring questions which covered non-target objectives). A proportion of questions answered correctly was calculated for each student.

Experimental group = 1) proportion of spaced objectives correct, and 2) proportion of massed objectives correct

Control group = 1) proportion of massed objectives correct

Within-Subjects Analysis

·      Spaced to massed objectives (Control versus Experimental)

·      Proportions correct submitted to repeated measures analysis of covariance (ANCOVA) – F(1,35)=7.96, p=.008, n²=.150

o   Proportion correct on spaced objectives (M=.71) was higher than on massed objectives (M=.68) – See figure 1 (bar chart)

Between-Subjects Analysis

·      Spaced objectives (in experiment group) to all-massed group

·      Proportions correct submitted to repeated measure analysis of covariance (ANCOVA) – F(1, 80)=9.01, p= .004, n²=.009

o   Proportion correct on spaced objectives (M=.71) was significantly higher than  massed ones (M= .63)

·      Also, calculated subjects’ proportion correct collapsed across spaced and mass objectives in the experiment group (M =.70) to target objectives in the control group (M= .62) – F(1, 80) = 6.58, p= .012, n² = .074.

Engineering Analysis 1 (EA1)

·      First Exam: Students in the experimental group (space retrieval practice) scored higher on the first unit exam (M= .62) than those in control (M= .56)

o   Only approached significance F(1, 64)=3.21, p= .078, n² = .043

·      Third Exam: Groups almost identical on target objectives

o   Experiment (M= .66) & Control (M = .67)

·      Cumulative Final: Significant advantage for students who had been in experiment group (M= .53) and control group (M= .43) F(1, 48)=4.68, p= .035, n² = .087

Other

o   Satisfactory and unsatisfactory grades in EA1

o   Also conducted satisfactory (C or better) versus unsatisfactory (C- or worse) overall grade analysis

o   Control (48.3 % = satisfactory) & (51.7% = unsatisfactory)

o   Experimental (68.0 % = satisfactory) & 32.0% = unsatisfactory)

o   Those in experiment group were more than twice as likely as those in control group to earn a satisfactory grade

Discussion

Implications

This was a well-designed study that controlled for student differences through the randomized assignment of groups. In addition, groups were statistically indistinguishable on race, gender, high school GPA, and Math ACT. Furthermore, these variables were used as covariates in all analysis.

I agree with the authors that the between-group differences in both EA1 and ICE performance differences can be attributed with some confidence to the manipulation of retrieval practice (experimental condition) in the pre-calculus course.

Contribution and future research

Future research focus

Additional studies should attempt to replicate these findings in similar classes, including similar requirements of retention of materials over longer periods of time.

This study has made a significant contribution providing evidence that the spaced retrieval practice can make be applied in real college courses and that that impact can have a significant impact on student performance over time.

This research should be expanded to include other learning resources and methods beyond Pearson’s MyMathLab, to confirm findings.

Implications for the technology-enhanced learning (TEL) environment

The main manipulation in this research was accomplished through technology, mainly Pearssn's MyMathLab. This is extremely interesting because this type of automated manipulation can be scaled and, therefore, can have a larger impact if TEL's are designed or utilized with these principles at the forefront.

This research should be expanded to include other learning resources and methods beyond Pearson’s MyMathLab, to confirm findings.

Un-Check List - Spring Break - ECD for dummies

Un-Check List (Spring Break) 

No assigned work over Spring Break. When you return Monday/Tuesday (or over break) take a look at the following

• I located a very useful online presentation titled Evidence Center Design (ECD) for Dummies. No, we are not Dummies - but this presentation is very informative and practical in orientation. Take a look at that online presentation here. 
• Think about how we can use this framework within our TEL toolkit 
• Carry over from March 9th: Locate articles on your learning principle (prepare to talk about what you found on March 23rd) 

Have a great break! 

Check List Feb. 26th (Due March 9th)

To do (lots of reading but you have two weeks to complete it :)

• Read Chapter 5 (book): Is Drilling Worth It? 
• Read Chapter 1 (Super Crunchers): Who's doing your thinking for you? (Library Reserves) 
• Read Chronicle of Higher Education article: As Big-Data Companies Come to Teaching, a Pioneer Issues a Warning 
• Read article "Impacts of the Digital Ocean on Education" 
• Attend the C-ALT meeting March 1st at 2:00 (TILT 104) 
• Continue to build your TEL toolkit (add articles found and tools located) 
• Locate articles on your learning principle that apply to classroom teaching or learning (prepare to talk about about you found on March 9th (in class) 

Don't forget 

• We will not have class next week (March 2nd) 

Spaced Retrieval Practice

This semester my goal is to learn more about "spaced retrieval practice" and model behaviors that will enhance my learning and long term retention of information about this practice through the use of several technologies. I will be using this blog (technology) to produce several posts over the course of this semester in an effort to synthesize what I learn about spaced retrieval practice (active content generation). In this effort I will attempt to write these posts by pulling content from my memory and referencing my notes only as necessary. Here we go... .

Spaced retrieval practice is the practice of applying two well established and frequently reproduced cognitive psychology effects the testing effect (retrieval practice effect) and the spacing effect.

The testing effect is translated into the practice of testing your ability to remember information through active recall. In other words, the behavior of independently thinking about a subject and brining content back to your working memory from long-term memory reinforces neurological pathways that are critical for memory. We should always recall that long-term memories are residuals of what we think about and if we don't think about something our brains are very efficient and systematic about forgetting (to learn more about forgetting search "Ebbinghaus forgetting curves" here is one interesting article).

Hermann Ebbinghaus

The spacing effect is translated into the practice of spacing your active retrieval (or testing effect practice) over time. There is consistent evidence that by spacing your practice out over longer periods of time you can cheat forgetting. As I mentioned in class there are technology tools that have been designed to help you apply the spacing effect (or spaced retrieval practice) to your learning. One such tool is Anki (see ankisrs.net). I will be building an Anki built around spaced retrieval practice and will continue to use that anki over an extended period of time.

In subsequent posts I plan to explore the classic research findings, how this practice is being studied within classroom settings, and what are the implications beyond retention and connection of these effects / practices to transfer of knowledge.

Check List Feb. 17th (Due Feb. 24th)

To do:

• Read chapter 4 (book): Why is it so hard for students to understand abstract ideas?

• Read about the learning principle that you selected (find resources) 

• Write a blog post that introduces your learning principle (brief introduction with sources) 

• Prepare to share an idea of how you might apply this principle in your classroom or in your own learning 

• Imagine how a technology could be used to leverage this learning principle (Be creative in your thinking)

• Add the above information to your TEL toolkit 

VoiceThread Activity

• Engage with the following VoiceThread (LINK) before class on Wednesday. 

As always if you have questions about any of these activities - don't hesitate to contact me.

Resources: 
Top 100 Tools for Learning 2015

Check List Feb. 10th (Due Feb. 18th)

To do: 

1. Read Chapter 3 (book): Why do students remember everything that's on television and forget everything I say.
2. Start Technology Enhanced Learning (TEL) Tool Kit (bring to class to share your ideas). 
3. Select a learning principle (see listing on previous blog post). The principles are first come, first serve. Place your selection in the comments section (comment here or on post with learning principles).

Think about this essay: 

• I have created a essay type quiz in our Canvas course (yes I did break down and create one :) 
• The quiz (essay format) is titled - Think about learning. 
• As you will see this is a no-stakes essay quiz that does not count for points. 
• The objective is to complete the quiz without referencing your notes or any other resources. 
• It opens Sunday morning (spacing) and closes on Tuesday at 11:59pm.  
• Please complete the essay during this window of time. 
• IMPORTANT NOTE: these essay responses do not need to be extensive or composed as formal complete essays, in fact these could be in a bulleted type format that capture your ideas. 

If you have questions about these "to do" items - don't hesitate to contact me. 

Next Steps - Additional Principles of Learning (Article)

I am writing this blog post in an effort to direct our thinking and work on additional principles of learning. Take a look at this article "Science support learning: The behavioral research base for psychology's top 20 principles for enhancing learning and teaching."  Here is the abstract...

Abstract: Psychological science has much to contribute to preK-12 education because substantial psychological research exists on the processes of learning, teaching, motivation, classroom management, social interaction, communication, and assessment. This article details the psychological science that led to the identification, by the American Psychological Association’s Coalition for Psychology in Schools and Education, of the “Top 20 Principles from Psychology for PreK-12 Teaching and Learning.” Also noted are the major implications for educational practice that follow from the principles.

Here is a list of the principles that they identified.

Principle 1: Students’ beliefs or perceptions about intelligence and ability affect their cognitive functioning and learning 

Principle 2: What students already know affects their learning 

Principle 3: Students’ cognitive development and learning is not limited by general stages of development 

Principle 4: Learning is based on context, so generalizing learning to new contexts is not spontaneous, but rather needs to be facilitated 

Principle 5: Acquiring long-term knowledge and skill Is largely dependent on practice 

Principle 6: Clear, explanatory, and timely feedback to students Is important for learning 

Principle 7: Students’ self-regulation assists learning and self-regulatory skills can be taught 

I couldn't locate Principle 8. (help anyone?)  

Principle 9: Students tend to enjoy learning and perform better when they are more intrinsically than extrinsically motivated 

Principle 10: Students persist in the face of challenging tasks and process information more deeply when they adopt mastery goals rather than performance goals 

Principle 11: Teachers’ expectations about their students affect students’ opportunities to learn their motivation, and their learning outcomes 

Principle 12: Setting goals that are short term (proximal), specific, and moderately challenging enhances motivation more than establishing goals that are long term (distal), general, and overly challenging 

Principle 13: Learning is situated within multiple social contexts

Principle 14: Interpersonal relationships and interpersonal communication are critical to both the teaching–learning process and the social–emotional development of students

Principle 15: Emotional well-being influences educational performance, learning, and development

Principle 16: Expectations for classroom conduct and social interaction are learned and can be taught using proven principles of behavior and effective classroom instruction

Principle 17: Effective classroom management is based on (1) setting and communicating high expectations, (2) consistently nurturing positive relationships, and (3) providing a high level of student support

Principle 18. Formative and summative assessments are both useful, but they require different approaches

Principle 19. Students’ skill and knowledge should be assessed with processes that are grounded in psychological science and that have provided well-defined standards for quality and fairness

Principle 20. Good use of assessment data depends on clear, appropriate, and fair interpretation

This article will be food for thought for next weeks class.

Checklist 2-3-2016

Spacing effect: What do we know about this remarkable learning behavior? 

1. Read Chapter 2 in book: How can I teach students the skills they need when standardized tests require only facts? 
2. Read the article: Dempster, F. N. (1988). The spacing effect: A case study in the failure to apply the results of psychological research. American Psychologist, 43(8), 627. 
3. Select an article related to Dempster's 1988 article, read it and write a blog post summarizing what you learned. 

No more what! Why TEL and Learning Analytics?

I think this is worth watching and contemplating. Most of what I see related to TEL and Learning Analytics at CSU focuses on the What. What capabilities do TEL and learning analytics provide. I spent the entire day in a workshop with well intentioned CSU advisors focused on change and implementation of data-driven tools. After listening to our outstanding advisors it was clear... they are beginning to grasp the what... but are uninspired because the why is simply missing. So the basic question is... why are we doing this?


 

NOTE: here is a link to a wikipedia article on Samuel Pierpont Langley (mentined in Simon's TED talk).

Checklist 1-27-2016

Here is a list of activities assigned on January 27th, 2016

• Due by Wednesday - 2 / 3 / 2016

1) Read the Introduction and Chapter 1 in the book Why Don't Students Like School

2) Complete assignment: Champions and critics of teaching machines (see previous post)
3) Subscribe to classmates' blogs and "Diggs" (Diggs RSS)
3) Read classmates blog posts and comment when inspired
4) Continue to curate your Digg account by adding and removing RSS feeds and "Digg" interesting items related to this course. 

Champions and critics of teaching machines (45-60 mins)

Watch this six-minute video on Teaching Machines, presented by B.F. Skinner (exact date is unverified but believed to be in the 1950s). To put it in historical context, you may find it useful to skim this short history of instructional design, which is itself a historical artifact from the early years of the World Wide Web.

Pick one or two of the following thinkers or approaches and read a bit about them, starting with the resources linked. What would they like about the Teaching Machines approach? What would they oppose, and what alternatives would they propose?

Socratic Method

Communities of Practice

     Etienne Wenger

     Jean Lave

Paulo Freire

Ivan Illich

Social Constructivism

Actor Network Theory

Emergent Learning Model

Assignment: 

Write a blog post about what you think of the “teaching machine.” How do the approaches or above thinkers support (or go against) your thoughts?

Read the blog posts of your classmates and comment on their thoughts.

 This work is licensed under a Creative Commons 
Attribution: octTEL - 2014

Checklist 1 / 20 / 2016

Photo by: Ewoud

Here is a list of activities assigned on January 20th, 2016

• Due by Wednesday - 1 / 27 /2016

1) Read the assigned articles (see previous post)

2) Create a short blog post about these readings (oh yes, create a blog in Google's Blogger or other)

3) Signup for a free digg account (digg.com) and subscribe to this blog (if you get stuck on this step don't panic - we will visit this in class). 

4) Use your digg account throughout this semester to subscribe to interesting content

5) Come prepared to discuss readings on Wednesday
6) Bring a laptop or device to class on Wednesday 

Readings for week #2 (assigned 1 / 20 / 2016)

Read the following articles:

Brown, J.S. (2000). Growing up: Digital: How the web changes work, education, and the ways people learn. Change: The Magazine of Higher Learning, 32(2), 11-20

Kop, R., & Hill, A. (2008). Connectivism: Learning theory of the future or vestige of the past?. The International Review of Research in Open and Distributed Learning, 9(3).

Siemens, G. (2008). Learning and knowing in networks: Changing roles for educators and designers. ITFORUM for Discussion, 1-26.

Write a blog post that synthesizes these articles. Please post before next Wednesday and come prepared to discuss.