Implementing Customizable Site-Wide Spaced Repetition

Hi all,

To give you a sense of the importance of this suggestion to me, here are the lengths I've gone to to try to get this idea implemented at Khan Academy:

• I've connected with and messaged roughly 30 employees of Khan Academy on LinkedIn.
• I've directly emailed Sal Khan.
• I've commented on and upvoted suggestions in this forum.
• I've made donations to Khan Academy.
• I've considered trying to get an interview at Khan Academy to try to get in touch with someone to pitch this idea to.

All of this has been done over a span of years, with no one really getting back to me. I find it pretty baffling that it's this difficult to get a good idea considered by a company with hundreds of employees and a (seemingly) fairly static platform. Personally, I'd highly recommend using a platform like Canny for managing feature requests because IMO it provides a much more transparent way of showing what the company is actively working on. Here are here are two good examples of this in action.

Anyway, that context/background aside, here is my idea.

----------

High Level Summary

Objective:

My objective is to learn as much as possible from Khan Academy and transfer that knowledge to long-term memory.

Issue:

Over time, concepts I’ve learned (or even mastered) are forgotten, but regular review of all content is not feasible, requiring a more targeted approach for review. To my knowledge, Khan Academy lacks a mechanism for time-optimized review of concepts that have been covered site-wide.

Proposed solution:

I propose implementing an Anki or SuperMemo-like mechanism employing spaced repetition to reinforce concepts at an optimal time for a given user, before they are forgotten.

----------

Additional Background and Suggestions

Elaboration on the Issues:

Currently, when content is completed on Khan Academy, it appears to stay completed indefinitely. For example, I haven’t reviewed calculus in years and couldn’t do a problem to save my life, but I remain at e.g., 18% completion:

This is excellent for tracking what I’ve covered on the site, but it doesn’t reflect the true state of my knowledge over time. Ideally, I should review “learned” concepts periodically to keep them fresh and to transfer them to long-term memory. 

I recognize that I could regularly revisit the courses I’ve previously completed and take the course challenges again. However, this requires a lot of manual effort and forces me to spend time completing problems for concepts that I’ve already thoroughly understood. In addition, when users complete course challenges on Khan Academy, if they get questions wrong, it decreases their mastery of the relevant concept. At times, this can be frustrating when the user actually understands the content but simply misreads the question or makes a typo in their answer. I believe Khan Academy is already well-poised to implement a solution to these issues by adopting an Anki-like mechanism for review.

Further information about Anki and Spaced Repetition:

Decent introductions to Anki can be found here, here, and here. In short, Anki uses an algorithm to predict when a concept is likely to be forgotten by a user and then quizzes them on that content before they do. This approach is based on the spaced repetition method pioneered by SuperMemo, which quizzes users at appropriate intervals based on their projected trajectory along the “Forgetting Curve”. This spaced-repetition technique was implemented in response to evidence that information is retained more effectively when reviews are spread out over time. In addition, by tracking a user’s understanding of individual concepts, the software can surface only the information that is most in need of review at a given time, which helps the user focus their efforts on the content they’re most struggling to learn or retain. In other words, it provides a targeted approach for review.

Here’s a brief overview of how Anki works. First, the user designs flashcards for concepts they want to learn. The flashcards can include text, formulas, images, etc. For example, here is a flashcard for quizzing the user on the anatomy of the brain.

After recalling (or attempting to recall) the missing label, the user can click a button to see the answer. They are then prompted to rate how difficult the question was for them. Based on their rating, Anki will resurface that flashcard for review anywhere from minutes to months later.

----------

How this might work for Khan Academy:

Anki (and SuperMemo) use a flashcard-based approach for quizzing users on various concepts. Flashcards could be implemented at Khan Academy; however, I think Khan Academy is already well-positioned to implement a slightly different approach. 

Khan Academy has already developed questions for quizzing users on just about every concept covered on the site. Ideally, a user could opt in to an Anki-like review system for any number of courses they’ve started and then complete questions in “review mode”. Preferably, in review mode, one could select specific areas to review or include all subjects. While I am not a researcher in this field, my experience leads me to suspect that being quizzed with questions from any part of the site (e.g., a math question followed by a history question, etc) would help reinforce knowledge more holistically. This would test the user’s knowledge of various subject areas “cold”, unlike the course challenges, where many similar questions are answered consecutively, which can artificially boost the user’s apparent understanding.

In review mode, I would implement an Anki-like approach of having the user indicate how difficult the question was for them with a post-question rating. Relative to a binary correct/incorrect outcome, this would provide more accurate information about the state of the user’s knowledge. This approach would also solve the issue of users getting answers “wrong” when they misread the question or made a typo in their answer. Ideally, users could opt to see the answer directly rather than having to enter it to continue. This would speed up review when a user encounters a question they know how to solve, but which would take some time to actually work out. If tracking all concepts for all users is computationally prohibitive, perhaps the user could select a limited set of subjects that they would like to review periodically.