Feedback on Piloting this program with 3, 6, 7, and 8h grade students ( sent this to Khan and they asked that I repost here)

We piloted the program with math in the 3rd grade and 6-8. We piloted the program with English in 7 and 8.  Here is a collection of observations:

 

1. 3rd graders did not use this tool well. They got Khanmigo to behave like a calculator, which was the skill they were supposed to be working on. 3rd-grade students who were skilled in math could leverage this tool well because they would ask for summaries of the videos, which allowed them to practice more advanced math. Moving forward with 3-5th grades, we are going to implement Learning Pathways, but only suggest parents opt in for Khanmigo if they have one such student. We will also recommend this option to any new-to-our-context families directly from France. We found that second language learners appreciated being able to talk with Khanmigo in their native language. Beyond these two classes of students, Khanmigo seems like something learners at this age cannot wield well and we deduced it is because there is a need for a certain level of metacognition necessary to be able to ask the right questions.

2. Across the board, this tool was a distraction for our ADHD populations.  They got all of the way distracted by chatting away on all sorts of inappropriate topics and never entered into the learning experience.  We also found this to be true for students who are resistant to the work. 

3. Digital literacy was absolutely a barrier to leveraging this tool. We noted that there needed to be explicit lessons on demonstrating where information existed, and how to use the predetermined engagement selections.  We also had to model how not to use it like Google.  Students who engaged by just looking for the answer languished during practice sessions. 

4. In the tutor me section, students found they could just get the right answer by providing the wrong answer first. I had a 6th grader show me that she knew the math, but when she put in an answer that is one integer off, Khanmigo would reply "That's close, but the answer is actually ____". This is anecdotally not the case when students are working in Learning Pathways.

5. Students loved using Khanmigo in Learning Pathways. They want to skip all of the videos and just get into the practice of math. This was the largest use case. 

6. We want a way to jump into editing with the Writing Coach. This is the most useful part of this tool for English teachers, and we had to work through copying and pasting information into Khanmigo to get to the editing section. None of my English teachers wanted to use Khanmigo to outline an early draft. 

7. Digital Literacy of the teacher also contributed to the usefulness of this tool. Our 8th-grade teacher used this tool with a great deal of success (This is the focus of my dissertation, so I will be writing more extensively on this reality over the summer). Students absolutely disappeared into this learning activity.  We had students actively editing their work for more than 80 minutes.  That is revolutionary. This was deeply useful for the mid-level achievers and the high achievers.  It was useless with struggling writers. This is still helpful because it meant that we were differentiating perfectly for an in-class activity. The teacher was able to work one-on-one with the students who demonstrated the most need while the rest of the class could work independently. 

8. The most useful piece of Khanmigo when editing was the direct feedback on student edits. They used this more than any other thing. The next most used function was to get Khanmigo to generate synonym lists. 

9. This is a big one. You need to be able to retain the editing history. If we are moving into a post-AI world, the editing history is the goldmine of information for a writing instructor. This does not exist in Khanmigo. If we are looking at tools like draftback and origionality.ai, theses systems work on the archeological presentation of how data was entered into a document.  Khanmigo does not preserve this data, so an unsuspecting teacher will look at a Google Doc, see a giant block of text appearing in the editing history, and assume it is plagiarism or AI academic dishonesty. We were excited to see the "Download as Google doc" option, but it is not a rich text format and has no editing history attached. 

 

We have one more meeting as a focus group later on this school year. Our 6th-grade teacher just saw the option to use the Writing Coach, so she is excited to try this tool out.  This is good because our 7th grade English teacher could not get this tool to work with his students, so this new view will help us understand if it is more than students are mentally able to leverage, or if the disconnect is associated with adult digital literacy.