2013 NCWIT Summit - Plenary Panel, Leveling the Playing Field with MOOCs

August 14, 2013

[music 00:05]

Valerie Taylor:  Welcome to our panel on leveling the playing field with MOOCs. I'm happy to have our distinguished panelists here. We have Ben Eater, lead exercise developer with the Khan Academy. We just heard the presentation.

We also have Maggie Johnson, director of education and university relations at Google. Maggie manages all technical training, content development and information management programs for Google engineers and operations staff as well as Google's K‑12 educational programs in STEM, and computer science.

She also manages the university relations for Google, building strategic research partnerships with faculty and labs globally. Maggie is also very passionate about rebuilding K‑12 computer science education. Prior to Google, Maggie was a faculty member and Director of Undergraduate Studies in the Department of Computer Science at Stanford University.

With respect to educational frameworks, Maggie brings the perspective of the Google Course Builder. There are five key features with the Google Course Builder. Open Source, in the cloud, free, easily create online courses with great flexibility for activities, multiple lessons and video, and also great support for multiple assessment types.

This is a list of some of the courses offered with the Google Course Builder, with an example from the UniMOOC on entrepreneurship that was offered at the University of Alicante.

Our next panelist is Pooja Nath Sankar, who is the CEO and Founder of Piazza. Before founding Piazza, Pooja was a developer at Kosmix and at Oracle, and worked at Facebook on the News Feed Team. She received her Undergraduate Degree in Computer Science from IIT Kanpur in India, a Master's in Computer Science from the University of Maryland College Park, and a MBA at Stanford Graduate School of Business.

Pooja developed Piazza to be a remedy for students who are not given the intellectual space, freedom, or support to fulfill their educational potential and desire for learning.

With respect to educational frameworks, Pooja will bring the perspective of Piazza, a framework to encourage student engagement and course discussions and questions. WitsOn is an example of the use of Piazza for online sharing among women in technology. The sharing was among Undergraduate Students pursuing STEM with women mentors from industry and academia for six weeks in October of last year.

There were 632 mentors and over 15,000 students involved for which 90 percent were women.

David Evans is an Associate Professor of Computer Science at the University of Virginia. David's research seeks to create systems that can be trusted, even in the presence of malicious attackers, and then empower individuals to control how the data is used. He won the Outstanding Faculty Award in 2009 from the State Council of Higher Education in Virginia.

While on leave from the University of Virginia, Dave was Udacity's Vice President of Education, and helped to develop a dozen Computer Science MOOCs. Dave has taught two large open courses for Udacity, including the Introduction Computer Science course that has enrolled over 300,000 since the first offering in February 2012. Here we have some statistics about the CS101, thanks Dave.

We can see that approximately 10 percent of those who sign up for the Computer Science course completed the course. The completion represents 38 percent of those who attempted at least the first homework problem. The demographics also indicate significant international participation and 12.5 percent women. What we're going to do for this session is that we're going to have it be interactive. We enjoy hearing from you.

As we promised from the last session, you'll have an opportunity to ask questions to Ben Eater as well as our other panelists, Maggie Johnson, David Evans, and also Pooja as well. I'll start off with a question, and then I'll open the floor for a question from you, and then we'll go back and forth, keeping track of time.

Starting with the statistics, with respect to diversity, which is our focus in looking at MOOCs. Also, other similar type of environments, as we focus on women and people with disabilities as well as traditionally underrepresented groups, being African Americans, Hispanics, and Native Americans, then what we want to start off talking about is the demographic data.

We recognize that this data is important, but we want to ask the question about the challenges in obtaining this data, and also opportunities for addressing these challenges. With that, I'll open up the questions. Maybe Dave, if you'd like to start?

David:  OK, the data that's here is from a small survey of students who've finished the first run of the class. In general, there are lots of questions about what data you should collect. One of the appealing properties of MOOCs is students can participate in them without sharing their identity if they don't want to, so students can participate anonymously.

When there's discussions in the forum, people don't know where they're from, who they are, male, female, all of these things. Unless someone decides to share that or picks a username that makes that clear, all those things are hidden.

I think in terms of understanding more about what works, MOOCs are a great environment for doing experiments. There are students who are willing to volunteer demographic data, and there are probably a great wealth of data and experiments that could be done to try to learn about how different types of students are succeeding in MOOCs or not succeeding in MOOCs.

You can do experiments at the level of individual lessons to find out whether this works or doesn't work, and whether the type of student affects that. I think there is a big opportunity here. There has not been a lot of data collected in the demographic side yet.

You've got to worry about different laws about how you collect demographic data. You don't want to require students to do anything. At least, the philosophy at Udacity was, other than providing an email address, we don't want to force students to provide anything to start taking a class.

Valerie:  OK. We'll go down with Pooja.

Pooja Sankar:  I guess one of the biggest things to think about is, the sentence or the discussion that sparked me wanting to start Piazza was actually at Facebook, among women engineers who all stood up and said, "When I went to MIT, or when I went to Stanford, or when I went to Harvard, I didn't feel like I had a support group."

That was really shocking to me, because I went to India, where social discussion among boys and girls wasn't very common. To think that women who had grown up in America where they did start interacting with the opposite sex from high school, they also felt they didn't have a support group.

One thing we should be very cognizant about is, maybe this problem could be exacerbated in a hundred thousand student class, or a ten thousand student class, where the women don't speak up enough and they actually feel more in minority. Thinking about what we could do, I think, is an area of interest for me specifically.

Ben Eater:  Yeah, to your question on challenges in collecting demographic data, one of the things, Khan Academy, our mission is a free, world‑class education for anyone, anywhere. We want to lower the bar to sign up or to engage with our content. All of our content is available without signing up on our site.

If you do sign up, we try to make that as frictionless as possible. We use third‑party authentication services from Google and Facebook. We don't try to share a lot of data with them, just because we want to keep that as frictionless as possible.

Relatively recently, we now will allow you to create an account with us, directly, without using those third‑party authenticators. When we did that, we had to ask for birth date because we need to ensure that for people that are under 13, we respect certain privacy things there for legal reason.

We also decided to ask for gender, and it's optional. We actually, until last week, had never looked at that data, but I knew I was coming here, and I figured I would be asked.


Ben:  I looked at that data last week. You guys have had some impact. I'm not sure if there are biases. You guys would probably understand that better than me. Of the data we've collected it looks like about 47 percent of our registered users are women.

I also know you guys are particularly interested in computer science, so I looked at our computer science platform. I showed you the programs earlier. We launched that less than a year ago. Over 600,000 programs have been created on the site. It looks like about 40 percent of those programs are created by women.

That's basically the data we've gotten. As far as demographics, our demographics skew fairly young. There's a nice normal distribution around 14 years old.

Valerie:  Maggie.

Maggie Johnson:  With Google there are really three sets of classes that we look at when we talk about Course Builders. The first set are the, what we call, internal for external. These are courses that we build internally to teach our users about products and services that Google offers. We did power searching last summer.

We've got a big MOOCs that we're going to be doing on maps, and Google Earth, which just announced IOLA last week. Those kinds of courses are ones that they actually represent Google product launches. We're very sensitive about the kind of data that we collect there and very limited due to our privacy policy about the data that we can collect with any students taking those courses. That's one set.

The second set are courses that our partners, or people externally, who are using Course Builder. The way it works is it all runs on App Engine. If you're a user of Course Builder, then they actually own your own instance of App Engine and you run your course. You can brand it any way you like. You own the relationship with your students. You own the data.

We know that many of our users, our external users, on Course Builder have collected demographic data. We can't access that either because that's their instance. We can't really get at it either way, because of our privacy policy, or because of the way the terms of service for App Engine are set up.

A third area that we use Course Builder for is for internal engineering education. I think we're close to 20,000 engineers now. A good percentage of them are women. We use Course Builder internally to do engineering training. In that sense we're able to do some of the experimentation to understand what are the best ways that we can support our women engineers in the training, and the types of environments that are going to be most conducive for them.

We take that information and we pass it on to the Course Builder engineers, so that can get built into the larger platform. We use our own engineering community as a lab for the things that we can build, even if we don't know who they are. We want to try and make it supportive for any women that are taking courses on Course Builder.

Valerie:  Now I do want to open it up for questions from the audience if we have some. Oh, a pioneer.

Audience Member:  I'm confused about the economics with regard to these MOOCs. Some I know are coming from colleges and universities. I don't understand the way in which the economics of this works. Why is it to the benefit of a major college or university to put one of these online, where, as far as I understand the student is able to use it for free?

Valerie:  David.

David Evans:  It's very early. There are many different business models that are being explored. Part of where the opportunity is that the cost can be very low. At least the cost per student can be very low that you don't necessarily need a big revenue stream to support it.

One of the opportunities is, if you have more resources, you can do better courses and put more into it and do the kinds of things that Ben showed us with the kinds of much more developed exercises, and put more research in the courses.

The different models that are being explored, some are recruiting models where your students who are interested in getting a job can indicate that. MOOC provider might be getting a recruiting fee from companies that they connect students who do well in the courses to.

There are also ongoing experiences where students are paying tuition to get credit in the class. Starting over the last few months there are more courses that you can take for credit. Students who want to get credit will be paying for the privilege of having some additional parts of the course, being able to take a proctored exam as part of that and get credit.

I think it's not clear yet what the right business model is. I think there's a huge opportunity in the sense that the cost can be very low. You don't need the bulk of students to pay anything. You can have a free and open course that the vast majority of students taking it won't have to pay anything for.

They'll still get a lot of value and contribute to the community if there's a small fraction of students that are getting some more value and paying something for that. It's very early. None of the providers, as far as I know, are profitable yet. The right business model, both for the universities, and any companies involved, certainly remains to be seen.

Maggie:  I think it's really interesting too if you look at how this all started. It was at Stanford. They took three or four of their courses that they were basically flipping the classroom, and using technology in order to provide a better experience in class for their students. Because the technology was already a part of the in‑class and on‑campus experience, it was very easy to open it up to the world and just see what would happen.

They did that experiment, and it just took off. All kinds of things happened. It happened in such a way that, yes, some for profit companies got involved. As David said, they're not quite sure what they're going to do yet. It was something that took off in a way that was surprising. That's part of it, too.

David:  I have one more thing to add to your remark. We talk about MOOCs as this new thing that's happened in the last year. There's been open education going back decades. One of the most successful projects of this was the MIT open courseware. MIT put courses online. They got tremendous value in terms of applications from around the world of students who had already taken MIT courses and were excited to come to MIT.

They increased their applicant pool and the quality of it. They viewed it as very valuable in terms of just spreading the message of the university. I think there are ways universities can benefit from being associated with open courses, even if there isn't a direct revenue stream coming from it.

Ben:  It is very early and hard to tell where this is going. I don't think any of us know really where it's going. One thing I do think is we are going to see, down the road, is that there is going to be this decoupling of the education component and the credentialing. You're seeing this with MOOCs where the MOOC is providing the course. In some cases they are starting to provide a credential. That, perhaps, is a modernization strategy for the for‑profit folks and even a sustainability strategy for the non‑profits, like Kahn Academy, perhaps, is if this credential is something separate.

I think as you see credentials start to emerge, like these micro‑credentials that say, "Hey, you are a level three programmer," and you're certified by this well‑respected credential that has emerged from this process, whatever happens, then you can go anywhere and you can do that. You can study yourself. You can take a MOOC. You can go to a university. All of those things are going to have different benefits, depending on how you do it.

Then I think the for‑profit MOOCs may actually be able to demonstrate value, say, "Hey, we can prepare you for this credential that has a real meaning in the world." They may actually start charging at that point, perhaps. It's hard to say where this will all go.

Maggie:  One last thing is Google is doing something that is not academic. All of our courses, the internal or external ones, the ones that we're doing, they're short courses. Teach somebody to be a better searcher, how to use Maps more effectively. There's obvious business value in teaching our users to use our products more effectively. There's another thing.

David:  Worried about Google running out of money?


Maggie:  No, but it's a business model, right?

Valerie:  Right. A follow on to that, as we talk about the economic model going back to the title, to the leveling the playing field, as we talk about the economic model. If you can comment on your view in terms of do you see that, especially being that it is open, leveling the playing field in terms of more diversity, more inclusion, more women involved, more under‑represented groups, also people with disabilities? If you could, comment on that aspect as well. Ben?

Ben:  I guess I'll take this. This is all pure speculation. I'm not exactly an under‑represented minority here. I do feel like in some sense the story that I just told you all in terms of how I got stuck at some point in my education. I really don't know where I got stuck. I don't know why. I just know that there were these gaps that formed in my knowledge that eventually led to me hitting a wall and led to me getting kind of frustrated. Maybe I was sick one day. I don't know what it was.

I can certainly imagine someone who perhaps has a teacher or is in an environment that is not empowering to them, they get stuck by it. They get left behind in some sense. As I just talked about, once you get left behind, once those gaps start accumulating, it's really hard to catch up.

What I see, in terms of these things that are online, these things that are self‑paced and available when you are, can allow people who might otherwise get stuck or left behind to catch up, or to go at their own pace and even race ahead.

Valerie:  What about frameworks that also help with the engagement?

Pooja:  I spoke to a professor, Tucker Balch, who teaches a computational investing class at Georgia Tech, and he uses Piazza for the engagement component. His class is hosted on Coursera, but all the discussion is done on Piazza. Just quoting him, his takeaway after teaching this class twice now is MOOC's biggest weakness is the lack of interaction between faculty and students and students and students.

Often, as I said, women or minorities, the feeling of isolation can be exacerbated in a large lecture class where you think everyone else is excelling, but actually everyone is doing as poorly as you are. I really should have a bigger expert than I am, because I haven't taught these classes.

He did say forums in general are an essential method for filling the gap. For him, he found Piazza is an essential component for the online course, reflected also in feedback from the students feeling connected to each other while they were doing this class.

If we think about promoting diversity in these classes, letting women be known that there are other women out there, and also the boys are struggling as well.

Valerie:  Now we can take a question, too, from the audience. We'll go back and forth.

Audience Member:  I have a question. I'm curious about the recent announcement of Georgia Tech and its offering, or its partnership with Udacity and the Master's program in computer science and the potential, or the opportunity that that represents to not only scale workforce training, and to scale the technical workforce, but also the opportunities that that might represent for diversity and for inclusion and that. I wonder if any of you have an opinion on that.

David:  You said it very well, better than I could say it. I think there is a real opportunity. One of the things that I had hoped, in the early days, it seemed like you didn't need a degree. That was one of the opportunities of opening education was to say, "You don't need a university degree. There are other ways to demonstrate that you're talented and capable."

It has become more clear to me that, yeah, there is still this currency in academia that degrees really do matter. Providing a low cost degree with the open format will open up opportunities to students that couldn't afford a high cost degree where they had to be on campus for four years, and pay the kinds of costs that it costs, to do two or three years, to do a Master's at an institution like Georgia Tech.

There is also a plan to keep those courses completely open as well. The students who are enrolled and paying in the degree will get some extra services. Get some human‑aided grading and assistance that you couldn't afford to provide every student. But most of the course content will be free and open for anyone.

Ben:  I think this goes right to my point of decoupling of the education from the credentialing. You can go and you can take the course, and you can get the value from it. But if you want that credential, then you can pay for it. I think that makes perfect sense. That seems like a reasonable business model. Obviously, as these things expand, we may even see that cost come down even more. Frankly, I think that is very exciting.

Valerie:  OK. I also want to comment in terms of the question about Georgia Tech that in the audience is Charles Isbell with Georgia Tech, who probably can answer very well for Georgia Tech.

Charles Isbell:  Two things. One, on the credentialing question, I think it is important to understand that the way that we are structuring it at Tech is that only the people in the degree will actually be tested and assessed by Georgia Tech faculty. Even though the course itself will be open ‑‑ just like every other MOOC is, almost every other MOOC is ‑‑ those people will have assignments. They will have to do many other things. But they will not take the final exams and the midterms and they will not be graded.

I do think there is a business model there where you can imagine sharing course content across multiple universities or multiple departments where each one decides how to test them, and each one decides what an A is or what a B is. I think that is actually an important distinction worth making.

On the question of diversity, this is an empirical question, our current belief is that it will increase in absolute numbers all of the groups that we happen to care about. It is unclear whether it will increase in terms of percentages.

Tucker Balch, in fact, in an internal argument among the faculty about exactly this issue of diversity, pointed out that although, as a percentage, the number of women and African Americans taking his course was smaller than his typical classes. He had through his one single course taught more women and more African Americans than he had the entire 12 years he had been at Georgia Tech through all of his other courses combined.

I do think there is a lot of potential and there is a lot of possibility there. But it is going to require a lot of work by everyone involved to make certain that we bring everyone along.

Valerie:  Thank you.

Maggie:  I love that point because maybe it is a glass half full view, where there has been a lot of writing about the controversy about how many people drop out after they register. There is a huge drop off of people. When we look at our search courses, we have over 40, 45,000 people that have actually finished it. That is 45,000 more than we would ever reach any other way.

There is something very positive about that. I think it really is important that of that 45,000, it is going to be a larger set of women and under‑represented minorities than you would have ever reached otherwise. There is something to that, even though the drop off rate, and there's all these other scary statistics, the numbers are just so big that it's really interesting.

Valerie:  That leads us to another question that I'll ask as we talk about the sheer numbers, and you can talk about the numbers versus the percentages. But if we look at the in classroom versus the online, the MOOCs, if you could talk about some of the differences, and how some of the differences may be more engaging or less engaging with respect to diversity.

We can talk about the number, the sheer numbers being much higher. You can talk about the percentages being lower, both of those, in terms of contrasts, if we could talk about those differences in terms of the two environments.

Pooja:  Sure, I'd love to share. I have had the fortune to sit in on many professors' classes, and see how they teach their class. Eric Mazur of Harvard, for example, I've seen how he has small groups of people at tables and they're doing problems. Then they revisit problems after the answer comes out. Lorena Barba of Boston University does the Flipped classroom.

It's really fascinating to see how all these professors get very creative in the more traditional style of classrooms given the size. Maybe it's 50 people, 100 people, 20 people. They're able to have their creative styles of teaching their particular subject that in MOOCs is just not possible if you've got a class of 10,000 online.

I think it's important to Maggie's point of what are you enabling? If it's net positive, that's a good thing. Of course, it may not be able to match to what you can enable in a more traditional style of classroom.

Ben:  You're definitely enabling more people over all, which is great. But you're absolutely right. One of the problems I think, particularly with a lot of the MOOCs we're seeing now, and I think Udacity has probably done a better job than most of this, but they are very traditional in the sense that it's a lecture. Even when you go into a physical classroom, thankfully we're starting to see fewer and fewer lectures, as Pooja mentioned. In this day and age, I think there is absolutely no reason to collect everyone in a 300 person lecture hall and just lecture at them. That's a very dehumanizing experience.


Ben:  It is actually, in some sense, more dehumanizing than sitting alone at a computer, and actually having a conversation on Piazza or on some of these online forums. There you're actually interacting with people. In a lecture hall you're not interacting with anyone, you're just probably falling asleep.

I think there is some advantages to getting out of that lecture and the MOOCs aren't necessarily always doing that. What I think would be great to see is, as these systems evolve and as there is more technology that goes into them, these online courses can become, in a sense, more adaptive.

One of the things we're trying to do at Kahn Academy, is when you get through part of this course, a traditional class and even a MOOC now, if you get part way through the course and you hit a wall and you get stuck, the course moves on. Maybe there are some forums and some resources there, but if those aren't doing it for you, you have to move at the pace or you're going to get left behind.

I think one of the things the technology can do is it can provide the ability to slow down and work at your own pace. For folks that may get discouraged if they get into something and then hit a wall, having that ability to work at their own pace can actually be enabling, to see that, "I actually can do this if I work at it."

David:  One of the things that you get with scale is also an amazing contribution from students. In my small, in person classes, students contribute in the class discussion, there's certainly a lot of value from some students.

Once you get to a big online class that's open, there will be a handful of students that just make amazing contributions. That for any point that I make briefly in one video, they'll write up a ten page post in the forum explaining it in lots of details and answering all the questions from 100's of students in the class about it. That's something that you get with this open format, and the other thing is that it's there, forever, for every student.

The next time any other student comes to that video, they'll see a link to here's all the discussions about this. You can read all the depth that goes far beyond what's in the actual class, as well as help for here's the other 20 students that got stuck on this problem. Here's the different ways that they understood it afterwards.

There's a lot of value that comes from the students in the class, and it builds over time. I think it leverages in two ways with scale because, the scale really motivates students to contribute. Because, they get the social validation of 1000's of other students reading their contributions, rather than if it's a small in person class, a few students might notice.

Pooja:  I'll share an anecdote that actually builds exactly on this. Dave Betterman was an engineer at Facebook while I was an engineer there. Just a few months I got a Facebook message from him saying Pooja, Piazza's awesome, congrats on building this. He was taking Tucker Balch's class. Imagine you're doing a large open online class, and you've got world class engineers doing the class with you, and you can read their answers and contributions. That in itself is so powerful.

Ben:  In that sense, there's far more diversity in open online classes than there is any university class. Because there's students taking my intro CS class, and you heard from Michael earlier today, that have 10's of years of experience as a top level programmer helping answering from students just starting in their first CS class. That doesn't happen in university classes, and when there is that kind of gap in university classes, it's often kind of intimidating and overwhelming. It's not helpful. But in the online classes, it often can be very helpful.

Maggie:  It all speaks to the potential of this, because it's in its early stages. It's tried to simulate a lot of lecture based kind of things, a lot of traditional ways of teaching. But the adaptive learning possibilities, the amount of data you get from even one MOOC is just enormous. The kinds of things you can do with that data, the social element, having many, many students.

We're just beginning to tap into the potential of what this could look like that could be completely different from the way teaching and learning is done now. That's the evolution that has to happen. We're not really quite there yet, but we have a traditional MOOC now, which everybody knows what it is. We need to break away from that, and think of ways that we can start taking advantage of some of these capabilities.

Valerie:  It's very interesting, too, as we talk about the difference between the in classroom, and the MOOCs, and the online. Just the sheer number allows you that you can have groups. You can have that critical number. At one with in classroom, you always said you need to have a critical number of women to feel comfortable in the classroom. Now, where the percentage may be low, the numbers are there, such that you...

David:  There's the Facebook group of women taking the class, there's the community of students in Cairo taking the class. That's one of the thing that you can get is these sub‑communities that you could never get in a small class. Because there's just not enough common people that have the same schedule, and have the same interests to do that.

Valerie:  Right, while you may have a percentage that's 20 percent, that 20 percent may be two or three. Whereas in the MOOCs, you may have a percentage that's one percent, but that may be 100's. That sheer number and I know there was a question.

Audience Member:  I was wondering to extent your applications are accessible to people who are blind or deaf.

Maggie:  With Course Builder, we definitely have made it accessible in terms of disabilities, blindness disabilities. I think that's the main place that we've focused right now. Course Builder out of the box does provide those capabilities. One of the things that's a challenge though is that. Because it is a platform that anyone can use there's a certain set of steps that one needs to take as a content developer to make sure that the content they create fits into certain templates we provide.

But we can't enforce that, because it's an open source system, it's a free system. We have put all the basic framework in place for that to happen, but we rely on the people, the content creators who are using Course Builder, to finish it up. They have to do that final set of things in order to make it work.

Pooja:  We've actually been very fortunate. A few folks from UCSD San Diego, they offered to test and help ensure we were conformant. From the very early days, once we opened up Piazza beyond one or two schools, we were able to work with them and get them to test our site and make sure we actually complied.

Ben:  Khan Academy, I would say, this is probably an area where we could do a little bit more. We've got all of the videos subtitled and that sort of thing. As far the website, as far as I know, we haven't done very extensive testing. We definitely try to make sure that you don't need color acuity and those sorts of things. But we haven't done really extensive testing. That's definitely an area where, if there are folks here who could point us in the right direction, I'd be happy to talk with you.

David:  Similarly, when the first courses were done with USC we really didn't understand this as well as we should have. We certainly thought about it, realized it was important. Doing captioning is not enough, you've got to understand more than that, and that's something that has improved, but it's definitely something people are still working on.

Valerie:  It's a really significant engineering thing to take on.

David:  To make all the visual stuff, you can have a voice over try to explain it, but that often is not enough. You really need to understand the spatial relationships in the video to be able to answer the quiz.

Maggie:  But that's a good area of opportunity, with the online.

David:  In many ways, MOOCs are much more accessible to many types of students that couldn't function well in a regular classroom. I think there is a big opportunity there, but it has not been advanced to the point where it should be yet.

Ben:  To that point, we've got tons of anecdotal evidence from kids with dyslexia or ADD, and all these things that say Khan Academy is somehow getting through. They've tried all sorts of other things, and nothing else has worked. We're getting anecdotal evidence that that is working. There are definitely segments where we're in some sense better than what they had before, definitely a long way to go.

Valerie:  I know there's another question.

Audience Member:  I'm wondering, given the enormous numbers, potentially, that you are teaching to, to what extent are you able to monitor, engage in the conversations that occur in the different discussion forums, to ensure that the content that's being shared among students is still accurate, that there is fidelity to the content? That seems like it would be a huge job.

David:  That's rarely a problem. Technical correctness, if someone gives an incorrect answer, another student is going to post a better one, and it's going to be very quickly resolved. I participate in the forums. I know sometimes I feel like I can add something. I think students appreciate the sense that there's a real teacher there occasionally, but it's never because there's some technical point that I thought I needed to correct.

There's students in the class that know more than me about everything in the class, and they will be happy to correct other students or correct me if something is wrong.

Audience Member:  Is it expected that as a part of the process, the instructor is following along with an open schedule?

David:  In my role at Udacity was involved in many MOOCs. Some instructors were very active, and built a real great relationship with students. Others, the instructors were not as involved in the class. I think it did have a great impact on the success of the class, whether it's a causality, or it's more of a correlation. But I think students definitely felt more like putting effort into the classes when they saw that the person teaching it was actually involved.

Little things like having short recorded office hours, or being in the forums, or answering email I think do go a long way to making it feel like it's more of a personal experience than consuming content that someone created, and isn't involved in.

Maggie:  We've actually measured this, to some extent in terms of having TAs. For our courses on products and services we'll always have TAs involved in watching on the forums. We've done some control groups, where we've run courses multiple times such that we can determine that if we have that engagement, then you will have a higher completion rate. It's a positive thing to have some interaction with either the instructor or someone representing the instructor. It helps a lot.

Pooja:  One of the core design decisions in Piazza is the notion of a Wiki Answers. There's a single Wiki Answers that all students keep editing and instructors could edit as well. The feedback I get from professors is that students end up policing each other and it's like Wikipedia in that it keeps getting better and better if there was any misinformation provided.

Audience Member:  Related to that, can you comment on maintaining a respectful and engaging environment among the students, such that students feel comfortable.

Pooja:  It's phenomenal. I peek in on some of the large open online classes on Piazza. Phenomenal. The students will chip in saying why are you giving professors a tough time. They're trying their best, they're working hard. I think we should be respectful here, if you don't want to be here just log off. Don't participate. I found professors don't have to stand up for themselves, and it's just phenomenal, getting that sense of students watching out for the professors, that scale, it's really heartwarming.

Maggie:  We've had the same experience, it's almost nonexistent. If it is, you catch them and you get them out of there. But it's very, very small numbers.

David:  It's amazing how rare it is, but there are occasionally bouts between students in the forums. One thing about having an open class is that you do end up with 13 year old boys in the class, and they sometimes get into ego battles with each other. That can get ugly, we try not to moderate, or delete stuff.

But there have been, in the entire history of Udacity, maybe three cases where someone had to be removed from the forums, because they were antagonizing too many people, and not contributing usefully. But it's remarkably rare for how many 13 year old boys there are contributing.

Ben:  By and large I would say, most of the discussion, or all of the discussion on Khan Academy is very productive. We do have the situation where our median age of our users is around 14 years old. We get a fair amount of interesting posts, but basically the way we handle that is anyone can flag posts, and we have moderators that are volunteers that go through and clean those up very quickly.

If something gets a certain number of flags, it's automatically removed. It is a job to police that, and to remove some of the things that are just clearly off topic. But the things that are posted that are actually discussion, are actually pretty phenomenal.

Valerie:  I know there's a question.

Audience Member:  I agree, I think it's really exciting that more and more people are choosing MOOCs as a way to supplement their education, when it comes to CS and tech. My organization always encourages us to think big, and that the future is always closer than we think it is. Keeping that in mind, how do we evaluate the folks who are graduating from these MOOCs now as they're entering the workforce?

What are your thoughts around the way that we interview these candidates and how do they compare to someone coming from a traditional CS background like from a Stanford program, versus coming from a MOOC? I just wanted to hear your thoughts.

David:  I think that there's still currency in traditional degrees, but I think it should be, and it is certainly the case that many of the best computer scientists don't have a degree. I think as a field, we're very fortunate that computer science is probably more merit based, there certainly are problems. But it is a field where someone can build something and demonstrate what they can do separate from needing a traditional degree.

I'm still employed by a traditional university, so I think there is a lot of value that comes from that four year traditional university experience, for those who have the opportunity and can afford to do it. But for students who have enough self‑motivation and initiative, they should be able to build a portfolio that's just as impressive as students who have gone through a traditional university degree.

Ben:  I agree with pretty much all of that, in the sense that we are going to see this world where these independent certifications come up. Some of the MOOCs you're already starting to see certificates, and they're partnering with universities to provide different things. I think we are going to see this world of micro credentials that pop up that say, "Hey, this person is a great programmer. This person is a great writer. This person is whatever it is."

You'll start to see people that come in, and employers, being the ones that drive this, employers are going to say hey we value these credentials, because they target what we want. I think that traditionally, a university degree has been that signal that someone is someone that an employer should consider.

But it's been a weak signal. I think these micro credentials could be a much stronger signal, because they're specifically targeted towards what an employer wants. But I think that's half of it. I think the other half is this portfolio of creative works.

At Khan Academy, we're starting to create that, in the sense that we have our computer science platform, and when you create programs, they show up on your profile. If you look at someone's profile, you see oh yeah, they know algebra, they know trigonometry and everything. But also, here's this portfolio of creative work that they've created.

Here's all these programs they've made. Here's all their discussion history, you can see all the discussion that they've taken place. You can look at that, and we've actually done this at Khan Academy, going on two people that we've actually hired based, pretty much solely, on their Khan Academy profile and their contribution to the Khan Academy community. Those are the things that we value.

Maggie:  There's some challenges here though. Because if I think from Google's perspective, the 1000's and 1000's of applications and resumes we receive, to modify the way that we do our hiring and recruiting, in order to take into account all these micro credentials is a really big deal. Right now we've got great filters in place, in terms of the academic credentials that you bring. That allows us to deal with the 1000's and 1000's of applications that we have to process.

There's a transition that going to have to happen over time, it's not something that's going to happen real quickly. It is a high touch thing that you have to do. You have to deal with the person individually. With the kinds of things we're discussing here you would have to assess them in really deep ways, in the moment during an interview. That whole structure has to get set up in some way to do it.

Ben:  See, that's what we're doing now though, we are doing these really deep assessments in the interviews. Our interview process I think is not too different from Google's and we get 100's of applications a month, and we're only a 40 person organization at Khan Academy.

Our interview process, essentially, we put you through five hours of writing code, and trying to see. Even that, it's not a great signal to us, it's not a great indicator what someone can do in five hours, as these portfolio of creative works, and these other things, which I agree, is much harder to evaluate.

But if the industry can come up with a gold standard credential, that is much more rigorous than anything we've seen so far, maybe it involves an oral examination, practical examination, things like that. I think that does help.

Pooja:  We may also find unique opportunities come up. For example, when I was trying to find some help with front end design work, I started looking through forums and seeing who was participating. You may find a four person team somewhere in India, trying to start a company. They go on to this large open online class, see some students who are participating, they're in India. They partner together, they start the company together. There could be really fascinating other benefits that are impossible today.

Valerie:  I know there was a question here.

Audience Member:  My experience is that the number one challenge when teaching a college course is when you have a wide variation in student's background, ability, and goals for taking the course. It seems like MOOCs are going to face a much, much bigger challenge because they have so many more students in the course.

I noticed the statistics you have up on the board, we're all very impressed with what's been done so far, but what you're showing over there is that, by one measure, only 10 percent of the students finish your course. By another measure only 38 percent, and that's even with the advantage of self‑pacing.

At a traditional university that would be considered a disaster. I'd be fired if only 38 percent of my students are finishing the course, because my Dean would ask me, what happened to those other 62 percent?

I wonder what your thoughts about when MOOCs become the centerpiece, if they do, of a traditional education where you're responsible for not just the 10 percent that finish, but the 100 percent that start? What techniques do you foresee being used to really do mass, not just enrollment, but mass education?

David:  I guess I should mention that my first in person Intro to CS course, the completion rate was about 40 percent as well, and that I somehow didn't get fired. But that's obviously not what we aspire to in, in person classes, or MOOCs. Some of this is that if the barriers to get started are really low, and there's no penalty to leave.

The completion rates are going to be lower than they are for an in person class where students have to go to their advisor, and are paying tuition, are much more tied into the class, and are in many ways a captive audience and have to go through a lot of hoops to get out of class.

The completion rates for online classes where students had to pay tuition at the beginning are comparable to what they are for in person classes where people have to pay tuition at the beginning. In that sense, I'm not saying we shouldn't be trying to make the completion rates better. But I think comparing them to in person classes, where students have paid tuition, and have to beg to their adviser to get out of their class, is a pretty different kind of scenario than making it very easy to start a class.

Certainly, part of making the classes easy to start means lots of people will start and quickly realize this is not the class for them, and they should take something else that counts.

In terms of the stats you have there, about half the students never do anything. Maybe that means the enrollment numbers are really inflated from what they should be, because you're counting students who didn't even watch a video. In that sense, the completion numbers are including lots of students who didn't do anything. It's also, definitely something that we're very early in.

One of the things we do have is tons of data to see where students are leaving. Trying to understand why from that is much harder. I think it's still very early in this. I'm not real concerned about the completion rates being low.

Pooja:  You touched on a subtle point, which is if MOOCs is replaced. I think that won't happen. Many students, their parents will make sure they go to a four‑year college. They pay a lot of tuition, and they actually complete the degree with no dropout rate through. This is maybe enabling 200 million people who can't go to those four‑year colleges in America. They can learn some courses.

I often am touched by women out there who now have kids. They are past their four‑year college degree. They didn't choose to study CS. Now, at the age of 30, they're wondering if they can start learning how to code because maybe they can pick up a job that's more flexible for them to have kids and be a mom and also have a job.

We should be thinking of this as opening up the size of the universe, not taking from one to the other and having the same struggles. Let that be its own struggle. Let there be good completion rates in class in traditional colleges. Let this now enable hundreds of millions of people around the world who otherwise maybe didn't get that opportunity when they were younger.

Audience Member:  Related to the retention rates, you realize this and also the numbers are very different by orders of magnitude in terms of in‑classroom versus an open and online.

In terms of when you start to look at the percentage in terms of retention, do you think that can be a discouraging factor for women or minorities? Do you find a difference in the retention with women or minorities, or students with disability with online platforms?

Ben:  I think that would be very interesting to study. We do have that data. We just haven't looked at it. We have so much data, as do all these MOOCs, and it almost overwhelming. It's fascinating that we could study that, and we could look at that. It goes back to my point, before, of the self‑paced nature of these things, and not all MOOCs are self‑paced. The synchronous ones are not, but if you do have something that is self‑paced, I think it can actually be encouraging, in the sense that you can move at your own pace.

You can progress through it, even if you're discouraged, and actually going into the previous question about dropout rates, I think there are things we can do to increase engagement and keep you coming back. Gamification is one thing, or even just getting better feedback, and showing that, "Hey, you are making progress. This work that you're doing, you're getting better. Something is improving."

Provide you some positive feedback, that what you're doing is providing its value. I think that's an important component of that.

Valerie:  To look at what's providing value for different groups, and to somehow tailor...

Ben:  I think that's an open question, for us anyway, is to figure that out exactly. That's like, we have this data, it would be possible for us. I think we should go back and look and say, "OK. Hey, is there a difference between these different groups, where one group gets stuck versus the other group. Or, if we change the way that this is messaged on the site, or that we've changed the way feedback is given to one group versus another group, do that affect their engagement?"

We're doing those tests now, but we're doing them in aggregate. There's certainly no reason we couldn't do them, based on different demographics and that would be fascinating to do that.

Valerie:  Right, I think especially the messaging.

Ben:  Yeah, that's a great point.

Valerie:  I know others may want to chime in there, about the retention.

David:  Yeah, it's something that should be studied, has not been studied. I think there are lots of opportunities to do experiments, as well as, to change aspects of MOOCs and find ways that work better for different types of students. That's something that is still very early in, but it's something that an organization like NC, which should be pushing providers and universities to do those kinds of experiments, and figure out if there are gaps in what MOOCs are doing now, that can be better understood.

Valerie:  I think I know there were questions from the audience.

Audience Member:  I think this is really interesting to hear from the for‑profit and nonprofit corporation perspective on the problem MOOCs. I understand that both are trying to make education more accessible to everyone. But it seems like for‑profit corporations are more focused on PC‑users and their own business products.

I was wondering if you guys could speak on the potential of both for‑profit and nonprofit corporations working hand in hand in solving this problem of open online courseware.

Maggie:  There's a couple of different things that we're doing at Google. One is providing different kinds of educational opportunities for our users. That is obviously core to our business and it has a business value to it. There is another aspect to what we're doing with Course Builder that we feel is much more supportive of just open online education. Because one of the things about Course Builder is that it is open source. It runs in the cloud. It's free.

The reason why Google did this, why we open source this platform, is because we really wanted to make it possible for everyone else to get involved with MOOCs, and get involved with online education. There are many colleges and universities that are not participating with Coursera and Udacity.

There are nonprofits that want to do MOOCs. There are K‑12 teachers that want to do them. There are people who want to teach their neighborhood how to cook. Everybody should be able to do this.

One of the reasons why we have put Course Builder out there is certainly to support everybody else other than higher ed, which is where the for‑profits are really focused but also to really make it possible for anyone to be able to teach and learn. It's very much a part of why we wanted to do this.

Ben:  Kind of your courses is a not for‑profit and our mission is free world‑class education for anyone anywhere. For now, a lot of the MOOCs, they are free and so in a sense that is completely aligned with our mission and so we're very supportive of that. It remains to be seen what happens in the future. There are investors that have put money into these for‑profit organizations and for a couple of years I'm sure they have some patience.

Four or five years down the road they don't want to see some type of return, and it remains to be seen how that works. For now we're very excited to work with anyone who is trying to get free education out there. That's certainly our mission.

Valerie:  I know there was a question.

Audience Member:  There's been some research that shown that when people have a bad experience in a course, if stereotype says they won't be good at the topic, they'll think that's confirming the topic where stereotypes do not say that. They'll think and say, "I just didn't work hard enough." As I look at these numbers, I see a hundred thousand people that had a bad experience from computing.

That seems to me very troubling. Are we doing anything for those people who drop out to try to make them realize that they could have succeeded?

Maggie:  One thing I think is really important to recognize with any of these numbers like the 94k are people who registered for the course. We have the same huge numbers for our courses, so you got this huge number of people that pressed the button to register and gave their email. End of story.

Then they have to realize that actually this is a real course and I'm going to have to put aside this much time. You're going to lose half of the people who registered, because they didn't realize how much time they're going to have to devote to it.

I just want to caution about reading too much into how easy it is to register. There's no barrier. That's why the much more useful statistic is the group that actually came and did something. They did the first homework. They watched the video. They got involved, because at that point you can start to think about is the content turning them off.

Is that why they are dropping out of that point? That huge drop‑off from registration, we're not concerned about that. It's the people who start and don't finish. That's the group we're much more concerned about.

Ben:  In fact, there are a number of MOOCs where if you register for the MOOC, then you have access to the content after the MOOC ends. If you don't register for the MOOC, then you don't have access to the content. There have been a number of MOOCs that I have registered for because I'm like, "Oh, that looks interesting."

I know I don't have time to take it, but I might later want to go watch those videos and the only way I'm going to have access to those videos is if I register for it. Maybe they're doing this to inflate their members, I don't know. That certainly is pushing those numbers up.

David:  At Udacity, courses are always open and that content is always open.

Ben:  Yeah, yeah, I know. I needed to be clear.

David:  The question is a good one, though. I agree with what Maggie said that the ones who don't put any effort into it, you just say well, that's unfortunate. But it is the 60 percent or so that put enough effort into it that it seems clear they've wanted to learn what the course was about, was willing to make some effort to it.

I don't think not finishing, is necessarily a bad experience. I've talked with many students who said "Yeah, I got what I wanted out of the first two units, the first three units, and that's enough to be able to start writing some simple Python programs, and understand something about how a search engine works." They didn't feel the need to learn the deeper computer science things that go into the later units.

As a computer scientist, I would like them to learn everything in those and stay. Even what it means to complete the course is fuzzy, because new content keeps getting added to it. In that sense, there certainly is a large number that, I wouldn't necessarily say are having a horrible experience, because if it is a little bit bad they can switch to watching cat videos.


David:  It's not like if they're enrolled in a course in a university where they have to stick through it and really have a horrible experience for three months. But there are students who could potentially be attracted to computer science and aren't being attracted by this particular course. That something certainly I hope we can learn from.

You can see anecdotes where students post in the forums, "Oh, I want to quit because of this reason." Often you see great responses from other students convincing them, "Oh, stick with it a little more," and then they come around. But that's a pretty small fraction of the ones. Most of the people who get frustrated are just going to go do something else and you'll never really understand why they left.

Pooja:  I think it's really also important to understand the background of these people and it goes into psychology. But I just had a baby 10 months ago and I was trying to lose all the weight. The people I'm working out with, they're like, "Yeah, often when you try to lose weight you try six times. The sixth time it sticks and you get on a diet that works or something."

The same idea, these people haven't given up their fulltime job and are back in school. They're also trying to juggle motherhood or trying to juggle a job. Maybe the sixth time when they go and sign up for this class they'll actually finish it. Understanding the background of these people is really important.

David:  Another thing I think that this has the potential of, and it has already happened, is that there are lots of alternative ways to get started. If they find my course too tedious, and dull with too much black and white, they can try the Kahn Academy course and get doing much more exciting things right away.

What I hope will emerge is many different paths. Then you'll find what's the best way to direct students to the path that is going to work well for them. I think we don't know enough how to do that yet. But certainly from anecdotal discussions and if they reach out to the right people they might get directed to, "Well, you're frustrated in this class for this reason. Maybe you should try a different approach."

Valerie:  Right. Well and related to that, let me just add how much data do you collect when students leave a course, I guess? In a classroom you know when a student has dropped your course, because of registration. You know what a student's grade was at the time of dropping your course. You have a lot of information available. If you can comment on how much data do you collect or is available when you start to talk about students just not participating.

Maggie:  We surveyed for a couple of search courses we've surveyed that big group that never came, that never started. They registered, but they never started. They don't really do the survey, then either.


Maggie:  Some of them did. We did get enough information on them. When you're talking about 150,000 people, if you get 20,000 of them to answer, that is a pretty good sample. By far the majority, it was "I don't have time." That was 85 percent or so of the group that did not start. We've done it a couple of times now and the same percentage, same reason. That was consistent.

Pooja:  We did a survey of the women at WitsOn. The pattern at WitsOn is most people read contributions, so they didn't necessarily ask a lot. They did name their fears. We had one post that said "Name your fears." It was phenomenal. All these women sharing all of their fears, and these other women, mentors in academia or industry were sharing their own perspective on how they had the same fears, actually.

But the exit survey, the two biggest takeaways, one was 47 percent felt more motivated to continue pursuing STEM. These were undergrad women in STEM. 50 percent felt equal, and I think it was one or two percent that felt less motivated.

The second interesting stat was 92 percent of the women would want to come back in a subsequent program around women and technology, and 8 percent were not interested. That was also really encouraging for us.

Valerie:  I don't know, Ben, if you wanted to add?

Ben:  Yeah, it's a little bit different with Kahn Academy because we don't have courses per se. Our activities are fairly discrete. You can come. You can engage and watch a video. You can watch a series of videos. You can come and engage on a couple of exercises and earn proficiency in a group of exercises or just one exercise, really.

We see a lot of different use cases. Some of them are just, "I have a test tomorrow. I need help." They come, they engage, and then they leave. That's not a bad thing. I think we're providing value there.

But we are looking at how can we get people to come back? We do collect a ton of data. We collect every video that someone watches, when they paused it, when they restarted it, things like that. That gives us a lot of information about, "Hey, everyone stopped watching this video at minute five. What happened?" That gives us some data there. We do the same thing for our exercises. We collect right answers, wrong answers.

We do have this data that we can look at. We're just starting to focus on getting users reengaged. We just recently started collecting emails from users. We're looking at ways to email them and predict, based on their activity on the site, other things that they might be interested in, and emailing them and saying "Hey, did you know we have this video on finance," or whatever, "this other thing that might interest you," and hoping that people might come back and reengage through that as well.

Valerie:  OK. As we wrap up our panel, looking at disruptive education technologies, if you can each comment on what you think are some good things to be done to increase inclusion for women, minorities, and people with disabilities in MOOCs or online environments. Also, if you can, comment on, in particular, NCWIT. If you can, comment on recommendations for NCWIT. What we'll do is we'll start with Maggie and come full.

Maggie:  OK, let's see. I think that there is a lot of experimenting that needs to be done. One of the things that's been really clear with the discussion up here is that we all have a lot of data. We all have the means for understanding what might be causing women and underrepresented minorities to either be engaged, or not engaged in some of these kinds of learning experiences.

I think we have to look at that data and really figure out what it is that we can do that goes way beyond what we're currently doing, which is a traditional video, exercise, whatever, that kind of linear sequencing.

What are the other ways that we can bring more of a social aspect to it, how we can really enhance that part of it? One of the things that we've learned just with our own internal engineering education is that our women engineers really want to work with study groups, whether it's just women together, or just to have a group that they can go through the content together with.

That's something that we really want to make it possible. If you're in 150,000‑person MOOC or a much smaller class, how can you find others that you can work with, whether it's in the same room or across the globe, but make that virtual room possible?

Once we are able to start moving in directions where we can really see who engages with that, and who actually takes advantage of those kinds of features, we can just make it that much better, and make it that much more conducive to supporting those kinds of collaborations.

That's one thing I know that at Google we're very much looking forward to and are starting to look at. What are the features that we can add to course builder or that can be added to MOOCs that are going to make this more conducive to different types of groups? NCWIT I think is obviously a group that has a lot of this kind of expertise. Being able to tap into that is going to be something really, really useful.

Ben:  Yeah, and I think from Khan Academy's point of view I would echo a lot of what Maggie said. We have a ton of data, and we really need to start looking at what that data can tell us, and find better ways to engage with users based on that data. See what differences exist and all of that.

I think one of the things that we are doing that has been quite effective is what I've been talking about all along, is this self‑paced learning where folks can engage and move at their own pace. We've seen that to be hugely empowering in the sense that someone who may have been left behind in a traditional classroom or maybe had some bad experience or some discrimination.

Or some type of event happened, where they were taken out or discouraged, or something and then they got caught behind and never caught up. I think the ability for them to re‑engage, move at their own pace, catch up, and get ahead is huge.

You're all change leaders. You have organizations that you have some influence in. One of the problems that we have as Khan Academy is when I speak to a room like this I asked everyone, "How many of you are familiar with Khan Academy?" Almost every hand in the room went up.

But if I go to an inner city or somewhere where there's underrepresented minorities, and it's not some kind of Khan Academy event, nobody knows who we are. We're providing free tutoring, and this is what those folks need. What I would ask of you all is really just to go out and raise awareness that, "Hey, there's this free tutoring thing that exists and you can engage in it."

Helping set up after school programs in the different organizations that you're involved in would be huge. That physical environment where students can come together and work together is really key, because you get that peer validation. You can work at your own pace through some of the things on Khan Academy.

You can master certain concepts, and now you are a peer tutor to your peers because you've mastered this thing. That's very empowering to be able to teach others. That's, I think, a big way that you all can help us.

Pooja:  I'd like to share again what inspired me to start Piazza was feeling isolated, and I never wanted any other woman specifically studying a stem science/engineering/math subject to ever feel isolated. Today I've noticed that our students on average spend three hours a night on the site, and this is boys and girls.

I believe that from my own empathy for a student who can feel very shy or scared I've been able to build a product today that brings students together, classmates together. It goes back to whatever movement and however we think about getting education to the masses, how we can create a very comforting environment for them so that they feel confident and want to finish.

I really believe women studying computer science who are able to code can create a future for themselves that is very favorable to a balance between family and work. I mean I've seen that in my own life. When I was pregnant in my first trimester, I would go home and sleep all the time. The team was very understanding.

I want to be able to create such opportunities for more women to being very understanding that minorities need a bit more of a gentler hand, and understanding that they perhaps feel shy in ways that the men may not be able to understand.

David:  I think open education has this tremendous promise and potential, and we've heard a lot. Michael and Ben and others have talked about privilege. If you look at the students that are being reached by open education now, a very large fraction of them are coming from privilege.

You can see on this slide that's been up for I guess for the last hour and a half or so, that little tiny blue slither is Africa. 2.2 percent of the students who've finished CS 101 are from Africa. About 17 percent of the world's population is from Africa.

In terms of looking beyond the superficial diversity that we tend to look at in the US, I understand NCWIT has a mission focused in the US. Looking at globally this opportunity for open resources, open education, to reach parts of the world that are underprivileged, and don't have the kinds of opportunities that almost all of us have in the US, that it's not happening yet.

But the opportunity is there, and I hope that people will put pressure on both the for‑profit companies as well as the non‑profits as well as the universities to be aware that this is the huge opportunity that open education has. Something's wrong if five years from now our mix of students still looks like what's on that slide.

The more narrow thing I'll mention that I think many of the people in this room can contribute to is the subtle messages about whether someone belongs or not that happen in IT. I think very few of the people teaching computer science are outright sexist. I hope that's the case, and I think it is true. But there is also a pervasiveness of language and examples in ways of talking that often give women the impression that they don't belong.

One of the things I tried to do when I was at Udacity, I was reviewing scripts for courses and was looking for things like using generic "he," using "ninja," using "you guys," using the kinds of language that is commonly used in every classroom in computer science in the country.

Maybe not every classroom, but a large fraction of them, that gives many women ‑‑ especially if they're in a class where they already look around and see that 10, 20 percent of the students are women ‑‑ the sense that, yeah, they don't belong here.

That's something where I think the openness and the public nature of MOOCs mean that there is more of an opportunity to say, "Well, this is a place where we can fix that, and pay close enough attention, because this is going to be visible to more people." I hope I succeeded in my courses in not doing that. There are probably places where I was not as aware of cultural references or things that sounded exclusive as I should have been.

That's something where I think there is an opportunity and a value that people that are attuned to that can provide to people creating courses. Once the courses are there, one of the values that a recorded class has it can keep being changed and improved. Noticing things like this, and putting pressure on people to fix them.

I think that's a fairly small thing, but I think that people in this audience have the right attitude and can play a big role in changing that.

Valerie:  OK, so with that, let me just give our NCWIT t‑shirts to say thank you to our panelists.



Transcription by CastingWords