For our 100th episode, we invite AI2 CEO Oren Etzioni to talk to us about NLP startups. Oren has founded several successful startups, is himself an investor in startups, and helps with AI2's startup incubator. Some of our discussion topics include: What's the similarity between being a researcher and an entrepreneur? How do you transition from being a researcher to doing a startup? How do you evaluate early-stage startups? What advice would you give to a researcher who's thinking about a startup? What are some typical mistakes that you've seen startups make? Along the way, Oren predicts a that we'll see a whole generation of startup companies based on the technology underlying ELMo, BERT, etc.
Hello and welcome to the NLP highlights podcast where we talk about interesting work in natural language processing.
This is Matt Gardner and Walleed Ammar. We are research scientists at the Allen Institute for Artificial Intelligence.
Okay. This episode is geared towards helping NLP researchers and practitioners who are interested in starting a company or joining an early stage startup. So today our guest is Oren Etzioni. Oren Is the chief executive officer of AI2 including the research lab and the incubator. I learned a lot from Oren during my time at AI2 and I can’t think of a better person to help researchers carve their way in the startup world. Over the years Oren co-founded several startups, including ClearForest, which got acquired by Thomson Reuters, Farecast by Microsoft and Decide by eBay, he is partner at the Seattle’s a venture capital Madrona and is actively helping severally IBS startups in the AI2 Incubator. So welcome to the program.
Thank you Waleed and I should say that I’ve learned a lot from you and from Matt. So pleasure to be here.
Thank you. One might argue that successful researchers and successful entrepreneurs have a lot in common. Both tend to be good at identifying important problems, come up with original solutions and work hard to see their ideas through and finally convinced the other stake holders of the significance of their contributions. At the same time, my anecdotal evidence suggests that only a small fraction of the researchers follow that path. What are some of the complimentary skills you think researchers need in order to also be good entrepreneurs?
It’s a great question and I would say that I very much agree with you and I would add that researchers are poised to be leaders in startups. Another huge thing with a startup is uncertainty, right? Are you comfortable in a situation where there’s all kinds of questions we don’t know the answer to? We have to try. We have to experiment. We have to iterate, and I think as researchers, we’re used to that. We’re used to defining our density. We’re used to not just answering the questions, but also posing the right questions. So I think that’s a great thing. I do think that sometimes people in academia have a certain degree of conservatism. I know this because you know both my parents were professors and as somebody who got into academia, I was a professor at UW for 25 years. Startups initially were the furthest thing from my mind.
I was like, Hey, what I want to do is get tenure, but I realized over time that there is a really fantastic and creative enterprise in doing a startup. It’s not all okay about the money. I’m going to do a startup because I’m going to make a lot of money because plenty of startups don’t necessarily become multibillion-dollar successes, but it’s about working together with a team to launch something very exciting. A second point that I would highlight is that I think people aren’t always aware of just how supportive and receptive of an ecosystem we have here in Seattle at the AI2 incubator, but more broadly in the Bay area, even in Pittsburgh and other places, there are a lot of Angels, VCs, people who founded companies before who were very eager to talk to super sharp technical people and to brainstorm startup ideas. So I think all those are things in favor of it.
I would say one reason that it’s not necessarily what say 50% of the researchers do is that it does require a level of commitment and a level of willingness, for lack of a better phrase, to get on a rollercoaster and to ride the ups and downs with tremendous speed while not letting go. So I think that there is a lot of stress in a startup that’s even more than the stress of trying to get a paper in by the deadline and you know, gnashing your teeth over what reviewer number two said. It’s the next level of stress and not everybody’s up for that.
Yeah, that makes sense. Another aspect that I would like to ask about your personal experience with, is how do you go from being a professor at UW thinking about writing grants, teaching advising students to creating the head space and time necessary to actually pursue, a new start up,
Well, I think the best startups come from passion. It’s so hard in the beginning to know, gosh, this is going to be a huge business and we can look at examples. For example, Google, which is a startup that started at Stanford. It started with a famous research paper on page rank. They didn’t have a business model, they didn’t know how it was going to end up, but they had a passion for the topic, for searching the web with super high quality. And so I think that that’s a great starting point and you have to realize as an entrepreneur, particularly early on, you’re not going to have all the answers the same way when you start digging into a dataset or a research question, you don’t know all the answers and that’s fine. You make it up as you go along. One of my favorite phrases is you’re building the airplane as you’re flying it. That’s, that’s a startup. And so where you want to start and that you can do easily, even in the context of all the academic issues as you described, you can do that if you have a passion for a particular subject matter for a particular technology and you say, wow, this could really make a difference.
Got it. So I guess the main additional skill that you suggested that people actually need is be willing to sacrifice a lot of their time and work, having a passion for the startup they’re interested in.
I think it’s, it’s a passion and it’s the desire to take these really cool thing that you’ve built or that you’ve contemplated and bring it to the real world. So it’s one level of reality to say, I’ve got a good idea, I’ve tested it on a dataset. I’ve gotten state of the art performance, I’ve analyzed it, I’ve written a really nice paper explaining it. Maybe, hey, I even have a demo that takes the work to a certain level, but then you say, I can use this to say, write better job descriptions or to build a better search engine or to build a machine translation program that millions of people will use. All of these are examples that have led to a product and start up. To take it from the paper and demo stage to a prototype and ultimately to a product requires not just passion for the subject, but passion for really getting it into the hands of people. Right? Because if ultimately people or organizations are not using your product, then nothing happens.
Yeah, it makes sense.
Yeah. I guess at some sense. A researcher’s job is to take an idea from inception to prototype and entrepreneur’s job is to take an idea from prototype to actual user’s hands. Is that fair? Like you, you need different skills and it’s different motivations for doing both of these things.
I think that’s a great observation. The thing that I would say is it’s still a journey, right? So it may take us a year or sometimes more, when we get from our initial research idea, initial exploration and intuitions to writing a paper. It can easily take a year or more before we get from, let’s say, our prototype to something that we can expose to users. So I don’t want people to think, I don’t have that. So I can’t do a startup. Let me mention a very specific case in point because we’re talking very abstractly. Another thing specifically for NLP is the emergence of BERT, right? We had a ELMo, which was created at AI2, and that led to BERT. BERT then led to Roberta and many of these other transformer based architectures. The point is that now we have a real discontinuity in the abilities of NLP systems, right?
These types of systems with some fine tuning are getting very strong results across a wide range of tasks. Usually if you follow up on that, the next thing that happens is products that are based on the success. The fact that we’ve created the success in NLP means that very soon, right now and in the near future, there’s going to be a generation of startups based on this technology and that’s a very fantastic current opportunity. So if you understand transformer based architectures and you say, gosh, I’d love to get NLP to make the world a better place. Maybe in the context of sentiment analysis and the context of translation, there’s so many potential applications. If you’re interested in that, then that’s the time to start thinking about a potential startup.
Yeah. So when you think about assessing early stage startups, what kind of factors do you consider as an investor or as a stakeholder?
Well, the first startup I did, spinning out of the university of Washington, we had really great technology. This is back in the mid nineties so this is eons ago. We had a phenomenal technical team. We had great ideas, but we didn’t have a business team and I thought, what does it matter? You know, we’ve got great technology and it turned out that we suffered a lot for that. It turns out that the business side is as important, if not more important than the technical side. So I would say that you look at first of all at the team, at the early stage, that matters a lot. And team means both the technical team and the business team. You look at how well they get along, right? You know the line “there’s no me in team.” If folks get along well, that’s important. If there’s a lot of tension between them or a conflict of discord early on, that’s a bad sign because things, right, once they get on the roller coaster, things are just going to get tougher from there.
So you look at that and then you look at what’s called the pain point. So do they just have a technology? Sometimes companies start that way or they also have an idea about how to use that technology to solve a problem, a problem that matters to people. Another famous saying with startups is; “is this a vitamin or a painkiller?” Sometimes people say, yeah, I’ve got this problem, but it’s like it’s kind of nice to have, Hey, it’d be nice to prioritize my email. Sure. But is that really something that I desperately need that I would pay for, that I would stop using Gmail or Outlook because you have the ability to sort my email, you know, using a NLP methods, maybe not, but there’s other situations where you say wow, this is a real pain point. So my most successful startup, which was called Farecast, predicted airfare fluctuations.
And it dealt with the question of when do I buy my airline ticket? And we wrote a paper on it initially back in 2003 it appeared in KDD. And in response to that paper, we got a lot of people saying, Hey, I’m really tired of the airlines messing with me. I want to know when’s the right time to buy my airline ticket. Because sometimes I look at it two days later after I bought it and the price dropped by $200 and that really, you know, makes me mad. So we got a sense that this was a real pain point. If we could build a product that would tell people when’s the right time to buy their airline tickets. That’s why the company was called Farecast combination of fare, you know, airfare and forecast. So if we could solve that problem for people, then they would really be interested. They would pay for it, they would come to our website, et cetera. So it’s that combination of a genuine pain point and the technology that could help solve that pain point and a team that suggests that you have some of the key ingredients for an exciting startup.
Yeah, thank you. That makes sense. So I guess when you think about some of the more recent startups, which look very successful, like Lime and BERT, which do like micro mobility kind of services, more of a vitamin, right? Not something that you need to do.
You know, that’s where some judgment really comes in. Let’s look at Uber taxis, you know, we use them, we use buses. I didn’t have a notion that could be radically disrupted by much cheaper, much more plentiful platform, but people who are better entrepreneurs than I looked at that and said there’s a tremendous opportunity here that has to do with the availability of GPS and phones and that sort of platform. And they designed the system that it turned out to be a real painkiller. It turned out that there was a lot of pent up demand for transportation that we didn’t realize was there. And of course the same is true for Lime. And so on. So I think part of the genius is to realize where pain points are that you might not see them. I saw a talk from a guy at Apple who pointed out how annoying the little stickers are on fruit and said, why do we have to have those?
Now again, I don’t know that we need a start up for getting stickers off of fruit, but when he said that, I was like, wow, I didn’t think about that. It was right in front of me and I didn’t think that if somebody solved this problem, I would be happy. Now again, that’s probably more of a vitamin, but I don’t know, take simultaneous translation. Again, going to NLP topics, wouldn’t it be nice? And of course, a lot of our best ideas come from science fiction and this is straight from the Hitchhiker’s Guide to the Galaxy series. Wouldn’t be nice if when I put my AirPod in my ear, it just translated simultaneously what the person in front of me was saying in any language. Right. That’s a great a startup idea. Wouldn’t it be great if we could use a lot of the gamification ideas that have come up over recent years to build a language learning apps that really keep you motivated and engaged.
Oops. Ah, that’s a company called Duolingo, spun out of CMU out of Luis von Ahn team, right. Who was one of the you know, huge giants of gamification. Now that’s last I look, it’s a billion company, billion dollar market cap. So those opportunities are around us everywhere. I could throw out lots of different ideas for NLP startups. Of course, some of these are proprietary, but let me just mention another one. Think about all the contracts that sit in Dropbox. Okay. You may not have a lot, but the typical organization, whether it’s AI2, or a bigger company like Zillow or Redfin in Seattle, which of course themselves were startups not to long ago. Really any, any organization a university, others has a whole bunch of contracts sitting there in some Dropbox or some other folder and you don’t have the key metadata associated with those.
You don’t know. When do they terminate? Who’s insuring it, who’s the customer, very basic entity in relation extraction type of ideas and if you were able to extract that information automatically with high precision and recall from the documents, you’d have a product that would be tremendously helpful to the organization, to the legal departments, et cetera, and so spun out of AI2 a company called Lexicon.AI does exactly that. They’re getting great traction with customers. They have a paper on their work in NeurIPS this year, so they’re pushing hard on that technology and off you go.
Makes sense. Other factors that people sometimes consider are things like IP or whether this team has published on this concept before or not. How do you evaluate and assess these aspects?
Again, I want to emphasize to the listeners that you often don’t have all the pieces in place, so if you have a passionate team, smart people, you have problems you’re interested in, you may develop the IP as you go. Even an amazing company like Google, how much IP that they really have. When they started at the very beginning, they had something called BackRub at Stanford. They wrote some paper, you know, it got into WWW. I mean it was not obviously the conglomerate it is today. So in terms of evaluating the IP, I think it’s great to have a demo system. I think people love leaders in the field. So if you’ve published, if you have strong credibility, even if you’re early in your career, I think that’s great. I think if you’re a grad student and you have a hankering for this, it may make sense to to join a startup particularly early on.
So, you know, not everybody has to be a founder has to grow through the very difficult phase of, you know, can we get funded that tremendous uncertainty. One way to get into it is to join a startup. You can join early stage. You can Join even, you know, series C there might be already a hundred people working there. There are many stages in which you can join. But the thing I really want to emphasize is that one of the things that still works great in this country is the whole startup ecosystem. A lot of startups being created all the time. A lot of passion, a lot of energy, a lot of funding, a lot of knowledge about how to take a startup from A to B to C. So it’s a wonderful opportunity when I’ve given talks about this at the university of Washington, I like to say to undergrads, the grad students, everybody should do a startup sometime in their lifetime. And I think as you both know, it’s always a lot easier to do it early. Once you’ve got kids and a mortgage and things like that, you’ve got responsibilities. It’s harder to say, I’m just going to launch myself into an exciting adventure with, you know, this big roller coaster ride. But since, so the best time everybody should do it and the best time to do it is early on in your career.
Yeah, that makes sense. And even though many of the very successful startups were co-founded by people later in their age I think several of them made several attempts before when they were younger.
Another piece of advice that I love to give people ask me. Okay. What’s your advice for having a successful startup? I say go immediately to your second startup because the first one is often a huge learning opportunity, so you’re absolutely right Waleed. Sometimes we see these success stories and we say, wow, how did that person do that? But often behind that is previous failures, previous iterations these startups aren’t like Athena springing full-blown from Zeus’ head and Tada. There’s a lot of twists and turns involved
And so what if I’m say a grad student applying for jobs or maybe a researcher who sometimes feels like the research cycle feels a little bit repetitive and not very impactful because most papers don’t get read. Someone who’s thinking about, well, maybe do I want to stay in research? Do I want to do a startup? What advice would you give to someone trying to think through what they should do.
Well, let me just do a quick side tour into one of my favorite rants, which Matt and welded you’ve heard many times before, but maybe some of the listeners haven’t. If you’re concerned about the impact of your papers, you should be, because we now have search engines like semantic scholar and Google scholar. That give us great transparency into who’s read these papers, who cited them, if the citations is actually in depth so that they actually understood the paper and most papers are written and quickly forgotten. It seems like their main impact is to be another line on somebody’s CV. So I would say if you’re a researcher and you feel that way, don’t buy into this notion of I have to publish lots of papers instead buy into the notion of what would be a really impactful paper a paper that would have a lasting scientific impact.
And if I don’t have a great idea for one, let me keep looking. It’s not about who publishes the most papers by the time they die, no more than it’s whoever has the most toys by the time they die wins. It’s really whose work has lasting scientific impact anyway. If you’re saying, okay, what about the start up world? I would say two things. One is I would say go for it. What have you got to lose? Right. I think that the thing that you lose is if you get to be my age and you say, you know, I could’ve done startups but I was too scared or I felt like it was risk in some sense. That’s the real risk is that I could’ve been a contender, I could have done something, I could have given it my best shot, but I didn’t, and now it’s too late. That’s a real bummer. If you go and you try to start up, you work hard, you find a great team, it doesn’t work out. It can still be a great experience and maybe you’ll do better next time, maybe you won’t, but at least you gave it a shot. So I would say if you’re thinking that way, you should consider giving a shot and then there are various things we can do to make that shot be more likely to succeed.
I guess on the flip side, what if it’s, I get to be your age and I think, Oh, I should have been a professor. I wish I had stuck with academia and gotten tenure and become someone really famous, inducted into various academic societies. I had that opportunity. But instead I did a startup.
Right? So that’s the beautiful thing about our academic system. It’s not an either or choice. So that’s really a, a false dichotomy. As a grad student, you can take a year off, no problem. As an academic, you can take a year off before you join academia or you can do start ups as professors. Many people have. And of course, as an undergrad before grad school, at every point along the academic path, there are opportunities to do startups and you can investigate those very vigorously without losing your academic career. Something that’s really special about AI2 in particular is that because we have both the nonprofit research arm and the incubator, we actually allow people to share their time very fluidly across both activities.
So for example, Xnor.ai which is a vision-based startup started by Ali (Farhadi) and Mohmmad (Rastegari) at AI2 for quite some time after they started it and up until the present time, they’ve been sharing their time between AI2 and the incubator company and then AI2 and the startup. And that’s something that’s really quite unique for AI2. Generally in the university, if you get more engaged, you have to take a leave of absence. If you’re in a big company like Google or Microsoft or what have you, then you really have to leave to pursue your startup. So at AI2, we really encourage people to have a positive impact. And really the only constraint we put on this activity is we believe in AI for the common good. So we don’t do startups and evolve ad targeting or surveillance user data in general. We want your startup to also help make the world a better place.
So let’s say I decided to take a dive in the startup world and took a year off of my faculty position or of my research work and went all in. What are some of the mistakes that you’ve seen many startups do over and over and you think maybe not everyone has to relearn the same lesson over and over?
I would say the two have to know your strength and weaknesses, right? So one of the hardest things in the world is to look in the mirror and really see what you’re seeing. So typically if you’re taking a break from your academic career, you don’t have a lot of business experience and so you want to ask, how do I partner with someone who does, somebody who maybe has background in raising money for startups, somebody who has background in assessing business ideas and that doesn’t have to come in day one, but you definitely want to look for that. I would say that’s one thing. The second thing I would say is that you really want to seek advice. Sometimes people think that the business world is very secretive. Okay, I’ve got this idea, I’ve got to protect my IP, I’ve got to hide it. But actually a lot like research, you want to talk to as many people as possible and get their feedback.
Plenty of them are going to be negative. When we did Farecast, a lot of people said, you can’t do it. Technically the airlines won’t let you do it from a business perspective, you won’t be able to get the data to build your predictive models, et cetera, et cetera. There were many objections and from very knowledgeable people. You also have to have the tenacity to believe that your idea or an evolution or pivoting of it will result in something exciting. But if you feel like you’ve got tenacity and you’ve got the passion, then go for an explorer. Look for teammates, look for advice from people where it’s not their first rodeo. Read as much as possible. There’s so much material on the web about how to build startups, how to test your ideas. Let me mention by the way, another key idea which comes from the lean startup literature.
Often we think as computer scientists, okay, I’ve got an idea, I need to first build it right, and then we can test it. The building something, even a prototype can take six months or more and you’re already like deep into your, you know, your one year leave. Often you can test the idea in clever ways. Maybe just by creating a PowerPoint, maybe by just showing people examples. You don’t have to build even the prototype before you can start getting feedback on whether this makes sense or not. So I think that getting experimental feedback, we’re all CMU alumni, right? Computer science is an empirical inquiry. We want to get that external feedback, not just for my own crazy ideas, but from a data from potential users as quickly as possible.
That makes sense. I think a startup really fails when you run out of money, right? That’s a condition that at which you say like, okay, let’s start up failed. How do we get there? What are the common failure modes that get us there?
The most common failure mode that I see is people getting excited about something that’s really a feature, not a product. So for example, I mentioned, you know, prioritizing emails. That’s a feature of an email system. It’s not really a product. And so people start to believing that this feature, everybody needs like oxygen and everything else will fall into place. So you do want to think about the fact that you want to build a product. Second one is this notion again of a painkiller rather than a vitamin. This needs to be something that people or organizations will care enough about to change their behavior, to try a new thing to pay for it. Another thing is lack of focus. So people say, wow, I’ve got this technology. It’s amazing. It’s good for this. It’s good for that. It’s horizontal, that’s fine. But typically you need to find what’s called a vertical, a particular market, a particular set of users, a particular product and application to use your product.
So I think the lack of focus is a real problem and lack of focus can take many different directions. Even if I’ve chosen a market, maybe I’m wanting to build too many features, right? I can’t prioritize effectively. I want to try all these things and however hard it is these days to get a paper into a ACL or EMNLP, the startup ecosystem is that much less forgiving. It’s not 25% of startups succeed. The reality is that most startups do fail. And that’s something you have to accept. But you have to be a sufficiently optimistic person to say, but I’m still going to go for it. Then once you’re up and running, hopefully you’ve gotten some initial team, initial financing. There is an element of careful management, right? You don’t want to run too fast and you know burn the money that you’ve raised too quickly. You want to have a pace in kind of like if you’re running a marathon or half marathon, right? You can’t just start at a sprint pace. You need to have a plan for the run and you need to pace yourself.
Yeah, it makes sense. So one of the things that you mentioned is partnering with people. Say If I’m a researcher, an NLP researcher, maybe I can cover the technical aspect or at least I know how to cover the relevant technical aspects. How do I find partners from the business side? I remember when I was at CMU, there were a lot of business students whose afternoon activity is to go hunt for computer science students who are interested in building startups. But it’s hard. As someone from the computer science side, it’s hard for me to assess how good they are. Right. So what are some of the considerations you’d recommend people think about when they’re taking their business partners?
Right. Well, I have to confess that I don’t think that a business student is an ideal partner, typically for a computer scientist. So no disrespect to business students, but they’re really quite inexperienced. And whereas as a graduate student or even an undergrad, you can build state-of-the- art models with tremendous performance. Business students still in business school don’t usually have the experience to build world class organizations to successfully fundraise for venture capitalists. So the first question I would ask my business partner is, well, what have you done in the past? So if they haven’t raised money in the past or if they don’t know the industry that you’re thinking of going into, then they’re going to be learning this along with you. And that’s not the ideal partner. The ideal partner is somebody who’s already poised to make a contribution, who already knows a bunch of things that you don’t know.
I think also incubators like AI2’s incubator help matching the skill sets that are necessary. And it seems like there’s a growing culture of incubators across the country. So
That’s very true. So in Seattle there are three. We’re the newest one where an AI first incubator, but also a Madrona venture labs. Pioneer square labs. There are absolutely incubators and meetup groups in every city in the US also all over the world. This is not by any means, just a US phenomenon. And I think that there’s a lot of opportunities. And by the way, people are typically very generous with their time. Often if you just email a person who’s had some experience and you say, Hey, can I meet with you for 30 minutes to discuss my idea or to ask you for some pointers, particularly, you’ve already done some of the basic research yourself on the web. People are very open to that. Very generous. I hope I don’t get a ton of emails, but actually if I do, I shouldn’t mention that our incubator is open to people outside of Seattle.
So while we’re based in Seattle, if people want to come here, you know, explore joining our incubator, we definitely have conversations everywhere and one huge conviction, which is pretty unique for our incubator. The conviction that because of this phase change and NLP specifically based on transformer architectures like BERT and so on, I believe that we’re facing a discontinuity in NLP and over the next five years there are going to be some incredible NLP startups and they’re going to have impact on translation, on search, on speech, new products, and so now is a great time to do an NLP startup.
All right. Is there anything else that you wanted to bring up before we wrap up?
I guess I would just point out that actually startups are not just moneymakers, my son, for example, works for a nonprofit startup trying to make the world a better place. This podcast is actually a startup and kudos to both of you for launching that. It started from nothing, just an idea an identification of a need in the community and then you’ve started it, you’ve grown it. It’s gotten a lot of listeners. Again, I don’t think we’re ready to monetize this, but that’s fine. It’s serving a need. It’s a wonderful thing. So I want to congratulate both of you on your quite successful podcast startup.
Yeah, thank you. I think this is going to be the hundredth episode, so it’s kind of like, it’s a milestone for our podcast, so.
Perfect. Well, thank you very much for talking to me and way to go.
All right. Thank you Oren.