Skip to content


What I Learned In 1 Year+ of Lean Startup Can Cut Your Startup Risk 10X

I spent about 15 months doing market research using the lean startup method and presented what I learned about the process to the Talking Lean meetup group, in Tel Aviv. Here’s the video based on my experience running lean. Watch it and learn from my experience to dramatically derisk your startup and increase your odds of making it big!

(FYI I’m still working on the startup part time, but as my savings were running low I took on a job with Internet Marketing Ninjas.)

Here’s the transscript:

Gab: Shalom. Hi, my name is David Goldenberg, and I’m here to talk to you about finding a problem, that you can solve. Because if you’re just going to create a technology and then hope afterward to find a market for it, chances are you are going to waste. And obviously, like Shai just said, the whole point of Lean is not to waste. So I’m here basically on my own behalf, although I’m in my employer’s sweater, and I’ve been working for most of the past year, year and a half on making a better Jewish matchmaking service, called “Good People Dating.” And I would really love your feedback. You know, either by Twitter, by email and essentially tell me, you know, what I did well, what I could do better. I would really be appreciative of that.

Gab: Thank you . I will just use this for now. Thank you, though, Shai.

Gab: So, just so that we’re all quickly on the same page. There’s two big aspects to how Lean works. So the first one is doing your market research properly. Mainly insuring that you’re solving a real problem, like I said. The other aspect. Is to offer a desirable product. So you’re going to do later on solution interviews, to talk to people and see what do they think of this idea, that idea. To make sure that you are offering the right solution, not just that you found a problem, but the solution is not “shelait“, is not relevant. One great example that we have from the real world of how this can work excellently, is franchises. So, I’m sure most of you have heard that small business’s start-ups usually have an eighty to ninety percent failure rate, but franchises have a seventy-five percent success rate. Which is “recctos” or I’m a fool. And the reason for that is that essentially they have built a system of a, finding the right problem. For the right market, and the solution that matches it. And of course systems to produce and measure the product all the way through. So, I’m not going to talk about the whole process, that would be too long of a presentation. I’m just talking about the first half. Which is: How do you find a problem? And for the most part, I’m going to be talking about my own experience doing this for “Good People Dating.”

Gab: Sorry. Okay.

Gab: So there are five different aspects in particular I’d like to talk about, The first one is: Why not use other alternatives to Lean research methods? So the Lean Method is really focused on problem interviews. But you must have heard of other tactics, for example, surveys. Or buying data from brokers. Why not use one of those? Also talk about the right mindset you need to have, in order to do Lean properly. Because as entrepreneurs, usually we have this great idea we are so excited about, but the market doesn’t necessarily know that that’s such a great idea.

Gab: And even when you talk about it with them, it’s not such a great thing. So, there’s a particular mindset that will help you be more successful, a different attitude. We’ll address that. And then what is it, that you are trying to learn ? What are you trying to figure out with this approach? With this lean research? Okay? So the tools for that are interviews and surveys. And we’ll see how they fit. And then finally as a little bit of a bonus, I‘ll talk about a particular way to use crowd sourcing. You probably heard what happens on Mechanical Turk, other websites, in order to make your research go faster and make it cheaper as well.

Speaker1: So there are a few different inferior alternatives to Lean Research and mostly they have to do with ways that you could start on the wrong foot. So the first one, I know from experience, is don’t have any plan. Who cares? You know. And I did a Commerce Diploma at this place called Dawson College. I learned a lot. But, unfortunately, most people aren’t entrepreneurs, and so they don’t know to teach entrepreneurship.

Gab: So, they taught us, like, all these expensive, implausible ways that an entrepreneur doesn’t have the means for, doesn’t have the time for. How could you possibly do these things? So I was like, okay, well, these ideas are ridiculous. Obviously nobody else is doing them, because I spoke to other friends. And I thought, okay, I will just wing it. So after about six or seven years, Bochina, I did end up selling the company. But, obviously for a lot less than if I planned. And had an approach I was following, a game plan as it were, rather than just being random and trying to wing it.

Gab: Okay? The second tactic, I’m sure you’ve heard of is phone surveys or online surveys. So for example, when there is an election, we hear about this party has fifty-three percent, that one has thirty-two percent, etcetera. So, essentially they have people calling in the phonebook, at random. And the idea is to get a representative survey of the population. Now this assumes a few things. First of all, it assumes that a random survey of the population is a good thing. And it’s not. Because your market is not everybody. Your market is a specific segment of the population. Another assumption is that you have all this money. This starts at least at two thousand, usually more. If you just starting out, or trying to be lean, that’s a lot of money. Especially when another assumption…they think you know what the right questions are. This ties into the attitude we will talk about later. You don’t really know the right questions to ask usually, when you starting out. In other words, data brokers, especially for people who want to do enterprise, want to do B to B. So there are companies like Dunn and Bradstreet, which will sell you information, such as: Who are the executives? What is their contact information? What other software or things are they buying? What’s the buying cycle like there? Again, this assuming, you have lots of money, you know what questions to ask., what data to buy, etcetera. This is wasteful, because, usually when you are starting out, you don’t know anything. And you have to take that assumption as your basis.

Gab: Brainstorming, So, brainstorming has a place. But, when it’s your whole process. It’s not the starting point for research , but it’s “I’m, here, just going to brainstorm”, then, you’re essentially playing the lottery and, you’re hoping that you have a guess, which is lucky. Sometimes you may be lucky, but sometimes you won’t be. And that’s an easy way to fail, as well as to waste time. But, more importantly, okay, even if you fail, you can’t optimize. You can’t improve, because there is no way to measure this. If you are just guessing. How do I know if I’m right or wrong? Like, unless I go to the market, and I do all this research afterwards to find out, I don’t know.

Gab: So, pre-selling. You guys maybe heard of Seth Godin and his book “Meatball Sundae,” or Tim Ferris’ book “ The Four Hour Workweek.” They talk a lot about pre-selling. It is important and has a place in the research cycle. But it’s not the starting point. And in “Meatball Sundae,” he talks about the toy industry. So, it’s really fun how they do this. They get together. Everybody, All the buyers, the sellers, whatever, the people who want to produce toys, at this big conference, in February. And what do they do? They sit there watching commercials. Watching TV ads. And the ads with the most selling appeal determine what products end up getting made for Christmas of that year. So, this is useful if you are already quite far down. You understand who the audience is. You understand what need you are solving for them and usually this is more for a iterative approach. We’re making a better remote control car. We’re making a better Barbie doll, whatever. But, if you start out with this. So before we said, you are guessing. And here you are guessing, checking. So, great! You’re measuring, you can tell whether or not you got sales. So this should be a lot better. I can tell you from my personal experience: I wrote the first book on advanced SEO. And , you know, I got names and emails from my list. And I pre-sold the book, so I was able to tell, you know, is there actually any topic interest. Are people giving me money or not. But the problem with this, is that you are just guessing. You are like little children, Who play hot and cold. I don’t know if you know this game. Like, there’s one person who leaves the room. You choose, A treasure item. And the person walks around, they’re like, colder, colder, hotter, hotter, hotter. And they’re trying to figure what the item is based on the audiences’ feedback. But, there is no cleveite. There’s no, like, ways, telling you, okay, five hundred meters straight, and then over to your left, you’re good, you found it. This just approximating, and again when you are approximating, you’re wasting time.

Gab: Another popular substitute is VCs. They’re like, oh this guy got experience, he was really successful. Last time he exited for a billion dollars. He must know what he is doing. So, I had no exit for a billion dollars. But, other people who have. You may have heard of “Color,” I think, recently, basically, just failed, very publicly. And this is a guy, who, you know, knows what he’s doing. He made a lot of money his first time around, or second time, whatever it was. Raised all this money, and then it was a blow-out. I can speak from my own personal experience. IBL is a word press plug-in I created. I promoted it, got, you know, twenty-five thousand downloads. Not bad. For a rookie. I was two years into internet marketing. And, you know, links from, basically, all the top sites. People I didn’t even know, in Russia, in Romania, Brazil, Japan, all over the world, literally. So you think, okay, yeah, lets, give Gab more money, he’ll make another word press plug-in. This is like a sure thing. Well, guess what/? I did another plug-in. Got a handful of links, a handful of downloads. I’m not bragging about this one.

Gab: So what are the some of the common problems between all these different issues? So one of them is obviously is cost. You are wasting your time, you’re wasting your money, your wasting your motivation. Like Shy brought up and we saw so last time with David Katz, if you guys remember, his presentation. You know, he was working in the HR field, and he was sort of getting closer, but after a year, he gave up. And, you know, people asked him, and I had this question too. Why did you give up? It seem like you were getting closer. You were talking to Mozilla, this is a potentially very big client. He said “nimas ly,” (I got fed up). You know, fair enough, the guy has been at it for a year. He’s not really got any headway…

Gab: Another problem is, this is probably the key one, which underlies all the other ones. Is you don’t really get insight. You’re not learning. Okay. If you guys have read the book “ Running Lean” by
Ash Maurya. I really encourage you too. You’ll see that the key thing he wants to bring up is that you need to learn. Because by learning, you can iterate closer to a plan that works. So, Febreeze is an interesting case study. If you guys don’t know what it is, it is basically a little chemical spray, that you put on your clothes, or other materials. And it removes the stench. So they thought, great, smoker’s smell really bad, and so we give them this. And they will be like, fantastic. You know. And since they keep on smoking. Because they are addicted, they are going to keep using Febreeze, we can’t fail. It’s logical. It’s a guess. But it failed. Their sales really sucked, out of the gate. This is a huge company. This is, I think, a Proctor and Gamble product or something. This is a multi-billion dollar company. So then they did the research, okay. And , they were like, okay, who is buying this? How are they using it? They went to people’s homes, and they filmed the women who were using this. And they found people were, lets say, cleaning the den, cleaning the bedroom, whatever. And when they were done, they were doing like sprits, sprits. Finish the den. Okay. Bedroom. Done. Sprits, sprits. Okay. It’s like their own little reward. You’ve had a to-do list, and you check things off. It feels real good. Same idea. So then they changed the ads around., and they showed people cleaning their homes and sprit sing as a sort of self-reward. And sales took off. Okay. If they had done this research in the first place, how much angst and pain would have saved for them.

Gab: Another issue I sort of refer to, is with my book.. So, I had all this data about emails and sales, and, you know, very quickly. I’m a very conversion oriented guy. . Conversion just means sales in Internet marketing slang. And I ‘m like, okay. How do I make this site more user friendly? How do I make the copy more persuasive? All this stuff. I’ split testing. I’m using user ability tests. But, what you get is this focus on scaling. How do I make this bigger? How do I get more people on my e-mail list? How do I get more sales? But. You are not focused on understanding your client. You don’t know what they care about? And if you don’t know what they care about, you can’t offer it to them. So this is a, I think, a common problem, and it is so tricky, because you have money, you’ve got traction. And if you know, read at all, especially for this entrepreneurship crowd. If you are reading about ANGELS and VCs,
and basically raising money. We want traction! Tell us about your users! Tell us about how much money you’re making! I’ve got traction. What more do you want? You know. But, if you got traction without understanding, you are going to reach what they call a local maximum. So, imagine a mountain range. And, So, traction, you’re making progress. You’re like a third a way up the hill, very nice. Then what happens is that. Okay, so you optimize. You did everything. You reach the peak. Oh Yeah, I’m the…You look over, and you’re like, oh wow, well, that mountain was so much more massive. If only I had done my research. I would have known that. And my market would have been ten times bigger. And all this work and effort would have paid off a lot better.

Gab: So in summary. What are the benefits of Lean as opposed to this? You are going to get real insight. You really are going to understand what the customers need. What is their problem? It’s obviously a lot cheaper, because your not wasting this time, this money, etcetera. Your motivation. It’s much faster learning. You’re not, not guess and checking. You are not trying to feel your way around. Trying to approximate what the solution is. You’re actually learning and starting out. If you guys have any internet marketing background. When you understand the problem. You can be like: Hey, is your, you know, baby carriage really low when you are bending over walking around like this the whole the time? We’ve got the solution for you. If you understand that, that’s really the problem. And your conversions start much higher. So that the scaling actually isn’t a problem, even though you’ve had a delay, because, you start out with a higher conversion rate.

Gab: Another point is, that this doesn’t assume anything. It assumes you know nothing. Which is the truth. When you are starting out, you don’t know your market. So that was the first part of five. We said we were going to start with four alternatives to Lean and why Lean is a better solution.

Gab: Now, let’s talk just about the mindset. So right now I’m doing “Good People Dating.” And I thought, you know. How could I be different from all the dating sites? And all these sites have one big problem in common. And that problem is, that once you make a match, you’ve lost your customers. So you can only get limited, repeat business, basically if you ‘re doing a poorer job for your customers. I don’t know about you, but that doesn’t sound like a very fun business model to me. So I thought, okay, I know. I’m going to serve couples too. So, I’m going to do something. To help them build great relationships and lasting marriages. They will be really happy. So I thought of an idea. I took the relationship bank account–If any of you read “Seven Habits of Highly Effective People” or “ Highly Effective Teams.” If you measure your relationship. Then this will be fantastic! And people will be able to improve their relationship, based on measuring it. People hated it. They laughed at it. You know, my family laughed at it. And if you can’t convince your family that you’ve got a good product, you’ve got a real problem. People were telling me like, “Oh that’s such a fun game, you know. I totally know someone else who could use it, but not me.“

Gab: So the first thing is…And it gets even funnier, because after I did all this and wasted all this time. I’m like, oh that was a bad idea. I’ve got a better one. Let me try to do all these interviews, and this one is really going to be validated by the interviews. You don’t know your market. Okay. So, we need to be humble and admit this, and instead of asking forward looking questions in the interviews. Would you like this? Does this feature sound like a good idea to you? What do think about this approach to solving the problem? Ask questions about the past. Because that is certain. People either did something or they didn’t. They either struggled with something or they didn’t. And in this way you are going to get real data. And, your humility will actually help you to be a big shot afterwards, and then you can boast ,once you have that billion-dollar exit.

Gab: The other point is that you have to be patient. A lot of us, and it’s just human nature, like to build. We want to plant a tree and harvest the fruits. And, We want to write code and get people to use it. But if you do that before you understand who you’re writing code for, and what you’re solving for them, you’re wasting. And you’re more likely to fail.

Gab: So this what we need to learn, now. What is the problem? Is it a broken swing set or is it a dropped snow cone? And who is my audience? Is it the five-year-old Irish-looking girl? Or is it the three-year-old Hispanic-looking boy. Okay? So, in order to do this, we want interviews. If you read “Running Lean,” it talks about is problem interviews. One thing he doesn’t really address: Is where do you get people to be interviewed from? So I saw a really interesting video on Customer Devlabs dot com. It’s a great blog about running lean. Where actually another dating entrepreneur went to Amazon Mechanical Turk and he crowd sourced loads of interviews. Like in four hours, he got 100 interviews. It blew my mind. I’m like, well, he’s in dating. I’m in dating. I needs interviews. He’s got interviews. Let me go to Mechanical Turk It was a total blow-up. Totally failed. Didn’t quite figure out why. But I went to Facebook. And my friends were great. They allowed me to interview them. They passed it on to other people who were willing to be interviewed. I also posted it in Facebook groups, as opposed to just messaging friends directly. You know, groups that I was apart of. I wasn’t just like spanning random groups. And both of these really got me interviews, got me more data, later for surveys as I will talk about too.

Gab: Another one, he talks about this in the book, and I can vouge for how valuable this is. Always ask, at the end of your interview, for a referral. So, you’ve got a script. You should be following a script, so that your interviews, more or less, are gathering the same data from each person you speak too. And at the end you ask: Please can you refer me to two or three other people, who should be, you know roughly in this target market? Similar demographics, similar whatever. You know, you know what the target audience is.

Gab: Now to get these interviews, You have to set a target and work backwards. So if you want five interviews a week, you need to get five of them approved. Because approving and scheduling. People who are happy to be interviewed, don’t necessarily have the time. So to get five scheduled, you need probably, let’s say ten or fifteen agreed. To get ten or fifteen agreed, you need ask, maybe thirty or forty people, let’s say. Maybe your friends like you more, maybe you won’t need to ask as many. For me, let’s say thirty or forty.

Gab: Another issue, especially for any of you if you want to target North America. The shoe key of the market here, let’s say eight-million people there. Here is eight. There is three hundred fifty, so probably a lot of us, who wants a big market, want to target there. Guess what? We’re seven hours ahead. That’s really difficult for scheduling. Because it means when somebody leaves their job at five pm in, you know, New York, anywhere on the east coast. I’m from Montreal, Canada, [?omeilia?]. From there. They’re leaving at five. It’s midnight here. So give them just a little time to breath, get out of the office. You know. Let their, let their brain, wander a bit. It’s twelve-twenty, twelve-thirty, midnight.
When you just start your interview, you’re going to finish at quarter past one. If you run one back to back, you’re going to bed at two, two- thirty. So that’s very difficult. And what I found is that mostly I was scheduling my interviews on Sundays, because people aren’t working. So, it was able to work out in that way.

Gab: One solution I found, is that the data isn’t necessarily different between Israel and the rest of the world. Okay? So, I didn’t know this in the first place, but, I thought, hey, if I can get the data anyways, you know, during my week. I obviously wasn’t interviewing people in North America, because they weren’t available. So if I can, you know, let’s say, on a Tuesday, on a Monday, Tuesday, Wednesday interview somebody in Israel. Well, fantastic, because that is more data, and who knows, maybe it will be useful, somewhere else down the line. To let me correlate. Let me better understand.

Gab: Another little trick: GMT. Stands for Greenwich Mean Time, as opposed Israel Standard Time. And that’s the time that England is on. If you know your geography, England is only two hours behind us. So, that makes your scheduling life a heck of a lot easier. Plus their culture, not completely the same as North America, obviously there is some distinctions. But it’s still an English speaking country, a lot of western culture is shared. They watch Hollywood movies, etcetera.

Gab: Another idea, okay, so up to here is experience. There next bullet-point is theory. So, take it, maybe, with a grain of salt. You can perhaps hire a foreign virtual assistant, to actually do the interviews for you. So you can’t be interviewing the people, but you can maybe hire someone locally over there, who is in the right time zone, that’ll be convenient for your audience and for them. Now, why didn’t I do this?
A few things, first of all I’m a perfectionist. I like things to be done really, really well. And I’m reluctant to trust a stranger, when it’s so hard in the first place to get these interviews. You know, if they screw it up, it’s with their friends, Ehz. Another thing is, I’m really passionate, I care about this. I want to help Jewish singles get married. And I didn’t know if people, who are just being paid basically, by the hour, would, you know, ask, the followed questions, and show the interest and, and the, you know, the ozen shamot, the listening ear, that I would show to my interviewees.

Gab: So, you know, some pros and cons. If you test it out, I would like to hear what your experience was like with that. So, now, once you know what you’re looking for. What the problems are, and who the audience is. You need to get in your customers mind. And the way to do this is to ask: What is the biggest problem preventing you from achieving your goal?

Gab: Yes.

Audience member: This might be a strange question. Because you have a probing mind, that solves a lot of latitude problems, and I’m going to ask generally what kind of problem do you have might get an answer that is not relevant to your project at all.

Gab: So the full question is: What is the biggest problem, preventing you from achieving your goal? And so you substitute out, preventing from achieving your goal for what’s relevant to the market. So in my case I was asking: What’s the biggest struggle, what’s the biggest challenge preventing you from finding your spouse? So, now they’re not going to start talking to me about how, I don’t know, Skype crashes on them. That’s obviously not, I mean, maybe if they’re talking to, you know, their girlfriend, their boyfriend, whatever. Okay, maybe, that’s a problem.

Audience member: How can you rely on such an answer? Like the person could have personal problems. Especially in this field.

Gab: Good question. So, I’ll just repeat it louder.

Audience member: They won’t share everything with you

Gab: Absolutely. How can you rely on such an answer? Especially in dating, it’s sensitive, people might be a bit shy. How can you rely on it? So a few things. First of all, you are not relying on any one persons’ answer. You are looking for the pattern in the data So, maybe one person is going to lie. Maybe somebody else is concerned, whatever. But you are going to notice patterns. Number one. Number two. Provided that you tell people upfront, this is private; I’m not out to share this with everybody. These are your friends. So there is some degree of trust. Or your friend’s friends. Because you asked for referrals. Then you do get that trust. I can tell you from my experience, I got fantastic data. So, when you ask this question. When you get into your customers mind, and you’re seeing what they are seeing. Okay. So it’s incredible. Like, it’s magical! It’s lovely! You get this insight. You figured out: What do I need to do? What do I need to help them with?

Audience: Examples.

[Cross talk]

Gab: Question?

Audience member: Okay. Just a small remark. It’s very important, how do you ask.

Gab: Yep

Audience member: To ask, what is the problem? It’s not good..

Gab: Okay. How would you ask it?

Audience member: Because it’s negative.

Gab: Okay.

Audience member: You might cause yourself a problem.

Gab: Okay.

Audience member: It’s good to go on the positive way.

Gab: Okay.

Audience member: Just like asking them: How can I help you to achieve, to get the date? Or to get the spouse?

Gab: Okay.

Audience member: If you go by the negative. Once we asking people, we have to put some of our thoughts and mind onto the way we are asking them Because in the way you are going to ask them. You ask them this way or another way, you going to get a couple of different answers.

Gab: You’re right. You’re right. So the comment is, “If you ask with a sort of negative phrasing, what is the problem as opposed how can I help you? Then you might get very different answers, depending how you phrase the question. So, you are right. Now the thing is, we’re looking to find out what their problems are, what they’re struggling with. If you ask: How can I help you? Then people are very solution focused. And the tell you, Oh you could do “JJBaDa” better like this. Or you could do matchmaking better like that. Okay. And that might be useful later on, to ask, but initially, you need to find that out. And I can tell you from my personal experience, it does work. It might be worthwhile to test and see, you know, how can I help you, sort of, split-test interviews and see if that gets you better data. I suspect though, that you would get, like, the famous quote from Henry Ford; “If I’d asked people what they wanted, they would of told me faster horses“.

Gab: So..

Audience member: Yeah, the same thing. Like, if you…

[Cross Talk]

Gab: Yep.

Audience member: … are trying to date. You’re saying I’m a great dater. This is your problem. You’re concentrating in dating, and not in finding.

Speaker1: Okay.

Audience member: Okay. So, it’s the same thing. Just like you say. You bring me a faster [?] Now you’re thinking with the people. How do I get more dates? Do I really need more dates? I need to get the right person to find it. This is the question.

Gab: So the question is not: How do I help you with the process? How do I help you with the goal? So, when you are talking about the goal. I didn’t say: How do I help you date better? In which case, that would be a very valid comment. And it’s easy to see how that nuance could be lost. It’s very easy to ask: How can I help you have more fun dates? Dates where the conservation is better. Well, if that’s the goal, fine. If you’re goal is just to make funner dates for people, where the conversation , you know, flows more freely, [djudelekem] Then, maybe that would be the question. But here you are talking about: What is the goal? In my case it was: What is preventing you , finding the right person for you. Okay. So, that’s not talking about the process , that’s talking about the goal. And this is a very important distinction. That’s great that you bring that up. Because you are helping see how in actually, We might have understood the theory in one way, but misapplied it. So, thank you for clarifying that point, so that we know to say top problem in reaching your goal, as opposed to following this process.

Gab: So, a little tip that you can do is you can have the crowd help you with cross-checking what you found out from interviews. And one amazing site is Quora. Because people ask questions. People are very, very candid, like almost painfully candid in their answers. I’ve seen people being addicted to cigarettes, about be thrown in jail. You know, loads of very personal things. Obesity, this is a very interesting question, because it’s such a–excuse the pun–large market. And, and people say, you know, my problem is not that it’s hard to lose weight. It’s not that I can’t do exercise. It’s that they are very sad. And they are eating as comfort. So, you could come with a more clever diet, and you could count the calories more intelligently and come up with more tasty foods for them to eat, and easier exercises to do. But it’s useless, because you don’t know what the problem is. The problem isn’t that it’s hard to do exercise, that it’s hard for them to count the calories It’s that they are sad.

Gab: So, you can read this yourself, and you can summarize it yourself. But chances are, your time is pretty valuable. So, depending on how you value your time, you might consider hiring some people on Amazon Mechanical Turk, and just have them go and find these questions, read them, and summarize the answers for you. Also I’ve mentioned Wiki, Statsbrain, forums, basically you can get similar data from Wikipedia, and Statsbrain in particular, obviously more statistical information. Also the U.S. census. All that information is freely available online.

Speaker1: So, once you’ve done these interviews. You’ve done one round of interviews. Edit your questions. See, okay. This maybe wasn’t so clear. I’m just going make sure my phrasing is better. That’s to your point. And I had that experience., where I realized, I need to change this around. I need to get rid of this question., because it’s useless. Now, you do that one or two times, you’ve got these patterns that you’re finding. So write a progress report. It’s going to do two things. First of all, for yourself, you are going to understand better, what you’ve learned. Where you are going. What are the patterns you’re starting to see. Who has this problem? What is the main problems, the common problems, that my audience has.

Gab: Then email the progress report. Because people are giving you your time. It would be very nice if you returned the favor. And shared: Look you helped me learn these things, okay. I’m not going to tell you all them are going to read it. In my case, I’d say, probably most of them ignored it. Maybe I can optimize how I wrote it. At least if you make it shorter, even In the email itself, include a little summary. Then it’s great, because, like we said, we want to plant the trees and harvest the fruits. So, here’s a way for the people that got involved and gave you their time, to pay them back a little bit. Incidentally, if you are interested in working with me on Jewish match making, I’d be happy to share my progress with you, even if you just want to help out. Question?

Audience member: How many interviews are enough?

Gab: Great, great question. How many interviews are enough? When do you know to stop? So it’s probably going to vary on a case by case basis. In my case, it probably took more, because I thought I had all these great ideas. I was trying to validate my product, rather as opposed to finding out what the problem was. I didn’t start out being humble. I’ve done, probably about forty interviews. I wish I had done more, but as you saw it was a real pain in the butt. If I had just done three interviews a week it would have been really great.

Gab: But, your situation might be different. Maybe you realize that, Israel versus, lets say, North America is not such a big deal and, you have more time. You can find interviewees much faster. What I found, you get, that after forty interviews I had a pretty good idea what the common problems were. But I didn’t have a ranking.

Gab: And that’s the next point I’m getting too. I have all these stories. , Why are you[ xxxx] Fantastic stories we are still reading it thousands of years later. But what is the number one problem? The [Jezanazal]. That’s one problem. She’s married to a total drunk, who killed his last wife in a fit of rage. That’s another problem. The Jews are supposed to be killed in, you know, in a few weeks. That’s another problem. So, how do you prioritize which of these problems you are going to address. So, you’ve got qualitative data. Now quantitative data. Now is when you want to use surveys. Okay. Because with surveys, you can get quantitative data. And I love, love, love Likert Scale Questions. That’s just fancy research slang to say: Rate this problem on a scale of one to five, where one is really hard, five, sorry, one is really easy. Five is really hard. Okay. And you’ve got your problem descriptions there. People click through–tot, tot, tot, tot. And this is also easier to get people to do., because it’s just a request of for, let’s say, three or four minutes of people’s time. As most interviews where thirty minutes, often forty-five, fifty minutes is a common situation.

Gab: Sorry, what’s the question?

Audience member: population

Gab: You can send it to the same population. I personally sent it to other people. But, so long as they’re roughly in the target audience. The reason I didn’t send it to the same people, is I was concerned that I would’ve influenced them somehow, or by talking about it, I would’ve guided their choices. So I just wanted people who were sort of , completely new to what I’m doing, to answer it. But I haven’t test it. It might just be that just speaking to people who already did the interviews, would be, you know, great way to kill the data. Question?

Audience member: interviews and the

Gab: Excellent point

Audience member:

Gab: You’re right. Okay. So the comment, the question is that I’m sort of skipping head , I’m asking people to rate these problems, but I don’t know who my audience is. All I did was I asked: What is/ was your biggest problem? And that’s just because I made a mistake with the presentation, and I skipped mentioning that in the interview stage, you should already be asking demographic questions. And also here obviously I’m just focusing on the core of the survey. But elsewhere in the survey. For earlier questions, later question, whenever. You also want to ask demographic questions. That’s a very good point. Because that’s going to help you understand. Which segment of the audience has which problem. And we will talk about that in a second. But this right here allows you to average out… I will get to your question in a second. To average out which is the key problem. So, problem A may rate three out of five on average, and problem B is four to five. Great! Now you know to focus on problem B. And I love doing this with Wufoo for two reasons. One, the Likert stuff is built in. It’s just a form or survey creation tool. So I don’t need to play around with my HTML and go crazy. It saves me time. It’s worth it at fifteen bucks a month. And you can export to CSV, which is great. Because when you export to CSV, then you can get into slicing and dicing the data, and you can compare your demographics. Your age, your gender, your location, education, marital status, income, whatever your key questions are to the problem ratings. And you can see what problem, A or B is worse for guys versus girls, let‘s say. So we said problem A has an average of three out of five. So that could mean that half the audience, guys, have, you know, have four out of five rating for problem A. But girls only have two out of five. So it comes out three out of five rating. And problem B is a []. Women have five out of five rating, and guys three out of five. So that, that averages out to four out of five. But if you talk to everybody the same way. Or if you choose to target the whole audience, men and women., well , one, first of all you’ll be buying a lot of useless traffic, because the guys definitely aren’t going to convert as well if you are telling them about solving B problem. But you can also buy all that traffic and convert it better as follows: So you send the women to a landing page that talks about problem B. And you send the guys to a landing page that talks about problem A. Because problem A is for them is four out of five, as opposed to B, which is three out of five. And that is where cementing the data. like you pointed out, the demographics, is very useful. So thank you, I’m sorry I skipped talking about that, with the entry stage. You want to ask about the demographics, there. And then also with your survey. You know, so that you can not just guess who the right audience is, but having in the data, to tell you this segment really scores higher on that problem.

Gab: Question?

Audience member: yes, So, I read yesterday in Quora that is actually in practice, easier to get people to answer your interviews than to answer your Likert Scale. Is that really so?

Gab: I haven’t seen the question, so I’m not sure really, what the context is, but…

Audience member: Somebody say they found out the surprising result, that people prefer to answer interviews, than to do a survey.

Gab: Right. So I’ll repeat the question. This gentleman here said that he saw in Quora, an interesting point. That it’s easier to get people to answer interviews questions, than to answer Likert scale questions. Which is surprising, because it’s just, right, click, click, click. Should be very easy. I think the context, probably of that question is an open ended question. Like you would have in an interview.

Audience member: What about interviews versus surveys.

Gab: Interviews over the phone?

Audience member: Yes.

Gab: I don’t know. I would have to look at that. But , thank you , because that might be a really great hack, to improve your research. Let you go faster. So if you could send me that link, I ‘d appreciate it.

So We’ve talked about problems with alternatives to Lean Research . We’ve talked about the right attitude to have. Being more humble. Being more patient. And we’ve talked about what you need to figure out.. What is the problem? Who is the audience? And we’ve talked about using interviews and surveys to figure that out.
Speaker : So now is for the like bonus material. How do you do this faster and cheaper with crowd sourcing? How to use Mechanical Turk for all this stuff, for example. So, just so everybody knows, Mechanical Turk is just basically a platform, where you can hire a bunch of people, to do a task that you specify. So you say I’m going to pay ten people to this survey. Each person that does the survey, gets a dollar. And you can also specify who does the survey. I’ll talk about that in a second. And it’s, according to academics who have studied this, they have compared census data, da, da, da. Roughly representative of the U.S. Population. Which is excellent for us. Because In the U.S. population, there is lots of different people. And you can figure out who is my segment. Who do I want to talk to? So, the problem with Mechanical Turk is you’re limited in terms of who you can pick. You don’t get to have anybody do your–excuse me–they will let anybody do your survey. Which isn’t what we really want. We don’t want to sell a product to everybody. Unless you are running out of money and needs to eat, but most of us are not. So they just let you narrow it down by country and proficiency So that means that somebody has done five-hundred tasks and had, you know, 450 approved . They got ninety percent approval rate. Wonderful! They are not totally incompetent and they live in the United States. That narrows it down So, how do we narrow things down further to make sure that we get the exact right audience we’re looking for. In Mechanical Turk slang, the right turkers. Right/ Turkers are just Mechanical Turk workers.

Here are a few advanced tricks. If you have read the book “predictably Rational” by Dan Arieli. He shows you that if you tell people that you are on the honor system, and I’m trusting you. Even if people have an incentive to cheat, a financial incentive, they’ll still do the right thing. Even…and this was really comical in the book. He was saying you were on the MIT or whatever the school was, honor system. The school had no official honor system. And still people with no invigilator in the class with being paid like ten or fifteen cent per correct answer. Didn’t cheat. So how I write a task when I want specific people to do it is as follow they say. If you are doing this task, I’m trusting you on the honor system, that you are, let’s say a mother with a child under the age of two. If I want the baby carriage audience, let‘s say. And you get mothers with a child under the age of two.

Another trick is let’s say you want to exclude some people from your audience, which would be really useful in the case of surveys. So, I have found very often that is not enough, to just set a task and forget it. Let’s say, you want to collect survey data from a hundred people. It’s not usually the best idea, to just say, okay, I’m going to pay one dollar to a hundred people. Hers’s a survey, go. Why? Because usually there is a little problem in how you phrase the questions. You need to do some quality assurace basically. You forgot to ask a question. So, instead you ask 10 people to do it. So you can figure out from their problems, from their mistakes. How do you improve the survey for next time. Butr what happens is, nest time Mechanicl Turk doesn’t just let you exclude those ten people who did the exact same task before. So you had Steve , and John and Thyme do your survey. Wonderful. But , now they are able to do it again, just because you added a question. That’s not really worth it, because I want to pay a dollar to ten new people. For that one extra question, I’m going to get, also, ten new answers. So the symbol code hack, you send people to a page that says: Whats your Mechanical Turk Id? Proceed. When they click proceed, you send them to a page with a very simple piece of code. Any coder can write this for you.
It says if Mechanical Turk Id equals the Ids of people who done this before, don’t let them do it. [] You did this. If it doesn’t equal a Turk Id that we’ve already collected, please proceed, it’s fine. Okay. Another trick that Mechanical Turk…

Gab: Sorry, question?

Audience member: If you can do that, can’t you, like, technology from the ad, all the ad business to also filter based on other demographics?

Gab: For Example?

Audience member: You know, if you can send them to a page of your own, then you can check using the technology…

[Cross Talk]

Gab: If they have third party cookies, and tell you the demographics.

[Cross talk]

Audience member: Yes, male, female, what kind of purchases does he make…

[Cross Talk]

Gab: That’s a very clever idea, yes. You could probably do that. The question was, if you can send them to your own page, then maybe you can, you know, also check ad serving cookies to see what are the demographics of these people, and you can get richer data on who your audience is. Good Hack.

Gab: So finally, right, you can use qualifications. Normally Amazon Mechanical Turk asks you to do this for people’s abilities. So he did his ability test, he did it well. Great! He has his ability testing qualification. But he can also use it for demographics. So how it works, is like this. You run a demographic survey. Age, gender, income. Whatever your questions are. Then you assign a qualificaton for anybody whose got income over x. Whose a guy, whose a girl. Who. You know. Whatever your questions are. Then afterwards when you have a task that you want people to do. You can just specify, okay, they have to have the thirty-plus qualification. They have to have the PHD qualification.

So we said this is useful for recruiting survey takers. Another thing it is useful for, is analysis. So you’ve gathered this data, now it’s time to slice and dice it, like we said earlier. I can tell you , how great of a return on an investment this is. This is probably the best RYI I got. So you export your spreadsheet. Okay.
You remove the personally identifiable information. And, you upload it so that people can access it down to the spreadsheet, and slice and dice it themselves. And what you’re doing is you pay a basic amount. Let’s say five bucks. Okay? Just to do the task.. But you’re bonusing, and here’s where people can really make money. For finding patterns in the data. What are the relationships? Between variable x and variable y. So for example, your demographics and your problems. Or you could perhaps find patterns in the demographics. So, rich men are happier with x, or struggle more with y. And, poor men have different problems, let’s say. So when you do this, you just learn so much more. And here’s a great example. I found out, that the struggles Jewish singles have here, are not so different—I don’t know if you can see the numbers, they‘re a bit small– are not so different from the struggles that Jewish singles have in Whozlazits. So only problem F has a somewhat intresting difference. But even there, it’s not even rated above a three on average. Where a five would be a must-have problem to solve. You look at all these other numbers. It’s very very close margins of difference. So that tells me what? I don’t care that I’m seven hours ahead of North America. What a great productivy advancement.

To summarize. Run your Lean Research, to maximize your odds. To be more like a franchise, where you got 75% chances of success. By cutting down your waste. By skipping the poor alternatives. Like surveys, which should come later in the process. Like pre-selling, which comes ever later, even after surveys. Because your research should start with interviews. With asking your friends, their friends. What is the biggest problem preventing you from achieving your goal? You use your surveys, you rank your problems. And of course if you can use crowd sourcing to go faster, to get more data. [] Wonderful.

So, I’m Gab Goldenberg. I ‘d love to find a partner, to work on this with, to make a better Jewish matchmaking service. And if you’re interested in the progress report. Happy to email to you. Again, I would really love your feedback. Anything I can do better in this presentation, please tweet me at Gab Goldenberg. Or write me at Gab dot Goldenberg at G-mail dot com. If you have any more questions, I’m happy to take them now, or later. Shia can tell us how much time we have.

Speaker 2: A few questions.

Gab: A few questions. Okay.

Audience member: First of all. Thanks for the talk, it was really interesting.

Gab: Thank you.

Audience member: I have a question about finding your target audience. Do you start out initially with everyone, and then you check which one, which audience is the best, or do you have some target audience in mind?

Gab: So, the question is: To find who is your audience–Do you start out with everyone? Or do you start out with some idea of who your target audience is? So, It’s really a question of degrees. You’re asking me, I think–What is the nuance? Where is the red line between I‘m assuming I know too much, or assuming I know too little. So, if I’m creating a Jewish matchmaking service, chances are, it’s not so useful to interview non-Jews about their dating experiences. Maybe it’s useful but, obviously, I’m going to get a better target speaking to Jewish singles. But…

Audience member: for age, example assume []
[Cross Talk]

Gab: Great question. For age.? Well this something you would find out in part by asking demographic questions, again, in the interviews, which our friend raised. So, I’ll tell you for example, I found, and also, if you’re familiar at all with the online marketing industry in the dating world, that they will pay a lot less money, for a lead who is, let’s say, twenty to twenty-five. And the reason is very simple. Because singles, who are twenty to twenty-five, are often still in college, and have a lot of single friends. And their single friends have single friends that they can introduce them to. And once you get past this college network, people have fewer single friends. More people are in relationships, as they get older and older. And there is fewer people that they can be introduced to. So, you can be informed from your interviews. How much you want to do that, how much you want to assume…I guess it depends on your resources and time, and money. And, also, how humble you want to be about it. Good question!

Audience member: There’s one hack I ‘d like to share.

Gab: Please do.

Audience member: My last question like you is: Can you please refer me to either other people? My once before last question., is always: So, what kind of audience, do you think, matches me best? What kind of people do you think…

Gab: [Sound of amazement]

Audience member: …would use my product best. And, well, it seems I’m already talking to very diverse people and I’m asking all them that so, I will get really good answers. eventually. That somebody will be able to convince me, that I was looking at the wrong audience.

Speaker: So the trick was, at the before last question in your interview with somebody. Just before you ask them for referrals, ask them: Hey ,who do you think my target audience should be? What are their target demographics?

Gab: Other questions? No? Thank you so much!

Posted in Business, Judaism.


0 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.



Some HTML is OK

or, reply to this post via trackback.