Guts matter; you’ve just got to test them. Instincts are experiments. Data is proof. — *location: 334* ^ref-59088 --- Analytics is about tracking the metrics that are critical to your business. — *location: 411* ^ref-10310 --- In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out. — *location: 415* ^ref-20209 --- A good metric is comparative. Being able to compare a metric to other time periods, groups of users, or competitors helps you understand which way things are moving. “Increased conversion from last week” is more meaningful than “2% conversion.” — *location: 419* ^ref-20131 --- A good metric is understandable. If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture. — *location: 421* ^ref-14038 --- A good metric is a ratio or a rate. Accountants and financial analysts have several ratios they look at to understand, at a glance, the fundamental health of a company.[5] You need some, too. — *location: 423* ^ref-4794 --- A good metric changes the way you behave. This is by far the most important criterion for a metric: what will you do differently based on changes in the metric? — *location: 440* ^ref-25936 --- “Accounting” metrics like daily sales revenue, when entered into your spreadsheet, need to make your predictions more accurate. These metrics form the basis of Lean Startup’s innovation accounting, showing you how close you are to an ideal model and whether your actual results are converging on your business plan. — *location: 442* ^ref-11470 --- “Experimental” metrics, like the results of a test, help you to optimize the product, pricing, or market. Changes in these metrics will significantly change your behavior. Agree on what that change will be before you collect the data: — *location: 445* ^ref-32329 --- But if you want to change behavior, your metric must be tied to the behavioral change you want. If you measure something and it’s not attached to a goal, in turn changing your behavior, you’re wasting your time. — *location: 463* ^ref-52329 --- One other thing you’ll notice about metrics is that they often come in pairs. Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something). Together, they tell you a lot about your cash flow. — *location: 467* ^ref-24897 --- Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations. — *location: 479* ^ref-36239 --- If you have a piece of data on which you cannot act, it’s a vanity metric. — *location: 505* ^ref-55502 --- Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” — *location: 507* ^ref-30498 --- In some cases, a lagging metric for one group within a company is a leading metric for another. For example, we know that the number of quarterly bookings is a lagging metric for salespeople (the contracts are signed already), but for the finance department that’s focused on collecting payment, they’re a leading indicator of expected revenue (since the revenue hasn’t yet been realized). — *location: 623* ^ref-44401 --- One of the criticisms of Lean Startup is that it’s too data-driven. Rather than be a slave to the data, these critics say, we should use it as a tool. We should be data-informed, not data-driven. Mostly, they’re just being lazy, and looking for reasons not to do the hard work. But sometimes, they have a point: using data to optimize one part of your business, without stepping back and looking at the big picture, can be dangerous—even fatal. — *location: 970* ^ref-1364 --- Collecting good qualitative data takes preparation. You need to ask specific questions without leading potential customers or skewing their answers. You have to avoid letting your enthusiasm and reality distortion rub off on your interview subjects. Unprepared interviews yield misleading or meaningless results. — *location: 500* ^ref-23882 --- Ultimately, quantitative data is great for testing hypotheses, but it’s lousy for generating new ones unless combined with human introspection. — *location: 1005* ^ref-4894 --- In the world of analytics and data, this means picking a single metric that’s incredibly important for the step you’re currently working through in your startup. We call this the One Metric That Matters (OMTM). The OMTM is the one number you’re completely focused on above everything else for your current stage. — *location: 1259* ^ref-38335 --- Capture everything, but focus on what’s important. — *location: 1266* ^ref-39087 --- While it’s great to track many metrics, it’s also a sure way to lose focus. Picking a minimal set of KPIs on which your business assumptions rely is the best way to get the entire organization moving in the same direction. — *location: 1293* ^ref-24821 --- Knowing which metric to focus on isn’t enough. You need to draw a line in the sand as well. — *location: 1357* ^ref-50762 --- You need to pick a number, set it as the target, and have enough confidence that if you hit it, you consider it success. And if you don’t hit the target, you need to go back to the drawing board and try again. — *location: 1361* ^ref-27078 --- The acquisition channel is how people find out about you. The selling tactic is how you convince visitors to become users or users to become customers. Generally, you either ask for money or you provide some kind of scarcity or exclusivity—such as a time limit, a capacity limit, the removal of ads, additional functionality, or the desire to keep things to themselves—to convince them to act. The revenue source is simply how you make money. Money can come from your customers directly (through a payment) or indirectly (through advertising, referrals, analysis of their behavior, content creation, and so on). It can include transactions, subscriptions, consumption-based billing charges, ad revenue, resale of data, donations, and much more. The product type is what value your business offers in return for the revenue. The delivery model is how you get your product to the customer. — *location: 1483* ^ref-5911 --- In an e-commerce company, a visitor buys something from a web-based retailer. — *location: 1546* ^ref-29993 --- Conversion rate The number of visitors who buy something. Purchases per year The number of purchases made by each customer per year. Average shopping cart size The amount of money spent on a purchase. Abandonment The percentage of people who begin to make a purchase, and then don’t. Cost of customer acquisition The money spent to get someone to buy something. Revenue per customer The lifetime value of each customer. Top keywords driving traffic to the site — *location: 1627* ^ref-29231 --- Those terms that people are looking for, and associate with you—a clue to adjacent products or markets. Top search terms Both those that lead to revenue, and those that don’t have any results. Effectiveness of recommendation engines How likely a visitor is to add a recommended product to the shopping cart. Virality Word of mouth, and sharing per visitor. Mailing list effectiveness Click-through rates and ability to make buyers return and buy. — *location: 1637* ^ref-37536 --- A SaaS company offers software on an on-demand basis, usually delivered through a website it operates. Salesforce, Gmail, Basecamp, and Asana are all examples of popular SaaS products. — *location: 1818* ^ref-39312 --- Attention How effectively the business attracts visitors. Enrollment How many visitors become free or trial users, if you’re relying on one of these models to market the service. Stickiness How much the customers use the product. Conversion How many of the users become paying customers, and how many of those switch to a higher-paying tier. Revenue per customer — *location: 1834* ^ref-62237 --- How much money a customer brings in within a given time period. Customer acquisition cost How much it costs to get a paying user. Virality How likely customers are to invite others and spread the word, and how long it takes them to do so. Upselling What makes customers increase their spending, and how often that happens. Uptime and reliability How many complaints, problem escalations, or outages the company has. Churn How many users and customers leave in a given time period. Lifetime value How much customers are worth from cradle to grave. — *location: 1841* ^ref-61616 --- Customer acquisition payback is a great example of a single number that encompasses many things, since it rolls up marketing efficiency, customer revenue, cash flow, and churn rate. — *location: 1886* ^ref-55841 --- If your business model most closely resembles a media site, then your primary focus is sharing advertisers’ messages with viewers, and getting paid for impressions, click-throughs, or sales. Google’s search engine, CNET’s home page, and CNN’s website are all media sites. — *location: 2266* ^ref-15280 --- Audience and churn How many people visit the site and how loyal they are. Ad inventory The number of impressions that can be monetized. Ad rates Sometimes measured in cost per engagement—essentially how much a site can make from those impressions based on the content it covers and the people who visit. Click-through rates How many of the impressions actually turn into money. Content/advertising balance The balance of ad inventory rates and content that maximizes overall performance. — *location: 2378* ^ref-46137 --- First, you need empathy. You need to get inside your target market’s head and be sure you’re solving a problem people care about in a way someone will pay for. That means getting out of the building, interviewing people, and running surveys. — *location: 3317* ^ref-13755 --- Second, you need stickiness, which comes from a good product. You need to find out if you can build a solution to the problem you’ve discovered. There’s no point in promoting something awful if your visitors will bounce right off it in disgust. — *location: 3319* ^ref-45635 --- Third, you need virality. Once you’ve got a product or service that’s sticky, it’s time to use word of mouth. That way, you’ll test out your acquisition and onboarding processes on new visitors who are motivated to try you, because you have an implied endorsement from an existing user. Virality is also a force multiplier for paid promotion, so you want to get it right before you start spending money on customer acquisition through inorganic methods like advertising. — *location: 3322* ^ref-976 --- Fourth, you need revenue. You’ll want to monetize things at this point. That doesn’t mean you haven’t already been charging—for many businesses, even the first customer has to pay. It just means that earlier on, you’re less focused on revenue than on growth. You’re giving away free trials, free drinks, or free copies. Now you’re focused on maximizing and optimizing revenue. — *location: 3326* ^ref-16228 --- Fifth, you need scale. With revenues coming in, it’s time to move from growing your business to growing your market. You need to acquire more customers from new verticals and geographies. You can invest in channels and distribution to help grow your user base, since direct interaction with individual customers is less critical—you’re past product/market fit and you’re analyzing things quantitatively. — *location: 3329* ^ref-36353 --- If the problem is real and known, people are dealing with it somehow. Maybe they’re doing something manually, because they don’t have a better way. The current solution, whatever it is, will be your biggest competitor at first, because it’s the path of least resistance for people. — *location: 3428* ^ref-40170 --- Don’t just ask questions. Know how the answers to the questions will change your behavior. In other words, draw a line in the sand before you run the survey. — *location: 3871* ^ref-62815 --- Your job isn’t to build a product; it’s to de-risk a business model. Sometimes the only way to do this is to build something, but always be on the lookout for measurable ways to quantify risk without a lot of effort. — *location: 4015* ^ref-47427 --- This is the reverse Field of Dreams moment: if they come, you will build it. — *location: 4022* ^ref-23636 --- It’s hard to decide how good your minimum product should be. On the one hand, time is precious, and you need to cut things ruthlessly. On the other hand, you want users to have an “aha!” moment, that sense of having discovered something important and memorable worth solving. You need to keep the magic. Clarke’s Third Law: Any sufficiently advanced technology is indistinguishable from magic. —Arthur C. Clarke, Profiles of the Future, 1961 Gehm’s Corrollary: Any technology distinguishable from magic is insufficiently advanced. —Barry Gehm, ANALOG, 1991 — *location: 4024* ^ref-28437 --- But sticking with Lean Startup’s core tenet—build→measure→learn—it’s important to realize that an MVP will go through numerous iterations before you’re ready to go to the next step. — *location: 4047* ^ref-13864 --- If you can’t instrument a feature or component of your product, be very careful about adding it in—you’re introducing variables that will become harder and harder to control. — *location: 4070* ^ref-18379 --- No feature should be built without a corresponding metric on usage and engagement. — *location: 4069* ^ref-28353 --- every experiment starts with a series of questions: What do we want to learn and why? What’s the underlying problem we are trying to solve, and who is feeling the pain? This helps everyone involved have empathy for what we are doing. What’s our hypothesis? This is written in the form: “[Specific repeatable action] will create [expected result].” We make sure the hypothesis is written in such a way that the experiment is capable of invalidating it. How will we run the experiment, and what will we build to support it? Is the experiment safe to run? How will we conclude the experiment, and what steps will be taken to mitigate issues that result from the experiment’s conclusion? What measures will we use to invalidate our hypothesis with data? We also include what measures will indicate the experiment isn’t safe to continue. — *location: 4340* ^ref-11656 --- The minimum viable vision is a term coined by entrepreneur and Year One Labs partner Raymond Luk. He says, “If you’re trying to build a great company and get others involved, it’s not enough to find an MVP—you need an MVV, too.” A minimum viable vision (MVV) is one that captivates. It scales. It has potential. It’s audacious and compelling. As a founder, you have to hold that huge, hairy, world-changing vision in one hand, and the practical, pragmatic, seat-of-the-pants reality in the other. The MVV you need in order to get funding demands a convincing explanation of how you can become a dominant, disruptive player in your market. — *location: 4398* ^ref-46136 --- In 1981, cognitive scientist and economist Herbert Simon observed that we live in an information age, and that information consumes attention—in other words, attention is a precious commodity, and its value grows as we’re flooded with more and more information. — *location: 5156* ^ref-59384 --- In your current business model, you have some fundamental assumptions, such as “people will answer questions,” or “organizers are frustrated with how to run conferences,” or “we’ll make money from parents.” Some of these may be platform assumptions too: “Amazon Web Services are reliable enough for our users.” Each assumption has a metric associated with it, and a line in the sand. This is your big bet. These are the cells in your spreadsheet that you obsess over as a board. They’re what you look at to see if you can make payroll, or how much investment you’re going to need, or whether the marketing campaigns are bringing in more than they’re costing, or whether your business model is hopelessly, fatally, doomed. — *location: 5237* ^ref-4329 --- Know, clearly, what assumptions underpin your fundamental business model. Then, with the approval of stakeholders, change one of them. Hand that change to the executive team: which features do you think will improve that basic assumption? Plan out your daily activities to test those features: have conversations with customers, run surveys, create a segment that tests the new code, try mockups. This combination of agility and methodical precision is what distinguishes great startups from stalled ones. — *location: 5267* ^ref-8430 --- As you grow, you’ll need to have more than one metric at a time. Set up a hierarchy of metrics that keeps the strategy, the tactics, and the implementation aligned with a consistent set of goals. We call this the three threes. — *location: 5280* ^ref-33651 --- The Startup Genome project has collected key metrics from thousands of startups through its Startup Compass site.[ — *location: 5448* ^ref-3228 --- Simplicity isn’t just an attribute of enterprise disruption—it’s the price of entry. DJ Patil, data scientist in residence at Greylock and former head of product at LinkedIn, calls this the Zero Overhead Principle: — *location: 6924* ^ref-62651 --- A central theme to this new wave of innovation is the application of core product tenets from the consumer space to the enterprise. In particular, a universal lesson that I keep sharing with all entrepreneurs building for the enterprise is the Zero Overhead Principle: no feature may add training costs to the user.[139] — *location: 6926* ^ref-41067 --- In 2005, IEEE (Institute of Electrical and Electronics Engineers) committee chair Robert N. Charette estimated that of the $1 trillion spent on software each year, 5–15% would be abandoned before or shortly after delivery, and much of the rest would be late or suffer huge budget overruns.[140] A similar study by PM Solutions estimates that 37% of IT projects are at risk.[141] — *location: 6941* ^ref-1401 --- For all these reasons, most B2B-focused startups consist of two people: a domain expert and a disruption expert. The domain expert knows the industry and the problem domain. He has a Rolodex and can act as a proxy for customers in the early stages of product definition. Often this person is from the line of business, and has a marketing, sales, or business development role. The disruption expert knows the technology that will produce a change on which the startup can capitalize. She can see beyond the current model and understand what an industry will look like after the shift, and brings the novel approach to the existing market. This is usually the technologist. — *location: 6948* ^ref-55671 --- Having said that, consulting companies struggle a great deal to transition from service providers to product companies because they need to, at some point, abandon service revenues and focus on the product. That transition can be extremely painful—from a cash flow perspective—and most service providers don’t make the jump. It’s also necessary to “burn the boats” of the services business to ensure that you commit to the product. After all, you’re going to neglect some of your most-loved customers in order to deliver a product the general market wants instead, and it’ll be tempting to do custom work to keep them happy. You can’t run a product and a services business concurrently. Even IBM had to split itself in two; what makes you think you can do it as a fledgling startup? — *location: 6985* ^ref-21238 --- In the B2C world, startups worry less about “Can I build it?” and more about “Will anyone care?” In the enterprise market, the risk is more, “Will it integrate?” Integration with existing tools, processes, and environments is the most likely source of problems, and you’ll wind up customizing for clients—which undermines the standardization you fought so hard to achieve earlier. — *location: 7046* ^ref-30981 --- In many B2B-focused companies, the top 20% of customers generate 150–300% of profits, while the middle 70% of customers break even, and the lowest 10% of customers reduce 50–200% of profits.[146] — *location: 7125* ^ref-3097 --- David Boyle is the Senior Vice President of Insight at EMI Music, one of the major labels in the recording industry. His job is to help EMI make decisions based on data, and to help the company navigate the choppy waters of an industry in transition. To get the company more focused on data and analytics, and less concerned with anecdotes and opinions, Boyle first had to choose which decisions needed to be made, then find ways to get the right evidence in front of the decision makers. — *location: 7404* ^ref-49544 --- Boyle wasn’t short on data. EMI has billions of transaction records from digital services, as well as usage logs from artist websites and applications. “But each of these data sources is very limited in scope and very skewed concerning the types of person that is represented in that data set,” Boyle explained. So EMI built its own survey tool. “We found that building our own data set based on asking people questions and playing them music was the way to go.” The result was over 1 million detailed interviews, and hundreds of millions of data points. — *location: 7410* ^ref-40167 --- To build support and report progress within EMI, Boyle used case studies. — *location: 7427* ^ref-41312 --- “We got lots of people who’d successfully used our data to help their artists tell their story. They were better and more creative than anything we could have organized centrally to spread the word.” — *location: 7428* ^ref-36860 --- Boyle didn’t tie the results of research to hard numbers. “We simply said: ‘Asking thousands of people what they think about something is better than not asking them, right?’ and we showed that we could do so at high quality and low cost, and we went for it. — *location: 7430* ^ref-58440 --- “Good data beats big data,” he concludes. “I am constantly surprised at how good it can be when done properly.” — *location: 7439* ^ref-13465 --- We started by saying that if you can’t measure something, you can’t manage it. But there’s a contrary, perhaps more philosophical, observation we need to consider. It’s a line by Lloyd S. Nelson, who worked at Nashua Corporation. “The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.” — *location: 7575* ^ref-55797 --- Don’t go after the most crucial issue your company is facing—that’s likely got too many cooks in the kitchen already (or worse, it’s mired in politics you don’t want to wade into). Instead, pick an ancillary issue, something that can add demonstrable business value but is being overlooked. — *location: 7595* ^ref-49379 --- Once, a leader convinced others to act in the absence of information. — *location: 7632* ^ref-8979 --- Today’s leader doesn’t have all the answers. Instead, today’s leader knows what questions to ask. — *location: 7634* ^ref-13036 ---