🌱 5-Bit Fridays: Running experiments that don't suck, successful sales calls, how to convince someone in <50 words, the power of daily touchpoints, and developing your technical chops as a PM

#31

👋 Welcome to this week’s edition of 5-Bit Fridays. Your weekly roundup of 5 snackable—and actionable—insights from the best-in-tech, bringing you concrete advice on how to build and grow a product.

---

---

Happy Friday, friends 🍻

Here’s some useless information I just came across to get us started. The world record just broke for solving a Rubik’s Cube. Can you actually imagine how your mind must work to solve one in 3.13 seconds? Nope, me neither. 🫡

Anyway, if you’re out having a drink with someone this weekend, you can impress/bore them with that trivia. In more relevant news…

I’m not sure how much of the Reddit debacle around their API changes (i.e. making people pay for it in the ramp-up to Reddit’s IPO) you’ve been following. But, originally expected to only go dark for 48 hours, more than 300 subreddits so far have already decided to go dark indefinitely. Read about it here.

As the world’s leading community platform…that’s pretty substantial.

And lastly, in case you missed Wednesday’s deep dive, a link is below. I had a lot of fun writing this. You might be able to tell. ⛹️‍♂️

Why Quibi Died: The $2B Dumpster Fire That Was Supposed to Revolutionize Hollywood

Unpacking 7 Reasons Why the "Netflix for your phone" shut down in just 6 months, and lessons from our second trip to the startup graveyard

Onto today’s post…

Here’s what we’ve got this week:

• Running experiments that don't suck

• How to run a successful sales call

• How to convince someone in <50 words

• The why and how of daily user touchpoints

• Developing your technical chops as a PM

Small ask: If you learn something new today, consider ❤️’ing this post or giving it a share. I’d be incredibly grateful, as it helps more people like you discover my writing.

(#1) How to run experiments that don't suck

Earlier this week,

Peter Yang

shared his conversation with Eric Metelka. Eric is the Head of Product at Eppo, an experimentation platform empowering product teams to run reliable and impactful experiments.

In Peter’s interview, they spoke about:

1. When to run experiments

2. How to run experiments that don’t suck

3. How to introduce experimentation to your company

Given how important running things like A/B tests are (as we saw with Duolingo), I thought I’d share some of Eric’s first-hand insights into experimentation.

Here’s a summary:

When to run experiments

You should run experiments if you need to…

1. Prove causality. We’ve all attended meetings where people question if a product moved a metric. An experiment removes this uncertainty.

2. Measure impact. For example, you may run an experiment that grows engagement by 3% but hurts revenue by 1%. Measure impact to make decisions.

You should not run experiments if you need to…

1. Make a decision quickly. Experiments usually take time and work to run. They shouldn’t be used to make every product decision.

2. Bet on a 0-1 product. If you’re building something new, you might have too few users or not enough data to run an experiment.

3. Do a big launch. If you need to make a big public launch, it’s hard to experiment.

— Erik Metelka

To add to that, here’s an awesome graphic

Aakash Gupta

put together:

Source; Aakash Gupta, Product Growth

How to run experiments that don’t suck

Three [best practices] come to mind:

1. Be precise about your outputs, inputs, and guardrails. Don’t try to move too many output metrics and be sure to include inputs and guardrails.

2. Estimate how much the output metric will move. This helps you determine if you have enough users to achieve the desired impact (e.g., sample size calculator).

3. Check your experiment's health. Check how many users are in the test and control groups and whether your numbers make sense before and after launch. A bad experiment can easily burn a month of your team’s time.

Erik also calls out 2 non-obvious effects that may dilute the results of your experiments:

1. The Novelty Effect. Like giving a new shiny toy to a kid, there’s a bias for them to play with it more in the beginning. A similar thing can happen when testing a new feature. To avoid this, aim to run the experiment for at least 2-3 weeks before drawing any conclusions. Another significance in a test is often measured by traffic volume, but it’s not just about the number of people seeing the test, it’s about how that variable performed over time. So, even if you get a high volume of users into your test very quickly…let it run. This also ensures you get a better spread in your data, as, for instance, Wednesdays may perform very differently from Fridays.

2. The Interaction Effect. Sometimes pulling a nob that affects one type of customer can impact (unexpectedly) another. This is especially true in two-sided marketplaces, where a change on one side could impact the other side. In the context of experiments, one experiment may start impacting the results of another. To avoid this, test different hypotheses and try to space your experiments out. That’s why Erik says that guardrail metrics (AKA high-level metrics or adverse effects to monitor, like overall revenue and the number of support tickets) are so important.

How to introduce experimentation to your company

At a later stage company, this shouldn’t really be a problem as your company should have some testing culture already. However, let’s say you’re the PM at an earlier-stage startup that hasn’t gotten there yet and you want to convince leadership on experimentation. Here are three approaches Erik suggests:

1. Share examples of companies that have grown (and are growing) through experimentation (like Duolingo). The more relatable the product is to yours, the more convincing.

2. Emphasized how experiments can help you find the truth faster, and make more data-informed decisions

3. Find 3rd party tools (like Eppo or Amplitude) that make it easy and affordable to start running experiments in a scrappy way.

And a last nugget of wisdom from Erik:

You shouldn’t introduce it if you are pre-product market fit or lack scale. A good signal that your company needs experiments is when people start asking:

1. “How should we optimize this page?”

2. “How do we move more people down this funnel?”

3. “How are we going to measure impact?”

Further reading 🧠

• How Duolingo Grows

• When to A/B Test

• AB Testing Mistakes (podcast)

---

(#2) How to run a successful sales call

Or more expertly said by Emery Rosanksy (VP at First Round Capital):

Founders are an innately curious bunch. But for some reason on sales calls, they’ll often just talk at the prospect for an hour, throwing them the kitchen sink versus getting deeply curious about what sorts of problems the prospect is facing and how their product might help.

I was very recently on the receiving end of a sales call with Amplitude. My team and I use Amplitude extensively at the moment for Analytics and Experimentation, but we’ve been interested in their Audiences product, so we’ve been going through the upsell sales process.

While on the call, I felt I was observing a really well-oiled, nicely polished, sales-led framework. Wanting to clarify my own understanding of what I was witnessing and learn more about what makes a good sales call, I did some searching and quickly came across an article by First Round Capital, where Emery Rosanksy shares some of her hard-won wisdom on sales.

Here’s a summary of what I learned, specifically, about what separates a great sales call—one that closes a customer—from a kicking-the-can one. AKA: “We’ll get back to you.” Founders who are selling (or need to start selling)… this one is for you.

1. Start the call by defining: “Purpose, Benefit, Check”

How many times have you been on a call with no clear purpose or agenda? It’s maddening, right? But time after time, folks dive right into their pitch deck without first laying out the path ahead for the duration of the call.

This is why Emery pushes the PBC method, namely, “Purpose, Benefit, Check”. Simply, it’s an excellent (often underutilized) tool that creates verbal acknowledgment and agreement with your prospect about why you’re having this call, and what the outcome should be. Here’s the example she gave:

Thanks again for taking the time to chat with me today. We initially set this call for 30 minutes, does that time still work for you? The purpose of the call is for me to learn more about your process and challenges for tracking SaaS metrics and share a bit about how [insert product] can solve that problem for you. At the end of the call, we can decide together if it makes sense to move to the next step in the process which is [insert next step details]. How does that sound to you?

Why do this?

• It confirms alignment with your prospect on how they want to spend the time on the call. If there’s any confusion, find out earlier rather than frustrating someone and wasting time.

• It creates a nice opportunity for you at the end of the call to ask for a decision on the next step.

• It earns you credibility and trust for the professional and polished approach. There’s a saying that goes something like “First tell them what you’re going to tell them. Then tell them. Then tell them what you just told them.”

2. Dig deep to find the impact of the problem, not just the problem.

Emery has found that most founder sales calls end up being situational discovery questions. For example, “What challenges do you have in your billing process today?” or “What tools are you using to manage your vendor billing?”

But, as with most things, digging deeper and pulling on threads is where the good stuff is buried. ⛏️

It’s not enough to know the pain the prospect has, you want to deeply understand the impact of that pain. That’s how you create urgency and turn a nice-to-have into a must-have solution.

3. Use peoples’ own words back at them.

If you do a great job with discovery on this first call, you’ve likely uncovered some really meaningful insights about your prospect. So what do you do with those insights? You leverage them to make an even stronger connection between the prospect’s challenges and the solution you’re building.

Let’s say during discovery your prospect told you they spend 20 hours per week dealing with vendor billing issues. Later in the call, when you’re demoing a feature in your billing platform you would simply say, ‘Earlier you mentioned you’re spending 20 hours a week dealing with vendor billing issues. With [xyz feature] you can eliminate those billing issues. How would this impact your workflow?’ Most sellers would just demo the feature and expect the prospect to make the connection on their own.

This touches on a powerful communication principle: mirroring.

4. People remember stories. So, use them.

Numbers don’t create emotional connections. Stories do.

That’s why it’s far more powerful to tell a story about a specific customer and how they went from struggling to scaling with your solution, compared to bombarding someone with numbers (even if they are really good).

A great sales call finds opportunities to integrate storytelling. As Emery describes:

This can be the company origin story, stories about other customers who had similar challenges to the one your prospect is sharing, stories of customers who had similar objections to your product that the current prospect has (and how they overcame them) or stories of customers who ‘won’ using your product.

5. Don’t rush the close.

There are multiple critical boxes to check before you close the call, and folks end up in a race to beat the clock. Leave yourself enough time — no matter where you are in the demo, stop at the five minutes left mark to start closing out the call.

Here are four steps to ending a sales call the right way.

1. Ask for feedback. A simple, ‘I’ve shown you a lot today, what are your thoughts right now?’ can tell you a lot about whether to continue investing in the relationship. This is the moment to close that PBC loop.

2. Outline the next steps. Always aim to schedule the next conversation while you’re still on the call. Once you hang up, it can be near impossible to get that person’s attention again and you’ll lose momentum.

3. Follow up the same day. Always get a follow-up email out before you go home. This should include a summary of the call, what you learned, plus any action items you owe them.

4. Share out. If you want candid feedback about how you did on the call, then send a recording (which you should be doing) to someone you trust to share a few notes.

Further reading 🧠

• Founder-led sales (podcast)

• The Greatest Sales Pitch I’ve Seen All Year

• How I’ve approached founder-led sale

---

(#3) How to convince someone in <50 words

Here’s a lovely short, and very true, quote by

Julie Zhuo

from The Looking Glass.

How do I convince X that my idea is great, that we should do A and not B, or that I deserve a promotion?

1. Do you know what X cares about?

2. Can you be genuinely helpful on what they care about?

3. If not, try asking:

   1. What do you care about?

   2. What can I do that would be helpful for you?

4. If your proposal fits #2, then pitch it.

5. If not, find a different route.

Further reading 🧠

• How to get better at influence

• Tobi Lütke: The Trust Battery

• How To Influence Without Authority As a Product Manager

---

(#4) The why and how of daily user touchpoints

A core part of the PM’s and designer's job is being close to the customer. But, things are always popping up to fill up our plates, and often “speaking to customers” gets pushed off the to-do list so we can make time to knock out that last minutes deck a manager asked for.

In the end, we often end up talking with users occasionally, and as Bart Krawczyk says, “That’s not enough.”

Daily touchpoints with customers change everything.

Why?

• They allow us always to be up-to-date with our customers. Okay, our users don’t change overnight, but our product, the market and the environment around us do change quickly, and we should be aware at all times how it impacts our users.

• They help us to stay user-centric. Deadlines, business priorities, competition, regulations, revenue, stakeholders, ambitious pet projects and yearly goals… in a complex world of product development, it’s easy to be carried away by other priorities and unconsciously push user advocacy to the background. Having daily user touchpoints helps us stay grounded in user-centricity.

• Daily interviews cultivate the continuous discovery culture. Building a product is setting one hypothesis after another and checking them constantly with experiments. Alas, the bigger the product gets, the more rusty the discovery habits tend to become — the backlog of things we have to take care of just keeps growing. Daily touchpoints are like a habit of morning runs — they make sure that however busy life gets, we stay in shape.

— Bart Krawczyk, The Incredible Power of Daily User Touchpoints

Okay, but having a user interaction every day sounds like an unattainable goal. And while Bart argues for it (likely as a hyperbole), frankly, it’s an impossible goal. However, what is feasible is probably increasing the number of touchpoints you currently have as a PM. I know I can.

So, how do you do that?

• User interviews (with automated scheduling). The main headache with having more interviews is the logistics of getting them on the calendar. The key is automation, and to automate, you can:

  • Recruit directly from your product/site. Try implementing a banner ad, or using a tool like Hotjar (they actually just launched a neat user interview tool), that invites users to schedule 30m calls using an automatic scheduler like Calendly.

  • Setup triggered emails. This is a personal favorite that drives most of my general calls. Say you’re using HubSpot with some decent contact properties in there. You can create lists of users based on some combo of audience and engagement requirements (i.e. hasn’t logged in in X days, just performed X event, etc), and send automatic emails personalized to that situation, asking for an interview.

  • Recruit from sales. If applicable to your company, ask your sales team to schedule some interviews as part of their sales process. For instance, people that just bought the product or refused to buy it are great sources of insights.

  • Recruit from customer support. Similarly to the above, ask your CS team to funnel certain people your way.

• Doing occasional customer service activities. Not all touch points have to be speaking to customers. There are other ways to keep your finger on the pulse, such as:

  • Actively answering customer requests (usually only doable at smaller companies)

  • Going through older (hopefully tagged) support tickets

• Sitting in on sales calls. Again, you need to apply judgment here, but if you’re able to get added to some sales calls, it gives you a chance to see directly (in the context of selling the product) what kind of objections or questions potential customers have. An extra benefit is you’ll see if there’s alignment between how the sales and product teams perceive and communicate the product.

• Reading reviews. For instance, on:

  • Reddit, app stores, FB groups, other sites, your own forum or community pages if you have them, or anywhere else your customers are talking about you.

• Surveys. Besides exploratory surveys you might send out to a segment of people when you’re trying to learn something specific, you can have different automatic surveys going, helping you get a stream of insights coming in. For example:

  • Regular NPS surveys

  • Setting up an exit survey for people who drop out of a conversion funnel

  • Using a churn survey to understand why people stop using your product (either direct cancellations or lack of product usage for extended periods)

  • Sending a welcome survey for brand new customers (or baking it into product onboarding)

• Go spend time where your users hang out. This could be:

  • Meetups

  • Conferences

  • Online forums

  • Communities (Discords, Reddits, Slack Groups)

Further reading 🧠

• Create an Automated Customer Research Machine in 2 Hours

---

(#5) Developing your technical chops as a PM

Product managers don’t need to be engineers. It’s not our job to step on toes and waste time getting too deep into the codebase. That’s what the engineers are for. But, the more we understand the technical side of things (like the development process), the better we can move things along, prioritize, and earn more trust from our engineering partners.

So, how do you level up your technical knowledge and find that sweet spot?

Justin Gage—author of the Technically newsletter (fittingly described as “a place for not software engineers to learn about software engineering”)—shared his actionable tips on becoming a more technical PM a while back on Lenny’s Newsletter. According to him, there are the top 5 things PMs should learn about engineering.

1. Learn your company’s tech stack

   1. What to know: The technologies your product runs on (i.e what front/backend languages, database, server infrastructure)

   2. Why it matters: The languages, frameworks, and infrastructure being used all come with important trade-offs. And tradeoffs can impact things like how long it might take to build a certain feature. Also, as this will apply to all the points below, the more you can speak the language of your engineering team, the more trust you’ll build.

   3. How to learn about it: The easiest thing to do is just ask a developer.

2. Learn sensitive points in the codebase

   1. What to know: You want to know which pieces of the codebase are tangled, sensitive, or just near-impossible to change.

   2. Why it matters: Knowing the hardest parts of the app to work on can help you plan more effectively, like resource allocation and road mapping.

   3. How to learn about it: If there are seasoned PMs on your team, ask them about projects that went surprisingly quickly or surprisingly slowly. Also, speak to developers.

3. Learn your build and deploy process

   1. What to know: How your app gets built and deployed to your users, what’s manual and what’s automated, and how long it usually takes.

   2. Why it matters: Larger teams and more complex apps can have multi-day deployment processes that may delay projects and add to lead time. Knowing these pressure points can also help you more effectively batch features and understand what’s worth doing and when.

   3. How to learn about it: Ask a developer.

4. Learn how to contribute some code

   1. What to know: How to make code changes and open a PR, specifically for small, self-contained changes like updating copy or changing colors.

   2. Why it matters: Being able to make very small changes yourself, like fixing a typo, will result in more polished products shipped faster. But, as Justin notes: “Don’t bite off more than you can chew, though; keeping the code you contribute to smaller copy- or design-related tweaks will win you points.”

   3. How to learn about it: Sit down with your favorite engineer and ask them to walk you through how to contribute code.

5. Learn technical basics

   1. What to know: Outside of your own codebase, you want to have some understanding of how apps, data, and infrastructure work together.

   2. Why it matters: This gives you a conceptual understanding of what pieces come together to build applications, and without it, prepare to listen to lots of gibberish in any engineering discussion. Whether you know how to code or not, you’ll want to know what your developer means when they say they can keep this feature to a “frontend change only.’”

   3. How to learn about it: Outside of getting hands-on experience, the easiest is to read engineering newsletters geared towards non-tech-heavy folks, such as Justin’s Technically or ByteByteGo , watch YouTube videos, or even take a basic (free) Codecademy course.

Further reading 🧠

• 5 Technical Skills Product Managers Should Have

• How do I get more technical? (more details by Justin)

---

🌱 And now, byte on this if you have time 🧠

By now, you know we don’t really have a problem getting to space “cheaply” anymore. SpaceX has laid down some essential infrastructure, with the ability for companies to use their rockets like an Uber to space.

So, the next question… what do we do up there now?

Well, earlier this week, Varda launched a commercial space factory into orbit.

Interested? 🌝

Read | Varda: The Space Drug Factory, by Packy McCormick

---

And that’s it for this week!

As always, thank you for reading and giving me some of your time today.

If you learned something new, or just enjoyed the read, the best way to support this newsletter is to give this post a like, share, or a restack — it helps other folks on Substack discover my writing.

Or, if you’re a writer on Substack, enjoy my work, and think your own audience would find value in my various series, I’d love it if you would consider adding HTG to your recommendations.

Until next time.
— Jaryd✌️

---