[00:00:00] Crystal Widjaja: Everyone wants quality traffic. What is quality? It's a mix of breadth and depth. So you have a big enough market, target market segment who will use the product enough. And there are enough of them to provide your company value for the amount of work that you spend. There's also that maybe very perfect ICP, ideal consumer profile that's very deep. They're very sticky. They never leave the platform. There may not be as many of them in terms of the breadth of the target market, but once you can identify them and you provide an onboarding experience that's tailored to them, they get it. They get the product. They stick.

[00:00:00] Andrew Michael: This is Churn.FM, the podcast for subscription economy pros. Each week, we hear how the world's fastest growing companies are tackling churn and using retention to fuel their growth.

[00:00:55] VO: How do you build a habit forming product? We crossed over that magic threshold to negative churn. You need to invest in customer success. It always comes down to retention and engagement. Completely bootstrapped, profitable and growing.

[00:01:08] Andrew Michael: Strategies, tactics and ideas brought together to help your business thrive ‘
in the subscription economy. I'm your host, Andrew Michael, and here's today's episode.

[00:01:20] Andrew Michael: Hey, Crystal, welcome to the show.

[00:01:21] Crystal Widjaja: Thank you for having me. It's great to be here.

[00:01:23] Andrew Michael: It's great to have you. For the listeners, Crystal is an angel investor and advisor to companies like Scale.AI, Bounce and Maze. Crystal is also an Executive-in-Residence, a Venture Studio Mentor at Stanford University, and was previously the chief of staff supporting the co-CEOs at the GoTo Group, the largest technology group in Indonesia. So my first question for you, Crystal, is what was the transition like from SVP of Business Intelligence and Growth at Gojek to Chief of Staff for the GoTo Group and what motivated it?

[00:01:52] Crystal Widjaja: So I think if you become the Chief of Staff, hopefully you've been doing the job already. And so I was already informally doing that job. If you look at the rest of my CV, it goes from data to fraud and risk to performance marketing to growth, whatever growth was at that time. And so it was already picking up the pieces of, you know, this thing is important, we need someone who has a lot of context and, you know, history of how the company works and who can do what to kind of fill in the gaps and patch things up so that the company can keep humming along.

[00:02:23] Crystal Widjaja: So I think there was less of a transition and more of a slow gradual bleed into it of saying, hey, the company has acquired and merged with three other companies, we have three VPs of sales, you've got two HR heads, what do we do? And having been there since the very beginning, it made sense to kind of look at the company as a overall portfolio of businesses that we had. We had transportation, logistics, food, payments, and 20 other things going on. And I had been there since the beginning when we only had three services.

[00:03:00] Crystal Widjaja: So patching it together, building systems, it actually was like building a growth organization internally where we had, internal communications was one of the first teams that I put together as a chief of staff, because if the company doesn't know what's happening, they certainly don't know how to communicate that message to the users, to the other third parties in our ecosystem, like the drivers and the merchants. So I kind of saw it as, like a macro zoom out of being a head of growth in Gojek.

[00:03:28] Andrew Michael:  Just internally and getting everyone aligned. Nice. Yeah. So I think the one thing I think about the chief of staff role is… was… there's often different definitions of it. Like one is, like this glorified assistant type role. And then there's this other one that's really this like strategic assets. It's sort of like this second arm to the CEO, really like filling in for the CEO when they're not available.

[00:03:51] Andrew Michael: And it's like, it's definitely a role that's interested me as well, like as a founder and like, would wanting to work at like different places, like potentially something more on the latter, and how did you go about sort of defining this role then to, like avoid these connotations.

[00:04:07] Crystal Widjaja: That's a great question. So my hot take on this, having been in that role, informally and formally, is that you actually have to want to do both. And one of my favorite chief of staffs or just CEOs in general is Akshay Kothari, who's at Notion. And he was like, if it means that I need to get lunch for the office for everyone to be more productive, like, that's just one of the things that I do. If it means I'm herding cats, trying to get everyone into a meeting, if I'm rescheduling things for other people, I'm playing a glorified therapist in a way, as the chief of staff to listen to other people's complaints, make sure that they get heard by the CEOs.

[00:04:44] Crystal Widjaja: That is part of the job as important as it is to pick up strategic projects and figure out, hey, we are spending a massive amount of money on cloud expenses. No one knows what's happening on AWS versus Google. Like this is a strategic project as well. And so I think I fully embraced and loved what I did and the company and the spirit of what we were all trying to do. And I knew everyone had wanted to do their best work. So how could I help facilitate that?

[00:05:12] Crystal Widjaja: And probably the first Chief of Staff project I did is as simple as making a map of where all the meeting rooms are in the office with 30 meeting rooms. And I could literally count the minutes that were wasted in the company every hour where people were just turning left and right, trying to find a meeting room. And so we just put a map together and that helped everybody. 100% impact.

[00:05:37] Andrew Michael: It sounds funny, like something like this, but like at Scale when you have a huge workforce as well, like there's minutes, makeup hours that–

[00:05:44] Crystal Widjaja: They do.

[00:05:45] Andrew Michael: And years and yeah. So it can become a very big impact overall to the bottom line. Very nice. And then prior to that, obviously, like you spend a lot of time around data, the SVP of business intelligence. Now as well as Executive-in-Residence at Reforge, you've been putting out quite a few interesting artifacts of late as well when it comes to data, but also specifically around activation and abandonment, which is one of the ones I think will be good about for us to chat about today.

[00:06:14] Crystal Widjaja: Definitely.

[00:06:15] Andrew Michael: First of all, what motivated you to put this sheet together? Obviously, you're now working with multiple different startups and you're advising. Where did the inspiration come from, from the outset to put it together?

[00:06:27] Crystal Widjaja: So the first time that I met with Brian Balfour, who's been on the podcast as well, a great episode. Yeah. I was a guest on the Growth series where I talked about hacky experimentation being the defining growth person at a company where you know what data exists, you know how much fraud is happening, what all of the levers are, what the KPIs are and what looks normal. And I think this is the key part is that if you are on a data driven startup, you need to know what normal looks like.

[00:07:01] Crystal Widjaja: And if you've been staring at dashboards every day, you build up this muscle. I knew when things went wrong and if something looked weird in terms of retention, activation, revenue, then you have this baseline. And so I ran that as a guest on Growth series, talked through, you know, one of the experiments that we ran. And one thing that came up from a lot of our members in Reforge was I don't know how to track this or know when things aren't normal because I'm not tracking any data.

[00:07:31] Crystal Widjaja: And so I'd start looking at the instrumentation specs that people would use for a segment or at the time things like Mixpanel and Amplitude. I'd say, well, you're tracking your KPIs, right? You'll know how many people landed on your home screen. You'll know how many people clicked order. But you have no idea of the segments in between these, such as the types of people who would click order, but versus the types that wouldn't. And that requires tracking event properties. So that led to the blog post, why most analytics efforts fail.

[00:08:05] Crystal Widjaja: And I think it's here where you start to get a sense of what data needs to be collected and how do you think about user journeys rather than KPIs when you're instrumenting this tracking in your event spec so that you can do analysis later. It's almost… It's not intuitive to go to, look for ways to track data so that I can analyze it later. Because you think that tracking the data is the project when it's not.

[00:08:32] Andrew Michael: No, I fully, 100% agree with that as well. And myself as well, previously at Hotjar, 100% there we were at a state where in the beginning, like the focus was just on tracking data. And then there wasn't much thought or effort put into the tracking plan or thinking through like what the next steps were. And I'd say it was like a huge unlock for us when we basically at some point just said, okay, we're going to start building this team now. We're going to scrap everything. And we're going to be really thoughtful and deliberate when we start to think about what we want to track, what are the properties we need to track? How is that going to unlock different segmentation and sorting and filtering?

[00:09:04] Crystal Widjaja: So the, I think the most obvious example for listeners that everyone will have to go through probably is a sign up. So I often see this mistake of signup, a Google signup selected, email signup selected, phone number signup selected. Whereas the right level of abstraction here should be sign up successful or sign up confirmed and properties of Google versus email versus phone, and then properties of success versus unsuccess, so that you can start to compare one event of sign up completed to AHA moment completed.

[00:09:47] Crystal Widjaja: So are certain authentication segments more likely to reach your aha moment. Let's say Google users get a head start because you were able to collect their calendar notifications. And now my Ask Gemini has now popped up and said, are you listening? Are you trying to get me to talk to you? But I am not.

[00:10:10] Andrew Michael: We're not.

[00:10:10] Crystal Widjaja: But that could be a real reason. And no one is able to do that analysis because they have three different events, Google sign up, email sign up, phone sign up, and they're not able to connect it to the AHA moment. They have to run three analyses instead of one.

[00:10:23] Andrew Michael: And your specific examples, I think this is where, like understanding user traits and if you're using, like group level traits as well.

[00:10:30] Crystal Widjaja: Exactly.

[00:10:31] Andrew Michael: Storing that information in the user trait as well as like, sign up source and then having it as like their email or so forth. ‘Cause then it allows you further down the funnel as well for more interesting analysis as well. Yeah, I think that's like, definitely one of the key missteps and actually this was a conversation where I had with Eleanor Verner on the show. I'm sorry, not Eleanor Verner. And I'm losing the name now as well. So I'm going to have to cut this afterwards and we'll fix it.

[00:10:54] Crystal Widjaja: We'll cut it, don't worry. You'll fix it in posts.

[00:10:57] Andrew Michael: Yeah. So, you know, this was a previous discussion we had on the show as well, where at Segment at some point, Segment Analytics, you referenced them earlier, is that during, in the early days, they just allowed anybody to get started with their product and get set up and get running. And then at some points they realized, okay, this wasn't working for them. Cause essentially what was happening was that the whole premise and value prop of Segment was like to have this good, clean datasets that comes out and gets sent to all the different tools.

[00:11:25] Andrew Michael: And it's unified across the board, but allowing people just their own free will, just to go, set things up. They ended up in the same position. They were in the beginning when they got started with this spaghetti mess of data, because there wasn't a thoughtful process into the tracking plan. And actually like adding a layer of friction into their onboarding experience.

[00:11:41] Andrew Michael: So they said, okay, in order to get started, you're going to work with the CS rep, you're going to put together a good tracking plan, and then you're going to go get started. Rarely had, like a huge impact on activation, on retention, overall. And I think this is like, goes to like speaking to how important really is the step that you're stating of having a good solid sense of understanding of what you're going to track. So.

[00:12:03] Crystal Widjaja: That makes sense. And I think it makes, sense to you more now that I'm reflecting, how many people are the first person to set up the entire Segment project and to set up the entire event tracking spec. Usually it's one person at a company. And so already you have far fewer people who are experts at this. They may have experience with adding or instrumenting new events for new features, but they've never had to look at it from an overall abstract level and try to create sanity before everyone else onboards onto the platform. And that is where you learn a hard way how to abstract.

[00:12:35] Andrew Michael: Absolutely. And yeah, I think that was one of the things as well that came out was the episode with Eleanor Dorfman we were discussing. And that was sort of one of the other strategic things is like, having a couple of people on that call and from different teams as well, allowed different perspectives to come in and like spot things that were maybe missed before they got started with the whole implementation exercise. And then it was too late at that point where you've already created this cloud of missed it. Then nobody trusts and what's the source of truth? And like you get all these irrelevant questions most of the time, but things that people tend to lean back on as excuses.

[00:13:08] Crystal Widjaja: And it's an exponential problem to fix. Bad data does not just stay bad data. It starts piling up and ruining your reputation at the company. The product that you're using, Segment or Mixpanel or whatever it may be, stops being trusted and no one wants to use it. It's all a nightmare.

[00:13:23] Andrew Michael: Honestly, I'm growing more and more to believe though, that's like using data, like issues as an excuse, is a very bad excuse for, like, bad performance. And it's like, it's very easy to hide behind the data and say, oh, well, we don't have the sources or we don't understand this, but it's, I think it's more important really just to address the issues head on and realize, okay, like there's some things just knowing the signal is enough to know that something's not working or not. And it doesn't need to be perfect for you to understand that things need to change.

[00:13:55] Crystal Widjaja: We always, Gojek and Kumu and all of the startups that I've worked with, even today at Naked Wines, not a startup, it's a public company in London. But I hear the same, you know, issues wherein we don't have enough data to be sure if this project is working, the tracking's not perfect. And I'll say, yes, tracking will never be perfect. This cannot be a permanent excuse. You'll always have GDPR, cookies issues. You'll always have some random bug on some random Android device. We all wish everyone used Google Chrome and iPhones, but that's not our reality.

[00:14:28] Crystal Widjaja: And so some companies would kill to have thousands of users every day versus their 10 or 1 user per day. And so you always have enough if the question is, is it directionally correct? So we would do things even as simple as, you know, have a person stand on a corner of a sidewalk and count how many drivers drove by on a Gojek, how many of them had a passenger and that was our competitor market share test.

[00:14:52] Andrew Michael: But it gives you a good signal and good point as well.

[00:14:55] Crystal Widjaja: Exactly.

[00:14:56] Andrew Michael: Yeah. And I think it's like these onboarding questionnaires as well, like, how did you discover us as well? And when we think about attribution, like sometimes those are just way more effective than trying to put together the sophisticated attribution model and figure out how things are.

[00:15:09] Crystal Widjaja: Randomize it, leave five options, don't make it hard to think, and it's good to go.

[00:15:14] Andrew Michael: It's good. Yeah. So let's talk about getting good to go and activation. You put together these Segment retention funnels, the cohort activation abandonment funnel. I'm keen to dive into a little bit more detail on this and what is the purpose of putting something together like this? Why should I start off speaking about putting together these cohort views?

[00:15:34] Crystal Widjaja: Okay. So what is this for those of you who aren't, you know, visually looking, I'm trying to describe it, it's you have your first signup event whenever you first see your user, your next step in this funnel is one or two setup moments. You know, what are the adjacent journeys or things that users have to do to get to your AHA moment, and it should be two or three steps. And then step four is going to be your AHA moment. Step five is going to be the second time the user does the AHA moment.

[00:16:04] Crystal Widjaja: And now normally in a cohort analysis, you'll look at one and five or one and four. I signed up, I did the AHA moment. In a funnel analysis, usually you'll see like 10 or 12 different steps and they're not all mutually exclusive. They're not all sequential. And so this is a best of both worlds. It's a sign up to, abandonment funnel or habit funnel. And the biggest difference here is that one, it's a very linear funnel that's cohorted by week. Two, it ends at the second time the user does the behavior.

[00:16:37] Crystal Widjaja: ‘Cause out of all of the companies that I've advised, it's the second time that's the hardest. First time, sometimes it's a miracle, right? That the user gets to your AHA moment. They paid or they experienced the product. The problem is when they come back, do they come back? Do they remember you? Are they, are you top of mind enough? Did you solve the problem well enough that they come back to you? And so this helps you see one, are people actually getting to that second or third behavior that you really care about, your AHA moment, and two, is that overall improving over time?

[00:17:07] Crystal Widjaja: And that's why this is cohorted. And so when you start to look at the bar chart, it stacks up to 100%, but what it is is kind of a trail of the dead. All of your users who abandoned the funnel at some point between steps one and five, where did they end up? And when you start micromanaging your product experience and you work on step three in the flow, add to cart or insert payment or add a friend, how many people are abandoning at that step? And if you tweak it a bit, if you fix it, did they end up making it to the AHA moment or did they end up failing? And sorry, we may have to re-record that part.

[00:17:51] Andrew Michael: No problem.

[00:17:52] Crystal Widjaja: Okay. So the question is, are you’re trail of the dead when you have optimized step three in your funnel? Did they make it all the way to the AHA moment, the second AHA moment? Or did you just pass the problem on to the next step to being able to reach the step right before the AHA moment? And that's the challenge is when we tweak the product and we don't see any movement at the end goal, we need to put these little pivot points in the middle in between so we can assess, did it work at this step, but now I have this new problem that's become the new kind of a blocker to engagement and retention.

[00:18:30] Andrew Michael: And maybe if we could talk to, like a very specific example, then like we could just take a random app and talk to like what some of those steps might look like. And I think to the point as well of like the repeats and side of things as well as that, I think that is critical because often most of the times while during the onboarding, like we get our users to take the activities that we think are going to be there, but it's really do they come back and actually do those activities again. So.

[00:18:52] Crystal Widjaja: Exactly. You might be force feeding them into doing some behavior, but the user, just because you changed the product to automatically perform that behavior for them, doesn't mean that they're going to suddenly experience success. So the example that I have here is one of an e-commerce startup where we have a very simple homepage landing page. You have a view, SKU, what Google Analytics and all e-commerce people would think of as, product listing page. You have an add to cart event. That's step three.

[00:19:21] Crystal Widjaja: Step four is, you confirmed your order. Step five is you have a second order confirmed. So you can time box this on a cohort basis. So every week that a user has their first home visit, you cohort them. And in this case, if you roll out the cohorts, over time, you might see that we changed the product listing page so that we immediately land you on view an SKU right when you reach the homepage. So 100% of people make it to view SKU because they already have SKUs on the homepage.

[00:19:53] Crystal Widjaja: And you'll see that giant lift in users abandoning, not at home, but now on view SKU, which initially sounds great. But if they never actually add to cart, you haven't really solved your problem. Like the goal of this product is to get people to confirm their orders. If you just hack it so that every user views an SKU and suddenly everyone is abandoning at that second step instead of the first step, like you haven't really made a dent in your product. And that's why the visualization here helps illustrate what percent of the users that I get at the top of the funnel are making it all the way to that last step, to the dark blue.

[00:20:29] Andrew Michael: And yeah, so to that point as well, you can easily sort of like, manufacture engagement as well, but this sort of highlights, okay, like actually the end goal is really the second order confirmed or the second state. And just like tweaking these steps in between if you're trying to manufacture things isn't really going to impact the final outputs. It's about striking a good balance between, like how do you educate, but also how do you get them to take action at the same time as well?

[00:20:56] Crystal Widjaja: Exactly. So the color mapping keeps us honest, right? It helps, it forces us to understand whether or not the right colored bar, the AHA moment bars are growing bigger as the last performed action. Like, did we basically get everyone over the finish line? But it also, if you're a great product manager with a really ambitious marketing team, this can also be a proof point that, Hey, I may have a lot of people dropping off at View SKU or Add to Cart, but if you look at the chart number three, which is how many users reach that end goal, and on an absolute numbers basis, it may be bigger.

[00:21:32] Crystal Widjaja: And so I'm getting better at pushing people to the end of the finish line, but maybe marketing is sending me users who aren't suited for my product, how do we connect and engage on that? Because I've been making adjustments to the product, I'm increasing the number of absolute users who reach an order confirmation page. How do we work together? So it helps create more of a conversation. And I find that visuals help a lot more with data teams, product teams, marketing teams, like we're all looking at the same thing now, like how do we commute, how do we align and cooperate on reaching a better end product state?

[00:22:08] Andrew Michael: Can you talk through that a little bit more in detail? Because I think this is an important point to understand when it comes to, especially around activation and the influence of marketing and how much marketing can really impact those metrics coming through. And also then, so the idea of maybe things may go down in activation, but if it's a net positive overall in terms of the growth rates, how do you then think about the decisions you make as a point of that?

[00:22:33] Crystal Widjaja: Exactly. So this is where event property instrumentation and segmentation is important. So you may even have this chart segmented by user persona. So if I'm a B2B SaaS product like Notion or Superhuman, like your previous guest, I could have this segmented by user persona. So we care a lot about working professionals. We don't care as much about people using this for our family email updates. So if we were to segment this out and we've added this friction or a better onboarding experience for working professionals and teams, we might see that they have a much higher activation rate.

[00:23:14] Crystal Widjaja: And because we have focused on them and because we've made their onboarding so great, they tend to retain and reach that second AHA moment, whatever that might be for Superhuman. It's probably, you know, second month retention, continuing to be active and use the platform or paying. And we'll see that they are staying on our platform. They're continuing to cascade in our overall growth numbers, even though we may be losing 80% of users who wanted to use Superhuman for family email updates. Cause that just wasn't our target market. So being able to segment this, you can see what sacrifices you are trading.

[00:23:46] Andrew Michael: Exactly. And also then on the, even on the campaign level as well, if like you have that as from attribution, the signup source and campaign, you might even see that like there's maybe specific campaigns that are driving a type of persona that are like exponentially exploding, like sign ups or visits to the site, but then ultimately like impacting metrics further down the funnel and really just not good quality traffic that you want to be driving from marketing perspective.

[00:24:11] Crystal Widjaja: I mean, everyone wants quality traffic. What is quality? It's a mix of breadth and depth. So you have a big enough target market segment who will use the product enough and there are enough of them to provide your company value for the amount of work that you spend. There's also that maybe very perfect ICP, ideal consumer profile that, very deep, they're very sticky, they never leave the platform. There may not be as many of them in terms of the breadth of the target market. But once you can identify them and you provide an onboarding experience that's tailored to them, they get it, they get the product, they stick.

[00:24:45] Crystal Widjaja: And one of my favorite ways to segment for this, I actually learned this from Cedric at Commoncog, he was sending emails where he would say, if you are a data practitioner, click here. If you are a finance professional, click here. If you're a VC investor, click here. And so that creates opt-in segments for you when users click on those emails or they click on an onboarding experience to just give you their role. If you haven't built that into the product, it's a cool hacky way to just tag users via email UTM codes.

[00:25:19] Andrew Michael: I like that as well. And I think like the onboarding experience is what we often think of it is just like those first three or four steps that happen in signup. But there's a lot of different moments along the user journey where you can actually be collecting this data a little bit more organically in context where it adds value from that.

[00:25:35] Crystal Widjaja: Exactly.

[00:25:36] Andrew Michael: On the quality front as well on the ICP, I think, like the area and thing I like as a measure to take a look at this sort of thing is looking at the MQL ratio to total number of leads. So if you have a definition for what your ideal customer profile is, and you're able to score the leads coming through the door, like being able to see what ratio, we'll have to cut that please. So if you're able to see the ratio of what MQL is to overall total lead volume, then you can see, okay, have things significantly shift in terms of the quality of leads that we're driving down the funnel? And then do we see an adverse impact as well to the activation metric?

[00:26:13] Andrew Michael: And oftentimes you'll see a direct correlation to those two things when it comes to it. But like you say, it's different stages and understanding data and maturity of the organization and what you're actually ready for at certain points in time as well is important.

[00:26:28] Crystal Widjaja: Yeah, but any company that's getting at least 30 sign ups a day and can use Google Sheets well, you can run a linear regression, you can run a correlation analysis. I use these on a bunch of my templates. I'm probably going to drop another one in the next day or two. So keep an eye out for that. And you can statistically validate metrics against themselves. So MQL ratio to lead ratio versus stickiness. That those are two metrics that can be quantified and you can compare against other metrics.

[00:27:02] Crystal Widjaja: So what I often find troubling is that you will define a metric and then never compare it to something else. Right. What about SQLs to leads? What about MQLs to leads? Are they the same in terms of quantifiable correlation? In terms of direction and strength to your overall retention metric? Then great. Like you don't have to pick something based off of a random threshold. You can pick something that's based on a relative, it's the best thing that we have.

[00:27:35] Andrew Michael: And ideally as well, like the MQL should be, the analysis should be done to understand that it's a direct relation to, like I think for something like that, I like to lean into the LTV and looking at, is it a good predictor of the lifetime value of a customer? And there's definitely some easy analysis you can do with just taking, customer base that are spending more than the average, longer with you in 12 months and then breaking down filmographic and demographic properties and seeing where those properties align and putting together the scoring model. It's often quite an easy and quick way to get to a lead scoring MVP, if you want to call it that.

[00:28:09] Andrew Michael: And then generally in that sense that you want to have it as a predictor of future success and LTV. We've also just for the lessons, what we've been discussing today is a template that was put together by Crystal for Reforge. So definitely like we'll leave a link in the show notes, you can check it out there.

[00:28:29] Crystal Widjaja: Or giving you a Google sheet link that you can access freely.

[00:28:33] Andrew Michael: Nice. Okay. So we'll have a Google link that you can access freely as well, just so you can follow along if you're interested to see what these cohorts look like and the analysis itself. Very nice. You get to work with a lot of teams now as well, like from a startup perspective, I think looking at LinkedIn, you're working with at any point in time around five different companies.

[00:28:53] Crystal Widjaja:  Yep.

[00:28:53] Andrew Michael: What would you say is like a common data challenge that these companies face going in, because they're also different sizes and scales as well, looking at them and is there anything, like, common that you see across the board?

[00:29:04] Crystal Widjaja: There is a tendency to want to build dashboards. So I think one of the, my biggest pet peeves and ways that I know a company's performing data theater is if the first response to, like an experiment or a question around a metric for our users is, oh, do we have a dashboard for that? And typically, if you're a really early stage startup, all you really need is a Google Sheet. Like I'll have companies that I advise from scratch, they maybe have Segment, maybe they have Amplitude, but they haven't really set anything up yet.

[00:29:37] Crystal Widjaja: And I'll say, great, you probably don't need that much. Most of this work is going to be ad hoc analyses, because you have no idea what the physics of your business are. Most of the analysis is going to be looking at very specific user cohorts, very specific user segments, and digging very deep into their behaviors, such that a dashboard will just not cut it. Because you need to look at, what did this one account do? Look at all these tiny, small data points across all of these tables, across all of these different platforms.

[00:30:06] Crystal Widjaja: The email, CRM tools that you use, your products production data, any payment data that you're using, payment providers. And so the common mistake is trying to create dashboards before you know what data is valuable, because dashboards tend to be blinders for us, right? You use a dashboard when you know exactly what KPIs to look at and you know what normal looks like and that this has been a checked off, well-defined metric. So it could be things like how many signups converted to our setup moment and AHA moment on a daily basis.

[00:30:45] Crystal Widjaja: That is something that I could use as a dashboard, but I'm not going to act on that dashboard because I have no idea why a number went down or went up. I'd need to connect that with a bunch of other things. So that's probably the biggest mistake I see. Biggest common mistake.

[00:31:01] Andrew Michael: Absolutely. I like it resonates so much as well. Cause I think there's, you can go very far in a Google Sheets. Like I don't think people know how far you can actually go into Google Sheets with a type of analysis. And to your points as well, like I think I have a pet peeve with dashboards because I think they do exactly what you say, they end up giving more questions than answers. And if it's just like company level metrics that you want to track and those are like health metrics that you want to be working with, but still at the end of the day, they don't provide you any answers.

[00:31:29] Andrew Michael: Like you said, like when something dips, like everybody scratches their head and said, like, why is this like, we need to go and figure this out now. And they're not particularly useful in the context of analysis and giving you answers to the questions you may have of how to improve the performance of the business.

[00:31:45] Crystal Widjaja: Dashboards are a form of data theater, which is basically, I feel like I'm being data driven. I'm being data savvy. I'm like checking the box of, of being, you know, a data driven PM. But it's just a dashboard and I'm not doing any real analysis. It's just there in the best, healthiest organizations, is there for you to get a sanity check on what normal is and to find out when something is no longer normal.

[00:32:09] Crystal Widjaja: If I know I ran a campaign yesterday or I changed my marketing strategy recently, and I need to be able to see if that made a visually perceptible change, a variation in my metrics, and then I like to run that through an XMR chart or risk ratio analysis of before and after. They are signals to do something, not to not use data.

[00:32:32] Andrew Michael: Absolutely. And I like as well, this, the sheet that you put together here for the retention analysis, I think like having the cohort views, having it visual here in like a table form, like you just get so much more context and then just like a big number at the end of it or like a bar chart. It really allows you to sort of spot patterns and see, okay, like, wait a second, like it was actually only 20 people on that day versus 120 every other day. Like maybe that's why the percentage is way higher on the other end. And it's not worth us looking into that spike or like, you get so much more context from views like this than you do from standard dashboards.

[00:33:09] Crystal Widjaja: My best practices are always one, share the absolute number with relative percentages to use a heat map. Like there's no reason not to use a heat map. Everyone can do conditional formatting. And three, make it visual, like make it a bar chart, make it at worst a pie chart, if you have less than three categories, three categories or less, but make it visual so that the numbers come to life and you can see visually because there are a lot of people who, you know, they look at numbers and it's not their first language. And so you need to provide it in a format where people can digest it.

[00:33:43] Andrew Michael: Exactly. Yeah. I think, like a lot of the work of data analysis is really like, how do you communicate this in the most effective form that it's easy to digest and people can grasp the point in a minute. And sometimes it's like maybe even aggregating things and putting things together to make a bigger statement or a bigger point. And I think there's like a little bit of an art in how you present data.

[00:34:04] Crystal Widjaja: That's exactly right. Sunburst charts are one of my favorite charts to visualize. It's hard to do. There are some, like free online tools. If you Google like sunburst chart creator, free, but the ability to kind of take a full segment. Like 100% of users signed up yesterday. Here is, you know, 30% said that they came from Facebook. 20% said they came from TikTok. 10% came from Google Ads.

[00:34:27] Crystal Widjaja: Of the ones who came from Facebook, like this layered onion that you can peel and peel and peel until you get more refined segments and you start to look at of the, you know, percentage of people who came in from Facebook, 40% made it to the setup moment versus the users who came, the 10% that came from Google Ads. 90% made it to the AHA moment. To be able to compare that in relative terms, because the sunburst is relative in sizing to that 100%, you get to see very intuitively what to work on and where your opportunities are.

[00:35:01] Andrew Michael: Opportunities. Yeah. Because yeah, I think to the earlier point as well, in these cases, you may even see, like a reduction in the conversion rates partly through the funnel, but like the net totals increasing at the end of it. And like, that also is one of those things like where do you then start to focus? Like is, are we?

[00:35:20] Crystal Widjaja: Exactly.

[00:35:20] Andrew Michael: And ultimately you're trying to increase the total number of people getting to that end state. So if it's a net positive, but maybe the activation rate slows down at the top of the funnel, that's not necessarily a bad problem if you've increased it so much at the very top. So.

[00:35:34] Crystal Widjaja: Yeah. And if you find that it was Google Ads, it's driving your best conversion from signup to AHA moment, why not double down on that best performing channel? And start doing paid ads on Google and double down on spend and see whether or not that holds. Or does the variation, does that metric change when you pump more money into it in a net, when it results in a net negative ROI, then you can make a different decision.

[00:36:03] Andrew Michael: Different decision, yeah. But having like, these views really helps, like, give you that idea and sense and understanding. So thanks so much for sharing that. We'll definitely add it to the notes and you should check it out if you're listening. So I want to save time and see, we have now as well. A couple of questions I ask every guest that joins the show. First question is, what's one thing that you know today about churn and retention that you wish you knew when you got started with your career?

[00:36:27] Crystal Widjaja: I wish I knew more about merging acquisition loops with engagement loops. So for every new user that is acquired, we all know that, you know, virality activation loop being for every new user I acquire, how does that one user bring in a new, new user? But one thing I didn't do enough of when I was at Gojek was thinking about for every new or re-engaging user, how do I get them to bring back other existing users back to the platform? And you see this really well with products like LinkedIn.

[00:37:00] Crystal Widjaja: Every time you get a message about someone having viewed your profile, someone posting an update, every user that comes back and re-engages with LinkedIn, re-engages other existing users really well. And this drives the engagement and retention factor twofold. So for every engaged customer I have, how do I design experiences or create excuses for them to re-engage other users on my platform?

[00:37:25] Crystal Widjaja: This may be hard when you're not a marketplace, but I like to design these experiences into even D to C products. When I advised AB InBev, one of our products was a boutique craft beer product, and we allowed users, eventually to post reviews and to share their reviews with their friends about these specific beers that they were drinking. And that helps drive more re-engagement. It gives people at least an excuse to come back to the platform and pushes and makes use of your user who is engaged to make them a value generator and value distributor of your own.

[00:38:05] Andrew Michael: And I think that's similar, like if you think in the B2B context, some just as you're talking about some companies that came to mind as always, like something like Loom, where you may not be active in Loom, but you're getting email updates to say like this week, this is what's happening in your organ. Those sort of emails are like results of other people's activity that's bringing you back into the app to check it out. So Loom does something similar and a few others. And I think there's a lot of powerful hooks that you can pull to bring people back in and reactivate them through the use of others.

[00:38:33] Crystal Widjaja: How do you make things, you know, a two for one?

[00:38:36] Andrew Michael: Two for one. Right. The last question then I have for you is obviously you advise a lot of companies now on growth and on data. What's one thing you wish more companies would ask you but they don't?

[00:38:48] Crystal Widjaja: What am I missing? Right? Like, what is the segment that I'm not targeting or that I forgot about? And I find this in every company that does B2B SaaS or subscription models. No one ever retargets the user who canceled their subscription on day zero within the first hour. It's almost always 10 to 20% of your users who sign up to a subscription. No one ever retargets them after the user has experienced some value proposition.

[00:39:16] Crystal Widjaja: So these are users that I call, you know, they are, they're fickle. They've subscribed to your product. They decided to pay for it. So there's some hope there, but they canceled right away because they don't want to worry about forgetting to cancel the subscription. What if it's not good enough? They're anxious and nervous and they're fickle. And they may continue to use the product for the month that they have subscribed, but no one ever retargets them and says, hey, you're using the product, you seem to enjoy it. Would you consider resubscribing before your period ends?

[00:39:49] Crystal Widjaja: And this has consistently gotten me at least two to 5% re-engagement or unturned users to come back and say like, Hey, actually, yeah, I have found value in this and I regret unsubscribing prematurely. My subject line for this has always been, did we get it wrong? And that has like 70% open rates.

[00:40:09] Andrew Michael: Yeah. I think that for these sorts of scenarios, it's what in criminal form, like the type of products, come to mind or like a Canva or a, like something where you have maybe a specific use case, you come in to get that done, but then you realize, Oh, wait a second. Like, I'm actually–

[00:40:22] Crystal Widjaja: I thought some other use case.

[00:40:23] Andrew Michael: I found some other use cases and maybe I'd need this more than just the initial image that I needed for the banner. Yeah.

[00:40:30] Crystal Widjaja: A month is a long time. So I like to keep time-based, triggered, automatic re-engagement campaigns going. So what do we... You have to put yourself in the shoes of these users. I canceled my subscription in the first hour. I did the thing that I needed to do in the first day. I came back three weeks later, before my subscription has expired. What should I tell this user to ask them to resubscribe? What can I offer them? How can I empathize with them?

[00:41:00] Crystal Widjaja: And so being the type of company that says like, did we get it wrong? Like, is it our bad? Like we get it, you cancel, that's okay. We're giving you a chance to resubscribe because we see that you're using the product. And this might actually be, this is a hugely beneficial thing to do for users because they may have forgotten that they canceled. This may just be part of their habit and they're not even thinking about it.

[00:41:22] Andrew Michael: Very nice, Crystal. It's been an absolute pleasure chatting to you today. We learned a ton. Is there any final thoughts you want to leave the listeners with? Anything they should be aware of to keep up to speed with your work?

[00:41:32] Crystal Widjaja: Follow my Substack. I'll be, and follow me on LinkedIn. I post free templates there. I'm going to be posting more Google Sheet templates because I think a lot of us just don't know, like what can we do with our data? I just need to see it in the format where I can just copy paste what I've got and give me some great visual to come out of it. So my hobby is creating new interesting designs in Figma and Google Sheets. Follow me on LinkedIn for more of those. It was great chatting with you today.

[00:41:59] Andrew Michael: Awesome. Yeah. Thanks. And we'll make sure to leave everything we discussed today in the show notes so you can check it out there. And yeah, wish you the best of luck now going forward. Thanks for joining.

[00:42:07] Crystal Widjaja: Awesome. Thanks so much.

[00:42:09] Andrew Michael: Cheers.

[00:42:16] Andrew Michael: And that's a wrap for the show today with me, Andrew Michael. I really hope you enjoyed it and you were able to pull out something valuable for your business. To keep up to date with Churn.FM and be notified about new episodes, blog posts and more, subscribe to our mailing list by visiting Churn.FM. Also don't forget to subscribe to our show on iTunes, Google Play or wherever you listen to your podcasts.

[00:42:43] Andrew Michael: If you have any feedback, good or bad, I would love to hear from you. And you can provide your blunt, direct feedback by sending it to Andrew@Churn.FM. Lastly, but most importantly, if you enjoyed this episode, please share it and leave a review as it really helps get the word out and grow the community. Thanks again for listening. See you again next week.

‍