Hannah
Oh, okay. Yeah. Awesome. All right. Hi everyone. Today we have three amazing leaders in the UX industry to talk about building analytical muscle memory with a UX team that's largely on relied on qualitative insights. So joining us, we have Andrew Branch, the director of design@rev.com where he's led the UX team for over six years. And he's also the co-founder of front Utah design conference. He's held various UX roles for over 18 years.
We also have with us today, Josh Seiden who many will recognize Josh as the co-author of lean UX, sense and respond, and the sole author of his recent book outcomes over outputs. He teaches product and UX teams to build outcome focused cultures.
Finally joining us in the conversation, Steven Cohn. Steven is a serial entrepreneur whose last company Validately the qualitative research and UX testing platform sold to UserZoom in 2019, Steven is starting his fourth company impact product.com, which is an analytics tool designed for the persona of the UX designer. And the core use case is to measure UX design outcomes and impact.
All right, everyone. Thanks so much for being here. So to kick off the conversation around embracing an outcome focus mindset, I'd love to dive into and state goals for becoming an outcome over output driven organization. So Josh, I'd love for you to kind of start the conversation.
So could you elaborate on what outcome over output means for culture at an organization? And what's the ideal state for an outcome focused UX team.
Josh
Yeah. Yeah. It's great to be here. Thank you for the question. You know, there are a lot of ways to, to think about this for me. You know, the question that I'm always asking is how do I make, how do I put great things in the world?
Right. I'm a designer first and I'm really interested in, in not just getting stuff into the world, but getting great stuff into the world. And so, so, so what I'm looking at is trying to understand, first of all, doing the best possible job of design is not enough. It's just not enough. You have to do a great job of collaborating with your cross-functional team, right?
Because designers alone can't make this happen. And, and then you, so you're working with product people and, and engineers and content, people, and stakeholders, and all of that. You have to figure out ways to collaborate and you need a common language. For setting your success targets. Right? And so for me , the target language over the years has moved from we're going to design and build this great thing to, we're going to design, build design and build this great thing, put it in the world. And it's going to have the results that we want. It's going to make the business more successful. It's going to make our customers more successful. It's going to make the users more successful and it's, it's going to make the world a better place. And so. Being able to, to assess whether our work has the result that we're seeking for me, that's the big move. And we moved from outputs, which is making cool stuff to outcomes, which is making cool stuff that makes a result.
Hannah
Yeah, that's really interesting. I was wondering if you could maybe dive in a little bit more, you mentioned this notion of having a common language, which is really key and fostering this cross collaboration with teams, you know, from your experience of working with many organizations, how have you seen teams find that common language?
I imagine it's, it's different for every team, but what are some ways teams can kind of get started to, to finding that. That magical language that speaks to everyone.
Josh
Yeah. Well, so that's, that's what the notion of outcomes that I write about, uh, is sort of the basis for that common language. So I think a lot of times when we talk about outcomes, we use that word as sort of a synonym for results.
Right. We said, we want to be goal oriented or goal focused or be more effective. But for me, an outcome is something that's related to that, but very specific an outcome is a change in behavior, the change in user behavior that creates value. Right. And so if I'm trying to generate an outcome on my team, what I'm trying to do is make it possible for a user to do something new or do something more effectively.
That, that gives that user value. Um, and in, in, in doing that thing creates value for my company, right. So I make it easier for them to accomplish a task and they, they love that so much that they subscribed to my service. And so that, that. Specific definition of an outcome, right? It's, it's a change in human behavior.
That creates value is, is something you can have a conversation about, uh, with your colleagues in other departments. Right. Um, and to say like, look, did we, did we manage to change user behavior here? In a way that that created value for everybody. And so that's the kind of, that's the fundamental unit of this. That's the word in the shared language, you know?
Hannah
Yeah. That makes a lot of sense. Um, Andy kind of joining the questions over to your side now. Um, I was wondering if you could be a little candid and share kind of the current state of where your design team is, you know, can you talk a little bit about how big your team is? How does your team kind of measure outcomes for their designs and how would you describe your current design culture?
Andy
Yeah, happy to, I'm happy to be very candid, uh, and be a little vulnerable with you guys. Um, so our group we're growing fast or rather this is growing quickly in the next, in this month.
We'll we'll have doubled within a year, um, which is pretty crazy growth for us. We're still a relatively, relatively small team that I think will be around 10. Um, which includes, uh, so-called data research as well as product design. Um, and I think if I were to look at what we do well and where gaps are, I think we are very good at speaking with users and doing the qualitative side of research.
Um, it is something we just it's been drilled into our DNA as a culture, you know, a design culture to speak and to, um, Uh, you know, to, to do that, that level of research, the problem is probably, um, I don't think we're outcomes focused. I wouldn't call us an outcomes focused group, unfortunately. And part of the reason is because, uh, it's the, the artifact of the design document is the, is the product. At least that's the way it's seen. Um, and, uh, that's what we test against, right? That's what we talk to our users about and with, um, bettering the whole iterative process. And once that's done often our product managers. Um, it's left to them to, to determine whether or not it was successful at the validated later.
And we do plenty of encouraging, but then we start over with the next problem often, um, you know, QC with engineering may be, you know, some of what we do, but th but certainly feeling accountable and responsible, and even more than that, Uh, just wanting to know how it landed is not inherently part of our DNA.
Um, and it's, it's something I've been realizing later. It's interesting. The timing, because you know, when Steve and I started talking about this as something that was kind of weighing heavily on my mind, um, and I think also the problem is, is that it's not just at the end, it's at the outset because it's the way that.
Okay, ours get defined. It's the way that a product manager brings the problem to a design group and asks, you know, and, and where we get involved. It's already past the point at that point at which we, uh, feel ownership over it to the same degree that they did. So if it gets validated, it's done completely by-product manager and, and all the accountability in art come. And I've seen this a lot of other companies too, the success or failure of a product and the responsibility for it rests almost entirely on the product manager. We get the wash our hands, um, from that problem. The other problem though, and so I guess there's three so far that that relate to this lack of outcomes is, that designers are not very good at looking in mixed panel or in, uh, um, the other tool. I can't even remember the name of it. That's it revealing how bad we are at it. Looker, Looker, um, or look back, I get them mixed up, whatever. One of, one of those tools that, that we don't use that we should.
Steven
So, yeah, that's, that's one of those tools that you don't look at riffing on the look.
Hannah
Yeah, that, that, that makes sense. So it's you recognize that there's a kind of, this, we'll call it a gap, I would say to becoming more outcome focused and you recognize it's important. Um, you know, I'm kind of curious if you could just dig a little bit more, like, why is it so important to you that your team really makes this the shift culturally, or just how they're thinking, um, and bringing some of that ownership between from product on to design as well.
Andy
Yeah. I mean, I think the biggest reason why it's important is because, um, if, if we, if we fail to do that, then, um, uh, design. Is, it becomes the end. Like the interface itself is, is not the jobs to be done. It's not the product. It's not the result. Like you said, Josh, it's the, but because that's the focus, um, we, we miss, we might miss the Mark.
It's very easy to miss the Mark. Um, or, uh, even though we're talking to users all the time, um, The cycle for realizing that we missed the Mark. It takes a long time. If we're only talking to users about what's right in front of us and we're not validating, um, uh, what happened right after it was implemented.
Steven
Right. I would think of it. The analogy of my mind is, and tell me if this resonates with you, imagine like you're the owner of a restaurant and you hire this. Amazing chef. We've got an amazing reputation. And you say to the chef, um, you know, we want you to make this the best menu possible. And obviously food is super, super important for this restaurant success.
If the food's no good, we're not going to be successful. Super important for the food success. We're just not going to tell you for any of the meals that are, that are served like. How frequently they're being ordered, uh, whether they're whether the meals are being finished or sent back. Um, you know, we're, I can tell you, uh, any of the costs of, um, of the meals.
Like if, if any of the entrees are profitable or unprofitable, um, if, if there's a connection between which meals people order and how frequently they come back to our restaurant, I'm going to tell you any of those metrics. Trust us we're we got this. Um, but, uh, but, but really, you know, make sure like the food's good. Okay. Just make sure that, you know, and it's like, uh, those are kind of important things for me to know, like how do you, how do I, if someone's not enjoying like this one meal, Brandy too. Like, it's important to know. I mean, you know, I can go out and talk to the people in the restaurant to, Hey, did you, did you like it? You know? Um, but you know, not always you get the right level of insights there. Um, and so, you know, from my perspective, it's like, it's, you know, it's, um, I think you said it right, Andy, you know, I think designers are. Uh, if you look at designers as all their job is to crank out prototypes that stakeholders, like, I think you're missing the Mark, you know, I mean, it's just, and, and you're under utilizing the value that, that, um, you know, they can bring through, through connecting all of that qualitative user insights, um, and, and the creativity that they naturally have.
Um, but I'll tell you, you know, I've been, I've been building technology for the UX. I'm not a designer myself. But I've been building technology for the UX industry for about 10 years now. Um, and what you described as, I mean, it's the standard.
Josh
Well, yeah. And, and, you know, honestly, this is a new muscle for the design world.
I think, you know, like we, we have a lot of tradition that, um, uh, about, uh, tradition and expertise about how to do research, to get us to the point of building something. Right. And that, that goes way back before software, you know, it's, it's the research we do before we make a physical product or the research that we do, uh, to understand, you know, who we're communicating with before we design something and send it to the printing press.
So we've got all of that tradition in the design world about how we figure out what the requirements are. And we're good at that. Right. Um, but you know, it's only kind of in recent years, as we've been dealing with software that we have this kind of new, uh, these new abilities to measure the results of our work.
In this other way, right. And measure it and measure it at scale, right. We used to be able to put a product in people's hands and observe whether they were successful or collect data, uh, about, you know, the safety of our cars over time. But, but this kind of design, this, this sort of measuring results as a design in the design addition, I just think it's a new muscle and we, we, we don't even think about it when we talk about product discovery.
Right. We, all of our product discovery is about the stuff we do before we ship, not the stuff that we're doing after we ship, you know,
Hannah
Yeah, I was going to say Josh. I was wondering if you could kind of go a little, a little further in talking about, you know, we have this new muscle and muscle memory, we need to learn, um, kind of outside the typical parameters you think about with UX. If you're taking Andy's company as, uh, an example, what are some things that you could recommend to uX leaders, you know, in specific, in this case, or maybe there's some general steps of tactical ways they can start implementing this change and really becoming more outcome focused.
Josh
Yeah. Well, that's, that's the, that's the, that you just served up a big fat softball to a consultant.
Let me, Andy, let me tell you exactly what to do. Um, so like I make that joke because, uh, you know, uh, every, every company is different. Right. But the, you know, there are a lot of obstacles that I, that I heard, uh, Andy talk about, right. One is the, the way the sort of work flows through the organization, right? Like that conversation about the way things are framed from product management to design. Right. Is there an opportunity there for a collaborative conversation around the way we frame that work? And then the other thing, Andy, that you said that kind of, you know, rang a bell for me is that the deliverable, right?
The design artifact is the product. Right. And so that implies to me something about sort of the development cycle, right? One of the, one of the, one of the things that happens when you start working with outcomes is you, you still want to be able to test early and often. So you want to be able to put something small in the market and see if it works.
And that implies a kind of a rapid, uh, cycle of, of delivering small things as opposed to a longer cycle of delivering big things. Right. And so, um, you know, I, I don't know how much of that is true in your context. Uh, Andy, but, but to me, those were two things that I immediately heard.
Andy
Um, yeah. And I should, and I should clarify that we do. Yeah. I mean, it's weekly, we release weekly and it is a lot of tests and filling out, but often that is completely product manager lead where it seems like there's not enough design involvement. And to Steven, I think you, you, you, uh, brought up the reason why that's a detriment. Obviously we're doing something and we're doing some things, right. But if, if there is no design input, then you're exactly right. Steven. Like it's the, it needs to taste good, but we have no data to verify that the, the pleasurable experiences there, th the aesthetic side of, uh, of, uh, I mean, we're very good at metrics, but not so good at the, at the other side.
Steven
Yeah. I mean, you know, w my experience, again, I have a different perspective. I'm not a designer. Um, Uh, or design consultant, um, and, uh, uh, never led a design team or anything like that. My experience is that I've worked with hundreds, thousands of design leaders, both initially Validately, and then, and now with my new company of our product.
And, um, what I find is if we're being candid, um, you know, in a lot of examples, um, the, uh, the design team, um, Is is asked to crank out prototypes to meet stakeholder approval. Um, they're given high level metrics of like overall engagement is up overall time on site is up overall retention rate is up.
Right. Um, but there is very little, if any traceability between. You know, I did user research for two weeks on this. I created this prototype and got stakeholder feedback and iterated on it for, for two more weeks. You know, the team released it, like is that, uh, you know, is, is that. Are people using it, you know, like what precision, if people are using it, the people who use it, are they using it more than once?
Do they like it? You know, um, you know, that type of, kind of round thing, and it goes back to something else I picked on us that you said it goes back to OKR and it goes back to, you know, sometimes you see the OKR is that somebody's nineteens, it's like talk to five users. You know, I mean, you know, that doesn't really drive an outcome, you know, it's like a check I'll do that. Right. But that doesn't how do I know whether this thing that I talk to these users is, is actually using it. Now, some teams do a great job with that and they'll say, you know, get, you know, 10% of our users to use this new feature or increase, uh, you know, increase this metric from 22% to 30 to 35%. Right. And it's specific and it's actionable. Um, but you know, it has to happen upfront too. Um, and I think a lot of design teams don't ask the right question upfront. What does success look like? No, like what, is there a goal here or is like, you know, what, what, how are we measuring, right. Are you measuring this product management team? What's the key metric? And like, how do we know what success looks like? Um, and I think, I think it's hard. I think it's scary if you have the wrong culture and getting a metric is hard because if you have the wrong culture and you don't hit that metric, Then are you punished? Are you fired? Like, you know what I mean?
Like, like, or do we learn from it? Do we have the culture of learning and experimenting and iterating and just, you know, that, and so I think that's a lot what goes into it.
Andy
PM's I think are getting that and doing it pretty well where our design group tend to have OTRs that are focused on ops ish stuff and improving how we, how we deliver, because we're just trusting that the PMs are doing it right. And it's hard for me to know, like, how do we get in there? Like how do we get designers to be part of that, that process?
And I just don't know where to, how to do it, I guess. Um, is that, that's the question I wanted to ask, you know, you, Josh and Steven is work design organizations that do this, right. What do they look like and how do they work with the product management team
Do you have OKR's that are shared with product?Uh, I mean,
Josh
in other words, are your, are your pods, do you have like these like cross-functional product teams and, and do they align around a product related OKR? They kind of departmental functional OKRS?
Yeah.
Andy
W w the part of the problem is that, I mean, yes, to a degree, but our designers are not one-to-one. We don't have any PMs that are one-to-one with designers. So designers have to share, I have to be shared. A lot of that is because a lot of the work that we do is so backend heavy that not everything has a UI layer. And so it's, it doesn't always make sense to have a dedicated, designed to it. So, because of that, we kind of just say, okay, what are your OKR's, and how can I support you? But it's not the sense of ownership. And that might also lead to some of this, you know, handed over the, the design is the, is the product from our perspective as designers.
Josh
Yeah. Yeah, that makes sense. Yeah. I mean, one of the things I think that that's, I've seen happen and, and I think it's kind of an OKR anti-pattern is that, um, that OKR's are really good for creating alignment in an organization. Um, but, but if you sometimes, OKRs get used sort of as departmental measuring tools. And then you're actually using them, not for creating the lineman, but for creating local optimization, you know? And so I think like actually doing okay, ours implies other changes, including kind of realigning, re realign people, uh, to support those OKR , you know? And so, okay. So that, that can lead you astray, you know, I, you know,
Steven
So, let me jump in and be, let me push on that and be super tactical. Okay. Cause like, Andy correct me if I'm wrong, but Andy doesn't have the, probably as a design leader, who's like all their designers hasn't had the power to basically kind of rearranged company-wide OKR is or product management wideOKRs. Right. He has, he has, he has control over the OKR, I would guess. Yeah. Right. Um, so, um, So like, let's get tactical here. Like let's like, what is like, like, I mean, is there a specific things that, you know, he, he said, he's here. He's saying I really want to be outcome my team, be outcome focused. Um, I have control over the OKR, but my OKR is, are not there.
We have a shared design, not a dedicated, which I run into a lot. I hear a lot of that and some that are dedicated pods, but others a lot. That are shared, whether it's because someone's back-end or not, but some of them are very project specific. Are there things that structural things that Andy can put in place, whether it's a set of questions that needs to be answered before his team gets allocated or whatever it is like, are there structural things that they, that Andy can put in place as a design leader, within his span of control that can help, um, move the team more towards an outcome focused initiative mindset.
Josh
So I think kind of stepping away from OKR is, cause I, you know, I don't, I don't know what, I don't know how they're structured in your organization, but I think in terms of things that a design leader can do, right? One of the things that we can do is we can, uh, we can have some influence on the way work is scoped. Right. And on the way work comes into the team on the way we agree on the work that, that, that, uh, that needs to be done. And so I, I think one of the things that you can start doing is, and all of this is about, um, there's sort of the, the science of this or the technology of, of, of what we're going to do.
And then there's the cultural persuasion stuff. So I'm just going to talk about methods like that. The technology of this there's, there's a question you can ask when, when work comes in, when people say, make me a up or help me design a feature, right? That the conversation that is possible to have is when we've built this feature, when we've built this feature successfully, what will people be doing differently?
When we've built this feature, what will people be doing differently? And that really is a question about the outcome, right? Because remember I said, an outcome is a change in behavior. And so then once you've had that conversation with your stakeholder or your product person, you know, well, people will be, you know, um, I don't know, pick something, they'll be, they'll be sending us more.
RTF files instead of dot doc files or whatever, right. Then you can say, well, is this, then you can start having a conversation about that feature. Is that feature the right way to get that outcome? Or do we have the opportunity to explore the solution space? Right. If we all agree that the outcome is what's important, right?
What we want China to do is get people to do a different thing. Now we can have a collaborative conversation where instead of just being sort of like, Oh, we're, we're catching the past that you've thrown instead, we're designing the play together. Does that make sense? Does that feel like it's for you, Steven? Does that scratch the itch in terms of like a kind of thing that a product leader can do? I mean, a design leader can do.
Steven
I mean for me, I think so, but I'm a design leader. So Andy in real, like in real life, like, does this, does this feel like something that you could, you know, you could, you can do or be like, Josh is still not there yet?
Andy
Yeah, no, it absolutely is. It's a, it's something that we can, um, um, you know, we have. We have full control of that pro that little slice of the product cycle. So absolutely. And I could see value there for sure.
Steven
Yeah. So to me, like it's, um, as a founder CEO, Uh, my last company, you know, we set ourselves up as pods, um, and as cross-functional pods and stuff like that. And so to me, you know, my, uh, I always wanted, um, an outcome focus, um, pod, you know, and everyone kind of having the same, the same metrics, uh, to focus on that said, you know, Eddie, if, if, if you're saying like, you know, product manager kind of has comes to the table with their own set of metrics, maybe their KPIs, maybe they're higher levels than the individual things. They, 50% of the stuff they don't even need design. And 50% they do need is not, um, you know, it feels to me like, you know, one, maybe an OKR could be make you make sure you have asked this before for every single design initiative. Right. Um, and then. You know, make sure you, and you fill it out. Like maybe it's literally just, you know, it's just, you have to, to make this, to meet your OKR.
You have to have shown this metric. Like this question answered Josh, the question to answer, um, for every single design initiative. And if you're having trouble getting an answer from product, you know that that's when you maybe use me as a, as your boss, right. And I can say. Hey, product manager. Why won't you give me, you know, and I can answer that question.
What does success look like? That seems like a fair question. Um, you know, but then I think it's also, you know, I think there's gotta be Josh tell me, but there's gotta be some sort of process or muscle memory around coming back to it later on. So yeah, we say, okay, success looks like 10% of our users clicking this button or completing this, this flow. Right. Um, I could find it. Okay, great. Am I done, you know, or is there, is there something in that muscle memory process afterwards to celebrate the success? Talk about the things that didn't work, you know, what's, what's that part of the process, right?
Josh
I mean, I think that's, and that goes into the, that sort of, that, that, that practice of, um, sort of what, what I think of is continuous, uh, continuous research and continuous discovery, right? That, that what, we're, what we're doing in, I mean, Andy talked about weekly releases, right? So this is, this is clearly a culture where you are working in a kind of a continuous way, right? You're designing and shipping and designing and shipping. And the question is, are you designing, shipping and validating, right?
Are you designing and shipping and then observing the results? Right? One of the things that I've done with startup teams is, is to, to turn the, um, uh, turn that sprint review. Meeting right. If, and I don't know, kind of, if you're doing ad Joel or what flavor of agile, but most, most versions of agile have a kind of, uh, uh, uh, um, a meeting at the end of every cycle where in scrum, it's called the sprint review.
And you look at everything that's happened in the last sprint, right? Typically you're looking at, uh, everything you've designed and built. But really what you should be looking at is all the things we've designed, built and learned during the last cycle. And one of the things that you're learning is okay, I'm learning from that upfront qualitative research, but I'm also learning about the stuff we just put in the market.
How is this new page performing? How is this new feature performing? Is it hitting the numbers that we agreed we were going to try to hit. Right. And that means things like, Oh, by the way, in the definition of done right at, in, in the acceptance criteria for every feature, we have to be able to measure it.
Right. And so we have to be able to collect that data and it starts to force, um, changes through the process, right? If you say like every week or every two weeks or every three weeks, whatever your cycle is, we're going to review. What we've designed, what we've built and what we've learned, and that learning is includes the performance of this stuff that we're making.
And to me, that's the forcing function. Right. And that, um, and, and, and that, hopefully that's a cross-functional conversation, right? That's your product team. That's your, that's your product, people, your designers, your engineers, and your interested stakeholders, uh, sitting down and looking at that at a, at a pretty detailed level, uh, you know, How did our red button work?
Steven
Can that be an OKR too? Meaning, uh, meaning an OKR is at every, um, sprint review or whatever it is, um, review your charts of outcomes for the specific metric. Right. So, okay. R one is. Ask this question. When you get a new project to find a metric, what does success look like? Write it down, you know, you know, make sure we can, we can measure it what success looks like. And then, okay. Or two is, um, make sure at each sprint review or every two sprint reviews or every four, like every four, maybe thought every one, I don't know, carve out 10 minutes to. Talk about, you know, your, your charts and review it with the team.
Josh
Yeah. I mean, yeah, absolutely. Right. I mean, what we're talking about is, is using OKR is for, for process transformation or for organizational transformation. And, um, that's a, that's a really, really good use of them. Uh, the one thing is that, you know, I, I would say that this, you don't want to, for me, individual OKR is, are sort of an anti-pattern.
And so this is something that you want your product development group to own. Right. You don't want it to be like, Oh, there's one designer whose job it is to bring this stuff up all the time. God, Andy's talking about that stuff again. You know, it needs to be like, OKRs to me are really useful tool for again, creating alignment. So this is something that we're doing with product and design and engineering. Right. And we all own this OKR. Right. And we're all responsible for it. Right.
Steven
What do you think about OKR that are extremely specific, uh, move this conversion rate from 22% to 35%? Is that an OKR that's reasonable when you're like super advanced? So my, my thought is. Like that level of an OKR specificity, especially being that specific for Andy's team right now it's an early transformation might be overkill and scare people, you know, it's like, wow, what if I get it wrong? What's going to go on there.
Josh
Right. Call that a KR, not an OKR. Right? Like what's the, what's the why there. Right. What's the why what's the, what's the inspirational part of that? What's the, what's the, what's the thing to get us rallied behind it. That's how we know we're succeeding at something, but what are we succeeding at some? I mean, you could argue this all day. We're succeeding at moving the numbers, but like what's the, what's the why where we're getting more where we're spending our time more effectively. We're only working on features that matter, right? We're not wasting time on features that don't work. Right? Like, to me, those are objectives that I could get excited about.
Steven
Yeah. And Josh, Andy, Andy was the reaction to that. Do you agree with all that or, or not, or candidly you're still okay.
Andy
Yeah. No, that does resonate. Um, for sure. Um, the, the, if I were to take away some action items or potential ideas that we can implement really immediately, it's that. Hey designers can, um, we can take accountability for like, we don't have to. Right now we're completely just trusting that the PM's that what they said they would do is happening. We're just, and we have no idea for a lot of times, if it is or not, we just, hopefully it is, but we can, we can find that out. We don't have to be completely beholden on a PM. We have the data. And there are other ways, Steven, your tool provides a way we can find that out for most things and we should be. And, um, regardless if there's a process for, I don't know that we have a very good, you know, kind of, um, post evaluation. Other than that, that I know that the PM's are, are very much held accountable for their own, uh, metrics that they're and, and goals that they're setting. Um, Th that's the kind of stakeholder level level, which we're not necessarily a part of that, so we can build in processes there, but even if we didn't, we can, we can find out and we should be.
Um, and then, yeah, I love that idea of, of, of, uh, implementing those sorts of OKR stats. The question upfront to measure at the back, even just doing those two things, I think can make it, make it, make a difference to our positive, uh, move in a positive direction.
Hannah
Awesome. I feel like, uh, Andy, you kind of give a good segway from that. Uh, talking about just insights into products and prominent analytics to talk about product analytic tools. Um, so I, I believe you mentioned before your company uses mixed panel, um, would love to hear just from, from your side, like how often your team is using it to measure outcomes and you know, what are some of the challenges your team has with the product analytics tools?
Andy
Yeah. I mean, I'll be kind of embarrassed that some of my team will ask this transcript for me to admit that I don't, we don't really use it. Use those, those tools much at all. We, we, we asked the PM's when w when we're curious and let them figure it out because it's intimidating and it's good. Uh, it's, it's a lazy answer. Unfortunately, that's the reality. And it's, I mean, I'll take responsibility because, you know, I was the first and only designer for three years, so I set the tone and the stage, and it's, it's my fault that we're, we are where we are, but we, we needed to do better.
Hannah
I don't think there's a lot to be embarrassed about. I think, um, Josh can probably say he, he might see a lot of teams in similar, uh, scenarios. Um, Josh, I would love, if you could also kind of expand on from your experiences, hearing the sentiment from Andy, you know, what are some of the challenges or roadblocks that you've observed with teams that you've worked with in regards to UX, trying to leverage analytical tools. Like they, they want the insights, they kind of just. Have a hard time getting them and going to the next step and really becoming outcome driven.
Josh
I, I think a lot of times the sort of first problem is that they don't control their own destiny, you know? And so you, you rely on someone else to implement a way for you to get the numbers you want. Right. And so, you know, the, the thing that we, we know how to do as designers and where we control our own destiny is we can do, uh, qualitative research upfront. Right. W we know how to do that. Nobody's in fact, we're the only ones in the organization who knows how to do that. Right. Um, if you include researchers in that, we, um, but we don't control our destiny in terms of getting the numbers for what people are doing in the system.
And typically the kinds of numbers that we want to get are things that other people in the organization are not super interested in traditionally. Right. So they'll, they'll be able to tell us either the easy things like, you know, that come out of other tools, like session data, or time on site or things like that, or we'll be able to tell, um, I used to work on wall street, uh, and I worked on trading systems and the transactional financial data.
What symbols were being traded at what prices. The whole system is built around that we needed to know that we could get that all day long. We could get trading volume and all of that stuff, which buttons, people press, who cares about that. Right. And so we needed, we needed a campaign of persuasion to get those kinds of things built into the system.
And, um, For me, that's the biggest consistently the biggest roadblock that design teams face, uh, they're onboard. They want to do this, they're ready to do it. And, um, you know, there are obstacles.
Hannah
Yes, Steven would love to hear your perspective. So as a, as a founder and CEO built four different companies conference from small startup and scale, you know, from your experience like what have you seen as challenges from the UX perspective and using these analytical tools, but also kind of just, uh, honing in on what Josh mentioned of. Kind of there's two different data sets. People are looking for. There's a qualitative and quantitative. How are you as a founder kind of balancing those two to make more data-driven decisions?
Steven
Yeah. I mean, one of the things that I've observed in the industry, I think it's really interesting. If you look at the portfolio websites of any designer, They go in great detail to show you wonderful, beautiful pictures, uh, with sticky notes, all over the place for their user interviews, um, and their collaboration sessions.
And then they show you their prototypes and it really they're beautiful and all this stuff. Um, and, uh, you know, they talk to you about their process. They talk to you, all these things. And at the end of, at the nowhere on that portfolio website is any type of outcome. No. I mean, I, I mean, I've looked at literally tens of thousands.
Andy, keep me honest. I mean, how many portfolio websites have you looked at? How many, how many outcomes now? If you look it up. Yeah, very few. There's a couple of you look at a marketing person's resume. It's every single measure. Every single line has some sort of metrics like, Hey, you know, campaign, this campaign has a 22% ROI everything's metric, right?
Salesperson everything's metric. Product managers resume. Everything is a KPI. Like we have these KPIs and this type of thing and stuff like that. Designers is like, Hey, I, you know, here's the output, right? Here's the process I did. And here's the output. And, um, you know, I always used to say in my interview, that was like, that looks good to me, but like, I'm not your user so like, does your user, like it. And, um, what I sound so consistently that I got blank stares. And so this is why, what I've been doing research on for the past year, I've interviewed over 300 companies and I can tell you, Andy, there's nothing that he saved up because no, like almost no teams, uh, in the 300 companies that I've interviewed big, small startup mid size growth, large enterprise FinTech, financial service, you name it across the board technology companies.
Um, Uh, there's basically two models. There's half the teams are, have no idea they're in the dark. Um, anything besides exactly what Josh said time on site sessions. It's just like, what does that do for me? Um, the other half the team had this hub and spoke model, so they'll have, they'll have a one person who's a data scientist who will log into the tool.
You can ask for information. Uh, that takes time to get that information back. It's almost never back correctly, the right, the first time you have to go back and forth multiple times, by that point, you're already onto a whole nother design. It sits in your inbox and, you know, whatever. And, um, there's three reasons that I've heard over and over and over again.
Uh, as Josh said, the instrumentation. Requirement is massive. It's a headache. And if we're being candid, design teams don't have access to development teams to schedule what they, what they can instrument. So the design teams have to go through product or data science to then get the instrumentation scheduled and organized, and I've heard over and over again.
That's the blocker that takes so much time. And when it gets done, it's not always instrumented properly. Et cetera. The second thing is, and Andy talked on this is that these analytical tools, very candidly are designed for, um, analytical, quantitative. You know, analysts, people who really want to get into the weeds, uh, they require a heavy amount of kind of training and, and, uh, there's a lot of jargon.
You got to know what these metrics mean. Um, and then a lot of cases, designers were never trained on this stuff. They were never trained on the, the numbers, the metrics and, and the, and the jargon. So when they're, when they're shown the UI of these dashboards or these analytical tools, there's a whole lot of stuff that, that.
Even if they know which button to click on, they're not always sure what that, what they're looking at, how to interpret the metrics and the charges, because that's not their training, that's not their background. Um, and so that's created a lot of friction. Uh, and then the third thing is, is, um, uh, just connecting outputs, the outcomes, um, you know, it's, it's, these tools are not designed for that.
They're not designed. Um, they're basically spreadsheets these tools. Max panel. It's like we have all this data, here's a little UI on top of this big structure database. Tell us what you want. Right. Tell us exactly what metric you want, which basis, you know, your assumption is that, you know what your key four, um, and, um, you know, uh, so to connect, you know, this work stream that I did to this outcome can be done.
Usually it's some sort of like dashboard that someone has to put together for you. Um, by the time it gets put together scheduled and put together and gets sent over, you're already onto something else. Um, and if you don't have the, the muscle memory, uh, like we're talking about earlier, uh, culturally around this, uh, you don't look at it, you don't do it.
And so these tools are just simply not designed for the designer. And if you go to their wet, don't take my word for it. Go to their website, go to Mixpanel's website and find where it says, use case designer. It doesn't, it says product manager. It says data scientist. So, you know, business owner. Right. But it doesn't say designer and it's because it's not the same.
It's not designed for that designer. So it's a classic example of if you design for a specific persona, this other persona, Doesn't understand what they're doing is not, it's not, it doesn't mesh with what they need. It doesn't, it doesn't align the jargon. Doesn't align with what they're used to the use case that w the flow, all that stuff.
Hannah
Yeah, I think, um, it's really interesting. We, we kind of, you know, started this conversation, talking about this muscle memory and be really becoming, um, outcome focused specifically for UX teams. But there's a theme in what everyone kind of said is, uh, there's almost these roadblocks you find with product and I was reading something a few weeks ago where, um, There was a conversation about how a product manager should really be focusing on designer calling design their design partner rather than my designer. And I feel like that idea kind of really resonates within this group of, you know, Maybe product isn't thinking about UX and design as a, as they really should be. So I kind of am curious and wanted to pose the question. Well, you know, Steven's going to solve this, this problem for the whole world and really thinking about how we design for the persona of the UX designer. What would you all want to see a product and product managers do to really help better empowering UX designers to become more involved or a better share metrics, or make sure that everyone's really aligned and talking about OKR circles or focusing on what behavior we want to change for the customer.
Josh
Well, uh, maybe I'll, I'll weigh in here, you know, when, um, when, when I sort of. Started my career. I'll just say, I, I worked as a junior product manager and I worked on some great products and I worked on some really terrible products that from the same company. And I couldn't understand, like, why is this one great and so successful?
And why is this one so terrible? You know, and at some point I, I, I discovered, um, the, this idea of, of what we now call user experience design. And I thought like, yes, this is what we need. All we need is design and nothing will ever be terrible again, you know? And, uh, and I, I learned a lot about design.
I've worked as a designer and then I got hired. To, uh, build a design department inside a company. And when I joined that company, I thought, you know, my mission is to make design successful at this company. And I realized like that was absolutely a hundred percent the wrong mission, right. It was not about making design successful at the company.
It was about making the company successful with what I knew as a designer. And, and, and that meant that I had to, it, that it was that I had to bring what I knew, but I also had to collaborate with all these people, bringing what they knew. And so we needed this kind of cross-functional collaboration, this deeply collaborative, uh, um, way of working.
And that in my mind, the best I've ever seen that work, um, was, uh, it was at a, um, for a while we ran a street, I ran a studio, uh, that built digital products and services. And we assigned every team. The starting point of every team was one designer, one product manager, and one engineer. There was no team that we built that was smaller than three people.
And they always had those three roles. And those three people were co-leads on the team. Right. They, they each owned, um, they each own their domain, but they were, they made decisions together. Uh, we, we put one person, eventually we ha we realized we needed one person who was going to be the decider. And so that was whoever was the senior person on the team could have been a technologist.
It could have been a product person who could have been a designer, but when those three people were working together as peers. Right. You got what you were talking about, Hannah, which is this notion of a product partner, a design partner and engineering partner. And it wasn't this notion of w w this notion that I really hate, which is the product manager as the mini CEO.
You know, that's, I think when you have that model, you're, you're setting yourself up and your team up, you're setting yourself up as a product person for failure. And so, um, I think the degree to which, and it's different in every organization, the degree to which you can build those peer relationships and those peer responsibility is, uh, is to me, that's the way you're going to win.
Hannah
Yeah. Any thoughts, Andy?
Andy
Yeah. Yeah. And I personally I'm, I'm fine with a product manager saying I'm I'm there's that's okay with me. Okay. But. Yes. Like, um, uh, if, if it is, if in our, in our use case, if, uh, if a designer does not feel the same level of ownership as a product manager, that the product will suffer challenges.
If not being one-to-one with the product manager, not insurmountable though, even though that is a pretty big reason for that not being the case, we've got to, we've got to have that same sort of, uh, Accountability responsibility, but really just ownership in that peer. And then I don't think anyone's, you know, in our organization, everyone would be aligned around wanting that to happen. So that's not a challenge. It's just, uh, um, overcoming the challenges that we've we've, we've talked about already.
Steven
I would add this last piece, which is, um, one of my greatest insights as a, you know, multiple time founder came , um, during my third startup, I used to think that I had to do everything and that it was my job to come up with all the right answers, my first two startups. And, uh, that is exhausting by the way. Um, and, uh, because it's, it's impossible. Um, and, um, uh, and I, I kept all the data, uh, to myself. Uh, you know, I didn't share a whole lot of data, certainly, unless it was really good. I shared the good stuff, but I never shared the bad stuff. Um, and, um, uh, and, uh, you know, frankly, you know, the, I had kind of mixed results.
And then my third company, which was Validately, uh, I started with that same mindset. And then I, we were struggling early on and I had this realization as a founder that, you know what this thing is not, we're going to fail if things don't change dramatically. And so, you know what, screw it. I'm going to tell the whole team exactly that I'm going to say, guys, Here's here's, here's where we are.
I'm not hiding the bad stuff anymore. Here's here's the metrics. Our turn rate is terrible. Um, you know, the conversion rate is not as good. The engagement on this stuff is not good. Like here's what I have all the data. Um, and, uh, I need your help. And if we're successful, We're going to watch these metrics.
These are the three max four metrics that we need to be successful on. And, um, if we're successful, like this is what's gonna look like, and we'll be able to raise our next round before we run out of bite. I said, we've got this much runway left to fix this. Um, and if we're not successful, then we're all looking for jobs in, you know, couple months, you know, or, you know, it's like six, eight months, whatever, not much.
And the thing that was the aha was they loved it. They loved it. They loved having all the data. They love having, sharing that responsibility, having a sense of ownership of the outcomes, th the whole culture of the company, the whole mood, the whole everything, cheese, all basically overnight. And I slept better.
It was so much less stressful. We were far more. Uh, productive and successful than I could have been when I was in, I had been in my prior model where I felt like I had to come up with all the answers. Cause I thought I had BC job and, um, and which I'm not, I'm not a founder, Steve jobs. Um, and, um, you know, uh, and that was kind of the aha thing.
So from my point of view, you know, I would flip the question, I would say, um, I couldn't imagine knowing what I know now, being a successful product manager, without the mindset of, I want my design partners, my engineering partners, to understand directly their outcome, their impact, the connection from their work product to.
Changing user behavior and or business impact, um, to have those discussions when things work and when they don't work, um, in a nonthreatening proper non-confrontational way. And, um, and I can promise you from experience for any PM's out there, who aren't sure about, uh, you'll sleep better. You'll be happier and you'll be far more successful.
Um, and so to me, it's not, you know, should they it's it's if they're don't, they're, they're, they're holding themselves back from, from success. Um, and they don't have to be the one that comes up with all the answers that, uh, the designers and by the way, also the engineers have so many creative ideas that if they, if they connect them to outcomes, we'll come, we'll come at them in ways that they had not anticipated.
In, in delightful ways that, um, and that really your job as the leader of the pod or the company or whatever it is, is to create that culture and share that data. So everyone feels that level of ownership, accountability, connection to outcomes, et cetera. And if you do that, I mean, it's just, it's magic.
It's magic. And that's why I started companies because I, I figured that out and it's like, it's so much easier being a founder when I can just say, here's, here's what we're doing. Like everyone, like, here's the problem with, like, we have this thing it's not working. What did we do? Let's talk about it.
Right. Awesome. Yeah. Well, thank you. Thanks. Yeah. As you wrap it up, um, at the one hour, Mark, did you have anything else, Hannah?
Hannah
No, I was just going to say, thank you everyone for your insights. I think it was really helpful. And, um, hopefully anyone listening or watching this later on, we'll have some good insights.
I think there's some really good nuggets in here for people to take away and actually do something actionable on their teams and make some changes that they want to see. Yeah. I loved it. Cool. Thanks everyone. Great to talk to you all. Thank you so much. All right. Thank you. Thanks.
