Embracing a Testing and Experimentation Mindset with Rachel Schiff of Intelycare

Rachel Schiff - S3E5

Kayla: [00:00:00] Thanks for tuning in to Product Chats. On today's episode, I talk with Rachel Schiff who's the SVP of product at Intelycare, and we talk about testing and experimentation mindset, and also knowing your users. So hope you enjoy the show and don't forget to leave us a review.

Hey Rachel. Thanks so much for coming on today.

Rachel: Thanks for having me.

Kayla: Yeah. So in a minute or less, can you tell us about yourself?

Rachel: I'm the SVP of product here at IntelyCare. We're an AI-based platform for matching nurses to short-term work opportunities, primarily in post-acute care. I've worked in tech my entire career.

I've been in product since I got my MBA at MIT Sloan. I currently manage a product team of eight PMs and four designers. And the VP of Data Science in our organization also reports into me. And we're adding a strategy function within product as well.

Kayla: So I kind of wanna back up and talk about how you actually got into product and what that looks like.

Rachel: Yeah, so I've worked in technology my [00:01:00] entire career. I did code for a year or two right at the beginning of my career. People always ask that. I was a developer, I was a project manager. Then I went to business school to get my MBA. I was someone who in college, I hadn't really thought about what the end state would be career-wise. So in my MBA I was like, Oh, what job is the best? And I realized that the people that I'd worked with in tech were so smart and that we got to actually build things that we could use and people could use. And I was like, Oh, that, that was really the best thing to do. So I basically came out of business school as a product manager. And I worked at MathWorks that makes very technical software products like matlab.

Then I worked at Microsoft and then at Virgin Pulse prior to this. And at Virgin Pulse I did a lot of behavior change design, figuring out how to make our product, help our users develop healthy habits.

Kayla: And with that behavior users, Just for my own knowledge, is it same practices as like product management, or are there different things, or what does that look like?

Rachel: Yeah, I think it goes into like, I mean, it it's product management cuz you're saying like, Oh, how, how should my product be?

Kayla: Mm-hmm. [00:02:00]

Rachel: But I want it to be a way that will keep users coming back every day.

Kayla: Yep.

Rachel: Right? And so some of that is about making it a daily habit in some way. And some of it is about prompting them, whether it's through a push notification or an email or something they do in their day to make it a habit.

Kayla: Something I wanna go back to is, you mentioned that you have product and design and data science under you, so I wanna kind of talk about why data science aligns and why it's not it's own function and what that looks like.

Rachel: I actually really think that in our case data science is in many ways the core of the product, right?

Cause we're a marketplace that matches nurses with nurse jobs. And the way that we do that differently and better than anyone else, right, ultimately comes back to data science. It comes back to complex pricing models that dynamically price each shift that we have. It comes back to matching models that figure out like, which shifts do we show to this nurse right now?

Right? Either at any particular time or if we know something that's just happened, like maybe a [00:03:00] different shift was canceled or maybe they haven't worked in a few days. So kind of taking into account everything about their behavior to present them with the right shifts. We also have liquidity models to tell us where do we need to recruit more nurses.

Where do we have enough nurses to support more facilities. So really data science is really integral to our entire offering.

Kayla: So with that, right, data science is really important. You also have products, so how do they work together? Is it product going and asking the questions and then data science kind of designing it through algorithms?

Or what does that look like?

Rachel: That's a great question. I think there's both, like, both parts, like initiate projects, if that makes sense. So yeah, sometimes it'll be product asking a question. Sometimes it'll be product asking a question and, and working with data science to get an algorithm to help with that.

And sometimes it'll be data science to just spending their days in the data and coming up with proposals from that.

Kayla: So I wanna pivot a little bit, and let's talk about kind of this testing and experimentation mindset.

Rachel: Great. That – [00:04:00] I wanted to talk about that . And I think it, it does really go to that collaboration between product and data science.

And so I think one thing that I really try to cultivate in my team is that sometimes as a product manager you have in your head that you wanna build something, but every time you can, you actually should test that first and see what you learn. And ideally, AB test. I think what we were just talking about is a good example. That cancellations is like the bane of this industry, right? That you have a nurse who's ready to work, you've recruited them, you've fully credentialed them, you presented them the right shift in the app, they're ready to work. And then for whatever reason, that facility says they don't need that nurse that day. You know, maybe their person before is working overtime.

Maybe their census is lower than they thought, right? And so like kind of all the work that went into that match falls apart. So one of the things we're really trying to do behaviorally in our platform right now is to reduce cancellations. Right? And there's so many different tactics that you can think of, right?

[00:05:00] That's a goal that we have in mind, and there are a lot of different ways to do it. And really structuring that as testing a number of different options, being able to measure the impact that they have and then building on that. And I think, I just think sometimes everyone gets excited about ideas and wants to go kind of build a final version right away, but it's really an art form to think about how can you do a relatively inexpensive test to learn if what you have in mind will change behavior.

And I think potentially, and I'm not working at your company, but a lot of times when we talk to product leaders, it's about the why, right? Understanding, like, why did you cancel? What are the reasons? And so getting that knowledge, and we all want a solution, like, yeah, many of us want a solution, right?

And it gets back to that like, What is that pain of why they're canceling? Right? Or what is that reason? And understanding that, and then building things around kind of that workflow and understanding, okay, here's the most common reasons. Are there any ways we can work around these kind of hurdles or the reasons why they're canceling?

Kayla: I think that's a really good [00:06:00] point, and that's something that we try to do a lot, is to put in data gathering mechanisms at every point we can like, Okay, you canceled. You know, tell us why, right? And now we're at the point of, again, typically it's because they already have someone there who can work. So now we're trying to help them earlier, you know, plan ahead and solidify to know who they'll have so that we don't get, you know, a nurse at their door who then gets turned away.

Rachel: Right? I mean there's obviously like kind of the AB testing of messaging, right? In our product in particular for this, they've kind of been a fair number of like more kind of educational messages like, did you know this is disruptive to someone's life? And, and we're kind of trying a range of tones now, right?

Maybe it's better to be funny, maybe it's better to be stricter, sort of. And just looking at all the different levers. I mean, that's one of the things that's exciting about a real time marketplace is there are so many levers that we can adjust to try to change this particular behavior. So definitely. Exactly. I think you certainly are much smarter in targeting your tests if [00:07:00] you have a better understanding of the motivation, which gets to like really the importance of understanding your users.

Kayla: Yes, yes. I think you also bring a good point of doing it in small ways. Right? Hey, let's discuss the messaging, right?

Instead of building out like this whole new product, right, around cancellations, it's maybe let's, maybe it's just in the messaging of helping them understand that these are like nurses and they have more of like need. You need to understand the persona better of what they're going through. So let's talk a little bit more about like what you do to better understand besides like AB testing.

Let's talk a little bit more about what you do to understand your users.

Rachel: Yeah. So I mean, the first thing I have to promote in this regard is not going to, you know, shock anyone. It's just asking them. And I really have been amazed. I mean, we have little popup surveys we can do in our app. I don't wanna do like product promotion.

I know there's, you know, lots of SDKs you can put in to, to do that, but it's just amazing. Like pretty much any time we ask our users, [00:08:00] they answer. Right? So then in like a day we get 5,000 responses and we, as you say, know so much more than we did before. I think last week we were kind of having a whole debate about like how many of our nurses are in school in addition to working for us.

And we were like, Oh yeah, let's ask them. Right? And we asked them. And a day later, you know, we know.

Kayla: And I think you, I think you bring up a great point with that, right? Like you have so much data and that's one of the differences, right? Like in yours, it's sort of like this B2C marketplace. And with B2B it's a lot more challenging because maybe they only, when they're starting out, they only have like five customers and they're basing the data off of this.

And so I'm sure this is like, it can be good and bad at the same time, right? Cause you can have so much data and be biased with the data. But one of the cool things is you have all of these data points, right? And you're like, oh, 10% of like, our nurses are going to school. So like, how can we make sure that their needs are met?

Right? Or does this change their customer journey? And so I think you get this really good sense of your customers because [00:09:00] you have customers already that you're solving a pain for. And those, so they're so much more willing to answer these questions and be engaged.

Rachel: Yeah, I mean that's definitely true. I won't say we're not lucky.

We certainly are lucky to have really engaged users. I do think so. A few other things, I mean we, there's obviously a B2B aspect of our platform also. We do have the same popup messaging. I think they're like many B2B companies. There's the disconnect that our users aren't necessarily kind of the overall decision makers.

Right? And so kind of depending on what we wanna ask about, if it's like a usability thing. Oh, that I do wanna promote this too. That we actually like will pop a message driving them to an online testing platform for our nurse users. It's like, oh, we have like a prototype we wanna test. Great. We just pop the message.

We can get like hundreds in like a small fraction of a day to, to try out a new experience and give us feedback. So that's been like life changing for us as we move from like trying to recruit people individually to just having the whole flow [00:10:00] be ours. But back to the B2B part. So one thing we do recognize is that in B2B we're, for usability things,

again, we can use some of those techniques to communicate with the end users. You know, do you like this flow better? Do you know that this thing is over here? Like how do you like to do this? But that is a little bit different from the purchaser. Right? So keeping those two roles separate. I do think that as you're saying for a smaller startup, you know, more B2B, fewer customers a lot of times, right, you can develop a close relationship with individual customers and figure out what incentives you can provide. Cuz recently we did launch a SaaS scheduling product for post-acute facilities. I mean, it's pretty flexible. It could kind of be for anyone, but, and in that case, we definitely looked to partner with pilot users and figure out what incentives would make sense to get their feedback on a really regular basis.

Kayla: So I wanna talk, I kind of wanna pivot into hiring. I know you managed, right? You have like eight product managers, you said four people in [00:11:00] design, and some data scientists. So A) I guess this is like a two-parter. The first part is like: as you manage this team holistically, are you looking for different qualities in design and product and data science?

Or like how do you make sure that you're creating a really cohesive team when you're bringing all of these parts together?

Rachel: Okay, and I'll say like I oversee all the products hiring, but that, you know, design mostly overseas, design, hiring, and for sure data science overseas data science hiring. I think there are different parts, right?

There's obviously kind of a culture fit part that's all the same. I would say the same value of experimentation, of wanting to test and learn and move fast and being able to think about really lightweight test scenarios to learn what you need to know. Those, I would say apply across all of them, but certainly we're hiring for different skill sets.

Yeah, I would say culture fit is consistent. The skill sets are obviously quite different, but I don't, I guess I [00:12:00] don't so much see a conflict there, but we do tend to interview across the group so that we'll get input, you know, input from design on product input from product on design, and the same with data science and with engineering as well, to kind of make sure that whoever we get has the skills to work cross-functionally in the ways that they need to.

Also with stakeholders, obviously critical.

Kayla: Yeah. Yes. So with that also, like what are the, when you're hiring, what are the skill sets that you look for?

I think it really needs to be someone with a passion for user experience. I guess that could be, that's where I think product and design are very similar, right?

In that it needs to be someone who's gonna come in and say to your point, like, how am I gonna solve the pain or need or job to be done of this user? And I'm gonna kind of stick with it until it's there. And so I, we do tend to talk a lot about projects they've done and have been proud of, apps that they like in the world, making sure that they have that, that sense.

And is there anything else, right, so kind of that like user experience. Is there [00:13:00] anything else, like genuine curiosity or like, I mean, these are things I hear a lot and I'm just curious if you look for them. A lot of times I hear a genuine curiosity or I hear like, obviously getting to the why, understand the user experience, but I also hear a lot of times like being willing to say no because everyone has their opinions and so.

Being able to say no. So what are some other things that you look for when you're looking for like that next product management hire on your team?

Rachel: Yeah, I mean, I think the point you just made around prioritization is critical. That's a huge critical piece of the product manager job – is basically figuring out what you need to say.

And we, yeah, I mean we just had some interesting situations with integrations where, when the product manager came in, she really took it up a level and said, Okay, like, we could integrate this product you've selected. But did we look across the big picture and say, If we're not really committed to this vendor, do we really wanna build out a custom integration?

Or is there something out there that, you know, preintegrates with one of the many [00:14:00] tools that are already in our ecosystem like Salesforce or Marketo or Snowflake or Domo? Um, and, and just really kind of brought that to the conversation. So I think there, there's a piece of flexible thinking and, and I think it goes related to saying no for sure, but it's also kind of being able to balance effort and reward.

Kayla: And I think something you bring up that's such a great point is a lot of times product teams wanna build everything out and sometimes it's, let's take a step back and look at our ecosystem, what exists already, rather than let's build, build, build. Because obviously we all know engineering resources are very expensive ,so what if there's this maybe something that exists already? And so I, I applaud that product manager for being like, hey, let's take a step back. Right? Like, and I think that's a big piece of product management – is taking that step back and thinking holistically, right? Like, here's the pain, but to get from that pain and solve it, it doesn't always have to be us building a [00:15:00] product or have to be this certain way, right?

Like for you, maybe it's in-app messaging in a different way of changing the thinking around someone. Or maybe it's, Hey, let's look at our ecosystem and what exists already.

Rachel: Yeah, definitely. And we're, I mean, to be honest, we're growing so fast that we've been forced into some of that. Like we just can't kind of bring on as fast as we wanna grow, so, so what are the pieces that we can buy? And I think, I think it is really healthy. I agree to have that mentality.

Kayla: One last thing I wanna kind of go back to is prioritization. And I think that all product teams think about this, right? There's so many options of what you could build and all the ideas you have around what you wanna build.

So how do you figure out what you prioritize and how do teams decide that that's what they're gonna build?

Rachel: I mean, I definitely think a piece of it is like good old fashioned revenue modeling, right? Like, okay, if we do this, what does it get us?

Kayla: Yeah.

Rachel: That, you know, that's kind of for some of those market expansion pieces.

I think for some of the user [00:16:00] experience pieces, that's where it does. Sometimes it helps to figure out like, okay, what would I ask the users to figure out if I should build X, Y, or Z first. Again, like the asking the users, I feel like you often get answers you wouldn't totally expect. And then that's where again, I think the effort and reward piece comes in.

Like, you don't wanna only do the little things so you're not moving the big boulders. But at the same time, like, again, in a market as fast moving as ours, it's kind of like, oh, you know, where can I really make our app a better experience in a pretty short timeframe? So picking, you know, some mix, I guess, of the big boulders and the quick hits.

Kayla: And I think you bring up a great point with that. Like a lot of times a lot of people just wanna do like the small things that can make a difference. But I think it's in the end you have to kind of do that balance of, hey, what are these big things that can make impact and maybe have a little bit more effort.

Obviously have some testing around that. Maybe do some user interviews, right? But do a mix of these little small wins and [00:17:00] some bigger boulders that kind of align with that strategic view of like, or the strategic goals of the company.

Rachel: Yeah, absolutely. And I think, I mean that's honestly even comes back to the revenue modeling of when you're in the hyper growth kind of mode that we're in and you're like, how can we be a totally different thing 18 months from now than we are now because we're gonna be five times as big? It does drive you to say, oh, well, to be five times big, we're gonna have to change this part and we're gonna have to change this part. And those are, those could be big boulders and let's staff them and keep those going while at the same time we deliver incremental benefit.

Kayla: Right.

Rachel: I also think like we do have an active social media community. Like mostly in their own groups. But I think being attentive to kind of anything that people complain about, I mean, obviously we ask for NPS all the time, but that, to be honest, that the Facebook groups, they set up themselves up in really valuable for just knowing anything people are complaining about. Obviously you wanna make it better right away.

Kayla: And I think you bring up a great point with that, right? About listening to your users and where they're at. Like it's not always about, hey, come fill up this NPS score, right? [00:18:00] It's maybe, hey, we know our users live in these Facebook groups, so let's actually be part of the Facebook groups and engage with them.

And so I think like at the end of the day, it's about listening to them. And then you get this information that you may not have gotten when they're filling this like score out or they're filling out some survey, you're actually hearing it through like organic conversations. So I think that's a great point that you bring up – it's meeting people where they're at.

So on that note, I wanna know, what is one piece of advice that you would give to an aspiring product leader?

Rachel: I do think that any opportunity you have to get deeper in tech will absolutely help you in the long term. I mean, I just interviewed someone for a senior job. And I was like, you've never coded, ever. And he was like, no.

And I feel like even if just a little bit, I think it's really helpful and valuable. That would certainly be one thing. And the other is to like truly apply that product mindset to products as you use them, right? Like, what's working for me? What's not? And think about why. And it really helps more than you think when you're able to say, oh wait, didn't I see a good version of [00:19:00] this in this other app?

Kayla: Great. And then where can people find you?

Rachel: Oh yeah. Well, LinkedIn is always good. Rachel Schiff, speak your product at Intelycare , but I'm also on Twitter, @rschiff on Twitter.

Kayla: Awesome. Well, thanks for coming on today.

Rachel: Yeah, thanks so much.

Kayla: Thanks again to Rachel for joining us on today's episode of Product Chats.

For more product management resources, head to canny.io/blog, and we'll see you next time.