Episodios
-
I’m excited to share this conversation with Ankit Raheja. Ankit is a lead product manager focused on AI, data, and APIs at CDK Global. During this conversation, Ankit discussed the AI product development lifecycle, metrics for AI products, and how product managers could start their AI journey with small steps.
Links
Ankit on LinkedIn
CDK AI Survey: What Automotive Leaders Think About Artificial Intelligence
DeepLearning.AI: Start or Advance Your Career in AI
Transcript
[00:00:00] Himakara Pieris: Welcome to the smart product show. My guest today is Ankit Raheja. , to start things off, could you tell us a bit about, , your current role and how you're using AI, as a product manager?
[00:00:11] Ankit Raheja: Absolutely. Currently I am a lead product manager at CDK Global.
[00:00:20] Ankit Raheja: CDK Global is the largest Car dealership software company in the United States, we power more than 15, 000 dealership location. So, so that's why it is one of the most biggest force but which you haven't, which you haven't heard about because you do not interact with it directly, but I'll tell you 15, 000 plus dealerships are using it.
[00:00:53] Ankit Raheja: And. We are embedded across the whole user journey, starting from [00:01:00] the front office. Front office is when you go to a dealership for purchasing a car and getting all the different warranties and insurance options. Second is the fixed operations. The fixed operations is the car services that you get done when you go to a dealership.
[00:01:21] Ankit Raheja: Then there is some back office. You can imagine dealerships need to take care of like inventory of the parts. And the vehicles and there are many more other things and last but not the least these dealerships need Massive infrastructures to run so we are embedded across all these four Parts of the user journey, the next question that you mentioned about Like where exactly we have used ai so so I have been in the ai space since 2013.
[00:01:55] Ankit Raheja: It was a combination of data and AI. In past, we [00:02:00] have used AI across companies such as Cisco, Visa, and state compensation insurance fund. We have worked number one in the customer support. Use cases, then we have worked in market segmentation, use cases that visa and finally healthcare fraud detection, use cases that state compensation fund currently where I'm using AI at CDK, we are leveraging it across multiple ecosystems.
[00:02:33] Ankit Raheja: Number one is we are trying to match potential customers with potential cars. So it's like a propensity to buy a model. Second is predictive service. Basically what we're trying to do is that when you go to a car dealership and, and sometimes you do not know what services. additional services that you need.
[00:02:56] Ankit Raheja: And, and, you know, you are a busy professional, [00:03:00] you have so many other things to worry about. So we want these car dealership employees to be able to recommend you additional services that you may have not even thought about. So that's the second use case. Last but not the least. We are also exploring benchmarking use cases where something like dealers like you, for example, you have one dealership group and you don't know whether, how are you doing?
[00:03:24] Ankit Raheja: Like, are you doing well? You need to back up on few of the things. So, so that's where the benchmarking comes in. So these are the current use cases. And as you know chatbots are becoming more and more prevalent now. So, yeah, but right now just want to focus on the current use cases and the use cases that I've worked on previously.
[00:03:47] Himakara Pieris: Great. And before this you had an interesting use case with chatbots at Cisco as
[00:03:54] Ankit Raheja: well. Absolutely. Yeah. I can definitely talk to you a little bit about the [00:04:00] chatbot at Cisco. The, let me tell you some... context around the issue. Basically Cisco has lot of switching products, router products, basically all B2B products.
[00:04:17] Ankit Raheja: And some of them as you can imagine will become defective and, and you want to return those products. However, Cisco identified that a lot of these products do not need to be returned. Some of them are avoidable returns. So technically we were trying to solve an avoidable returns problems. This existing way to solve that was that these customers would reach out to the technical assistance center engineers.
[00:04:55] Ankit Raheja: who are technical customer service engineers, if [00:05:00] in, in more layman terms, and they troubleshoot these problems from them and then decide whether this product should be returned or not. We realize. AI could be a really big help to these technical assistant center engineers because you can basically have a lot of skill.
[00:05:25] Ankit Raheja: Number two it's like an intern. AI is like an intern, which is trying to learn new, new things. So as it learns more and more, it will get, become better and it will become a lot more helpful for them. And sometimes these technical assistance engineers are not available, that's where this chatbot can come in.
[00:05:43] Ankit Raheja: So, multiple use cases, why we thought AI made sense, and, and we really had great impact by leveraging AI for this use cases.
[00:05:56] Himakara Pieris: So Cisco and CDK, these are very large companies [00:06:00] with a ton of use cases.
[00:06:02] Ankit Raheja: How did you decide
[00:06:04] Himakara Pieris: the use cases and when to use AI, when to not use AI and what kind of framework do you use for that?
[00:06:12] Ankit Raheja: Absolutely. I'll have a spicy take on this. The first rule of AI is not to use AI in the first place when you're in the discovery stage. You should be able to understand how. A human can do this work better for example, I'll give you two examples, autonomous driving car, what could happen right now, instead of autonomous driving car, what's happening, you're the one who are driving, so you're the one looking around, hey, here's the signal, here's this pedestrian, here's this road, so you should be able to do that first.
[00:06:51] Ankit Raheja: Another thing for chatbot, right? So we had this technical assistance engineers who were doing it. So, so this is a very, [00:07:00] the framework is pretty simple and universal. AI is only one of the ways that may solve this customer's problem while ensuring its need to drive business value. We have seen so many times right now, as you've seen with the chat GPT hype, more and more products are coming out, but the time will tell how many of them will really be retained.
[00:07:25] Ankit Raheja: Right now there's big hype, but eventually retention is the key. So to think about this, I have a very simple framework and this is overused a lot, but there's a bit nuance to it. The number one is user value. Are you providing real value to customers? Why should these customers hire your solution? Are you helping them with their jobs to be done?
[00:07:52] Ankit Raheja: So that's the first thing. That's the first constraint that you'll look at. Number two, which is very important. You may not even get [00:08:00] funding if you don't have a good answer for it. That's your business goals. Just because your c e O said, Hey, I see the chatbot chat gpt is doing really well. You need to really start from the vision.
[00:08:12] Ankit Raheja: Go to the strategy, goes to the goals and come with your KPIs. And what are your KPIs? Do you want to acquire more users? Number two, you want to retain more users. Number three, you need to monetize these user more by upscale or cross sell. Or last but not the least you need to drive more word of mouth, net promoter score.
[00:08:33] Ankit Raheja: So that's the second thing, the business goals. The last constraint that we need to think about is the The technical component of it, like how comfortable are you? Okay. Using a predictive solution versus a deterministic solution. Sometimes, if you can imagine [00:09:00] there like you can make a machine go through and read one medical chart for cancer.
[00:09:10] Ankit Raheja: Would you give all the... Onus on the machine to make a call. I would not say that. So you still need to have a human in loop. However, in some cases like recommendation engine for Amazon, there are so many different permutation combination that can, can, can come with the long tail option. So that's where the the AI makes sense.
[00:09:33] Ankit Raheja: So it all depends from case to case basis. If you want me to go more into detail, I can definitely go more into detail about the AI use cases.
[00:09:41] Himakara Pieris: generally speaking, start with with a focus on customer value and then map it to your business goal and strategy and have clear KPIs. And make sure that your proposed solution could deliver on those KPIs. Absolutely.
[00:09:59] Ankit Raheja: So,
[00:09:59] Himakara Pieris: how [00:10:00] would you compare, let's say, more of a deterministic solution? So, if you have a, I'm sure at all these companies, you have a very large and long backlog of things that you could do.
[00:10:10] Himakara Pieris: Does this mean that AI solutions are possibly going to sink to the bottom of the backlog? because they are relatively more difficult to quantify or the, you know, the, the time to value might be not as quick as more of a deterministic solution.
[00:10:28] Ankit Raheja: Sure. So it all depends on the use cases as. We have made this possible in this world of building and launching something fast and getting feedback.
[00:10:42] Ankit Raheja: You can always build a minimum viable product. What I call it is minimum viable algorithm. You can always build a simple model. For example, if you think about LLM use cases. [00:11:00] You can always, there are still, there are so many other machine learning libraries which are already available that you can use to prove out the value quickly.
[00:11:10] Ankit Raheja: And then you can get a buy in from your leadership. It's all about influencing without authority and how will it drive value? And then like after you get a little bit of buy in, you start putting more and more bodies on this problem. And so it's a little bit different from the normal product development life cycle.
[00:11:33] Ankit Raheja: There's another product development life cycle called AI product development life cycle, which makes sense a lot here compared to the normal other products.
[00:11:43] Himakara Pieris: Very cool. Let's talk a bit about the AI product development life cycle. And also on the back of that, I'd love to pick your brain a bit about designing and building AI MVPs as
[00:11:56] Ankit Raheja: well.
[00:11:57] Ankit Raheja: Absolutely. Yeah. So. I [00:12:00] think the good example would be to talk about my experience at Cisco. Yeah, I think let's share the case study here. I think that will make a lot more sense here. For AI product life cycle, the number thing and, and it is universal. That's why we should really start from first principle.
[00:12:21] Ankit Raheja: Problem identification, whether this problem makes sense for us to solve, whether we have the the, the value that we're able to get from it, and third, it aligns with the strategy. For example Amazon is not going to start sending rockets in the, in the, in the universe, it will be the other product groups.
[00:12:47] Ankit Raheja: So it all depends, like, how does it align? So number one is. Problem identification. Number two is since it's a data product is about your data sourcing and data preparation strategy. [00:13:00] You can start small with taking some sample data and see how it's generating value. So but as you know 80 percent of time goes into cleaning the data.
[00:13:15] Ankit Raheja: And 20 percent time gets into building the model. So, so data sourcing and data preparation is the number number two step. The third is the model building. You build the model, you launch the product the small product in the market, or you do a beta test depends on you. And then you do. Tracking on top of it.
[00:13:36] Ankit Raheja: So as you start doing tracking on it, you'll get more and more idea. You will iterate over it and either change the problem, change the data set or change the model. So it all depends. So at Cisco, basically we had a triple track agile process. I had a track which was working on discovery of [00:14:00] different machine learning models, because at Cisco, when we started with our accuracy was not that great. So it was a little bit lower than the human benchmark. So you can imagine that there was some hesitation for these technical assistance support engineers to adopt. this product wholeheartedly.
[00:14:25] Ankit Raheja: So one team was working on discovery of the the, the new and latest models and how we can improve the accuracy. The second, the track was all about data acquisition. You live and die with this data. That's why you will see here. The big tech is spending so much time building their data mode. So. So the next track which can work in parallel is data acquisition.
[00:14:53] Ankit Raheja: They need to start sourcing the data. They need to start cleaning the data so that can be fed to a module. Last, but not the least [00:15:00] delivery of these models. It's not like building a model in the like in a Jupiter notebook. It's about deploying this mo model in production so that you can get feedback.
[00:15:13] Ankit Raheja: So there were three different tracks. If you have thought about a normal product development lifecycle, you'll be putting all of them in one track itself. And then you can imagine AI engineers, AI infrastructure, data cleansing. is not a cheap affair. That's why 90 percent of products fail because we are not thinking about setting our processes better to really drive quick value and have a quick iterative step.
[00:15:46] Himakara Pieris: It sounds very interesting. It sounds like you had them grouped under different sort of competencies as well from a team structure and organization standpoint because when you think about model discovery I'm thinking of ML engineers and Did [00:16:00] acquisition clean up data scientists and, and delivery off the model, MLOps and CICD folks.
[00:16:06] Himakara Pieris: So, so how did those three groups sort of collaborate in that kind of environment? Like, you know since these are three parallel tracks, I'm, I'm guessing the deliverables or the sprints are not necessarily aligned at all times because they might be making progress at different, different
[00:16:24] Ankit Raheja: speeds.
[00:16:25] Ankit Raheja: Perfect. Yeah. Thankfully I, and it all depends from companies to companies. So context, as you know, is the most important thing in industry. Like you cannot just use best practices or Facebook and apply in a startup. You can't even take some time, best practices of company like Google and put into Facebook.
[00:16:45] Ankit Raheja: They are so different. So thankfully. The way it worked really well in our favor at Cisco was there was a ceremony called Scrum of Scrums. We had one program manager who used to own [00:17:00] these three different tracks and we will have a weekly meeting where we, we talk about like what went well, what help we need, any, any blockers, et cetera, et cetera.
[00:17:10] Ankit Raheja: So, so that's why there was a sync up. At a regular cadence and so scrum of scrums that made sense like so that was more of a Cisco process but like if you're a startup sometime the same person is the mlops is done by the data analyst As well as the data center. It's the same person doing everything.
[00:17:31] Ankit Raheja: So it all depends.
[00:17:41] Himakara Pieris: I want to talk a bit about bringing these solutions more from a, like a go to market and distribution standpoint. Are you working with 15, 000 car dealerships now, right? How does that process look like? Do you do like, you know, incremental releases going out to these folks? Are they part of like, you know, product [00:18:00] discovery?
[00:18:01] Himakara Pieris: Could you talk a bit about that as well?
[00:18:04] Ankit Raheja: You know you've touched on Extremely important point and I'm realizing that industry is still in the, the discovery and development stage and we don't give a lot of weightage to the GTM, but you have seen the, the beauty of the GTM strategy that open AI had.
[00:18:29] Ankit Raheja: When they first of all launched the product really quickly, they already had a tie in with Expedia of the world and, and, and, and Instacart also. So they had their GTM ramped up really well. So, and for us also at CDK as a B2B SaaS giant, GTM is taken extremely seriously. So for a few of the products, what we have done, number one, we take help of these customers to get an early access.[00:19:00]
[00:19:00] Ankit Raheja: to do some kind of a co development with them. We did it for one of our product offering called Data Your Way. It's a data product, it's not an AIML product, but that's what we did. We got their feedback, and we launched the product within a few months after working with them. After the co development phase, next came for us, the beta stage.
[00:19:25] Ankit Raheja: There we expanded our sample set to around 15 dealer groups. There's a difference between dealerships and dealer groups. One dealership group can have multiple dealerships under them. So we worked with 15 dealerships for our beta stage. And finally, We launched our product after they being in beta for for, for a few months and now the product is in GA.
[00:19:52] Ankit Raheja: So it all depends. The same thing applies to AI ML products. Also, you, you co create with them give [00:20:00] them some like extra credits or, or give them on a discount so that they can help you decide because it's a skin in the game for them also, and then you can put in your proper telemetry in place.
[00:20:11] Ankit Raheja: And then you can expand and make it a lot more better. How do you
[00:20:15] Himakara Pieris: facilitate communication and collaboration during that process? What kind of metrics do you look at? What does the feedback loop look like?
[00:20:25] Ankit Raheja: Absolutely. How the first one to how to facilitate this conversation is like, I think it again, it all depends from company to company like depending on the size of the company I was spread across multiple products.
[00:20:42] Ankit Raheja: So there I leverage. My amazing customer success management team and customer sales team who really had a one to one relationship with these customers. I would also go into the meeting, but they will be the project manager. We'll have our spreadsheet [00:21:00] where we'll be talking about, Hey, these are the feedback that we got from the, from these customers.
[00:21:06] Ankit Raheja: These were the, the, the pluses, these were the deltas. And then we will be having these bi weekly meeting with these Select customers and we'll tell them that hey, this is something that we are working on to to keep them In communication metrics what we were tracking were that we had a really good funnel system that hey, we started first with 30 prospective beta dealers, dealership groups, thinking that many of them will be busy with so many other things.
[00:21:40] Ankit Raheja: Then we knew that this will number will go down. We wanted a critical mass of 15 plus. So we, we got it. We have seen some failures in some products. What has happened was that we only start with one or two dealership groups. I think that's a recipe for disaster because if you start with two or three, it's so obvious, but as you know, hindsight is always 20, [00:22:00] 20, like always start with a big group and, and expect that your customers have busy lives.
[00:22:05] Ankit Raheja: You are just embedded in part of their solutions, it all depends.
[00:22:12] Himakara Pieris: So what kind of like specific business and customer metrics do you track? And are they any different from your traditional SAS
[00:22:20] Ankit Raheja: products? Oh yeah, definitely. So that's why there's a nuance to these different metrics. So first of all, the first one remains exactly the same.
[00:22:33] Ankit Raheja: These are your business and customer metrics. To give you an example again for the Cisco number one was that like how many additional cases. That you're able to handle with a chatbot. Imagine what can happen is that I, the chatbot can come in and can try out some use cases some cases for you before sending it to a human being.
[00:22:59] Ankit Raheja: So like, [00:23:00] like how much uplift you can do with this. Number two sometimes you don't need to staff so many additional customer service engineers. So how much it's helping reduce in, in the personal costs. Another business metric could be that how much reduction in the avoidable returns that you are able to get through that.
[00:23:24] Ankit Raheja: So these are the two high level metrics. The third metric, again, the third business metric is your net promoter score. Alexa does a great job in it. Like sometimes what happens is Alexa would be asking you, how much would you rate this response from one to five? So we thought, why don't we learn from Alexa and start leveraging it so that we had a net promoter score also going in.
[00:23:47] Ankit Raheja: So these were just the business metrics. What changes In the AI ML space is the next thing that shows up is your, your algorithmic metrics, [00:24:00] which is like when you're trying to do like the, the modeling, you need to worry about like, Hey, What's the simple metrics like accuracy, precision, recall, it all depends like, and it depends from you, what do you care the most about?
[00:24:16] Ankit Raheja: Do you care more about accuracy versus whether you care about precision, whether you care about recall? What I have seen is that like sometimes people forget about it that which metric is the most important from models and you can solve the wrong thing. For example, for some places, like when you're doing cancer detection you need to be really careful about the false positive metric.
[00:24:44] Ankit Raheja: Sorry false negative metric. The false negative metric is if somebody has cancer and if you don't tell them that they have cancer That's a bigger problem than they're going into the the cancer treatment and getting a chemo They will not be happy but hey, [00:25:00] they're still alive. But if somebody takes a lot of time to get the cancer detection done I think that's the bigger problem.
[00:25:08] Ankit Raheja: So we have to be really careful about this, which model metric we should optimize for. Last but not the least is the ML infrastructure metrics. Basically sometimes you need to worry about latency, right? Sometimes do you want a fast model? Or you want an accurate model, because there are two different things.
[00:25:35] Ankit Raheja: Do you want your model to be available on edge or you want it to be on cloud? So you need to worry about the infrastructure metrics also. To summarize three kinds of metrics, business and customer metrics, algorithmic metrics, and ML infrastructure and production based metrics.
[00:25:56] Ankit Raheja: So
[00:25:56] Himakara Pieris: I presume the first and last buckets customer metrics [00:26:00] and infrastructure metrics are more visible, but algorithmic metrics tend to be less so. And sometimes they reflect in the other two. Do you keep the customer, especially early adopters or like, you know, development partners in loop about algorithmic metrics?
[00:26:17] Himakara Pieris: And what is the level of conversation there as you share this with the business groups?
[00:26:22] Ankit Raheja: Oh, perfect. Again Cisco was an amazing playbook that I can talk to you about like, so and this came so often number one rule like I follow this product management guru and I'll give a shout out to him.
[00:26:39] Ankit Raheja: His name is Shreyas Doshi. He says you are not dev complete. unless you have your telemetry built in. So the step one for making sure that you are GA, in fact, we have it at CDK also, that you need to build your telemetry in place. If you have your telemetry built in place, things become a lot more [00:27:00] easier later.
[00:27:01] Ankit Raheja: I'll give you an example for Cisco, a chatbot, right? A chatbot is there and it is giving a prediction of, of, of saying that, hey, this product should be returned. We will be having this weekly meeting with our developers and we'll tell the developers that, hey, this is where the chatbot is telling that we should return this product.
[00:27:28] Ankit Raheja: The human, when he's coming in and he's checking it, they're saying, no, it should not be written. So we had a Tableau dashboard that was capturing the model scores as well as the, the human recommendation and whenever there was a delta. We will be surfacing that to the development team, they they will take it, they will retrain the model and then like that's how we will keep on iterating over it.
[00:27:58] Ankit Raheja: So, but the first thing is if you [00:28:00] don't have telemetry in place. You will be thinking you'll be regretting why I don't have that built in first place. So step one in ML model, have your telemetry built in place, store it in some database, have some kind of a way to surface those results because otherwise it's just opinions.
[00:28:19] Himakara Pieris: That's really good advice. On the back of that, are there any other advice you'd like to offer to product leaders, product managers who are interested in getting into AI and, and building AI-powered solutions?
[00:28:34] Ankit Raheja: Yeah definitely. The, the first thing that I will request our product leaders is like I'm all about being as transparent and as inclusive as possible here.
[00:28:49] Ankit Raheja: Do not think machine learning is some ivory tower. It was. Thought of SM I've retired earlier. And now you have seen with [00:29:00] the amazing GTM strategy and the product strategy of open AI, like so many folks are jumping on the bad wagon and I do not want the product leaders to be left behind. I want them to have it as one of the items.
[00:29:15] Ankit Raheja: in their toolkit, like, but this is not your front and center. The first thing that you need to understand is how and where can you use AI? I'll give you a few examples here. Something like a SaaS product. Are you in a SaaS product layer? For example like Amazon Recommendation Engine, Coda AI, Notion AI.
[00:29:38] Ankit Raheja: That's like a SaaS product adding AI to their products. Second one could be algorithmic products. Anthropic, OpenAI, Facebook Llama, like that's the second place where you can think about like where you can use AI. Third is [00:30:00] AI infrastructure software companies. There's a company called Cube. That's one example.
[00:30:06] Ankit Raheja: The fourth is AI infrastructure hardware companies. As you can imagine, NVIDIA's stock is at an all time high. Again, it's a trillion dollar company, all because of its GPUs and then with AI and with Snowflake partnership, yeah, it's taken to the next level. So, so think first of all, which What space do you operate in or where do you think that you can use AI?
[00:30:32] Ankit Raheja: Number two, start small by leveraging data because at least you can have some kind of proof of concept to decide whether AI makes sense. Because what's happening now in the industry is that we are able to, number one use a lot of annotation services, which can annotate data for you.[00:31:00]
[00:31:00] Ankit Raheja: Number two, you can also create your data synthetically. There are a lot of use cases there. Last but not least, check out these amazing websites. One is artificialintelligentnews. com. You can look at Andrew Ng's batch at Deep Learning AI. He really comes up with amazing content and last but not the least, you can always ask chat GPT, which is trained on billions of parameters to, to tell you that what could be some use cases.
[00:31:30] Ankit Raheja: At least it's the first step.
[00:31:32] Himakara Pieris: very much for coming on the podcast today. Is there anything else you'd like to share as well?
[00:31:40] Ankit Raheja: The only thing that I would probably like to share is about if you are a car dealership company and so CDK has shared an amazing AI survey and I will provide you Imkara in the case notes.
[00:31:59] Ankit Raheja: So if [00:32:00] any car dealerships is listening to it, they can always look at that link and yeah, and we are excited to talk to you.
-
I’m excited to share this conversation with Faizaan Charania. Faizaan is an AI product lead at LinkedIn. During this conversation, Faizzan discussed the potential of Generative AI and its applications, the importance of keeping GenAI solutions simple, and how to think about trust, transparency, and managing costs as a product manager working in Gen AI.
Links
Faizaan On LinkedIn
Transcript
[00:00:00] Faizaan Charania: I love the analogy with cloud because cloud can make experimentation so easy. And you're just like trying to set up something new. Test it out. See, see if it works. Will I get product market fit? What are my users thinking about this feature? All of these things are also possible with GenAI. So for any PM who's thinking about GenAI, my recommendation would be test it out.
[00:00:24] Hima: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate and learn from industry leaders who are solving real world problems. Using AI.
[00:00:34]
[00:00:35] Himakara Pieris: My guest today is Faizan Charania. Faizan, welcome to the show.
[00:00:40] Faizaan Charania: Thank you so much for inviting me, Hima.
[00:00:43] Himakara Pieris: To start things off, could you tell us a bit about your background
[00:00:46] Faizaan Charania: Yes. I am a product manager at LinkedIn. My main focus is around machine learning and artificial intelligence.
[00:00:53] Faizaan Charania: And obviously these days I've been looking into gen AI as well. I've been in the machine [00:01:00] learning field for around Eight, over eight years now started on the research side, worked with startups, uh, was a machine learning engineer for a bit. And then I switched to product management.
[00:01:12] Himakara Pieris: There is a lot of attention on generative AI at the moment. Could you tell me a bit about the way you see it? What is generative AI and how it's different from all the various other types of AI that we have seen so far?
[00:01:24] Faizaan Charania: Yeah, definitely. There is so much hype around gen AI. Uh, one thing, uh, one code that I've heard multiple times is. Uh, this is like the iPhone moment or this is the desktop to mobile moment of technology again. To answer your second question around, uh, how is it different from all other kinds of AI?
[00:01:46] Faizaan Charania: Because it's like so many things that we can qualify as AI, right? So a simple explanation that I try to go with is. Differentiate these two types of AIs, analytical AI and generative AI. [00:02:00] So analytical AI is where you, where you have some specific features or data points or like historical input, and you're trying to make one single decision based on that.
[00:02:12] Faizaan Charania: So the decision can be, Hey, is this email spam or not spam, spam classifiers? It can be a ranking decision. So say you log into Facebook or Instagram or like any of these applications and what post should appear first? What should be first? What should be second? What should be third? And this is based on the text in the post, the images.
[00:02:35] Faizaan Charania: It's based on what you like, what you don't like. So this is like a ranking problem. So ranking, decision making, all of these are a part of analytical AI and generative AI. As the name says, it's about taking, uh, generating new content. So if it's about post completion and everyone has heard about child GPD, so I'll just like [00:03:00] use that as one of the examples.
[00:03:02] Faizaan Charania: Like, Hey, I asked you a question and give me a response in natural language format. So natural language generation is generative AI generating new images, images that did not exist before is generative AI. So like even for images, if you were to classify an image, Hey, is this. Safe for children or not safe for children, that's analytical, but if you want to generate a cartoon image, that's generative.
[00:03:30] Himakara Pieris: From an overall landscape standpoint. So we have a ton of startups that are out there and then there are a couple of, in a way, key gatekeepers, Microsoft slash open AI. Um, I would say one of them, and then there is an emerging, emerging rivalry with, with Google, um, or refresh rivalry with Google on this front.
[00:03:53] Himakara Pieris: And then there are also chip makers. How do you. So if a map out this landscape, [00:04:00]
[00:04:00] Faizaan Charania: yeah, so, uh, when you're thinking about landscape, yes, Google and Microsoft are big players, but then there's like so many more important players over there. So if you're just thinking about the flow of generative AI at the base layer, you will have the infrastructure companies, these chip companies, and they are the ones who actually make gen AI possible.
[00:04:25] Faizaan Charania: So that's one thing then at the top level, you will have applications that are using generative AI. And in the middle, you would find all of these other players who are building new features and new utilities to even make gen AI, um, efficient. So to give you one example for prompt engineering, there's new companies that are just focused on prompt engineering, making prompt engineering easy.
[00:04:52] Faizaan Charania: Versioning of it, iteration, structures of it. Um, there's a prompt engineering [00:05:00] marketplace now. So people can sell prompts and people can buy prompts. So, I, yes, Microsoft and Google are the popular ones because they're like big players so there's like more Um, media limelight around them, but I think they're, they're just like one of the initial pioneers and there's just so many players and there's so much scope for everyone to be a part of this.
[00:05:24]
[00:05:24] Himakara Pieris: So I think what we're talking about is there is the foundational layer, right? Which Microsoft and Google's of the world are going to provide similarly to how they provide cloud computing today. And there's going to be a huge ecosystem that is getting built on top of it. And prompt engineering sounds like.
[00:05:42] Himakara Pieris: One big part of it prompt prompt engineering and everything that's that goes around prompt engineering Are there any other ecosystem participants at that layer
[00:05:54] Himakara Pieris: in your view
[00:05:56] Faizaan Charania: In the initial days The market is going to evolve a lot [00:06:00] So when these new models were launched and again, I'm talking about November and December you You might have seen, um, a large list of startups that just like came about.
[00:06:13] Faizaan Charania: So those are the ones who are early adopters and who are just making these things, uh, making like new applications possible. I think that's just the spur and that's the wide net that we are casting. But as time progresses, this is going to become business as usual. Gen AI won't be exciting anymore. Then the problems to solve are, hey.
[00:06:34] Faizaan Charania: How do I scale this? How is it going to be efficient? How do I do it for cheaper? And there are many different players who are playing in the infrastructure side of this. There are many new startups. I, there's this one startup that I. Sort of from, I can't remember their name, but, um, they've been working on making Gen AI more efficient for like three years now.
[00:06:59] Faizaan Charania: So Gen AI for the [00:07:00] public, it's, it seems like a new word and all of us are talking about it right now, but the early seeds were sown in 2017. And actually even before that, everyone has been building on the top of giants that came before them. But yeah, the concept has been around for a while and there are new marketplaces.
[00:07:19] Faizaan Charania: There are no new ecosystem players that are just going to solidify even more as time passes.
[00:07:25] Himakara Pieris: Let's say you are a product manager, , for a product that exists in the market today. Where do you see opportunities and threats and challenges, , someone should look out for, , as a PM?
[00:07:39] Faizaan Charania: . My approach to Gen AI is to just think of it as a tool as it is. I've been doing this for like AI for a while and now Gen AI is just a flavor of it, right? So think of it as a tool and see how this tool can help me or my customers.
[00:07:54] Faizaan Charania: Solve their, solve for their opportunities or solve the challenges that they're facing, more easily. [00:08:00] And that is the core of how we should approach all kinds of product solutions. And then see where can Gen AI come in? How can we solve problems using Gen AI? Is there some flow or some funnel that my user is going through right now?
[00:08:15] Faizaan Charania: Where's the friction? Can Gen AI solve that? Can Gen AI make something possible which would make my users happy? But it was too difficult to do in the past. So there are many ways to think about this. The core of all of this should be the jobs to be done, the user needs, and then see where the unique capabilities of Gen AI are going to be useful for them.
[00:08:41] Himakara Pieris: What I'm seeing is that you can use. Generative AI for summarization, , expansion, style translation, I think I can put. Graphic stuff for diffusion into into one of those three buckets as well.
[00:08:56] Himakara Pieris: Am I missing something here?
[00:08:58] Faizaan Charania: Summarization, [00:09:00] expansion, style translation. There's obviously all kinds of like generation. When you say style transformation, this could be just text style transformation.
[00:09:09] Himakara Pieris: It could be anything from turning Drake's voice into JC's voice. I think I see all those as some kind of a, , a transfer operation, right? It's essentially any kind of transductive problem you can, approach with, , generative AI in, in some ways.
[00:09:25] Faizaan Charania: Yes. Yes. And, , to go one step deeper into summarization, even summarization can be done on, uh, like in a very basic manner where you're just like summarizing one document or it could just 10x the speed of research. So a doctor who's looking at new symptoms and trying to find out a diagnosis or a prognosis, or a lawyer who's working on a case and wants to find, um, or precedent, like other cases that are similar to theirs, what happened and consuming all of this information and coming up with [00:10:00] takeaways and next steps.
[00:10:02] Faizaan Charania: These places are also where I see ChargPT and I like ChargPT has just become like a pseudonym for All Gen AI these days. But Gen AI play a very big role.
[00:10:13] Himakara Pieris: What would be a framework or rubric PMs could use, when they approach Gen AI?
[00:10:21] Faizaan Charania: I really like this topic. So the framework is actually almost independent of Gen AI, and then we have to adapt it to Gen AI.
[00:10:31] Faizaan Charania: So, whenever we are building any new product, your product has to be rooted in, again, what the member, what the customer wants, what the user needs, but also some principles. Like how do you want to build things? So if I were to come up with some principles for generative AI products, how I would put them, I would put them in like three categories.
[00:10:54] Faizaan Charania: So the first thing, because Gen AI can do so many things and it's possible to, [00:11:00] uh, it's possible to be like used in many different ways. One core aspect that I would want to obsess over is keep it simple because not every user is going to be an advanced user. Not every user is going to be able to use Gen AI to the full of its capabilities if you just like give the model to them.
[00:11:20] Faizaan Charania: So your product or my product, whatever product you're building should keep it simple for the user. So that's one thing. Um, the second would be is create unique value, create new value. This goes to the look at the members needs and opportunities thing. If, if someone doesn't need Gen AI to solve their problems, we shouldn't just force Gen AI into a product just because it's a new cool thing to do.
[00:11:48] Faizaan Charania: And we've seen this happen with some other technologies in the past. So, again, this is just one thing to be very mindful of when we are thinking about, okay, this is a new technology, GenAI, and how we do [00:12:00] this. So that's two. And I can go into depth of like these two as well. But the third one is going to be build with trust.
[00:12:08] Faizaan Charania: And I'm focusing on trust over here because GenAI, at the end of the day, it is AI. And this is generating new content. So now people are going to read this, people are going to consume this, and for, for a lot of instances. It becomes a black box and we don't know why it is saying whatever it is saying.
[00:12:28] Faizaan Charania: Yes, it depends on all of the training data. Why one particular output came out in this particular instance? We, we can't really answer that. So we have to build with trust. We have to, um, make sure that we proactively think about avoiding bias. inclusion, diversity in our data sets. Um, whenever we do go wrong and we will go wrong, it's AI.
[00:12:53] Faizaan Charania: It makes mistakes. All kinds of AI do have a feedback mechanism. Let your users interact with your [00:13:00] products see where it's going wrong And then as a PM or like whatever company we are so that we can actively work on it And this is going to build trust with GenEA, build trust with your company, build trust with your product Otherwise, it's just a black box and people can be apprehensive towards technology as well Maybe there's yeah, a lot of things can go wrong.
[00:13:23] Faizaan Charania: So having a Continuous conversation is going to be very useful. Okay to summarize keep it simple meet the users where they are keep it Um easy to use second create unique value create new value Only use genii when it is actually useful for your members users customers Whoever they are and third build with trust Avoid bias.
[00:13:48] Faizaan Charania: Think about inclusion in your data set and build for feedback because this is new. You're going to make mistakes. You just need to keep improving.
[00:13:58] Himakara Pieris: You talked about [00:14:00] removing complexity, can we break it down a bit more
[00:14:02] Faizaan Charania: To break it down, let's look at particular examples, like example products that are already out there so that we can contextualize these understandings.
[00:14:10] Faizaan Charania: Okay. So one product that I truly love is Notions AI Enhanced Editor. They were one of the early adopters of Gen AI and launching it in production. So I don't know if anyone listening has used Notion's AI product before or even Notion. So just to give you some context, Notion is for note taking, but it does like so much more than note taking.
[00:14:37] Faizaan Charania: And the AI enhanced editor. Can help you use all of these genii capabilities that uh, he might just mention like summarize translate, uh Transform like change the tone of things so there are many features available, but I love how they keep it simple So the first thing is They have fixed [00:15:00] entry points, like, Hey, if you want to translate, click over here.
[00:15:02] Faizaan Charania: If you want to change, don't click over here. Everything is just one click away. I don't have to worry about talking to a gen AI model and asking for asking in detail what I want from them. I just click and it's done on the backend. It's still using gen AI. But as a user, I don't have to worry about what's happening on the backend.
[00:15:24] Faizaan Charania: I just say, Hey, make it more conversational, make it more professional, make it more polite. And it happens. So that's one thing. Remove barriers. The second, uh, I had mentioned the, I had mentioned abstract the complexity again, by putting everything on the backend, you abstract the complexity. Uh, one thing is the, you could give your, uh, users just.
[00:15:51] Faizaan Charania: What you enable using one click, but then if you expose a Uh, gen ai model and give the user all kinds of hyper parameter [00:16:00] options like hey adjust the temperature adjust the max tokens adjust Uh frequency penalty repetition, but there's like so many things That you can do with, uh, even like GPT 3, the base model and like the new models are have like more capabilities.
[00:16:16] Faizaan Charania: But if you give all of this to the user, like, um, a separate form that where you keep editing things, editing hyperparameters, it's just going to confuse them. So abstract the complexity of it. And in the UI, I don't have to leave my editor for anything. I can just press forward slash. It gives me a section to enter what I want, whatever AI command I want to enter.
[00:16:40] Faizaan Charania: And the output comes in my, comes in my document where I was already writing things. So meet the user where they are. This again, if anyone hasn't used Notions AI Enhanced Editor, check it out. It's a great example of how you can build a simple UI.
[00:16:57] Faizaan Charania: While still doing a lot of powerful things.[00:17:00]
[00:17:00] Himakara Pieris: Let's build on the Notion example then. How do you think Notion created new value, , using this AI powered editor?
[00:17:09] Faizaan Charania: The new value over here is we are reducing a lot of friction that writers have to go through when they are creating any new content.
[00:17:19] Faizaan Charania: Okay. When I come up with ideas, it also has a brainstorm section, by the way. Say, if I want to come up with ideas. Right now, without GenAI, how would I do it? Hey, I would look at, I would search it online, search the topic that I'm writing about, read a few things, or even look at what the industry trends are, what people in the field are talking about, and then get all of that information.
[00:17:42] Faizaan Charania: But with this notion, AI, I just go in the brainstorm section, enter what I want to enter. Um, say I want to talk about how people can transition to product management. Again, this is just a very specific topic, but how people can [00:18:00] transfer to product management as a career. And there could be a 10 other ideas that are similar to this.
[00:18:06] Faizaan Charania: So, okay. Advice for early career product managers or what transferable skills are useful when you want to transfer into product management and like eight other ideas. So that's new value. When I want to expand my blog posts to from like English to Spanish, French, German, there's a translation option.
[00:18:27] Faizaan Charania: Right there. Now I don't have to go to some other product and do it over there. So that's new value. And yeah, new value comes with all of these new features that I as a Notion user couldn't use before, but now I can.
[00:18:42] Himakara Pieris: There are a number of challenges I could imagine right off the bat in the case of Notion, everything from inappropriate suggestions to incorrect, , outputs to hallucination and all that. How do you think Notion is handling the concept of build with trust? [00:19:00]
[00:19:00] Himakara Pieris: Okay.
[00:19:01] Faizaan Charania: A few things that I did notice were one, there's like always a disclaimer that like, Hey, this is AI generated. If anything seems off, please provide feedback.
[00:19:11] Faizaan Charania: So that shows me that Notion is willing to, is. has potentially thought about where the AI could go wrong. So there's like good intentions and they're collecting feedback and are willing to improve. So, um, for, for one of the things I was trying to like write, what I was working on was like write a blog post with a particular topic and I was stress testing it.
[00:19:37] Faizaan Charania: So as a new user, as a PM who works in AI, I was interested in how they did things. So I was just trying to play around with it. And I tried to make it say, um, some not so appropriate things. I, I have done similar things with like chat GPD as well. So, uh, one thing that I've seen is they have content policies [00:20:00] in place.
[00:20:00] Faizaan Charania: So if an output is, um, is against the content policies, it will get flagged. So this is, these are extra layers on top of the Gen AI, uh, model. So that is something that's very important. Um, Fairness, Inclusion, Bias, I didn't see any of those issues in the Notion AI outputs. So maybe they have cleaned all of their data already.
[00:20:26] Faizaan Charania: Maybe my prompts weren't, um, scandalous enough for them to break, but in my experience, they were doing fine. So maybe they've thought about those things as well. The third thing that I won't have much information on, like how notion did is data collection and security. So whenever people. Are talking to gen ai models.
[00:20:47] Faizaan Charania: They are providing a lot of input. So we should actively think about hey Where is this data stored? Is it is it in a secure spot? Is it unsecured? Is it encrypted? There are government regulations [00:21:00] around what data you can keep what data can you can you not keep like based on california law ccpa gdpr So I have to think about those things And, God forbid, if your company gets hacked, that data is going to leak, that sensitive data about members.
[00:21:15] Faizaan Charania: So, you have to be very careful about, uh, what you store, what data you store.
[00:21:26] Himakara Pieris: If you are looking into adopting Gen AI or exploring Gen AI. It's not a cheap thing to do especially if you're adopting OpenAI's, , API. Those, token based pricing could add up very quickly. How do you think, as a PME, you should, you should plan for, , cost and scale?
[00:21:45] Faizaan Charania: This is a very interesting problem that even I am thinking about, um, during my day job because Gen AI is new. It can do a lot of exciting things for our users, but at LinkedIn we have [00:22:00] millions and millions of users. I guess the last number that we had released was monthly active users around 800, 900 million.
[00:22:07] Faizaan Charania: So that's a huge number of people that you have to build a solution for. Okay. So when you're thinking about cost and scale, you have to first think about what the problem is. Like what is the problem for the user that you're trying to solve and in a lot of use cases you won't have to Create an extremely personalized solution for each and every member Members users like whatever your product is at linkedin.
[00:22:38] Faizaan Charania: We like to call them members but Members can be a part of a cluster. They can be similar users. So if I am a PM, the things that I would value are going to be slightly similar to what other PMs value. So whether it's a job application or whether [00:23:00] it's a post that I would want to write. A topic that I would find interesting if there's like, so I'm taking gen AI aside right now.
[00:23:08] Faizaan Charania: I'm just thinking of solutions. So it doesn't have to be extremely personalized. Now if you can put me in a small cohort of 10 users who are extremely similar to me and have very similar attributes and create a solution for them and you do this for everyone at LinkedIn. Now your cost is reduced 10 X.
[00:23:29] Faizaan Charania: So that's one way of going about it. So this is one thing. The sec, again, cluster users and you don't have to go the extremely personalized route. So that's one. The second way you can bring down costs is not everything will need G P T for, or like whatever the biggest bad model is out there. A lot of problems that we generally attempt to solve are going to be, uh, solvable with much smaller models.
[00:23:57] Faizaan Charania: And again, this is how machine learning research goes [00:24:00] in general. If there's a big new model out there, it's the state of the art, it's performing best on all benchmarks. Six months later, it doesn't even take a year generally, but six months to one year later, you'll have models half the size, one fifth the size, one tenth the size doing the same thing now.
[00:24:19] Faizaan Charania: So GPT 3 launched in 2020. ChatGPT launched last year. We have so, like once ChatGPT gained all the hype, there have been so many new models that are out there. We have the Facebook model at 13 billion parameters compared to 370 billion. So cost can be adjusted based on what model you're using. And I know you had alluded to this a while ago, people can train their own models as well, like fine tune their own models.
[00:24:52] Faizaan Charania: And once you fine tune, that's going to be a fixed cost and upfront cost, but then it becomes very cheap because you can host it [00:25:00] yourself. You don't have to pay someone else to host it. So these are some ways like model selections, how you're doing your targeting. All of this can really, really help with the cost and how you can scale your solution.
[00:25:11] Himakara Pieris: You know, when, , cloud computing was new, everyone was trying very hard to estimate costs. I feel like that's going to be a similar exercise with as well. , especially if you're working on a free product, if you have a free component, um, it's going to be. A process to figure out how much it's going to, cost
[00:25:29] Faizaan Charania: I love the analogy with cloud because cloud can make experimentation so easy. And you're just like trying to set up something new. Test it out. See, see if it works. Will I get product market fit? What are my users thinking about this feature? All of these things are also possible with GenAI. So for any PM who's thinking about GenAI, my recommendation would be test it out.
[00:25:54] Faizaan Charania: At a small scale, only release it to 1 percent of your users. See how it works. If it works, then you can [00:26:00] think about, okay, how do I make the scale? How do I get the cost down?
[00:26:05] Himakara Pieris: It would be similar to any other product launch process. I presume in, in that way, if you create enough value, you can charge enough money and then you have the spread there.
[00:26:14] Himakara Pieris: So it becomes a viable product. It's, it really comes down to finding a big enough pain point to solve with this new, hammer,
[00:26:23] Faizaan Charania: Absolutely. There's, um, there's one thing different. Like for Specifically machine learning models, uh, before gen AI, we had to build our own machine learning models.
[00:26:34] Faizaan Charania: That would mean get data, clean data, build a model, iterate, release something, gather information, gather feedback, and then do something. So this was a very big pipeline and that has been cut short because these large models already have a lot of, , data and they're kind of pseudo intelligent.
[00:26:55] Himakara Pieris: Faizan, thank you so much for coming on Spot Products today. Is there anything else you'd like to [00:27:00] share with the audience?
[00:27:02] Faizaan Charania: Yes. , one thing I'd love to share is just my excitement around generative AI. But, um, anyway, so I write about product management. I write about generative AI on LinkedIn. You can find me as like Faan Nia on LinkedIn and if you have anything interesting that you're working on, if you find some interesting products that you want to talk about, just reach out.
[00:27:25] Himakara Pieris: Great. Thank you very much. And we'll share those links in show notes as well.
-
¿Faltan episodios?
-
I'm excited to share this episode with Don Rosenthal. Don is a seasoned product leader with extensive experience in AI and large language models. He has led product teams at Google AI research, Facebook Applied AI, and Uber's Self-Driving Technology division. During this conversation, Don shared his insights on the anatomy of an LLM, ways to incorporate LLMs into products, risk mitigation strategies, and taking on LLM-powered projects.
Links
Don on LinkedIn
Attention is all you need
The Illustrated Transformer
Transcript
[00:00:00] Don Rosenthal: please, please, please do go out and do come up with unique and exciting, uh, important new applications, build stuff that solves important problems we couldn't even try to address previously. I just want you to be sure that, uh, you're going into this with your eyes open and that you've prepared your stakeholders properly.
[00:00:21] Don Rosenthal: There, um, there are a lot of successful applications that have been built with these LLMs and a lot of the pioneers have discovered all the pitfalls and where all the dragons hide so that we can We can avoid them.
[00:00:35] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate and learn from industry leaders who are solving real world problems. Using AI.
[00:00:46]
[00:00:47] Himakara Pieris: , today we are going to talk about large language models. And I can't think of a better person to have this conversation with than Don Rosenthal. So Don has [00:01:00] spent most of his career in AI.
[00:01:02] Himakara Pieris: He started out as a developer. building ground support systems for the Hubble telescope, um, including being part of the team that built the first air ground system ever deployed for a NASA mission. He then went on to build and manage NASA's first AI applications group, where his team flew in, flew the first two AI systems in space.
[00:01:22] Himakara Pieris: And he worked on prototype architectures for autonomous Mars rovers, done then commercialized. Uh, the air technology from Hubble Telescope in two of his air companies that he founded. He was the group product manager for autonomy at Uber 80 G. Uber's autonomous vehicle spin off in Pittsburgh. He was the PM for face recognition at Facebook.
[00:01:43] Himakara Pieris: And most recently, Don was the group product manager for conversational at a I research
[00:01:50] Himakara Pieris: done. Welcome to the smart production.
[00:01:53] Don Rosenthal: Thank you very much. I'm really, really excited to be here. You might. Thank you for inviting me.[00:02:00]
[00:02:01] Himakara Pieris: So let's start with the basics. What is an LLM?
[00:02:05] Don Rosenthal: Good place to start. Um, let me start out by saying that, uh, LLMs have finally solved, and I don't think that's really an exaggeration.
[00:02:14] Don Rosenthal: They finally solved one of the longstanding foundational problems of natural language understanding. Understanding the user's intent. Um. What do I mean by that? Um, uh, any one of us who's used the recommender system for movies, TV, music, which pretty much all of us, um, we know how frustrating it can be to try to get the system to understand what we're, we're looking for.
[00:02:40] Don Rosenthal: These systems have all trained us to dumb down our queries. Uh, in order to have any chance of a successful retrieval, you can't talk to in the way you would to a friend or or to any other person. You can't, for example, say, Hey, um, I like all kinds of music. Uh, the genre is not [00:03:00] important, jazz, pop, classical, rock, even opera, as long as it's got a strong goosebump factor, put together a playlist for me with that kind of vibe for the next 30 minutes while I do chores.
[00:03:13] Don Rosenthal: But you can, in fact, say that to, uh, something that's got a large language model in it, like chat gbt. And go ahead and try it. When I did, I even asked it if it understood what I meant by goosebump factor, assuming I'd have to explain it, but it said, Sure, I know what it is and gave me a perfectly reasonable explanation and definition of it.
[00:03:36] Don Rosenthal: So... Why and how is it able to do that? Um, we can get into the technology a little bit later, but for the 3, 000 foot level to start with, the point is that through an absolutely enormous amount of training, um, these systems have internally created a highly nuanced model of language. Which they can [00:04:00] then use for the semantic understanding of language that is input to it, as well as to craft highly nuanced and natural sounding language responses.
[00:04:09] Don Rosenthal: Um, and it's important to, to, uh, to underscore that these are the two things that large language models do really well. Um, semantic understanding of language and its input to it, and Uh, highly nuanced and natural sounding land, which responses and yes, they hallucinate and they make up stuff out of thin air.
[00:04:30] Don Rosenthal: But the interesting thing is that they always seem to hallucinate within the correct context of your query. So, you know, if you ask them about strawberries, it might make stuff up about strawberries, but it's not going to make stuff up about fire engines. And as for the highly nuanced Natural sounding responses.
[00:04:53] Don Rosenthal: Um, just, uh, remember, for example, the response to the, uh, the query of generating instructions for [00:05:00] removing a peanut butter sandwich from a VCR written in the style of the ST James Bible, which kind of broke the Internet last November.
[00:05:10] Himakara Pieris: Take us inside an LLM. Um, what makes this technology so transformative, if you will?
[00:05:17] Don Rosenthal: Um, I'm not going to go into the, the technical details of how they work, but, um, it'd be great to be able to cover. Why they're so important and what has enabled them to become the agent of change in LLP to become so transformative. Um, and if you are interested in, in more details, the original paper from 2017 is attention is all you need.
[00:05:42] Don Rosenthal: It's all over the internet. You can find it easily. I'd also recommend, um, the Illustrated Transformer by Jay Alamar, A L A M m a R, who is well known for his, uh, incredible capability of helping you to easily understand complicated, [00:06:00] uh, concepts. And if you'd rather watch a video than, than read an explanation to check out his video.
[00:06:06] Don Rosenthal: The narrated transformer anyway, six transformers were able to. Help us leapfrog into the current generation of NLP tools. It's kind of important to first explain the state of the art just prior to their introduction, if that's okay. So, at that time, uh, NLP, the NLP world was using a set of technologies which were, uh, grouped together under the subfield of recurrent neural networks.
[00:06:34] Don Rosenthal: Um, not a very descriptive name, um, But the TLDR is that these technologies took the input sequence, any type of sequence, but let's say with language, so sequence of words in the sentence, um, and, uh, the RRN took the sequence of words, fed them in, in order, one at a time, the, quick, brown, fox, etc. [00:07:00] But they included a really novel component, which enabled feedback connections that allowed them to inject information from previous time time steps.
[00:07:09] Don Rosenthal: And this is what enabled them to capture contextual dependencies between words in a sentence instead of just having a look at one particular word. But so when quick was input, you get some feedback from the When brown was input, some feedback from the quick problem with this was, I mean, it was, it worked well for the time, but the problem was that the farther along in the sentence you got, the weaker the feedback was from the early previous steps.
[00:07:39] Don Rosenthal: So by the time you got to the end of the input sequence, um, the system may have been left with so little signal from the initial inputs. that they had very little effect on the evaluation of the sequence. So, uh, put that all together. Words that were closer to each other affected each other more than words that were farther apart in [00:08:00] trying to understand what the sentence meant.
[00:08:02] Don Rosenthal: And obviously that's a problem because language isn't constructed that way. Um, it also meant that sequences could only be evaluated. Sequentially, one at a time, and that made RNN processing really low. So the two stripes against RNNs, although they were really valuable for the time, was that they focused more on words that happened to be closer together in a sentence, and that they only processed sequentially, uh, one word at a time.
[00:08:31] Don Rosenthal: Um, so then along came transformers with a new idea, which was, let's present all of the words in the sequence to the transformer at once, all at the same time. Thank you. And this lets the system evaluate the connections between each word and every other word, um, regardless of where they show up in the sentence, um, and it can do this to figure out which words should pay particular attention to which other words.
[00:08:58] Don Rosenthal: And that's the intention part [00:09:00] of Attention is all you need. So no longer the words have to be close to each other to capture the contextual relevance between them. But it also meant, and this was the other key, uh, improvement. You could now evaluate all of the words in the input in parallel instead of analyzing one word at a time in order.
[00:09:18] Don Rosenthal: And I'm being a little bit hand wavy and imprecise. But, um, I'm trying to give you the intuition about how these work rather than, than teach you how to build one. Um, but at this point now, we could analyze semantic information equally between all combinations of words, no matter where they appeared in this, in this sequence, and we could do this in parallel.
[00:09:39] Don Rosenthal: So. NLP is solved, right? Um, unfortunately, not so fast. Uh, transformers, yes, they could analyze the semantic connections between all pairs of words, and yes, you could do this, you could do a lot of the work in parallel, um, but if you look a little closer, you see [00:10:00] that we've actually created a lot of extra computing for ourselves.
[00:10:04] Don Rosenthal: Transformers evaluate semantic connections between every word in the sequence with every other word in the sequence. Which means that as the sequence goes longer, uh, the number of pairs that you've got to analyze not only grows, but it grows incredibly quickly. Um, uh, a sense of two words. It's one communication path between one and two.
[00:10:26] Don Rosenthal: Uh, three words, three communications. One and two, two and three, and one and three. Uh, you get to 10 words, it's 45 communication paths. And, um, as the group size grows, The speed at which the number of communication pads grows, uh, um, that accelerates. And if you get to 10 24 people, the number of paths is, uh, over half 1,000,523, uh, 776 to be precise.
[00:10:57] Don Rosenthal: And I know that's correct 'cause I asked Chad g p t [00:11:00] to calculate it for me. So, um, So when you input, uh, a large document, your resume, uh, for example, you're inputting a very large sequence, which then requires a lot of computation. Um, and, um, you probably run into the term context window and that roughly maps to the size of the, the input, um, uh, uh, uh, size, the input length.
[00:11:25] Don Rosenthal: And um, now you kind of understand how, even though we can parallelize it, um, Give one core of each GPU each word, uh, to do it to evaluate in parallel with other words. Um, uh, even if you could do this, it requires, uh, a lot of GPUs. A lot of G TPUs to, to enable the parallel analysis. Uh, and, um, while the attention mechanism, um, has enabled some incredible advances in N L P, you never get something, uh, for nothing.
[00:11:59] Don Rosenthal: So [00:12:00] we've taken a big, taken a big step. We've got new problems to solve, but, um, we've getting, we've gotten a, we've gotten a lot of value out of this new, new approach.
[00:12:10] Himakara Pieris: , so we talked about what makes. LLMs to transformative. And we are seeing a lot of LLMs coming to market. Some of them are from big tech companies like OpenAI, Azure, from Microsoft.
[00:12:22] Himakara Pieris: And then we have Titan from Amazon and Palm from Google. Some of them are open source, Eleuther, Falcon, et cetera. And we're also seeing large language models coming to market from. Not so big tech companies and also not so very tech companies like, um, from the not very big company standpoint, Databricks, Dali, Dialpad, GPT from Dialpad and Bloomberg GPT where they built something from ground up.
[00:12:48] Himakara Pieris: So could you talk us through what it takes to build an LLM and also who should consider building an LLM on their own?
[00:12:58] Don Rosenthal: That's a critically important [00:13:00] question. I'm really glad you asked this. To train up an LLM from scratch requires a lot of compute resources. Which means you need a lot of money. Um, Microsoft invested 12 billion or more in open AI.
[00:13:16] Don Rosenthal: I'm pretty sure that they're not hiring 12 billion worth of AI researchers. I'm guessing that much of the money is covering the ongoing costs of compute resources. And in addition, you need a lot of training data and that represents a lot of time you've got to do a lot of, of, of training, even though you don't have to do with what you did in the old days of discriminative AI, you don't have to label it.
[00:13:45] Don Rosenthal: It's still, if you're taking the full, full corpus of the Internet and trading on that, it's going to take a lot of time to do that. So, uh, if you have one takeaway from this conversation today, I hope it's that, um, you should [00:14:00] lead the development of LLMs, uh, to the well heeled, uh, big tech companies who have a lot of money and a lot of people, and many of them also design and build and own their own compute resources.
[00:14:14] Don Rosenthal: Or leave it to the open source community, which at least Has a lot of people, uh, it's really hard to overestimate the amount of work, uh, the cost and the time required to develop these on your own. Plus, once you've done the initial training of the model, you then need a very large user community to evaluate what's being generated.
[00:14:35] Don Rosenthal: So people can say, Oh, we like the way it responds here, but we don't like the way it responds here. So those preferences can be fed back into the training of the models. Uh, and it creates, um, the naturalness of the responses today. That all has to be done manually, which again takes a lot of money and a lot of time.[00:15:00]
[00:15:00] Don Rosenthal: That makes, um, that makes a lot of sense. So the alternate approach to building your own LLM, um, is the closest alternate is fine tuning your own. And matter of fact, DALI and, and, um, Dialpad GPD seems to be fine tuned large language models. What are your thoughts on fine tuning? Great. Another really, really important question.
[00:15:23] Don Rosenthal: Um, first off, it's a, it's a term that's used very imprecisely these days. So think of fine tuning in terms of transfer learning. If you have trained a system, how to process English, it should be easier to train it, to process a second related language. You don't have to start from scratch. If you've trained a model, not allowed to language model necessary, but if you train a model to play one card game, It already understands what face cards are, what suits are, etc.
[00:15:51] Don Rosenthal: It's easier to train it on, um, a second game. But, um, if there's a second thing that I, that I hope you take away [00:16:00] from this conversation, it should be to, uh, uh, try to avoid fine tuning as well. Uh, the process of fine tuning is exactly the same process as that of the initial training. But at a somewhat smaller scale, you're retraining a previously trained model.
[00:16:17] Don Rosenthal: Um, at a smaller scale, yes, but when your initial size is enormous, this doesn't necessarily get you out of the problem of needing a lot of time. And money, um, remember that what fine tuning actually creates is a new variant, a new variation of the original next word, but it's basically a continuation of the, of the model training.
[00:16:41] Don Rosenthal: And in addition, there's some potential downsides, such as there's a thing called catastrophic forgetting, which we've known about in deep learning since the early days. And this is where the changes made in the weights of a neural network, stable neural network that's been trained when you teach it, [00:17:00] when you train it on a new task, can affect its ability to perform on tasks it was previously trained on.
[00:17:09] Don Rosenthal: But I also want to underscore this is not the city that fine tuning. is a bad, uh, idea for every situation. For example, a really reasonable, uh, use case for fine tuning an LLM is if you'll be operating in a domain where there is a very specialized vocabulary and a vocabulary that's reasonably stable, such as, uh, medicine, uh, or pharmacy.
[00:17:36] Don Rosenthal: So for many practical reasons, A lot of the companies, especially small to medium sized companies, are now going to build their own LLM and fine tune their own LLM. So what are some other mechanisms available? for these product teams to incorporate LLM features into their products. Great, great, great, great.
[00:17:56] Don Rosenthal: So, um, many of the capabilities that you might think [00:18:00] require fine tuning, uh, can be addressed by using other techniques, um, in systems. external to the LLM. So instead of trying to teach one monolithic model to do everything, um, connected to models, uh, out or, or other systems, even conventional systems, external to the LLMs.
[00:18:24] Don Rosenthal: One of those, which is getting a lot of attention these days is something called retrieval augmented, uh, generation. And. I'm happy to go into that particular example. If that makes sense. Yeah, that would make sense. Great. Great. Um, so, uh, let's work through an example. Um, let's say you're a law firm and you've got a huge amount of unstructured data.
[00:18:47] Don Rosenthal: You've got All of the contracts, all the litigation, all the presentations, you've got emails, you've got slack conversations, all of that, that have been generated over the life of the company. Um, [00:19:00] and you can't put these in a, an SQL database. They're, they're unstructured. So you'd like to be able to refer back to these specific documents that they're really useful.
[00:19:09] Don Rosenthal: Um, but you'd like to be able to do it without having to manually sort through. Mountains of documents, either physical documents or documents stored in the cloud, you want to be able to automate the retrieval of the appropriate documents in response to a query. So, um, let's say at first you think, ah, I'm going to fine tune my LLM on all of these documents, and that will eventually encode all of this information that all the information that's in the documents into the weights Of your neural net and the weights of your LLM, and that would allow you to use one single end to end model to solve the problem.
[00:19:45] Don Rosenthal: An interesting idea, but there are a few issues. Money and time, like we talked about before. Uh, hallucinations, which we can talk about later. But since you're working with one monolithic model, these are hallucinations that can't be caught [00:20:00] prior to sending the reply to the user. But most importantly, um, the thing that's really gonna, uh, hang you up here is that you'll need to constantly continue to fine tune your LLM as new documents are generated.
[00:20:16] Don Rosenthal: Because if the information is not encoded into the weights of the network, it has no idea that it never exists. I mean, I'm sure those of you who have used, uh, I've played around with, with, uh, chatbots and stuff. Now, um, you'll ask it a question about, uh, uh, you know, what's the, what was the score in the last, uh, Niners game?
[00:20:37] Don Rosenthal: It'll have no idea and it'll answer something like, you know, my training ended in. September of 21. I can't tell you anything about that that information. So, um, this particular use case, um, um, is really well suited for retrieval of automated augmented generation. Um, [00:21:00] the way this works. And again, I'll start at a high level.
[00:21:04] Don Rosenthal: We can get into more details. You feed your documents into an embedding engine, um, and there are plenty out there. Uh, there's good ones that are even open source like Pinecone. That encodes the semantic information of each document into an embedding, sometimes called a vector, which can then be stored in a vector database.
[00:21:24] Don Rosenthal: And these vectors incorporate, incorporate learned semantic features. Which enabled the ability to compare the semantics similar similarities of two documents with a simple arithmetic comparison. You remember back when we were talking about, uh, attention and you were comparing, um, semantic correlations between two words in a sentence.
[00:21:48] Don Rosenthal: This is, uh, this is something something akin to that. It's not exactly the same thing, but you can think of it in in the same way. Um, so you can ask, for example, uh, what VC or [00:22:00] startup contracts have we written for seed round funding using cryptocurrency and which currencies were used in each of them. And, um, you give that query to the LLM, it analysis, it analyzes it, uh, to find the important components in it.
[00:22:16] Don Rosenthal: funding, contracts, cryptocurrency, etc. It won't be just keywords, but, um, let's, let's just give you an idea of what it's talking about. And it, um, it generates an embedding of the analysis of the query with the same Embedding generator that's been used on your documents by, um, using one of these, um, uh, um, systems that allow you to build pipelines, uh, of, of systems.
[00:22:47] Don Rosenthal: Things can, an LLM can call an external system, like the embedding generator. And then once you've got the embedding, you can feed it into the vector database. And then that gets compared with the encoded query. [00:23:00] Uh, and it finds the documents that are. Uh, related to that query semantically, and that retrieval system grabs the those documents, the ones that are most closely related to the query, it sends that back to the LLM, which generates a response to the user and maybe attaches the docs or the links to them.
[00:23:23] Don Rosenthal: And. Important point here. If tomorrow a new funding document for crypto seed rounds is generated instead of having to re fine tune the model with a new document, you just feed the new document into the embedding generator, store it in the vector database, and you're, you're completely, uh, up to date. And as I said, there are pipeline, Pipeline platforms like lang chain that make it easy to connect together the external systems, uh, and the LLM.
[00:23:56] Don Rosenthal: I went through that kind of, um, quickly, but I hope that that was at least a [00:24:00] good introduction, uh, to using external systems, uh, as opposed to fine tuning.
[00:24:06] Himakara Pieris: Yeah, absolutely. Um, so the gist of that is using, um, written work meditation, you have the capability to tap into data that's, that's not in the weights.
[00:24:18] Himakara Pieris: Of the model. And this is a great way to tap into your internal systems or other parts of the ecosystem within your application and open up capabilities to your users. And it's a much easier, much faster thing to do compared to fine tuning, etc. And also, it's a way to keep your information current without having to fine tune every time.
[00:24:38] Don Rosenthal: Exactly, exactly. And a rule of thumb is remember what LLMs are good for. They're really good at understanding the query of the user. And they're really good at, um, generating highly new, nuanced, natural sounding language. All the other stuff in between, see if you can ship that out to external systems like, uh, SQL databases, vector [00:25:00] databases, math lab, uh, you know, uh, search engines, whatever, whatever is needed in your particular use case.
[00:25:08] Himakara Pieris: Great. So we talked about what LLMs are, what's inside an LLM. ways to incorporate LLM capabilities in your product. Let's talk about risks and challenges because if you are to go and pitch using these capabilities in your product, I think it's very important to understand what you're getting yourself into, right?
[00:25:29] Himakara Pieris: Exactly. Let's talk a bit about what kind of risks are there, what kind of challenges are there, and how you can plan on mitigating them as well.
[00:25:36] Don Rosenthal: Okay, so this is my personal analysis. Your mileage may vary, but there's a lot of other folks in the industry, um, that think along the same lines. Um, and, um, there are, in my mind, there are four key challenges.
[00:25:53] Don Rosenthal: Uh, for those of us who want to incorporate LLMs into their products, um, um, and as Hema [00:26:00] said at the beginning, this is an amazing time to be an AI PM because you can really quickly and really easily prototype systems, uh, that demonstrate their potential use, um, to your end users, to your company, et cetera.
[00:26:14] Don Rosenthal: But the real challenge is the next step after prototyping. getting them ready for real world use and then scaling. So let me just start by listing these top four challenges. The first one is factuality and groundedness, um, sometimes called hallucination. Second is the high cost of inference servability.
[00:26:37] Don Rosenthal: It costs a lot to train these models. It also costs a lot to run data through them and get an answer. Um, uh, the third one is implementing guardrails against inappropriate content, which we sometimes refer to as content moderation. And the fourth one, um, is, is that this, these systems [00:27:00] represent. a major shift in product development, uh, from a series of short, plannable, progressive steps to loops of experimental, uh, experimentation, trial and error, et cetera, because these systems are non deterministic.
[00:27:17] Don Rosenthal: Um, and it's critically important to be aware of these issues and go into a project. With your eyes open, um, and have buy-in from all, uh, of the stakeholders because these challenges are very likely to manifest as, uh, increased costs, uh, and increased time to market compared to what your stakeholders are used to.
[00:27:40] Don Rosenthal: So make sure these are discussed and that you derive the consensus from the, from the very beginning. So if it's okay, I'm going to start with the brave new world of non deterministic systems because folks are likely likely familiar with the other three. Uh, but this one is the least talked [00:28:00] about and the least acknowledged, and maybe the first time some folks have, have heard, I've heard it, uh, brought up.
[00:28:08] Don Rosenthal: Um, so, um, uh, so the first thing to come to grips with is that, um, productizing large language models does not neatly fit into our, our tried and true software engineering practices. Uh, LLMs are non deterministic. Instead of being able to predict the expected output for given out input, um, uh, uh, and even, um, you know, the, the, the exact same inputs can even produce, uh, different outputs.
[00:28:42] Don Rosenthal: Uh, you, you, um, this is something that we're not, we're not used to in developing products. And, um, in addition to their, the outputs not being predictable, the evaluation of these systems is more art than science. So, Um, [00:29:00] there are academic benchmarks, and the academic benchmarks work really well for academic pace papers where you're trying to prove progress against the last published state of the art architecture.
[00:29:12] Don Rosenthal: We beat this by, you know, X amount of X percent, but these academic benchmarks don't add very much for Evaluating user experience, um, which is something that's critical for us as product managers. Um, and evaluating the user experience these days, um, typically involves subjective human rating, which is lengthy and expensive.
[00:29:39] Don Rosenthal: So it can be hard to one measure progress on your project and also to objectively. Define success. Um, again, that was kind of high level. Let me go into a little bit more detail. Um, we're used to being able to, uh, as product managers uncover a significant business [00:30:00] problem, generate the requirements, then developers code up the solution and Q.
[00:30:05] Don Rosenthal: A. Test it to uncover any bugs. Developers Fix the bugs, you QA again, rinse and repeat, and finally you get to a point where for any given input, you can predict what the output will be, and if the output isn't that, it's a bug, um, and you can generate regression tests to ensure that any new changes, uh, haven't broken old capabilities, and the entire modern discipline of software engineering Is built around this type of sequence, but with deep neural nets in general, um, and especially with the enormous models like LLMs, this goes almost completely out of the window.
[00:30:48] Don Rosenthal: Um, so let's look at a completely new environment in the lab where, um, the LLMs are being developed research. And again, this is true for any deep network, but let's stick with these [00:31:00] really, really enormous ones. Researchers laboriously train these models over and over and over again. Each time of the random starting point and they keep looking for generating one that they That they like depending on what particular metrics they're using at that point You start doing experiments and it's really Empirical it's like a biologist With a finding a discovering a new organism, you poke it, you prod it to see what it does, and it doesn't do and how it behaves.
[00:31:34] Don Rosenthal: Um, I'm not exaggerating here. This is really the way you get to know your new model. And in addition, you are likely to discover, um, what are called emergent behaviors. These are capabilities that through experimentation, you discover even though you never explicitly trained the model to do that. Okay, then [00:32:00] you step back and assess what you've got and see how useful it is for the things you hoped it would be useful for and specifying what on effect unexpected things it could be good for that you hadn't anticipated.
[00:32:14] Don Rosenthal: And there's no debugging in the classic sense, uh, because there's no human readable code, right? This is all machine learning. This is all, um, uh, generated, uh, on its own. And, uh, the deep dark secrets are hidden in the weights and biases of the model. It's all about things that the system is, has learned from analyzing, uh, the training data.
[00:32:39] Don Rosenthal: Um, and forget regression testing because at least in the classic sense. Because these models are not not deterministic for any. Input. You cannot predict the output. So the best you could do is evaluate if the output is reasonable, and that takes a lot of time and more [00:33:00] money than the stakeholders might have expected.
[00:33:03] Don Rosenthal: So even though you're hopefully not the ones building these models from scratch, when you're working on the addition of any LLM to a product, or even just evaluating which one out of all the choices we have these days will be the best for your application, These are likely the things that you have to be prepared to deal with, and I want to make a really important important point.
[00:33:25] Don Rosenthal: Now, after painting that, uh, picture of doom and gloom, please don't misunderstand. I'm not trying to scare you away from using this technology. You can build. Really, really important, impressive products with them. There will likely be very few opportunities in your professional lifetime, uh, to apply such transformational technologies.
[00:33:48] Don Rosenthal: Uh, thank you again for the, for the pun, um, but
[00:33:52] Don Rosenthal: please, please, please do go out and do come up with unique and exciting, uh, important new applications, build stuff [00:34:00] that solves important problems we couldn't even try to address previously. I just want you to be sure that, uh, you're going into this with your eyes open and that you've prepared your stakeholders properly.
[00:34:13] Don Rosenthal: There, um, there are a lot of successful applications that have been built with these LLMs and a lot of the pioneers have discovered all the pitfalls and where all the dragons hide so that we can We can avoid them.
[00:34:31] Himakara Pieris: Very good. I think it's it seems to be also a function of picking the use case as well. Correct. You know, you have in some use cases, you have human surprises who could be built into the loop and you know, those would be the low hanging fruit. Could you talk a bit about how to think about picking the right use cases as well?
[00:34:50] Don Rosenthal: Yeah, that's a really important, important point. So all of the low hanging fruit, um, yeah. Uh, the things that you could build just with one single [00:35:00] monolithic, uh, uh, LLM, those have all been been gobbled up, um, and we're looking at at more, uh, uh, we're looking at more complicated problems now and more interesting problems, more valuable problems to solve.
[00:35:15] Don Rosenthal: Um, and one good way to evaluate the use case that you're looking at as was just mentioned is if your use case. Already has, um, built in human, uh, supervision or evaluation or, uh, for example, why was, why was copywriting one of the first, um, one of the first, uh, low hanging fruit use cases that was gobbled up?
[00:35:42] Don Rosenthal: Because built into the workflow of copywriting is a human editor who always reviews, corrects, sends back to the junior copywriter. Um, uh, you know, all of the copy that's been generated. [00:36:00] Before it goes out for publication, um, uh, the, um, uh, the senior editor comes up with a, with something that needs to be written, sends it off to a copywriter, the copywriter comes up with a version, the editor marks it up, sends it back, they iterate, eventually they get one that is, uh, yeah, that passes, uh, muster for the, uh, The editor, and, um, it goes to, to publication.
[00:36:29] Don Rosenthal: Um, if you think in terms of that, you'll understand that, okay, you don't need to get rid of the junior copywriters. They can use these LLMs to help them, but if you have an application where there is something Um, like a senior editor already in the workflow. Um, this is a really good, uh, application to start with.
[00:36:54] Don Rosenthal: You've got somebody there that can check for hallucinations, for inappropriate, uh, [00:37:00] uh, content. Et cetera, et cetera. Great. So you are a product leader with decades of experience in AI. Um, in addition to picking the right use case, what other advice would you have for PMs who are interested in getting into AI?
[00:37:16] Don Rosenthal: Um, well, uh, This may not be for for everybody, but I'm I'm a PM that really likes to get his hands dirty. I like to stay really technical. I don't have to be good enough to code these things, but I want to understand in some depth the, um, the technical details. That's important for a couple of reasons. Um, one of them is Uh, you want to be able to, um, properly represent what can be built in this to that can be built with this technology to your end users.
[00:37:51] Don Rosenthal: So you're not just making up stuff and you want to be able to have discussions with your technical teams. Um, in a way that makes [00:38:00] sense for them. It doesn't waste their time when you get new ideas. Uh, Hey, we might be able to use it. Uh, use LLMs for this particular application. Um, here's some more details.
[00:38:13] Don Rosenthal: Uh, how does it Give it a sanity test to check. Um, uh, does this make sense? Is the technology here already? Is there anything that we need actually, um, uh, fundamental research? Are there any new ideas that we have to develop in order to make this possible? Or is this something that with, uh, with some hard Working and good, good, uh, attention to detail we can we can build great.
[00:38:43] Himakara Pieris: I want to transition and take some questions from our audience. So there is a question about. Um, LLMs in healthcare domain specifically, are there any specific use cases in healthcare where LLMs could be used effectively, especially in healthcare diagnostics? [00:39:00] I love this question. Uh, my second company was in, uh, in, in healthcare.
[00:39:06] Don Rosenthal: Um, and yes, absolutely. Um, uh, you know, um, medicine is fundamentally a, uh, a multimodal field. You have images, you have Um, uh, you have, uh, text, et cetera. Let's stay on the text for, um, the certain for the time being. So, for example, um, we've we've seen an example that I that I discussed already that when you have a lot of unstructured text data.
[00:39:38] Don Rosenthal: This is where things like LLMs and the stuff of processors shine. Do we have that in, uh, in, in medicine? Um, absolutely. Uh, every patient's chart is now in an ERM, an electronic, uh, EMR, electronic medical records system. Most of [00:40:00] the good information has been written into the comments, um, and there's no way to automatically retrieve it.
[00:40:07] Don Rosenthal: Just like, um, there was no way to automatically retrieve, uh, the legal documents. If it's possible to get access to that and store that in, um, a vector database. Um, so that you could when you're working with a particular patient on a particular problem, you could retrieve that information for them. Um, that would be really, really useful because a lot of that information is just stuck in some text field in some database and unusable.
[00:40:39] Don Rosenthal: Uh, not accessible to the to the medical medical staff and the clinical staff. Um, it's that's also would be extremely useful if you're doing longitudinal, longitudinal studies. Um, or if you've got, uh, um, a rash of new cases that all seem to be, uh, [00:41:00] related somehow, but you don't know what is the common factor for it.
[00:41:04] Don Rosenthal: You can get the data, uh, if you have, um, uh, uh, uh, stored in embeddings, um, all of this, uh, comment data, uh, for each of the patients, you can search through that and try and, uh, you know, some common threads. And again, this is all, uh, text data. This is all language data. So this is a really good application for it.
[00:41:30] Don Rosenthal: If you're talking about specifically, um, uh, helping, uh, to diagnose based on x rays, there are good systems in computer vision that can do that. And they generate data, but they may be a bit difficult to use. And it would be really nice if they had, uh, conversational interfaces with them. So that the radiologists could talk to their data or could talk about the data in the same way they talk to their colleagues about it, instead of [00:42:00] having to learn SQL or or come up with, uh, you know, understand the limits of some other, uh, interface they have for retrieving, uh, this data.
[00:42:09] Don Rosenthal: Um, that's a really short answer. I would love to, uh, talk more about this if you're interested. But yes, I think, uh, the medical field is a really, really good opportunity for LLMs.
[00:42:24] Himakara Pieris: And we have another great question about QA. So since there wouldn't be any deterministic output, does it mean there would be no QA involved with LLMs?
[00:42:34] Don Rosenthal: Well, I, I hope not. Um, what it means is that it's gotta be a different type of QA. Uh, we've got, uh, people that are trained up and are really amazing. Um, uh, uh, For, um, products, software products that that were were, uh, that go through Q. A. Today. Um, they may or may not be the people that, um, we would use for this type of Q.
[00:42:58] Don Rosenthal: A. But remember, there are [00:43:00] always going to be systems outside of the L. L. M. S. That will be coded up by your, uh, by your, um, uh, colleagues in whatever, in C and Python, whatever, uh, that will still have to go through standard QA, um, the, um, the problem, it's, it's still open questions about the right ways to do QA.
[00:43:24] Don Rosenthal: For the output of L. L. N. S. Uh, open a I has, uh, uh, been using, uh, reinforcement learning from human feedback. Um, it's been around for a while, but they're showing how valuable it is. But that is basically getting a lot of human Raiders. reading through the outputs and saying this one's good, this one's bad, or, or, or, um, marking them, uh, grading them on a particular scale, then that information goes back to reinforce the training of the model, um, to try and, and have it, um, tend [00:44:00] more toward the, uh, the types of, uh, outputs that, um, that got good, good scores.
[00:44:12] Himakara Pieris: We have another question here about, um, data sourcing. Um, what are the best ways to get data to tune LLMs for SMS phishing attacks into that one? But, uh, it's actually, I think the larger question of how to source data for LLMs is, I think, particularly an interesting one. So, um, the. Go on a short side trip.
[00:44:36] Don Rosenthal: This is a really great question. I don't know enough about that field to really give you a hard and fast answer. But, um, the interesting thing about working with data for these systems. Is that unlike in discriminative models, like for computer vision classification, for example, where you have to generate a bunch of data, you have to clean it up.
[00:44:58] Don Rosenthal: Then you have to [00:45:00] label it. So the, um, the, the model can learn whether it's doing a good job classifying it or not, uh, language data for LLMs, uh, is self supervised. What does that mean? Um, you grab, uh, data. Um, you grab text data from the Internet, from Wikipedia, whatever. And you give it to it, uh, one, one word at a time, for example.
[00:45:28] Don Rosenthal: Um, uh, you give it the first word in a sentence, and it's trying to, going to try and predict what the second one is. It's not going to do a very good job, but that's okay. You go on to the next one. Um, what's the, what's the third word? What's the fourth word? And little by little. After, um, an astonishing amount of training.
[00:45:48] Don Rosenthal: It's able to home in on the connections between words, even if those words are not ones that are right next to it, and make a good prediction about a reasonable, um, [00:46:00] a reasonable, uh, um, prediction for the next word. You can also do this for full sentences, give it a paragraph with a sentence missing, try and fill in the, the, the, um, the sentence, et cetera, et cetera.
[00:46:14] Don Rosenthal: You still need a lot of data. Um, and you still need to clean that data, but there are really good sources for it. And you don't have to spend the time and money to label it. So, uh, if you're in the medical field, for example, um, you can, if you get the right to use them, you can go, you can use medical texts, um, to, uh, fine tune.
[00:46:41] Don Rosenthal: Uh, for the particular vocabulary, uh, for medicine and use it by, um, in the, in the cell, in the self supervised, uh, uh, ways of, uh, training.
[00:46:55] Himakara Pieris: Great. Uh, we are a few minutes over, so let's do like one final question before we [00:47:00] wrap things up. Um, so the last question is the biggest challenge for me to advocate for AI in our product is quality, more specifically uncertainty about.
[00:47:08] Himakara Pieris: Um, quality of outputs. What are some ways to address quality and predictability concerns with LLM?
[00:47:16] Don Rosenthal: Okay. Um, if I understand the question correctly, it's, it's a really good question. We know we have these, these problems. Um, we know we have, uh, we want to solve this really important problem. How do we get it to how we get it to market today?
[00:47:32] Don Rosenthal: Today? It's a lot of manual effort. Um, uh, you can, um, find, uh, workflows where there are people in the workflow to prevent, uh, inappropriate content from getting out and such, or to catch hallucinations. You'd like to be able to, uh, generate, um, text where you don't have to worry about that. That's currently beyond the state of the art, [00:48:00] but there's a ton of really good research going on.
[00:48:03] Don Rosenthal: Um, for example, uh, just yesterday, um, from, uh, DeepMind, um, there was a paper about, um, a, uh, a new model. That could be used to evaluate the quality of language generated from an LLM, and it showed that in many cases, not all cases yet, but in many cases, it was as it generated, uh, its evaluations were as good or better.
[00:48:34] Don Rosenthal: Then what people came up with, um, and, um, haven't read the whole paper. So don't ask me how they how they decided that if you're doing human evaluation, who evaluates whether it's better than human evaluation. I'll read the paper. I'll give you an answer. Um, but, um, for now, it's a lot of manual work.
[00:48:53] Don Rosenthal: There's a lot of really, really important research being done. Keep your fingers crossed. Going forward, there'll either be new [00:49:00] architectures or new models that'll help us get out of this manual mode. Great question, though.
[00:49:05] Himakara Pieris: Thank you, Don. Thank you so much for coming on the pod today.
[00:49:08] Himakara Pieris: We'll share a recording of the video and also links to the things that Don mentioned in this. And a big thank you to also everyone who signed up and shared this with the network. If you found this interesting, please go to www. smartproducts. show to listen to other episodes as well. We'll publish this one there.
[00:49:28] Himakara Pieris: Um, there as well. It's available on Apple Podcasts, Google, Spotify, and wherever you listen to your podcasts. Don, thanks again, and uh, thank you everyone.
[00:49:38] (Outro)
[00:49:38]
[00:49:41] Himakara Pieris: Smart products is brought to you by hydra.ai. Hydra helps product teams explore how they can introduce AI powered features to their products and deliver unique customer value. Learn more at www.hydra.ai.
-
I’m excited to share this conversation with Khrystyna Sosiak. Khrystyna is a product manager at TomTom. Before that, she was a lead AI coach at Intel and a senior data scientist at mBank. During this conversation, Khrystyna shared her approach to navigating the complex landscape of AI projects, which includes investing in research, strategically placing bets, fostering stakeholder support, and embracing transparency.
Links
Khrystyna On LinkedIn
Transcript
[00:00:13] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we recognize, celebrate and learn from industry leaders who are solving real-world problems. Using AI.
[00:00:25] Himakara Pieris: Khrystyna welcome to smart products.
[00:00:27] Khrystyna Sosiak: Thank you. I'm super excited to be here. Thank you for having me.
[00:00:30] Himakara Pieris: To start things off, could you tell us a bit about your background, um, what kind of environments that you've worked in, and also what kind of AI projects that, that you've been part of?
[00:00:39] Khrystyna Sosiak: Yes. So, uh, currently I'm a product manager at TomTom. I'm working on the external developer experience and uh, and analytics and billing topics. And in past I was working on the machine learning operations platforms and, uh, in my previous experience was a data scientist. So I was actually working with, [00:01:00] uh, with machine learning and with artificial intelligence before I moved into product.
[00:01:05] Himakara Pieris: What would be a good example of an AI project that you worked on?
[00:01:10] Khrystyna Sosiak: Probably one of the most, Exciting and interesting, , products that we've been working on that was very powerful is, , understanding the customer's behavior and, and the patterns.
[00:01:22] Khrystyna Sosiak: And then based on that ing uh, the right products. So I was working in banks, so we would analyze. All the data that we can find about our customers, right, of course, with two G D P R and making sure that we only use the right data, but, and then making sure that all the communication that goes to the customers is the right communication about the right products and in the right way.
[00:01:46] Khrystyna Sosiak: So really understanding the customer needs and, uh, the stage of the customer life and saying that's, that's what the customer need at this point, and that's how we. Understand that and how we can communicate and [00:02:00] sell it to the customers. So it's not about only making money, but it's understanding how we can actually.
[00:02:06] Khrystyna Sosiak: Go through this journey of life with the customer and supporting them. So, and understanding that by the data that they're generating and by the insights that we can find in this data. And sometimes, you know, and like data that you have like that generated by your transactions and by your history, like, It's a really specific data that show a lot about the person that probably some people even don't know about themselves.
[00:02:33] Khrystyna Sosiak: And the real goal is how we can use it for the benefit of the customer and not to harm the customer, right? And, um, we really change the way that we approach them. Uh, we approached the, the marketing communication with the customers, what was very interesting and transform transformational to see how very old fashioned organization would really move in direction into the [00:03:00] AI and making sure that all the decisions and the marketing strategies are powered by ai.
[00:03:06] Khrystyna Sosiak: So yeah, that was very interesting. It took us a long time. We made a lot of mistakes on the way, but it was a super interesting learning experience.
[00:03:17] Himakara Pieris: If I take a step back, so we're talking about mbank a consumer banking operation and reaching out the customers at the right time is something very important to, to become that part of the customer's daily life or, or their journey.
[00:03:32] Himakara Pieris: How was that done before and what point. Did the bank decide to explore AI as a possible, , solution to, possible tool to improve the, communications with the customers?
[00:03:46] Khrystyna Sosiak: I think the turning point was understanding that where the, you know, not only trends, but like the industry goals, right? And really AI powers the financial industry and the financial industry thing [00:04:00] in general.
[00:04:00] Khrystyna Sosiak: It's been very innovative in, uh, Trying to adopt the new technology and trying to make sure that the customers get the best experience before it was all triggered by the events. So you can imagine, I mean, it's still used widely, right? And when we talk about recommendation systems and like how the communication is done, right?
[00:04:20] Khrystyna Sosiak: You open the webpage, you open the app, and you, you scroll through some pages, you know about the credit card, for example, and then, Next day you would receive the email saying, Hey, here's the discount. Or in today, someone would call and say, Hey, we saw that you are interested in a credit card. Do you want to order the credit card?
[00:04:41] Khrystyna Sosiak: We have this discount for you. And usually it was triggered by one event, right? Or the, the sequence of events. But it's also very event triggering, right? So you only can. You only can base your recommendations on what customer actually does on the webpage. You don't really go into details [00:05:00] of like, okay, what are the factors about the customers that can affect that and what is actually the things that they need?
[00:05:07] Khrystyna Sosiak: It's, um, so yeah, it was something that was used. For years and, uh, it worked. You know, there was some success rates there, so I cannot say it didn't work, but we know that moving forward expectations of the customers are higher because when we live in the era of ai, when you have, you know, Netflix and Facebook with the recommendation title, your.
[00:05:30] Khrystyna Sosiak: You know, reactions and like what you see, what you like, what you don't like. Really we need to be there as well. And just saying you clicked on something and that's why we think it's could be interesting for you. It's not good enough anymore.
[00:05:45] Himakara Pieris: Sounds, like the previous, , approach for doing this is purely driven by specific events.
[00:05:51] Himakara Pieris: You have a rule-based system. If you click on this page, then you must be interested in this product. Let's unleash all the marketing communication , to sell that product [00:06:00] towards you. Whereas now, , the idea is we can possibly make this better by using ai. , To make sure that we are making more personalized and more relevant recommendations to the customer.
[00:06:10] Himakara Pieris: And by doing that, you improve the customer's experience and you would also improve the sort of the clickthroughs or, or, or signups for that product that you're, that you're positioning for the customer. , so when you start there, so it sounds like it started more with a. With an experimental approach.
[00:06:26] Himakara Pieris: Is that right where you're saying, okay, we have this way, we are doing things now we have all these new tools that are coming to the market, coming to the world. Let's pick them up and see whether we can move the needle, , with these tools rather than the, the method that we are doing now, which is our baseline.
[00:06:42] Himakara Pieris: Is that a fair assessment?
[00:06:44] Khrystyna Sosiak: It's for assessment and to be honest, it's for assessment not only about this project and not only about this experience, about almost all of the experiences that I had with the big companies or even small companies trying to get into the AI and trying, [00:07:00] you know, if it's. Not like the, the companies that actually build it, right?
[00:07:03] Khrystyna Sosiak: That they're trying to adopt it. It's really about, we have some data, we see the trends, we see that our competitors are using it, so how can we benefit from it? And I can see very often, like also talking to my colleagues and to my friends that there's very. There's a lot of companies that would hire like, uh, machine learning or, uh, engineer or data scientists say, that's the data we have.
[00:07:26] Khrystyna Sosiak: We have no idea what we can do with it. You know, try to figure something out. And I think sometimes there is some wrong expectations about Right. What we can do and what we cannot do. So yeah, it's all started like that, right? We have the data. Here's the set of the business. Problems that we have, and then let's iterate.
[00:07:46] Khrystyna Sosiak: Let's see what gonna work, what not gonna work. And a lot of things fails before something starts working. Right. And I think that's a learning experience that once you, you cannot, like, you cannot get there. [00:08:00] If you, they make mistakes and learn on a way, because then your experience and your success is much more meaningful because you actually understand what you've done and how you've done it and why you made those informed decisions about some steps of the machine learning process that we have.
[00:08:18] Khrystyna Sosiak: And that was very important also for the data scientist and for the product manager to understand better how this industry works. And how building these products are different and why they're failing.
[00:08:33] Himakara Pieris: So I imagine you're in a conference room on, there are two whiteboards on either side. On one whiteboard you have a whole set of business priorities and all the other side you have a catalog of all the data services that's available to you.
[00:08:45] Himakara Pieris: And then in the middle you have a data scientist and a machine learning engineer with a, , with a, with a toolkit, right? So, so you're running through a bunch of experiments using the toolkit you have and the data you have to see where you can impact, , the business priorities that you've identified.
[00:08:59] Himakara Pieris: Is that a good [00:09:00] way to look
[00:09:00] Himakara Pieris: at it?
[00:09:01] Khrystyna Sosiak: Yeah, it was like that definitely. It was also like someone from the business coming saying, that's the problem we have, we need support and we need to, to sort it out and the help. Uh, right. It was also like, Hey, that's the data we never used.
[00:09:16] Khrystyna Sosiak: Maybe we can, there's some opportunities in this data that not discovered that can actually bring the value to the co to the, uh, company, or it's for. Selling more or automation, right. So it's really different range of how those initiatives can start and they usually start from very different directions, right?
[00:09:37] Khrystyna Sosiak: But I think one is definitely you have a set of priorities and business priorities that you want to achieve, and then you ask yourself, right, that's where my company wants to go and. That's like us, and that's like what we have as a asset is data. How can we help the company to get where they want to be?
[00:09:54] Khrystyna Sosiak: You know? And the goal of the company could be very far from like adopting ai, [00:10:00] right? It could be like, you know, growing the revenue or going to the end number of customers this year. And then you try to. Understand what actually this goal means and how you can operationalize it, and how you can use the assets that you have in a team.
[00:10:16] Khrystyna Sosiak: And that's usually the people. And you need to have the right people. And that's very important to have the right set of people, but also to have the right data, right? And then it's this combination. You can, you can deliver.
[00:10:31] Himakara Pieris: So you're starting with a problem definition or series of problem definitions. Right? What was the next step for you, in this type of project? , is it a building, a prototype or where do you go
[00:10:41] Himakara Pieris: from there?
[00:10:43] Khrystyna Sosiak: So once we have the business, business requirements, right, and understanding the questions, and I think one of the problems that's sometimes. Uh, machine learning projects fails because we ask the wrong set of questions and, uh, we then, then [00:11:00] we start to tie to those questions, right?
[00:11:02] Khrystyna Sosiak: And, uh, yeah, so you would set the, the expectations and the business goals or the problem that you want to solve. And the next step for us always was, and first of all, really understanding and deep diving into the problem, and also understanding the customer behind that problem or the process behind that problem.
[00:11:20] Khrystyna Sosiak: So it's not all like saying, Okay, our like acquisition process or our like, I dunno, customer support process doesn't work. Need to understand why. Where is this breaking thing that is not working that you actually want to optimize for and you want to improve? And once you really know that, then you understand and you like passionate about your problem that you're trying to solve and you really understand the difference it can make, then you deep dive into the data you have and that is such a.
[00:11:50] Khrystyna Sosiak: Critical point, like, uh, I used to do the, the classes of machine learning, uh, in one academy. It's like a lot of different [00:12:00] people from different backgrounds that they're trying to learn machine learning and ai, and I was always there. No matter how exciting you are, like about all this cool algorithms, you know, and the machine learning models you can build, you always need to start with data because the data is the key to successful product.
[00:12:17] Khrystyna Sosiak: When we talk about AI and machine learning and we really need to make sure that this part is set and we, most of the time we'll spend more, most of our time in. Understanding and gathering and preparing the right data because you know, that's why machine learning models failed. That's why so many of the projects I was working on failed is because we didn't have right data or it was the bad quality, right?
[00:12:47] Khrystyna Sosiak: Or it was. Something else, but it was, um, it was the data, right? That would never generate you the good results to the problems that you said. You could have a set of right questions, [00:13:00] but if data is not there, there's not gonna be an answer.
[00:13:06] Himakara Pieris: ,what would be a good approach to validating?
[00:13:09] Himakara Pieris: That your dataset is of good quality, you are acceptable for the kind of problem that you're looking at.
[00:13:16] Khrystyna Sosiak: There is the couple of the criteria that you can look at. Definitely the first one is you need to understand what is the problem you're trying to solve and what is type of the data that you need, right?
[00:13:28] Khrystyna Sosiak: So the first one always says like, start with the question, do you actually have any data available? And then do you have a fresh data available, , do you have a good. Quality data available, , the system that are generating the data are reliable because all of those things that are really like, it's.
[00:13:45] Khrystyna Sosiak: It's not even starting at the data, starting of what is the system that is generating the data? And that's where things start. And that's where also sometimes when you think about the bias in data that starts, it's not because of [00:14:00] the data, it's because of the system or the person that is generating the data.
[00:14:05] Khrystyna Sosiak: And I would say, look at the system and ask, can I trust this system with the quality of the data that the system is generating? And, um, yeah, definitely fresh data is super important. You know, I saw the, some people that would like, okay, we have this 10 years old data set. Let's build something for, we're gonna do the prediction about, you know, tomorrow it's not gonna work because the world is changing all the time.
[00:14:34] Khrystyna Sosiak: You know, the, the behavior that customers had one. Week ago, one month ago, could never be the same anymore. I had a very interesting example of the project that failed during Covid. So we had a very good model. It was doing the predictions, for the churn. And we were about to launch it for like, some pilot, right?
[00:14:57] Khrystyna Sosiak: It was, it was quite costly. [00:15:00] We wanted to be very, uh, strict on like, what is the customer base? We would launch it. We still decided to launch it, and then the covid started and you know, we said, okay, we, we still gonna see. We just gonna validate the model. And our model crashed on the new data. Because the customer behavior changed completely.
[00:15:19] Khrystyna Sosiak: And that's the thing that you need to have a recent data. You need to validate are my data representative to the reality that I live in? You know, is the data that I generated, actually the data that. Would correspond to what customers do and how they think at this particular moment.
[00:15:38] Khrystyna Sosiak: So that's, that's an important one as well. Right. There's also a lot about security and whether the data actually should be used because we live in a world when you know, the data, it's one of the. Very valuable assets that a lot of people trying to get and trying to use, not in the right way. And we as the product managers [00:16:00] that are building the products on top of the data and then responsible for the data that we use and for the, you know, Security and privacy of our customers is really important.
[00:16:12] Khrystyna Sosiak: Not only to think about the business metrics such as, you know, new customers, the revenue, but thinking about the quality, uh, uh, the security and the privacy of our customers first. And if there's the risk or you have a risk that, okay, it feels like something may go wrong, then I would say stop it before you start it because, um, The reputation laws or the, you know, defines even like all like very financial things.
[00:16:43] Khrystyna Sosiak: You could actually harm your company more. Doing some things like that than benefiting. And I think that's also something to remember is. Can I use this data? Is that the right data to use? Right. Do I have, for example, the consent, the rightly corrected consent of the [00:17:00] customers that I can use this data, right?
[00:17:02] Khrystyna Sosiak: It's my data anonymized in the right way. Really a lot of things I think data we can talk about it for, for a long time, but it's uh, it's the key.
[00:17:11] Himakara Pieris: So what is the next step from there?
[00:17:15] Khrystyna Sosiak: When we talk today, , about why, machine learning products and projects fail. One of them is because there is unrealistic expectations and there's no clear communication. And aligning the expectations between what the business usually or product expects, and what technically can be delivered. And I think the next step is once we are done with the data and once the data is prepared, then we have the feature engineering and the data is clean. It's the experimentation phase.
[00:17:47] Khrystyna Sosiak: And you know, sometimes it can take one. Quick to build the model and we say, wow, it works. You know, like we can go to the next phase and sometimes it can take months. [00:18:00] There's no result. And I think having this transparency with the stakeholders saying, Hey, that's not a software project. You know, I, I cannot tell you.
[00:18:09] Khrystyna Sosiak: Like, okay, that's gonna take two sprints and that's gonna take one sprint and we are gonna be done because that's a very different paradigm of. Building something and thinking about something. And I think bringing, not bringing your stakeholders, your managers, the people that the sponsors of the project align with that.
[00:18:29] Khrystyna Sosiak: And understanding and being on board before you start could really cause, you know, disagreement, but also just failing what you're doing just because there's no support anymore. And I think that's one of the important things, , is just. When you do the experimentation, and it can take a lot of time, right?
[00:18:49] Khrystyna Sosiak: Make sure that you have the set of boundaries that's saying, okay, that's what we aligned with the stakeholders and that's the metric that we are gonna optimize for and that's when we are gonna [00:19:00] stop. So there's, I always say when you think about experimentation and building the model, and like there's need to be two things.
[00:19:07] Khrystyna Sosiak: You look at the one. What is the metric? Like? What is the metric value that you are optimizing for and what is like your north star that you say it's good enough, you know, we can move on. We can try to validate it on the real data. We can try to see whether we can put it in production. That's one. But there's another one, and this one is actually much more important is saying.
[00:19:35] Khrystyna Sosiak: Honestly sitting with your stakeholders and saying, how much time do we have to do the experimentations it to say that after this time, we're not gonna try anymore? You know, I. We say we not gonna do it anymore because we don't see any progress. And I think having this boundary set at the beginning, before you start invest, being invested in [00:20:00] the project is so important because you know, the more you invest, it's a, it's from the psychology.
[00:20:06] Khrystyna Sosiak: The more you invest in a project, the more difficult is for you to say. It's over, even though everyone knows it's over and there's nothing gonna be out of it. And that's how also, uh, a lot of projects fail, right? But also fell with having these bad feelings that someone is killing something that is so close to your heart.
[00:20:28] Khrystyna Sosiak: But when you have this set of expectations and when you're very clear, we have this goal, and if you are not achieving this goal, or we're not close to this goal in this particular timeframe, We gonna, we gonna just kill it, you know? And it's, it's good because then you know it and you know, you work hard to make it work, but it doesn't work.
[00:20:50] Khrystyna Sosiak: That's something you agreed on at the beginning.
[00:20:54] Himakara Pieris: You touched on this earlier. It sounds like part of that conversation is having a good way to validate the [00:21:00] results or the impact I. , of the model, , and compare that with some real world as of right now, results. So you have a very clear comparison
[00:21:08] Himakara Pieris: point.
[00:21:10] Khrystyna Sosiak: Absolutely. I think that understanding, because that's, that's the one thing I always said, that you can build the best model in the world, but there's no impact in building the model if it's not gonna land on production and actually being used.
[00:21:24] Khrystyna Sosiak: And I think that's, that's so important to make sure that what we build is line on production and stakeholders. The key in making sure it's there and it's used. And I think that having the expectations about the time bound and when we, when, you know, we call it off and we said, you are not gonna continue.
[00:21:45] Khrystyna Sosiak: But also having the real expectations with the stakeholders about. How the machine learning works and the mistakes that it can make. You know, the error rate and the what is the cost of the [00:22:00] error. We failed one project. The model was really good. There was impressive, like it was so scientifically interesting build, like we literally spent like months reading all the research papers and trying to understand how to solve one problem and we built the model that was really good.
[00:22:17] Khrystyna Sosiak: And it was like very close to someone's, you know, the results were very close to the results that someone like wrote the PhD on and it was really good. And, but because we, at the beginning, we haven't really. Talk and calculated what is the cost of an error for us and how we as the company are ready to take this risk and this cost.
[00:22:40] Khrystyna Sosiak: Saying that we know that's the value the model can bring, but that also the. The the risk that we can take, uh, we need to take. Are we happy with that or not? And I think having this conversation with the sponsors, right, with the senior leadership that would sponsor your project [00:23:00] and also be the decision makers.
[00:23:01] Khrystyna Sosiak: Whether at the end is the most important step gonna happen and the model is gonna land on production or not. And I think that's very important. And sometimes as the data scientist or the product manager that works with the data science, we are so invested and we are trying to sell our idea and our, you know, what we do.
[00:23:22] Khrystyna Sosiak: That we are so focused on the benefits that we don't explicitly talk about the risks, and I think it's very important to talk about those two things.
[00:23:34] Himakara Pieris: You talked about having this time boxing or having a clear understanding of how much time are we.
[00:23:42] Himakara Pieris: Love to spend on this problem . What would be a good way to estimate what's acceptable or reasonable? Because if you say, okay, you have two weeks to solve this problem, right? Then if it's not done in two weeks, you're gonna kill it. That doesn't sound quite reasonable. Maybe it is for some problems.[00:24:00]
[00:24:00] Himakara Pieris: What is your way of figuring out what is the right amount of time? What is the right number of cycles to burn through for any given problem?
[00:24:09] Khrystyna Sosiak: So there is the couple, there's no, like, I don't think there's like the one formula you can apply and you have the right estimation. And I think that estimations in general with, with the eye, it's very difficult, right?
[00:24:21] Khrystyna Sosiak: Because it's hard to estimate when it's gonna work. So the couple of things that I would look at and I would use like as a frame reference is the first one. And I always like start with the business because I'm, you know, the product manager. So I'll start with the business question and I'll say, For how long we can afford that, right?
[00:24:42] Khrystyna Sosiak: Like for how, actually, for how long we can afford trying to solve this problem in this way. Because you know, when you say yes to something, you say no to something else. And if you say yes to one opportunity and one project, it means that those [00:25:00] resources not gonna be used for something else. And that's the question of also the, the budget and the risk that we are ready to take and for how long we.
[00:25:09] Khrystyna Sosiak: Can sustain that and for how long it's. It's okay for us to take the risk that at the end it's not gonna work out right. And I think that's a clear conversation that we need also to calculate the cost. And I really like to understand on like, you know, when you have a numbers, it's much easier because you can calculate the cost of your people, you can calculate the cost of the processing power that you need, and you can say that's the cost of one week of doing it.
[00:25:37] Khrystyna Sosiak: And that's the cost of one month of doing it. If we know, let's not go with this happy scenario, let's go with the best scenario and say, we know it's gonna fail for how long? Like what is the risk? And like in like money, you know, that we are ready to take. And I think that's something that the first thing we would do, right?
[00:25:57] Khrystyna Sosiak: It's just to understand what is the [00:26:00] risk we are happy to take and we know that the reward that we can get. It's much higher, right? Because if we say that, okay, the return of investment like for the year is gonna be this percentage, but if we move on like for one month, like with doing it, it's, or like for six months, right?
[00:26:19] Khrystyna Sosiak: It's, it's gonna take us like five years to return the investment of that, then I would say, you know, we probably shouldn't be doing it in the first place. That's the first one. Experience. I think talking to the data scientists and to the engineers and also understanding, okay, looking at the problem that we have and the, the complexity of the problem that we have.
[00:26:43] Khrystyna Sosiak: How much time is the reasonable amount of time to invest to see the first results? And I would never say, Hey, let's do one iteration. And we, and we decide because I think it's not enough, right? We need to try different things. And then [00:27:00] I would optimize for how we can reduce the, the, the time. Of trying new things, right, of trying new algorithms, of adding new data.
[00:27:10] Khrystyna Sosiak: So then, you know of it's not gonna take us weeks, but it take, gonna take us days, right? Maybe just to see whether we can find something where actually works and then optimize and then it rate on something. We actually start. Working. Uh, but yeah, the time the estimation is, is difficult most of the time. I think that you look at the resources that you have and the technical complexity of the problem that you have, because, you know, sometimes you are such a complex problem.
[00:27:40] Khrystyna Sosiak: Like if someone would ask. Q to build the chart G p T in one week. Like, I mean, probably there's some people that can do that. But you know, if you look at just like normal data science team in some company, they would not do that in one week. Right? And that's like realistic. There's no way, right? So you need to say what is the time that you're comfortable with delivering the first results, [00:28:00] right?
[00:28:00] Khrystyna Sosiak: And, and then going from there and understanding whether there is actually the positive change. In the next situations or it all stays the same because if you try 10 times and it all fails, then probably that's, you know, the time for us to stop.
[00:28:18] Himakara Pieris: This sounds like you're placing a series of betts on probability of success in r o i, in the technical feasibility and in the team, and also the kind of adoption you could, you could get, right? So you have to decide case by case basis.
[00:28:31] Himakara Pieris: How much are I, you willing to bet that, , you can, you can deliver x times return on investment and how much you wanna bet , this is technically feasible, , et cetera. So are there any other things you would put into that mix of considerations other than r o i, technical feasibility, your team's capabilities and adoption?
[00:28:54] Khrystyna Sosiak: , let me think about it. So I. I would, again, I think that [00:29:00] it's also important to, always do the market research and understanding what is on the market. And also there's so many use cases explained, right? And I think just getting this information and understanding, so what is the reasonable amount of time to spend on something like that, right?
[00:29:17] Khrystyna Sosiak: I think that's very, that's also key. To understand and also set the right set of expectations and the time bound. So yeah, you, and also like, okay, when we, when we look usually like when you in commercial, not in the research, right? And when you have one problem, so for example, you build recommendation model for one product, right?
[00:29:40] Khrystyna Sosiak: I dunno for, for dresses then it's very easy to replicate it and build it for, you know, shoes. And once, if you have the similar set of problems you're trying to solve, for example, with different data or for different segments, then it becomes much easier because with experience, then you understand, okay, that's probably the [00:30:00] amount of time that we would need to validate and then we would need to build.
[00:30:04] Khrystyna Sosiak: So it's always the question whether I'm doing it and I already done something like that in a previous that. , more or less similar. It's never going to be the same, but the problem statement and the problem space and the data space is something that we know, do we know it? And then it's easier to have the set of expectations.
[00:30:26] Khrystyna Sosiak: But then when we think about something completely new, we never touched, right? Like if you take someone who all the time for their career used to build recommendation systems and, I don't know, text analysis, and you tell them now, you know, to do the generative AI of. Uh, videos of some popular singers that is very different problem space, right?
[00:30:49] Khrystyna Sosiak: And then it's very different, difficult to estimate. So then I would also give a bit higher buffer, right? And say, okay, we, we will need more time than usually, [00:31:00] right? If usually for the. Simple sim uh, problem space. We'll say, okay, it's two weeks, and if in two weeks doesn't work, then we move on. Then for something very complex and new, we will say, okay, it's gonna be one month just because it's, we don't know it yet.
[00:31:16] Khrystyna Sosiak: We need to discover, we need to learn. We need to trade.
[00:31:20] Himakara Pieris: Say you have a functional model that's performing well to an acceptable level, could you take us through the process of productionalizing that and what kind of pitfalls you would look out for? , what would you flag as high risk factors that could cause a project to fail?
[00:31:37] Khrystyna Sosiak: The one is like not being able to pro, uh, put in, in production. That's like the, the highest, uh, I think problem that a lot of companies think more than we think of, like they really have. But when we say, okay, we are ready for that, I think the one that is.
[00:31:53] Khrystyna Sosiak: Very common for like maybe smaller companies as well, but also for the big ones. Not all of them have the [00:32:00] right strategy, like have a strategy for the ML ops and how would you actually deploy the machine learning model, right, and have the right infrastructure. To maintain it. And I think that's a very important thing that we need to understand that, , deploying machine learning model and also the processes and the monitoring that is required is a bit different from the normal deployment of, you know, some A P I or the uh, or the application.
[00:32:27] Khrystyna Sosiak: And we need to be able to do that and we need to have qualified people that know how to do it. And for me, I think the biggest problem. In doing it, in my previous experience in my teams that I had was not having the right people in place. You know, because usually when you hire, and also like I started quite some time ago, not that long ago, but like six years ago, and.
[00:32:56] Khrystyna Sosiak: At that time, like you would just hire data [00:33:00] scientists. There was no profile of like mops. There was no profile of, you know, machine learning engineer. You would just hire someone who works with data. And when you have a set of like five people that know how to do the data preparation and build machine learning model itself.
[00:33:18] Khrystyna Sosiak: It's not the same skillset as needed to deploy the model and make it work in production. And I think that's the biggest problem is not having the right people and also having the expectation that, you know, oh yeah. We, we will, we, you know, someone will, will come and do it. Because usually it's difficult to find those people and we need to, to make sure that we have them in place and they know what to do, right?
[00:33:44] Khrystyna Sosiak: Because when we just approach that, you know, we don't have, for example, the right monitoring in place. And I'll say that's the most important for me. And that's why some projects, a lot of them fail is because we would put something in production and then, you know, it works and. [00:34:00] Actually there was like one month, six months, one year, and we still operating under the assumption of what we validated one year ago that it worked.
[00:34:12] Khrystyna Sosiak: But we don't know if it works now. And if you have a webpage and you build it once, it's gonna work, you know, until something new. So you gonna break something in the code. But with machine learning, it's different because. What works today doesn't necessarily is gonna work tomorrow or in one year. And we need to have all the monitoring in place to make sure that actually your machine learning model is still helping your business and not harming your business.
[00:34:41] Khrystyna Sosiak: And uh, I think that's one of the very important aspects about. Having a running machine, learning models and AI in production is having the right monitoring and alerting in place, and also knowing what are the actions I'm gonna take once I [00:35:00] receive that alert. You know, like what it means. Like what it means.
[00:35:05] Khrystyna Sosiak: Not only, okay, I need to retrain this model, but if it's gonna stop working, if we are gonna turn it off. What is the cost of that for one, one minute, one second, one hour, right? Or if I will continue, it's, we still continue working while I'm fixing something. What is the cost of that? And I think those things is very important because we can say we are gonna monitor, you know, we have a set of metrics we're gonna monitor and we are gonna receive an alert.
[00:35:32] Khrystyna Sosiak: And then you receive alerts in the middle of night. And then what? And what is the next step? And I think having this plan and strategy in place of not only exciting part of building the model, but the part when actually customers interact with that and something goes wrong and the world is changing and the data is changing and things are breaking.
[00:35:54] Khrystyna Sosiak: It's very important.
[00:35:57] Himakara Pieris: I know you have like [00:36:00] seven key questions or seven areas that you look at, , as a way to mitigate the risk of failure in AI projects. Could you talk us through those seven key items?
[00:36:11] Khrystyna Sosiak: There's a lot of projects that we have done that failed, and that's why we learned from that. And it's like, okay, there's some things that you can watch out to, like, not to make those mistakes and prevent your product from failing. So the first one would be you ask the wrong questions. You don't understand the problem or you don't understand the customer, or you don't understand the data, and then the question and the metric you're optimizing for, and the problem you're trying to solve is actually not the right, and then no matter what you do is not going to work because the question is wrong.
[00:36:46] Khrystyna Sosiak: Uh, so you need to invest in that. You need to invest time in understanding that. Um, the last, the next one, it kind of, not very, not technical, right? You say, okay, machine learning. Products fell because [00:37:00] there's like something with technology or data? No, actually most of them fell because there is no support from stakeholders.
[00:37:08] Khrystyna Sosiak: There's no understanding, there's no, uh, sponsorship for the things that we do. There's no willingness to change the approach That was. Used for years. And I think bringing all of them on board and making sure that you get buy-in from them, from the stakeholders that you need to work with. And it's, I'm not only talking about the sponsors that gonna give you money to do that, right.
[00:37:33] Khrystyna Sosiak: Or like, Green light, but also about the people that would, for example, at the end of the day, will need to use the machine learning model. Are they willing to do that? Right. Because if you have department of 100 people and you come to them and you tell them, you know, we want you to use that, and they say, no.
[00:37:49] Khrystyna Sosiak: There's not, there's not a lot you can do. Um, yeah, the data, data is the key. So having the data quality, um, in the right place, checking the [00:38:00] data quality, having the, the right data set in place, it's very important. And if you don't have it, it's a very big problem that can cause the failure. Uh, the data science team, having the right people in place and having the right set of people, when you have the team that every single person would know only one thing, and it's all the same thing, it's not gonna work because with the way the product works and also there's different stages and you need different set of skills, um, Going for something super complex that you not necessarily understand and having like so super complex models, it's also very often like you'll fail and you'll not even know why you failed because it was so complex that you don't even know what optimize for and like that's another one.
[00:38:50] Khrystyna Sosiak: Right? And also over promising, overselling, setting the wrong expectations. It's another one because. You are, there's [00:39:00] always the risk of failing and you need to go and talk about that and make sure that cus uh, stakeholders know about it. And there's always the return of investment, there's always the risk.
[00:39:12] Khrystyna Sosiak: And just making sure that you and people you work with align on that and fine with that. That's, uh, that's the things I would look for.
[00:39:22] Himakara Pieris: Thank you for coming on the podcast and sharing insights today. Khrystyna, is there anything else that you'd like to share with the audience?
[00:39:29] Khrystyna Sosiak: I think that just making sure that what you do in life, you bring some impact or to your customers or to your business or to the world, and making sure that we use our power of using technology in the right way.
[00:39:43] Khrystyna Sosiak: I think that's, that's very important and there's so much power that we have right now and opportunities, so yeah.
-
I'm excited to bring you this conversation with Ali Nahvi. Ali is a Sr. Technical Product Manager for AI and Analytics at Salesforce. During this conversation, he shared his thoughts on championing AI initiatives as a product manager, translating business needs into AI problem statements, and how to positioning yourself for success.
Links
Ali On LinkedIn
Transcript
[00:00:00] Ali Nahvi: to get there, to build that success, success story. You need to fail. And failure is part of the process and sometimes it's not easy for people to see that,
[00:00:09] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate, and learn from industry leaders who are solving real world problems. Using AI.
[00:00:19] Himakara Pieris: I'm excited to bring you this conversation with Ali Navi. Ali is a senior technical product manager for AI and analytics at Salesforce. During this conversation, he shared his thoughts on championing air initiatives. As a product manager, translating business needs into air problem statements. And how to position yourself for success.
[00:00:37] Himakara Pieris: Check the show notes for links. Enjoy the show.
[00:00:42]
[00:00:43] Himakara Pieris: Ali, welcome to Smart Products.
[00:00:47] Ali Nahvi: Thank you so much Ima, for having me
[00:00:49] Himakara Pieris: to start things off could you share a bit about your background and how you got into AI product management? I.
[00:00:58] Ali Nahvi: I'm an accidental [00:01:00] product manager. I started my career journey with business intelligence and I guess it was around 2012 or 13. It was the first time I've heard the board data science. Before that we simply called it math. And I love the idea. I decided to move from BI to ai and that was the major figure for me to come to us do a PhD.
[00:01:27] Ali Nahvi: And I did my PhD in application of ai m ml in the context of project management. And after that I started as a data science consultant. In a consulting company and yeah. And, and, and one day out of blue my roommate from grad school called me at the time who was working at Amazon and he told me that, Hey, I mean, we have this thing in product manager and I think you should, should become one of them.
[00:01:56] Ali Nahvi: I did some research and very quickly I [00:02:00] also. I've got the same impression that, well, this can be an ideal job for me. I love helping people. I love solving business problems. I love ai. And I also love business development and communication and being around people.
[00:02:16] Ali Nahvi: So I thought, well, that might not be a bad idea. So I joined iron Mountain in my first for like manager role. And then I joined another company after a while Cengage, which was mainly focused around online education. And recently I've joined Salesforce as a senior technical product manager for AI analytics.
[00:02:45] Himakara Pieris: What is the primary difference you see going from, bi to data science, to ai as a product product manager? Do you need a different skillset? Are those, BI skills transferable across all these verticals?[00:03:00]
[00:03:00] Ali Nahvi: Yeah, business intelligence definitely still helping me a lot.
[00:03:04] Ali Nahvi: And from data science perspective, I'm one of those PMs who thinks that PMs should be technical and have the ability to have that super technical discussions with the teams especially in data sciences space. In data science, in AI ward, understanding the problem, understanding business requirements is, in my opinion, is solving half of the problem.
[00:03:31] Ali Nahvi: If you get there, if you can really digest the problem statement and have the ability to transfer that into a data science language then you are a really good PM and, and to do that for me, Having that technical background around data science have been extremely helpful.
[00:03:51] Himakara Pieris: What would be a hypothetical example for translating a business requirement into data science or machine learning language?[00:04:00]
[00:04:00] Ali Nahvi: Let's say I'm assigned to work with a stakeholder in sales or marketing. And I sit with them, set up a call and say, Hey, what's your pain point?
[00:04:12] Ali Nahvi: And they say, okay, I wanna increase sales and productivity. And so I would say, okay so can you explain what you're doing on a day-to-day basis? And they, they explain, this whole sales process that they go through from lead generation to sales calls to closing deals, and I might be able to find some opportunities there.
[00:04:36] Ali Nahvi: To use AI to help them to do a better job. For example, the lead generation piece. Maybe you don't need to call all the customers, all, all the leads coming to your way. Maybe you can optimize that. Okay? But then you need to build a bridge. Between that really weight business problem into a very solid, robust data science problem.[00:05:00]
[00:05:00] Ali Nahvi: The business requirement doesn't give you anything like dependent variable, independent variable, the data structure, anything like that. So as a product manager, it's my job to help the team to kind of define that problem. And another thing that I believe that, that, that's why I think data, data science, product managers should be technical, the feature engineering.
[00:05:22] Ali Nahvi: That's extremely delicate thing to do in my opinion. It's, it's something that where you tie business with science and you really need to have good understanding about how data scientists would do feature engineering. And at the same time, you really need to have a robust understanding of how business operates to, in incorporate all the important features in your feature engineering and make sure you capture all the important elements.
[00:05:51] Himakara Pieris: You talked about, doing these customer interviews or user interviews looking for opportunities, these might be data, sort of [00:06:00] curation opportunities or recommendation opportunities or clustering opportunities or, or what have you, that sort of.
[00:06:09] Himakara Pieris: Buried in, in the story that they're saying.
[00:06:11] Himakara Pieris: You identify that and then you transform it from there to a, a problem statement that machine learning and DataScience folks can understand. Right. Could you talk me through the full workflow that you're using? So what are the key steps? So sounds like you're always starting with a use interview.
[00:06:28] Himakara Pieris: How does the rest of the process look like?
[00:06:31] Ali Nahvi: Let's go back to that sales problem again. For example, on the late generation, they say that, okay, we generate 2000 leads per day, but we can only call. 500 of them. So the, the lead optimization problem that I mentioned before that would pop up or on the sales calls, they say that we have limited number of sales mentors who can help salespeople.
[00:06:54] Ali Nahvi: So maybe we can leverage AI to listen to some of the recorded calls and provide some [00:07:00] insights. So these are all hypotheses that could come up and I will write them down, all of them as potential initiatives. And then I would ask these questions from my stakeholders all the time. Let's say we are six months from now, a year from now, let's say we are done with this and we build this model that is a crystal.
[00:07:20] Ali Nahvi: Al can tell you this lady's gonna make it, this lady's not gonna make it. How would you use it in your day-to-day, how it's going to change your workflow? Okay. And, and based on that, I, I try to basically come up with an estimate, ideally a dollar value around the, the potential added value that initiative can have.
[00:07:44] Ali Nahvi: And then I would work with my team engineering managers, data science managers, try to understand visibility, data accessibility, data availability, and level of effort. , and based on that, I create a diagram [00:08:00] in, in one axis we have value. In the other we have level of f effort. And when you build something like that, it, it, it would immediately pop up and, and the, the, the high highest priority initiatives would, would show themselves to you.
[00:08:19] Himakara Pieris: Sounds like you're identifying opportunities and then solutions, and then you are going through an exercise of validating these solutions. Right? And then it moves to the implementation part. I want to go through and discuss how if it is different from a traditional software development process.
[00:08:41] Ali Nahvi: Absolutely. There are major differences between data science and software engineering and lots of intersections. So intersections are obvious. They both need coding. They both need infrastructure.
[00:08:54] Ali Nahvi: They both need data. But there is a, a delicate [00:09:00] difference between them that. It's, it's, it's kind of hidden in the name of data science as well. It's science, it's not engineering. So element of uncertainty is there. All of these initiatives that we came up with, they are just hypothesis. We have a hypothesis that based on the current data, based on the current evidence, we might be able to build a prediction model to meet whatever requirement that we have in mind.
[00:09:26] Ali Nahvi: But for there might be a chance that, that, that hypothesis. Wouldn't be right or even there might be a chance that we build a model, but it's not really usable or explainable for the user. So these types of uncertainties I think significantly different. Differentiate data science from software engineering work.
[00:09:55] Himakara Pieris: How do you account for and plan for the probability of [00:10:00] failure? There is a probability that your moral can't make predictions with enough level of accuracy. How do you put in guardrails to make sure that this kind of failure, probability of failure is accounted for and planned for in that process?
[00:10:15] Ali Nahvi: That's a fantastic question. And I have two mitigation plans for that one on the soft side of the business and on the other is quantitative.
[00:10:28] Ali Nahvi: The quantitative side is basically, I look at what they have right now. Okay. Do they have any system in place? Let's go back to that late generation problem. Is it 100% random? So if it's 100% random, then I just need to beat the random mistake which, which is not a super challenge. But on the other hand if they have a system already that is able to predict things by 90% true positive, I'm [00:11:00] not gonna touch that. I don't wanna yeah, compete with that because the chances of success is, is not really high.
[00:11:08] Ali Nahvi: I always try to kind of educate my stakeholders on how data science work and I, I try to show them some stats around the failure of data science.
[00:11:18] Ali Nahvi: Nine out of 10 would fail for different reasons, and I, I try to kind of be honest about some of the pitfalls and shortcomings of data science in advance and say, Hey, this is just the hypothesis we have. It may not work.
[00:11:36] Himakara Pieris: Do you have a probability of success for each experiment that you're running and parallel track number of experiments to make sure that you have something that's functional at the end,
[00:11:48] Ali Nahvi: at least at the qualitative level. Yes. I try to. Qualitatively capture based on the discussion I have with engineer managers and also my my own [00:12:00] experience and my own feelings about the problem and the evidence that I see. Ideally, we should be able to get into some quantitative level. Even if it's, it's not possible, you still can do it qualitatively.
[00:12:16] Himakara Pieris: Sounds like this. Structure of it makes it difficult to follow something like agile as part of the development cycle. How does the development, workflow, or the development cycle look like in machine learning projects
[00:12:31] Himakara Pieris: for you?
[00:12:33] Ali Nahvi: Well to be honest, I think all these, these, these things would, would happen before we start development and I think we should be very picky.
[00:12:44] Ali Nahvi: Developing What? That's why I set up all these rules for myself because there are tons of business problems out there. Okay. One rule that I used to set up in my former company that if we cannot put a dollar value on an initiative, we don't do it. Even if it's a strategic, if [00:13:00] and if it's intuitive that, that, wow, it has lots of value.
[00:13:04] Ali Nahvi: We have to be able to put a dollar, we have to be able to quantitatively measure that because there are lots of opportunities out there. We can do lots of things and we have limited resources. So I, I try to be some sort of a goalkeeper for the team so when things get into that developmental stage, none of these questions would come up again.
[00:13:25] Ali Nahvi: And, and then when things get into that developmental stage, we try to follow Agile as much as we can. Many people say that Agile is not really working for data science. My perspective is a little bit different there. I still think that majority of the things we do in data science, they have a engineering function.
[00:13:46] Ali Nahvi: I mean lots of pre-processing, post-processing, lots of network is just simple data engineering. And the data science piece model development is maybe 20%, 30% of the [00:14:00] whole work. For that piece, definitely we need to have some sort of a contingency plan in case things won't go as we expected and we need additional time to try different models and different iterations of models.
[00:14:15] Ali Nahvi: Beside that, I'm still loyal to, to Agile
[00:14:18] Himakara Pieris: you talked about data science modeling component. You talked about engineering component. So this sort of brings me to thinking about the structure of a AI machine learning data science team within a product organization.
[00:14:36] Himakara Pieris: So what are the different approaches to put these teams together? What are some of the roles and responsibilities and what advice you have when you think about team formation?
[00:14:47] Ali Nahvi: Well in my carrier, I've been working with teams in, in totally different structures. But the thing that worked for me, I cannot say that is ideal [00:15:00] across all companies, but something that worked for me is.
[00:15:05] Ali Nahvi: Some sort of a separation of church and estate. I want product and engineering to be very separated from each other because there's obviously some, some conflict of interest going on there. As a product manager, when they ask me to estimate a point for a. Story, I even unconsciously I tend to go with one or two all the time.
[00:15:28] Ali Nahvi: But a data science manager may, may feel totally, totally different about that, which is totally, totally understandable. Cause they are eventually have to deliver. So at, at the high level, I think they should be separated. And then within a data science team ideally it'll be great if you have, if you can have some, some data engineers.
[00:15:50] Ali Nahvi: To help with some of the initial transformation. Building some of the pipelines help with centralization, having data analysts to help with some of [00:16:00] the pre-processing cost processing, doing some analysis and helping data scientists to build some hypothesis and validating them, and eventually data scientists to help with lots of statistical modeling, machine learning and bringing explainability to the models.
[00:16:17] Himakara Pieris: I think there is a massive amount of interest in embedding AI and machine learning into products across the board. So in this context, we have a large number of. Establish software companies, that are profitable, serving clear market needs. And we have product managers inside these organizations who are now looking to find the optimal path to adopt, and experiment ai within their own products. What do you think is the best path?
[00:16:48] Ali Nahvi: It's all about customer. And customer should be in the center of what you build and what you design. So we need to understand the customer pain points [00:17:00] and for, for doing that, our different methods If, if you have limited number of customers, I mean, for example, if you are working in a big, big corporation and you have one product, but you only sell it to 12 giant banks, for example, in that case you have the privilege of going to customers and, and talking with them and figure out what's the pain point.
[00:17:25] Ali Nahvi: But if you have a product that is kind of bigger in a scale and it has for example 100,000 users. Then, then you have to look at the usage data, try to understand some of the pain points that, that your customer is facing using your product. That piece is not really going to change.
[00:17:48] Ali Nahvi: Okay? But the thing that would be different from software engineering to AI ml is when you want to build those [00:18:00] features and capabilities. Let's say I'm in a software company that didn't have a data science AI ML practice. Now I wanna add a feature there. To my product. And and, and for that I, I'm going to start with some of the architectural challenges.
[00:18:19] Ali Nahvi: For that I need data. Okay? In, in software engineering, especially the companies who have, have been a legacy of software engineers, they have robust processes for data distribution and data governance, which is a good thing. But when you wanna build data science models, sometimes you need all the data.
[00:18:38] Ali Nahvi: I remember I was having this, this conversation with one of the solution architects that I said, Hey, you mean we need data from this database, this database, this database, and we need data from these tables. And it was like, okay, but can you specify which columns? And I was like, I don't know. And he was like, what?
[00:18:58] Ali Nahvi: What do you mean you don't know? [00:19:00] And I don't like, I mean, I need all the data. I mean, I, I have no clue which I want till until we don't, we won't examine it. You know, we won't know which data we need. And and yeah, that was kinda funny problem that happened. So architectural challenges definitely would be one major challenge there.
[00:19:19] Ali Nahvi: Another challenge is human resources mean data science resources are expensive. Some, sometimes company cannot see the immediate value to higher. 10 new data scientists. And sometimes you kind of need to align leadership and business with your vision and say, Hey, I mean, if we do this, it can bring this value.
[00:19:44] Ali Nahvi: So try to convert that to some sort of a benefit cost analysis problem for them. But another challenge, as I mentioned earlier, is uncertainty in a company like Salesforce. Failure is perfectly fine. I mean, you can fail when you do AI because [00:20:00] it, they've been doing AI for so many years and I mean, Salesforce users only heard about stories because they're using the successful features.
[00:20:13] Ali Nahvi: But to get there, to build that success, success story. Yeah. You need to fail. And failure is part of the process and sometimes it's not easy for people to see that, especially when they were building softwares. The way software features would fail, would be totally different from data science features.
[00:20:35] Ali Nahvi: It won't completely fail, but in data science, you have a chance to completely fail and come up with no outcome that's a possibility. So these are some of the challenges that could be
[00:20:49] Himakara Pieris: In nutshell, you have architectural challenges because the way you perceive data where you store your data and manage data, go on data could be very different.
[00:20:58] Himakara Pieris: And [00:21:00] you often don't get unrestricted access to things. What would be a way to solve that kind of challenge?
[00:21:09] Ali Nahvi: So ideally, I think data platforms should be designed as a product, not as engineering solution. There is a certain difference between a product and engineering solution, and that distinction is experience data platforms have persona have users, and those users are human being. Although they can code, although they are technical, but they are still human big and there is an element of exper experience that needs to be taken into consideration.
[00:21:39] Ali Nahvi: And I've seen lots of, and lots of data platforms, lots of system designs and solution architectures that completely forget about that piece because they say, okay, the persona, our person is data sign. They can code. Yeah, of course they can code, but, but it's still, I mean, they don't wanna struggle. With that.
[00:21:59] Ali Nahvi: So [00:22:00] ideally we should have some sort of a platform that me as an analyst, as a data scientist, as a data engineer can be able to choose the data tables I need. I I should be able to have access to some sort of a business glossary to tell me what each column in each data table would really mean.
[00:22:20] Ali Nahvi: That's the ideal data platform to me. I mean, element of experience, element of customer journey, all those things would be considered.
[00:22:32] Ali Nahvi: But things are not ideal. So what I've done before and helped me that I started with some champions.
[00:22:42] Ali Nahvi: Because these companies are really big. Some of these software companies can be very big and they have lots of teams, different databases, tons of solution architects, and among those folks, there might be some people who really get that. There might be some business leaders and also solution [00:23:00] architects who had some exposure with data science before.
[00:23:02] Ali Nahvi: So one thing you can do, start building something small with them and showcasing that capability to others. Telling that success stories to others and telling that success story to leadership. To gradually build this culture, cultivate this culture around centralization and around the proper data platform for data science, and eventually conquer different business units within the company and, and finally get to that automate centralized data solution.
[00:23:35] Himakara Pieris: Another thing you talked about was cultural challenges, not having a cultural failure at some of these companies . How would you overcome cultural challenges?
[00:23:47] Ali Nahvi: In companies like Salesforce, AI has been very well understood even on the business side, even within the non-technical folks and whenever new capability. [00:24:00] Comes up in terms of ai. People look at it as an opportunity, but, but I've, I've been in, in companies that the narrative was quite a bit different.
[00:24:10] Ali Nahvi: . So couple of things I think can, can happen there. First the alignment with leadership. Understanding the priorities would be a key, identifying low hanging fruit, and by low hanging fruit, I mean AI initiatives that doesn't require that much building on your end.
[00:24:29] Ali Nahvi: Things that you can leverage technology to do that with the current resources you have. So if you, if you have 10 software engineers in your team and don't have any data scientists, you still can do ai. So, I mean, there are multiple tons of services offered by AWS G C P Azure. Lots of companies like DataRobot Data, aiq Salesforce understand those things that can be leveraged.
[00:24:58] Ali Nahvi: So you can, you can still leverage the [00:25:00] skill sets of the people you have by technology in form of pay as you go, rather than bringing 10 data scientists on board and then wanna show, show value. So you can start small with current resources you have leverage technology, prototype it again, go back again.
[00:25:17] Ali Nahvi: Come up with a story, tailored success story, cultivate the culture, and then you can attract then when, then, then when you have business team, there's attention, then you can ask for budget for building things that you cannot easily get from out of the shelf.
[00:25:36] Himakara Pieris: That sounds like a good approach to overcoming human resource challenges,
[00:25:40] Himakara Pieris: You can go from like more of a pass offering, pay as you go, try it out using off the shelf products get some traction, show some value, and use that to justify acquisition of new resources to build out your team and go from there.
[00:25:56] Ali Nahvi: Yeah,
[00:25:57] Himakara Pieris: Let's say you are a product manager at a material [00:26:00] software company and you have identified number of areas that you could effectively use AI and machine learning to possibly deliver unique value, and now you had to go and present this to the leadership to get their buy-in.
[00:26:15] Himakara Pieris: What would you recommend as another thing they should do to secure buy in? Assume that the leadership is not very current with all the latest in AI and ml and they are possibly even more exposed to the challenges and pitfalls than the opportunities.
[00:26:35] Ali Nahvi: You need to propose something with whatever we have that would be able to generate revenue. When you do that, when you do that couple of times, then, then you earn the trust and then you can do other stuff. And I think that rule of thumb I can confidently say that it's consistent across all the businesses because they wanna make money.
[00:26:57] Ali Nahvi: They don't really care about AI or [00:27:00] technology. They care about their own business. They care about their own customers. And that, that should be the mindset you should have. You should always ask that. So what question from yourself? Let's say we built this. So what who is going to get benefit from is what's, what's gonna change?
[00:27:16] Ali Nahvi: After building this. So I think that mentality helps a lot with getting alignment with business priorities, building something that building some sort of a prototype to showcase to leadership to attract their attention with limited amount of resources and cost and yeah. And eventually a sustainable data science development.
[00:27:44] Himakara Pieris: Any success stories or failure stories that you'd like to share as
[00:27:49] Himakara Pieris: well?
[00:27:50] Ali Nahvi: So in terms of successes, stories when I was with iron Mountain my former [00:28:00] company we wanted to Basically build some workflows, processes to use AI and based on that be able to parse some of the documents.
[00:28:18] Ali Nahvi: And this is no secret. I'm not giving any insight. Inform, this is our mountains business. You can find this on their website. And one of the challenges that, that I faced immediately when I got the job. Oh gosh. We need lots of AI to do this. And we are not an AI company. At the time, they hired 12 machine learning engineers.
[00:28:42] Ali Nahvi: But I've done some quick estimation, one of the engineering managers, and we learned that even if we had 100, we cannot deliver these things in a year. So I, I did some research and I found a company who had all that AI and that, that company's name [00:29:00] surprisingly, was Google. So we reached out to GCP folks and talked about the problems.
[00:29:07] Ali Nahvi: I talked with our G C P representative and we found that majority of the AI components, the features that we want they already have, and we can start using them immediately and. The, some of the things that the G CCP didn't have in their offering, then we can leverage our own resources to do that. So eventually the cost of development significantly decreased from zillions of dollars to a couple of millions.
[00:29:38] Ali Nahvi: And also the time and also the quality of the de deliverables significantly improved.
[00:29:46] Himakara Pieris: That sounds like a good success story. Are there instances of failure that you can share as
[00:29:50] Ali Nahvi: well?
[00:29:51] Ali Nahvi: So yeah, I had some ambitious ideas before when I was in consulting. [00:30:00] We had, lots of legal documents that was manually passed by folks at the time. And I've, I've tried to showcase some of the AI email capabilities to leadership to build to basically help with parsing some of those legal documents for our customers.
[00:30:27] Ali Nahvi: And the prototype that I put together, it was fantastic. Honestly, I was doing a great job, but I didn't think about the scalability problems and the volume of data. Again, I was thinking about this as a data scientist, not as a product manager later, and it was one of the most challenging things that I've learned in my life in a very hard way.
[00:30:49] Ali Nahvi: That when you are a data science product manager, you have to think about this whole system end-to-end data science. Ai, ML is just 10% of it. 90% are other things that [00:31:00] you should consider in your plans. And and I didn't plan for that. So it it failed. It, it failed. It was a catastrophe.
[00:31:08] Himakara Pieris: So essentially think about how you productionalized something, how you scaled something that goes beyond showing value in a sandbox environment.
[00:31:17] Himakara Pieris: That sounds like the biggest takeaway there.
[00:31:19] Ali Nahvi: Exactly. Yeah.
[00:31:21] Himakara Pieris: Ali, thank you so much for sharing your insights today. Is there anything else that you'd like to share with our audience?
[00:31:27] Ali Nahvi: Thank you so much, Hima. no, I really enjoyed talking with you. Also had a chance to listen to the previous podcast and I loved it.
[00:31:36] Ali Nahvi: I really appreciate what you're doing. I think this is gonna help a lot with, with data science community and I hope to see more and more folks to become data science product managers.
[00:31:47] (Outro)
[00:31:47]
[00:31:50] Hima: Smart products is brought to you by hydra.ai. Hydra helps product teams explore how they can introduce AI-powered features to their products and deliver unique customer value. Learn more at www.hydra.ai
-
I'm excited to bring you this conversation with Sayanti Ghosh. Sayanti is a Sr. AI/ ML product manager at Teck Resources — one of Canada's leading mining companies. Sayanti manages a recommender systems product at Teck to support clean coal processing. During this conversation, she shared her thoughts on assembling an AI/ ML team, build vs. buy decisions, and the types of risks/ KPIs she monitors.
Links
Sayanti On LinkedIn
Transcript
[00:00:00] Sayanthi Ghosh: if you wanna go for build, very important to see where the company stands in AI product, maturity level. Is it just starting? Is just in an experimentation phase? Is it in the level of using AI in few of the products? Or it is in a phase, or it is in a phase where it is into the DNA of the organization.
[00:00:21] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate and learn from industry leaders who are solving real-world problems. Using AI.
[00:00:33] Himakara Pieris: I'm excited to bring you this conversation. Shanti gauche Shanti is a senior AI ML product manager at tech resources. One of Canada's leading mining companies. Shout the managers recommended systems product at tech to support clean coal processing. During this conversation, she shared her thoughts on putting together an AI ML team build was this by.
[00:00:53] Himakara Pieris: Types of risks and KPS. She monitors. Check the show notes for links. Enjoy the show.[00:01:00]
[00:01:00]
[00:01:01] Himakara Pieris: Shanti, welcome to the Smart Product Show.
[00:01:03] Sayanthi Ghosh: Thanks, Hima. Thanks for giving me this opportunity
[00:01:07] Himakara Pieris: tell us a bit about what Tech Resources does and also how you use AI and ml.
[00:01:14] Sayanthi Ghosh: Tech Resources is a mining company. Tech is a hundred plus years old company who works in copper, zinc, and , steel making coal.
[00:01:25] Sayanthi Ghosh: Tech resources also has another wing, which is race. And tech digital. So that's the part where they work with all the AI and ML products. The whole idea is to increase production and there are various other problems in supply chain, in mining, in blending. So there are various aspects and opportunities at Tech where AI and ML and other software engineering products help them solve these critical problems to, , grow their mining, to grow their production, and make it much more sellable product for their customers.
[00:01:59] Himakara Pieris: [00:02:00] What kind of AI and non-AI products are you involved in, and how do you draw that line?
[00:02:08] Sayanthi Ghosh: It's an interesting question . So before we jump into the kind of AI and non-AI products, let's just, In one line, just give an idea of what we mean by AI and what we mean by non-ai. So anything that you would have a machine to train to and a machine could learn, we broadly put them into ai and anything that is rule-based, which doesn't have any learning capacity those type of things, we broadly put them into non-ai.
[00:02:38] Sayanthi Ghosh: So at Tech, what I do specifically with ai, we run recommendation systems. So think about it as a factory and I am in coal processing, so my work is in the domain of coal cleaning the coal. So think about there is a factory, you mine some coal. You need to clean that coal before you sell it to [00:03:00] your customers.
[00:03:01] Sayanthi Ghosh: So when you are mining that coal, And when you are cleaning that coal, your goal is to maximize the production of the coal. So you do not wanna lose clean material while you are cleaning it. So, If there is a factory to do that, there are several machines, ? And you want those operators to run the machine in a, in its most efficient way, so that you clean and get the maximum amount of coal.
[00:03:28] Sayanthi Ghosh: Here you have a digital product which recommends these operators. What should be specific? Set points or parameters that you would put in each of these machines so that your machines are optimized. There are trade offs. I'm not going into too much technical detail, but there are trade offs, and then at the ultimate goal is to get maximum amount of clean coal.
[00:03:55] Sayanthi Ghosh: There are a few parameters. Also, we have to meet few [00:04:00] specifications, so the idea is to meet those specification and also maximize the coal amount. So that's where my AI product comes in. So it's a recommendation system. So it has got a bunch of machine learning programs underneath and an optimizer on top, and then it sends out recommendations.
[00:04:19] Sayanthi Ghosh: So this is one of the AI product, and to your question, the non-AI product. Now think about you clean the coal. So your machine learning recommendation system did great job. You cleaned the coal, you have lots of clean coal, now you need to send it to your customer. So there is a whole supply chain method running so you put it in a train, you first load it into the train. Train goes into the port. From the port, it goes to your customer. So there is a chain of events going on, and there is. Non-AI software engineering based product, which helps us optimize the amount of coal that we put into our trains.
[00:04:58] Sayanthi Ghosh: So this is a very [00:05:00] high level though, but this is an example of my AI and non-AI product that I work with.
[00:05:06] Himakara Pieris: How do you decide when to use AI and when to use a traditional or rule-based system?
[00:05:12] Sayanthi Ghosh: The first thing I would always say, if you see that it can be solved without ai, don't overkill with ai. If it can be rule-based, go for rule-based solution.
[00:05:24] Sayanthi Ghosh: Then the second thing you need to look into is data. It's very, very critical. You need a lot of amount of good training data because ai, without good data, it's like garbage in, garbage out. So you need to make sure you have relevant data, good amount of data, and the third important pillar is, Is your organization and your user ready for it, the cultural readiness to have an AI solution.
[00:05:51] Himakara Pieris: I also wanna start at the. Point of recognizing whether you need AI for something, is that based [00:06:00] on inability to describe , the outcome effectively using a set of rules, what kind of criteria goes into making that determination?
[00:06:11] Sayanthi Ghosh: It depends. So what is the problem that you are solving and what is the goal that you wanna achieve? Now, it could be that the goal that you wanna achieve is not at all possible by a rule-based system. Why it is not possible. If you would have a lot, if you have a data and you want your system to learn.
[00:06:33] Sayanthi Ghosh: Get trained and then behave in a certain way and provide an outcome. In that case, I don't think you can end up writing so many rules, but you can also think of like there were chat bots in past, or even now they have rule set up and the chat bot is working fine, but then you need. Much more advanced. So now with modernization, with time, AI is a lot more [00:07:00] understanding and adapting as well,
[00:07:02] Sayanthi Ghosh: so if you need that system to learn, Then probably a rule-based solution is not an ideal way. So it depends upon what is the problem, and what do you have? What kind of data do you have? It could happen that you know that you need ai, you know that you need a system which should learn, but then you don't have the data, or it is extremely expensive to get to that data, and you need a lot more time to even acquire the data. In that case, even if you want an ai, probably you have to think it in a different way. That you probably need more time to find the AI solution till you reach that solution.
[00:07:44] Sayanthi Ghosh: Until you gather that data, you need a non-AI solution to sustain.
[00:07:49] Himakara Pieris: (lets discuss your framework ) Let's go into the framework , , love to learn the process that you, use and follow.
[00:07:55] Sayanthi Ghosh: As I mentioned, starting with the problem, so always. You understand who [00:08:00] are your user, customer segment, and then you go deep dive into the problem,
[00:08:04] Sayanthi Ghosh: you need to check if you have enough information or enough data available in case your team has suggested that AI is the only solution or the best solution.
[00:08:15] Sayanthi Ghosh: If you see there is an option or a solution that can go without AI fulfills the business needs. Fulfills the value or solve the customer pain point. Go for non-ai
[00:08:28] Sayanthi Ghosh: once you do that, now you are in a space where you know about the problem. You have your vision ready. Try to figure out if, how easy or difficult it is to access the data. And how expensive it is.
[00:08:44] Sayanthi Ghosh: Understand how can you access the data? How can you integrate with your current system? That's the second checkpoint.
[00:08:51] Sayanthi Ghosh: Third is checking the current state of the data. So what do you have right now, what amount of data that you have, and if you need [00:09:00] more information, is it an open source information that you can find?
[00:09:04] Sayanthi Ghosh: Do you have to buy? Do you have to spend money on that? Do you need to invest? For me, I had to in like, my company has to invest on sensors. They had to put sensors in place so that we get the data. So that's an investment, you need to check what is the current state of data and what you need.
[00:09:23] Sayanthi Ghosh: Finally, , this I have seen happen many times. People think we have to create and innovate in our company, but sometimes it's a question of builder purchase, if there is off the shelf products that would work for you, if that fits with your investment and if you see and check your roi, if that works.
[00:09:45] Sayanthi Ghosh: I would say rather than inventing the wheel, Let's purchase that product. If you think that no, that won't work, then make a decision based on the company the problem and your investments and every and your [00:10:00] budget that it, either you will build it or you will purchase
[00:10:03] Sayanthi Ghosh: So this is a framework to reach to that point, but then you have to sustain it,
[00:10:08] Sayanthi Ghosh: you need right people, you need right skills. So creating that team. What do you need with respect to team and infrastructure? When we talk about research, that's something else. But when you are talking about a business, you need to deploy it, you need to move fast, as fast as you can with your AI ML products.
[00:10:31] Sayanthi Ghosh: You need to see what is the infrastructure that you need. So cloud computing, your ML ops, your whole architecture has to be adapted in a situation that you can deploy faster you need to have right skills, data scientists, machine learning engineers, data engineers, DevOps, app devs.
[00:10:50] Sayanthi Ghosh: And once you have your right team, the other big factor is security and ethics. Either in your company you might have a [00:11:00] team or that role falls into either in your product managers role understanding or program manager. There will be either, it will be. Given or dispersed into these roles, or you will have a, a team who works with security and governance and ethics.
[00:11:18] Sayanthi Ghosh: So those are high level. Steps that I follow in my own work and product discovery before I decide whether it is an ai, whether it is best option is a non-ai. And if it is an ai, then what are the checks that these are like prerequisite before you invest anything in your experiments or your MVPs.
[00:11:40] Himakara Pieris: To do a high level recap you understand the problem and the US needs, then you propose a solution. You work on securing access to data and also understand the current state of data, and then you make a build or purchase decision, purchase off the shelf product or deliver the product. After that point, you [00:12:00] build out a team.
[00:12:01] Himakara Pieris: Choosing the right skills, and then you implement the governance and security around that. Implementation?
[00:12:07] Sayanthi Ghosh: Yes.
[00:12:07] Himakara Pieris: What are some, high level considerations that you would advise? People to follow as they're navigating, build versus buy.
[00:12:18] Sayanthi Ghosh: If there is a question coming in that you do not know if you wanna build or you wanna buy, so take one at a time and see how much you are investing and how is the ROI coming out.
[00:12:34] Sayanthi Ghosh: So if I have to think, do I need to build. Take in account, do you have the strength? Do you have the team? If your company doesn't even have a team and you don't just take a team for temporarily and then dissolve it that's a heavy cost you are adding. And if you do not see much of a AI products coming in, if you do not have a company vision [00:13:00] to go in that direction, probably that's not a great idea.
[00:13:03] Sayanthi Ghosh: Eventually, you won't know what to do with the data scientist and lops, so there are a number of things that you need to consider if you think about build
[00:13:12] Sayanthi Ghosh: then your current architecture if it is not ready. So you have to shift everything. You have to get into cloud computing.
[00:13:23] Sayanthi Ghosh: If you, if, if the company doesn't have so many of products like that or it is not looking towards AI transformation, in that case, probably build is not a great idea because you need to sustain it. Is it worth sustaining one or two products and making such big change? Because it's also a question of change management,
[00:13:46] Sayanthi Ghosh: if you wanna go for build, very important to see where the company stands in AI product, maturity level. Is it just starting? Is just in an experimentation phase? Is it in the level of [00:14:00] using AI in few of the products? So it is in a active phase, or it is in a phase, or it is in a phase where it is a D into the DNA of the organization.
[00:14:11] Sayanthi Ghosh: So you need to be very careful to understand which level your organization and which level it wants to head to. What is the business outcome or business goal here? In that way, you can make a decision with BUILD when it comes to purchase. For us also, we had this discussion in our company on few cases, like, do we wanna purchase few of the things off the shelf?
[00:14:33] Sayanthi Ghosh: So in that decisions or discussions, what we generally see is the roi. If I purchase, how much autonomy I will have, do I need to customer a lot of stuff? Will it like shift and shift? And that should be great and work for us. And then that's okay. If not, if you see lot of customization because I have four different sites.
[00:14:54] Sayanthi Ghosh: All are coal size, but they have different problems. They have different material. So if I [00:15:00] buy a product and if I have to keep on customizing on everywhere, is that cost effective? If not, then I would rather build something which works for us. So, so these kind of decision on build and purchase and then maintaining maintenance is a big thing, right?
[00:15:18] Sayanthi Ghosh: You can purchase, you can build, but everything comes out at the end of the sustainability. A solution is not a solution if it is not sustainable. So if you are not able to sustain it for various reason, and if that adds to your cost and if it is not giving you the r i that you thought or envisioned, then you need to make a trade off there.
[00:15:39] Sayanthi Ghosh: So those are the few points I would say. Keep in mind when you are talking about builder purchase decision,
[00:15:46] Himakara Pieris: If you're making a long-term investment in ai with the understanding that is core to your business strategy, what would be the team structure that you're thinking about? What are the key roles? And [00:16:00] share some insights on the sequence of these roles,
[00:16:05] Sayanthi Ghosh: this is how I would tackle this. So I have a engineering team so I can get. My hands dirty with the data if I have the data, but then I would need someone who understands the problem, so I will pick someone from that company who has a business acumen and understands the business either.
[00:16:30] Sayanthi Ghosh: It's a business it's a product owner or a product manager, a program manager. Different companies have different roles who work with business. So somebody from business who understands the business and the problem, who can frame the problem. I would first get one of that person and I will get a person from the engineering team who is typically a data engineer who can work and transform the data, like get the data.
[00:16:55] Sayanthi Ghosh: Translate it, transform it, and then help another [00:17:00] person to work with the data and that another person will be a data scientist for sure. I'm gonna get a data scientist I need,
[00:17:08] Sayanthi Ghosh: so why do I need to do that one? I need someone to understand the business and understand the problem and frame the problem. So that's why I need someone from business.
[00:17:18] Sayanthi Ghosh: I need someone who is pro with. Dealing with data, accessing data, working with data transforming that data, knowing where it is, what, and putting it in my system, right? Because I need that data. That's why I'll get someone from data engineer. Now, I need someone who can do the analysis from the insight, and that's a typical data scientist.
[00:17:40] Sayanthi Ghosh: I won't, , take a machine learning engineer into my team right now because I don't know if I need it. So I, first, I need to crack the problem, understand the problem, get my hands dirty with the data, see if I have enough data, and I need someone who can tell me that. And that's why I need a data scientist.
[00:17:58] Sayanthi Ghosh: I will make my team like [00:18:00] this let's say a product manager or a product owner. A data scientist, a data engineer, and what I would do is maybe not in the first, very first go, but after I get the hang of the data and understand, yes, we have some sort of opportunity here where we can work with, probably the next person I would get into my team is ux.
[00:18:29] Sayanthi Ghosh: You have great data, you have state-of-the-art ml, but at the end of the day, it goes to the user. If today's AI is not user experience improved or equipped, your solution falls flat. Think about chat, G P D, just the big thing that is going on, it does amazing thing.
[00:18:52] Sayanthi Ghosh: But today, If you didn't have a simple user experience where you can just type in, what are you gonna do with that [00:19:00] solution? If it is too complex, no one is gonna use that. So, usability. So for any product, there are four key pillars,
[00:19:06] Sayanthi Ghosh: value, risk, business risk, usability, risk, and feasibility. So I need to go through these pillars to make sure, so I would build my team who can work on these pillar. For business, I need that business acumen. So product manager or owner. Feasibility risk. I would get the data scientist and the data engineer so that they, the data engineer will help the data.
[00:19:31] Sayanthi Ghosh: Scientists without these two cannot work alone in silos. They have to work together. And then for the usability risk, I would need my ux. To create or give a prototype or understand the user journey, forget about prototype first, first to understand the pain point and user journey. So probably my team will become a product manager, a UX designer, a data engineer, and a data scientist where I check these, , three pillars.
[00:19:58] Sayanthi Ghosh: And that's how [00:20:00] I'll start.
[00:20:00] Himakara Pieris: What are the different stages of. Maturity from their on. If that's your base state, , how would you think about growing the team and what are those different stages?
[00:20:08] Sayanthi Ghosh: This team. Right now we are in a very, This is a very beginning stage, this is the first stage where you are actually seeing, do I even have a problem and do I even know if AI is a solution? If that is the case, do we, is there any possibility to form that solution?
[00:20:29] Sayanthi Ghosh: Do I have the enough data? So I am in a very, very level, zero stage right now. Let's assume that we cross that stage. We do have a co problem in hand. AI is the solution, and we do have data, maybe not everything, but we have enough data so that we can form a solution for this, and then eventually iteratively, we will keep gathering on the data, ?
[00:20:53] Sayanthi Ghosh: Because with time you gather data and you then retrain your model with the new relevant data. So that's your first [00:21:00] stage, and once you cross that stage, and if I assume yes to all these questions, my next stage is experimentation. What is that? I need to understand that the solution that I am proposing or thinking is the right thing to build, which will solve my problem.
[00:21:17] Sayanthi Ghosh: Now when I talk about experimentation. Depending upon the problem and the breadth of the solution. If you need to add more team members, that could be another data scientist or a data engineer given what is the breadth of the problem and how much is the work? Probably, I'm not gonna add any UX designer.
[00:21:36] Sayanthi Ghosh: One is enough. Product manager one is more than enough. And then what I need to see if it is machine learning and if my data scientists have all the skills to. Pull that, or if I need a machine learning engineer. So that depends on your solution.
[00:21:50] Sayanthi Ghosh: I won't load with too many team members at this time because I'm still in experimentation.
[00:21:56] Sayanthi Ghosh: After this stage, let's again assume your outcome of [00:22:00] your experimentation is positive, where you are building the right thing. You validated with your stakeholders. You validated yourself, you have your set kpi.
[00:22:09] Sayanthi Ghosh: Let's say you validate and you have cleared your experimentation. Next, you are going into productionizing IT active stage here. You need to have a full SLEDGED team of DevOps active because now you need to build a ui. So far, you just had mocks, you had prototypes. UX designer was perfect, but now somebody needs to build it data science or ai ML is not out of software engineering. So you still need that software engineering layer to it, right? So now you need the whole team. Now, once I have satisfied with my experiment, then you would need a whole team of your DevOps app devs your So for us it's React dashboard. So I have UI developers and I have machine learning.
[00:22:53] Sayanthi Ghosh: I do not have machine learning engineers. I have data scientists who does the machine learning bit also. So that's why I said it [00:23:00] depends how you wanna coin it. Then data engineer, very, very critical. Another part. Now, once you are productionizing, slowly, you are moving into a phase of maintaining and sustain. How do you do that? You need to gather not just your KPIs, but you need to keep gathering your feedback from customer. You need to see the customer behavior for that. You would build some dashboards, depends upon your company if they wanna build it in react.
[00:23:29] Sayanthi Ghosh: Then probably the same engineer who work for your front end could help you. But for us, we build it in Power bi. So I have two, BI developers in my team, depending upon different, various products they are attached with. So their job is to create those dashboard, which helps me understand customer behavior.
[00:23:48] Sayanthi Ghosh: My machine learning models behavior, my overall product behavior and everything, like different tabs and et cetera. So you would need bi developers or you would [00:24:00] need some person who would help you create those dashboard given what technology you use. So that's when you are in the level where you have productionized it and you are in the last leg where you are sustaining your AI solution.
[00:24:16] Himakara Pieris: When you think through the project lifecycle, how does the lifecycle compare to your traditional software development lifecycle
[00:24:24] Sayanthi Ghosh: That's a very, interesting question. I have seen this confusion happening a lot in my organization as well. So we come, so majority of people experience they come with an experience of software development lifecycle, I don't wanna bring waterfall into it, which is a different way. So let's talk, let's stay in Agile. So so in, even in Agile, the softer lifecycle and having a ai, it's a different lifecycle. The reason it is different, because think about it. You [00:25:00] have an understanding of problems. Same for both. Great. Now you know your customer.
[00:25:04] Sayanthi Ghosh: You know the pain points, you prioritized it. You did a product discovery. You did a continuous discovery. You created your opportunity solution tree and et cetera, and then you prioritize and now you are all set. But if your solution contains ai, you have to be very heavy on data because you need to train.
[00:25:28] Sayanthi Ghosh: But if your solution is software development, it's a rule-based system, you still need data, but then you do are not training something, you are writing rules and the system or the workflows through those rules and giving you some outcomes. So it's still a rule-based, so that's a difference. Second, and the most important difference that not just the product manager, but also the organization or the higher leadership has to understand is time.
[00:25:56] Sayanthi Ghosh: For a software development lifecycle, writing your [00:26:00] development time and effort is writing those rules and testing, which we know, and then figure getting the feedback and iterating. However, when you come to ai, if you have a machine learning model, first you need to get that data to an e d a, and you have to.
[00:26:17] Sayanthi Ghosh: Train the model and then you go to production. If there is something different, then you come back and you again change the hyper parameters, you change the model and you again have to probably train them or retrain the model. So when I say train and retrain, what I'm tr essentially saying, you need data, you need relevant data.
[00:26:36] Sayanthi Ghosh: If you do not have it handy, you need to wait. So the time, this layer of training, this layer of build is different than, very, much different than the normal software development model. And then maintenance. Let's come back to maintenance again. That's the very critical part. Sustaining your solutions think about it. You have a rule-based system. Some business has changed. You went [00:27:00] in, you checked, checked the impact, you went you changed your rule, you. Like updated your program and, and all the impacted other modules and everything. But if there is no change in the rule, no other feedback, you can pretty well sustain it the way it is.
[00:27:19] Sayanthi Ghosh: But for a machine learning model, we have to retrain our model every three months. We have to, if we don't, so it does, it is not relevant anymore because something that. Things have changed in the plant, whole body has changed. So maintenance of ai, or AI or ML product is different than what you would do for a rule-based
[00:27:42] Sayanthi Ghosh: system.
[00:27:44] Himakara Pieris: How do you think about success? How do you define success and how do you measure success?
[00:27:48] Sayanthi Ghosh: When you talk about KPIs or success, You have to make sure that you categorize your KPIs. One category will be leading to revenue KPIs. Those are typical the business [00:28:00] KPIs. How will you know that the outcome is achieved?
[00:28:02] Sayanthi Ghosh: However, if you have a AI product or an ML product, you need to make sure that if you are performing it right or not, and then there will be another category that will be usability of the product. So let's talk about the business and the link with the ml, . For me to give an, , easy example, so I have a recommendation system.
[00:28:24] Sayanthi Ghosh: The goal of the recommendation system in high level is to increase the production of, , clean coal, but then I have three machine learning models. Think about it. If I just look into the clean call production, and then suddenly I start seeing, okay, my percentage of production is going down, do I know where exactly is the problem?
[00:28:47] Sayanthi Ghosh: I don't, because it could be wrong data. It could be something has changed in the plan. It could be my ML models, which is not doing what it is supposed to do. So I need to understand and I need to measure the performance [00:29:00] of my machine learning model. And there are typical KPIs like. R Square, mean absolute error.
[00:29:05] Sayanthi Ghosh: There are very typical KPIs we look into. We also look into sharp plots. This is very, very hardcore on the data science section, but I also have to measure and understand those KPIs and keep a check on them to know that. Oh, now it is the time to retrain it, or now the model is behaving something different.
[00:29:24] Sayanthi Ghosh: We have the same data. We don't need a retrain, so what's going on? There must be something. So to drive those type of action, the one of my KPI is measuring those models. Then the ultimate goal. Your success criteria, the business level success criteria. You can have a great machine learning model with amazing R Square, very low error and et cetera.
[00:29:46] Sayanthi Ghosh: But then if my business goal or business KPI is not coming to the target, which I have, it's pointless. So either the solution has to change, something has changed, there has to be some action. So it's, it's all the time. You have to keep [00:30:00] keep on measuring your business KPIs. That is your leading to revenue KPIs, your model performance KPIs.
[00:30:06] Sayanthi Ghosh: Those are the typical model performance and data science, KPIs, different business, different sort of KPIs. And then for me, usability kpi, like because it is a recommendation system, I need to keep a check acceptance criteria, acceptance percentage. If they have accepted and they have complied, then compliance percentage.
[00:30:24] Sayanthi Ghosh: That's my usability. K P I.
[00:30:27] Himakara Pieris: Shanti, thank you so much for coming on the podcast today and the insightful conversation. Is there anything else that you'd like to share with our audience?
[00:30:35] Sayanthi Ghosh: Thank you Hima, for giving me this opportunity. It was great having this conversation, and the only thing that I would highlight all the time is start with your customer and their pain points from the problem, and then work backwards from it and rest of the things will fall in place automatically.
[00:30:52] (Outro)
[00:30:53]
[00:30:56] Hima: Smart products is brought to you by hydra.ai. [00:31:00] Hydra helps product teams explore how they can introduce AI-powered features to their products and deliver unique customer value. Learn more at www.hydra.ai
-
I'm excited to bring you this conversation with Spurthi Kommajosula. Spurthi is a portfolio product manager at IBM. During this conversation, she shared her thoughts on the types of exploratory questions an AI PM could ask to discover optimal use cases, how to communicate effectively with other stakeholders, and AI governance.
Links
Spurthi On LinkedIn
Transcript
[00:00:00] Spurthi: The first question as product managers, we should ask ourselves, and really anybody who you're involving within your AI conversation, is what data is available right now?
[00:00:10] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate, and learn from industry leaders who are solving real-world problems. Using AI.
[00:00:20] Himakara Pieris: I'm excited to bring you this conversation with
[00:00:24] Himakara Pieris: is a portfolio product manager at IBM. During this conversation, she shared her thoughts on the types of extra for the questions an AI PM could ask to discover optimal use cases, how to effectively communicate with all the stakeholders, and AI governance. check the show notes for links. Enjoy the show.
[00:00:42] Himakara Pieris: welcome to the show.
[00:00:44] Spurthi: thank you so much for having me. It's very exciting.
[00:00:47] Himakara Pieris: Tell us a little bit about your background and what you're working on.
[00:00:51] Spurthi: I currently am a PR portfolio product manager for business analytics at IBM. So essentially what that means is I manage a portfolio [00:01:00] consisting of Cognos Analytics, planning, analytics, analytics, content hub, business analytics, enterprise and Cognos controller. If you've been in the analytics space , you probably have heard of at least one of these products given at their longevity and how how involved they are in the analytics space.
[00:01:16] Spurthi: It's been an exciting role. I get to touch upon a lot of these products. They fall under the scope of data and AI at IBM which is a very exciting place to be in. Apart from that, I'm also an adjunct professor at Conestoga College where I focus in on analytics and data and AI based courses, so that's been really exciting to convey to people the importance of AI in this world.
[00:01:40] Himakara Pieris: Sounds like the product Suite is focused on making the power of AI available to the masses to the sems, et cetera. Is that right?
[00:01:48] Spurthi: Yes, that's exactly what it is. These products are more so b2b. So they target medium and large base card organizations or enterprises that really want to, in, in like improve their, [00:02:00] I would say, BI ecosystems their planning ecosystems. So that way not only are you just planning and forecasting and improving how you handle your data within your organization, but also you're incorporating AI when you do that.
[00:02:13] Spurthi: So what that essentially means is when you do your forecasting or when you do your budgeting, you're involving AI to in ensure and help you predict and get more information from the data and insights that you have, and use that to really drive the decision making within your organization. So the best case example that I can give you is say for instance, you are a small time retailer that essentially retails in, say, coffee.
[00:02:37] Spurthi: So if you're in the coffee space, you really wanna know how much coffee products you need to buy, how many cost of what's the cost of goods sold within this month. Et cetera. But at the same time, you need to plan how much to buy for the remaining year. What are my main months of sale? What are my sales looking like?
[00:02:53] Spurthi: How should I be targeting to meet my goals by the end of the year, et cetera? So when you [00:03:00] do that kind of level of planning, you really want to be able to use reliable insights to get that. Level of confidence to make your decisions in a more effective way. So that's essentially where my type of products come in.
[00:03:12] Spurthi: We really ensure that your entire business intelligence suite and your data and AI suite is set up to give you as reliable insights as possible. Insights that can really drive to make your decision making easier.
[00:03:27] Himakara Pieris: What would be a good example use case that we can get into?
[00:03:30] Spurthi: Let's take the example of L'Oreal Paris a big manufacturer and retailer of cosmetic products within the world. So typically during red carpet events, you would see a lot of different looks, a lot of different trends that come up in makeup and fashion, et cetera. So what L'Oreal saw this as a business opportunity, but essentially in order for them to know what was trending, what was Going to be sold later on.
[00:03:54] Spurthi: What would essentially be the product of the year that people would go after? They would need to analyze thousands, [00:04:00] if not millions, of data trends of trends within the red carpet at a very real time basis. So that's where business analytics comes in, helps them essentially target these data sources, these trends, and give them the insights that they need.
[00:04:16] Spurthi: To do their demand and supply planning so that way they're set up for the rest of the year following the trends that have been seen in red carpet events, et cetera. So that's essentially given them that opportunity of pursuing that business that business opportunity in a very, very reliable and fast paced manner with data backing up their decisions.
[00:04:35] Spurthi: That way they know what product to make more of, what product to essentially stock up on. So that way when they do go to market, they have enough sources to sell enough products to sell without any issues. And their planning and budgeting for all of this is done in a more seamless manner. So I think that's one of the best case examples that I can give you.
[00:04:54] Spurthi: Another really key one that's used AI in the last couple of years to make their [00:05:00] processes better has been this financial institution that's incorporated data discovery. Within their audit modules, so that way when auditors come in and take and do their audit processes, et cetera, they're able to automate and rely on AI to drive certain in insights and automate some of their redundant tasks and administrative tasks, which is essentially save them thousands of hours.
[00:05:23] Spurthi: And thousands of auditors have been able to essentially incorporate that and improve their audit processes. So not only are they able to audit better, but they're also able to audit faster and essentially save billable hours for a lot of the organizations. So if you look at, you know, AI right now in any of the spaces, you know, you pick supply chain or if you pick retail consumer goods, really just any aspect outside or even inside your organization, you can see that there is a space for ai.
[00:05:52] Spurthi: It's just how you kind of. Pave the path to get that space set up and utilized for you to ensure that [00:06:00] AI really just adds a value to your organization.
[00:06:02] Himakara Pieris: What are some of the challenges that you experience as a PM in the, AI space?
[00:06:10] Spurthi: Everybody wants to be involved with AI because it's become such a big buzzword. And what that essentially means is as product managers, you wanna cater to this demand, but you also want your clients to incorporate their data and AI-based solutions in the best way possible.
[00:06:26] Spurthi: Taking the example of say the audit example that I'd mentioned with the financial institution, they knew that they had to incorporate AI in some form or the other, but showing that they're such a niche place and incorporating it with audit. What's the best way for them to go forward? And the best way you can do that and have your clients understand where they can incorporate AI is through.
[00:06:49] Spurthi: Constantly having a conversation with them, understanding where the pain points are, and then trying to see where they, where your products can add value to it. If, you know, you go [00:07:00] into incorporating AI with the mindset of this is going to solve all of my organization's problems, then you're not really using AI to its capacity or its full potential.
[00:07:09] Spurthi: Really understanding the vision of what you wanna do with AI is, The best way to go forward, you know, and that may mean you have to incorporate it at some form or the other, or that may mean you may not exactly need ai, but you might need like an analytics based solution or you might need maybe a data management solution.
[00:07:27] Spurthi: So incorporating it really just depends on. Where your pain point is as an organization and how effectively can data and AI solve that? And the second aspect, I think the second challenge of it is, while people wanna get into the space of ai, they're also very scared of what that means. And they're very, very of of what the AI piece could mean, especially with the conversations of AI being unpredictable or not trustworthy or not explainable being so relevant in this day and age.[00:08:00]
[00:08:00] Spurthi: Again, all of these just come back to the conversation of having an understanding of where AI can sit within your organization and having a strong understanding of what goes into your ai. What type of insights are you getting out and how rele relevant and reliable are those insights. The best case example that I can give you is when you use Watson Studio, which is a product offered under the data fabric solution space under the data and AI banner.
[00:08:29] Spurthi: At I B M, as soon as you come up with your ML model or machine learning model, you get to see exactly the accuracy scores of the model. So that essentially tells you where the, the model stands in terms of its reliability, where the model is faulty, where the model is not accurate. You get a real understanding of what's going right and what's going wrong with your model.
[00:08:51] Spurthi: So take that and expand that example, and you need to make sure. Your AI and your machine learning and analytics tools are really [00:09:00] incorporated in a way that they're explainable to people. So the minute, you know, you explain and you understand something, you are less scared of it. You're always likely to be more scared of things when you don't understand them.
[00:09:11] Spurthi: So ensuring that your AI and your machine learning and your analytics solutions are really in a place where they're explainable to people, it really takes out the challenge of people being scared or wary of them in the future. But in this, in the space of ai, I think it's just a two, it's just two sides to the same coin of people wanting to get into and use ai, but people also being very of AI and always, you know, the best example of that is to, the best case for that is to have a conversation, understand the pain point, see where the use case is, and then incorporate ai.
[00:09:46] Himakara Pieris: If I unpack that sounds like you sort of touched on three core issues there. The first one is picking the right use case. Second verse, reliability slash accuracy. And the last one was explainability. Right? [00:10:00] Right. I'd like to get into each of them in a bit more detail, starting with picking the right use case.
[00:10:06] Spurthi: Yes.
[00:10:07] Himakara Pieris: Do you have a simple framework or rubric that someone could use to understand whether a given use case is appropriate or not? For ai?
[00:10:18] Spurthi: Let's touch back on the L'Oreal example. In the example that I'd mentioned.
[00:10:23] Spurthi: L'Oreal saw a business opportunity. They were able to seize a business opportunity that existed when they saw that a lot of trends were getting marketed out through the red carpet. So there was a clear business opportunity there. And then they used AI to follow that business opportunity. Now let's, you know, circle back on the financial institution incorporating AI into their auditing tracks.
[00:10:46] Spurthi: So when you incorporate AI into, or auditing tracks, what they essentially did was reduced cost. And that's another aspect of how you can. Get value out of ai. You can also have it in the sense of time savings [00:11:00] because now auditors spent less time on redundant administrative tasks and more time on actually properly auditing and utilizing their time more efficiently, which reduced billable hours.
[00:11:11] Spurthi: And as a result of that, saved a lot of money for the organization. So their two aspects of it, the first one is revenue management and incorporating revenue to drive with your ai. And the second aspect of it is increasing your efficiency and reducing cost as a result. These are, I would say, the two main ways that people like to play around with ai.
[00:11:33] Spurthi: But again, you need to really realize where you can go about it with. AI through understanding your pain points within the organization. So when you look at ai, it can just be as a bandaid solution to everything. It needs to be a niche way for you to know exactly what the pain point is and what the vision is for you to incorporate AI and utilize it better.
[00:11:56] Spurthi: Taking the example of L'Oreal, they saw [00:12:00] that this was a trend and their vision was incorporate ai. To increase the way that we market and improve our su supply and demand planning and business planning to cater to these trends. So they had a very clear vision and a clear problem that they were trying to solve.
[00:12:18] Spurthi: Taking the example of the financial institution, their main vision was reduced, time spent on redundant tasks. And increase efficiencies through ai. Another example that I can give you within IBM actually was including and incorporating ai as a way for you to reduce interactions for hr. So what they did was essentially create a chat bot that would answer every single question that a lot of HR representatives and business partners were answering continuously and often.
[00:12:52] Spurthi: So instead of going to an HR partner, now you have a one stop shop. Where it essentially answers everything relating to your payslips, to your [00:13:00] tax information, your, your personal information, any updates you'd like to make. Really just a very automated bot incorporating AI to answer and reduce redundancies within the organizations.
[00:13:13] Spurthi: So when you have a use case or when you're looking for a use case, Your first thought should be, what is the organization's problem that I'm trying to solve, or what is the pain point that I'm trying to attach AI to? So once you have that strong grip, you can go about it. And the only way to get that strong grip.
[00:13:31] Spurthi: Is eighth to have multiple conversations, client interviews, and really just have as many conversations with the stakeholders affected by that problem. Or have a little workshop where you can have a very open dialogue on what the current situation or the scenario is, how the business process flows, and then identifying a pain point through that.
[00:13:53] Spurthi: Typically consulting goes through a workshop, whereas I would say product relies more so on the first aspect of [00:14:00] it where you have a lot of conversations with stakeholders client interviews where you're in, in basically interviewing stakeholder clients and understanding where the aspect or the pain point lies.
[00:14:12] Spurthi: And once you have that understanding, and that's when you start incorporating ai. I think that's really just a very simple framework to go through. Once you have that understanding, then maybe an AI readiness checklist that a lot of organizations have would be a good way to go about it. But the first aspect of it, if not clear, it would not it would not essentially drive as much value as AI can potentially promise.
[00:14:40] Himakara Pieris: What would be some exploratory questions? Product manager could ask to discover features that would lend themselves well for AI and lend themselves for creating a, considerable business impact using
[00:14:54] Himakara Pieris: ai.
[00:14:55] Spurthi: I think when you go through these kind of questions, they would be essentially dropped into two main [00:15:00] buckets.
[00:15:00] Spurthi: The first one would be strategic, and the second one would be technical. Now, in your strategic questionnaire, I think the first question that you should essentially ask when you're thinking about AI is, or, you know, creating a product, managing a product, improving features on a product is, what do I want to drive or what issue am I trying to resolve?
[00:15:20] Spurthi: Once you start at having your questions based around this philosophy of what problem am I trying to solve, I think that would be, A good way to get started and then it only builds up on that because then you can have a little bit more of a deep dive and understand the niche area that you're trying to resolve.
[00:15:39] Spurthi: With the technical questions, I think you can start with the data. It always should start, in my opinion, with the data because inherently AI is just data science, which is inherently against stats and math. So when you think about it in that lens data, which should, should, and would be the starting point.
[00:15:57] Spurthi: And the first question as product [00:16:00] managers, we should ask ourselves, and really anybody who you're involving with in your AI conversation is what data is available right now? And data by itself in this lens can be, I think, categorized into transactional, operational, and master data. And master data consists more so with like personal information and so on.
[00:16:19] Spurthi: And once you have these like data sources identified, where does the data within your organization flow from? The next question should be, what data in this sense can I use? And where exactly is my data going to be used? How is it going to be used and what data usage and purpose is there really at it, at, at the crux of it?
[00:16:41] Spurthi: So say for instance, coming back to the supply chain example that we gave. If a client comes up to you and says, I want to incorporate AI in my enterprise warehouse management system, well that's great. Okay. Then what questions should you ask them in the first place is where do you, what, what is the vision?
[00:16:59] Spurthi: Where do [00:17:00] you see AI help you? And once you have that answered, then asking the questions of, okay, what are your data sources? And in most cases for supply chain, it would be operational and transactional data. That's how information flows in and out. And then asking the question of how do you wanna use this data?
[00:17:16] Spurthi: And this ties back to the question of what is the purpose that you're trying to drive? Because say for instance, they come back and say, I wanna use this to. Forecast better. I wanna use this to predict how much inventory I need to keep on hand. Then the answer to what is the vision and what is the problem you're trying to solve ties back in and it ties back in seamlessly.
[00:17:36] Spurthi: And that should be a check for you. Because if that answer ties back to what is your vision and purpose, then you're on the right track. If the answer doesn't tie back in, then maybe you're trying to solve more problems with ai which is also okay. But then each problem will have a unique AI solution.
[00:17:53] Spurthi: So I think that those ways of questioning and really managing your answers and [00:18:00] evaluating the answers should be a good checklist to understand what you need to build, how you need to build it, what features need to be prioritized first, and so on. I know typically we use a lot of enterprise design thinking also to incorporate these thoughts and the feature prioritization, especially.
[00:18:18] Spurthi: But in all honesty, these checklists really ensure that you are going on the right direction.
[00:18:25] Himakara Pieris: It sounds like there are two parts of entry into this process. One is starting with your data, taking a holistic look at what data you have because that's your key driver, and figuring out ways you can create value and then assessing whether that value is going to actually create.
[00:18:41] Himakara Pieris: Significant enough business impact . And another track could be looking at pain points. Yeah. , and starting from there. And I can imagine both these trucks running permanently in, certain cases as well.
[00:18:54] Spurthi: Exactly, yes. Even if they, for instance, you know, you do one and then come to the other, or you do one [00:19:00] and then you do the other and come back to the first one.
[00:19:02] Spurthi: Both aspects are fine as long as they tie to each other. It can be so that, you know, my pain point is data discovery and intelligent data understanding and augmentation. And then you coming back and saying, my data needs to be in a place where I can predict better information, because then those two don't tie to each other.
[00:19:22] Spurthi: You're trying to solve two different problems and you're taking one single singular approach. So as long as you know either questions are being asked and they tie to each other, I think that's the good way to check whether you're going in the right direction.
[00:19:36] Himakara Pieris: How do you think about minimum acceptable performance levels for an AI product that you're working on?
[00:19:43] Spurthi: That really depends on the use case that you have. And also I would say it depends on the industry . Now, let's start with the industry and then we'll get to the use case.
[00:19:53] Spurthi: Minimum acceptable levels for an industry such as banking would be far higher than [00:20:00] say, With supply chain or with consumer goods. And then minimum acceptable testing levels for healthcare would be very different to how you would expect it to be for retail. So again, firstly, understanding whose problem you're trying to solve.
[00:20:16] Spurthi: What industry are you working with? Is the first key understanding of setting that baseline level. Because if, you know, you're dealing with healthcare data, you want it to be at like the highest level of possible accuracy. Even for that matter when you're working with retail, you do want it to be at a higher level, but it's just, you know, you're playing in completely two different arenas with healthcare finance.
[00:20:40] Spurthi: Retail. So understanding which, you know, industrial, you're playing in first and then going into the use case would be a very good way. Now when you go into the use case, again, it would change from, say, when you're working with like a use case that's more customer facing versus a use case that is a little bit more backend, [00:21:00] the minimum acceptable levels of testing and the, the people and the culture you're testing with.
[00:21:05] Spurthi: Change. So as a result of that, again, keeping in mind the use case that you have would be a very good way to set that baseline. And once you have that baseline ready, communicating it with your data science teams and ensuring that you know you are setting that index at the right place would be a good way to get that validated and checked going forward as well.
[00:21:27] Himakara Pieris: Speaking of communicating with the data science team do you have any suggestions or advice to your fellow PMs on, on how to effectively communicate with, with your technical counterparts?
[00:21:41] Spurthi: Oh, a hundred percent. I think one of the key aspects of product management is essentially taking a piece of information.
[00:21:48] Spurthi: And then curating it to ensure that it's palatable to all of your stakeholder teams. So the way that I would speak to my marketing focals would be very different from how I would speak to my technical teams, [00:22:00] and that would be very different to how I'm speaking to sellers. So keeping in mind the audience that you're speaking and what the audience is looking for and what you know, what value they're trying to drive from that conversation is.
[00:22:12] Spurthi: Really the crux of product management. And that's why, you know, it's such a interesting intersection of all of the different stakeholder management groups that you have within the entire industry. Now taking the example of speaking to a technical team member, I, yeah, like, I mean, taking that information and then communicating it to a marketing focal would, you know, be.
[00:22:36] Spurthi: I would say like, you know, you would have to change the terminologies and, you know, you need to really understand the audience group that you're talking to because if not information can just go over people's heads or information might not be interpreted with as much like deep level of understanding that you're seeking.
[00:22:54] Spurthi: And also like each agenda, each group has a different agenda. You know, technical teams wanna get the product out [00:23:00] with as much, you know, speed and as much accuracy as possible. Marketing would really like to hone in on the features and what value added drives to the sellers. And the sellers also would have a similar mindset, but they're essentially looking for a way or like a value add that you know they can seek.
[00:23:17] Spurthi: So when you're as a product manager, standing in the middle of all of it, When you get your information, you really need to take a second, absorb it. As much as you can because then you need to feed that information in different verbiages and different strategies to all of your consumer groups. So that way the product by itself is set for success.
[00:23:36] Spurthi: Because if your marketing focal is not in alignment with what's being sold, then your seller groups are also not going to be in alignment, and then the product is not going to do as well as it can. So first and foremost, I think the best advice that I can give is know your audience. Know how to. Know how to contribute to them and know how to interact with them, and that's how you're only going to seek your value out in the future.
[00:23:59] Himakara Pieris: Is there anything [00:24:00] else you'd like to share with our audience?
[00:24:02] Spurthi: I'd really like to touch upon the piece of AI governance. I think it's a very interesting area right now with especially the, the conversations that we have that are happening about the reliability and explainability of ai. When you think of AI by itself, you really, you know, the best way to like translate AI governance is Like the value add is, you know, when you write a book, you're not going to take it straight to publishing.
[00:24:29] Spurthi: There's going to be an editor, there's going to be somebody who does the spells and the checks and then it gets published. AI governance is in the same place where, you know, there are people who are essentially, or, you know, technologies that are essentially in place to ensure that your AI piece or your machine learning piece is in a place where it's checked.
[00:24:53] Spurthi: Validated and then sent off to the market or sent off to deployment, however you choose to use your [00:25:00] product. The importance I think of AI governance now more than ever is at its peak. And the only way to make AI less scary to people and more trusting to people is through AI governance and having it incorporated in, as in every aspect of, you know, when you include yourself in data and ai.
[00:25:18] Spurthi: Every aspect of that needs some level of governance, be it on the data side or on the AI governance side. So I think, you know, when you think of ai, you should always go hand in hand with AI governance.
[00:25:30] Spurthi: So what would
[00:25:30] Spurthi: be your framework that PMs could operate within or think about in terms of AI governance?
[00:25:38] Spurthi: Like what are some of the pitfalls we should plan on avoiding?
[00:25:41] Spurthi: I think, you know, firstly, Okay. I would say firstly, when, you know, we humans by in our nature have biases. So say for instance, you know, when you write your AI models, you're writing it with, not with the intention of having your bias, but the bias inherently falls in by itself.
[00:25:59] Spurthi: [00:26:00] So ensuring that, you know, I would say the framework that I would, you know, particularly touch upon is just threefold. A, you need to make sure that you are explainable. When you put in whatever aspects of data or whatever variables you're using to create a model or to create a chat bot, you really need to think about how can I explain this and should I be using this data?
[00:26:24] Spurthi: It's not just that data's available and then you should go about using it. The minute data's available. Take a look at it and ask yourself, should I be using all of this or should I be using this? Is it ethical to use it? And that's where you know, if it's a yes, then it's a yes. If it's a no, that's where.
[00:26:41] Spurthi: You know, you should step back and not use that data. So the first step of that, Is explainability. So knowing exactly what variables came in and exactly what variables came out. Number two, validity. To ensure that your AI models are reliable, they need to be validated. Now, the validation can happen through the AI governance tools that are there, or through a group of data [00:27:00] scientists.
[00:27:00] Spurthi: Really just however you choose to validate. It should still be validated regardless. The validity portion of it, I think would be a very key IT audit point also for your AI chat bot or, you know, intelligent document, understanding knowledge catalogs, really wherever your AI touches upon. I think it's an very important point for ai audit or IT audit.
[00:27:22] Spurthi: So ensuring that the information's validated and checked off is always a key aspect of it. And lastly, making sure your outcomes are reliable or trustworthy. You really need to ensure that your AI is in a place that's trustworthy and your AI is, your outcomes are trustworthy by itself. And that always happens if the first two aspects of your framework are met.
[00:27:48] Spurthi: When the first two aspects of your framework are met by nature, your last aspect of it being trustworthy would be met. But to ensure that these frameworks are maintained, you need to constantly go back, [00:28:00] tweak, look at your model and see what it's missing and improve your model over time. And you should always continue doing that as much as you can to ensure that your model is wholesome and not inheriting any biases that are pre-existing.
[00:28:14] Spurthi: I also know that a lot of organizations have their AI governance boards. So having a constant conversation with them and incorporating them on your AI journey should always be a go-to as well.
[00:28:25] Himakara Pieris: Thank you, Spurthi, it was great having you on the show today.
[00:28:28] Spurthi: Thank you so much for having me.
[00:28:29] Spurthi: It was a pleasure.
[00:28:36] Hima: Smart products is brought to you by hydra.ai. Hydra helps product teams explore how they can introduce AI powered features to their products and deliver unique customer value. Learn more at www.hydra.ai.
-
I'm excited to bring you this conversation with Anand Natu from Meta, where he is a product manager working on responsible AI.
Before Meta. Anand was at Amazon, where he worked on brand protection using AI. During this conversation, Anand shared his thoughts on building large-scale AI systems, how to think about risk, and common mistakes AI product managers make.
Links
Anand On LinkedIn
Transcript
[00:00:00] Anand Natu: Big data walked so that AI could run in the sense that it was the democratized language of , using data to drive decision-making that eventually became more sophisticated forms of artificial intelligence.
[00:00:12] Hima: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate, and learn from industry leaders who are solving real world problems. Using AI.
[00:00:22] Himakara Pieris: I'm excited to bring you this conversation with Ananta natto. Lotto. Annette is a product manager working on responsible AI at meta. Prior to meter on and was at Amazon where he worked on brand protection using AI. During this conversation on unshared, his thoughts on building large ScaleIO systems, how to think about risk and common mistakes. AI product managers make check out the show notes for links.
[00:00:45] Himakara Pieris: Enjoy the show.
[00:00:46]
[00:00:47] Himakara Pieris: Anand, welcome to the Smart Product Show.
[00:00:50] Anand Natu: Thank you for having me.
[00:00:51] Himakara Pieris: To start things off , could you tell us a bit about your background, what you're doing now, and your journey, , in product management in ai?
[00:00:58] Anand Natu: Academically, my background's in engineering traditional engineering. So my undergrad was in chemical engineering, and then I spent the first few years of my career as a management consultant working across a variety of industries, but with some focus on the technology space.
[00:01:14] Anand Natu: I first got into product management. And got interested in AI kind of around the same time because I, I, number one, kind of wanted to switch into product management to be closer to building things, which is one of the things that I really missed after a few years working in consulting where I was a little bit further from that.
[00:01:32] Anand Natu: And then the, you know, secondarily, I was, at the time at least interested in robotics and autonomous vehicles, and obviously AI is a big driver of innovation in that space and. That was kind of the motivating factor for me to do my master's in, in robotics at Stanford, which is what I did to kind of transition from consulting into product.
[00:01:52] Anand Natu: And as it turned out, I never really ended up doing that much professional work in that space as my first camera role aside from an internship in, [00:02:00] in the mobile game space, was actually at Amazon. After finishing my Masters and at Amazon, I spent several years working in the consumer business, specifically on a team called Brand Protection, which is among other things, you know, the, the overarching mission of the team is to develop security and commerce features that power the experience for brands on Amazon.
[00:02:24] Anand Natu: Mostly for the third party selling business. And so, you know, Basically the, the purpose of what our team's work was, was to create a better environment on Amazon for brands to organically grow and develop their presence and connect with the type of buyers and the type of audience that they're interested in marketing to.
[00:02:39] Anand Natu: And, and basically compete with direct to consumer. Options and channels in the process. During that time, I worked that, that was kind of my first experience working with AI in a product management capacity. We worked on a number of different AI driven initiatives, including, you know, a big one that we'll get more into detail [00:03:00] on later involving figuring out how to use AI to basically identify the branding of products at massive scale massive scale, in this case being the entire Amazon catalog.
[00:03:11] Anand Natu: And then after Amazon, I transitioned into my current role at Meta, which I've been at for just over a year now on the responsible AI team here, which is used to be just part of the central AI org and is now part of the central social impact organization in Meta. And I specifically work on fairness within responsible ai.
[00:03:30] Anand Natu: So, We sort of research, develop, and ship software that's designed to ensure that all of the production AI systems used at meta work fairly and equally for different user groups on our services. And the, the scope of our team is fully horizontal, but in the last few years we've worked on just some of the more high priority business products that meta operates, like our ads, ads, products for, you know, ads, personalization, targeting some social.
[00:03:57] Anand Natu: Some models that power social [00:04:00] features on Instagram, so on and so forth. And, and we also do some stuff in more sensitive areas like content integrity and moderation and, and things like that.
[00:04:09] Himakara Pieris: Let's talk about the brand protection use case. Can you tell me about what was the driver, what did the original business case look like ?
[00:04:17] Anand Natu: I'll start with a bit of context on kind of what I was working on and why it's important to to my org to Amazon in general. So at the time I was owning a program within my team called ASIN Stamping. So ASIN stands for Amazon Serial Identification Number. It's basically just the unique ID for a given product within the catalog.
[00:04:37] Anand Natu: The point of the ASIN stamping program is to basically identify. For every single product in the catalog, the brand that that product belongs to, and stamp it or just basically populate a field in the catalog with a unique identifier called the brand id. And there's a separate data store called Brand Registry that is the sort of authoritative source of all brand IDs that exist within [00:05:00] the Amazon store worldwide.
[00:05:02] Anand Natu: The reason why that's important is there are many reasons why that's important, but they fall into two big buckets. The first is catalog protections. If we know if we have an authoritative signal for what brand each product belongs to, we have a much more robust mechanism through which to vet and validate authentic branded selection on our platform.
[00:05:23] Anand Natu: Prevent counterfeits, prevent malicious sellers from representing themselves as brand agents, so on and so forth. This is, this becomes really important for major brands who. Frequently have their stuff counterfeited, like luxury brands the N F L, things like that. There are several high profile examples of counterfeits becoming prevalent on Amazon in the past, which is part of the reason my team was created in the first place.
[00:05:48] Anand Natu: So the security side is one big incentive for the business case of this program. The other is commercial benefits, which basically are related to. You know, features [00:06:00] ranging from ad targeting to advanced sort of content creation tooling that allow brands on Amazon to better represent and sort of.
[00:06:10] Anand Natu: Show off their brand within the context of the Amazon e-commerce experience. And the purpose of those features is to basically better, like I said at the beginning, better allow brands to develop their brand equity and brand image within the Amazon ecosystem and connect to the kind of audience that they want to connect to.
[00:06:29] Anand Natu: So that they start seeing Amazon as an actual home for brand development as opposed to what sellers historically have viewed Amazon as, which is, as you know, this, this e-commerce channel that's helpful for pushing volume and getting sales, but not necessarily a great place to like develop your brand image or find your really loyal, kind of like high retention customers.
[00:06:50] Himakara Pieris: The two scenarios that you talked about there, one sounds like you're addressing a pain point, counterfeiting of products and which has a lot of implications. And then the second one sounds like you're [00:07:00] trying to create new value by providing a platform where the vendors could create brand equity.
[00:07:06] Himakara Pieris: What was your process to assess the impact of these two sort of drivers at the start?
[00:07:15] Anand Natu: Each of those two cases are directly sort of enabled or provided by features that basically leverage the data that the ASIN stamping program creates in the catalog. So the point of the stamping program is basically just to look at the entire catalog, which at the time was about three and a half billion products in total, give or take, and make sure that as many of them as possible have a brand id.
[00:07:40] Anand Natu: Stamped on them. And that globally, the, the data that lives like the brand ID data across the entire catalog maintains an accuracy bar of 99% or higher. So that was the internal quality bar reset. And the mandate of the team was to basically push towards full coverage of the catalog with this high fidelity brand identity information [00:08:00] on all products listed on Amazon.
[00:08:04] Anand Natu: The value of, so, In terms of the business impact of the downstream features, the power we typically represent that in in different ways depending on what we're looking at. If we're looking at commerce features, like things that help brands connect to customers, things like that, we would typically use things like downstream impact studies that demonstrate incremental lift in G M s or gross marketplace sales that result from brands identifying themselves as brand representatives.
[00:08:30] Anand Natu: Basically going through all the relevant steps to. Onboard their brand into Amazon and leverage the benefits that we make available to vetted branded merchants on Amazon. And I, I forget the exact numbers, but we basically had some DSI studies that then turned into public facing marketing material to basically demonstrate the incremental revenue that brands stand to gain by engaging with the commercial side of the feature set that we provide to brands, which is powered again by the, the data that the [00:09:00] ASIN stamping program.
[00:09:01] Anand Natu: Puts into the catalog for all of their products on the security side. This, again, this is mostly done with the intent of protecting catalog quality and catalog integrity and reducing instances of basically like bad action. And we would choose to represent that through a few different measures.
[00:09:20] Anand Natu: But the most prevalent were. Just the overall rate of catalog abuse, which can look like a number of different things. But so, so we'd, we'd want catalog abuse rate to go down customer perceived duplicate rate, which is basically if you, if you don't have high fidelity brand identity differentiation, you end up with a lot of undifferentiated samei products, which harm harm the catalog experience and basically like make the selection look.
[00:09:47] Anand Natu: Lower quality overall. And then the third is customer complain rate, which just basically is just anything relating to a customer ticket that's filed after they receive a product.
[00:09:56] Himakara Pieris: When you go
[00:09:57] Himakara Pieris: through that process of essentially quantifying [00:10:00] the downstream impact of what you're building , what was the output like? Is that a business case document? Can you talk about. How that's presented internally?
[00:10:13] Anand Natu: Typically the impact is presented in the form of, of docs. You know, at Amazon, pretty much every business process is powered and aligned on through narrative style docs.
[00:10:22] Anand Natu: So I obviously I worked on this program after its inception. We were, at the time that I joined it, we had basically started the program, but we're in the process of meaningfully scaling up coverage because we still had huge gaps of like large swats of the catalog that didn't have High Fidelity brand identity information stamped onto them, which was basically what my work at that time related to.
[00:10:49] Anand Natu: But in terms of presenting the business case, yeah, that's basically done through through docs. It's done through goal reporting metrics that are presented in weekly business reviews, things like that. [00:11:00] So the, the direct impact metric that, that the stamping program uses is coverage. So basically the proportion of.
[00:11:09] Anand Natu: Products that are stamped both on a raw basis and on a weighted like impressions basis. Cuz you obviously want to prioritize products that have proportionally higher impressions that we'd count impressions on a trailing 12 month basis to make sure that we're focusing on the important products first.
[00:11:26] Anand Natu: And then that data, but obviously feeds into a lot of downstream. Team businesses that each use their own metrics as measures of impact, like, you know, brand. The brand program is the partner team we work with who manages a lot of the commercial sides of the features and their gold. More on direct like revenue metrics like G M s and things like that that sellers generate on, on the marketplace.
[00:11:52] Anand Natu: Processing , I think you said three and a half billion items in the catalog. That sounds like a huge undertaking. What did that look like? [00:12:00] What were some of the challenges that you experienced as you're going through that process?
[00:12:05] Anand Natu: I think when you start working at a scale that massive, you begin to realize that deterministic or like absolute methods of solving a problem.
[00:12:17] Anand Natu: Work to a degree, but they will never work exhaustively. So another way of putting that is every single, pretty much every single edge case you can think of about how a product behaves in the catalog or why we, why it's easy or hard or confusing to identify the brand. It probably exists and it probably exists to a degree that is significant.
[00:12:40] Anand Natu: Like we're not talking. Five or six products that you know, you can't quite figure out. And so you can just label them. We're talking hundreds of millions of products that firmly fall into the, I have no idea how to deal with this category. And so you rapidly start to adjust the way you think about things in terms of once you've taken that first 80 20 pass and solved for all of the [00:13:00] really easy problems, which are like, you know, figuring out which Nike, which products belong to brands like Nike or Louis Vuitton, you know, really obvious high signal brands.
[00:13:10] Anand Natu: You end up having to really change the way you look at the problem in order to make meaningful and fast progress critically on those, those more gray area segments, which again, are huge but won't obey any of the sort of elegant solutions that, that got you. 80% of the way there. And, and that's kind of where.
[00:13:31] Anand Natu: At least in this case, ML came into play. But I'd say that was really the number one learning.
[00:13:37] Anand Natu: it sounds like you really had to implement a process where there's continuous iteration, continuous deployment and release and, and sort of bringing some of the learning back from the, from the production, from the wild into, into the open cycle.
[00:13:51] Anand Natu: And can you push this thing out?
[00:13:55] Anand Natu: Absolutely. And, and I think building off of that [00:14:00] iterative and continuous nature of product improvement, another big thing that became at least rapidly apparent to me as we started working on this program was working at that scale also requires you to basically become comfortable, not necessarily with failure, but it, it inc it forces you to recognize the difference between the existence of a risk.
[00:14:23] Anand Natu: And how much you as a business leader or a product manager, care about that risk because you will always have risks and it is probably impossible for you to come up with a design that is elegant and perfect and fixes everything. So when you're doing things like writing product requirements docs or business proposals I had to do something that I was previously a little bit less familiar with doing, which is flagging risks.
[00:14:46] Anand Natu: And then for some of those just saying, We don't, we're gonna call this irrelevant and take, like, eat the loss or eat the error that we potentially stand to incur by not doing anything about it. Because if you aren't willing to make those calls, you just won't get anything done [00:15:00] because the scale is so large.
[00:15:01] Anand Natu: And like I mentioned, you know, the, the, there is always gonna be an edge case that throws a wrench in, in whatever perfect design you try to build to make everyone happy.
[00:15:10] Himakara Pieris: What is. A rubric if there was one that you used to assess risk and figure out what's acceptable and, and what's not acceptable.
[00:15:20] Anand Natu: I think it's important.
[00:15:22] Anand Natu: Typically, I number one is to figure out what your. Measures of success or failure are. So usually I do that by identifying two things. Number one is just the goal metric, which you should already have a pretty good understanding of. As a pm it's probably the business metric that you report, which in this case it was, it was again, just a regular coverage metric.
[00:15:44] Anand Natu: So that was number one. And then number two are basically like trust buster type events, which even if they happen at low prevalence, aren't necessarily acceptable. So in our case, that might have been things like, you know, if we're meaningfully exposing the catalog to incremental [00:16:00] abuse, that's not acceptable because even small lifts in abuse rate are hugely problematic from the perspective of eliminating trust with sellers, potentially destroying relationships with brands, sometimes major brands, things like that.
[00:16:14] Anand Natu: So once you have a good, and those are basically the ax axis along which you can measure risk predictably. Once you do that, you look at a particular treatment that you've prescribed. You know, you look at your problem, you come up with what you think is a good answer, and you say, if I do this, what, what risks exist as a result of me taking this action?
[00:16:36] Anand Natu: You then kind of list those risks and quantify them according to the two different measures of risk that you identified in the first step. So you say, if I do this, Counterfactually what's gonna happen in terms of, you know, like maybe goal metric movement that I'm leaving on the table or that I'm potentially losing and what's the probability, and you're never gonna know this completely, but like back of [00:17:00] the envelope, what's the likelihood that I'm exposing myself to one of those major kind of trust buster events?
[00:17:05] Anand Natu: Based on that, you can kind of combine those two measures and develop a more holistic but, but still quantitative view of how much risk you're exposed to. Based on each scenario, and that'll help you as a leader make a call on whether it's acceptable to sort of green light a launch or a change while being exposed to it or not.
[00:17:30] Anand Natu: And I wouldn't say there's hard and fast rules for like, what the cutoffs are in terms of looking at the numbers. I think that is really subject to the PM's discretion, team's discretion and, and honestly, the business case, like it, it really depends on context, but I think all of that pre-work. At least gives you a consistent basis of information to look at different risks through the same lens, if that makes sense.
[00:17:52] Anand Natu: And I think that's like very important. That's probably the number one most important part of that whole framework, is that if you're looking at different risks, you need to [00:18:00] do whatever you need to do to make sure that you can look at them all the same way. Which is why I think selection of risk measures is.
[00:18:06] Anand Natu: The most important part of that, that process, which ends up being the most consequential for what your outcomes are.
[00:18:14] Himakara Pieris: So it sounds like two key measures of success for this initiative. One was coverage, the second I presume was accuracy. , coverage is relatively easier thing to measure. How was the price of measuring. Accuracy look like considering there are a ton of edge cases, as you talked about before.
[00:18:34] Anand Natu: Accuracy ends up being an exercise in compromise with respect to how many sort of operating resources you want to throw at measuring accuracy. Versus how much uncertainty you accept in order for your metric to be useful from a business standpoint. So the context for data quality is we would basically perform audits on a monthly basis on samples of our catalog data.
[00:18:59] Anand Natu: And those [00:19:00] numbers get reported out in our monthly business reviews as a measure of data quality to inform, you know, whether we're. Hitting R S L A in terms of quality. And if we see defects, then we, you know, we sometimes break the audit out by different, different methods of, like, stamping, cuz we obviously stamp using different signals in order to see if specific channels are more error prone than others, so on and so forth.
[00:19:24] Anand Natu: So tipping typically, again, this is something where I think the, the making that trade off. Is very dependent on the PM's discretion and the partnering with your DS or whoever your like business intelligence person is to figure out what the right balance is. And I think what, what this really highlights as an important skill, not necessarily just for ml, but I think for any large scale data work is it's really helpful for the PM to at least have a basic understanding of the stats [00:20:00] component of the problem that you're trying to solve.
[00:20:02] Anand Natu: And I think data quality in this specific example was a really good example of that because I think the, the confidence interval we ended up arriving at was something like 2%. So we would basically measure a proportion of our population that produced an uncertainty of plus or minus, like between one and 2%.
[00:20:20] Anand Natu: I, I don't remember the exact value. But that was important information because if I didn't know that, or I didn't have an explicit understanding of the error that is inherent in the measurement. I might look at our monthly readout and see that it's 98.2% and say, oh, we're below sla. We have to do something about this.
[00:20:40] Anand Natu: But in reality, your measurements are always subject to a band of uncertainty. And so that really helps you manage expectations and determine when it's appropriate to take action versus when it isn't. And, and basically just, just helps you separate signal from noise and stats is the fundamental language that you will use to be able to do that consistently and reliably.
[00:20:59] Anand Natu: And I think [00:21:00] it, it really. Behooves you as a product manager in these largest data spaces, to have at least a basic knowledge of how to use statistical language to make sense of data and make sense of how data's used to drive decision making. It saves you a ton of time and it saves you from having to go to a DS or a b i e for every little question you have about how to measure something or how to infer based on data.
[00:21:26] Himakara Pieris: What other advice would you offer to PMs who are interested in getting into AI and machine learning?
[00:21:35] Anand Natu: That's a good question. So if I had to pick the top few, I'd say, you know, number one is remember that machine learning is just a tool and you should feel empowered to be creative about how you use it to, to deliver value to your users or your customers.
[00:21:52] Anand Natu: I think that's really where. The skill and craft of the product manager shines is, you know, it is just a tool. It's not [00:22:00] gonna do anything revolutionary on its own. You need to make sure that it is geared up in a, in a manner, and, and shipped in a manner that allows it to accomplish its intended. Business purpose effectively.
[00:22:13] Anand Natu: Number one. Number two, I think is be explicitly aware of the human factors that go into u using ML in the real world, like labeling, data quality, ml ops, all of that stuff. And, and explicitly consider them in your design because they're important and they will inevitably end up being part of how you deploy and maintain an ML system over time, especially for large data like large data.
[00:22:39] Anand Natu: Problems. And then I think the third would probably be like, don't clinging too hard to specific frameworks that you might read in books or on LinkedIn or wherever you might find them. They're helpful guides, but they're not authoritative, and your problem is, is the king. Like you, you should always respect your problem [00:23:00] and its unique needs more than you respect anything you read on LinkedIn or in any book.
[00:23:04] Anand Natu: So have a good practical understanding of the useful thing that you are trying to build an ML model to help you accomplish and orient all of your decision making around that. If you see choices that a framework prescribes, but that don't help you use ML to achieve that useful thing, then you should strongly reconsider whether you should even do that.
[00:23:24] Himakara Pieris: I'd love to get into the workflow that you use to take something from idea to production, starting with how do you identify a good opportunity for ai?
[00:23:36] Anand Natu: I'd broadly divide that process into the steps that happen before you even start building, and then the steps that happen once you begin building.
[00:23:45] Anand Natu: So before you start building, and this is like a mix of validation plus diligence work. I think number one is present the business problem as an ML problem and see if it makes sense to solve using ml. Makes [00:24:00] sense is kind of a loaded term. There's a few different checks you can do to verify that. I can go into that now or come back to it later, but, That's step one is basically look at the thing you're trying to do in a, in practical terms.
[00:24:13] Anand Natu: See if you can at a high level, turn it into a machine learning problem and see if that representation in, as a machine learning problem seems like a feasible thing to, to solve. And you would, you might measure feasibility based on, you know, the volume of data you have, the, the quality of the data, whether the features are actually good predictors of the thing that you're trying to.
[00:24:36] Anand Natu: Train the model to do. But, but that's step one. If you don't clear that common sense sanity check, then that's a problem. And it's not necessarily reason to just abandon it all together, but you should really get to a point where you can convince yourself that the ML version of this problem is make sense and is solvable to a high degree of performance.
[00:24:59] Himakara Pieris: So it [00:25:00] sounds like there is a, you know, component of whether you can do it and whether you should do it . Whether you can do it seems to be associated with the availability of data, cleanness of data, and I'm sure based on your organization, some of the other factors like, you know, availability of resources and timelines and other business pressures as well.
[00:25:18] Himakara Pieris: When you, think about whether you should do it or whether you should consider alternatives, what is your way of figuring that
[00:25:26] Himakara Pieris: out?
[00:25:27] Anand Natu: I think the weather you should part is, is more subjective. The weather you can often falls on more hard determin like objective determinations.
[00:25:36] Anand Natu: Like do I even have enough data and is it high enough quality for me to build a model out of that doesn't make mistakes all the time because like, you know, you can't, you can't make high quality predictions using low quality training data. I think the weather you should part falls on more of whether you.
[00:25:55] Anand Natu: Can develop some, a strong conviction that the, the [00:26:00] features or like the data that you're looking at is a good, is at some level a good predictor variable for the thing that you, the model is trying to accomplish. So, as an example, you know, just to make something up, if you were gonna build a model that looks at features to determine the median price of a home.
[00:26:17] Anand Natu: But you didn't have important things like square footage or the zip code that it's located in, but you had things like, I don't know, weather data or something. You know, there, there may be a correlation there, but you sh that that common sense check that you make at the start should tell you this is probably a low quality predictor.
[00:26:35] Anand Natu: And going through the effort of building a solution that's indexed on this low predictive, like low predictive power training data isn't really gonna help me at all because, Y you know, it's, it's, I'm picking features that are ill conditioned to the problem I'm trying to solve. And, and I think that's really like the level of technical competency that is, is important for a PM to have in this setting is like, you should be able to look at the kind of data [00:27:00] you're using and come to a common sense conclusion about whether it's directionally right or wrong for the problem you're trying to solve.
[00:27:07] Anand Natu: And if it's, if it feels like it's the wrong kind of data, then that's a good signal that maybe you shouldn't be using ML for this kind of. Problem
[00:27:17] Himakara Pieris: sounds like a good place to start is spending time with data, becoming familiar with the data, and there will be some intuition about, you know, as someone who is looking through the data, whether you can come to certain conclusions based on what you have in front of you or not. And if you can't, then you definitely can't do a model to do it as well.
[00:27:35] Anand Natu: Absolutely. And, and I don't think that's any that needs to be any overcomplicated, you know, understanding of statistical inference and like the mathematical techniques that go into doing that. But I think this really goes back to like linear regression, which you learned in high school map.
[00:27:49] Anand Natu: Like develop this good instinctive understanding of. What, how data is used in practice to build a model that relates inputs and outputs and use that to common sense your way through looking at [00:28:00] your data and looking at your desired output and seeing if they empirically are related in a way that seems reasonable.
[00:28:08] Himakara Pieris: From that point, I presume you're going to go and evaluate the technical feasibility. What would you recommend as a good approach for doing that?
[00:28:18] Anand Natu: I think I would do two things there. Number one is to the extent that you can, it might not be perfect the first time, but, define , what I would call the operating variables of your problem, which are fall usually in the two buckets. Number one is the objective objective function.
[00:28:32] Anand Natu: The thing that your model can hold onto and drive either like maximize, minimize, whatever. Basically optimize and, and find that one metric that you think is the right objective function for the model to. Used as its North Star. And then the second bucket of operating variables is constraints.
[00:28:53] Anand Natu: Like, are there any hard design constraints that, that come in from the practical side that, that, you know, are going to bound [00:29:00] your problem in some way? Once you have those two, that's a great start for just giving some substance to what the modeling exercise will look like. And then the, the second thing that's good for the PM to do is, at a high level, again, doesn't have to be perfect, but put the ml task in context within the business use case.
[00:29:20] Anand Natu: This can be. Literally just a flow chart on a slide or something. Again, nothing overly fancy, but the, the question that I would use to guide this exercise is, you know, you're gonna, you're basically trying to build a model. It's gonna train itself on data. It will perform some inference where it takes signal and it produces either a label or an action or something, some kind of output.
[00:29:43] Anand Natu: The, the contextualizing exercise you should do is say, what are the steps I need to go to, to go, go through, to go from that output, that raw output data that the model spits out to the useful thing for my user that I'm trying to get it to do. And then just providing that context [00:30:00] is really helpful because then your, your engineering team is gonna understand how it actually gets used in practice, which may, may give them just from a system design perspective.
[00:30:09] Anand Natu: At least a broad understanding of where they might have to integrate and, and you know, what other services they might have to manage around or be, be mindful of.
[00:30:18] Himakara Pieris: Would you also think through in advance about various releases or various iterations starting from the lightest M V MVP that you could think of? At this stage,
[00:30:32] Anand Natu: Inherently it is a good idea to take an iterative approach, but with respect to what modeling method the, the best MVP should be, I think that's a choice best made in partnership with your engineering team. I think it's, it's good for the PM to have a really firm, clear representation of.
[00:30:52] Anand Natu: What the business case is and how it's represented as like an ML problem. But when it comes to actually [00:31:00] solving that ML problem I think it's really helpful to work directly with your engineering partners on that because it, it saves you as a PM from having two strain to territory where you may not a, be as comfortable or as confident and Basically come up with solutions that, that end up conditioning your, you basically, you wanna avoid injecting bias into your engineering team's design process.
[00:31:28] Anand Natu: So I think that that first turn is best left as a collaborative exercise with your engineering partners.
[00:31:34] Himakara Pieris: So do you see that as sort a collaborative exploratory period where you kind of, you know, working, looking through the data with your engineering team to get their guidance on how to best navigate the.
[00:31:47] Himakara Pieris: Technical implementation of it.
[00:31:49] Anand Natu: Absolutely. And, and I think the, the reason I mentioned that as a collaborative process is because I think it's, it's obviously the PM's job to bring the, the sort of viewpoint [00:32:00] of business urgency and customer utility and stuff to the table and how much you have to do that or how you have to do that will just depend on the direction that your engineering team is like naturally inclined towards.
[00:32:10] Anand Natu: So it may be that you work with ENG partners who are also very scrappy and like to. Ship fast and you know, and then that's fine. Maybe you, you might even have to push them in the opposite direction to, to build something more robust when they want to do a really sort of rough first iteration of something.
[00:32:30] Anand Natu: On the other hand, you might have an ege team who wants to have the first go around, be something that is very, that is more complex and takes more time to build and you may have to reign them in and say, Well, no, let's, let's sort of bring this back to the core problem and, and figure out what the cheapest way is to develop a first solution, test it, and, and figure out where to go from there.
[00:32:52] Anand Natu: So we don't waste waste time on something that might just be like directionally the wrong a protel together.
[00:32:58] Himakara Pieris: What are some of the most common [00:33:00] mistakes that you have seen people make at this stage?
[00:33:05] Anand Natu: Probably the most common mistakes are number one, the thing that I mentioned previously, which is an over injection of bias into the engineering process from the product manager.
[00:33:17] Anand Natu: It's important for the PM to be confident up to the limit of what they're expected to know on the business side of the problem. So that's all the stuff I mentioned before, like, have a good understanding of stats, have a good understanding of, you know, how data is actually used in ML and, and you know, how inputs and outputs are related through ml.
[00:33:36] Anand Natu: But don't have too strong an opinion about. What the right way is to solve that problem because again, depending on the disposition of your engineering team, they might sort of defer to your opinion more than they should or more than they need to when it comes to figuring out what the right answer is.
[00:33:57] Anand Natu: And so number one is, [00:34:00] you know, like basically run up to the finish line, so to speak, and then stop there and, and. Once you've stopped there, don't make any further progress without doing so indirect. Collaboration with your end partners because they're the subject matter experts and getting the highest quality results is is a matter of letting the people who know certain parts best.
[00:34:25] Anand Natu: Own those parts and own the final sort of call the shots on those parts, if that makes sense. I think the second one is, the second biggest mistake area actually happens in the steps that we talked about before, which is in the framing and, and sort of problem development part, which is it's very tempting to look at a lot of lofty aspirational business problems and.
[00:34:48] Anand Natu: Think that, you know, I can, I can use ML somehow to solve this and, you know, through some black box magical engineering [00:35:00] process, get to a AI driven tool that's gonna do something amazing for my users. It's, it's important upfront to abandon that thinking and be very discreet and, and sort of prescriptive about how you make that jump from the business world.
[00:35:17] Anand Natu: Into the, like, basically jump, make the jump from, I'm solving a business problem that's useful for my customers to, I'm solving a machine learning problem that minimizes the loss function or takes this metric and, you know, tries to maximize it or takes these features and predicts the probability of something.
[00:35:35] Anand Natu: So, so being rigorous about how you navigate the boundary between those two spaces is important, and I think it's a common stumbling point for. For people, which is why they end up with solutions that are either weak, they have weak and french power. They don't do the thing that they were imagining in their head, or they're just solving, they're pointed in the wrong direction altogether.
[00:35:55] Anand Natu: And they, and they don't sort of, [00:36:00] they, they're basically non-starter models that, that do this like contrived thing really well, but don't fit into the broader business case that the PM actually wanted them to.
[00:36:11] Himakara Pieris: There has been a lot of news in this field lately. What are you most excited about are the things that you're hearing?
[00:36:21] Anand Natu: I think the biggest change that I've seen, so, you know, to be clear, I think machine learning as a topic has actually been a thing for longer than I think people realize. If you were to rewind maybe 10 to 15 years, Machine learning was actually still a fairly relevant topic in business. It just wasn't called machine learning, it was called Big data.
[00:36:46] Anand Natu: And big data kind of walked so that AI could run in the sense that it was the democratized language of, you know, using data to drive decision making that eventually became more sophisticated forms of artificial intelligence. [00:37:00] And I think in the very recent past we've started to see really powerful models.
[00:37:06] Anand Natu: Become sort of more democratically available and comprehensible to a much wider audience of users than we saw maybe four or five years ago. When, when I would say kind of comparable things were probably available or, or existed to an, you know, close to that degree of power, but were just not at the level where they could, you could put them in the hands of a regular person and.
[00:37:34] Anand Natu: And have them be useful. So I think that's what excites me the most is that I, I do feel like we've really reached a step function change in the ability for the average person to comprehend, make use of, and derive value from AI tools. So that's, that's the part I'm the most excited about.
[00:37:54] Himakara Pieris: There are a lot of PMs who are interested in figuring out how to use ai, how to apply AI in their own [00:38:00] work. What would you recommend they should do as the first step?
[00:38:04] Anand Natu: As a first step for learning about how to work in the space and, and I think I'm, I'm probably a little bit biased, but I do maintain that.
[00:38:12] Anand Natu: I think the easiest way to learn. In this space is by doing. And, and it's obviously important to keep in mind that you know, you're not, you're not doing the work of coding or building an ML model because that's gonna be part of what you do as a product manager. That's not the intent at all. But I think going through that process will, will give you a really hands-on and, and personal understanding of.
[00:38:38] Anand Natu: What that input output relationship actually looks like in practice. And that reasoning is really helpful for making decisions. When you're in more of a business or product context. I think it's much easier and much faster to just go through the process yourself for like a really common sense, simple business problem, like predicting home prices or you know, something that's easily comprehensible and just work [00:39:00] through the work, through the process of trying to build something that works.
[00:39:06] Anand Natu: And in doing so, you'll equip yourself with not necessarily the technical skills, because those aren't actually important in the context of a product management role, but you'll equip yourself with the knowledge of sort of how you go from, from one to the other in the sense of starting with something that is fundamentally just a technical problem that you can represent in code or in mathematical formulas or something like that into the world of.
[00:39:35] Anand Natu: I'm doing something that's useful for my users. I think it's, it's very important to force yourself to walk through all those, those steps in order to really be able to, as a pm be a, be a good partner to your engineering team and jump between those two places. Because at least for myself, you know, I, I feel a lot more comfortable working through that process myself in my head.
[00:39:58] Anand Natu: Because I've done [00:40:00] that work at some point on my own, and I am by no means a machine learning engineer, nor could I, you know, hope to be one anytime soon. But the fact that I've built kind of, you know, scrappy thrown together versions of models that solve practical problems allows me to, gives me a much more concrete reference point off of which to reason through the, the decisions that I have to make on the product side.
[00:40:28] Himakara Pieris: Is there anything else you'd like to share with our audience?
[00:40:32] Anand Natu: In general I think the it's obviously a fast moving field. There's a lot going on and there's a huge amount of potential in the space that has yet to be realized, I do think that product managers have a big role to play in figuring out how to turn.
[00:40:53] Anand Natu: Very promising technology into very valuable products and services. I do maintain that, that [00:41:00] that side of the equation remains as valuable as ever. That's really the point I'd wanna drive home is like, you know, ML is, is. Really, it's incredible what it's become capable of over the last decade or so, and there's a whole bunch of green greenfield space yet to explore.
[00:41:19] Anand Natu: And the, the, but the value of craft and the value of original high-quality decision making on the business side that's informed by a good enough understanding of how things work on the technical side is as important today as it was however far back you look.
[00:41:37] Himakara Pieris: Aan. Thank you very much for your insights today.
[00:41:39] Himakara Pieris: It was great having you on the podcast.
[00:41:41] Anand Natu: Thank you for having me.
[00:41:42] (Outro)
[00:41:42]
[00:41:46] Hima: Smart products is brought to you by hydride or AI. Hydro helps product teams explore how they can introduce AI-powered features to their products and deliver unique customer value. Learn more at www.hydra.ai.
-
I’m excited to bring you this conversation with Tarig Khairalla. Tarig is an AI product manager at Kensho; Kensho is the AI and innovation hub for S&P Global. They are focused on helping customers unlock insights from messy data.
In this episode, Tarig talked about the day-in-life of an AI product manager, how to overcome adoption challenges, and his approach to collaborating with both engineering partners and customers.
Links
Tarig on LinkedIn
About Kensho
Transcript
[00:00:00] Tarig Khairalla: being a product manager space right now is very exciting. Things are moving fast. There are opportunities everywhere. To leapfrog competitors and other companies out there. And I think it's an opportunity for a lot of product managers now to get into this space and, really, make a difference for a lot of customers.
[00:00:20] Hima: I'm, Himakara Pieris. You're listening to smart products. A show where we recognize, celebrate, and learn from industry leaders who are solving real-world problems. Using AI.
[00:00:30] Himakara Pieris: I'm excited to this conversation with Tarig Khairalla. Tarig is an
[00:00:35] Himakara Pieris: AI product manager. Kensho. Kensho is the AI and innovation hub for s and p Global. They're focused on helping customers unlock insights from messy data. In this episode, Tarig talked about the day-in-life of an AI product manager, how to overcome challenges, and his approach to collaborating with both engineering partners and customers. Check the show notes for links.
[00:00:54]
[00:00:56] Himakara Pieris: Welcome to the Smart Product Show, Tarig.
[00:00:58] Tarig Khairalla: Thank you for having me[00:01:00]
[00:01:00] Himakara Pieris: Could you tell us a bit about your background and how you started and how you got into managing AI products?
[00:01:06] Tarig Khairalla: Interestingly, coming out of school, I did not immediately become a product manager. I graduated with an accounting, finance, and economics degree. And I worked in the accounting space actually for the first few years at Ernest and Young. And so sometime during those first few years of working, there is when I got involved in the AI and machine learning space. And you know, from there, got stuck to it. I worked at Verizon afterwards for a little bit, and then here I am now with Kensho Technologies as a PM in the AI and machine learning space.
[00:01:38] Himakara Pieris: Tell us a bit about what Kensho does and specifically what Kensho does with AI and machine learning to help companies work with s e data.
[00:01:47] Tarig Khairalla: As you mentioned Kensho, we develop AI solutions that unlock insights that are hidden in messy data. If you think about the business and finance space, the majority of data created has no standard format, [00:02:00] right? You know, your typical professional gathers information from images.
[00:02:05] Tarig Khairalla: Videos, text, audio, and, and so much more. And unfortunately, as a result of that critical insights are generally trapped and, and are hard to uncover in that data. And so the reality is that, you know, the data today is being created at a rate much faster than before. And so the solutions we build are tackling problems that many organizations today are, are, are facing.
[00:02:30] Tarig Khairalla: You know, we build a variety of machine learning. Products that serve to structure, unify and contextualize data. We have products that are used in a daily basis for things like transcribing, voice to text, extracting financial information from pdf, f documents, identifying named entities like. People in places within text understanding sentences and paragraphs to tag them with a topic or concept that are being discussed.
[00:02:59] Tarig Khairalla: Right? So, [00:03:00] you know, at the end of the day, what we're looking to do is, is, is make information more accessible, easier to use, and allow our customers to discover hidden insights much faster than that they could before and ultimately, you know, enabled them to make decisions with a conviction.
[00:03:15] Himakara Pieris: Do you target specific industry verticals or specific business processes?
[00:03:21] Tarig Khairalla: Yeah, our products are mainly geared towards you know, your finance workflows. . A lot of the models and products that we built, were trained on financial data, like the extraction capabilities that we have or trained on financial documents or the transcription products that we we provide are trained on earnings calls, for example, and many other types of financial related data.
[00:03:44]
[00:03:44] Himakara Pieris: I presume you expect higher level of accuracy because your training data set is focused on domain specific data.
[00:03:51] Tarig Khairalla: That's the angle, yes. That we are focused on business and finance to make sure that we're the best at developing machine [00:04:00] learning solutions for the finance professional.
[00:04:02] Himakara Pieris: As a PM how do you think about ways to improve the product or add extra value? Is it primarily around, increasing accuracy or pushing into adjacent use cases? How do you build on top of what you already have as a pm?
[00:04:19] Tarig Khairalla: It's a little bit of both, definitely.
[00:04:21] Tarig Khairalla: Making sure that we are continuing to push the boundaries in terms of what's possible from an accuracy perspective across all our products. But you know, the other thing we do too is make sure that we can provide value beyond just what we offer with one product. For example, you know, a lot of our.
[00:04:36] Tarig Khairalla: Kind of capabilities sometimes are synergistic in nature. So you can add something like, and you know, the product called Kero extract to some of, some, some other Kero product that's called Kero Classify. To be able to now provide something that is beyond just one product or one solution that that users can drive value from.
[00:04:56] Himakara Pieris: How is it different being a product manager working [00:05:00] in in an AI product versus being a product manager who is working in a traditional software product?
[00:05:06] Tarig Khairalla: I think that there's a lot of parallels and similarities, but with being in the AI space, you add a layer of now you have to work really closely with machine learning and data scientists, and you also add an element of uncertainty,
[00:05:21] Tarig Khairalla: because as we're building AI and machine learning products, a lot of times we're uncertain whether a given experiment is gonna succeed. And so there's a little bit more uncertainty around it. There's a little bit more in terms of the discipline, right? You have to learn a little bit how to build machine learning models,
[00:05:37] Tarig Khairalla: the life cycle of a machine learning model. You have to learn that and really kind of be able to implement it in your day-to-day and, and build processes around that to make sure that you're still delivering what you need to del deliver for your customers and, and clients.
[00:05:51] Himakara Pieris: How do you plan the uncertainty is common in air and machine learning product life cycles?
[00:05:57] Tarig Khairalla: certainly what's important to do [00:06:00] is have specific measures of success and targets in mind before starting, let's say, for example, a specific experiment. But I'll also say that it's, it's important to also tie inbox your activities. So you know, when you're scoping out a specific experiment or a specific project, understanding what your no star is going to be.
[00:06:20] Tarig Khairalla: Understanding some of the measure of success that you're looking to go after and making sure that you're timeboxing what you're looking to achieve in a reasonable amount of time. Right? Typically what happens with machine learning experimentation is that, you know, it can, you, you can, you can experiment for, for a very long time, but is that going to be valuable?
[00:06:38] Tarig Khairalla: Is there a return of investment in, you know, spending two to three to four months of time experimenting or something? Or are you better off pivoting to something else that can potentially drive value elsewhere?
[00:06:50] Himakara Pieris: Do you have a data science and machine learning team that's embedded inside a product team, or is there a separation between the data science machine learning team and the product team?[00:07:00]
[00:07:00] Tarig Khairalla: The way we think about things is, there's three kind of main key players in developing a product.
[00:07:05] Tarig Khairalla: We've got the product manager we've got the tech lead on the engineering side application side, and then there's the tech lead on the machine learning side. So the three of them combined with a, a designer is usually how we approach building products at Kin Show.
[00:07:20] Himakara Pieris: How does that interface look like between let's say the machine learning lead and the engineering lead, and also between machine learning and product so those, different interfaces..
[00:07:31] Tarig Khairalla: . , we work really closely together. You know, say that we touch base regularly on a weekly basis to kind of talk about what we're looking to do. Right. Whether it's like a new product that we're building out if, if machine learning team is in kind of the research mode we make sure that we're continuing to talk to our tech leads to make sure that We build expectations ahead of time.
[00:07:56] Tarig Khairalla: So if a model is being built out, it's in research [00:08:00] mode. Maybe there's not a lot of effort that's needing to be done on the kind of backend side of things on the application side. But once that model graduates to being more of a potential one that's gonna be productionized, we're already kind of established.
[00:08:14] Tarig Khairalla: We're already on the same page as far as like, well, there's a model coming your way. We have to think about setting up that service up front. So I would say it's very collaborative in nature as we're scoping out. Kind of product early on. Everybody from a design to product, to applications to machine learning is involved in that process.
[00:08:33] Himakara Pieris: What does the input to the machine learning team look like from you as the pm.
[00:08:38] Tarig Khairalla: As pm I think the largely still remains very similar to how a software product manager would work, right? The voice of the customer, right. Thinking about exactly what problems we're trying to solve for and, and, and how we wanna optimize the algorithms and the models that we're building out, right?
[00:08:54] Tarig Khairalla: The measure of success. So my input is around kind of the [00:09:00] We think about them, kind of the, the domain expertise, right? What do the models need to do in collaboration with the users and customers that we're going after? My input is also around evaluation, right? How do we collect the right data to be able to evaluate the models and make sure that we're comfortable with what they're doing?
[00:09:18] Tarig Khairalla: And yeah, anything around coordinating between customers and, and, and clients, right? Being able to kind of pass feedback back and forth between the two groups of, of, of, of disciplines.
[00:09:30] Himakara Pieris: When you initially engage with the mission learning team on, let's say a new feature idea Is it that you have a data set and you outline a requirement that says, we want to accomplish X, Y, and Z with this data set, and here's the performance criteria that we expect. Or is it a lot more free form than that?
[00:09:50] Tarig Khairalla: I would say we work together on scoping all of that out. So the requirements are things that both myself on the product side and the machine learning side [00:10:00] work together to get really comfortable with. It's a little bit more collaborative in nature, so I, you know, I don't.
[00:10:07] Tarig Khairalla: Directly provide requirements as is because I know there may be some flexibility and flexes in how machine learning team can help develop something. And so you know, what I contribute to that is, is understanding the problem really well, understanding the domain knowledge behind it, really well, understanding where we want to go.
[00:10:24] Tarig Khairalla: So the actual North star, the strategy behind we're achieving, and then all this stuff that's in the middle. The requirements and, and exactly how we wanna execute on it is something that we work together to scope out and make sure we're all comfortable with.
[00:10:37] Himakara Pieris: If someone as a PM is working, let's say setting up their, first AI powered feature that they're putting together a team how do you think they should structure that conversation?
[00:10:47] Tarig Khairalla: You know, I would say that there's a, there's a few things to think about when you're first starting off trying to structure a project.
[00:10:55] Tarig Khairalla: You know, other than just knowing exactly what you're trying to solve for and some of the actual conversations that [00:11:00] you've had with clients and customers, to be able to really explain and talk about what we're trying to solve for with our, with our machine learning team. Right? That's number one. And then number two, I think it's important to know some of the inputs and outputs that we're looking for.
[00:11:14] Tarig Khairalla: Is there existing training data or data sets that we can go after somewhere to be able to build this model? Right? Feasibility, essentially just high level feasibility of that. What are the outcomes we're going after, right? What are the outputs gonna look like? Is this something that's gonna be embedded into a platform or is this something that's standalone, freeform product that we're building out?
[00:11:33] Tarig Khairalla: So just having an idea of like what it is. Where we want to deploy or build it and when, so timing is really important to have, right? Is this something that we want to build tomorrow in, in a week from now, two months from now, three months from now? Those are things that you need to be able to answer to then help guide and help structure how you wanna build it out with the machine learning team.
[00:11:57] Himakara Pieris: Considering there already sort of the feasibility [00:12:00] analysis phase at the very top of it do you recommend people to run multiple pilots in peril or is it best tackled sequentially?
[00:12:10] Tarig Khairalla: I think it depends on the use case, but I do think that Multiple pilots will help. I wouldn't say we take on too much because I think the idea here is the way I like to execute in myself is to start really small.
[00:12:25] Tarig Khairalla: So we're trying to validate something really quickly at a very small scale. And so starting off potentially with, with one pilot is perfectly fine if you can validate very quickly and then scaling up to the next and the second and the third. But if, if. If, depending on the use case, I don't see a problem of why we couldn't do multiple pods at the same time as well.
[00:12:46] Himakara Pieris: If you had to timebox these pilots, what do you think is a reasonable time box window to kind of think about?
[00:12:53] Tarig Khairalla: I'll say that it, this probably something that really depends on the, the existing team, but [00:13:00] generally I would say, We've used things like for from between four to six weeks as, as a tie in box.
[00:13:08] Tarig Khairalla: In some instances we do. If it's something that's really, really quick and we're in, we don't know the value of it yet. We do two weeks.
[00:13:14] Himakara Pieris: Could you share a bit about what your day in life looks like as an I P M? It's interesting. I think the role that you're in, because when we think about AI products, there's this sort of division between co ai products where there is a research you know, research in European component, and then there's the applied AI products where you are embedding these, you know, capabilities into.
[00:13:38] Himakara Pieris: A user application and user facing application. And then there's the machine learning m and ops side of things. And it sounds like you have to touch all three of them in a way and at least the first two for sure. So curious how, how your day is structured.
[00:13:55] Tarig Khairalla: That's actually, a fair assessment.
[00:13:57] Tarig Khairalla: Right? My, my role does touch on [00:14:00] things like research, like actually time, you know, allowing time for research to be done in terms of experimenting with models, and then ultimately how do we take them and actually productionize them. A day in the life of a PM is generally hard to, to pinpoint, and I think especially in the AI space, but the reality that it, it varies, right?
[00:14:18] Tarig Khairalla: I personally. I like to make sure that even though it's a very varied you know, schedule and, and, and, and week that I may run into, I like to make sure that I have a fixed you know, fixed activities that I do on a daily basis. So, for example, I start my day usually with three things, right? I.
[00:14:36] Tarig Khairalla: Read up on articles. The first thing I go on just to make sure that I'm continuing to be up to date on the latest developments in this space, especially nowadays. The this space moves really fast. Then I go through and organize my emails. I I don't immediately respond to emails, right? But I go in the morning and, and try to make sure that I prioritize and find the ones that I need to make sure to respond to during the day.
[00:14:58] Tarig Khairalla: And then the same goes for [00:15:00] things like Slack and teams and things of that nature. And the third thing I do is I finally go through analytics. Look at whether there's a user that's signed up to our product. Look at some of the metrics that we're tracking internally just to make sure that I am staying up to date on, on the product and, and everything that happened overnight, essentially.
[00:15:21] Tarig Khairalla: From there, I think the, the bulk of the day can vary, but what I will typically say that there's a lot of context switching. And so I may start off with a client call, for example, to discuss a use case or a project that we wanna explore with them. And then maybe I'll jump into a leadership call to present on something.
[00:15:42] Tarig Khairalla: Right. And then maybe go into like a technical brainstorming call with our machine learning team, followed by another technical running, brainstorming call with our engineering team, right, of software engineers. So these are just examples of what a typical day could look like. But there's other aspects for me personally that I run into, like [00:16:00] design.
[00:16:00] Tarig Khairalla: So working with design teams to introvert users or come up with concepts, sales. Marketing, finance, legal implications, right? In the AI space specifically, there's a lot of legal implications that we have to think about. And so as a PM in the AI space, you typically have to get involved in a lot of these different aspects on a daily basis.
[00:16:22] Tarig Khairalla: And so, you know, at the end of the day, I, I like to then structure myself as well. So I start with some structure and then I wanna make sure that I end with some structure. So I go through and figure out, What I wasn't able to accomplish during the day. And then I create a a to-do list of things I need to accomplish the following day.
[00:16:40] Tarig Khairalla: What I will say though, that if, if, if you're somebody who likes variety in their day, then I, I think being an AI pm is something that, you know, you'd, you'd like, you'd enjoy.
[00:16:53] Himakara Pieris: How do you prepare for a brainstorming session with a mission learning team? Would you [00:17:00] recommend a pm sort of a read up and get up to speed on technical aspects of things to be able to have a conversation there? Or are there any other recommendations?
[00:17:11] Tarig Khairalla: I think that N n having a high level understanding of the different types of machine learning approaches is really helpful in terms of being able to communicate with your counterparts and the machine learning space or, and machine learning team.
[00:17:28] Tarig Khairalla: But I certainly don't think it's a requirement, right? I, again, I started my career in the finance and accounting space, so I didn't have a technical background, but through kind of learning on the job and through reading up and, and staying up to date on the. The industry, I was able to kind of learn that as, as I went along.
[00:17:44] Tarig Khairalla: And so certainly I think it helps to, to understand some of the high levels of how a machine learning model is constructed. Some of the different techniques to get to, to, to a solution. But keep in mind as in those brainstorming sessions when we're [00:18:00] talking about how potentially things can be developed what you bring or what I bring as, as a pm is again, the customer.
[00:18:07] Tarig Khairalla: Right, understanding exactly the pain points and the, the measure of success that we're trying to optimize for. And that's a valuable input in terms of being able to develop and build something that's important and it's gonna solve a problem for users.
[00:18:19] Himakara Pieris: Moving on to the good market side of things what are the big challenges that you face? Considering. Finance is a big target vertical for you. , very risk of this industry. I I can't imagine there are, there are a lot of questions that you have to answer.
[00:18:39] Tarig Khairalla: Certainly many challenges that come our way when we try to productionize deploy and commercialized products. One of them that first comes up is general legality. The data that's underlying a lot of these models sometimes is. Proprietary nature. And a lot of our customers and users don't want us to use their data, for example. But I think [00:19:00] one thing that's, you know, more challenging and I, from my experience have run into many challenges around is actual adoption of our products.
[00:19:08] Tarig Khairalla: If you think about some of the users in the finance space, they've likely gotten accustomed to workflows they've been following for, for years. Right. These are individuals that are involved in validating every single detail or every single line item to get comfortable with financial decisions that are being made.
[00:19:27] Tarig Khairalla: And, and that's for good reason, right? There's a significant downside risk to negligence in this space that leads to penalties, liabilities, and things of that nature. And so as a result of that, our users are. Typically and generally very conservative in nature and relatively skeptical in terms of automation, which means that a, as a product team, you, you almost have to ensure that early stages, early stage models are either fairly accurate in nature, or if they're not, then your software that you're building around these models allows [00:20:00] for checks and balances to instill trust early in your product.
[00:20:05] Tarig Khairalla: You know, from my experience there. A few types of mindsets that you'll run into being a product manager in this space when you're introducing new products in the finance space, right. You know, the first one is gonna be around you. You'll run into users that are generally skeptical, but willing to try and adopt new products.
[00:20:25] Tarig Khairalla: And so as they're starting to use a new product, they're generally gonna run into like a learning curve, right? You know, this is something that. They keep in mind they've, they've been following these processes for a very long time and maybe with adopting a new product, the time that's being spent in the perceived complexity of the task at hand is changing and maybe it's going up and typically that will impact their performance and whether they're willing to continue using that product.
[00:20:55] Himakara Pieris: Can you talk a bit about explainability and [00:21:00] how much of a role they play when you're try and drive adoption?
[00:21:03] Tarig Khairalla: The second kind of mindset, actually, going back to the point earlier, a after, you know, other than skepticism and, and kind of the willingness to try is actually gonna be around.
[00:21:14] Tarig Khairalla: The black box nature of, of these sub of sub-machine learning techniques, right? Users will typically find it hard to wrap their head around how a machine is doing what it does. And again, going back to the finance practitioner they've been used to being able to validate the details, tracing back and make sure they're really comfortable with how things are being done.
[00:21:36] Tarig Khairalla: And now you're introducing something that is hard to validate. It's, it's, you know, hard to explain, right? And so what ends up happening is, is due to the lack of that explainability in those machine learning models, what ended up happening is that they go back to the org processes and resign to using their existing solutions.
[00:21:53] Tarig Khairalla: And so, These are, you know, some of these things are important to know as, as being a product manager in the space and you have to be irv. [00:22:00] And part of that is, again, being deeply empathetic and understanding user pain points really well, to be able to address and mitigate some of those issues.
[00:22:10] Himakara Pieris: How do you balance between Model complexity and explainability because there is, you know, there is a big focus and interest in building larger and large models investing or adopting the latest deep learning frameworks.
[00:22:27] Himakara Pieris: Whereas, you know, if you stay on the simplest side, things are easy to explain. Tend to be more consistent in, in, at least in most cases. How do you balance that? Staying on the cutting edge of technology and, and and the complexity of the product?
[00:22:46] Tarig Khairalla: The way I think about it, it's slow slow integration or slow introduction of, of complexity into the system. This is where kind of understanding users is really important as a pm, is if [00:23:00] you're looking for a user ultimately to adopt a product I like to start from a simple point of view, right? What's the simplest thing that we can do at a very fast rate to be able to drive value for that user without getting to the space where they can't, you know, they're experiencing rapid change.
[00:23:18] Tarig Khairalla: Right. And so I think it's totally fine to start with something that's simple, maybe that's less performant in nature, but it's, it's a trade off between trust and and, and performance essentially. Right? And so from there, once we have that initial user that's starting to use the product a little bit, they're, they're.
[00:23:36] Tarig Khairalla: They're trusting a little bit more. It's then you can kind of start to add complexity and add cutting edge technology and things that ultimately will drive value in the long term. But it is, it is really a trade off between balancing how users perceive your products, right? And instilling trust in them with ultimately really complicated cutting edge technology that maybe is less explainable in nature.
[00:23:58] Tarig Khairalla: And at some point, it's a [00:24:00] journey that you take with your users, right? You wanna take that journey and make sure that you're hand in hand. Adopt, you know, building your product up at a rate where users are able to keep up and, and really comprehend, you know, fo follow along essentially with the journey.
[00:24:15] Himakara Pieris: How do you think about success? I think there are two aspects I'd like to get into. The first one is, the system performance metrics like the F1 schools, et cetera, that you touched on earlier. And the second one is the, the real world success measures. How do you think about these various access measures at various stages of the products lifecycle?
[00:24:38] Himakara Pieris: From the initial validation to first version out of the door to, to like, you know, various. Version iterations in the future.
[00:24:47] Tarig Khairalla: Success measures are, to me a little bit more involved in the AI space compared to your traditional you know, building software. I, I, I always like to anchor our products [00:25:00] to user facing metrics.
[00:25:02] Tarig Khairalla: Rather than, you know, strictly model performance to start. And I think it's even important to do that as you're starting to build a model or like during the early phases in the kind of life cycle of the product, right. To, to illustrate that. For example, you know, the product I work on we process and extract information from hundreds and thousands of PDFs in a given day, for example, right?
[00:25:27] Tarig Khairalla: And so to support and so. Well, you may essentially support very large document viewing and research platforms. Well, you may have a really accurate model in terms of your extraction performance, right? But what users really care about when using those platforms is how quickly they can pull information out from a or from the source they're looking at, right?
[00:25:51] Tarig Khairalla: So if you optimize for accuracy, Maybe end up with a very you know, deep neural network of, of some sort. Then [00:26:00] you sacrifice processing time cuz you're a little bit slower in terms of being able to provide outputs to users. And so you'll run into adoption challenges, right? You'll face you'll hopefully face that realize that the business value that you thought was gonna be generated is not being generated because again, the success measure.
[00:26:17] Tarig Khairalla: In this example, to me would've been the time. You know, one of the success matters, there's many that you can track, but one of them would be around the time it takes for somebody to get information outta the system. And so that's why I like to kind of look at both sides of the equation, forgot the user point of view and some of the things that are important to them.
[00:26:37] Tarig Khairalla: To track and then on on the internal side of things. Yes, the model performance is important to look at the health of the model, and ultimately as a product manager, we're looking to solve problems for customers and making sure that we're balancing both the performance of a model from an accuracy perspective is also being balanced with looking at the user and what they're really coming to us for and what problems we're looking to solve for them.[00:27:00]
[00:27:00] Himakara Pieris: You talked about adoption challenges with AI products. As a pm how do you, mitigate these adoption challenges?
[00:27:09] Tarig Khairalla: Yeah. It's, it's a great, great question and I think from my experience, I, I talked a little bit about. The kind of black box nature of some of the techniques that we use.
[00:27:20] Tarig Khairalla: But at you know, as we're building our pro, you know, our AI products, there's, there's a few strategies they can use to make sure that your users are adopting your products. I think the first one that's really important is involving them as early and as often as you can, right? Continually as you're developing your products, get feedback from your users to instill this, this idea of, of, you know, the, the.
[00:27:46] Tarig Khairalla: Co-creation with your users, right? They're, they're understanding how you're building your product and why you're building it the way you're building it. And, and it makes it easier for them to ultimately use the product in the long term. I [00:28:00] think the second piece is around establishing an onboarding or piloting process where you know, you understand that your users are coming from.
[00:28:09] Tarig Khairalla: You know, a workflow that they've been used to for quite a long time. And so if they're interested in easier product, making sure that there's some sort of onboarding program or some sort of pilot or process that they can go through to help them adopt a product you know, easily. I think the, the last thing I'll say here is build feedback into the process and when feedback is given, make sure that you are able to address that feedback really fast.
[00:28:37] Tarig Khairalla: Specifically in the finance space, users aren't going to wait and so they'll, they'll revert back to using their old processes because they have jobs to get done. And so if we're not able to address critical feedback quickly enough, then you know, you're more likely to see churn in in customers relatively quickly.
[00:28:56] Himakara Pieris: There has been an avalanche of new developments coming [00:29:00] our way over the last few weeks and months at this point. What are you most excited about out of all this new stuff that's happening out there?
[00:29:08] Tarig Khairalla: Very excited about all the development in, in language models large language models specifically.
[00:29:15] Tarig Khairalla: The space is moving really, really fast. Y you know, and, and as part of Kensho we're, we're involved in that headfirst. There's a lot of work that we're doing to actually explore some of the work in terms of language models. It's, it's been a, it's been a lot of developments in a very short amount of time.
[00:29:31] Tarig Khairalla: And I think being a product manager space right now is, is very exciting. You know, things are moving fast. There's opportunities everywhere. To leapfrog you know, competitors and other companies out there. And I think it's an opportunity for a lot of product managers now to to get into this space and, and, and really make a difference for a lot of customers.
[00:29:53] Himakara Pieris: Great. Is there anything else that you'd like to share with our audience?
[00:29:57] Tarig Khairalla: Look, if things like speech-to-text, [00:30:00] extraction from documents, classification, and entity recognition and linking seem like types of solutions you're looking for, come find us at kens show.com. We have a friendly team that will talk to you in-depth about our products.
[00:30:14]
[00:30:18] Hima: Smart products is brought to you by hydra.ai. Hydra helps product teams explore how they can introduce AI-powered features to their products and deliver unique customer value. Learn more at https://www.hydra.ai
-
My guest today is Vibhu Arora. Vibhu’s product career spans both startups and top-tier tech companies. Vibhu is a group product manager focused on AI, M a production at Walmart. Prior to that, he was a product manager at Facebook focused on AR VR e-commerce experience. In this conversation, Vibhu shared with us some of the ways Walmart is using AI and machine learning to improve the customer experience. His advice to product managers who want to get into managing AI and ML products and his thoughts on how to build a business case.
Links
Vibhu on LinkedIn
Transcript
[00:00:00] Vibhu Arora: we created a system where Walmart would actually predict your most likely to use payment vehicle for the given transaction. We make a directed bet that the customer is actually going to use a specific payment method and auto select that, which basically removes one friction point from the journey. So one less friction point. So better conversion, better for customers, better for the business.
[00:00:28] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate and learn from industry leaders who are solving real world problems. Using AI.
[00:00:39] Himakara Pieris: My guest today's Vibhu Arora. Vibhu’s product career spans across both startups and top tier tech companies. Vibhu is a group product manager focused on AI, M a production at Walmart. Prior to that he was a product manager at Facebook focused on AR VR e-commerce experience.
[00:00:52] Himakara Pieris: In this conversation. Vibhu shared with us some of the ways Walmart is using AI and machine learning to improve the customer experience. His advice to product managers who want to get into managing AI and ML products and his thoughts on how to build a business case.
[00:01:07]
[00:01:08] Himakara Pieris: Vibhu welcome to the smart Fredette show.
[00:01:10] Vibhu Arora: Thank you Hima for having me here. I'm excited to be part of the podcast today.
[00:01:14] Himakara Pieris: We've seen. Walmart investing heavily in AI and ML to transform the customer experience. What are some of the interesting AI use cases you've had the chance to work on at Walmart?
[00:01:24] Vibhu Arora: I started out with a product called extended Warranties. It's a very neat concept especially in, electronics and home business verticals where you're buying like a TV.
[00:01:38] Vibhu Arora: The TV comes with one year warranty, but at the time of purchase, you get the option to buy extended warranty. And this option exists only at the time of purchase. You cannot buy it later. So it, it has kind of like this timing component and. It happens to be a winner for both the customer and the business.
[00:02:01] Vibhu Arora: The customer wins because it's peace of mind. You have like four kids in your home, you're buying a thousand dollars tv. You want peace of mind, put in 50 bucks more. And even if the TV is broken into pieces, you know, it'll be replaced, the business is working off of statistics. How many TVs get broken and, how the pricing works accordingly.
[00:02:22] Vibhu Arora: But ultimately , 90 to 95% of warranties never get claimed. Means like things don't break. So it's a highly profitable business.
[00:02:34] Himakara Pieris: Sounds like it's a good example of how you can essentially use AI enabled service products to increase the cart value and the bottom line. Are there any other common types of finance or service use case examples where you're using machine learning?
[00:02:50] Vibhu Arora: One of the most common like use cases for machine learning is fraud detection. If you generalize it one step further, it could even be called anomaly detection.
[00:03:03] Vibhu Arora: One of the use cases, which we used for the credit card application process, was that of anomaly detection or more specifically fraud detection. We actually apply for the credit card. A bunch of inputs are taken in your name and, you know sort of like financial information.
[00:03:25] Vibhu Arora: And based on that the machine learning model actually performs a lot of checks and assesses the risk and, and then spits out kind of a binary answer. Yes or no, you know? Yes. Whether Walmart and partner Capital One in this case are willing to take the risk to give credit to you versus no this profile is, is too risky to give credit.
[00:03:54] Vibhu Arora: The other interesting use case. We, unlocked in payments, in checkout. We built a model where if you actually have a lot of payment vehicles in your account, which means, you know, you have your debit card, you have your credit card, you have your PayPal, you have, you know, whatnot, you have like a lot of payment vehicles in your account. And guess what? Some of, some of these you know, are running low on balance, et cetera.
[00:04:24] Vibhu Arora: So there's like different states of each of these payment vehicles. So we created a system where Walmart would actually predict your most likely-to-use payment vehicle for the given transaction. So instead of like, instead of being super neutral about the transaction that, okay you're, you're going through this.
[00:04:50] Vibhu Arora: And now you can select from any of these options. We used to make we, we used to make like a directed bet that the customer is actually going to use a specific payment method and auto-select that, which basically removes one friction point from the journey. Like the most likely-to-use payment method is automatically selected.
[00:05:12] Vibhu Arora: So one less friction point. So better conversion, better for customers, better for the business.
s[00:05:17] Himakara Pieris: Walmart, I think pioneered trending curries in e-commerce as well. Can you share a bit about that?
[00:05:24] Vibhu Arora: If you start your search journey on the app, you, you start with the auto complete screen. And on the auto complete screen, there is actually a stage when you have not even entered a single letter, which we call like the starting screen or the landing screen.
[00:05:43] Vibhu Arora: Which is the blank slate screen. So on this screen, we envisioned, the strategy here was, we, we want people to, to learn and know more about, trending products, [00:06:00] which, you know, other people are using. And it had like, again, like most good products, it's a win-win for both customers in Walmart.
[00:06:07] Vibhu Arora: So customer gets to learn about the trends and they don't sort of like, they can, they can get to come out of their FOMO because they caught on the trend. and Walmart actually wins as well. Cause, it means like more product sales and, and a happy customer. and the feature is like on this, this, auto complete landing page, below your previous suggestions.
[00:06:32] Vibhu Arora: We started showing inquiries and, we weren't actually expecting it to be, as big of a, a win, but, it actually did, take off pretty well. and we, we, we did experiments with it and we actually got like, you know, pretty solid feedback , from the experiments and ended up launching it.
[00:06:57] Vibhu Arora: So it's now launched fully, so you can actually like [00:07:00] see it on the app as well. and again, this feature also, is built on, on machine learning models. It takes in a bunch of signals from the crowd, and has like a, puts them into, through a definition of trending versus non trending. And depending on whether yes or no, a particular query gets.
[00:07:26] Vibhu Arora: Stamped as a trending query or not. So, again, you know, kind of like a powerful, cool, fun, implementation and use of ai, in, in Walmart search ecosystem.
[00:07:40] Himakara Pieris: In couple of those use cases, they seem to be directly tied with improving UX and by doing that, removing friction and by removing friction increasing revenue generation, .
[00:07:52] Himakara Pieris: One of the challenges that we hear all the time is difficulty that PM's experiencing in making a business case [00:08:00] because they're having trouble quantifying the impact at the start. Especially if you are at an organization that doesn't have lot of machine learning and data science resources, you're trying to venture into this area,
[00:08:12] Himakara Pieris: Given this type of context, what's your advice to a product manager? On how to build a business case.
[00:08:19] Vibhu Arora: What your questions sort of highlights is if you're, if you're not like a mature ai culture or a mature AI organization it can be very challenging to, to sort of like build this or cultivate this sort of mindset. So Absolutely. I think, you know, it all, it all begins with like empathy and it begins with the awareness.
[00:08:48] Vibhu Arora: If I'm a senior product manager in a, in a situation where we want to try our first AI product and, and AI sort of understand, never [00:09:00] existed before, so I would, I would want to have like an awareness and empathy of my. Organization , because obviously, you know, I would need out of many things I.
[00:09:11] Vibhu Arora: Lead alignment and funding from stakeholders peers and senior leadership and you know, how these people are thinking and where their minds is at. Is, is fundamentally going to be material in, in unlocking you know, this chart or not. And by the way, this, this also like, is one of the biggest learnings I had working in startup versus working in large companies is like in startup, like generally the strength of the idea wins and the the time to it takes to test something and pivot is so [00:10:00] rapid.
[00:10:00] Vibhu Arora: It's, it's, it's unbelievable. You have a good idea. You, you test it and it doesn't work and you pivot. So it's like a small boat. You can make turns easily, but in large companies, like the, the boat is bigger. So pivoting is extremely costly. Like changing direction is one of the hardest things to do for big ships.
[00:10:22] Vibhu Arora: So in bigger companies, Aligning with, with multiple organizations, multiple team members, stakeholders, peers, leaders on what direction, actually, sometimes longer than, than holding that line. So it's kind of like a crazy, like a oxymoron where. Alignment can be more painful than actual doing of the work.
[00:10:46] Vibhu Arora: So, so just going back to the question of like, you know, how to build a business case for ai I would not, I would not sort of like, you know, view this lightly from, from this perspective. How mature [00:11:00] is my leadership? How majority are my stakeholders and peers to absorb this concept because, you know, in a large organization that that can make or break the, the ongoing success of this direction.
[00:11:14] Himakara Pieris: given all the recent interest in AI. There are a lot of early career project managers who are interested in getting into AI and ML product management. What's your advice to them?
[00:11:28] Vibhu Arora: One of the challenges that, you know early PM has that they don't necessarily have a lot of influence and the nature of tech organizations is you know, they're a, they're very busy. B they're very matrixed and c like all organizations, they, they work on the principle of influence.
[00:11:55] Vibhu Arora: You. 10 emails from some, like with less influence [00:12:00] will probably reach the same fate as one email from someone with influence. And this is the reality of the situation is like, if you're starting fresh in a company or a junior spot, you won't have influence. So how do you build it? So one way to build influence is like through credibility and, and earning respect.
[00:12:21] Vibhu Arora: And for the AI landscape, I think maybe it's gonna sound like a little bit tactical, but one of the easiest way to build respect, trust, trust and eventually influence is going to be through documentation and documentation. It's, it's not a lot of fun for most people, right? It's a lot of like work.
[00:12:43] Vibhu Arora: But it's, it's going to be one of those areas where No one is going to come in your way, stand in your way and say, Hey, I am not going to let you document more. Right? So if you want to like document more, [00:13:00] you know, everyone in the company will come behind you. Like even if you are, let's take an example.
[00:13:05] Vibhu Arora: You know, let's take, there's a new update to a board model that the whole team is, is, is, you know, looking into you know, somebody from outside will be surprised at the lack of clarity and documentation in large companies. Like, people will be surprised, like, you know so if you come in with a point of view that, hey, I know we are working on this Burr model.
[00:13:27] Vibhu Arora: I know nothing about this, but let me put together what is going on in this release. So these are the five changes we are working on. And, and you know, this is change from the previous version and, and that's the configuration in the previous version and so on so forth. So, You essentially start learning and things together.
[00:13:50] Vibhu Arora: So, so, you know, that's, I, I think could be a very, very powerful way in, in building bridges, building connections gaining [00:14:00] trust with the team and ultimately getting influence. Another way is for, Junior PMs, I would say, is you know, deconstruct and simplify the low level implementations to your executives.
[00:14:17] Vibhu Arora: Meaning, you know, every two months, let's say you have a one-on-one with your vp, right? It's up to you how you frame that one-on-one. So one of the ways you could frame that one-on-one is it could be about your career, right? So one on one-on-ones could be about your career. That's okay. But maybe like 1, 1, 1 more session.
[00:14:39] Vibhu Arora: It could be a lot about, hey, I know you are a vp, I know you are super busy and you don't have access to a lot of like details. Let me try to explain a very complicated concept to you in a very simple manner, right? Which is, how does, I'm just taking an example. How does query [00:15:00] understanding in your company or, you know some take some, some it doesn't need to be like a search engine concept.
[00:15:07] Vibhu Arora: It could be like everybody has a checkout checkout system. How does fraud detection work for checkout system in, in, in my company? And take that concept and explain it to the vp and you can, in order to do that, you will need to learn it yourself. So you will need to sit down with your team. I need to tell the team, Hey, I'm going to explain your work to the vp.
[00:15:35] Vibhu Arora: So, Help me simplify it and pat it in a way so that I can tell your story to this influential leader. So that's another way, you know, teams will open up and the cost and the burden of sharing information. You will see some of that go away. Team will start like sharing information. You will put together something, you know, nice, like a deck [00:16:00] or a document, and then you share that with your, with your senior leader.
[00:16:05] Vibhu Arora: That's when you will establish credibility and with your leader and you will establish bridges. This person will start relying on you. So some of these things, you know, you can do as a junior product person which is a lot around like you know, documenting and synthesizing information. And I finish off with like this, this us example for the longest time in our org.
[00:16:36] Vibhu Arora: We had the challenge of documenting how the Walmart search engine works, right? And we had like bits and pieces of this, but we never had like a grand simple to understand narrative and the grand, like simple to understand like storyline around this. And after like many, like years and years, [00:17:00] it took a junior product person in, in the role to actually come together, bring all the teams together, and to structure that simple story, like a very simple story.
[00:17:13] Vibhu Arora: This is how the search engine works at Walmart. These are the five pieces, these are the algorithms, these are the signals, these are the inputs outputs. So one of the most, most difficult concepts was actually simplified by a junior product person. So I guess the, the, the thing here is, Also don't underestimate your, your value and, and you know, the value you can bring by doing such an exercise.
[00:17:42] Vibhu Arora: So that's, that could be pretty powerful for the organization too.
[00:17:45] Himakara Pieris: what's your advice to a senior product leader who wants to focus on AI and ML products.
[00:17:52] Vibhu Arora: Let's, let's do role play here. So let's say I suddenly find myself leadership position. And there could be [00:18:00] like two scenarios, right?
[00:18:01] Vibhu Arora: Scenario A could be, I have risen through the ranks, off the AI landscape, meaning I know what it takes. And, and I was, I was actually the pm at the PM level. I was launching features, based on ML models. So I, I, I know the landscape roundup, that's scenario a, for a leader, where they have the background, the, the, the lowest level of background needed.
[00:18:28] Vibhu Arora: Scenario B could be you just happen to be a sharp, savvy leader. And, you somehow found, find yourself leading a large team, who is using technology, this technology heavily, but not necessarily. You might not have had the, the experience with, with. You know, getting your dirty and understanding the nuances and ins and out of, of machine learning landscape.
[00:18:55] Vibhu Arora: So I think both can be interesting. Like I think, scenario a, [00:19:00] probably is for that person is going to be more about like the, the usual leadership. Issues and problems than anything else, which is like, you know, how is the org structured? How is the org thinking? How do I lead teams? How do I motivate teams?
[00:19:17] Vibhu Arora: how do I, get people to focus? How do I get people to like, work together like, like, you know, like the usual leadership challenges because this person does not have the challenge of understanding the stack, understanding the, the, the domain. So I think it's probably like this person just needs to uplevel the, the, the leadership, the usual sort of leadership, you know, gap.
[00:19:41] Vibhu Arora: But I think for the other person, I think it could be, it could be, you know, like fun and challenging at the same time. because I think the, the main thing this person needs to overcome is a, their own sort of like understanding. off, off the, the stack, the ML stack [00:20:00] and the ecosystem. and b, they need to build trust, in their teams that their teams actually trust, them as a second step.
[00:20:08] Vibhu Arora: So I think these could be like the two, two, you know, big, big sort of like, areas for this leader to focus on. And if you are, if you are like, you know, falling into this space, then one of the things which, which, you know, I can sort of like, say is, you know, jump right in. You know, let's say, I'm just making an example.
[00:20:28] Vibhu Arora: Let's say you are the, the chief product officer of Facebook newsfeed, right? Today, starting today. And you never had like an experience in ai. Well, the, the easiest way to get to the bottom of this is to dock food like crazy. Meaning like, use your product very heavily. And when you start using your product, really you will start having questions.
[00:20:53] Vibhu Arora: Hey, why do I see the post from, interest I have in soccer [00:21:00] being prioritized over my childhood friend's, birthday photo, right? It's, it's a, it's a very, you know, simple curious inquiry. And through this, dog fooding or using your own product, you will form these questions and through these questions, you will actually open up dialogue with your team and this channel, I know it's kind of like sounding a bit tactical, but usually this channel is really, really, productive and it's also like very disarming for the team to like, Share more with you because you actually came in with a question.
[00:21:40] Vibhu Arora: and it could be a valid question. So team, any, any sort of like good intention per person will welcome a alert question. So team will actually start sharing more. You will start learning more. so that could be like a really, really good way, to, to, you know, understand and, and dig deep.[00:22:00]
[00:22:00] Vibhu Arora: The, other thing I think at some point, you will need as a leader to create some sort of a high level map of your ecosystem. So just as an example, right? Like for for search, you know, there's different components of Sergio. You start with, expression. then from an expression there's an understanding component from understanding.
[00:22:29] Vibhu Arora: There is like a recall, where the records are pulled in from the recall step. There's a ranking, re ranking, and there's re-ranking. Then there is presentation just like a super basic, you know, like, the five tips of the first, search result that you come up with. So any new leader will probably need to quickly break down the stack into its sub-components and, and have like a visceral understanding of [00:23:00] these sub-components.
[00:23:00] Vibhu Arora: Like what does each of these sub components do, number one and number two, which is gonna be super crucial. Understanding the majority of each of these sub-components compared to where you want to be. So let's just take that example of expression, and understanding, you know, let's just take those two pieces.
[00:23:23] Vibhu Arora: So expression, meaning am I able to express something easily into a search engine, which is, you know, component number one. And component number two is once I have expressed something, is the search engine able to understand it? So these two components can, can actually be pretty different in terms of maturity, in terms of where the organization will be.
[00:23:47] Vibhu Arora: So then the new leader can actually bring a lot of focus to the team, that out of these five component. Hey, these two or three seem to be doing fine. They're okay. Yes, we can make some [00:24:00] incremental bets, but guess what? These two components are really sub subpar performing, not performing well, and we should double down and invest on these two components.
[00:24:12] Vibhu Arora: And that's another angle where the leader can actually bring a lot of like, experience and strategic focus to the org and the team. and through this exercise again, you know, they will sort of like start going, they will get opportunity to dive deep into each of these, like, you know, unpack these boxes, unpack these components, learn more.
[00:24:36] Vibhu Arora: but they, they will actually add a lot of value to the organization as well, on the way by, by empowering. To focus on the most important components and just like, forget about the, the components which are working well. Because, you know, the, the main thing in, in, in product is like focusing and prioritizing the right problem.
[00:24:57] Vibhu Arora: Those could be , few tactics the leaders can [00:25:00] adopt.
[00:25:00] Himakara Pieris: Is there anything else that you'd like to share with our audience?
[00:25:03] Vibhu Arora: Yes, absolutely. We are hiring and our team is always growing. This time we actually have one role open on my team and for anybody who is listening to this podcast whoever this is reaching out to, if you or someone you know is interested in joining a stellar team, creating a lot of impact please don't hesitate to reach out.
[00:25:30] Vibhu Arora: We would love to have a conversation with you.
[00:25:36] Hima: Smart products is brought to you by hydra.ai. Hydra helps product teams explore how they can introduce AI powered features to their products and deliver unique customer value. Learn more at www.hydra.ai/smart-products.
-
Waleed is a product manager at Untether AI.
Untether AI is building a processor that is designed specifically for AI workloads.
During this conversation, Waleed shared his thoughts on why we need a new processor architecture for AI, the cost implications of running AI workloads, and what it would mean to reduce the cost of operating AI.
Links
Waleed on LinkedIn
Untether AI
-
Sid is an AI product manager at GE HealthCare. In this episode, we talk about Sid’s approach to identifying AI use cases, his workflow from idea to production as an AI product manager, and his advice for product managers interested in leading AI products. Please check the notes section for links.
Links
Sid on LinkedIn
CRISP-DM: Towards a Standard Process Model for Data Mining
AIR Recon DL paper on arXiv
-
Wissem is the VP of innovation at BusPas — a company that is creating a building block for smart mobility. BusPas uses their SCiNe AIOT device to collect data and make predictions on the edge.
Links
Wissem Maazoun on LinkedIn
Learn more about BusPas
Wissem showing us BusPas' SciNe device
-
Vinod Subramanian is the Chief data and product development officer at Syapse.
Syapse collaborates with health systems, life sciences companies, and Food And Drug Administration to advance care for patients with serious diseases through precision medicine powered by AI.
During this conversation, he shared his thoughts on how to identify the right use cases for AI, how to approach explainability, and how to think about talent when you are starting off with an AI project.
Links
Vinod Subramanian (@VinDNA) / Twitter
Syapse on the web
-
In the fall of 2022, we set out to interview hundred product leaders to learn what they perceive to be the challenges and opportunities associated with embedding AI into their own products.
When we compiled all their responses, it turned out there are ten types of challenges.
The goal of this podcast is to interview people who have already gone through these challenges and have come up with ways to work through these challenges.
We look forward to identifying, celebrating, and learning from industry leaders who are using AI to deliver real-world value and sharing some actionable advice and frameworks with all of you.
Links
Get the research report => https://www.hydra.ai/report
Learn more about the podcast and submit guest suggestions => https://www.smartproducts.show
Get in touch: [email protected]
