Folgen
-
âLast week was a great year in GenAI,â jokes Mark Ramseyâand itâs a great philosophy to have as LLM tools especially continue to evolve at such a rapid rate. This week, youâll get to hear my fun and insightful chat with Mark from Ramsey International about the world of large language models (LLMs) and how we make useful UXs out of them in the enterprise.
Mark shared some fascinating insights about using a companyâs website information (data) as a place to pilot a LLM project, avoiding privacy landmines, and how re-ranking of models leads to better LLM response accuracy. We also talked about the importance of real human testing to ensure LLM chatbots and AI tools truly delight users. From amusing anecdotes about the spinning beach ball on macOS to envisioning a future where AI-driven chat interfaces outshine traditional BI tools, this episode is packed with forward-looking ideas and a touch of humor.
Highlights/ Skip to:(0:50) Why is the world of GenAI evolving so fast?(4:20) How Mark thinks about UX in an LLM application(8:11) How Mark defines âSpecialized GenAI?â(12:42) Markâs consulting work with GenAI / LLMs these days(17:29) How GenAI can help the healthcare industry(30:23) Uncovering usersâ true feelings about LLM applications(35:02) Are UIs moving backwards as models progress forward?(40:53) How will GenAI impact data and analytics teams?(44:51) Will LLMs be able to consistently leverage RAG and produce proper SQL?(51:04) Where can find more from Mark and Ramsey InternationalQuotes from Todayâs EpisodeâWith [GenAI], we have a solution that weâve built to try to help organizations, and build workflows. We have a workflow that we can run and ask the same question [to a variety of GenAI models] and see how similar the answers are. Depending on the complexity of the question, you can see a lot of variability between the models⊠[and] we can also run the same question against the different versions of the model and see how itâs improved. Folks want a human-like experience interacting with these models.. [and] if the model can start responding in just a few seconds, that gives you much more of a conversational type of experience.â - Mark Ramsey (2:38)â[People] donât understand when you interact [with GenAI tools] and it brings tokens back in that streaming fashion, youâre actually seeing inside the brain of the model. Every token it produces is then displayed on the screen, and it gives you that typewriter experience back in the day. If someone has to wait, and all youâre seeing is a logo spinning, from a UX experience standpoint⊠people feel like the model is much faster if it just starts to produce those results in that streaming fashion. I think in a design, itâs extremely important to take advantage of that [...] as opposed to waiting to the end and delivering the results some models support that, and other models donât.â- Mark Ramsey (4:35)"All of the data thatâs on the website is public information. Weâve done work with several organizations on quickly taking the data thatâs on their website, packaging it up into a vector database, and making that be the source for questions that their customers can ask. [Organizations] publish a lot of information on their websites, but people really struggle to get to it. Weâve seen a lot of interest in vectorizing website data, making it available, and having a chat interface for the customer. The customer can ask questions, and it will take them directly to the answer, and then they can use the website as the source information.â - Mark Ramsey (14:04)âIâm not skeptical at all. Iâve changed much of my [AI chatbot searches] to Perplexity, and I think itâs doing a pretty fantastic job overall in terms of quality. Itâs returning an answer with citations, so you have a sense of where itâs sourcing the information from. I think itâs important from a user experience perspective. This is a replacement for broken search, as I really donât want to read all the web pages and PDFs you have that *might* be about my chiropractic care query to answer my actual [healthcare] question.â - Brian OâNeill (19:22)âWeâve all had great experience with customer service, and weâve all had situations where the customer service was quite poor, and weâre going to have that same thing as we begin to [release more] chatbots. We need to make sure we try to alleviate having those bad experiences, and have an exit. If someone is running into a situation where theyâd rather talk to a live person, have that ability to route them to someone else. Thatâs why the robustness of the model is extremely important in the implementation⊠and right now, organizations like OpenAI and Anthropic are significantly better at that [human-like] experience.â - Mark Ramsey (23:46)"Thereâs two aspects of these models: the training aspect and then using the model to answer questions. I recommend to organizations to always augment their content and donât just use the training data. Youâll still get that human-like experience thatâs built into the model, but youâll eliminate the hallucinations. If you have a model that has been set up correctly, you shouldnât have to ask questions in a funky way to get answers.â - Mark Ramsey (39:11)âPeople need to understand GenAI is not a predictive algorithm. It is not able to run predictions, it struggles with some math, so that is not the focus for these models. Whatâs interesting is that you can use the model as a step to get you [the answers]. A lot of the models now support functions⊠when you ask a question about something that is in a database, it actually uses its knowledge about the schema of the database. It can build the query, run the query to get the data back, and then once it has the data, it can reformat the data into something that is a good response back." - Mark Ramsey (42:02) LinksMark on LinkedInRamsey InternationalEmail: mark [at] ramsey.internationalRamsey International's YouTube Channel -
Guess what? Data science and AI initiatives are still failing here in 2024âdespite widespread awareness. Is that news? Candidly, youâll hear me share with Evan Shellshearâauthor of the new book Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analyticsâabout how much I actually didnât want to talk about this story originally on my podcastâbecause itâs not news! However, what is news is what the data says behind Evanâs findingsâand guess what? Itâs not the technology.
In our chat, Evan shares why he wanted to leverage a human approach to understand the root cause of multiple organizationsâ failures and how this approach highlighted the disconnect between data scientists and decision-makers. He explains the human factors at play, such as poor problem surfacing and organizational culture challengesâand how these human-centered design skills are rarely taught or offered to data scientists. The conversation delves into why these failures are more prevalent in data science compared to other fields, attributing it to the complexity and scale of data-related problems. We also discuss how analytically mature companies can mitigate these issues through strategic approaches and stakeholder buy-in. Join us as we delve into these critical insights for improving data science project outcomes.
Highlights/ Skip to:(4:45) Why are data science projects still failing?(9:17) Why is the disconnect between data scientists and decision-makers so pronounced relative to, say, engineering? (13:08) Why are data scientists not getting enough training for real-world problems?(16:18) What the data says about failure rates for mature data teams vs. immature data teams(19:39) How to change peopleâs opinions so they value data more(25:16) What happens at the stage where the beneficiaries of data donât actually see the benefits?(31:09) What are the skills needed to prevent a repeating pattern of creating data products that customers ignore??(37:10) Where do more mature organizations find non-technical help to complement their data science and AI teams? (41:44) Are executives and directors aware of the skills needed to level up their data science and AI teams?Quotes from Todayâs EpisodeâPeople know this stuff. Itâs not news anymore. And so, the reason why we needed this was really to dig in. And exactly like you did, like, keeping that list of articles is brilliant, and knowing whatâs causing the failures and whatâs leading to these issues still arising is really important. But at some point, we need to approach this in a scientific fashion, and we need to unpack this, and we need to really delve into the details beyond just the headlines and the articles themselves. And start collating and analyzing this to properly figure out whatâs going wrong, and what do we need to do about it to fix it once and for all so you can stop your endless collection, and the AI Incident Database that now has over 3500 entries. It can hang its hat and say, âIâve done my job. Itâs time to move on. Weâre not failing as we used to.ââ - Evan Shellshear (3:01)"What we did is we took a number of different studies, and we split companies into what we saw as being analytically matureâand this is a common, well-known thing; there are many maturity frameworks exist across data, across AI, across all different areasâand what we call analytically immature, so those companies that probably arenât there yet. And what we wanted to draw a distinction is okay, we say 80% of projects fail, or whatever the exact number is, but for who? And for what stage and for what capability? And so, what we then went and did is we were able to take our data and look at which failures are common for analytically immature organizations, and which failures are common for analytically mature organizations, and then weâre able to understand, okay, in the market, how many organizations do we think are analytically mature versus analytically immature, and then we were able to take that 80% failure rate and establish it. For analytically mature companies, the failure rate is probably more like 40%. For analytically immature companies, itâs over 90%, right? And so, youâre exactly right: organizations can do something about it, and they can build capabilities in to mitigate this. So definitely, it can be reduced. Definitely, it can be brought down. You might say, 40% is still too high, but it proves that by bringing in these procedures, youâre completely correct, that it can be reduced.â - Evan Shellshear (14:28)"What happens with the data science person, however, is typically theyâre seen as a cost centerâtypically, not always; nowadays, that dialog is changingâand what they need to do is find partners across the other parts of the business. So, theyâre going to go into the supply chain team, theyâll go into the merchandising team, theyâll go into the banking team, theyâll go into the other teams, and theyâre going to find their supporters and winners there, and theyâre going to probably build out from there. So, the first step would likely be, if youâre a big enough organization that youâre not having that strategy the executive level is to find your friendsâand there will be some of the organization who support this data strategyâand get some wins for them.â - Evan Shellshear (24:38)âItâs not like thereâs this box you put one in the other in. Because, like success and failure, thereâs a continuum. And companies as they move along that continuum, just like you said, this year, we failed on the lack of executive buy-in, so letâs fix that problem. Next year, we fail on not having the right resources, so we fix that problem. And you move along that continuum, and you build it up. And at some point as youâre going on, that failure rate is dropping, and youâre getting towards that end of the scale where youâve got those really capable companies that live, eat, and breathe data science and analytics, and so have to have these to be able to survive, otherwise a simple company evolution would have wiped them out, and they wouldnât exist if they didnât have that capability, if thatâs their core thing.â - Evan Shellshear (18:56)âNothing else could be correct, right? This subjective intuition and all this stuff, itâs never going to be as good as the data. And so, what happens is, is you, often as a data scientistâand Iâve been subjected to this myselfâcome in with this arrogance, this kind of data-driven arrogance, right? And itâs not a good thing. It puts up barriers, it creates issues, it separates you from the people.â - Evan Shellshear (27:38)"Knowing that youâre going to have to go on that journey from day one, you canât jump from level zero to level five. Thatâs what all these data maturity models are about, right? You canât jump from level zero data maturity to level five overnight. You really need to take those steps and build it up.â - Evan Shellshear (45:21)"What weâre talking about, itâs not new. Itâs just old wine in a new skin, and weâre just presenting it for the data science age." - Evan Shellshear (48:15)LinksWhy Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, without the Hype: https://www.routledge.com/Why-Data-Science-Projects-Fail-the-Harsh-Realities-of-Implementing-AI-and-Analytics-without-the-Hype/Gray-Shellshear/p/book/9781032660301 LinkedIn: https://www.linkedin.com/in/eshellshear/ Get the Book:Get 20% off at Routledge.com w/ code dspf20 Get it at AmazonWhy do we still teach people to calculate? (People I Mostly Admire podcast) -
Fehlende Folgen?
-
Ready for more ideas about UX for AI and LLM applications in enterprise environments? In part 2 of my topic on UX considerations for LLMs, I explore how an LLM might be used for a fictitious use case at an insurance companyâspecifically, to help internal tools teams to get rapid access to primary qualitative user research. (Yes, itâs a little âmetaâ, and Iâm also trying to nudge you with this hypothetical exampleâno secret!) ;-) My goal with these episodes is to share questions you might want to ask yourself such that any use of an LLM is actually contributing to a positive UX outcome Join me as I cover the implications for design, the importance of foundational data quality, the balance between creative inspiration and factual accuracy, and the never-ending discussion of how we might handle hallucinations and errors posing as âfactsââall with a UX angle. At the end, I also share a personal story where I used an LLM to help me do some shopping for my favorite product: TRIP INSURANCE! (NOT!)
Highlights/ Skip to:(1:05) I introduce a hypothetical internal LLM tool and what the goal of the tool is for the team who would use it (5:31) Improving access to primary research findings for better UX (10:19) What âquality dataâ means in a UX context(12:18) When LLM accuracy maybe doesnât matter as much(14:03) How AI and LLMs are opening the door for fresh visioning work(15:38) Brianâs overall take on LLMs inside enterprise software as of right now(18:56) Final thoughts on UX design for LLMs, particularly in the enterprise(20:25) My inspiration for these 2 episodesâand how I had to use ChatGPT to help me complete a purchase on a website that could have integrated this capability right into their websiteQuotes from Todayâs EpisodeâIf we accept that the goal of most product and user experience research is to accelerate the production of quality services, products, and experiences, the question is whether or not using an LLM for these types of questions is moving the needle in that direction at all. And secondly, are the potential downsides like hallucinations and occasional fabricated findings, is that all worth it? So, this is a design for AI problem.â - Brian T. OâNeill (8:09)âWhatâs in our data? Can the right people change it when the LLM is wrong? The data product managers and AI leaders reading this or listening know that the not-so-secret path to the best AI is in the foundational data that the models are trained on. But what does the word *quality* mean from a product standpoint and a risk reduction one, as seen from an end-usersâ perspective? Somebody whoâs trying to get work done? This is a different type of quality measurement.â - Brian T. OâNeill (10:40)âWhen we think about fact retrieval use cases in particular, how easily can product teamsâinternal or otherwiseâand end-users understand the confidence of responses? When responses are wrong, how easily, if at all, can users and product teams update the modelâs responses? Errors in large language models may be a significant design consideration when we design probabilistic solutions, and we no longer control what exactly our products and software are going to show to users. If bad UX can include leading people down the wrong path unknowingly, then AI is kind of like the team on the other side of the tug of war that weâre playing.â - Brian T. OâNeill (11:22)âAs somebody who writes a lot for my consulting business, and composes music in another, one of the hardest parts for creators can be the zero-to-one problem of getting startedâthe blank pageâand this is a place where I think LLMs have great potential. But it also means we need to do the proper research to understand our audience, and when or where theyâre doing truly generative or creative workâsuch that we can take a generative UX to the next level that goes beyond delivering banal and obviously derivative content.â - Brian T. OâNeill (13:31)âOne thing I actually like about the hype, investment, and excitement around GenAI and LLMs in the enterprise is that there is an opportunity for organizations here to do some fresh visioning work. And this is a place that designers and user experience professionals can help data teams as we bring design into the AI space.â - Brian T. OâNeill (14:04)âIf there was ever a time to do some new visioning work, I think now is one of those times. However, we need highly skilled design leaders to help facilitate this in order for this to be effective. Part of that skill is knowing who to include in exercises like this, and my perspective, one of those people, for sure, should be somebody who understands the data science side as well, not just the engineering perspective. And as I posited in my seminar that I teach, the AI and analytical data product teams probably need a fourth member. Itâs a quartet and not a trio. And that quartet includes a data expert, as well as that engineering lead.â - Brian T. OâNeill (14:38)LinksPerplexity.ai: https://perplexity.ai Ideaflow: https://www.amazon.com/Ideaflow-Only-Business-Metric-Matters/dp/0593420586 My article that inspired this episode -
Letâs talk about design for AI (which more and more, Iâm agreeing means GenAI to those outside the data space). The hype around GenAI and LLMsâparticularly as it relates to dropping these in as features into a software application or productâseems to me, at this time, to largely be driven by FOMO rather than real value. In this âpart 1â episode, I look at the importance of solid user experience design and outcome-oriented thinking when deploying LLMs into enterprise products. Challenges with immature AI UIs, the role of context, the constant game of understanding what accuracy means (and how much this matters), and the potential impact on human workers are also examined. Through a hypothetical scenario, I illustrate the complexities of using LLMs in practical applications, stressing the need for careful consideration of benchmarks and the acceptance of GenAI's risks.
I also want to note that LLMs are a very immature space in terms of UI/UX designâeven if the foundation models continue to mature at a rapid pace. As such, this episode is more about the questions and mindset I would be considering when integrating LLMs into enterprise software more than a suggestion of âbest practices.â
Highlights/ Skip to:
(1:15) Currently, many LLM feature initiatives seem to mostly driven by FOMO (2:45) UX Considerations for LLM-enhanced enterprise applications (5:14) Challenges with LLM UIs / user interfaces(7:24) Measuring improvement in UX outcomes with LLMs(10:36) Accuracy in LLMs and its relevance in enterprise software (11:28) Illustrating key consideration for implementing an LLM-based feature(19:00) Leadership and context in AI deployment(19:27) Determining UX benchmarks for using LLMs(20:14) The dynamic nature of LLM hallucinations and how we design for the unknown(21:16) Closing thoughts on Part 1 of designing for AI and LLMsQuotes from Todayâs Episode
âWhile many product teams continue to race to deploy some sort of GenAI and especially LLMs into their productsâparticularly this is in the tech sector for commercial software companiesâthe general sense Iâm getting is that this is still more about FOMO than anything else.â - Brian T. OâNeill (2:07)âNo matter what the technology is, a good user experience design foundation starts with not doing any harm, and hopefully going beyond usable to be delightful. And adding LLM capabilities into a solution is really no different. So, we still need to have outcome-oriented thinking on both our product and design teams when deploying LLM capabilities into a solution. This is a cornerstone of good product work.â - Brian T. OâNeill (3:03)âSo, challenges with LLM UIs and UXs, right, user interfaces and experiences, the most obvious challenge to me right now with large language model interfaces is that while weâve given users tremendous flexibility in the form of a Google search-like interface, weâve also in many cases, limited the UX of these interactions to a text conversation with a machine. Weâre back to the CLI in some ways.â - Brian T. OâNeill (5:14)âBefore and after we insert an LLM into a userâs workflow, we need to know what an improvement in their life or work actually means.â- Brian T. OâNeill (7:24)"If it would take the machine a few seconds to process a result versus what might take a day for a worker, whatâs the role and purpose of that worker going forward? I think these are all considerations that need to be made, particularly if youâre concerned about adoption, which a lot of data product leaders are." - Brian T. OâNeill (10:17)âSo, thereâs no right or wrong answer here. These are all range questions, and theyâre leadership questions, and context really matters. They are important to ask, particularly when we have this risk of reacting to incorrect information that looks plausible and believable because of how these LLMs tend to respond to us with a positive sheen much of the time.â - Brian T. OâNeill (19:00)Links
View Part 1 of my article on UI/UX design considerations for LLMs in enterprise applications: https://designingforanalytics.com/resources/ui-ux-design-for-enterprise-llms-use-cases-and-considerations-for-data-and-product-leaders-in-2024-part-1/ -
Ben Shneiderman is a leading figure in the field of human-computer interaction (HCI). Having founded one of the oldest HCI research centers in the country at the University of Maryland in 1983, Shneiderman has been intently studying the design of computer technology and its use by humans. Currently, Ben is a Distinguished University Professor in the Department of Computer Science at the University of Maryland and is working on a new book on human-centered artificial intelligence.
Iâm so excited to welcome this expert from the field of UX and design to todayâs episode of Experiencing Data! Ben and I talked a lot about the complex intersection of human-centered design and AI systems.
In our chat, we covered:
Ben's career studying human-computer interaction and computer science. (0:30)'Building a culture of safety': Creating and designing âsafe, reliable and trustworthyâ AI systems. (3:55)'Like zoning boards': Why Ben thinks we need independent oversight of privately created AI. (12:56)'Thereâs no such thing as an autonomous device': Designing human control into AI systems. (18:16)A/B testing, usability testing and controlled experiments: The power of research in designing good user experiences. (21:08)Designing âcomprehensible, predictable, and controllableâ user interfaces for explainable AI systems and why [explainable] XAI matters. (30:34)Ben's upcoming book on human-centered AI. (35:55)Resources and Links:People-Centered Internet: https://peoplecentered.net/Designing the User Interface (one of Benâs earlier books): https://www.amazon.com/Designing-User-Interface-Human-Computer-Interaction/dp/013438038XBridging the Gap Between Ethics and Practice: https://doi.org/10.1145/3419764Partnership on AI: https://www.partnershiponai.org/AI incident database: https://www.partnershiponai.org/aiincidentdatabase/University of Maryland Human-Computer Interaction Lab: https://hcil.umd.edu/ACM Conference on Intelligent User Interfaces: https://iui.acm.org/2021/hcai_tutorial.htmlHuman-Computer Interaction Lab, University of Maryland, Annual Symposium: https://hcil.umd.edu/tutorial-human-centered-ai/Ben on Twitter: https://twitter.com/benbendc Quotes from Todayâs EpisodeThe world of AI has certainly grown and blossomed â itâs the hot topic everywhere you go. Itâs the hot topic among businesses around the world â governments are launching agencies to monitor AI and are also making regulatory moves and rules. ⊠People want explainable AI; they want responsible AI; they want safe, reliable, and trustworthy AI. They want a lot of things, but theyâre not always sure how to get them. The world of human-computer interaction has a long history of giving people what they want, and what they need. That blending seems like a natural way for AI to grow and to accommodate the needs of real people who have real problems. And not only the methods for studying the users, but the rules, the principles, the guidelines for making it happen. So, thatâs where the action is. Of course, what we really want from AI is to make our world a better place, and thatâs a tall order, but we start by talking about the things that matter â the human values: human rights, access to justice, and the dignity of every person. We want to support individual goals, a personâs sense of self-efficacy â they can do what they need to in the world, their creativity, their responsibility, and their social connections; they want to reach out to people. So, those are the sort of high aspirational goals that become the hard work of figuring out how to build it. And thatâs where we want to go. - Ben (2:05)The software engineering teams creating AI systems have got real work to do. They need the right kind of workflows, engineering patterns, and Agile development methods that will work for AI. The AI world is different because itâs not just programming, but it also involves the use of data thatâs used for training. The key distinction is that the data that drives the AI has to be the appropriate data, it has to be unbiased, it has to be fair, it has to be appropriate to the task at hand. And many people and many companies are coming to grips with how to manage that. This has become controversial, letâs say, in issues like granting parole, or mortgages, or hiring people. There was a controversy that Amazon ran into when its hiring algorithm favored men rather than women. Thereâs been bias in facial recognition algorithms, which were less accurate with people of color. Thatâs led to some real problems in the real world. And thatâs where we have to make sure we do a much better job and the tools of human-computer interaction are very effective in building these better systems in testing and evaluating. - Ben (6:10)
Every company will tell you, âWe do a really good job in checking out our AI systems.â Thatâs great. We want every company to do a really good job. But we also want independent oversight of somebody whoâs outside the company â someone who knows the field, whoâs looked at systems at other companies, and who can bring ideas and bring understanding of the dangers as well. These systems operate in an adversarial environment â there are malicious actors out there who are causing trouble. You need to understand what the dangers and threats are to the use of your system. You need to understand where the biases come from, what dangers are there, and where the software has failed in other places. You may know what happens in your company, but you can benefit by learning what happens outside your company, and thatâs where independent oversight from accounting companies, from governmental regulators, and from other independent groups is so valuable. - Ben (15:04)
Thereâs no such thing as an autonomous device. Someone owns it; somebodyâs responsible for it; someone starts it; someone stops it; someone fixes it; someone notices when itâs performing poorly. ⊠Responsibility is a pretty key factor here. So, if thereâs something going on, if a manager is deciding to use some AI system, what they need is a control panel, let them know: whatâs happening? Whatâs it doing? Whatâs going wrong and whatâs going right? That kind of supervisory autonomy is what I talk about, not full machine autonomy thatâs hidden away and you never see it because thatâs just head-in-the-sand thinking. What you want to do is expose the operation of a system, and where possible, give the stakeholders who are responsible for performance the right kind of control panel and the right kind of data. ⊠Feedback is the breakfast of champions. And companies know that. They want to be able to measure the success stories, and they want to know their failures, so they can reduce them. The continuous improvement mantra is alive and well. We do want to keep tracking whatâs going on and make sure it gets better. Every quarter. - Ben (19:41)
Google has had some issues regarding hiring in the AI research area, and so has Facebook with elections and the way that algorithms tend to become echo chambers. These companies â and this is not through heavy research â probably have the heaviest investment of user experience professionals within data science organizations. They have UX, ML-UX people, UX for AI people, theyâre at the cutting edge. I see a lot more generalist designers in most other companies. Most of them are rather unfamiliar with any of this or what the ramifications are on the design work that theyâre doing. But even these largest companies that have, probably, the biggest penetration into the most number of people out there are getting some of this really important stuff wrong. - Brian (26:36)
Explainability is a competitive advantage for an AI system. People will gravitate towards systems that they understand, that they feel in control of, that are predictable. So, the big discussion about explainable AI focuses on whatâs usually called post-hoc explanations, and the Shapley, and LIME, and other methods are usually tied to the post-hoc approach.That is, you use an AI model, you get a result and you say, âWhat happened?â Why was I denied a parole, or a mortgage, or a job? At that point, you want to get an explanation. Now, that idea is appealing, but Iâm afraid I havenât seen too many success stories of that working. ⊠Iâve been diving through this for years now, and Iâve been looking for examples of good user interfaces of post-hoc explanations. It took me a long time till I found one. The culture of AI model-building would be much bolstered by an infusion of thinking about what the user interface will be for these explanations. And even the DARPAâs XAIâExplainable AIâproject, which has 11 projects within itâhas not really grappled with this in a good way about designing what itâs going to look like. Show it to me. ⊠There is another way. And the strategy is basically prevention. Letâs prevent the user from getting confused and so they donât have to request an explanation. We walk them along, let the user walk through the stepâthis is like Amazon checkout process, seven-step processâand you know whatâs happened in each step, you can go back, you can explore, you can change things in each part of it. Itâs also what TurboTax does so well, in really complicated situations, and walks you through it. ⊠You want to have a comprehensible, predictable, and controllable user interface that makes sense as you walk through each step. - Ben (31:13)
-
Wait, Iâm talking to a head of data management at a tech company? Why!? Well, today I'm joined by Malcolm Hawker to get his perspective around data products and what heâs seeing out in the wild as Head of Data Management at Profisee. Why Malcolm? Malcolm was a former head of product in prior roles, and for several years, Iâve enjoyed Malcolmâs musings on LinkedIn about the value of a product-oriented approach to ML and analytics. We had a chance to meet at CDOIQ in 2023 as well and he went on my âneed to do an episodeâ list!
According to Malcom, empathy is the secret to addressing key UX questions that ensure adoption and business value. He also emphasizes the need for data experts to develop business skills so that they're seen as equals by their customers. During our chat, Malcolm stresses the benefits of a product- and customer-centric approach to data products and what data professionals can learn approaching problem solving with a product orientation.
Highlights/ Skip to:Malcolmâs definition of a data product (2:10)Understanding your customersâ needs is the first step toward quantifying the benefits of your data product (6:34)How product makers can gain access to users to build more successful products (11:36) Answering the UX question to get past the adoption stage and provide business value (16:03)Data experts must develop business expertise if they want to be seen as equals by potential customers (20:07)What people really mean by âdata culture" (23:02)Malcolmâs data product journey and his changing perspective (32:05)Using empathy to provide a better UX in design and data (39:24)Avoiding the death of data science by becoming more product-driven (46:23)Where the majority of data professionals currently land on their view of product management for data products (48:15)Quotes from Todayâs EpisodeâMy definition of a data product is something that is built by a data and analytics team that solves a specific customer problem that the customer would otherwise be willing to pay for. Thatâs it.â - Malcolm Hawker (3:42)âYou need to observe how your customer uses data to make better decisions, optimize a business process, or to mitigate business risk. You need to know how your customers operate at a very, very intimate level, arguably, as well as they know how their business processes operate.â - Malcolm Hawker (7:36)âSo, be a problem solver. Be collaborative. Be somebody who is eager to help make your customersâ lives easier. You hear "no" when people think that youâre a burden. You start to hear more âyesesâ when people think that you are actually invested in helping make their lives easier.â - Malcolm Hawker (12:42)âWe [data professionals] put data on a pedestal. We develop this mindset that the data matters moreâas much or maybe even more than the business processes, and that is not true. We would not exist if it were not for the business. Hard stop.â - Malcolm Hawker (17:07)âI hate to say it, I think a lot of this data stuff should kind of feel invisible in that way, too. Itâs like this invisible ally that youâre not thinking about the dashboard; you just access the information as part of your natural workflow when you need insights on making a decision, or a status check that youâre on track with whatever your goal was. Youâre not really going out of mode.â - Brian OâNeill (24:59)âBut you know, data people are basically librarians. We want to put things into classifications that are logical and work forwards and backwards, right? And in the product world, sometimes they just donât, where you can have something be a product and be a material to a subsequent product.â - Malcolm Hawker (37:57)âSo, the broader point here is just more of a mindset shift. And you know, maybe these things arenât necessarily a bad thing, but how do we become a little more product- and customer-driven so that we avoid situations where everybody thinks what weâre doing is a time waster?â - Malcolm Hawker (48:00)LinksProfisee: https://profisee.com/ LinkedIn: https://www.linkedin.com/in/malhawker/ CDO Matters: https://profisee.com/cdo-matters-live-with-malcolm-hawker/ -
Welcome to another curated, Promoted Episode of Experiencing Data!
In episode 144, Shashank Garg, Co-Founder and CEO of Infocepts, joins me to explore whether all this discussion of data products out on the web actually has substance and is worth the perceived extra effort. Do we always need to take a product approach for ML and analytics initiatives? Shashank dives into how Infocepts approaches the creation of data solutions that are designed to be actionable within specific business workflowsâand as I often do, I started out by asking Shashank how he and Infocepts define the term âdata product.â We discuss a few real-world applications Infocepts has built, and the measurable impact of these data productsâas well as some of the challenges theyâve faced that your team might as well. Skill sets also came up; who does design? Who takes ownership of the product/value side? And of course, we touch a bit on GenAI.
Highlights/ Skip to
Shashank gives his definition of data products (01:24)We tackle the challenges of user adoption in data products (04:29)We discuss the crucial role of integrating actionable insights into data products for enhanced decision-making (05:47)Shashank shares insights on the evolution of data products from concept to practical integration (10:35)We explore the challenges and strategies in designing user-centric data products (12:30)I ask Shashank about typical environments and challenges when starting new data product consultations (15:57)Shashank explains how Infocepts incorporates AI into their data solutions (18:55)We discuss the importance of understanding user personas and engaging with actual users (25:06)Shashank describes the roles involved in data product developmentâs ideation and brainstorming stages (32:20)The issue of proxy users not truly representing end-users in data product design is examined (35:47)We consider how organizations are adopting a product-oriented approach to their data strategies (39:48)Shashank and I delve into the implications of GenAI and other AI technologies on product orientation and user adoption (43:47)Closing thoughts (51:00)Quotes from Todayâs Episode
âData products, at least to us at Infocepts, refers to a way of thinking about and organizing your data in a way so that it drives consumption, and most importantly, actions.â - Shashank Garg (1:44)âThe way I see it is [that] the role of a DPM (data product manager)âwhether they have the title or notâis benefits creation. You need to be responsible for benefits, not for outputs. The outputs have to create benefits or it doesnât count. Game overâ - Brian OâNeill (10:07)We talk about bridging the gap between the worlds of business and analytics... There's a huge gap between the perception of users and the tech leaders who are producing it." - Shashank Garg (17:37)âIT leaders often limit their roles to provisioning their secure data, and then they rely on businesses to be able to generate insights and take actions. Sometimes this handoff works, and sometimes it doesnât because of quality governance.â - Shashank Garg (23:02)âData is the kind of field where people can react very, very quickly to whatâs wrong.â - Shashank Garg (29:44)âItâs much easier to get to a good prototype if we know what the inputs to a prototype are, which include data about the people who are going to use the solution, their usage scenarios, use cases, attitudes, beliefsâŠall these kinds of things.â - Brian OâNeill (31:49)âFor data, you need a separate person, and then for designing, you need a separate person, and for analysis, you need a separate personâthe more you can combine, I donât think you can create super-humans who can do all three, four disciplines, but at least two disciplines and can appreciate the third one that makes it easier.â - Shashank Garg (39:20)âWhen we think of AI, weâre all talking about multiple different delivery methods here. I think AI is starting to become GenAI to a lot of non-data people. Itâs like theirâeverything is GenAI.â - Brian O'Neill (43:48)Links
Infocepts website: https://www.infocepts.ai/Shashank Garg on LinkedIn: https://www.linkedin.com/in/shashankgarg/ Top 5 Data & AI initiatives for business success: https://www.infocepts.ai/downloads/top-5-data-and-ai-initiatives-to-drive-business-growth-in-2024-beyond/ -
Welcome back! In today's solo episode, I share the top five struggles that enterprise SAAS leaders have in the analytics/insight/decision support space that most frequently leads them to think they have a UI/UX design problem that has to be addressed. A lot of today's episode will talk about "slow creep," unaddressed design problems that gradually build up over time and begin to impact both UX and your revenue negatively. I will also share 20 UI and UX design problems I often see (even if clients do not!) that, when left unaddressed, may create sales friction, adoption problems, churn, or unhappy end users. If you work at a software company or are directly monetizing an ML or analytical data product, this episode is for you!
Highlights/ Skip to
I discuss how specific UI/UX design problems can significantly impact business performance (02:51)I discuss five common reasons why enterprise software leaders typically reach out for help (04:39)The 20 common symptoms I've observed in client engagements that indicate the need for professional UI/UX intervention or training (13:22)The dangers of adding too many features or customization and how it can overwhelm users (16:00)The issues of integrating AI into user interfaces and UXs without proper design thinking (30:08)I encourage listeners to apply the insights shared to improve their data products (48:02)Quotes from Todayâs EpisodeâOne of the problems with bad design is that some of it we can see and some of it we can't â unless you know what you're looking for." - Brian OâNeill (02:23)âDesign is usually not top of mind for an enterprise software product, especially one in the machine learning and analytics space. However, if you have human users, even enterprise ones, their tolerance for bad software is much lower today than in the past.â Brian OâNeill - (13:04)âEarly on when you're trying to get product market fit, you can't be everything for everyone. You need to be an A+ experience for the person you're trying to satisfy.â -Brian OâNeill (15:39)âOften when I see customization, it is mostly used as a crutch for not making real product strategy and design decisions.â - Brian OâNeill (16:04) "Customization of data and dashboard products may be more of a tax than a benefit. In the marketing copy, customization sounds like a benefit...until you actually go in and try to do it. It puts the mental effort to design a good solution on the user." - Brian OâNeill (16:26)âWe need to think strategically when implementing Gen AI or just AI in general into the product UX because it wonât automatically help drive sales or increase business value.â - Brian OâNeill (20:50) âA lot of times our analytics and machine learning tools⊠are insight decision support products. They're supposed to be rooted in facts and data, but when it comes to designing these products, there's not a whole lot of data and facts that are actually informing the product design choices.â Brian OâNeill - (30:37)âIf your IP is that special, but also complex, it needs the proper UI/UX design treatment so that the value can be surfaced in such a way someone is willing to pay for it if not also find it indispensable and delightful.â - Brian OâNeill (45:02)LinksThe (5) big reasons AI/ML and analytics product leaders invest in UI/UX design help: https://designingforanalytics.com/resources/the-5-big-reasons-ai-ml-and-analytics-product-leaders-invest-in-ui-ux-design-help/ Subscribe for free insights on designing useful, high-value enterprise ML and analytical data products: https://designingforanalytics.com/list Access my free frameworks, guides, and additional reading for SAAS leaders on designing high-value ML and analytical data products: https://designingforanalytics.com/resources Need help getting your productâs design/UX on trackâso you can see more sales, less churn, and higher user adoption? Schedule a free 60-minute Discovery Call with me and Iâll give you my read on your situation and my recommendations to get ahead:https://designingforanalytics.com/services/ -
Welcome to a special edition of Experiencing Data. This episode is the audio capture from a live Crowdcast video webinar I gave on April 26th, 2024 where I conducted a mini UI/UX design audit of a new podcast analytics service that Chris Hill, CEO of Humblepod, is working on to help podcast hosts grow their show. Humblepod is also the team-behind-the-scenes of Experiencing Data, and Chris had asked me to take a look at his new âListener Lifecycleâ tool to see if we could find ways to improve the UX and visualizations in the tool, how we might productize this MVP in the future, and how improving the toolâs design might help Chris help his prospective podcast clients learn how their listener data could help them grow their listenership and âtrue fans.â On a personal note, it was fun to talk to Chris on the show given we speak every week: Humblepod has been my trusted resource for audio mixing, transcription, and show note summarizing for probably over 100 of the most recent episodes of Experiencing Data. It was also fun to do a âlive recordingâ with an audienceâand we did answer questions in the full video version. (If you missed the invite, join my Insights mailing list to get notified of future free webinars).
To watch the full audio and video recording on Crowdcast, free, head over to: https://www.crowdcast.io/c/podcast-analytics-ui-ux-design
Highlights/ Skip to:Chris talks about using data to improve podcasts and his approach to podcast numbers (03:06)Chris introduces the Listener Lifecycle model which informed the dashboard design (08:17)Chris and I discuss the importance of labeling and terminology in analytics UIs (11:00)We discuss designing for practical use of analytics dashboards to provide actionable insights (17:05)We discuss the challenges podcast hosts face in understanding and utilizing data effectively and how design might help (21:44)I discuss how my CED UX framework for advanced analytics applications helps to facilitate actionable insights (24:37)I highlight the importance of presenting data effectively and in a way that centers to user needs (28:50)I express challenges users may have with podcast rankings and the reliability of data sources (34:24) Chris and I discuss tailoring data reports to meet the specific needs of clients (37:14)Quotes from Todayâs EpisodeâThe irony for me as someone who has a podcast about machine learning and analytics and design is that I basically never look at my analytics.â - Brian OâNeill (01:14)âThe problem that I have found in podcasting is that the number that everybody uses to gauge whether a podcast is good or not is the download numberâŠBut thereâs a lot of other factors in a podcast that can tell you how successful itâs going to beâŠwhere you can pull levers toâŠgrow your show, or engage more with an audience.â - Chris Hill (03:20)âI have a framework for user experience design for analytics called CED, which stands for Conclusions, Evidence, Data⊠The basic idea is really simple: lead your analytic service with conclusions.â- Brian OâNeill (24:37)âWhere the eyes glaze over is when tools are mostly about evidence generators, and we just give everybody the evidence, but thereâs no actual analysis about how [this is] helping me improve my life or my business. Itâs just evidence. I need someone to put that together.â - Brian OâNeill (25:23)âSometimes the data doesnât provide enough of a conclusion about what to doâŠThis is where your opinion starts to matterâ - Brian OâNeill (26:07)âIt sounds like a benefit, but drilling down for most people into analytics stuff is usually a tax unless youâre an analyst.â - Brian OâNeill (27:39)âWhereâs the source of this data, and who decided what these numbers are? Because so much of this stuffâŠis not shared. As someone whoâs in this space, itâs not even that itâs confusing. Itâs more like, you got to distill this down for me.â - Brian OâNeill (34:57)âYour clients are probably going to glaze over at this level of data because itâs not helping them make any decision about what to change.â- Brian OâNeill (37:53)LinksWatch the original Crowdcast video recording of this episodeBrianâs CED UX Framework for Advanced Analytics SolutionsJoin Brianâs Insights mailing list -
In this week's episode of Experiencing Data, I'm joined by Duncan Milne, a Director, Data Investment & Product Management at the Royal Bank of Canada (RBC). Today, Duncan (who is also a member of the DPLC) gives a preview of his upcoming webinar on April 24, 2024 entitled, âIs that Data Product Worth Building? Estimating Economic ValueâŠBefore You Build It!â Duncan shares his experience of implementing a product mindset within RBC's Chief Data Office, and he explains some of the challenges, successes, and insights gained along the way. He emphasizes the critical role of understanding user needs and evaluating the economic impact of data productsâbefore they are built. Duncan was gracious to let us peek inside and see a transformation that is currently in progress and Iâm excited to check out his webinar this month!
Highlights/ Skip to:
I introduce Duncan Milne from RBC (00:00)Duncan outlines the Chief Data Office's function at RBC (01:01)We discuss data products and how they are used to improve business process (04:05)The genesis behind RBC's move towards a product-centric approach in handling data, highlighting initial challenges and strategies for fostering a product mindset (07:26)Duncan discusses developing a framework to guide the lifecycle of data products at RBC (09:29)Duncan addresses initial resistance and adaptation strategies for engaging teams in a new product-centric methodology (12:04)The scaling challenges of applying a product mindset across a large organization like RBC (22:02)Insights into the framework for evaluating and prioritizing data product ideas based on their desirability, usability, feasibility, and viability. (26:30)Measuring success and value in data product management (30:45)Duncan explores process mapping challenges in banking (34:13)Duncan shares creating specialized training for data product management at RBC (36:39)Duncan offers advice and closing thoughts on data product management (41:38)Quotes from Todayâs EpisodeâWe think about data products as anything that solves a problem using data... it's helping someone do something they already do or want to do faster and better using data." - Duncan Milne (04:29)âThe transition to data product management involves overcoming initial resistance by demonstrating the tangible value of this approach." - Duncan Milne (08:38)"You have to want to show up and do this kind of work [adopting a product mindset in data product management]âŠeven if you do a product the right way, it doesnât always work, right? The thing you make may not be desirable, it may not be as usable as it needs to be. It can be technically right and still fail. Itâs not a guarantee, itâs just a better way of working.â - Brian T. OâNeill (15:03)â[Product management]... it's like baking versus cooking. Baking is a science... cooking is much more flexible. Itâs about... did we produce a benefit for users? Did we produce an economic benefit? ...Itâs a multivariate problem... a lot of it is experimentation and figuring out what works." - Brian T. O'Neill (23:03)"The easy thing to measure [in product management] is did you follow the process or not? That is not the point of product management at all. It's about delivering benefits to the stakeholders and to the customer." - Brian O'Neill (25:16)âData product is not something that is set in stone... You can leverage learnings from a more traditional product approach, but donât be afraid to improvise." - Duncan Milne (41:38)âData products are fundamentally different from digital products, so even the traditional approach to product management in that space doesnât necessarily work within the data products construct.â - Duncan Milne (41:55)âThere is no textbook for data product management; the field is still being developedâŠdonât be afraid to create your own answer if what exists out there doesnât necessarily work within your context.â- Duncan Milne (42:17)LinksDuncanâs Linkedin: https://www.linkedin.com/in/duncanwmilne/?originalSubdomain=ca -
This week on Experiencing Data, I chat with a new kindred spirit! Recently, I connected with Thabata Romanowskiâbetter known as "T from Data Rocks NZ"âto discuss her experience applying UX design principles to modern analytical data products and dashboards. T walks us through her experience working as a data analyst in the mining sector, sharing the journey of how these experiences laid the foundation for her transition to data visualization. Now, she specializes in transforming complex, industry-specific data sets into intuitive, user-friendly visual representations, and addresses the challenges faced by the analytics teams she supports through her design business. T and I tackle common misconceptions about design in the analytics field, discuss how we communicate and educate non-designers on applying UX design principles to their dashboard and application design work, and address the problem with "pretty charts." We also explore some of the core ideas in T's Design Manifesto, including principles like being purposeful, context-sensitive, collaborative, and humanisticâall aimed at increasing user adoption and business value by improving UX.
Highlights/ Skip to:
I welcome T from Data Rocks NZ onto the show (00:00)T's transition from mining to leading an information design and data visualization consultancy. (01:43)T discusses the critical role of clear communication in data design solutions. (03:39)We address the misconceptions around the role of design in data analytics. (06:54) T explains the importance of journey mapping in understanding users' needs. (15:25)We discuss the challenges of accurately capturing end-user needs. (19:00) T and I discuss the importance of talking directly to end-users when developing data products. (25:56) T shares her 'I like, I wish, I wonder' method for eliciting genuine user feedback. (33:03)T discusses her Data Design Manifesto for creating purposeful, context-aware, collaborative, and human-centered design principles in data. (36:37)We wrap up the conversation and share ways to connect with T. (40:49)Quotes from Todayâs Episode"It's not so much that peopleâŠdon't know what design is, it's more that they understand it differently from what it can actually do..." - T from Data Rocks NZ (06:59)"I think [misconception about design in technology] is rooted mainly in the fact that data has been very tied to IT teams, to technology teams, and theyâre not always up to what design actually does.â - T from Data Rocks NZ (07:42) âIf you strip design of function, it becomes art. So, itâs not art⊠itâs about being functional and being useful in helping people.â - T from Data Rocks NZ (09:06)"Itâs not that people donât know, really, that the word design exists, or that design applies to analytics and whatnot; itâs more that they have this misunderstanding that itâs about making things look a certain way, when in fact... Itâs about function. Itâs about helping people do stuff better." - T from Data Rocks NZ (09:19)âJourney Mapping means that you have to talk to people... Data is an inherently human thing. It is something that we create ourselves. So, itâs biased from the start. You canât fully remove the human from the data" - T from Data Rocks NZ (15:36) âThe biggest part of your data product successâŠhappens outside of your technology and outside of your actual analysis. Itâs defining who your audience is, what the context of this audience is, and to which purpose do they need that product. - T from Data Rocks NZ (19:08)â[In UX research], a tight, empowered product team needs regular exposure to end customers; thereâs nothing that can replace that." - Brian O'Neill (25:58)âYou have two sides [end-users and data team] that are frustrated with the same thing. The side who asked wasnât really sure what to ask. And then the data team gets frustrated because the users donât know what they wantâŠNobody really understood what the problem is. Thereâs a lot of assumptions happening there. And this is one of the hardest things to let go.â - T from Data Rocks NZ (29:38)âNo piece of data product exists in isolation, so understanding what people do with it⊠is really important.â - T from Data Rocks NZ (38:51)LinksDesign Matters Newsletter: https://buttondown.email/datarocksnz Website: https://www.datarocks.co.nz/LinkedIn: https://www.linkedin.com/company/datarocksnz/BlueSky: https://bsky.app/profile/datarocksnz.bsky.socialMastodon: https://me.dm/@datarocksnz -
This week on Experiencing Data, something new as promised at the beginning of the year. Today, Iâm exploring the world of embedded analytics with Zalak Trivedi from Sigma Computingâand this is also the first approved Promoted Episode on the podcast. In todayâs episode, Zalak shares his journey as the product lead for Sigmaâs embedded analytics and reporting solution which seeks to accelerate and simplify the deployment of decision support dashboards to their SAAS companiesâ customers. Right there, we have the first challenge that Zalak was willing to dig into with me: designing a platform UX when we have multiple stakeholder and user types. In Sigmaâs case, this means Sigmaâs buyers, the developers that work at these SAAS companies to integrate Sigma into their products, and then the actual customers of these SAAS companies who will be the final end users of the resulting dashboards. also discuss the challenges of creating products that serve both beginners and experts and how AI is being used in the BI industry.
Highlights/ Skip to:
I introduce Zalak Trivedi from Sigma Computing onto the show (03:15)Zalak shares his journey leading the vision for embedded analytics at Sigma and explains what Sigma looks like when implemented into a customerâs SAAS product . (03:54)Zalak and I discuss the challenge of integrating Sigma's analytics into various companies' software, since they need to account for a variety of stakeholders. (09:53)We explore Sigma's team approach to user experience with product management, design, and technical writing (15:14)Zalak reveals how Sigma leverages telemetry to understand and improve user interactions with their products (19:54)Zalak outlines why Sigma is a faster and more supportive alternative to building your own analytics (27:21)We cover data monetization, specifically looking at how SAAS companies can monetize analytics and insights (32:05)Zalak highlights how Sigma is integratingAI into their BI solution (36:15)Zalak share his customers' current pain points and interests (40:25) We wrap up with final thoughts and ways to connect with Zalak and learn more about Sigma (49:41) Quotes from Todayâs Episode"Something Iâm really excited about personally that we are working on is [moving] beyond analytics to help customers build entire data applications within Sigma. This is something we are really excited about as a company, and marching towards [achieving] this year." - Zalak Trivedi (04:04)âThe whole point of an embedded analytics application is that it should look and feel exactly like the application itâs embedded in, and the workflow should be seamless.â - Zalak Trivedi (09:29) âWe [at Sigma] had to switch the way that we were thinking about personas. It was not just about the analysts or the data teams; it was more about how do we give the right tools to the [SAAS] product managers and developers to embed Sigma into their product.â - Zalak Trivedi (11:30) âYou canât not have a design, and you canât not have a user experience. Thereâs always an experience with every tool, solution, product that we use, whether it emerged organically as a byproduct, or it was intentionally created through knowledge data... it was intentionalâ - Brian OâNeill (14:52) âIf we find that [in] certain user experiences,people are tripping up, and theyâre not able to complete an entire workflow, we flag that, and then we work with the product managers, or [with] our customers essentially, and figure out how we can actually simplify these experiences.â - Zalak Trivedi (20:54)âWe were able to convince many small to medium businesses and startups to sign up with Sigma. The success they experienced after embedding Sigma was tremendous. Many of our customers managed to monetize their existing data within weeks, or at most, a couple of months, with lean development teams of two to three developers and a few business-side personnel, generating seven-figure income streams from that.â - Zalak Trivedi (32:05)âAt Sigma, our stance is, letâs not just add AI for the sake of adding AI. Letâs really identify [where] in the entire user journey does the intelligence really lie, and where are the different friction points, and letâs enhance those experiences.â - Zalak Trivedi (37:38) âEvery time [we at Sigma Computing] think about a new feature or functionality, we have to ensure it works for both the first-degree persona and the second-degree persona, and consider how it will be viewed by these different personas, because that is not the primary persona for which the foundation of the product was built." - Zalak Trivedi (48:08)LinksSigma Computing: https://sigmacomputing.com
Email: [email protected]
LinkedIn: https://www.linkedin.com/in/trivedizalak/
Sigma Computing Embedded: https://sigmacomputing.com/embedded
About Promoted Episodes on Experiencing Data: https://designingforanalytics.com/promoted
-
In this episode of Experiencing Data, I speak with Ellen Chisa, Partner at BoldStart Ventures, about what sheâs seeing in the venture capital space around AI-driven products and companiesâparticularly with all the new GenAI capabilities that have emerged in the last year. Ellen and I first met when we were both engaged in travel tech startups in Boston over a decade ago, so it was great to get her current perspective being on the âother sideâ of products and companies working as a VC. Ellen draws on her experience in product management and design to discuss how AI could democratize software creation and streamline backend coding, design integration, and analytics. We also delve into her work at Dark and the future prospects for developer tools and SaaS platforms. Given Ellenâs background in product management, human-centered design, and now VC, I thought she would have a lot to shareâand she did!
Highlights/ Skip to:I introduce the show and my guest, Ellen Chisa (00:00)Ellen discusses her transition from product and design to venture capital with BoldStart Ventures. (01:15)Ellen notes a shift from initial AI prototypes to more refined products, focusing on building and testing with minimal data. (03:22)Ellen mentions BoldStart Ventures' focus on early-stage companies providing developer and data tooling for businesses. (07:00)Ellen discusses what she learned from her time at Dark and Lola about narrowing target user groups for technology products (11:54)Ellen's Insights into the importance of user experience is in product design and the process venture capitalists endure to make sure it meets user needs (15:50)Ellen gives us her take on the impact of AI on creating new opportunities for data tools and engineering solutions, (20:00)Ellen and I explore the future of user interfaces, and how AI tools could enhance UI/UX for end users. (25:28)Closing remarks and the best way to find Ellen on online (32:07)Quotes from Todayâs EpisodeâIt's a really interesting time in the venture market because on top of the Gen AI wave, we obviously had the macroeconomic shift. And so we've seen a lot of people are saying the companies that come out now are going to be great companies because they're a little bit more capital-constrained from the beginning, typically, and they'll grow more thoughtfully and really be thinking about how do they build an efficient business.â- Ellen Chisa (03: 22) âWe have this big technological shift around AI-enabled companies, and I think one of the things Iâve seen is, if you think back to a year ago, we saw a lot of early prototyping, and so there were like a couple of use cases that came up again and again.â-Ellen Chisa (3:42)âI don't think I've heard many pitches from founders who consider themselves data scientists first. We definitely get some from ML engineers and people who think about data architecture, for sure..â- Ellen Chisa (05:06) âI still prefer GUI interfaces to voice or text usually, but I think that might be an uncanny valley sort of thing where if you think of people who didnât have technology growing up, theyâre more comfortable with the more human interaction, and then you get, like, a chunk of people who are digital natives who prefer it.â- Ellen Chisa (24:51)[Citing some excellent Boston-area restaurants!] âThe Arc browser just shipped a bunch of new functionality, where instead of opening a bunch of tabs, you can say, âOpen the recipe pages for Oleana and Sarma,â and it just opens both of them, and so itâs like multiple search queries at once.â - Ellen Chisa (27:22)âThe AI wave of technology biases towards people who already have products [in the market] and have existing datasets, and so I think everyone [at tech companies] is getting this big, top-down mandate from their executive team, like, âOh, hey, you have to do something with AI now.ââ- Ellen Chisa (28:37)âI think itâs hard to really grasp what an LLM is until you do a fair amount of experimentation on your own. The experience of asking ChatGPT a simple search question compared to the experience of trying to train it to do something specific for you are quite different experiences. Even beyond that, thereâs a tool called superwhisper that I like that you can take audio content and end up with transcripts, but you can give it prompts to change your transcripts as youâre going. So, you can record something, and it will give you a different output if you say youâre recording an email compared to [if] youâre recording a journal entry compared to [if] youâre recording the transcript for a podcast.â- Ellen Chisa (30:11)LinksBoldstart ventures: https://boldstart.vc/LinkedIn: https://www.linkedin.com/in/ellenchisa/Personal website: https://ellenchisa.comEmail: [email protected] -
This week, I'm chatting with Karen Meppen, a founding member of the Data Product Leadership Community and a Data Product Architect and Client Services Director at Hakkoda. Today, we're tackling the difficult topic of developing data products in situations where a product-oriented culture and data infrastructures may still be emerging or âat oddsâ with a human-centered approach. Karen brings extensive experience and a strong belief in how to effectively negotiate the early stages of data maturity. Together we look at the major hurdles that businesses encounter when trying to properly exploit data products, as well as the necessity of leadership support and strategy alignment in these initiatives. Karen's insights offer a roadmap for those seeking to adopt a product and UX-driven methodology when significant tech or cultural hurdles may exist.
Highlights/ Skip to:
I Introduce Karen Meppen and the challenges of dealing with data products in places where the data and tech aren't quite there yet (00:00)Karen shares her thoughts on what it's like working with "immature data" (02:27)Karen breaks down what a data product actually is (04:20)Karen and I discuss why having executive buy-in is crucial for moving forward with data products (07:48)The sometimes fuzzy definition of "data products." (12:09)Karen defines âshadow data teamsâ and explains how they sometimes conflict with tech teams (17:35)How Karen identifies the nature of each team to overcome common hurdles of connecting tech teams with business units (18:47)How she navigates conversations with tech leaders who think they already understand the requirements of business users (22:48)Using design prototypes and design reviews with different teams to make sure everyone is on the same page about UX (24:00)Karen shares stories from earlier in her career that led her to embrace human-centered design to ensure data products actually meet user needs (28:29)We reflect on our chat about UX, data products, and the âproductyâ approach to ML and analytics solutions (42:11) Quotes from Todayâs Episode"Itâs not really fair to get really excited about what we hear about or see on LinkedIn, at conferences, etc. We get excited about the shiny things, and then want to go straight to it when [our] organization [may not be ] ready to do that, for a lot of reasons." - Karen Meppen (03:00)"If you do not have support from leadership and this is not something [they are] passionate about, you probably arenât a great candidate for pursuing data products as a way of working." - Karen Meppen (08:30)"Requirements are just friendly lies." - Karen, quoting Brian about how data teams need to interpret stakeholder requests (13:27)"The greatest challenge that we have in technology is not technology, itâs the people, and understanding how weâre using the technology to meet our needs." - Karen Meppen (24:04)"You canât automate something that you havenât defined. For example, if you donât have clarity on your tagging approach for your PII, or just the nature of all the metadata that youâre capturing for your data assets and what it means or how itâs handledâto make it good, then how could you possibly automate any of this that hasnât been defined?" - Karen Meppen (38:35)"Nothing upsets an end-user more than lifting-and-shifting an existing report with the same problems it had in a new solution that now theyâve never used before." - Karen Meppen (40:13)âEarly maturity may look different in many ways depending upon the nature of business youâre doing, the structure of your data team, and how it interacts with folks.â (42:46) Links Data Product Leadership Community https://designingforanalytics.com/community/Karen Meppen on LinkedIn: ââhttps://www.linkedin.com/in/karen--m/HakkĆda, Karen's company, for more insights on data products and services:https://hakkoda.io/ -
This week Iâm chatting with Caroline Zimmerman, Director of Data Products and Strategy at Profusion. Caroline shares her journey through the school of hard knocks that led to her discovery that incorporating more extensive UX research into the data product design process improves outcomes. We explore the complicated nature of discovering and building a better design process, how to engage end users so they actually make time for research, and why understanding how to navigate interdepartmental politics is necessary in the world of data and product design. Caroline reveals the pivotal moment that changed her approach to data product design, as well as her learnings from evolving data products with the users as their needs and business strategies change. Lastly, Caroline and I explore what the future of data product leadership looks like and Caroline shares why there's never been a better time to work in data.
Highlights/ Skip to:
Intros and Caroline describes how she learned crucial lessons on building data products the hard way (00:36)The fundamental moment that helped Caroline to realize that she needed to find a different way to uncover user needs (03:51)How working with great UX researchers influenced Carolineâs approach to building data products (08:31)Why Caroline feels that exploring the âwhyâ is foundational to designing a data product that gets adopted (10:25)Carolineâs experience building a data model for a client and what she learned from that experience when the clientâs business model changed (14:34)How Caroline addresses the challenge of end users not making time for user research (18:00)A high-level overview of the UX research process when Carolineâs team starts working with a new client (22:28)The biggest challenges that Caroline faces as a Director of Data Products, and why data products require the ability to navigate company politics and interests (29:58)Caroline describes the nuances of working with different stakeholder personas (35:15)Why data teams need to embrace a more human-led approach to designing data products and focus less on metrics and the technical aspects (38:10)Carolineâs closing thoughts on what sheâd like to share with other data leaders and how you can connect with her (40:48)Quotes from Todayâs EpisodeâWhen I was first starting out, I thought that you could essentially take notes on what someone was asking for, go off and build it to their exact specs, and be successful. And it turns out that you can build something to exact specs and suffer from poor adoption and just not be solving problems because I did it as a wish fulfillment, laundry-list exercise rather than really thinking through user needs.â â Caroline Zimmerman (01:11)âPeople want a thing. Theyâre paying for a thing, right? And so, just really having that reflex to try to gently come back to that why and spending sufficient time exploring it before going into solution build, even when people are under a lot of deadline pressure and are paying you to deliver a thing [is the most important element of designing a data product].â â Caroline Zimmerman (11:53)âA data product evolves because user needs change, business models change, and business priorities change, and we need to evolve with it. Itâs not like you got it right once, and then youâre good for life. At all.â â Caroline Zimmerman (17:48)âI continue to have lots to learn about stakeholder management and understanding the interplay between what the organization needs to be successful, but also, organizations are made up of people with personal interests, and you need to understand both.â â Caroline Zimmerman (30:18)âData products are built in a political context. And just being aware of that context is important.â â Caroline Zimmerman (32:33)âI think that data, maybe more than any other function, is transversal. I think data brings up politics because, especially with larger organizations, there are those departmental and team silos. And the whole thing about data is it cuts through those because it touches all the different teams. It touches all the different processes. And so in order to build great data products, you have to be navigating that political context to understand how to get things done transversely in organizations where most stuff gets done vertically.â â Caroline Zimmerman (34:37)âData leadership positions are data product expertise roles. And I think that often itâs been more technical people that have advanced into those roles. If you follow the LinkedIn-verse in data, itâs very much on every data leaderâs mind at the moment: how do you articulate benefits to your CEO and your board and try to do that before itâs too late? So, Iâd say thatâs really the main thing and that thereâs just never been a better time to be a data product person.â â Caroline Zimmerman (37:16)LinksProfusion: https://profusion.com/Caroline Zimmerman LinkedIn: https://www.linkedin.com/in/caroline-zimmerman-4a531640/Nick Zervoudis LinkedIn: https://www.linkedin.com/in/nzervoudis/Email: mailto:[email protected] -
This week, Iâm chatting with Steve Portigal, who is the Principal of Portigal Consulting and the Author of Interviewing Users. We discuss the changes that prompted him to release a second version of his book 10 years after its initial release, and dive into the best practices that any team can implement to start unlocking the value of data product UX research. Steve explains that the key to making time for user research is knowing what business value youâre after, not simply having a list of research questions. We then role-play through some in-depth examples of real-life experiences weâve seen from both end users and leadership when it comes to implementing a user research strategy. Thhroughout our conversation, we come back to the idea that even taking imperfect action towards doing user research can lead to increased data product adoption and business value.
Highlights/ Skip to:
I introduce Steve Portigal, Principal of Portigal Consulting and Author of Interviewing Users (00:38)What changes caused Steve to release a second edition of his book (00:58)Steve and I discuss the importance of understanding how to conduct effective user research (03:44)Steve explains why itâs crucial to understand that the business challenge and the research questions are two different things (08:16)Brian and Steve role-play a common scenario that comes up in user research, and Steve explains an optimal workflow for user research (11:50)The importance of provocation in performing user research (21:02)How Steve would handle a situation where a member of leadership is preventing research being done with end users (24:23)Why a consultative approach is valuable when getting buy-in for conducting user research (35:04)Steve shares some of the major benefits of taking imperfect action towards starting user research (36:59)The impact and value even easy wins in user research can have (42:54)Steve describes the exploratory nature of user research and how to maximize the chance of finding the most valuable insights (46:57)Where you can connect with Steve and get a copy of v2 of his book, Interviewing Users (49:35)Quotes from Todayâs EpisodeâIf you donât know what youâre doing, and you donât know what you should be investing effort-wise, thatâs the inexperience in the approach. If you donât know how to plan, what should we be trying to solve in this research? What are we trying to learn? What are we going to do with it in the organization? Who should we be talking to? How do we find them? What do we ask them? And then a really good one: how do we make sense of that information so that it has impact that we can take away?â â Steve Portigal (07:15)âWhat do people get [from user research]? I think the chance for a team to align around something that comes in from the outside.â â Steve Portigal (41:36)On the impact user research can have if teams embrace it: âThey had a product that did a thing that no one [understood], and they had to change the product, but also change how they talked about it, change how they built it, and change how they packaged it. And that was a really dramatic turnaround. And it came out of our research, but [mostly] because they really leaned into making use of this stuff.â â Steve Portigal (42:35)"If we knew all the questions to ask, we would just write a survey, right? Itâs a lower time commitment from the participant to do that. But weâre trying to get at what we donât know that we donât know. For some of us, thatâs fun!" â Steve Portigal (48:36)LinksInterviewing Users (use code DATA20 to get 20% off the list price): https://rosenfeldmedia.com/books/interviewing-users-second-edition/Personal website: https://portigal.comPublisher website: https://rosenfeldmedia.comLinkedIn: https://www.linkedin.com/in/steveportigal/ -
In this episode, Iâm chatting with former Gartner analyst Sanjeev Mohan who is the Co-Author of Data Products for Dummies. Throughout our conversation, Sanjeev shares his expertise on the evolution of data products, and what heâs seen as a result of implementing practices that prioritize solving for use cases and business value. Sanjeev also shares a new approach of structuring organizations to best implement ownership and accountability of data product outcomes. Sanjeev and I also explore the common challenges of product adoption and who is responsible for user experience. I purposefully had Sanjeev on the show because I think we have pretty different perspectives from which we see the data product space.
Highlights/ Skip to:
I introduce Sanjeev Mohan, co-author of Data Products for Dummies (00:39)Sanjeev expands more on the concept of writing a âfor Dummiesâ book (00:53)Sanjeev shares his definition of a data product, including both a technical and a business definition (01:59)Why Sanjeev believes organizational changes and accountability are the keys to preventing the acceleration of shipping data products with little to no tangible value (05:45)How Sanjeev recommends getting buy-in for data product ownership from other departments in an organization (11:05)Sanjeev and I explore adoption challenges and the topic of user experience (13:23)Sanjeev explains what role is responsible for user experience and design (19:03)Who should be responsible for defining the metrics that determine business value (28:58)Sanjeev shares some case studies of companies who have adopted this approach to data products and their outcomes (30:29)Where companies are finding data product managers currently (34:19)Sanjeev expands on his perspective regarding the importance of prioritizing business value and use cases (40:52)Where listeners can get Data Products for Dummies, and learn more about Sanjeevâs work (44:33)Quotes from Todayâs EpisodeâYou may slap a label of data product on existing artifact; it does not make it a data product because thereâs no sense of accountability. In a data product, because they are following product management best practices, there must be a data product owner or a data product manager. Thereâs a single person [responsible for the result]. â Sanjeev Mohan (09:31)âI havenât even mentioned the word data mesh because data mesh and data products, they donât always have to go hand-in-hand. I can build data products, but I donât need to go into theâdo all of data mesh principles.â â Sanjeev Mohan (26:45)âWe need to have the right organization, we need to have a set of processes, and then we need a simplified technology which is standardized across different teams. So, this way, we have the benefit of reusing the same technology. Maybe it is Snowflake for storage, DBT for modeling, and so on. And the idea is that different teams should have the ability to bring their own analytical engine.â â Sanjeev Mohan (27:58)âGenerative AI, right now as we are recording, is still in a prototyping phase. Maybe in 2024, itâll go heavy-duty production. We are not in prototyping phase for data products for a lot of companies. Theyâve already been experimenting for a year or two, and now theyâre actually using them in production. So, weâve crossed that tipping point for data products.â â Sanjeev Mohan (33:15)âLow adoption is a problem thatâs not just limited to data products. How long have we had data catalogs, but they have low adoption. So, itâs a common problem.â â Sanjeev Mohan (39:10)âThat emphasis on technology first is a wrong approach. I tell people that Iâm sorry to burst your bubble, but there are no technology projects, there are only business projects. Technology is an enabler. You donât do technology for the sake of technology; you have to serve a business cause, so letâs start with that and keep that front and center.â â Sanjeev Mohan (43:03)LinksData Products for Dummies: https://www.dataops.live/dataproductsfordummiesâWhat Exactly is A Data Productâ article: https://medium.com/data-mesh-learning/what-exactly-is-a-data-product-7f6935a17912It Depends: https://www.youtube.com/@SanjeevMohanChief Data Analytics and Product Officer of Equifax: https://www.youtube.com/watch?v=kFY7WGc-jFMSanjMo Consulting: https://www.sanjmo.com/dataops.live: https://dataops.livedataops.live/dataproductsfordummies: https://dataops.live/dataproductsfordummiesLinkedIn: https://www.linkedin.com/in/sanjmo/Medium articles: https://sanjmo.medium.com -
Today I am sharing some highlights for 2023 from the podcast, and also letting you all know Iâll be taking a break from the podcast for the rest of December, but Iâll be back with a new episode on January 9th, 2024. Iâve also got two links to share with youâdetails inside!
TranscriptGreetings everyone - Iâm taking a little break from Experiencing Data over December of 2023, but Iâll be back in January with more interviews and insights on leveraging UX design and product management to create indispensable data products, machine learning apps, and decision support tools.
Experiencing Data turned this year five years old back in November, with over 130 episodes to date! I still canât believe itâs been going that long and how far weâve come.
Some highlights for me in 2023 included launching the Data Product Leadership Community, finding out that the show is now in the top 2% of all podcasts worldwide according to ListenNotes, and most of all, hearing from you that the podcast, and my writing, and the guests that I have brought on are having an impact on your work, your careers, and hopefully the lives of your customers, users, and stakeholders as well!
So, for now, Iâve got just two links for you:
If youâre wondering how to either:
support the show yourself with a really fast review on Apple Podcasts,to record a quick audio question for me to answer on the show, or if you want to join my free Insights mailing lists where I share my bi-weekly ideas and thoughts and 1-page episode summaries of all the show drops that I put out here on Experiencing Data.âŠjust head over to designingforanalytics.com/podcast and youâll get links to all those things there.
And secondly, if you need help increasing customer adoption, delight, the business value, or the usability of your analytics and machine learning applications in 2024, I invite you to set up a free discovery call with me 1 on 1.
You bring the questions, Iâll bring my ears, and by the end of the call, Iâll give you my best advice on how to move forward with your situation â whether itâs working with me or not. To schedule one of those free discovery calls, visit designingforanalytics.com/go
And finally, there will be some news coming out next year with the show, as well as my business, so I hope youâll hop on the mailing list and stay tuned, thatâs probably the best place to do that. And if you celebrate holidays in December and January, I hope theyâre safe, enjoyable, and rejuvenating. Until 2024, stay tuned right here - and in the words of the great Arnold Schwarzenegger, Iâll be back.
-
In this conversation with Klara Lindner, Service Designer at diconium data, we explore how behavioral science and UX can be used to increase adoption of data products. Klara describes how she went from having a highly technical career as an electrical engineer and being the founder of a solar startup to her current role in service design for data products. Klara shares powerful insights into the value of user research and human-centered design, including one which stopped me in my tracks during this episode: how the people making data products and evangelizing data-driven decision making arenât actually following their own advice when it comes to designing their data products. Klara and I also explore some easy user research techniques that data professionals can use, and discuss who should ultimately be responsible for user adoption of data products. Lastly, Klara gives us a peek at her upcoming December 19th, 2023 webinar with the The Data Product Leadership Community (DPLC) where she will be going deeper on two frameworks from psychology and behavioral science that teams can use to increase adoption of data products. Klara is also a founding member of the DPLC and was one ofâif not the very firstâdesign/UX professionals to join.
Highlights/ Skip to:
I introduce Klara, and she explains the role of Service Design to our audience (00:49)Klara explains how she realized sheâs been doing design work longer than she thought by reflecting on the company she founded, Mobisol (02:09)How Klara balances the desire to design great dashboards with the mission of helping end users (06:15)Klara describes the psychology behind user research and her upcoming talk on December 19th at The Data Product Leadership Community (08:32)What data product teams can do as a starting point to begin implementing user research principles (10:52) Klara gives a powerful example of the type of insight and value even basic user research can provide (12:49)Klara and I discuss a key revelation when it comes to designing data products for users, which is the irony that even developers use intuition as well as quantitative data when building (16:43)What adjustments Klara had to make in her thinking when moving from a highly technical background to doing human-centered design (21:08)Klara describes the two frameworks for driving adoption that sheâll be sharing in her talk at the DPLC on December 19th (24:23)An example of how understanding and addressing adoption blockers is important for product and design teams (30:44)How Klara has seen her teams adopt a new way of thinking about product & service design (32:55)Klara gives her take on the Jobs to be Done framework, which she will also be sharing in her talk at the DPLC on December 19th (35:26)Klaraâs advice to teams that are looking to build products around generative AI (39:28)Where listeners can connect with Klara to learn more (41:37)Linksdiconium data: http://www.diconium.com/LinkedIn: https://www.linkedin.com/in/klaralindner/Personal Website: https://magic-investigations.com/Hear Klara speak on Dec 19, 2023 at 10am ET here: https://designingforanalytics.com/community/ -
This week Iâm covering Part 1 of the 15 Ways to Increase User Adoption of Data Products, which is based on an article I wrote for subscribers of my mailing list. Throughout this episode, I describe why focusing on empathy, outcomes, and user experience leads to not only better data products, but also better business outcomes. The focus of this episode is to show you that itâs completely possible to take a human-centered approach to data product development without mandating behavioral changes, and to show how this approach benefits not just end users, but also the businesses and employees creating these data products.
Highlights/ Skip to:
Design behavior change into the data product. (05:34)Establish a weekly habit of exposing technical and non-technical members of the data team directly to end users of solutions - no gatekeepers allowed. (08:12)Change funding models to fund problems, not specific solutions, so that your data product teams are invested in solving real problems. (13:30)Hold teams accountable for writing down and agreeing to the intended benefits and outcomes for both users and business stakeholders. Reject projects that have vague outcomes defined. (16:49)Approach the creation of data products as âuser experiencesâ instead of a âthingâ that is being built that has different quality attributes. (20:16)If the team is tasked with being âinnovative,â leaders need to understand the innoficiency problem, shortened iterations, and the importance of generating a volume of ideas (bad and good) before committing to a final direction. (23:08)Co-design solutions with [not for!] end users in low, throw-away fidelity, refining success criteria for usability and utility as the solution evolves. Embrace the idea that research/design/build/test is not a linear process. (28:13)Test (validate) solutions with users early, before committing to releasing them, but with a pre-commitment to react to the insights you get back from the test. (31:50)Links:
15 Ways to Increase Adoption of Data Products: https://designingforanalytics.com/resources/15-ways-to-increase-adoption-of-data-products-using-techniques-from-ux-design-product-management-and-beyond/Company website: https://designingforanalytics.comEpisode 54: https://designingforanalytics.com/resources/episodes/054-jared-spool-on-designing-innovative-ml-ai-and-analytics-user-experiences/Episode 106: https://designingforanalytics.com/resources/episodes/106-ideaflow-applying-the-practice-of-design-and-innovation-to-internal-data-products-w-jeremy-utley/Ideaflow: https://www.amazon.com/Ideaflow-Only-Business-Metric-Matters/dp/0593420586/Podcast website: https://designingforanalytics.com/podcast - Mehr anzeigen
