Welcome to episode 24 of EvidenceTALKS! We've introduced a new structure this month, where we divide the podcast into different sections including:Think Like a Scientist - we discuss different aspects of evidence-based practice as well looking at how data is used and misused! Evidence summaries - with our friends at ScienceForWork we provide trustworthy and useful insights from the science of organisations and people management.
In this episode we talk about how data is used to support big claims and the importance of following the evidence trail.
We also talk to Lorenzo Galli, found of ScienceForWork who shares what the evidence really says about millennials.
Pilar Orti shares her reflections on two examples from the healthcare sector illustrating how data can be misleading.
Download the evidence summary 'Generational Differences: Myth or reality?' https://scienceforwork.com/blog/generational-differences/
Health Check podcast:
https://www.bbc.co.uk/programmes/w3cswjk9 - Myth about the MMR/Autism link
https://www.bbc.co.uk/programmes/w3cswjkb - talking about correlation Vs. causation in relation to behaviour change.
In this episode we are delighted to interview Rob Briner, Professor of Organisational Psychology at Queen Mary, University of London, Founder and Scientific Director of Centre for Evidence-based Management (CEBMa). We talk about his journey into evidence-based management, explore what evidence-based management is and how to overcome some of the barriers to putting it into practice.
Pilar and Claire also talk about a introducing an exciting new format for EvidenceTALKS for 2018! We'd love to hear you feedback, ideas or if you would like to take part in a future episode and you can get in touch on email at email@example.com or @FW_Centre.
Claudia Nuttgens is a Chartered Occupational Psychologist and Principal Consultant at the Future Work Centre. In this episode, we interview Claudia about the practitioner case study she presented at the annual Division of Occupational Psychology Conference in January.
The case study focused on a recent project with an NHS Trust where we helped the organisation take meaningful action from the results of their employee survey. This involved in-depth analysis, communication of results and close engagement with senior stakeholders. Claudia shares some of the lessons learnt which have broader application for those using employee research, including stakeholder management, problems versus polarities and organisational awareness.
Claudia also took part in a panel discussion exploring the challenges of embedding evidence-based practice in an organisation and the value we bring as occupational psychologists.
(Apologies for the rustling noise during the podcast!)
In this episode we're delighted to welcome Tiffany Poeppelman, Head of Sales Productivity at LinkedIn. Tiffany is an Industrial Psychologist, passionate about learning and development and applying research into the workplace.
In this interview, Tiffany shares her views on the importance of being evidence-based and what it can bring to the workplace. She discusses some of the practical challenges she has experienced as a practitioner including staying up-to-date with the latest research, the importance of communication and working closely with the business.
Tiffany sees herself as a bridge between science and practice, which is why we were so excited that she agreed to take part in our podcast.
It's our 20th episode! Richard and Pilar take a few minutes to look back on their experience of hosting the podcast over the last 20 episodes. The podcast is an important channel for the Future Work Centre to promote and communicate the value of evidence-based practice, making it actionable and accessible.
When we're not recording podcasts, the Future Work Centre works directly with organisations on a commercial basis. This revenue is used to help us fund our wider programme of activity to promote and advocate evidence-practice to organisations, practitioners and the wider public. Our client work keeps us on our toes and helps us learn the best ways to engage and support organisations to be more evidence-based, whilst being sensitive and pragmatic to their specific working environment.
Richard shared three anonymised client project examples:Understanding root cause of sickness and absence - we helped one organisation to scratch the surface and better understand what was driving employee absence and worked with them to identify small, but impactful ways to address this issue. Analysis of staff survey data - this is a topic we've talked about many times on the podcast! We've helped a number of organisations to go beyond descriptive analysis to understand what their data was saying, where to target action most effectively and engage with senior stakeholders in feedback and action planning. As occupational psychologists we are well placed to not only understand data, but can use our knowledge of the workplace to put this into context. Evaluation of a leadership development programme - this is an ongoing project for the Centre and our role is not the design and delivery of the programme, it's to act as an independent evaluator. We will go 'beyond the happy sheet' to understand what's working, what's having the biggest impact and how. We've also looked at what the scientific evidence says about the effectiveness of leadership development interventions in their sector, so that we help the client further refine the programme in the future.
These examples illustrate some of the real challenges of being evidence-based and the importance of being pragmatic!
In this episode we discuss the gig economy, prompted by a recent event Richard chaired: 'Managing the Gig Economy 2017'. We explore the themes covered in the event, why it's receiving lots of attention in the press and examine the implications for organisations.
Richard and Claire also reveal why episode 19 has taken to so long to produce!! Issues with sequencing, technology and a lost voice!
In this episode we explore and ask questions about the use of technology in the workplace, specifically wearables and 'speakables'. We're increasingly used to using devices in our personal lives e.g. fitness trackers and virtual assistants (Alexa, Siri, Cortana), and we anticipate seeing a rise in their use in the workplace.
Technology such as this seems to be advancing at an ever increasing pace, but our understanding of the benefits and possible drawbacks is less clear. We think it is important to ask questions before adopting new technology, so before we need to think carefully before introducing new technologies.
Recommended resources referenced in the podcast: http://www.bakadesuyo.com/
Barking up the Wrong Tree, Eric Barker
We are delighted to welcome back Paul Thoresen on this episode of EvidenceTALKS. We first interviewed Paul on episode 9 about employee surveys and the concept of engagement and it proved to be a very popular episode indeed! So, we thought we would invite him back to discuss in more details some of the practical aspects of conducting surveys.
Paul and Richard start with the fundamental question 'Do you really need to run a survey?'. Paul argues that any survey should be part of a wider strategy, rather than a tactical exercise. He recognises the challenge of moving away from an existing provider where you are familiar with the content, the process, the supplier and the difficulties of benchmarking old and new data, but if you need to do something differently, then it is worth reviewing the whole process.
Another area discussed is being clear what you're going to do with the survey data. If you're asking employees to invest their time, energy and effort into taking part, what happens afterwards is extremely important. This leads on to a common challenge around budget and resources, where organisations invest their money in the administration of the survey, but don't think through the whole process such as feedback, action planning or communications. Survey is a process, not an event, therefore allocating resources and budget in a more balanced way is key.
Paul and Richard go on to discuss what makes a good survey, exploring issues such as the variety of concepts they can measure (engagement, satisfaction, commitment), being clear about strategic objectives, the technical properties of the survey, transparency of vendors and quality of how the survey is implemented (pre & post survey).
If you missed Paul's first interview you can listen here: http://directory.libsyn.com/episode/index/show/evidencetalks/id/5239508
If you have questions about your own organisational survey, please do get in touch: firstname.lastname@example.org
Recommended books and resources:
"Getting Action from Organizational Surveys: New Concepts, Technologies, and Applications':
'Designing and Using Organizational Surveys: A Seven-Step Process': edited by Church, Waclawski , and Kraut.
"Organization Development: A data Driven Approach to Organisational Change'.
"Employee Engagement Survey Readiness Checklist" https://www.linkedin.com/pulse/20140322174642-29417413-employee-engagement-survey-readiness-checklist by
Employee Engagement Survey Readiness Checklist | LinkedIn
Before making decisions, it is useful to have a process to see if you are prepared to take the next step. Are you ready to buy a house? Are you ready to ...
How many times have you heard someone kick off a presentation or a conversation with the phrase 'As we all know...', closely followed by a strong assertion or claim?
In this episode Richard and Pilar meet face-to-face to talk (so forgive the background noise!) about how disruptive and dangerous myths and fads can be in the workplace, and whilst there may be a nugget of truth in there somewhere, it's been over-played or over-simplified.
We discuss a four common workplace myths:Millennials - we've seen an explosion of focus on this population group, highlighting how we need to do things differently because they need and want different things. This is often seen as the rationale to do things differently whether that's your recruitment strategy, succession planning or provision of development. But what does the evidence say? Engagement - As we all know...increasing employment leads to increased productivity. Quite apart from the challenges associated with its definition, this is often used as the rational for focusing on engagement i.e. that in doing so, there's a certainty that increased productivity will follow. Is that really true and is that the direction of causality? Flexible working - As we all know everyone wants to work flexibly! Flexible working means different things to different people - it's not always about working from home. We need to understand the context, the requirements of the role and your personal circumstances. It can be misleading to refer to it as a singular phenomenon and that it's wholly positive. Rise of the machines - As we all know robots are going to take our jobs! A huge topic, which is often associated with the fear factor, but automation offers lots of opportunities to do things better. But futurology can make big predictions about what we should or shouldn't be doing. As psychologists we have an important role in understanding the complexities between technology, people and organisations.
So, the next time you hear 'As we all know...' take the opportunity to challenge and ask why!
Gallup research, flexible working: https://trello-attachments.s3.amazonaws.com/57fbb722f5bdb288e7a854da/58fe36562ad84483cb4493bf/b9a304cc6c650ea2f567350d0e29a9de/State_Of_Telecommuting_U.S._Employee_Workforce.pdf
As part of the Beyond Measure initiative where we're looking to help organisations and candidates improve their understanding of workplace assessment, with the aim of improving practice and quality of its use. In this episode we look at assessment through the eyes of the person taking the test, why their experience is important and what they need to know.
If you're interested in our Beyond Measure campaign, have a listen to episode 8, where we introduce the initiative, and re-visit in subsequent episodes.
Richard and Pilar put themselves in the shoes of someone applying for a job and walk through a 'typical' process from the assessment invitation through to the test itself, feedback and how organisations use assessments to make decisions.
Given that many assessments are completed online, remotely, we talk about the importance of giving focused attention to completing the questionnaire. As a candidate, be free from distraction and give it your full attention, rather than trying to complete it on the run, in a noisy environment.
Feedback is also an important part of the process, and not always provided by organisations. Pilar shares her experience of a hiring process where she was provided with constructive feedback which she really valued. In contrast another organisation failed to get in touch after a month, leaving a lasting, poor impression.
We also discuss the danger of creating an overly idealised view of the job and the organisation during recruitment, where the reality bears little resemblance to the recruitment pitch. This disconnect can leave candidates with a false impression of what life is actually like and increase their intention to leave.
Throughout the whole process there's a balance to be struck between being fair and objective and being human.
Pilar recently attended the Work 2.0 conference where she listened to a presentation by Vodafone during which they shared their use of assessment games during the assessment process. The games are used to help applicants feel at ease but also because they collect vast amounts of data to measure cognitive ability and decision-making style. Technology is playing an ever-increasing role in the measurement of people at work, but also enriching the experience. However, it's still vital to be clear about what you're measuring, why you're using it and ensuring the process is fair and robust.
We are very grateful to Alan Redman, Chartered Occupational Psychologist and Director Criterion Partnership, for taking the time to share his perspective on workplace assessment, as part of our Beyond Measure initiative.
Alan is a highly experienced developer of workplace assessment and over the last 15-20 years, has witnessed the impact that technology has had on the way assessment is designed and implemented. He is also an independent reviewer of assessment on behalf of the British Psychological Society so is extremely well-placed to share his perspective on this topic. Given the explosion in technology, he often finds himself helping organisations make tricky choices about assessments, differentiating those which are underpinned by science and those which look slick, but have little depth or utility.
Richard and Alan talk about the advantages of technology in terms of making it easier to standardise an assessment process, easier to administrate for organisations and easier for candidates to take an assessment. However, there are concerns that sometimes, what gets lost along the way is the science and evidence underpinning the assessment.
Alan shares his practical advice and tips for spotting the symptoms of 'snake oil assessment' (referencing the work of Rob Briner at the Centre for Evidence-based Management). Beware of marketing speak, gurus, over-simplistic solutions and the 'next big thing'!
Useful reference for an evidence-based enquiry process: https://www.cebma.org
So what does it mean to make an evidence-based approach to using assessment? Given the changes in the market, Alan believes it's time to move our critical judgement on from the traditional quality markers - for example 'validity' has long been seen as a property of a test, but the lens we need to view it as a property of the way the test is used. Important to note that it's not enough to see technical data in a test publisher's technical manual, you need to understand whether it works in your organisation.
We particularly liked Alan's cautionary comment: 'What gets lost in technology is the science and the evidence.'
Other useful links:
British Psychological Society - https://www.bps.org.uk
British Psychological Society Psychological Testing Centre - https://ptc.bps.org.uk
Psychometrics Forum - http://www.psychometricsforum.org
In this episode we're continuing our Beyond Measure campaign and talking about whether the automation of workplace assessment has resulted in the de-skilling of practitioners.
This week Richard and Pilar are joined by Claire Rahmatallah, Managing Director, Future Work Centre. The topic of discussion is how technology has played a huge part in the way assessment is designed, implemented and interpreted. Much of the process is now fully automated which brings lots of advantages including process improvement, reduction in administration time, enhanced candidate experience and innovation. However, it does prompt some challenge and debate about the quality of assessment and how it is being impacted by technology.
Richard also shares his reflections after presenting at the CIPD Future of Work Conference in London where he got the opportunity to see different perspectives on the workplace of the future. He shares some research from Edelman showing low levels of trust in organisations, but also that 60% of people felt ill equipped for work in the future, and 54% feared their jobs were at risk because of automation and robots.
Claire introduces a new FWC free resource called EvidenceBITES - a series of webinars which will focus on a range of topical HR issues from an evidence-based perspective - http://www.futureworkcentre.com/evidencebites/
In this episode Richard shares his reflections on a recent conference, organised by the Division of Occupational Psychology in Scotland, where they brought psychologists together from a range of disciplines to discuss the core question 'What counts as evidence?'.
As we all know...
The episode kicks off with a teaser for a future podcast titled 'As we all know...'. Have you ever attended an event or conference where someone says 'As we all know...' followed by a big claim, often with no evidence to back it up? We've come across this a lot and think it's time to ask some more challenging questions! Here are just a few:
As we all know...everybody likes to work remotely increasing employee engagement leads to increased productivity learning styles dictate the success of training
Whilst there may be some useful ideas, these claims do not invite critical appraisal or a deeper exploration of where the assertion came from. So, we'd love to hear from our listeners about similar claims they've come across, and we'll take a critical look during a future podcast episode.
You can get in touch at email@example.com or via Twitter @FW_Centre.
We're also turning an evidence-based spotlight on workplace assessment, so if you have any questions about assessing people in the workplace, how it works, how to choose the right instrument, is it fair, does it work, we'd be delighted to hear from you and we'll do our best to answer your questions.
You can get in touch at firstname.lastname@example.org or via Twitter @FW_Centre.
What counts as evidence?
Richard was recently invited to speak at a conference organised by the Division of Occupational Psychology in Scotland. Also at the conference we psychologists from a range of disciplines including forensics, educational, clinical and sports psychology, all of whom were there to discuss the question 'What counts as evidence?'.
During the conference Richard found that many of the challenges experienced by occupational psychology (prevalence of myths ands fads) also took place in other fields.
There was lots to learn from the other psychologists such as methodology and how to evaluate the effectiveness of an intervention. He shares one example from a neuro-pscyhologist who evaluated an intervention with a sample size of just one.
Another example came from an educational psychologist who presented their work not the topic of reducing alcohol abuse in school-age children. They were able to demonstrate the impact of their intervention which led to securing additional funding to roll out to other schools. Similar to occupational psychologists, who work in a live, complex environment, they conducted their evaluation in a school and were able to point to the positive outcomes of their work.
A Forensic psychologist from the National Offender Management Service shared their work in the treatment of sex offenders. In high stakes environments, such as this, it's more important than ever to be able to defend decisions and understand the effectiveness of interventions.
Overall it's clear that there is a common ethos among psychologists to understand what works in order to make an improvements, it was also highlighted that it's equally if not more important to understanding what doesn't work or is potentially harmful. But what is clear is the notion of 'one-size-fits-all' rarely applies, whatever the field of psychology.
In this episode we discuss two things:Explore the use, limitations and challenges associated with the assessment of personality at work. Respond to a question from a listener about how to help managers better understand and take action from employee survey data.
As part of our 'Beyond Measure' initiative we will be sharing information with listeners about the world of workplace assessment. Workplace assessment touches a lot of people, whether you've been asked to complete a test or questionnaire, or whether you're responsible for using assessments as part of a talent programme. We want to help demystify this subject, busting some myths along the way and hopefully bring some clarity on how it can be used most effectively.
Before diving into this topic, Richard and Pilar respond to a question raised by a listener, following our interview with Paul Thoresen (episode 9) about the challenges of using survey data and how to get managers to understand the data and take action.
We then return to the topic of personality assessment - what is it, and why and how organisations use it? Richard provides deeper insight about these assessments work and how they differ from an opinion survey. Personality assessments seek to identify strength of preference on a continuum, rather than a binary category.
Personality assessments are commonplace in organisations, but it's important to first understand why you're using it. Given the logistics and cost associated with its use, being clear about the problem you're trying to solve is an important first step.
One of the misconceptions about personality assessment, commonly held by non-psychologists is to over-estimate the power of these assessments and that they can perfectly predict how someone will behave in an organisation. They are not perfect, but used appropriately can be a valuable input to help make a decision about and individual, whether that's in a hiring context or for development purposes.
In talking about the importance of feedback, Richard points out that personality assessment is 'not a diagnosis', it's a starting point for your feedback or development discussion. Therefore, being clear with a candidate about why you're asking them to complete the questionnaire and how the information is going to be used is really important.
We'd love to hear from you about your experiences of using personality assessment, whether you're a candidate or responsible for its use - get in touch on Twitter @FW_Centre or email email@example.com
Some useful resources:
BPS Psychological Testing Centre - https://ptc.bps.org.uk
Psychological testing - a test taker's guide - https://ptc.bps.org.uk/sites/ptc.bps.org.uk/files/Documents/Guidelines%20and%20Information/A%20Test%20Takers%20Guide_0.pdf
International Test Commission test taker's guide - https://www.intestcom.org/page/21
In this episode we interview Eric Barends, MD and founder of the Centre for Evidence-based Management (CEBMa) and explore his journey into evidence-based management and the evolution of the organisation.
CEBMa (www.cebma.org) started as a network of like-minded people from all over the world who shared a passion and interest in the evidence-based practice in organisations. Since its inception 6 years ago, the organisation has grown its presence significantly including a fantastic website of tools, resources and guides for practitioners as well as teaching and supporting individuals and organisations to make better decisions.
Eric shares his personal journey into evidence-based management. Driven by a frustration about the way decisions were being made in his organisation, he looked to evidence-based medicine for guidance on how to apply a more systematic and rigorous approach to making better decisions. As a practitioner he talks about his experiences of using an evidence-based approach to reduce uncertainty when addressing and tackling people challenges.
Richard and Pilar move on to talk about recent articles and reports about the fact that IBM, pioneers of remote working, have now decided that workers in one department must work from one of six locations. Pilar references some research, used to 'sell' the benefits of working in an office which was conducted in the 1980s and 90s and questions its relevance, given the changing nature of work and how technology enables us to work remotely. What do you think? What problem is IBM trying to address and does this mean remote working isn't working?
(Great comments in this other article https://www.washingtonpost.com/news/on-small-business/wp/2017/03/24/ibm-is-ordering-its-work-from-home-employees-to-stop-working-from-home/?utm_term=.5c29536c3a80#comments)
In this episode we are delighted to share an interview with Paul Thoresen, an industrial organisational psychologist based in the US. Paul works with organisations helping them to design, implement and take action from employee surveys.
Richard and Paul discuss some of the challenges around employee engagement and its measurement, such as the definition of the concept and the importance of being clear about the problem you are trying to address before looking at a solution.
They also explore whether surveys are sometimes seen as the solution, rather than a mechanism to gather the views of employees. As a tool they can be over-used and if no action is taken in between by the organisation, it can be a fruitless exercise. Paul encourages organisations to explore the data they may already have in their organisation, or think about other ways of collecting data such as focus groups and interviews.
More broadly we also look at the impact that social media is having on the speed and ease of sharing scientific information, however, it therefore requires us to critically appraise what we see and read.
You can read more about Paul's perspectives on evidence-based practice, organisational change and employee research on Twitter: @surveyguy2 or his blog: https://medium.com/@SurveyGuy2
If you're interested in reading more about employee engagement why not download our white paper: Employee engagement: the emperor's new clothes?
In this episode we do two things:Discuss a recent article about the prevalence of 'neuromyths' in education and the workplace Introduce an initiative we'll be running for the remainder of the year, where we'll be focusing on workplace assessment.
Richard highlights recently came across this article in the Guardian: https://www.theguardian.com/teacher-network/2016/feb/24/four-neuromyths-still-prevalent-in-schools-debunked?CMP=share_btn_tw and thought it was a good example of how misunderstandings of science persist as beliefs despite a lack of evidence, for example 'we only use 10% of our brains' or 'learning styles'. It's easy to see how this popular shorthand of science becomes common knowledge, but we should be extremely wary about using these 'theories' as the basis for shaping policies, educational practices or learning and development.
Pilar and Richard move on to discuss the Future Work Centre's upcoming initiative about workplace assessment. Used by many organisations across the world to help recruit and develop staff, it is an industry fraught with poor practice and can be complex and challenging to navigate. The aim of our campaign is to bring clarity and de-mystify the world of assessment, both for organisations and the people who are on the receiving end.
If you have any questions about the use of assessment, whether you're being asked to complete a questionnaire or if you're an organisation wondering if you're getting the most value from your assessment process, please get in touch - @FW_Centre or firstname.lastname@example.org.
An interview with David D'Souza Head of London & Head of Engagement (branches) at the CIPD, to explore his perspective on introducing and developing an evidence-based perspective in organisations Building on his recent blog post.
We talk about this article which challenges Amy Cuddy's research.
Articles on replicating research:
Employee engagement: the emperor's new clothes?
In this episode, we do two things - introduce the theme for 2017 around assessment in the workplace and what we'll be doing and how to get involved. Then for the majority of the episode, we focus on employee engagement. What does it mean and what does the evidence say?
You can access our own whitepaper on engagement here (http://www.futureworkcentre.com/knowledge-centre/employee-engagement-emperors-new-clothes-2/).
Something a bit different for this episode, as we have an interview with Lorenzo Galli, founder of ScienceForWork.
Scientific evidence is one of the core pillars of evidence-based practice, but for organisations and practitioners there are several challenges in this regard. The first is being able to access it in the first place, whether that’s because of a paywall or knowing where to locate studies. But even if you do manage to access academic research, do you understand what the data is telling you?
ScienceForWork, is a not-for-profit organisation, who address these challenges directly, by aggregating and evaluating research on a range of people-related issues and make their findings available freely, for all.
Pilar is beginning to think that "science" and "research" can become a bit of a fad in themselves, specially anything that's brain related. She recalls a blog article which referenced a study conducted in Colorado State University where three groups of students were shown some fake data about brain imaging, one group with text only, one included a bar graph showing brain-scan results, and one showed pictures of brains. The articles accompanied by brain images were rated significantly higher than the other groups, despite being based on fake data!
You can have a read of the study on http://scienceblogs.com/cognitivedaily/2008/06/04/whats-more-convincing-than-tal/
Science can also be miscommunicated for a variety of reasons, especially in the media. Richard highlights an article from the Independent about personality traits and wellbeing, as an example of an over-simplification of a complex issue.
What do you think? We’d love to hear from you – email@example.com @FW_Centre.