Episodes

  • #77 – Elizabeth Seger on Open Sourcing AI

    #77 – Elizabeth Seger on Open Sourcing AI

    · Hear This Idea

    Elizabeth Seger is the Director of Technology Policy at Demos, a cross-party UK think tank with a program on trustworthy AI.

    You can find links and a transcript at www.hearthisidea.com/episodes/seger In this episode we talked about open source the risks and benefits of open source AI models. We talk about:

    What ‘open source’ really meansWhat is (and isn’t) open about ‘open source’ AI modelsHow open source weights and code are useful for AI safety researchHow and when the costs of open sourcing frontier model weights might outweigh the benefitsAnalogies to ‘open sourcing nuclear designs’ and the open science movement

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

    Note that this episode was recorded before the release of Meta’s Llama 3.1 family of models. Note also that in the episode Elizabeth referenced an older version of the definition maintained by OSI (roughly version 0.0.3). The current OSI definition (0.0.8) now does a much better job of delineating between different model components.

  • #76 – Joe Carlsmith on Scheming AI

    #76 – Joe Carlsmith on Scheming AI

    · Hear This Idea

    Joe Carlsmith is a writer, researcher, and philosopher. He works as a senior research analyst at Open Philanthropy, where he focuses on existential risk from advanced artificial intelligence. He also writes independently about various topics in philosophy and futurism, and holds a doctorate in philosophy from the University of Oxford.

    You can find links and a transcript at www.hearthisidea.com/episodes/carlsmith

    In this episode we talked about a report Joe recently authored, titled ‘Scheming AIs: Will AIs fake alignment during training in order to get power?’. The report “examines whether advanced AIs that perform well in training will be doing so in order to gain power later”; a behaviour Carlsmith calls scheming.

    We talk about:

    Distinguishing ways AI systems can be deceptive and misalignedWhy powerful AI systems might acquire goals that go beyond what they’re trained to do, and how those goals could lead to schemingWhy scheming goals might perform better (or worse) in training than less worrying goalsThe ‘counting argument’ for scheming AIWhy goals that lead to scheming might be simpler than the goals we intendThings Joe is still confused about, and research project ideas

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • Missing episodes?

    Click here to refresh the feed.

  • #75 – Eric Schwitzgebel on Digital Consciousness and the Weirdness of the World

    #75 – Eric Schwitzgebel on Digital Consciousness and the Weirdness of the World

    · Hear This Idea

    Eric Schwitzgebel is a professor of philosophy at the University of California, Riverside. His main interests include connections between empirical psychology and philosophy of mind and the nature of belief. His book The Weirdness of the World can be found here.

    We talk about:

    The possibility of digital consciousnessPolicy ideas for avoiding major moral mistakes around digital consciousnessProspects for the science of consciousness, and why we likely won't have clear answers in timeWhy introspection is much less reliable than most people thinkHow and why we invent false stories about our own choices without realisingWhat randomly sampling people's experiences reveals about what we're doing with most of our attentionThe possibility of 'overlapping minds'How and why our actions might have infinite effects, both good and badWhether it would be good news to learn that our actions have infinite effects, or that the universe is infinite in extentThe best science fiction on digital minds and AI

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #74 – Sonia Ben Ouagrham-Gormley on Barriers to Bioweapons

    #74 – Sonia Ben Ouagrham-Gormley on Barriers to Bioweapons

    · Hear This Idea

    Sonia Ben Ouagrham-Gormley is an associate professor at George Mason University and Deputy Director of their Biodefence Programme

    In this episode we talk about:

    Where the belief that 'bioweapons are easy to make' came from and why it has been difficult to changeWhy transferring tacit knowledge is so difficult -- and the particular challenges that rogue actors faceAs well as lastly what Sonia makes of the AI-Bio risk discourse and what types of advances in technology would cause her concern

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • Bonus: 'How I Learned To Love Shrimp' & David Coman-Hidy

    Bonus: 'How I Learned To Love Shrimp' & David Coman-Hidy

    · Hear This Idea

    In this bonus episode we are sharing an episode by another podcast: How I Learned To Love Shrimp. It is co-hosted by Amy Odene and James Ozden, who together are "showcasing innovative and impactful ways to help animals".

    In this interview they speak to David Coman-Hidy, who is the former President of The Humane –League, one of the largest farm animal advocacy organisations in the world. He now works as a Partner at Sharpen Strategy working to coach animal advocacy organisations.

  • #73 – Michelle Lavery on the Science of Animal Welfare

    #73 – Michelle Lavery on the Science of Animal Welfare

    · Hear This Idea

    Michelle Lavery is a Program Associate with Open Philanthropy’s Farm Animal Welfare team, with a focus on the science and study of animal behaviour & welfare.

    You can see more links and a full transcript at hearthisidea.com/episodes/lavery

    In this episode we talk about:

    How do scientists study animal emotions in the first place? How is a "science" of animal emotion even feasible?When is it useful to anthropomorphise animals to understand them?How can you study the preferences of animals? How can you measure the “strength” of preferences?How do farmed animal welfare advocates relate to animal welfare science? Are their perceptions fair?How can listeners get involved with the study of animal emotions?

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #72 – Richard Bruns on Indoor Air Quality

    #72 – Richard Bruns on Indoor Air Quality

    · Hear This Idea

    Dr Richard Bruns is a Senior Scholar at the Johns Hopkins Center for Health Security, and before that was a Senior Economist at the US Food and Drug Administration (the FDA).

    In this episode we talk about the importance of indoor air quality (IAQ), and how to improve it. Including:

    Estimating the DALY cost of unclean indoor air from pathogens and particulate matterHow much pandemic risk could be reduced from improving IAQ?How economists convert health losses into dollar figures — and how not to put a price on lifeKey interventions to improve IAQAir filtrationGermicidal UV light (especially Far-UVC light)Barriers to adoption, including UV smog and empirical studies needed mostNational and state-level policy changes to get these interventions adopted widely

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #71 – Saloni Dattani on Malaria Vaccines and Missing Data in Global Health

    #71 – Saloni Dattani on Malaria Vaccines and Missing Data in Global Health

    · Hear This Idea

    Saloni Dattani is a Researcher at Our World in Data, and a founder & editor at the online magazine Works in Progress. She holds a PhD in psychiatric genetics from King’s College London.

    You can see more links and a full transcript at hearthisidea.com/episodes/dattani.

    In this episode we talk about:

    The history of malaria and attempts to eradicate itThe role of DDT and insecticide spraying campaigns — and why they were scaled downWhy we didn’t get a malaria vaccine soonerWhat comes after vaccine discovery — rolling out the RTS,S vaccineNew funding models to accelerate similar life-saving research, like vaccines for TB and HIVWhy so much global health data is missing, and why that mattersHow the ‘million deaths study’ revealed that about 50,000 deaths per year from snakebites in India went uncounted by health agencies

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #70 – Liv Boeree on Healthy vs Unhealthy Competition

    #70 – Liv Boeree on Healthy vs Unhealthy Competition

    · Hear This Idea

    Liv Boeree is a former poker champion turned science communicator and podcaster, with a background in astrophysics. In 2014, she founded the nonprofit Raising for Effective Giving, which has raised more than $14 million for effective charities. Before retiring from professional poker in 2019, Liv was the Female Player of the Year for three years running. Currently she hosts the Win-Win podcast (you’ll enjoy it if you enjoy this podcast).

    You can see more links and a full transcript at hearthisidea.com/episodes/boeree.

    In this episode we talk about:

    Is the ‘poker mindset’ valuable? Is it learnable?How and why to bet on your beliefs — and whether there are outcomes you shouldn’t make bets onWould cities be better without public advertisements?What is Moloch, and why is it a useful abstraction?How do we escape multipolar traps?Why might advanced AI (not) act like profit-seeking companies?What’s so important about complexity? What is complexity, for that matter?

    You can get in touch through our website or on Twitter. Consider leaving us an honest review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #69 – Jon Y (Asianometry) on Problems And Progress in Semiconductor Manufacturing

    #69 – Jon Y (Asianometry) on Problems And Progress in Semiconductor Manufacturing

    · Hear This Idea

    Jon Y is the creator of the Asianometry YouTube channel and accompanying newsletter. He describes his channel as making "video essays on business, economics, and history. Sometimes about Asia, but not always."

    You can see more links and a full transcript at hearthisidea.com/episodes/asianometry

    In this episode we talk about:

    Compute trends driving recent progress in Artificial Intelligence;The semiconductor supply chain and its geopolitics;The buzz around LK-99 and superconductivity.

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #68 – Steven Teles on what the Conservative Legal Movement Teaches about Policy Advocacy

    #68 – Steven Teles on what the Conservative Legal Movement Teaches about Policy Advocacy

    · Hear This Idea

    Steven Teles s is a Professor of Political Science at Johns Hopkins University and a Senior Fellow at the Niskanen Center. His work focuses on American politics and he written several books on topics such as elite politics, the judiciary, and mass incarceration.

    You can see more links and a full transcript at hearthisidea.com/teles

    In this episode we talk about:

    The rise of the conservative legal movement;How ideas can come to be entrenched in American politics;Challenges in building a new academic field like "law and economics";The limitations of doing quantitative evaluations of advocacy groups.

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!Key links:

  • #67 – Guive Assadi on Whether Humanity Will Choose Its Future

    #67 – Guive Assadi on Whether Humanity Will Choose Its Future

    · Hear This Idea

    Guive Assadi is a Research Scholar at the Center for the Governance of AI. Guive’s research focuses on the conceptual clarification of, and prioritisation among, potential risks posed by emerging technologies. He holds a master’s in history from Cambridge University, and a bachelor’s from UC Berkeley.

    In this episode, we discuss Guive's paper, Will Humanity Choose Its Future?.

    What is an 'evolutionary future', and would it count as an existential catastrophe?How did the agricultural revolution deliver a world which few people would have chosen?What does it mean to say that we are living in the dreamtime? Will it last?What competitive pressures in the future could drive the world to undesired outcomes?Digital mindsSpace settlementWhat measures could prevent an evolutionary future, and allow humanity to more deliberately choose its future?World governmentStrong global coordinationDefensive advantageShould this all make us more or less hopeful about humanity's future?Ideas for further research

    Guive's recommended reading:

    Rationalist Explanations for War by James D. FearonMeditations on Moloch by Scott AlexanderThe Age of Em by Robin HansonWhat is a Singleton? By Nick Bostrom

    Other key links:

    Will Humanity Choose Its Future? by Guive AssadiColder Wars by GwernThe Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter by Joseph Henrich (and a review by Scott Alexander)
  • #66 – Michael Cohen on Input Tampering in Advanced RL Agents

    #66 – Michael Cohen on Input Tampering in Advanced RL Agents

    · Hear This Idea

    Michael Cohen is is a DPhil student at the University of Oxford with Mike Osborne. He will be starting a postdoc with Professor Stuart Russell at UC Berkeley, with the Center for Human-Compatible AI. His research considers the expected behaviour of generally intelligent artificial agents, with a view to designing agents that we can expect to behave safely.

    You can see more links and a full transcript at www.hearthisidea.com/episodes/cohen.

    We discuss:

    What is reinforcement learning, and how is it different from supervised and unsupervised learning?Michael's recently co-authored paper titled 'Advanced artificial agents intervene in the provision of reward'Why might it be hard to convey what we really want to RL learners — even when we know exactly what we want?Why might advanced RL systems might tamper with their sources of input, and why could this be very bad?What assumptions need to hold for this "input tampering" outcome?Is reward really the optimisation target? Do models "get reward"?What's wrong with the analogy between RL systems and evolution?

    Key links:

    Michael's personal website'Advanced artificial agents intervene in the provision of reward' by Michael K. Cohen, Marcus Hutter, and Michael A. Osborne'Pessimism About Unknown Unknowns Inspires Conservatism' by Michael Cohen and Marcus Hutter'Intelligence and Unambitiousness Using Algorithmic Information Theory' by Michael Cohen, Badri Vallambi, and Marcus Hutter'Quantilizers: A Safer Alternative to Maximizers for Limited Optimization' by Jessica Taylor'RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning' by Marc Rigter, Bruno Lacerda, and Nick Hawes'Quantilizers: A Safer Alternative to Maximizers for Limited Optimization' by Jessica TaylorSeason 40 of Survivor
  • #65 – Katja Grace on Slowing Down AI and Whether the X-Risk Case Holds Up

    #65 – Katja Grace on Slowing Down AI and Whether the X-Risk Case Holds Up

    · Hear This Idea

    Katja Grace is a researcher and writer. She runs AI Impacts, a research project trying to incrementally answer decision-relevant questions about the future of artificial intelligence (AI). Katja blogs primarily at worldspiritsockpuppet, and indirectly at Meteuphoric, Worldly Positions, LessWrong and the EA Forum.

    We discuss:

    What is AI Impacts working on?Counterarguments to the basic AI x-risk caseReasons to doubt that superhuman AI systems will be strongly goal-directedReasons to doubt that if goal-directed superhuman AI systems are built, their goals will be bad by human lightsAren't deep learning systems fairly good at understanding our 'true' intentions?Reasons to doubt that (misaligned) superhuman AI would overpower humanityThe case for slowing down AIIs AI really an arms race?Are there examples from history of valuable technologies being limited or slowed down?What does Katja think about the recent open letter on pausing giant AI experiments?Why read George Saunders?

    Key links:

    World Spirit Sock Puppet (Katja's main blog)Counterarguments to the basic AI x-risk caseLet's think about slowing down AIWe don't trade with antsThank You, Esther Forbes (George Saunders)

    You can see more links and a full transcript at hearthisidea.com/episodes/grace.

  • #64 – Michael Aird on Strategies for Reducing AI Existential Risk

    #64 – Michael Aird on Strategies for Reducing AI Existential Risk

    · Hear This Idea

    Michael Aird is a senior research manager at Rethink Priorities, where he co-leads the Artificial Intelligence Governance and Strategy team alongside Amanda El-Dakhakhni. Before that, he conducted nuclear risk research for Rethink Priorities and longtermist macrostrategy research for Convergence Analysis, the Center on Long-Term Risk, and the Future of Humanity Institute, which is where we know each other from. Before that, he was a teacher and a stand up comedian. He previously spoke to us about impact-driven research on Episode 52.

    In this episode, we talk about:

    The basic case for working on existential risk from AIHow to begin figuring out what to do to reduce the risksThreat models for the risks of advanced AI'Theories of victory' for how the world mitigates the risks'Intermediate goals' in AI governanceWhat useful (and less useful) research looks like for reducing AI x-riskPractical advice for usefully contributing to efforts to reduce existential risk from AIResources for getting started and finding job openings

    Key links:

    Apply to be a Compute Governance Researcher or Research Assistant at Rethink Priorities (applications open until June 12, 2023)Rethink Priority's survey on intermediate goals in AI governanceThe Rethink Priorities newsletterThe Rethink Priorities tab on the Effective Altruism ForumSome AI Governance Research Ideas compiled by Markus Anderljung & Alexis CarlierStrategic Perspectives on Long-term AI Governance by Matthijs MaasMichael's posts on the Effective Altruism Forum (under the username "MichaelA")The 80,000 Hours job board
  • #63 – Ben Garfinkel on AI Governance

    #63 – Ben Garfinkel on AI Governance

    · Hear This Idea

    Ben Garfinkel is a Research Fellow at the University of Oxford and Acting Director of the Centre for the Governance of AI.

    In this episode we talk about:

    An overview of AI governance space, and disentangling concrete research questions that Ben would like to see more work onSeeing how existing arguments for the risks from transformative AI have held up and Ben’s personal motivations for working on global risks from AIGovAI’s own work and opportunities for listeners to get involved

    Further reading and a transcript is available on our website: hearthisidea.com/episodes/garfinkel

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #62 – Anders Sandberg on Exploratory Engineering, Value Diversity, and Grand Futures

    #62 – Anders Sandberg on Exploratory Engineering, Value Diversity, and Grand Futures

    · Hear This Idea

    Anders Sandberg is a researcher, futurist, transhumanist and author. He holds a PhD in computational neuroscience from Stockholm University, and is currently a Senior Research Fellow at the Future of Humanity Institute at the University of Oxford. His research covers human enhancement, exploratory engineering, and 'grand futures' for humanity.

    This episode is a recording of a live interview at EAGx Cambridge (2023). You can find upcoming effective altruism conferences here: www.effectivealtruism.org/ea-global

    We talk about:

    What is exploratory engineering and what is it good for?Progress on whole brain emulationAre we near the end of humanity's tech tree?Is diversity intrinsically valuable in grand futures?How Anders does researchVirtue ethics for civilisationsAnders' takes on AI risk and whether LLMs are close to general intelligenceAnd much more!

    Further reading and a transcript is available on our website: hearthisidea.com/episodes/sandberg-live

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #61 – Rory Stewart on GiveDirectly and Massively Scaling Cash Transfers

    #61 – Rory Stewart on GiveDirectly and Massively Scaling Cash Transfers

    · Hear This Idea

    Rory Stewart is the President of GiveDirectly and a visiting fellow at Yale’s Jackson Institute for Global Affairs. Before that, Rory was (amongst other things) a Member of Parliament in the UK, a Professor in Human Rights at Harvard, and a diplomat. He is also the author of several books and co-hosts the podcast The Rest Is Politics.

    In this episode, we talk about:

    The moral case for radically scaling cash-transfersWhat we can do to raise governments’ ambitions to end global povertyWhat Rory learned about aid since being Secretary of State for International Development

    Further reading is available on our website: hearthisidea.com/episodes/stewart

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #60 – Jaime Sevilla on Trends in Machine Learning

    #60 – Jaime Sevilla on Trends in Machine Learning

    · Hear This Idea

    Jaime Sevilla is the Director of Epoch, a team of researchers investigating and forecasting the development of advanced AI. This is his second time on the podcast.

    Over the next few episodes, we will be exploring the potential for catastrophe cause by advanced artificial intelligence. Why? First, you might think that AI is likely to become transformatively powerful within our lifetimes. Second, you might think that such transformative AI could result in catastrophe unless we’re very careful about how it gets implemented. This episode is about understanding the first of those two claims.

    Fin spoke with Jaime about:

    We've seen a crazy amount of progress in AI capabilities in the last few months; even weeks. How should we think about that progress continuing into the future?How has the amount of compute used to train AI models been changing over time? What about algorithmic efficiency?Will data soon become a bottleneck in training state-of-the-art text models?

    Further reading is available on our website: hearthisidea.com/episodes/sevilla

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • #59 – Chris Miller on the History of Semiconductors, TSMC, and the CHIPS Act

    #59 – Chris Miller on the History of Semiconductors, TSMC, and the CHIPS Act

    · Hear This Idea

    Chris Miller is an Associate Professor of International History at Tufts University and author of the book “Chip War: The Fight for the World's Most Critical Technology” (the Financial Times Business Book of the Year). He is also a Visiting Fellow at the American Enterprise Institute, and Eurasia Director at the Foreign Policy Research Institute.

    Over the next few episodes we will be exploring the potential for catastrophe cause by advanced artificial intelligence. But before we look ahead, we wanted to give a primer on where we are today: on the history and trends behind the development of AI so far. In this episode, we discuss:

    How semiconductors have historically been related to US military strategyHow the Taiwanese company TSMC became such an important player in this space — while other countries’ attempts have failedWhat the CHIPS Act signals about attitudes to compute governance in the decade ahead

    Further reading is available on our website: hearthisidea.com/episodes/miller

    If you have any feedback, you can get a free book for filling out our new feedback form. You can also get in touch through our website or on Twitter. Consider leaving us a review wherever you're listening to this — it's the best free way to support the show. Thanks for listening!

  • Show more