Episodes
-
Is Apache Spark too costly? Amazon Principal Engineer Patrick Ames tackled this question during an interview with The New Stack Makers, sharing insights into transitioning from Spark to Ray for managing large-scale data. Ames, described as a "go-to" engineer for exabyte-scale projects, emphasized a goal-driven approach to solving complex engineering problems, from simplifying daily chores to optimizing software solutions.
Initially, Spark was chosen at Amazon for its simplicity and open-source flexibility, allowing efficient merging of data with minimal SQL code. The team leveraged Spark in a decoupled architecture over S3 storage, scaling it to handle thousands of jobs daily. However, as data volumes grew to hundreds of terabytes and beyond, Spark’s limitations became apparent. Long processing times and high costs prompted a search for alternatives.
Enter Ray—a unified framework designed for scaling AI and Python applications. After experimentation, Ames and his team noted significant efficiency improvements, driving the shift from Spark to Ray to meet scalability and cost-efficiency needs.
Learn more from The New Stack about Apache Spark and Ray:
Amazon to Save Millions Moving From Apache Spark to Ray
How Ray, a Distributed AI Framework, Helps Power ChatGPT
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
In this New Stack Makers, Codiac aims to simplify app deployment on Kubernetes by offering a unified interface that minimizes complexity. Traditionally, Kubernetes is powerful but challenging for teams due to its intricate configurations and extensive manual coding. Co-founded by Ben Ghazi and Mark Freydl, Codiac provides engineers with infrastructure on demand, container management, and advanced software development life cycle (SDLC) tools, making Kubernetes more accessible.
Codiac’s interface streamlines continuous integration and deployment (CI/CD), reducing deployment steps to a single line of code within CI/CD pipelines. Developers can easily deploy, manage containers, and configure applications without mastering Kubernetes' esoteric syntax. Codiac also offers features like "cabinets" to organize assets across multi-cloud environments and enables repeatable processes through snapshots, making cluster management smoother.
For experienced engineers, Codiac alleviates the burden of manually managing YAML files and configuring multiple services. With ephemeral clusters and repeatable snapshots, Codiac supports scalable, reproducible development workflows, giving engineers a practical way to manage applications and infrastructure seamlessly across complex Kubernetes environments.
Learn more from The New Stack about deploying applications on Kubernetes:
Kubernetes Needs to Take a Lesson from Portainer on Ease-of-Use
Three Common Kubernetes Challenges and How to Solve Them
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Missing episodes?
-
Valkey, an open-source fork of Redis launched in March, introduced its multithreaded Version 8.0 in September, now available through AWS ElastiCache. At All Things Open 2024 in Raleigh, AWS's Kyle Davis explains that Valkey was developed after Redis changed to a restrictive license, drawing contributors from companies like AWS, Google, Alibaba, and Oracle. Notably, some contributors emerged independently, including a significant contributor from Vietnam. Version 8.0 differentiates itself from Redis by leveraging multithreaded CPUs, addressing the efficiency of I/O operations in modern hardware. Additionally, data structure refinements were made to improve memory efficiency by up to 20%, particularly benefiting large-key databases.
Looking ahead, Valkey plans two annual updates, with the next release expected in 2025. New modules are anticipated, including a JSON module for efficient data manipulation and a Bloom filter for probabilistic data presence checks. Version 9.0 may bring substantial changes to clustering, updating it to better leverage modern technologies. The Valkey project aims to continue evolving its capabilities to meet the demands of advanced data storage needs.
Learn more from The New Stack about Valkey:
Valkey Is a Different Kind of Fork
AWS Adds Support, Drops Prices, for Redis-Forked Valkey
Valkey: A Redis Fork With a Future
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Deb Nicholson, executive director of the Python Software Foundation, attributes Python’s popularity to its minimal syntactical complexity, which appeals to beginners and seasoned developers alike. Python allows flexibility for those exploring coding without a specific focus, unlike purpose-built languages. Since her leadership began in 2022, Nicholson has overseen the foundation’s role in managing Python’s fiscal and operational needs, including the package index that hosts over half a million add-ons. This open ecosystem enables contributions from large corporations and individual developers while demanding vigilant security measures.
Nicholson envisions Python's future advancements, particularly in improving multi-threading and expanding usage in mobile development. She acknowledges Python’s critical role in AI and data science but remains cautious about AI’s pervasive application, likening it to a temporary trend. On open source in the enterprise, Nicholson critiques companies profiting from open-source tools while adopting restrictive licenses. Instead, she admires models like Red Hat’s, which leverage open source sustainably without compromising accessibility or innovation.
Learn more from The New Stack about Python:
Python 3.13: Blazing New Trails in Performance and Scale
The Top 5 Python Packages and What They Do
Python Mulls a Change in Version Numbering
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Platform engineering will be a key focus at KubeCon this year, with a special emphasis on AI platforms. Priyanka Sharma, executive director of the Linux Foundation, highlighted the convergence of platform engineering and AI during an interview on The New Stack Makers with Adobe’s Joseph Sandoval. KubeCon will feature talks from experts like Chen Goldberg of CoreWeave and Aparna Sinha of CapitalOne, showcasing how AI workloads will transform platform operations.
Sandoval emphasized the growing maturity of platform engineering over the past two to three years, now centered on addressing user needs. He also discussed Adobe's collaboration on CNOE, an open-source initiative for internal developer platforms. The intersection of platform engineering, Kubernetes, cloud-native technologies, and AI raises questions about scaling infrastructure management with AI, potentially improving efficiency and reducing toil for roles like SRE and DevOps. Sharma noted that reference architectures, long requested by the CNCF community, will be highlighted at the event, guiding users without dictating solutions.Learn more from The New Stack about Kubernetes:
Cloud Native Networking as Kubernetes Starts Its Second Decade
Primer: How Kubernetes Came to Be, What It Is, and Why You Should Care
How Cloud Foundry Has Evolved With Kubernetes
Join our community of newsletter subscribers to stay on top of the news and at the top of your game. game. https://thenewstack.io/newsletter/
-
Rohit Choudhary, co-founder and CEO of Acceldata, placed an early bet on data observability, which has proven prescient. In a New Stack Makers podcast episode, Choudhary discussed three key insights that shaped his vision: First, the exponential growth of data in enterprises, further amplified by generative AI and large language models. Second, the rise of a multicloud and multitechnology environment, with a majority of companies adopting hybrid or multiple cloud strategies. Third, a shortage of engineering talent to manage increasingly complex data systems.
As data becomes more essential across industries, challenges in data observability have intensified. Choudhary highlights the complexity of tracking where data is produced, used, and its compliance requirements, especially with the surge in unstructured data. He emphasized that data's operational role in business decisions, marketing, and operations heightens the need for better traceability. Moving forward, traceability and the ability to manage the growing volume of alerts will become areas of hyper-focus for enterprises.
Learn more from The New Stack about data observability:
What Is Data Observability and Why Does It Matter?
The Looming Crisis in the Observability Market
The Growth of Observability Data Is Out of Control!
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Rust has maintained its place among the top 15 programming languages and has been the most admired language for nine consecutive years. In a New Stack Makers podcast, Joel Marcey, director of technology at the Rust Foundation, discussed the language's growing importance, including initiatives to improve its security, performance, and adoption in various domains. While Rust is widely used in systems and backend programming, it’s also gaining traction in embedded systems, safety-critical applications, game development, and even the Linux kernel.
Marcey highlighted Rust’s strengths as a safe and fast systems language, noting its use on the web through WebAssembly (Wasm), though adoption there is still early. He also addressed Rust vs. Go, explaining that Rust excels in performance-critical applications. Marcey discussed recent updates, such as Rust 1.81, and project goals for 2024, which include a new edition and async improvements.
He also touched on government interest in Rust, including DARPA’s initiative to convert C code to Rust, and the Rust Security Initiative, aimed at maintaining the language’s strong security reputation.
Learn more from The New Stack about Rust
Could Rust be the Future of JavaScript Infrastructure?
Rust Growing Fastest, But JavaScript Reigns Supreme
Rust vs. Zig in Reality: A (Somewhat) Friendly Debate
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
In a New Stack Makers episode, Ashley Williams, founder and CEO of axo, highlights how the software world depends on open-source code, which is largely maintained by unpaid volunteers. She likens this to a CVS relying on volunteer-run shipping companies, pointing out how unsettling that might be for customers. The conversation focuses on open-source maintainers’ reluctance to be seen as "suppliers" of software, an idea explored in a 2022 blog post by Thomas Depierre. Many maintainers reject the label, as there is no contractual obligation to support the software they provide.
Williams critiques the industry's response to this, noting that instead of involving maintainers in software supply chain security, companies have relied on third-party vendors. However, these vendors have no relationship with the maintainers, leading to increased vulnerabilities. Williams advocates for better engagement with maintainers, especially at build time, to improve security. She also reflects on the growing pressures on maintainers and the underappreciation of release teams.
Learn more from The New Stack about open source software supply chain
2023: The Year Open Source Security Supply Chain Grew Up
Fortifying the Software Supply Chain
The Challenges of Securing the Open Source Supply Chain
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
In this New Stack Makers podcast, Xun Wang, CTO of Bloomreach, brings insights from his time at Nvidia, particularly lessons from its founder, Jensen Huang, to his current role in e-commerce personalization. Wang emphasizes structuring organizations to reflect the architecture of the products they build, applying a hands-on, detail-oriented approach that encourages deep understanding of engineering challenges.
He credits Huang for teaching him the importance of focusing on fundamental architecture rather than relying on iterative testing alone. Wang highlights the impact of generative AI (GenAI) on Bloomreach, explaining how AI-driven search is essential to understanding human language and user intent. As GenAI reshapes application development, Wang stresses the need for engineers to adopt new skills in AI manipulation, while still maintaining traditional coding expertise. He advocates for continuous learning, acknowledging the challenge of staying updated in a rapidly evolving field. Wang, himself, reads extensively to keep pace with innovations, underscoring the importance of staying curious and adaptable in today’s tech landscape.
Learn more from The New Stack about Entrepreneurship for Engineers:
How to Grow into Leadership
Engineering Leaders: Switch to Wartime Management Now
How Teleport’s Leader Transitioned from Engineer to CEO
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Code reviews can be highly beneficial but tricky to execute well due to the human factors involved, says Adrienne Braganza Tacke, author of *Looks Good to Me: Actionable Advice for Constructive Code Review.* In a recent conversation with *The New Stack*, Tacke identified three challenges teams must address for successful code reviews: ambiguity, subjectivity, and ego.
Ambiguity arises when the goals or expectations for the code are unclear, leading to miscommunication and rework. Tacke emphasizes the need for clarity and explicit communication throughout the review process. Subjectivity, the second challenge, can derail reviews when personal preferences overshadow objective evaluation. Reviewers should justify their suggestions based on technical merit rather than opinion. Finally, ego can get in the way, with developers feeling attached to their code. Both reviewers and submitters must check their egos to foster a constructive dialogue.
Tacke encourages programmers to first review their own work, as self-checks can enhance the quality of the code before it reaches the reviewer. Ultimately, code reviews can improve code quality, mentor developers, and strengthen team knowledge.
Learn more from The New Stack about code reviews:
The Anatomy of Slow Code Reviews
One Company Rethinks Diff to Cut Code Review Times
How Good Is Your Code Review Process?
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
In the New Stack Makers episode, Adi Polak, Director, Advocacy and Developer Experience Engineering at Confluent discusses the operational and analytical estates in data infrastructure. The operational estate focuses on fast, low-latency event-driven applications, while the analytical estate handles long-running data crunching tasks. Challenges arise due to the "schema evolution" from upstream operational changes impacting downstream analytics, creating complexity for developers.
Apache Iceberg and Flink help mitigate these issues. Iceberg, a table format developed by Netflix, optimizes querying by managing file relationships within a data lake, reducing processing time and errors. It has been widely adopted by major companies like Airbnb and LinkedIn.
Apache Flink, a versatile data processing framework, is driving two key trends: shifting some batch processing tasks into stream processing and transitioning microservices into Flink streaming applications. This approach enhances system reliability, lowers latency, and meets customer demands for real-time data, like instant flight status updates. Together, Iceberg and Flink streamline data infrastructure, addressing developer pain points and improving efficiency.
Learn more from The New Stack about Apache Iceberg and Flink:
Unfreeze Apache Iceberg to Thaw Your Data Lakehouse
Apache Flink: 2023 Retrospective and Glimpse into the Future
4 Reasons Why Developers Should Use Apache Flink
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Bob Wise, CEO of Heroku, discussed the impact of generative AI (GenAI) coding tools on software development in a recent episode of The New Stack Makers. He compared the rise of these tools to adding an "infinite number of interns" to development teams, noting that while they accelerate code writing, they don't yet simplify testing, deployment, or production operations. Wise likened this to the early days of Kubernetes, which focused on improving operations rather than the frontend experience. He emphasized that Kubernetes' success was due to its focus on easing the operational burden, something current GenAI tools have yet to achieve.
Heroku, acquired by Salesforce in 2010, is positioned to benefit from these changes by helping teams transition to more automated systems. Wise highlighted Heroku’s strategic bet on Postgres, a database technology that's gaining traction, especially for GenAI workloads. He also discussed Heroku's ongoing migration to Kubernetes, aligning with industry standards to enhance its platform.
Learn more from The New Stack about Heroku
The Data Stack Journey: Lessons from Architecting Stacks at Heroku and Mattermost
Kubernetes and the Next Generation of PaaS
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
After the XZ Utils backdoor vulnerability was uncovered in March, the OpenJS Foundation saw a surge in inquiries from potential open source JavaScript contributors. Robin Ginn, executive director of the foundation, noted that volunteer-led JavaScript communities often face challenges in managing these contributions. The discovery that a single contributor, "Jia Tan," planted the backdoor heightened vigilance, especially when new contributors requested admin privileges. Ginn emphasized that trust is not synonymous with security, especially in open source projects where maintainers must be vigilant about who can access their repositories.
The XZ vulnerability highlighted broader concerns about the security of open source software, particularly in projects with only a single maintainer. Despite receiving a significant grant from Germany's Sovereign Tech Fund, the foundation remains under-resourced, with just two full-time staffers supporting 35 projects. Ginn urged companies that rely on open source software to invest in it by hiring maintainers, ensuring these critical projects are properly supported.
Learn more from The New Stack about open source vulnerability
Linux xz Backdoor Damage Could Be Greater Than Feared
Unzipping the XZ Backdoor and Its Lessons for Open Source
Linux xz and the Great Flaws in Open Source
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Paige Bailey, who began coding at age 9 in rural Texas, now leads the GenAI developer experience at Google. In a conversation with Chris Pirillo on The New Stack Makers, Bailey reflected on the evolving role of software development in the era of generative AI. While she once urged her nieces and nephews to pursue computer science degrees, Bailey now believes that critical thinking and problem-solving may be more crucial for future tech careers.
She emphasized that generative AI is democratizing software development, making it more accessible and enabling developers to focus on creative tasks rather than the minutiae of coding. Bailey's experience at Google highlights this shift, as she now acts more as a reviewer and overseer of AI-generated code. She sees GenAI not as a replacement for developers but as a tool to accelerate their creativity and tackle longstanding backlogs. Bailey believes the key is ensuring everyone understands how to effectively apply generative AI to their work.
Learn more from The New Stack about the future of development:
7 Ways to Future Proof Your Developer Job in the Age of AI
The Future of Developer Careers
4 Forecasts for the Future of Developer Relations
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Anne Currie, a leading expert in sustainable tech and part of the Green Software Foundation, discusses practical steps for building resilient, sustainable software in an episode of The New Stack Makers. With 30 years of experience, Currie co-authored Building Green Software, emphasizing the tech industry's role in the energy transition. She highlights the complexity of adapting technology to renewable energy, involving extensive research and debunking misinformation. Currie discusses the importance of energy proportionality—the idea that increased utilization improves a computer's energy efficiency—and how this concept aligns with modern DevOps practices that reduce carbon emissions while enhancing speed, cost efficiency, and security.
Currie also emphasizes architecting systems to operate on renewable power and draws parallels between managing variable grid power and internet bandwidth. Using examples like video conferencing, she illustrates how software can adapt to fluctuating resources. The episode also touches on potential pitfalls like greenwashing and the challenges in accurately naming concepts like energy proportionality.
Learn more from The New Stack about sustainability:
Sustainability: How Did Amazon, Azure, Google Perform in 2023?
Sustainability Focus: Cloud Efficiency, Not Carbon Emissions
Developers Should Press Cloud Providers on Sustainability
Join our community of newsletter subscribers to stay on top of the news and at the top of your game. https://thenewstack.io/newsletter/
-
In an era marked by complexity, the golden path is essential for software architects, asserts James Watters, senior director of R&D at VMware Tanzu, Broadcom. This approach, emphasizing fewer application patterns, simplifies life for security personnel, developers, and infrastructure teams. VMware defines the golden path as streamlining software development, crucial in today's economic climate. Watters highlights this in the Broadcom report: State of Cloud Native App Platforms 2024, noting that 55% of organizations favor this method for its consistency and security.
Watters, a pioneer in platform as a service since 2009, helped establish Cloud Foundry and now drives VMware Tanzu. Tanzu's golden operations offer standardized, consistent processes across platforms, crucial for efficiency and security. Watters advocates for minimal DIY in favor of operational consistency, providing commands for building, deploying, and scaling applications.
Tanzu’s focus is on integrating AI to enhance user interfaces and data access, impacting platform engineering significantly in the coming years. This integration aims to offer a better developer experience while maintaining security and efficiency.
Learn more from The New Stack about golden paths:
Golden Paths Start with a Shift Left
Platform Engineering Not Working Out? You’re Doing It Wrong.
How to Pave Golden Paths That Actually Go Somewhere
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
Maintaining and ensuring the success of a microservice-based system can be challenging. Sarah Wells, a seasoned tech consultant with over 20 years of experience, offers valuable insights in her book "Enabling Microservices Success" and a discussion on The New Stack Makers podcast. Drawing from her tenure at the Financial Times (FT), Wells illustrates how transitioning to microservices and adopting DevOps and SRE practices enabled FT to accelerate software releases from 12 annually to over 20,000.
This transformation required merging IT organizations, investing in automation, and fostering team autonomy. Wells emphasizes that successful microservices adoption depends not only on developer expertise but also on organizational structures. She highlights the importance of continuous delivery and proactive communication, especially during critical periods like major news events. Additionally, she discusses the evolving roles of senior engineers and the need for flexibility in defining architectural responsibilities. Wells advocates for "engineering enablement" over "platform teams" to better support effective service management and evolution.
Learn more from The New Stack about enabling successful outcomes of microservices:
What Is Microservices Architecture?
4 Strategies for Migrating Monolithic Apps to Microservices
Continuous Improvement Metrics for Scaling Engineering Teams
Join our community of newsletter subscribers to stay on top of the news and at the top of your game. https://thenewstack.io/newsletter/
-
In August 2023, the open source community rallied to create OpenTofu, an alternative to Terraform, after HashiCorp, now owned by IBM, adopted a restrictive Business Source License for Terraform. Ohad Maislish, co-founder and CEO of env0, explained on The New Stack Makers how this move sparked the initiative. A few hours after HashiCorp's license change, Maislish secured the domain opentf.org and began developing the new project, eventually named OpenTofu, which was donated to The Linux Foundation to ensure its license couldn't be altered.
Maislish highlighted the importance of distinguishing between vendor-backed and foundation-backed open source projects to avoid sudden licensing changes. Before coding, the community created a manifesto, gathering significant support and pledges, but received no response from HashiCorp. Consequently, they proceeded with the fork and development of OpenTofu. Despite accusations of intellectual property theft from HashiCorp, OpenTofu gained traction and was adopted by organizations like Oracle. The community continues to prioritize user feedback through GitHub.
Learn more from The New Stack about OpenTofu:
OpenTofu vs. HashiCorp Takes Center Stage at Open Source Summit
OpenTofu Amiable to a Terraform Reconciliation
OpenTofu 1.6 General Availability: Open Source Infrastructure as Code
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
-
In the early days, the internet was a decentralized space created by enthusiasts. However, it has since transformed into a centralized, commerce-driven entity dominated by a few major players. The promise of the fediverse, a decentralized social networking concept, offers a refreshing alternative.
Evan Prodromou, OpenEarth Foundation's director of open technology, has been advocating for decentralized social networks since 2008, starting with his creation, Identi.ca. Unlike Twitter, Identi.ca was open source and federated, allowing independent networks to interconnect.
Prodromou, a co-author of ActivityPub—the W3C standard for decentralized networking used by platforms like Mastodon—discusses the evolution of the fediverse on The New Stack Makers podcast. He notes that small social networks dwindled to a few giants, such as Twitter and Facebook, which rarely interconnected. The acquisition of Twitter by Elon Musk disrupted the established norms, prompting users to reconsider their dependence on centralized platforms.
The fediverse aims to address these issues by allowing users to maintain relationships across different instances, ensuring a smoother transition between networks. This decentralization fosters community management and better control over social interactions.
Check out the full podcast episode to explore how tech giants like Meta are engaging with the fediverse and how to join decentralized social networks.
Learn more from The New Stack about fediverse:
FediForum Showcases New Fediverse Apps and Developer Network
One Login: Towards a Single Fediverse Identity on ActivityPub
Web Dev 2024: Fediverse Ramps Up, More AI, Less JavaScript
Join our community of newsletter subscribers to stay on top of the news and at the top of your game. https://thenewstack.io/newsletter/
-
In a recent episode of The New Stack Makers, recorded at the Open Source Summit North America, Matt Hartley, Linux support lead at Framework, discusses the importance of the "right to repair" movement. This initiative seeks to allow consumers to repair and upgrade their own electronic devices, countering the trend of disposable electronics that contribute to environmental damage. Framework, a company offering modular and customizable laptops, embodies this philosophy by enabling users to replace outdated components easily.
Hartley, interviewed by Chris Pirillo, highlights how Framework’s approach helps reduce electronic waste, likening obsolete electronics to a form of "technical debt." He shares his personal struggle with old devices, like an ASUS Eee, illustrating the need for repairable technology. Hartley also describes his role in fostering a DIY community, collaborating closely with Fedora Linux maintainers and creating user-friendly support scripts. Framework’s community is actively contributing to the platform, developing new features and hardware integrations.
The episode underscores the growing momentum of the right to repair movement, advocating for consumer empowerment and environmental sustainability.
Learn more from The New Stack about repairing and upgrading devices:
New Linux Laptops Come with Right-to-Repair and More
Troubling Tech Trends: The Dark Side of CES 2024
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
- Show more
