Is Claude Mythos “Terrifying”? (According to Experts: No.)

About this video

• Video Title: Is Claude Mythos “Terrifying”? (According to Experts: No.)
• Channel: Cal Newport
• Speakers: Cal Newport
• Duration: 00:25:01

Overview

This video debunks the claim that Anthropic's new LLM, Claude Mythos, presents a terrifying new cybersecurity threat. Cal Newport analyzes independent testing and a specific study to argue that while LLM cybersecurity capabilities are improving, Mythos does not represent a revolutionary leap, and the "terrifying" narrative was primarily a marketing strategy by Anthropic.

Key takeaways

• Public Perception vs. Reality: The announcement of Claude Mythos created significant public fear of its cybersecurity capabilities, amplified by media like Thomas Friedman's column, comparing it to the "WOPR" supercomputer from War Games.
• LLMs and Vulnerability Discovery: Security researchers have been using LLMs to find vulnerabilities for years; this is not a new capability exclusive to Mythos. Studies from 2024 showed GPT-4 could already exploit numerous vulnerabilities.
• Zero-Day Vulnerabilities Not New: Anthropic's claim that Mythos can find zero-day vulnerabilities is also not new; their previous model, Opus 4.6, was also capable of this, and the infrastructure has remained secure.
• Independent Testing Findings: Independent security researchers found that smaller, cheaper, open-weight models could replicate the exploit capabilities showcased by Anthropic for Mythos, suggesting Mythos's advantage is not as profound as claimed.
• AIS Study Shows Incremental Improvement: A study by the UK's AI Security Institute (AIS), which had direct access to Mythos, indicated an improvement in its ability to exploit vulnerabilities and navigate complex security scenarios compared to previous models like Opus 4.6, but not a revolutionary leap.
• Marketing Over Substance: The "terrifying" narrative surrounding Mythos was largely a marketing decision by Anthropic, designed to generate attention and fear, rather than a reflection of a groundbreaking cybersecurity threat.
• Skepticism Towards AI Company Claims: The video advocates for skepticism towards claims made by AI companies, emphasizing the need for independent verification, as previous groundbreaking advancements touted by these companies have not materialized.
• Cybersecurity Matters, But Don't Overreact: While LLM cybersecurity capabilities are improving and warrant attention, the level of dread generated by the Mythos announcement was disproportionate to the actual demonstrated advancements.

The speaker believes Anthropic's marketing of Claude Mythos as a cybersecurity threat is bad news because it shifts the focus away from the truly groundbreaking advancements they have been hyping for years, such as automating vast portions of the economy and moving towards AGI. Instead, they are highlighting a capability (finding cybersecurity bugs) that was already present in earlier models like GPT-3 and is considered by some to be a less significant application of AI. This marketing choice suggests that Anthropic may not have the revolutionary capabilities they've been promising investors, leading to questions about the justification for their substantial investment.