LLMs: The Wall Is Now a Mirror
From The Information — Dec 5th 2024
Back in November, I wrote about how Large Language Models (LLMs) seem to be hitting a wall. My piece, “LLMs Are Hitting the Wall: What’s Next?”, explored the challenges of scaling these models and the growing realization that brute force and larger datasets aren't enough to push them closer to true intelligence. I argued that while LLMs excel in pattern recognition and syntactic fluency, their lack of deeper reasoning and genuine understanding exposes critical limitations.
Today, Gary Marcus has weighed in with his own sharp insights in his Substack post, “Which CEO Will Be the Last to See the Truth About Generative AI?”. Marcus not only reinforces the idea that LLMs are at a plateau but goes further to question the industry's fixation on them as a panacea for AI advancement. He points out how the obsession with generative AI could be a distraction from addressing the structural issues that hold these systems back from true reliability and utility.
Gary’s piece resonated with me because it highlights a key issue I’ve been grappling with: the tendency of AI leaders to overpromise on capabilities while underdelivering on robustness. As he notes, this misalignment between hype and reality risks undermining the broader field of AI.
While my November article was primarily about technical limitations, Gary shines a light on the cultural and strategic blind spots within the industry. He calls out the “tech FOMO” (fear of missing out) that drives many CEOs to double down on flawed systems, even as the cracks become more visible.
The real question, as we both seem to agree, is not whether LLMs can keep generating clever outputs but whether the next leap in AI will require a fundamental rethinking of what intelligence means. I believe this pivot could involve exploring interdisciplinary approaches — combining insights from cognitive science, neuroscience, and even philosophy — to build systems that do more than mimic language.
Gary’s post reminds me of a valuable lesson: sometimes hitting a wall isn’t just a signal to stop; it’s an opportunity to look in the mirror and ask the hard questions about where to go next.
What do you think? Are we ready to look beyond the wall — or are we too captivated by its reflection to see what’s on the other side?