A Rational Baseline for AI
These are certain AI truisms and thoughts, according to me, that help me contextualise the limitations of what the current tech called AI (an LLM) has, where it’s good and bad at, and where the outcome is yet to be determined.
Ground truth beliefs #
- Generally speaking, LLMs are fundamentally reconstituted publically and sometimes non-public knowledge available usually from the internet, which then does 2 things really well:
- Pattern match existing text and understand intent, connections and sometimes hidden patterns, beyond what a normal human can do at just a cursory glance (within their context window; think of it as the current working memory for a single LLM turn). This has resulted in what we call inference AI.
- Based on the previous inference capability, but not always limited by it, it can understand input prompt triggers + current or past injected history in context + (optional) supplemental or tool-searched data and provide the next token(s), whether as an answer or as a question to the user.
- By themselves, LLMs are a great piece of tech, and training them to be better, more responsive, and smarter is a key aspect (and expense) for labs.
- However, raw LLMs can be tedious to use and don’t offer any unique advantages over a productive human attention-span timeframe.
- To help, people build what’s called harnesses, deterministic “generalised event loop” style interfaces on top of LLMs, which make things possible.
- Primarily, it enables inference and next-token (and remember token here is generalised; a list of tokens saying “yes, that is blue” and saying “please call google sheets api with these parameters” are the same) to work together to form higher-order workspaces for users to interact in. Claude Code, Codex, etc. work in this space.
- They allow users to compose raw LLM calls beyond what frontier labs provide and build custom harnesses tailored to a specific, point-in-time use case, such as refactoring a codebase, then throw them away.
- The rate of advancement of harnesses is traditionally faster (in part driven as a self-recursive loop using the underlying LLM) than the rate of advancement of a model, which requires significant resourcing (usually in the billions) to build. AI labs funnel their work case studies into building harnesses and expect the general public to follow that for places where the product is not an LLM.
Good things #
- Harnesses and LLMs, by and large, make it much easier and faster to do the < P50 knowledge work, which was always questionable and variable output.
- They also allow for simplifying the more tedious parts of many jobs such as coding, where we don’t necessarily need to write the code compilation symbols anymore; however, for reasons not directly related to LLMs (such as spatial code memory, better design heuristics when typing vs. reading code), people may still prefer some limited amount of typing. In these scenarios, I think of LLMs as really strong, spiky autocomplete. For example, give it a structure by hand or by imitating existing structures, and let it copy and paste. This, imo, isn't going away and is an instantly great, exciting part of these models.
- LLMs lower the bar for doing the work, elevating an existing piece of technology beyond what humans and general capital allocation systems across industries would allow. This is also a great thing, especially for people who want to take pride in what they put out in the world.
- LLMs also lower the bar to learn. This is no different from when the internet was introduced, with books as the source of truth. People could still reference books when they wanted to, but Wikipedia made it easier. Similarly, we can still reference the internet and click through multiple links if we want to, but LLMs make it easier. This does not mean we should not use the previous tools. The internet didn’t suddenly make everyone well-informed, so neither will LLMs provided by the big labs.
Neutral/Bad things #
- The strengths of LLMs lead to 3 core flaws
- Since they are superhumanly good at inference and generating text or some form of pattern-matched input, they can only ever form connections or pattern-match from the existing set of data. Architectural, technical and physical constraints limit LLMs from metacognition aspects that require actually thinking about problems to solve that meaningfully move the needle in how humans work or experience life (and this should be the basic tenet, as LLMs for LLMs' enjoyment is not useful to anyone).
- They are limited by the myopia and hyperfocus on their training data and weights. LLMs fundamentally behave differently based on their training data and weights; while reinforcement learning exists, it doesn’t help with things LLMs don't even know how to reason about, since that isn't baked in during training.
- Finally, this is a weakness of the harness and a current problem (although perhaps a lot easier to fix in the moment than the last two): LLMs will not actively help you fight entropy. For the purposes dictated by their owning corporations, they are incentivised to output tokens, which adds more entropy to things like code, writing, and context. Depending on your domain, this is always a bad thing. All of human progress has been predicated on us fighting entropy to add order to our local reality, and LLMs tend to do the opposite for now.
- I don't buy the narrative from frontier labs and vested interests that LLMs will make most people redundant and useless. Yes, there will be short-term, 3–5 years' worth of pain where everyone who’s fully “online” gets caught up in the hype whirlpool. We have people (CEOs, students, workers) reacting to the hysteria, but in the long term, the core architecture of LLMs will not actually result in any meaningful overall changes. We will ebb to a smaller number of people using LLMs for a while, but once we hit critical mass of AI-driven output and we see its effects, people will rejoin as higher-order abstraction drivers and verifiers. This always happens across many technological shifts. People need to upskill and plan for any local headwinds.
- Finally, discount the narratives coming out of frontier labs and their suppliers. Even by late-stage capitalism standards, their capital requirements create massive incentives to manufacture both hype and fear
- Nvidia has a multi-trillion-dollar incentive to treat compute demand as permanent and infinite, consistently pushing right up against export controls and geopolitical guardrails to keep the silicon moving.
- Anthropic leans heavily on the "doom for all white-collar work" refrain, amplified by a steady stream of employee social posts that conveniently double as marketing for how powerful their models are and a case for regulatory moats.
- OpenAI operates like a hyper-growth fundraising machine, with leadership routinely making sweeping, untethered claims in interviews to justify the next multi-billion-dollar round.
- Finally, I strongly believe if LLMs were a clear god-like superintelligence candidate and long-term driver of complete economic collapse, a few signs would have been true by now:
- AI labs would already have built fully replaceable and maintainable software for every existing 80% value vertical in the world and not need financing.
- AI labs would not be hiring or proposing ever more blog posts and creative ways to use their AI to deal with the harsh shortcomings.
- Leaders wouldn't be drumming up mass paranoia for money or increasing regulation if they truly had an infinite-money glitch.
To summarise, I think LLMs are as significant as, and perhaps more so, in terms of rate of advancement, as the open internet was for the world, and we will see perhaps an explosion of hitherto unprecedented economic opportunity and output from it. However, by their very nature, they act as a compression function on a wide band of knowledge work, and frontier labs have little incentive to make that transition painless. They won't make humans obsolete, but they will force a messy shift up the abstraction stack.
Writing this down is my way of establishing a rational baseline. Parts of it will inevitably be wrong as the tech evolves, and that’s fine, since this is meant to be my own mental model, not an agenda.
- Previous: AI Is Retrocausal