Some More Words About AI and Value

Now that I've had some time to think and cool off about all this AI stuff, I thought I'd revisit the subject one last time, and then just forever hold my peace.

Looking at the situation, I've noticed very few articulated opinions either for or against the AI boom, HPC datacenters, and massive scaling. No one is referring the literature or the research, no one has any reasons to think anything. So, let me give you the gist of what I think now, and that'll be it. I've basically had enough talking, hearing or even thinking about AI anyway.

What I see

I see that politicians like talking about computing infrastructure in terms of highly abstract buzzwords. I see people mix networking infrastructure with raw computing infrastructure. And of course, I see people mix all types of AI algorithms together, even though most people seem to be talking about LLMs, and LLMs alone. Some might be enlightened, in that they actually consider drug discovery or something (oftentimes failing to name what that something is; a common placeholder is "abundance"). Or maybe they're just lusting after completely photorealistic, computer-generated porn. I seriously don't even know at this point. And there's simply no dumber way to goon than giving a text prompt to a computer in exchange for it.

Whatever the case may be, the nonsense lingo -> reasonable translation goes approximately like this:

AI Factory = A regional, multi-tenant HPC data center equipped with some NVIDIA, AMD, or Intel AI accelerators.

AI Antenna = A localized hub or a gateway. Has nothing to do with wireless or RF. Basically just a network access point to an AI Factory.

AI Gigafactory = A datacenter cluster for training LLMs. Totals about a 100,000+ AI accelerators. Sometimes called "hyperscale", whatever the hell that's supposed to mean.

"Four times more compute" = EU wants to match the US in cluster size. Literally no other reason for that particular scaling metric. It's a contest of who has the bigger cock.

Now, you'd think the good thing about all this is that there might soon be an alternative to US-monopolized infrastructure. Purely on principle, that makes total sense to me. But the secret is in the company names there: NVIDIA, AMD, Intel. You have to seriously ask whether you in fact know what goes on in the bowels of your dearly beloved Intel processor, and whether it has the ability to phone home (Intel Management Engine (ME), AMD Secure Processor (PSP)).

And what about the nerdy CS perspective?

To a theoretical computer scientist, the goal of political decision-makers is not about achieving specific mathematical elegance, but about shifting the Pareto frontier of practical computability to capture economic rents, geopolitical leverage, and strategic sovereignty.

Strategic sovereignty of course only means something if we already presume everyone is spying on you, by the way. In case that needed to be mentioned.

The politicos say they want to increase computing power by four times; they are betting that scaling laws mean that it soon becomes possible to solve optimization problems currently intractable within European borders.

So you do a bit of a simulation there, and the computation has high time complexity, something like O(2^n) or O(n!). The goal is to use machine learning models in order to come up with accurate simulations and a lower time complexity.

Still, knowing what I know, I'm not at all yet convinced that there are algorithms for which essentially the constant factor becomes within-reach with just a bigger datacenter, so to speak.

Namely, from a strict complexity perspective, moving from a standard supercomputer to a hyperscale "Gigafactory" is just a 4x to 10x constant factor increase in hardware throughput. If an algorithm scales exponentially, O(2^n), a 4x increase in compute capacity yields an insultingly small gain in the solvable problem size: you can merely handle n + 2 inputs in the same wall-clock time.

And before you say it, I have indeed noticed some subtle interest in galactic-looking algorithms rising lately, but I'm not too sure how relevant that is.

Now, I've heard that scaling dense transformers, the empirical cross-entropy loss L behaves like a perfectly smooth power law relative to compute^* L(C) ≈ ( C_{c} / C )^{𝛼_{C}}. So the hope here is to drop the complexity even a little to gain more capabilities downstream.

^*I'll have to double-check that later, once I have time. I'm thinking maybe I'll implement transformers of some kind, if I have time and transformers are still relevant then. Might be a fun little learning project.

Moreover, there's an interest in chopping up the computation into phases so that you first do the expensive stuff to establish a base LLM prior (or some such), and then run verification loops and other approximations at test time, and then maybe save even more for later, for inference time.

Still, that's like an analogue to lazy evaluation, and knowing how that works, what this means is actually less total efficiency in exchange for a kind of just-in-time computation. Meaning, energy bills just keep on risin' in exchange for a gamble that this approach'll prove irreplaceable by earlier hardware solutions.

Put it another way, people basically hope that just having a machine learning model there it can somehow take a shortcut no traditional algorithm would, and that that shortcut is accurate and trustworthy.

So, simply following the current trend like a blind donkey, the European Commission is betting over €30 billion on the assumption that brute-force statistical approximation via hyperscale compute will achieve things that clever, highly optimized, deterministic algorithms simply cannot. But I guess someone's gotta do it; is China doing it?

But here's the irony, though: if this brute-forcing is intended to be a temporary patch before more efficient techniques are discovered – namely, a more efficient algorithm that achieves the same reasoning power – then these multi-gigawatt facilities instantly become if not obsolete, then quickly repurposed, with the GPU stuff being the first to go. I suppose the plan is the accelerators become obsolete every 3 to 5 years anyway. However, they will have spent billions building massive power lines and cooling systems for hardware that was only necessary to prop up an inherently inefficient, brute-force statistical paradigm.

What it looks like

Thing is, writing a highly efficient deterministic algorithm requires deep human domain expertise and years of R&D for every single specific problem.

The policy makers are paying a massive premium in electricity just to avoid the human bottleneck of algorithmic discovery. In the AI Gigafactory era, we are using the most advanced hardware in human history to run some of the most theoretically sloppy, brute-force search heuristics ever conceived.

If a "Gigafactory" can use brute force to discover a critical defense technology, a new battery chemistry, or a semiconductor material six months faster than a human team writing a targeted algorithm, the economic and strategic advantage of those six months is perceived to outweigh the cost of the electricity bill – regardless of whether the human-made solution could be reused elsewhere!

And, regardless of whether market forces can be accurately predicted. First-mover advantage, yeah I've heard of that.

If a breakthrough algorithm reduces the computational complexity of frontier AI from a massive brute-force requirement to a lightweight polynomial workload, the physical infrastructure undergoes a brutal re-commoditization. A mid-sized business can run what used to require a Gigafactory on a local rack of 8 GPUs. The Gigafactory loses its monopolistic pricing power...

Either way, those are some big ifs.

Which is to say nothing about the qualitative uses for all that "generated intelligence". Personally, I'm of the mind that those practical uses are incredibly few.

Now, I've yapped about this in my own chaotic way in this here blog, and I mentioned this incredibly weird opinion of mine that the CPU actually trumps GPUs, but I didn't bother to explain why I think that. Here's why: it is the combined reason of pointer-chasing, power-efficiency, and how time complexity works. Long story short, most writing is all about chasing pointers around and performing the write in a scanning, left-to-right fashion. Because of I/O speeds, you won't usually gain much from parallelization. For in-memory ops though, especially with proper memory reuse, that speedup is real. I'm thinking every CS guy or gal knows this, so it's weird practically nobody has even mentioned it.

That is, as models shift toward test-time reasoning (MCTS, graph-of-thought, tree-search verification loops), they start introducing complex branching and pointer-heavy tree structures, and so you get what? Pointer-chasing.

And what's there to say about power-efficiency except GPUs cost big bucks both at purchase-time and runtime? I dunno. I still like to play games though, so there's that.

And time complexity? Well, put it this way: the same computational task that runs in a minute using one algorithm can sometimes be completed in six seconds with another; but when it already takes less time due to parallelization, then what you have is the same computation that runs in 10 seconds can be completed in 4 at best by a better algorithm. That's it in a nutshell: 6 seconds cheaply vs 4 expensively. There's a famous thing called Amdahl's law, look it up here.

And if the algorithmic breakthrough does not come, well, it'll cost you anyway. It's the poor plebs that absorb all the cost, being treated like dumb cattle.

Some bits about memory

What about memory speed? Everyone's horny for memory now with all the shortage going on.

When looking at marketing material, you will see bold claims like "DDR5 7200 MT/s is vastly faster than DDR3 1600 MT/s." This statement is only true for sequential bandwidth – the maximum amount of contiguous data the memory can stream per second. To a computer scientist, the more critical metric for overall execution flow is random access latency (the structural fetch delay). This is the absolute time, measured in nanoseconds, that elapses between the processor requesting a memory address and receiving the very first burst of data. While raw throughput and data transfer bandwidth have expanded by nearly 450% from DDR3 to DDR5, the structural time it takes to pluck an isolated byte from a random memory address has remained largely frozen at roughly 10 to 11 nanoseconds for over fifteen years.

This ceiling exists because the underlying physics of the DRAM storage matrix have not changed. The micro-capacitors and transistors on the silicon die still charge, discharge, and sense voltages at roughly the same physical speed.

In other words, when running algorithms that traverse non-contiguous data structures – where the processor cannot pre-fetch memory because the address of node B is unknown until node A finishes reading – modern hyper-fast DDR5 RAM drops back to the speed of a decade-old system.

For the GPU and its own GDDR or HBM memory, the same is even more true.

Now, I also recently found some hottie babe talking about channel-hedged DRAM reads. This was interesting.

Channel-hedged DRAM reads are basically a tradeoff between p99+ tail-latency and bandwidth: DRAM cells are tiny capacitors that constantly leak charge. Every few milliseconds, the memory controller must lock a bank and issue a Refresh Command to refill them; if a single-channel read hits a bank during a refresh cycle, it stalls for up to hundreds of nanoseconds, and so you can slash that by issuing the same operation on the same exact memory address. Because physical channels operate on independent internal clocks, Channel A and Channel B almost never run refresh cycles at the exact same picosecond anyway.

That's nerdy and cool, but it slashes your bandwidth too. Moreover, in the HFT world they use FPGAs that bypass RAM entirely, SRAM, or QDR.

Back to AI

The only justification in my mind for this nonsense is if the big countries basically have a pipeline prepared, ready to dish out new deadly viruses or deadly mutagens or something at a moment's notice, so that you actually need to be able to create a counter-medicine for that in six months rather than ten, so your population isn't wiped by an even deadlier COVID.

That's to say practical uses for generated intelligence are incredibly few.

But we'll see. "Never say never", as the saying goes. I myself would rather trust people than some hallucinating statistical algorithm, but you know how it is. Humankind is sick, sick with wanting to be better than everyone else.

Let me ask you just one final thing I've tried to desperately answer for myself the last few years: what difference does it make whether we create intelligence to exterminate the Russians or whether we create intelligence to exterminate the US? Or China. Or anyone. Or whether we subjugate those damn Africans. Huh?

No but we'll get that abundance that was promised. Thanks for keeping me informed. Goddamn fucking bozos.