Storing Lists in Registers

While working on my Wordle solver, I found a trick for treating integers as little lists. To keep memory usage down, I decided to pack the 5 letter words into a 32 bit integer. I use 6 bits per letter, leaving the top bit as a zero. This allows doing some SIMD tricks without having to expand the nibbles later.

However, sometimes it’s convenient to treat each Wordle word as a variable length list of characters, rather than the fixed size, 5 letter words that the game uses. Since the game uses up to 5 letters per word, and each letter consumes a 6 bit nibble, that leaves 32 - (5 * 6) = 2 bits left for use. 2 bits isn’t enough to encode a length field. However, we can use a trick similar to NULL terminated C strings: 1-bit prefixed ints.

This trick works by treating each 32 bit int as a [0, 5] length string of 6 bit values, prefixed by a 2 bit prefix. For example:

bits:  01EE EEEE DDDD DDCC CCCC BBBB BBAA AAAA
index: 1098 7654 3210 9876 5432 1098 7654 3210

The A-E bits represent the nibbles representing the letters, while the 01 prefix indicates the end of the string. To detect the length of the String, we count the number of leading 0 bits and divide by 6 to get the length:

MAX_CODES_PER_WORD = 5;
BITS_PER_CODE = 6;

int zeros = Integer.numberOfLeadingZeros(codeWord);

return MAX_CODES_PER_WORD - ((zeros - 1) / BITS_PER_CODE);

This is nice because Integer.numberOfLeadingZeros() is a single instruction on Intel architectures. Additionally, while dividing by 6 would normally be slow, the compiler optimizes this to a “magic number” division, i.e. a multiplication by (2^32) / 6. On my machine, calculating the length is around 3ns.

The End of Ads

While playing around with local LLM inference, I started to internalize how the web is going to look in the future. This isn’t a hard bet, but the incentive structure is now clear that Ads will no longer be the preferred way of monetizing the web. ¹ Instead, I see the web continuing to exist as a pay-per-view model, where most servers reject initial requests with a 405 Payment Required error along with a price.

How We Got Here

Up until about 2023, Google, along with other search engines, was the way people found information online. Google interleaved ads in the results, which received clicks, and thus revenue from advertisers. Google’s rise to prominence stemmed from their ability to distinguish fraudulent clicks from real clicks from real people. Advertisers picked Google’s ad network over others due to higher conversion rates, and were happy to pay the premium. This product is called AdWords.

For the non advertising links, users were redirected to the many websites across the web. These websites, as they became more popular, needed money to pay for the rising cost of servers, bandwidth, and staff to keep things running. For the mass majority of sites, the cost (and traffic) was so low as to be pennies, and it was just a fun way to be part of the Internet. For sites that were larger, there were two ways to make money:

1. Show ads. These started off as text ads, evolved to flashing GIF ads, and finally as video and picture ads.
2. Require payment. If a user pays for access, they can participate in the site.

Making payments on the Internet has always been a barrier to entry. For any early stage website, ads were the only feasible way to go. Charging for access would alienate the original user base for a site, making it impractical to ever switch from free or ads-based, over to a payment model.

Google also made this model of sustaining possible. By putting their ad snippet in your HTML, Google could show ads in the header, footer, or any other high visibility space. They would figure out who your site’s audience is, what ads would do best in their slot, and keep track of payment. Once a month, if you had enough clicks, Google would cut you a check for your share of the ad revenue. This product is called AdSense.

How We Broke Down

With the proliferation of LLMs, the virtuous cycle of making sites, getting clicks, earning money, and making more sites, broke down. OpenAI, followed by Anthropic and others, showed the world there is a better user experience. AI companies boil down the entirety of the Internet, (and books and everything else) into a machine that generate text (a.k.a. GenAI). There are two phases to this process:

1. Consume all available information. This is the training phase.
2. Use the trained model, and occasionally query for outside data, to generate text. This is the inference phase.

As the world turns, so too do trained models become out of date. They must constantly be re-trained with new information (such as news) to be able to answer topical questions.² This has resulted in immense load on the websites of the Internet, with utterly insatiable demand for all the data available. Website proprietors saw colossal increases in cost across the board, but with no humans looking or clicking on their ads.

In other words, almost every site out there is paying higher cost, with no additional ad revenue to cover.

This might be sustainable in the long run, but I’m betting not. Many websites have tried to exclude bot traffic, i.e. the machines scraping them, by using technologies such as Cloudflare and reCaptcha (ironic?). However, while this may help limit the costs, it doesn’t help with the other virtuous-cycle breakdown: No more ad revenue.

Google, adopting the UX of OpenAI, started putting their own generated text at the top of the search results. The starting point for users to visit the rest of the world wide web is being spawn-camped. I’d bet heavily that Google’s own outbound clicks from the search engine page is crumbling. With no more [human] traffic being steered into websites, there is no more ad revenue to sustain the site. With the ever increasing bot traffic to websites, the cost is going up. This will not last!

(Aside: I think this change in our technology world is not necessarily a bad thing. The Tech Industry evolves faster and faster, with outdated ideas falling before the new innovation. As a user of AI, the LLM snippets are a better experience. I just want to make sure the Golden Goose isn’t killed in the transition.)

What Can Be Done To Fix This

Until I started playing around with local LLMs, I kinda knew this world we live in was not going to last. It seems hopeless to solve, with all the incentives being upended and no clear part which needs to be changed. The key idea that made me change my thinking was who was making all these requests to websites?

LLMs have a weakness: They only know approximate things. Like the compression artifacts of a JPEG, precise information is hard to capture.
For example, what was the daily high temperature in Los Angeles on August 29th, 1997? The machine knows generally, what temperatures there are, and generally what temperatures there are in August, and generally what temperatures there are in Los Angeles. But for specifics, you need to go the the NOAA [FTP] site to answer this.

Running locally, this problem becomes a lot more apparent. The model itself is only 20-100 GB in size. Think about that. All the worlds knowledge boiled down into something that can fit on a thumb drive. Something has to be removed for it to fit. Unlike the production grade models that Anthropic, OpenAI, and others run, the local models suffer more from imprecise knowledge.

To fix this problem, harnesses (like pi.dev) know how to issue queries to websites and other sources to fill in gaps in their memory. And, crucially, who is making this query? It’s me. Carl. Not some faceless megacorp making millions of queries per second, but just me.

We have a chance! Looking back at the virtuous cycle above, website owners now have a way to ensure they can withstand the influx of load. Not in a “personalized, hyper targeted panopticon” kind of way, but in a “this is a real request for knowledge” way.

Carl’s Bet for the Future.

With this in mind, here is what I see for the (hopeful) survival of the web:

Payment Required

Most websites will require a small, almost insubstantial payment for their services. When accessing a site, it will reject it with 405 Payment Required. Just like a challenge for authentication, our agents will pay a tiny bit and retry the request with a receipt, finally getting the data. Scraping the web will flip to being encouraged, with website owners doing everything they can to invite traffic.

Subscriptions

Rather than every website implementing their own payment solution, I expect there to be a standardized payment mechanism, perhaps built into the HTTP Spec itself. Rather than go through the challenge each time, a user will buy a pass to query an unlimited amount, (perhaps tokens?), and have the cost deducted.

Bulk Data Deals

Reddit was probably the first to sell bulk access to their data. If I recall, Google paid like $60 Million for unabridged access, in order to train their model. I suspect deals like this will make sense for much larger sites.

Boutique Data Brokers

Not all data is easily fetched, and some data can only be discovered in person. I can see a world where custom data is collected and organized by small shops, and resold for usage by AIs. There will be a large open market for niche data that LLMs don’t know about, but whose users would like to know. For example: “Is this restaurant open on Labor Day? Someone can go in person to find out and publish it via an API.

Where We Go From Here

This change in the world didn’t click for me until I tried using local LLMs myself. When my harness’s requests were rejected, I thought “I’d be happy to pay for that, just give me the data”. Problems such as overscraping evaporate when crawlers have to pay for their data. So many problems just disappear. I think we as a tech community need to agree that the web is in a precarious state, agree on how we can enable it going forward, and build our tools with empathy in mind. Website owners need an incentive to keep providing us with valuable services if we want them to stick around.

Finding AI’s Niche

Local LLMs have been really interesting to me recently, so much so that I picked up an NVidia DGX Spark. While I still maintain that reading AI slop is utterly demoralizing, there are some parts of it that I find help me a lot. Partly as an exercise in self discovery, and partly to find where AI is overwhelmingly useful, I have explored using it on some personal projects. I hate the idea of letting a clanker have all the fun of programming, but not all programming is unique and exciting.

The biggest litmus test for “Should I use AI for this?” is “Would I do this myself?”. If you, the reader, wouldn’t personally do something, then don’t have an AI do it for you. Your agent acts as an extension of you. Having it do something you wouldn’t means it isn’t you, yet so many people allow it to use their name, their logins, and thus their personal brand.

Unit Tests

For example, writing unit tests is pretty dull work. The coding standard for tests is much lower, with increased grace for things like copy-pasting code, repetitive set up, barely any documentation, excess field and method visibility, no immutability rules, etc. We don’t hold test code to the same standard as regular code.

But, we still write tests. We all agree tests are useful to have. If we did have time, we would write them. Both at work and at home. Thus, AI is okay. It still needs a lot of hand-holding, but it has accelerated my coding allowed me to alt-tab to other things while it churns in the background. I get the most satisfaction out of writing the main code itself, and the (unit) tests are just insurance that the code does what I think it does. It doesn’t really make me a better programmer, and it takes time. It’s easy to zone out while writing unit tests: something that is hard to do while writing real code.

Caveat: Don’t have the AI write both the main code and the tests. It’s the fox watching the hen house. Do one, or the other, but not both.

Build Files

I have not, and probably will never, learn how Gradle works. Despite having used it for nearly a decade, and having hand written thousands of lines of it, I still don’t get it. I’ve tried. Gradle’s documentation is impenetrable. It seems like it’s written as a reference, rather than for people who just kinda know what they want. There is some obscenely complex model for projects, configurations, tasks, plugins, closures, and what not. However, knowledge of Gradle, in my experience, can never be boiled down to something that earns its keep in my working memory. Every few weeks, Gradle releases another minor version that breaks something one of my plugins depends on.
The huge amount of churn in APIs means there’s pretty much no point in learning it; it will be different next month.

Thus, my next litmus test: Do I want to get better at this skill? Am I okay with slowly getting worse at it?

For build files (Gradle, Make, Bazel, etc.), the answer is pretty much no. It is not my life’s purpose to master building projects. LLMs are extremely good at reading through all the build documentation for these tools. Hunting down where this documentation even is is something they excel at. Did I look at the Gradle blog? Did I make sure to check the git commit notes for the most recent Bazel release? Was there some note posted on a far off, newly created git repo that boldly clarifies the new best practices of Bazel? The answer is no, and it’s a waste of time to go on an expedition every time I encounter some build issue. LLMs have none of these problems. They are machines and love rote, methodical work. Best of all, they shed some human biases, such as assuming they have already checked something. Or misremembering important details.

Fast Reconnaissance

Recently, I debugged an OOM with a machine learning product I built. While spelunking through process boundaries and unfamiliar libraries is fun, I also need to be careful about yak shaving. The issue was how Java, XGBoost, and OpenMP manage threads. Java code, and the JVM, are both pretty familiar to me and I can quickly zip through where they might have bad behavior. However, I only have a novice understanding of how XGBoost and OpenMP interact.

To understand how these pieces work, and if the calendar read “2023”, I would have Google’d my questions. Hopefully some curious person would have asked the same question, and some grizzled pro would answer. I would find my answer on a badly HTML-rendered mailing list, or a Stack Overflow page, closed for having been useful-but-not-objective. It’s in places like these that small sparks of insight are found. Even if they aren’t precise matches to my query, they are enough to unwedge my mind.

Therefore, the last litmus test I’ll proffer is Would I have Google’d this?. Generally the answer is “yes, obviously!” I do greatly fear losing my skills, such as no long being able to diagnose and overcome technical problems. But, if I am being honest, using the LLM seems substantially equivalent to searching online for someone else having solved it.

I eventually found out that XGBoost uses OpenMP to create teams of threads, but has no way to globally limit the number of threads. OpenMP’s docs confirmed this, and XGBoost provides no way to close the thread pool down. I worked around the issue by managing the top-level Java threads better, seeing as neither XGboost nor OpenMP provided a way to fix it. Without being able to ask complex questions and using local copies of their git repos, it would have taken a very long time to track down the OOM. That said, I don’t take the LLMs response for granted. It’s just a machine. We still need to retain a hint of doubt, and verify what it says is true. This is especially critical when working in an area I am not familiar with, as I am more susceptible to subtle, but believable, lies.

Respectful AI Usage

Using AI day to day is still immature in the workplace today. I think we need to establish etiquette around code generation, commit and PR messages, and Slack thread chatter. I find it rude when someone’s bot spews a vomitrocious amount of text into the commit message. It’s de-humanizing how I asked people questions, and they respond with “Claude said…” (great, why do you think I couldn’t do that myself?)

Using the first litmus test above (Would I do this myself?) I can’t in good conscience say I would write a novela’s worth of text in my commit message, or respond to people’s inquiries with mechanically-separated drivel.

Humans come first. AI is so new, it’s like we are deer staring into headlights. We are amazed that it can do so much, but yet we forget how using it makes other people feel. The Golden Rule comes in to play here: how would I feel if someone hurled slop across the Internet into my face? If I wouldn’t read it myself, why would I write it? If I wouldn’t trust it myself, why would I expect others to?

Aside: unfortunately, some companies have proclaimed “Use as much AI as possible, or else…” I think this has pressured otherwise respectful people to use LLM slop in improper places.

Conclusion

AI and LLMs have been a marvelous addition to humanity, but we should be prudent about when and where they are appropriate.