Finding AI’s Niche

Local LLMs have been really interesting to me recently, so much so that I picked up an NVidia DGX Spark. While I still maintain that reading AI slop is utterly demoralizing, there are some parts of it that I find help me a lot. Partly as an exercise in self discovery, and partly to find where AI is overwhelmingly useful, I have explored using it on some personal projects. I hate the idea of letting a clanker have all the fun of programming, but not all programming is unique and exciting.

The biggest litmus test for “Should I use AI for this?” is “Would I do this myself?”. If you, the reader, wouldn’t personally do something, then don’t have an AI do it for you. Your agent acts as an extension of you. Having it do something you wouldn’t means it isn’t you, yet so many people allow it to use their name, their logins, and thus their personal brand.

Unit Tests

For example, writing unit tests is pretty dull work. The coding standard for tests is much lower, with increased grace for things like copy-pasting code, repetitive set up, barely any documentation, excess field and method visibility, no immutability rules, etc. We don’t hold test code to the same standard as regular code.

But, we still write tests. We all agree tests are useful to have. If we did have time, we would write them. Both at work and at home. Thus, AI is okay. It still needs a lot of hand-holding, but it has accelerated my coding allowed me to alt-tab to other things while it churns in the background. I get the most satisfaction out of writing the main code itself, and the (unit) tests are just insurance that the code does what I think it does. It doesn’t really make me a better programmer, and it takes time. It’s easy to zone out while writing unit tests: something that is hard to do while writing real code.

Caveat: Don’t have the AI write both the main code and the tests. It’s the fox watching the hen house. Do one, or the other, but not both.

Build Files

I have not, and probably will never, learn how Gradle works. Despite having used it for nearly a decade, and having hand written thousands of lines of it, I still don’t get it. I’ve tried. Gradle’s documentation is impenetrable. It seems like it’s written as a reference, rather than for people who just kinda know what they want. There is some obscenely complex model for projects, configurations, tasks, plugins, closures, and what not. However, knowledge of Gradle, in my experience, can never be boiled down to something that earns its keep in my working memory. Every few weeks, Gradle releases another minor version that breaks something one of my plugins depends on.
The huge amount of churn in APIs means there’s pretty much no point in learning it; it will be different next month.

Thus, my next litmus test: Do I want to get better at this skill? Am I okay with slowly getting worse at it?

For build files (Gradle, Make, Bazel, etc.), the answer is pretty much no. It is not my life’s purpose to master building projects. LLMs are extremely good at reading through all the build documentation for these tools. Hunting down where this documentation even is is something they excel at. Did I look at the Gradle blog? Did I make sure to check the git commit notes for the most recent Bazel release? Was there some note posted on a far off, newly created git repo that boldly clarifies the new best practices of Bazel? The answer is no, and it’s a waste of time to go on an expedition every time I encounter some build issue. LLMs have none of these problems. They are machines and love rote, methodical work. Best of all, they shed some human biases, such as assuming they have already checked something. Or misremembering important details.

Fast Reconnaissance

Recently, I debugged an OOM with a machine learning product I built. While spelunking through process boundaries and unfamiliar libraries is fun, I also need to be careful about yak shaving. The issue was how Java, XGBoost, and OpenMP manage threads. Java code, and the JVM, are both pretty familiar to me and I can quickly zip through where they might have bad behavior. However, I only have a novice understanding of how XGBoost and OpenMP interact.

To understand how these pieces work, and if the calendar read “2023”, I would have Google’d my questions. Hopefully some curious person would have asked the same question, and some grizzled pro would answer. I would find my answer on a badly HTML-rendered mailing list, or a Stack Overflow page, closed for having been useful-but-not-objective. It’s in places like these that small sparks of insight are found. Even if they aren’t precise matches to my query, they are enough to unwedge my mind.

Therefore, the last litmus test I’ll proffer is Would I have Google’d this?. Generally the answer is “yes, obviously!” I do greatly fear losing my skills, such as no long being able to diagnose and overcome technical problems. But, if I am being honest, using the LLM seems substantially equivalent to searching online for someone else having solved it.

I eventually found out that XGBoost uses OpenMP to create teams of threads, but has no way to globally limit the number of threads. OpenMP’s docs confirmed this, and XGBoost provides no way to close the thread pool down. I worked around the issue by managing the top-level Java threads better, seeing as neither XGboost nor OpenMP provided a way to fix it. Without being able to ask complex questions and using local copies of their git repos, it would have taken a very long time to track down the OOM. That said, I don’t take the LLMs response for granted. It’s just a machine. We still need to retain a hint of doubt, and verify what it says is true. This is especially critical when working in an area I am not familiar with, as I am more susceptible to subtle, but believable, lies.

Respectful AI Usage

Using AI day to day is still immature in the workplace today. I think we need to establish etiquette around code generation, commit and PR messages, and Slack thread chatter. I find it rude when someone’s bot spews a vomitrocious amount of text into the commit message. It’s de-humanizing how I asked people questions, and they respond with “Claude said…” (great, why do you think I couldn’t do that myself?)

Using the first litmus test above (Would I do this myself?) I can’t in good conscience say I would write a novela’s worth of text in my commit message, or respond to people’s inquiries with mechanically-separated drivel.

Humans come first. AI is so new, it’s like we are deer staring into headlights. We are amazed that it can do so much, but yet we forget how using it makes other people feel. The Golden Rule comes in to play here: how would I feel if someone hurled slop across the Internet into my face? If I wouldn’t read it myself, why would I write it? If I wouldn’t trust it myself, why would I expect others to?

Aside: unfortunately, some companies have proclaimed “Use as much AI as possible, or else…” I think this has pressured otherwise respectful people to use LLM slop in improper places.

Conclusion