Gusarich's thoughts

There is nothing out-of-distribution

AI turned out to be very simple if you think about it. You just make a model that works with something generic, and feed as much training data as you can into it. The generic data I mean here is text. People had writing for thousands of years and the whole world is built on it — we write, we speak, we read, and we listen our entire lives. It's so deep in our brains that it's hard to think of something that cannot be described in text.

Some AI pessimists love to say that "LLMs do not generalize out-of-distribution", but if they are built to understand and write text — they can generalize to anything that can be described as text, so pretty much everything. What people actually refer to when pointing out to generalization problems is intelligence.

Why couldn't LLMs solve simple puzzles 2 years ago if they can generalize? The reason is simple: LLMs were stupid. Bad data, bad models, and bad training produced models with very limited intelligence. They simply lacked IQ to generalize well. Nothing was changed conceptually on a fundamental level in the past couple of years — we just gathered better data, designed better architectures, and improved training algorithms.

Things that LLMs fail right now usually aren't "boolean" in nature. If it only solves 1% of some specific tasks right now, it means that it can solve them sometimes and next generations of models will solve more. And it's hard to find a task where LLMs couldn't complete at least the simplest version of it today.

Inability to solve 100% of given tasks of some kind right away just means the lack of intelligence. In order to generalize some problem, you need the intelligence to understand and solve that problem well. There is no data limitation in LLMs in the sense of generalization ability, only an intelligence limitation that is being improved rapidly.

I believe that LLMs are one of the valid paths to ASI. There could be other paths, even better and more general ones, but LLMs can do the thing too. ASI is not far away already, and from what we'll see in 2026 we won't have to change much to reach a superhuman level of general intelligence. I don't see anything that could stop LLMs from progressing further at the current exponential pace.

Writing with AI

If you scroll up through my Telegram channel, or open my first blog posts, you will easily notice how they were completely written with AI. Those very obvious patterns that are very easy to spot, like "this isn't just X, it's Y". I was mostly writing drafts myself, but using AI to "finish the paragraph", and also to completely rewrite the whole draft in the end for the purpose of fixing grammar errors and "improving writing".

I'm now ashamed of that. A lot of time has passed and only recently I stopped using AI like this. Writing has to come fully from myself, otherwise it makes no sense. It also helps to actually think more about what I write. Before I could just drop a bunch of random thoughts into AI and ask it to write a nice post out of it, but as a result I skipped the whole stage of thinking in a structured way that happens when you write things yourself.

I think I changed my mind on all that after seeing hundreds of fully AI-generated posts on X for months. They all look the same. My first posts look this way too when I reread them now. It's soulless, feels cheap, and often provides less value to readers.

I'm not using AI this way anymore, at least for things that actually require some thinking from me. I'm now only using it to fact-check, spot grammar errors and give feedback, but never to rewrite whole chunks of text or to finish my thoughts. I'm writing everything myself, then asking AI to give feedback to me, then changing things myself. For random posts on X, I don't even do these last steps.

But I definitely use AI for boilerplate stuff, like prompts for AI itself or some Slack messages. Those things don't require much thinking from my side anyway. And I think for such cases, it's fine.

Channel photo updated

This February I made a https://x.com/GusarichOnX/status/1889624242611741123 that AI would surpass top human performers in competitive programming by the end of the summer.

For context, at that point, the best models we had were o1, Sonnet 3.6, and Gemini 2.0. They couldn't reliably solve even simple tasks, and the only hint we had was the late-December preview of o3 (which eventually turned out to be a different version from what we got in April). This "o3-preview", when benchmarked on Codeforces contests, ranked in the 99.8th percentile, which is extremely good, but it was still far behind top performers, positioned at about ~130th place.

In the past couple of months, Google and OpenAI have been https://x.com/OpenAI/status/1945959016653598816 their experimental reasoning models' skills by participating in various math and coding olympiads. Just today, they both posted about their results at ICPC, which is the most prestigious competitive programming olympiad, where university students from different countries participate in teams of 3.

Google's system https://x.com/GoogleDeepMind/status/1968361776321323420 10/12 problems, which is enough for a gold medal already. And OpenAI https://x.com/OpenAI/status/1968368133024231902 a stunning 12/12, which is even more than the top-1 team from Saint Petersburg solved. Basically, AI could solve all those extremely difficult problems given the same time and constraints as human contestants, and actually achieved first place, which I think can count as a successful resolution to my prediction.

Even earlier, in December, I made a more general for 2025, saying that 1. AI would surpass humans in different technical fields (mostly meaning math/coding); and 2. AI would discover a lot of new science.

We still can't say that AI has "surpassed" humans in all those things, and it's actually quite hard to define that outcome. And AI hasn't discovered much this year either, but I think with these recent results in various olympiads we're definitely getting there very quickly. Overall, I still believe in those timings, and the general timing for ASI in 2027, or even earlier.

Why did many people have a bad first impression of GPT-5?

Actually, the reason behind that is absurdly stupid. OpenAI fucked up with UX. That's it. The model is actually good; all variants of it are. But OpenAI rushed the release for some reason, and their goal of making the UX better made it worse for a lot of users.

The key detail here was the model router that they added to ChatGPT so that users don't have to manually choose a model, and it can just choose the appropriate one on its own. For example, if you ask it how to pronounce a word, that can easily be answered with a non-thinking model, with lower latency and the same accuracy. But if you give it a math problem, ask something about coding, or just generally give it a task that requires more reasoning - it is better processed by a thinking variant of the model.

And the idea is good, especially for the average user who doesn't know much about how these models work and doesn't want to think about which model to choose for every query. But the implementation was very bad in the first couple of days, and OpenAI confirmed that themselves. The router was working poorly, not choosing a thinking model for complex queries when needed, and not only that, the information about which model answered the query was also hidden, so, for example, when your request (as a free/plus user) was routed to "GPT-5 mini", you couldn't know that. There's not even a "mini" model in the model picker.

And another factor is the "reasoning effort" parameter that OpenAI's models have. It determines "how hard" the model thinks before answering. If you need quicker answers, choose "low"; if you need more reasoning for more complex tasks, use "medium" or "hard". And the thing is that in ChatGPT you can't choose that setting yourself. It's part of the router, too. And the information about this setting is also hidden from users.

It turned out that most of the requests from free/plus users were processed either by a non-thinking model, or by a thinking model but with the "low" reasoning effort setting. And sometimes also by "mini" models when limits for the regular ones were exhausted. And the performance is, expectedly, bad under those circumstances.

So, even when paying users tried out GPT-5, they were often getting bad results. And that was their impression of GPT-5. And that's why there was so much hate for it online.

But OpenAI is fixing those problems, and some are already fixed. So, if you tried out GPT-5 in the first couple of days and didn't like it, consider trying it out again now or in a few days, as it might be much better.

My impression of GPT-5

This was an extremely anticipated release. Literally the whole AI bubble waited for it and watched closely. It's been 2 years since GPT-4, and people expected something extraordinary. Me too.

I raised my expectations for GPT-5 in the past few months - hoping that it would basically be "o4" but under a new name. And I expected a capability jump similar to the jump from o1 to o3.

I was also watching the whole rollout extremely closely and had tried out GPT-5 before the official release for a few days. First, when it was being tested on LMArena under the codenames "Zenith" and "Summit", and another time when it was available on Perplexity due to a bug.

I didn't try it out heavily on real tasks in those days, but I still sent many prompts for testing purposes. And I had a "taste" of it at that time. It felt similar to o3 in vibe, but smarter, more precise, and just generally better. My expectations rose again after trying it out there. I was almost sure it would be an "o4" moment.

But then the day of the release came, and I was watching the livestream. It was boring. I turned it off halfway through. And I didn't even try GPT-5 that day and went to sleep. I was already disappointed with the boring presentation.

The next day I was scrolling X and looking at feedback from other people, and it was mostly bad. People said it was either on par with o3, or a step down.

Then I finally tried it out. And it actually felt better than o3. And GPT-5 pro felt better than o3 pro. I still can't exactly say how much better they are, but it's definitely noticeable and significant in many scenarios.

It's hard to notice a difference in simple casual chats, but once you give it something complex or put it in an agentic environment - you'll see how it just does a better job than o3 could, and in many cases - much better than any other model could.

It also translates to agentic coding. For a whole month beforehand I was extensively using Claude 4 Opus for coding in Claude Code full-time, and it was great. I liked that model and its taste. It was nice coding with it. But honestly, it was pissing me off very often.

And so I tried downloading the Codex CLI with GPT-5 inside. The UX of the CLI itself is poor compared to Claude Code at the moment. It is not that developer-friendly. But after trying to code with GPT-5 the same way I did with Opus, I started to notice how GPT-5 was just better.

It's not always about the quality of the code, and definitely not about the speed. My first impression is that it not only writes better code overall, but that it's much better at instruction following and tool calling. Those are the things that people liked the most about Claude models. And I liked that too. And many people thought that no model would match Claude in these metrics anytime soon.

The thing is that GPT-5 just follows your instructions extremely precisely. And it doesn't do things you don't ask it to do. Claude was pissing me off so much by starting to go off track from instructions in long coding sessions, or even in simple queries when it just did something I did not ask it to do. GPT-5 is just better in this regard. Sometimes it follows instructions so well that you understand that your instructions were bad.

And it works so well with long context. I can mention something once early in a coding session, and then I just see how it still remembers, references, and follows that for so long. Opus was missing those things very often, especially in long sessions.

It might sound like too much ass-licking for OpenAI, but that's my honest experience with GPT-5. I was sceptical too, especially after seeing that boring livestream and seeing so much hate on X. But after trying all of it out myself, I was really amazed. Is it "o4" level? I'm not sure. More like o3.5.

The Complexity Threshold of AI

We see dozens of new LLMs heavily tuned for software engineering tasks, and they're becoming very good at it, very quickly. As models evolved, I started using them more and more for writing code, eventually reaching a point where I almost completely stopped writing code myself. The last time I wrote code manually (or rather, with AI-assisted tab completions) was around four months ago. However, once tasks become larger and more complex, these models quickly become inefficient. They seem to have a certain complexity threshold, beyond which their efficiency rapidly declines.

I was mostly using AI either to quickly take projects "from 0 to 1" by iterating on MVPs, or to build small Python scripts for working with LLMs and data. About a week ago, I needed to rapidly build another MVP while iterating on ideas, so I used Claude Code and completed the whole thing within a single day. I wanted to keep developing it, but the code became so messy that changing or adding anything was nearly impossible. Even minor adjustments, like updating the UI, caused other parts to break. At that point, I decided I was done with this MVP and needed to re-implement everything from scratch with better structure and architecture.

When I started the second implementation attempt, with my "plan" ready, I gave it to Claude Code and watched it fail for hours. It was producing code, but it didn't match my vision and wasn't working as expected. Many architectural and code-level issues remained. I tried slightly adjusting the plan and re-implementing everything multiple times over the span of three days, but it didn't help. After three unsuccessful attempts, I almost lost hope.

But then I decided to spend more time refining the specification itself before starting the implementation. I spent two days writing and iteratively refining the specification through feedback loops with the smartest models, while also giving my own feedback on each part. It covered almost everything needed for implementation, from high-level architecture to logic for specific use cases and solutions for the hardest implementation parts that I struggled with the most. Suddenly, when I gave this new specification to the agent and started slowly implementing things one by one, it just worked.

I told the agent to implement the next thing, waited 10 minutes, tested the implementation, asked it to fix any issues, and then moved to the next step. A few times during those two days, I also asked another agent to carefully read the entire codebase and strictly compare it against the specification, then passed its feedback back to the first agent to resolve any differences. After about two days, the project was mostly finished. It still has some rough edges (which are easy to address), and I haven't thoroughly tested everything yet (I even decided not to write any automated tests at all at this stage), but all the core functionality just worked. When I asked Claude to change something, it usually did so accurately, without breaking other parts.

The thing is that AI is much better at following instructions than at coming up with practical stuff on its own. Smarter models can handle more complex tasks independently, but in larger projects, their limit of "acceptable complexity per step" is lower. Therefore, when working on a bigger and more complex project, it's important to keep all individual steps at the same level of complexity and depth as you would when working on smaller projects. The complexity of each individual step matters more than the complexity of the whole project.

✍️ New Blog Post: "Billions of Tokens Later: Scaling LLM Fuzzing in Practice"

I've spent months running LLM-powered fuzzing at production scale—processing billions of tokens, discovering practical scaling laws, and developing effective deduplication strategies. Here’s what I learned along the way:

https://gusarich.com/blog/billions-of-tokens-later/