Some background

I studied CS and was part of the few batches that experienced the whole AI transition within my college lifetime. In year 2, I was coding a 2D RPG game from scratch without AI - building the game engine, tracing bugs through the physics engine, and in year 4, I was running 6 Claude Code sessions in parallel building ren.

I have some thoughts on AI, but I am still trying to consolidate them, so maybe I’ll start by focusing more on the practical side of things from the perspective of someone who has been using it quite extensively while building a startup. It’s a dump so it might be a bit all over the place, but I hope it can be useful to someone.

Is AI going to replace software engineers?

A heuristic I have been using is: can a non-technical Claude Code power user that I know do the things that I do at ren?

If the answer is yes, that would be quite worrying, because it either means that software engineering is solved, or that I have a skill issue.

But if the answer is no, then why? What are the things that I do that a non-technical power user can’t do?

When “vibe coders” confront me with such questions, as much as I can throw around terms like “design patterns”, “software architecture”, “maintainability”, it’s very difficult to put across the value of these things concretely.

“But I can just ask Claude Code to suggest the best design pattern for this problem, and it will do it for me, right?”

I think what differentiates an engineer and a non-technical power user is one’s ability to internalize the tradeoffs and pushback if necessary. But instead of trying to explain in abstract terms, what I find to be more effective is to show specific examples where my input as an engineer has altered the course of Claude Code (in a good way).

And to have this “bank” of examples, I created a skill that I call pushback vault. Every time I have to push back on Claude Code, an entry is automatically written to the vault, which contains:

  • context of the problem at hand
  • verbatim quote of the user message that “pushed back” on Claude Code
  • what changed as a result

The most recent example is where we are working on an experimental feature where we want to give interns different permissions to our admin dashboard. Claude Code suggested we add boolean flags for the permissions, and I pushed back to use JSON. Here is the entry:

## Context

We were changing Rubric manage access so product interns could view Overview and Insights without getting access to content, grants, or other admin actions. I started implementing fixed boolean columns on `PrepUser` for each capability, while preserving `Teacher.isAdmin` as a superuser fallback.

## The pushback(s)

### 1. Match the existing permission model

> "do we want to set in stone the different permissions as db columns, or deal with it the way we deal with the apikey and organisation permissions, where we use json?"

I had moved too quickly from capability design into concrete boolean columns. The user pointed out that the repo already has a precedent for flexible permission arrays in org roles and API key scopes, which is the more appropriate model for a growing Rubric permission surface.

The state of the codebase is a cumulation of many such “small” decisions. It requires technical knowledge, a good mental model of the core system, and intuition about tradeoffs to maintain the integrity of the codebase, especially with AI.

If I ever reach a point where there are no longer any pushbacks, one of these must be true:

  • I have figured out how to break tasks down into small enough problems that I can completely trust Claude Code’s judgment (in which case the act of breaking down the problem is my value add as an engineer)
  • I have become lazy, which means the entropy of the codebase is bound to increase (that is a sign to slow down and start focusing on technical debt)
  • I have reached a skill ceiling where I can no longer recognize the problems that need to be pushed back on (which is a sign that I need to start learning more and leveling up my skills)
  • Software engineering is solved

As an engineer, a good mental model to have is: “is what I’m doing now replaceable by another Claude Code tab?”

If the answer is no, you don’t have to worry too much (yet). If the answer is yes, I’m not sure if I’m in a position to give advice - but it might be worth reflecting and seeking advice elsewhere.

Sidenote: I also have a skill that auto runs in the background to provide bite-sized and non-trivial SWE concepts to mitigate some of the brain rot from using Claude.

On productivity with AI

I scroll TikTok (a lot), and I’ve been seeing a lot of comments calling AI “overhyped” and claiming it provides “no value”. I suspect that a lot of these comments are from people who did not go beyond (or found a use case for) entering prompts into ChatGPT’s web interface.

Maybe I’m “AI pilled”, but as someone running a small startup, I realised that we have been able to do so much more with the same amount of resources, and I’m not just talking about coding.

For instance, our accounting bookkeeping is hosted on a private GitHub repo. Because we use beancount, entries are just text files, which means we can just say “Justin paid $20 for Vercel this month” and Claude Code can:

  • interpret the human language, translate it into double entry bookkeeping format, and add it to the ledger
  • run beancount commands to check if the ledger is balanced
  • version control using git

We have Openclaw set up where we could just send the receipt to a Telegram channel, and Openclaw will help us commit the change to the ledger.

ren finance

I’ll go as far as to say that this is more “correct” and “maintainable” than having an Excel sheet or accounting software for startups at our scale.

Likewise, we use a variant of Andrej Karpathy’s LLM Wiki for our internal docs, which also allows us to query it in natural language on Telegram through Openclaw.

We built a crawler that runs in the background for hours or days, scraping and indexing past year papers, and we have open sourced it as rex.

We taught our non-technical marketing interns to create custom skills for marketing and operations tasks, which has been a huge force multiplier for us. We created a brand kit to make it easy to create documents using LaTeX / HTML to PDF.

For the tech interns, they just need to clone a “master” repo, which contains all the custom skills we created, including skills to clone other repos, install dependencies, set up the environment (like creating the local database), and it even ends off with “check Bitwarden for the env variables”.

Using AI has reduced the cognitive overhead of doing a lot of ad hoc tasks (who wants to waste time maintaining an Excel sheet for bookkeeping?) and allowed us to focus our mental energy on the things we care about.

On AI coding workflow

The bottleneck of AI coding is not the speed of writing code, but verifying correctness.

So a substantial effort should be put into building the correct harness, such that we can improve the agents’ ability to run long-running tasks without human intervention without undermining the quality of the output.

Treat CLI as first class citizen

For simple applications, AI can verify correctness trivially - start the server, curl the endpoints, done.

The harder case is when correctness depends on side effects. Consider a feature where a user uploads a PDF on the frontend: the backend uploads it to S3, writes metadata to the DB, and enqueues a job. A worker then picks up that job, downloads the file, fetches the metadata, and does complex processing.

To test the correctness of the “complex processing”, you’d have to: go to the client, upload the file, wait, copy the error to Claude Code, repeat. That loop has too much human-in-the-middle friction for AI to iterate quickly.

The fix is to expose a CLI wrapper over the core logic using the adapter pattern or otherwise, so the worker’s domain logic can be invoked directly, without the queue, S3, or frontend:

$ python -m some_module.some_logic --file some_file.pdf --metadata some_metadata.json

This is the same principle as designing server code to be injectable for unit tests, except we are designing it to be injectable for the CLI too. AI can then run the command, observe the output, and iterate without any human interaction in the loop. CLI-Anything might be a good start to convert some of your side projects to CLI.

One example we have with this pattern is rex, where domain logic is shared between the API and the CLI, except in this case the app is probably trivial enough for us to not need a CLI.

server architecture

Likewise, when dealing with infra (e.g. provisioning servers or DBs in AWS), I prefer to use CLI over GUI.

Debugging

It’s crazy how much AI agents can help with debugging (without human intervention) when you give them the correct tools.

We use AI to debug production issues by:

  • giving Claude Code a read-only role for our prod DB
  • giving Claude Code access to our AWS CLI with read-only access to CloudWatch and S3, where it can query our backend logs
  • integrating Sentry MCP (fallback to Vercel CLI) to query Next.js error logs and events

All these are noted down in a custom skill.

For local development:

  • full psql access to our local DB
  • Chrome DevTools MCP to spawn headless browsers for frontend testing (we bypass and hardcode auth for local development)
  • a curated directory of resources (since we deal a lot with grading, we need sample PDFs for testing), and making our backend CLI-friendly allows for this

We also have a custom skill for local debugging.

For instance, this is one example of a debugging workflow:

  1. Bug is reported, human invokes custom skill
  2. Claude Code starts investigating. If it is a frontend or Next.js backend bug, it filters Sentry or Vercel logs in the past 1 hour. If it is our backend services bug, it uses AWS CLI to pull backend logs
  3. Based on the information, it traverses the codebase to find the cause
  4. If needed, it psqls into prod DB using the read-only role to find exact data with the bug, and downloads the exact files from S3
  5. It spawns a headless browser using Chrome DevTools MCP if it’s a frontend bug, and injects the exact data or file to our local backend to recreate the bug
  6. Once the bug is recreated and identified, it uses red-green TDD to make the code changes
  7. It verifies the fix using Chrome DevTools for frontend or CLI for backend
  8. It commits, pushes and creates a PR using the GitHub CLI
  9. Human reviews and merges

As we can see, the human intervenes at steps 1 and 9, and steps 2 to 8 can run autonomously in loops by agents.

Building with AI

Coding agents

I’ve tried Claude Code, Codex, Opencode (with Kimi K2.5), Pi (with Kimi K2.5), Cursor, Gemini (now Antigravity).

Anthropic models used to be the SOTA, but I think OpenAI models are equivalent now. We have a 5x Claude Code subscription but if we ever need to increase usage, I would get a 5x Claude and a 5x Codex. The Codex harness is way less buggy and the bug report I made was responded to in one day. On the other hand, the Claude Code bug report I was monitoring has not been addressed after almost a year, and it was even marked as complete!

I stopped using Cursor a while ago. I only use Opencode and Pi when I have free credits to burn but they look promising. Gemini was bad. I will try and form my opinions on Antigravity soon.

Skills and guardrails

For non-trivial features, we use the superpowers skill. gstack is an obnoxious bloatware.

For UI/UX, apart from the commonly known frontend design skill, I find impeccable to be quite useful. For instance, you can run the npx impeccable detect command to detect AI slop.

I am still experimenting with this, but we use CLAUDE.md to enforce certain standards, and include patterns where we keep having to correct AI mistakes in lessons.md. The default models are very smart - don’t oversaturate them with trivial information like tech stack, patterns, etc. since those can mostly be inferred from your codebase.

For reference, here are some rules we have included in our frontend CLAUDE.md and lessons.md:

  • other than type linting, we run yarn impeccable, which runs npx impeccable detect to flag out potential cases of AI slop
  • we want features to be localized - e.g. UI, logic, types, tests for a specific feature or page should be in the same directory. The point is to clearly differentiate the “leaf nodes” as referenced in Anthropic’s talk
  • never use any for TypeScript
  • JSON objects that cross the frontend to backend boundary (e.g. DB columns) need to have contract tests
  • user-facing copy (e.g. tooltips, labels, error messages) are written for [insert target audience], since AI has a tendency to add very descriptive text using developer vocabulary. For example, we have features where we save to local storage first before saving to DB. The AI has a tendency to write toasts like “Changes persisted to database.” instead of “Your changes are saved.”
  • domain-specific vocabularies
  • validate with Zod instead of typecasting
  • prefetch on hover, and other rules
  • enforce permission gates server-side instead of client-side
  • use the specified source of truth for routes or S3 key builder
  • prefer parallel execution for independent async operations

For backend:

  • use Pydantic models or TypedDicts instead of Dict[str, Any]
  • modularise to make injection for testing or CLI easier
  • some framework-specific quirks like eager-loading when using SQLAlchemy

Evals and goal-driven tasks

I’m sure some of you have heard of AlphaEvolve or OpenEvolve. The reason they work is because there is a quantifiable metric for “good” and “better”.

Surprisingly, AI agents are quite good at solving optimisation problems, taking inspiration from OpenEvolve and Andrej Karpathy’s autoresearch. The trick is identifying such problems which we can turn into optimisation problems.

Suppose I want to create a service that takes in a PDF with handwritten text and extracts the words or sentences that are cancelled out. Like any other GPT wrapper, I would probably call OpenAI’s API with something like:

“Hey ChatGPT, extract the cancelled out words, make no mistakes.”

alongside the essay PDF.

But the AI would probably perform badly.

Interestingly, we can apply principles of machine learning to prompt engineering (I’m not kidding).

The first step is creating the dataset. We can manually look through a few handwritten PDFs and extract the words that are struck through, and put them in a structured format (JSON or YAML). This will make up our gold standard.

Depending on the use case, we can define an evaluation criteria. For instance, if we want to grade the PDFs, false positives and false negatives are probably equally bad, so we could use F1 score as the evaluation metric.

The AI agent can then call the service with the PDFs, compare the LLM’s identified crossed-out words against the gold standard, and calculate the score accordingly.

You can then use the /goal command to run it with the necessary guardrails - for instance:

  • giving it the freedom to modify the prompt (without hardcoding)
  • giving it the freedom to modify the parameters, such as setting the detail field, verbosity, reasoning, and adding a code interpreter
  • giving it the freedom to modify the architecture, add phases, add tools to zoom in, etc.
  • giving it the freedom to change models and providers (just make sure you have the API keys of the other providers in your .env)
  • giving it web search ability (e.g. using Firecrawl) to do research for new experiment ideas

Make sure it writes the different experiments and findings to a markdown file.

It will then be able to run for hours, doing research, conducting experiments, running evals, and exploring and hill climbing until it finds the most optimal prompt or parameters to extract crossed-out words from PDFs.

Of course, just like how you would treat machine learning tasks, you would have to set proper guardrails to prevent overfitting, such as leaking gold standard details into the prompt.

End of sentence

I’m still learning how to make use of AI effectively, but these are what I have for now. As for my actual thoughts on AI, I’m still thinking - so I might have to write another piece.