Building AI Wasn't the Goal. Understanding It Was.

When I first started exploring Generative AI, my instinct was the same as everyone else's. Pick an application, connect a language model to it, and start building.

I decided to build a customer support ticket classifier. The idea was intentionally simple. A customer writes, "My battery drains within an hour," and the model predicts "Battery Issue." It felt like the perfect project because it was practical, measurable, and small enough to build without getting overwhelmed by infrastructure.

At the time, I thought the interesting part of the project would be building the application itself.

I couldn't have been more wrong.

The application stopped being the interesting part surprisingly quickly. What fascinated me instead were the questions that kept appearing every time I tried to move forward. Why did the model classify some tickets correctly but fail on others? Would changing the prompt actually improve performance, or was I simply convincing myself it did? Would adding examples make a measurable difference? If retrieval improved one category of tickets, would it help every category or only specific ones? More importantly, how would I even know whether an improvement was real?

Without realizing it, I had stopped thinking like someone building software and started thinking like someone designing experiments.

The Project Changed Before the Code Did

Originally, the repository was called Customer Support AI. It described exactly what I thought I was building: a small AI application for automating ticket classification.

The more I planned the project, however, the more obvious it became that customer support wasn't actually the problem I wanted to solve. It was simply a convenient dataset that allowed me to ask questions about AI systems.

That realization completely changed the direction of the project.

Instead of building a single-purpose application, I started designing something that could answer a different research question every week. Customer support became the first laboratory rather than the final product. The project gradually evolved into what I now call AI Product Lab, an experimentation platform where every experiment is designed to answer one specific question about how AI systems behave.

The objective is no longer to ship another chatbot. The objective is to replace assumptions with evidence.

Asking Better Questions

One lesson became obvious very early. Most AI projects begin with a solution. I wanted mine to begin with a question.

• Can a local language model classify customer support tickets?

• Does prompt engineering actually improve accuracy?

• How many examples are useful before few-shot learning stops helping?

• What information do embeddings really capture?

• Does Retrieval-Augmented Generation always improve results, or can it sometimes make them worse?

• Can confidence scores actually be trusted when making automated decisions?

Every one of these questions can be answered experimentally, but only if the experiment itself is designed carefully. That became the philosophy behind the entire project.

One Variable at a Time

One of the easiest ways to fool yourself while experimenting with AI is to change several things simultaneously. A different model, a different prompt, a different dataset, and a different evaluation method might all produce a better result, but you have no idea which change was actually responsible.

I wanted to avoid that from the beginning.

Every experiment in AI Product Lab changes only one variable while everything else remains constant. That makes improvements measurable instead of anecdotal.

Rather than saying, "I changed a few things and the model became better," I want to be able to say something much more specific. For example, adding retrieval improved warranty classification by a measurable amount but had almost no effect on shipping-related tickets. That kind of conclusion is far more valuable because it explains not just what improved, but why.

Building a Framework Instead of Scripts

As the project evolved, I realized I wasn't building individual Python scripts anymore. I was building an experimentation framework.

Every experiment now follows exactly the same lifecycle. A dataset is prepared, passed through preprocessing, evaluated by a model, and measured using consistent metrics. Predictions are analyzed, failures are categorized, and the results are translated into product insights before deciding what the next experiment should investigate.

That consistency is important because it allows every future experiment to be compared against the previous one. Instead of isolated projects that are forgotten after a weekend, each experiment becomes another step in understanding how AI systems actually work.

Why I'm Not Starting With RAG

Like many people learning modern AI, I was eager to jump straight into Retrieval-Augmented Generation. It seemed like the most exciting topic, and almost every tutorial suggested that RAG was the next logical step.

The more I thought about it, the more I realized I would be skipping over concepts that deserved to be understood first.

Before experimenting with retrieval, I wanted to understand baseline classification. Before retrieval came prompt engineering. Before retrieval came embeddings, semantic search, clustering, and evaluation. RAG shouldn't feel like magic. It should feel like the natural outcome of understanding everything that comes before it.

That realization completely changed how I planned the roadmap. Instead of treating RAG as the destination, I began treating it as just another experiment in a much larger journey.

Accuracy Isn't the Most Interesting Number

One of the biggest changes in my thinking came when I stopped treating accuracy as the final goal.

It's easy to celebrate a model that achieves eighty-nine percent accuracy. Most benchmarks stop there.

I became much more interested in the remaining eleven percent.

Why did those predictions fail? Was the ticket ambiguous? Did it contain multiple intents? Was the prompt poorly designed? Did the retrieval step return irrelevant information? Was the model missing context that simply wasn't available?

Those questions turned out to be far more valuable than the overall accuracy itself. Every incorrect prediction reveals something about the system that a correct prediction never can. Failure analysis has gradually become one of the most important parts of the entire project because failures often point directly toward the next experiment.

Product Thinking Matters More Than Model Size

Perhaps the biggest lesson so far has very little to do with language models.

When an AI system performs poorly, the natural reaction is to replace it with a larger or newer model. That certainly solves some problems, but it doesn't solve all of them.

Imagine a model consistently struggles with warranty-related tickets. The obvious solution is to upgrade to a stronger model. Another solution, however, is to retrieve the customer's order history before asking the model to make a decision. Suddenly the problem isn't about choosing a better model at all. It's about designing a better product workflow.

That distinction fundamentally changed how I think about AI products. Models perform reasoning. Products provide context. Good AI systems usually need both.

Looking Ahead

At the time of writing this, AI Product Lab is still in its early stages. Most of the work so far has gone into designing the experimentation framework rather than rushing into implementation. That might seem slower than immediately building features, but I believe it will pay off over time.

The roadmap is intentionally incremental. It begins with establishing a baseline using local language models, then gradually introduces prompt engineering, few-shot learning, model benchmarking, embeddings, semantic search, Retrieval-Augmented Generation, context engineering, explainability, confidence calibration, and human-in-the-loop workflows. Each experiment builds upon the previous one while introducing exactly one new idea.

Eventually, I'd like to apply the same framework beyond customer support. Product feedback, feature requests, customer interviews, PRD reviews, resume screening, and product analytics all present similar opportunities to study how AI systems behave under different conditions. The customer support dataset simply happens to be where the journey begins.

Final Thoughts

When I started this project, I believed I was building another AI application.

Today, I think of it very differently.

The application is simply a vehicle for asking better questions. What I'm really building is a way to understand how AI systems behave, why they fail, and what product decisions can make them more reliable.

I don't expect every experiment to produce dramatic improvements. In fact, many of them probably won't. But even a failed experiment teaches something valuable if it's designed carefully and evaluated honestly.

For me, that's what AI Product Lab has become. Not a customer support chatbot, but a place to explore AI one question at a time.