TL;DR: AI agents can write incredible code, but they can also write absolute garbage. Because there is no industry standard yet for managing agent skills or running evaluations (evals), developers must rely on collective, peer-to-peer learning at events like AI Native DevCon to establish best practices.
Remember the early days of Docker? Before Kubernetes won the container wars, we were all just trial-and-erroring our way through configuration files, desperately trying to figure out how to orchestrate microservices without blowing up our staging environments.
That is exactly where we are with AI agents today.
Our entire profession is being flipped on its head. In the last year, AI agents have become incredibly capable of writing software. But whether they write clean, maintainable systems or compile a mountain of unmitigated technical debt depends entirely on how we use them. The reality? Nobody has all the answers yet. We are collectively designing the playbook in real time.
How do you make AI agents write good code instead of garbage?
To make AI agents write high-quality code, you must move away from generic prompting and instead build constrained environments with modular skills and robust evaluation frameworks. Without these boundaries, an agent acts like a junior developer with infinite speed and zero oversight, quickly producing chaotic boilerplate.
Think of an AI agent as a highly capable but radically unpredictable contractor. If you give them a vague ticket and unrestricted access to your codebase, they will make a mess. Instead, we have to transition to an architectural mindset where we design:
- Strict Context Windows: Giving the agent only the exact files and APIs it needs.
- Sandboxed Runtimes: Allowing the agent to run, test, and break code in an isolated environment before it ever hits a pull request.
- Deterministic Guardrails: Forcing the agent's output to conform to strict type definitions and linting rules.
What is the difference between AI agent skills and evals?
Agent skills are the specific, sandboxed tools and APIs an agent can execute to perform tasks, while evals (evaluations) are the automated test suites used to measure the accuracy and safety of the agent's output. Think of skills as what the agent is capable of doing, and evals as the grading rubric that proves they did it safely and correctly.
When we build agentic workflows, we generally break the architecture down into three core concepts:
| Concept |
Definition |
Production Challenge |
| Agent Skills |
Modular, pre-defined functions an agent can call (e.g., "read_file", "execute_test", "search_db"). |
Restricting scope so agents do not execute destructive or runaway operations. |
| Evals (Evaluations) |
Automated testing frameworks designed to score LLM outputs for correctness, security, and style. |
Creating reliable, deterministic assertions for non-deterministic model outputs. |
| Dark Factories |
Fully autonomous, end-to-end software delivery pipelines where a ticket goes in and a verified PR comes out. |
Eliminating the need for human code review while maintaining 100% system trust. |
Why is peer-to-peer learning essential for AI native development?
Because AI native engineering is shifting daily, no single documentation page, tutorial, or textbook has the "correct" answers yet. Sharing hard-won production secrets—like how to handle agent drift or structure evals—with other developers is the only way to establish reliable industry standards.
I recently attended the London AI Native DevCon and spent time working the "Agent Bar," chatting with engineers and troubleshooting their setups. The disparity in where teams are at is wild. Some developers are brand new to the concept of agentic skills. On the exact same day, I talked to an engineer who had already implemented a fully automated "dark factory" where a ticket comes in, the agent writes the feature, runs the tests, and deploys it end-to-end.
That developer didn't want to talk about basic prompting; they wanted to discuss advanced eval strategies. This is why getting together in the same room—or the same virtual chat—is so valuable. We have to learn from each other's experiments.
If you want to get involved, the next AI Native DevCon is happening in New York this November. You can attend in person or stream the entire event virtually for free.
If you want to head to New York, use my discount code DOOGA30 to get 30% off. If you are heading over with your team, drop me a DM and I can grab you 50% off.
FAQ: Navigating the AI Agent Landscape
What is an AI "dark factory" in software engineering?
An AI dark factory is a fully automated software development pipeline where human intervention is minimized or entirely removed. A user story or bug ticket is fed into the system, and a network of coordinated AI agents autonomously writes the code, runs the test suite, fixes its own errors, and submits a verified deployment.
How do you write reliable evals for non-deterministic AI code?
Writing evals requires shifting from exact-match assertions to semantic and functional testing. Instead of checking if the code looks exactly like a reference solution, your eval framework should execute the generated code in a secure sandbox, verify that it passes unit tests, analyze it with static analysis tools (like AST parsers), and run LLM-as-a-judge prompts to grade code quality.
What are the best resources for learning AI agent architecture?
Because the field is changing rapidly, the best resources are active developer communities, open-source agent frameworks (like LangChain, AutoGen, or CrewAI), and peer-to-peer technical conferences like AI Native DevCon where engineers share real-world production post-mortems.
