Our Blog

A synthetic user is a constrained AI decision agent defined by twelve fields, from functional role and context to assumptions and abandonment rules.

In the previous post I built an early, working implementation, and the next question was whether the same rules could hold up in a repeatable, automated test.

This post is that next step: how I turned the framework into a Claude Code plugin, and the technical decisions behind adapting methods designed for people into something an AI can execute without cheating.

Why “find the usability issues” is not enough

Give a model a URL and ask it to “find the usability issues.” It works halfway. And the “halfway” is the interesting part, It gives you a generic list, correct in the abstract, useless in practice.

A usability issue matters because of who encounters it and under what conditions.

Using an app from bed is not the same as using it on a factory floor. Urgency changes, lighting changes, attention changes, previous knowledge changes. The same confusing button can be irrelevant to a power user and an abandonment point for an operator wearing gloves.

The whole design comes from that observation: the AI does not evaluate the interface. It acts as a specific person in front of the interface.

The person brings the context with them. And the context turns a list of defects into a list of priorities.

Anatomy of a simulation

An orchestrator controls the browser through Playwright MCP. It reads each screen as an accessibility snapshot: text, roles, states, no guessing pixels. Then it acts on specific elements.

The decision on each screen is made by an isolated subagent, which returns a JSON for each step:

{

  "action": "...",

  "clarityLevel": "High|Medium|Low",

  "doubtDetected": true,

  "reason": "...",

  "abandoned": false,

  "estimatedTimeSeconds": 40,

  "emotionalState": "...",

  "memory": "..."

}

Two rules make this look more like a person and less like an oracle.

1. The evaluator never sees the end.

The evaluator receives one screen at a time, without knowing how many are left or what comes next in the flow.

If the interface leaves room for a mistake, the synthetic user makes the mistake. It clicks where a person would click, not where it is convenient to click in order to complete the test. This is where the framework’s forbidden assumptions live. The agent cannot assume backend logic or mentally complete what the screen does not show.

2. Emotion is memory, not decoration.

The memory field travels from one step to the next. The emotional state is inherited and accumulates. A frustration +1 persists. This detects something that is structurally invisible to any test that evaluates screens separately.

Screen five does not necessarily fail because of screen five. It fails because the user gets there with accumulated frustration.

Evaluated alone, that screen passes. Evaluated by someone carrying three doubts and one broken promise, it triggers abandonment. In the first post, I wrote that doubt is not failure. It is the signal that reveals structural friction.

Emotional memory is that idea turned into architecture.

Eight subagents, one job each

Each subagent gets a clean context. It knows the minimum required to do its job.

That ignorance is deliberate.

The agent acting as the user does not know what the orchestrator knows. It cannot compensate for bad design with knowledge a real person would not have.

Subagent

What it does

Adapting a human test: the heuristic evaluation

A textbook heuristic evaluation uses three to five human evaluators because each human finds different problems.

My first experiment was literal, and it went meh.

I iterated until I reached two synthetic detection runs with different agents, coverage was extremely high, but it exposed another problem: an unmanageable list. Dozens of valid issues, very few important ones.

The final design separates those two jobs.

1. An expert finds violations.

Based on Nielsen’s literature, an expert goes through each screen and is forced to produce a verdict for every heuristic: 

• Violation
• Clean
• Not observable

Each verdict includes textual evidence from the snapshot, forced enumeration breaks the habit of reporting only the things that stand out.

2. Three synthetic personas decide what matters based on what they bring with them: context, emotions, urgency, and constraints.

Three synthetic personas are generated according to the business being evaluated: 

• power user
• average user
• low digital literacy

They score the findings without seeing the expert’s conclusions. The same issue can matter very differently depending on what each persona brings to it.

The formula is business impact × usability impact, with agreement between personas as the tiebreaker.

This keeps issue detection and user impact as separate jobs: the expert identifies the violations, and the personas help determine which ones deserve attention first.

Three modes, and a tool for building users

The plugin currently has three modes.

simulation-run (custom)

You build a profile field by field in the Synthetic User Builder, the tool I built to materialize the framework.

First come the attributes: 

• Role in relation to the product
• Boundaries
• Initial emotional state
• Context
• Forbidden assumption

Only after that, and separately, comes the task.

The profile describes how someone decides, never what they have to do. That is why the same profile can be reused across tests.

simulation-auto (inferred)

You only give it the URL.

It researches the business, infers the typical roles, proposes users with tasks, and you adjust that proposal in natural language before anything runs.

heuristic-test (inspection)

The heuristic test described above, for one screen, one flow, or the entire site.

Everything run becomes a file

Every run leaves Markdown artifacts inside the project:

user-simulation-tests/

├── simulation/

│   ├── profiles/    ← users: the .md used for simulation + a .builder.json

│   │                   that can be imported back into the Builder and edited manually

│   └── results/     ← one report per run + the consolidated report from auto mode

└── heuristic/

    ├── personas/    ← the 3 raters + business research, reused across runs

    └── results/     ← reports with the prioritized findings table

Simulation reports include the full step by step flow, the emotional arc, risks, and a single “Fix this first.”

The consolidated report classifies findings by convergence: did one user suffer from this, or did all of them?

The decision to keep everything as accumulating .md files is strategic.

These are different runs, using different lenses, that can be analyzed together later, crossing heuristic violations with simulated emotions answers something no individual test gives us:

Of everything that is wrong, what actually matters?

Models and costs

What worked for me for the synthesis subagents:

• For reports, consolidation, and the heuristic expert, the best available model makes sense. That is where the judgment lives.
• For the screen evaluator, a medium and fast model is enough. There are many short, constrained calls, and the profile already restricts the decision.
• The raters are the lightest case.

A complete run consumes between 100k and 400k tokens, depending on the model and mode, in around 20 minutes.

That is the cost of a test that previously required coordinating the schedules of three professionals, and that can now run against every iteration of the product.

See it in action

Here's a complete run against our site, kzsoftworks.com: a skeptical "Business Leader" profile, five live browser steps, and a full Markdown audit in under three minutes that names the exact moment the executive persona lost trust.

It is still early, but it already runs

Every rule in the framework became an architectural constraint: clean context, one screen at a time, emotional memory, forbidden assumptions.

The plugin is open source: github.com/PabloManzoni/user-simulation.

Three commands, and the inferred mode only needs your URL.

If you try it and your synthetic user abandons on screen three, you already know what it means:

It is not failure. It is the signal.

‍

Generative UI assembles the interface around what each user is trying to do, instead of showing everyone the same fixed screen. That flexibility comes with real considerations: keeping the experience consistent, secure, and easy to support once it's live. This post covers what generative UI is worth building for, what it costs, and how teams keep it under control.

Generative UI works best when the experience is dynamic, but the system behind it stays tightly controlled.

Start by defining which parts of the interface can change, which cannot, and what must be validated before anything reaches the user.

TL;DR

• Interfaces can adapt to user context, support more variations without designing every screen by hand, and reduce unnecessary steps in a workflow.
• The trade-offs include inconsistent experiences, unreliable or unsafe output, added latency and infrastructure cost, and harder analytics and debugging.
• Better prompting can reduce unwanted behavior, but it cannot guarantee reliability, security, or consistency. Those controls need to exist around the model: a stable interface shell, a closed component catalog, validation of model output, session-level logging, and model routing with fallback options.
• Every control introduces a trade-off. No architecture maximizes flexibility, reliability, privacy, performance, and cost at the same time.

What does generative UI make possible?

Interfaces that adapt to context

The interface can adapt to what a person is trying to do instead of relying only on a persona defined at design time. Steps can reorder or disappear based on intent. It can change how much information it shows and what it emphasizes. Copy can adapt to the user's locale and context instead of relying on literal translation.

More interface variations with less custom development

A small set of components can support many variations without designing each screen separately. The system can also support workflows the team did not design as individual screens, as long as the required components and actions already exist.

Fewer steps between intent and action

The interface can hide controls a task does not need, reducing the number of steps required to complete it. Generative UI can also help teams test different ways of presenting the same task. Whether that improves completion or conversion depends on the workflow.

What can go wrong with generative UI?

Experience consistency risks

When layouts change between users or sessions, they can break muscle memory and make support harder. They can also drift from the design system or disrupt accessibility patterns that depend on consistent structure.

Reliability and security risks

The system should not trust model output by default. A model can render a button that does nothing, display fabricated data in a component, or produce a state the team never tested. Prompt injection can push it toward components, content, or actions the system should not allow. Weak controls can expose sensitive data or allow actions and interface states the product should block.

Performance and infrastructure risks

A generative interface also inherits the model layer's latency, cost, and availability risks. Waiting on an LLM to generate a layout adds delay before a page renders. Each generation uses processing resources, and hosted models usually add usage-based cost. Relying on one provider also exposes your product to outages, API changes, price increases, and deprecations.

Analytics and debugging risks

Standard analytics often assume a fixed set of screens. Heatmaps and funnels become harder to compare when users see different layouts. Reproducing a bug also gets harder when you cannot reopen the exact screen the user saw.

How do you control these risks?

Prompts can reduce unwanted behavior, but they cannot enforce which components the system may render or which actions it may allow. Those limits need to be enforced in the architecture around the model.

What parts of a generative interface should remain fixed?

Keep global navigation, account and security controls, primary actions, critical transaction controls, and accessibility-critical structure fixed. Let the model modify only the content and controls that benefit from adaptation.

Fixed navigation preserves familiar interaction patterns. A stable structure also makes accessibility testing, branding, and support more predictable.

How do you stop generative UI from creating broken interfaces?

Do not let the model generate arbitrary UI code. Have it return structured configuration instead. The schema should specify the component, its data, and its position. Validate that output against a closed catalog before rendering it.

The model should not write HTML, CSS, or JavaScript or choose anything outside that catalog. This reduces invalid layouts and unsupported combinations. This is the declarative approach we covered in Part 1.

How should teams test and secure generative UI?

Treat model output as untrusted input. Validate it against the schema and component allowlist, sanitize content, and keep authorization outside the model.

Add content security policies and prompt-injection defenses based on what the model can access and what actions it can trigger. Pay particular attention to user-provided content, privileged actions, sensitive data, and external tools.

Limit valid component combinations, then use visual regression and property-based tests to exercise unexpected inputs and edge cases.

Minimize sensitive data sent to the model. Mask or anonymize it before generation when the task does not require the original values.

How do you monitor a UI that looks different for every user?

Record enough context to reconstruct each generated interface. That includes detected intent, model version, generated configuration, rendered components, task completion, and errors, all tied to the session.

That record lets teams segment analytics by generated experience and reconstruct what a user saw during a specific session.

How do you control latency, cost, and outages?

Cache reusable results where freshness and privacy allow. Show a skeleton layout immediately and stream the rest in. Route simpler requests to smaller or local models, and reserve larger ones for complex requests. Put providers behind the same integration layer so you can switch models or fall back to a static experience during an outage.

What do these controls cost you?

Keeping more of the interface fixed protects consistency but limits personalization. Limiting combinations makes the system easier to test but reduces how much it can vary. Caching lowers cost, but cached output can go stale.

Running models locally can reduce how much sensitive data leaves your infrastructure, but it adds systems your team has to operate and maintain. Detailed session logs can make support easier, but they also create storage, retention, and privacy requirements.

No architecture maximizes flexibility, reliability, privacy, performance, and cost at once. You need to decide which trade-offs matter most for each workflow and design around them.

Somewhere between a prospect Googling your company and a prospect never visiting your site at all, a new kind of visitor showed up.

It doesn't click. It doesn't scroll. It doesn't show up in Google Analytics. But it scans your website, decides what matters, and quietly influences whether your business gets mentioned the next time someone asks ChatGPT, Perplexity, or Google's AI Overviews for a recommendation.

We had no real way to know what these AI bots were finding on our own site. So, before telling anyone else what to do about it, we built something to find out for ourselves.

The blind spot in your analytics

Google Analytics tracks human sessions, not server-side crawler activity. That's the blind spot. A person searches, sees a list of links, clicks one, lands on your site; that's the journey it was designed to track.

That journey is changing. Fewer people start their research by typing a query into Google and scanning ten blue links. Most of them are asking an AI assistant directly: "who are good software partners for X," "what's the best tool for Y," and trusting the shortlist it hands back. To build that answer, the AI first sent something to read the web on its behalf: a bot with a name like GPTBot, PerplexityBot, or ClaudeBot, crawling pages much like search engines have for decades.

None of that shows up in your dashboards. Those bot visits don't count as sessions, don't trigger conversion tracking, and don't appear anywhere you're already looking. If your site is hard for those bots to read, poorly structured, or quietly blocking them without anyone realizing it, you're not losing a ranking position. You're being left out of a conversation you never knew was happening. It's a new kind of competitive risk. Not "we got outranked," but "we were never in the running, and nothing told us."

That's the gap we set out to close, starting with our own site.

Are AI bots even visiting our site? We stopped guessing.

Inside our Innovation Hub, the group that experiments with new tools and workflows before we bring them into client work, someone asked a simple question: are AI bots even visiting our site? And if they are, what are they actually able to see?

Nobody could answer that with confidence. Not because it's a hard problem to reason about, but because the tool to answer it didn't exist among the tools we already had. So instead of guessing, or buying something built for someone else's website, we built a small internal dashboard for our own.

What we built: a dashboard that tracks AI bot visits

The idea is simple, even if getting there wasn't: a small piece of code sits quietly in front of our website and notes every time a known AI bot stops by. It records which one it was, which page it looked at, whether it got a clean response or hit an error, and how deep into the site it went.

Right now we're tracking bots from OpenAI (the ones behind ChatGPT), Anthropic (Claude), Perplexity, Google, Microsoft's Bing, Meta, and Apple. That list will keep growing. New AI crawlers show up faster than anyone can keep a definitive catalog.

All of that gets pulled into a dashboard the team can check the same way we'd check any other business metric: how much of the site is actually getting crawled, where bots are hitting dead ends, whether they're respecting the instructions we leave for them, and how that changes over time.

What the dashboard caught in the first two weeks

We didn't have to wait long to see the point of building this. Two things came up in the first few weeks alone.

The file we thought was working

An llms.txt is a simple file some AI models look for to understand what a site is about. Like a lot of sites getting ready for an AI-driven web, we added one, checked it was live, and moved on, assuming that box was checked.

The dashboard said otherwise. Weeks in, not a single bot had requested it.

So we went digging, and read that crawlers rely on robots.txt to know an llms.txt file exists in the first place, and ours didn't reference it. We added the missing line. Bots still weren't picking it up.

Third attempt: we added plain, visible links to the file in the site's header and footer, the same way we'd link to any other page. That's what did it. Two weeks of zero requests, and on the exact day we shipped that change, the file got six requests from five different AI companies.

The detail we only noticed because the dashboard breaks bots down by type: those six requests were all from indexer and training bots, the ones that crawl the web to build a general picture of it, not yet from retrieval bots, the ones that fetch a page in real time to answer someone's specific question right now. That's a useful distinction. It's the difference between "we're now on the map" and "we're being pulled up live," and it tells us what to check for next.

None of that would have surfaced anywhere else. Not in Analytics, not in Search Console. We would have gone on believing the file was doing its job, simply because we remembered adding it.

The high-value pages AI bots were quietly skipping

The second finding was less comforting: several of our most important pages, the ones describing what we actually do, were barely being crawled at all. Not blocked, not broken. Just quietly skipped by many bots.

We built a graphic on the dashboard specifically for this: crawl coverage per bot, broken down page by page. Now, instead of assuming coverage is even across the site, we can see exactly which high-value pages each AI bot is actually reading, and which ones it's ignoring.

We're still working on closing that gap. The first fix we tried didn't move things the way we expected, so for now the coverage graphic itself is doing the real work: telling us, page by page and bot by bot, whether the next attempt actually helps instead of just hoping it does.

Neither of these was something we could have reasoned our way into. We only found them because we were finally looking.

Before you optimize, measure

It's tempting to jump straight to fixes: restructure content, add an llms.txt file, rewrite pages to be more "AI-friendly." We did some of that too. But our own llms.txt sat unused for weeks and we had no idea, because we had nothing telling us otherwise. Without a baseline, you can do all the "right" things and still have no idea whether any of them worked.

Our approach here mirrors how we tend to approach any technology problem: understand what's actually happening before deciding what to change. It's a small dashboard, built quickly, answering one honest question. It's already paid for itself twice over, and we're still early.

We'll keep sharing what we find as the picture gets clearer. If you're curious what your own numbers might look like, that's a conversation we're happy to have.

A while ago we noticed something pretty common: everyone wanted to share more knowledge internally, but nobody wanted another heavy corporate ritual.

Internal talks usually start with good intentions and slowly disappear. They take time, preparation, and energy. And at some point people start feeling like they need to be experts before presenting anything.

So we tried the opposite.

15 minute talks.

Small topics.

Low pressure.

And one important rule: every session had to leave something useful behind. A tool, a workflow, an idea, a shortcut, a new way to approach a problem. Something people could actually use after the talk ended.

We didn’t want theory that went nowhere.

Somehow, that ended up working much better than we expected.

The idea was to reduce friction

Tiny Knowledge Bytes is intentionally simple:

• anyone can suggest topics
• anyone can end up presenting
• you don’t need to master the topic
• talks can come from experiments, client problems, tools or random discoveries
• sessions should leave something practical behind
• if nobody volunteers, the system picks someone for us

The goal was making knowledge sharing feel lightweight instead of exhausting.

Some of the best talks start with:

“I tried this yesterday and it was weird.”

The topic pool started growing on its own

Over time, topics started coming from everywhere.

Sometimes someone took a course and used a Tiny Knowledge Byte as a way to give something back to the team. Other times, a client problem triggered research into new tools, workflows or AI approaches.

A lot of sessions start from curiosity or necessity more than planning.

The pool slowly filled up with things like:

• Synthetic Users
• Google AI Studio
• Design.md
• Computer Vision
• MCP + Figma
• V0 workflows
• AI orchestration
• Figma plugins
• comparing AI tools using the same prompt

And honestly, the mix is part of what makes it interesting.

Sometimes a UX session drifts into Computer Vision. Sometimes someone technical shares a visual workflow that half the design team ends up adopting later.

There’s not much curation. It behaves more like a constant exploration system.

Then another problem appeared: choosing who presents

And this is where things became unnecessarily dramatic.

Nobody wanted to be “the person who chooses”. So we started adding absurd layers of randomness until we somehow ended up building a full internal app called 2FS.

Two Factor Sorteo.

Yes, it’s real.

The wheel proposes. The oracle decides.

The logic is simple.

First, a wheel picks someone.

Then a Magic 8 Ball decides whether destiny approves the selection.

If the oracle rejects the person, the process starts again.

That’s it.

The app accidentally became part of the learning loop too

2FS originally started as an excuse to experiment with:

• Claude Code
• Claude Design
• design systems
• editorial interfaces
• motion and microinteractions

Eventually those same explorations turned into future Tiny Knowledge Bytes.

The tool we used to select speakers started generating new topics itself.

The system started feeding itself

One of the most interesting side effects is that people started building things outside their usual role because of previous Tiny Knowledge Bytes.

2FS itself is a good example. A designer saw sessions about Claude tooling and AI workflows and thought:

“Maybe I can actually build this.”

What started as a ridiculous speaker selection tool became a real product experiment involving Claude Code, interface systems and interaction design.

Then it came back into the Tiny Knowledge Bytes circuit as a new talk.

That loop became surprisingly valuable:

someone learns something,

tries it,

builds something with it,

and eventually inspires someone else to do the same.

What ended up mattering most

Over time we realized knowledge sharing works much better when:

• it doesn’t require huge preparation
• it’s allowed to be imperfect
• it mixes different disciplines
• it leaves something practical behind
• and somehow involves a mystical wheel connected to a Magic 8 Ball

At that point, it stops feeling like another internal obligation and starts feeling like something people genuinely want to keep alive.

Just this month, I built a full design system in about 20 hours.

What used to take weeks, sometimes months, is now dramatically faster. So… what actually changed? And more importantly: what didn’t?

Design systems take time. On complex platforms, they can take hundreds of hours.

We were working with a large and complex product where inconsistencies had started to pile up. Different modules had evolved in isolation, teams were making independent decisions, and there were no shared guidelines. The answer was clear: we needed a design system.

AI tools were just starting to emerge back then. They were mostly useful for simple tasks as they tended to hallucinate when things got complex. Developers had started using them earlier than designers, MCP didn't exist yet, and Figma plugins were the best automation we had.

But the context has changed. Fast.

The Manual Era

We did what most teams did. We stopped, and we built it. Manually.

Picture two designers, a mountain of inconsistencies, and no map. We had to cross-reference information manually, digging through the code, detecting what could be merged, agreeing on naming conventions, deciding how to name components. Hours and hours of discussion until we finally landed on a solution.

In the end, we got there. A cleaner system, faster workflows, and for the first time, both teams speaking the same visual language. Hard-won, but it worked.

But now every month a new AI model seems to be released. Design is finally catching up with what developers faced about two years ago. New tools arose, and with that, the scope of our work as designers completely changed.

The Human Factor

For an internal project, I used our Kaizen site as a reference, combined with documentation from industry leaders as a guideline.

I started in v0, which is essentially a chat interface where you can generate UI components through prompts. I fed it the colors, typographies, and a reference image, and from there it was a back-and-forth: the AI generated, I reacted, adjusted, and pushed until the output matched what I had in my head. And just like that, I started prompting my way through a Design System.

Once a component was ready, I used the html.to.design plugin to bring it into Figma (yes, plugins are still alive!). Think of it as a bridge: the plugin exports designs directly from the browser into a Figma file.

Inside Figma, the intervention was more hands-on. First, I checked that everything was visually consistent with what was defined in v0: colors, typography, styles. Then I used Figma's built-in AI to rename all the component layers using BEM convention (something that would have taken a significant amount of time to do so manually).

BEM, which stands for Block Element Modifier, is a widely adopted naming convention in CSS. It structures layer names hierarchically and predictably, for example: button__label--disabled.

Using it keeps the code clean, readable, and consistent, especially when you're working alongside a developer who needs to understand what came out the other side.

Beyond naming, I also made sure the layer structure would generate the right properties when building component sets in Figma, so that all the variants would be correctly exposed and usable. My team also pointed out that adding descriptions to components and variants was key as context for any agent using them through an MCP.

The last step was connecting everything to Windsurf via MCP. With a frame selected in Dev Mode, Windsurf could read the Figma file and use the components to build more complex screens.

We worked closely with a developer throughout this phase. Not just for the technical knowledge, but because having someone who reads code fluently meant catching things we wouldn't have spotted otherwise. The design role here was direction and supervision: making sure the AI used the components correctly and didn't invent solutions where context was missing.

Every step of the process had a human decision behind it.

An Unexpected Discovery

At one point, before we had any of the naming conventions figured out, I selected a frame and asked Windsurf to build a form using the components inside it, styled to match a specific card. The developer next to me was skeptical until he saw the result, and then he was just as surprised as I was.

What we realized is that the MCP wasn't reading layer names to understand context. It was reading everything inside the frame, even the loose text sitting alongside the components. Good naming is still worth doing. But the MCP doesn't need it to understand what it's looking at.

Learning to Talk to an AI

The more specific and contained your prompt, the better the outcome. We started with the most atomic component: the button, and worked outward from there. Each approved component became context for the next one, so the system gradually picked up the visual language we were building.

At some point I got ambitious and asked for five cards in a single prompt: blog card, service card, testimonial card, stats card, feature card… structures, states and all. The AI delivered.

Visually, everything looked fine. Then the developer looked at the code and pointed out that all five cards were independent components instead of variants of one. For a design system, that breaks everything.

One correction prompt fixed it. But it was a good reminder: the AI does exactly what you ask, not what you mean. And fixing it after the fact can cost more than getting it right from the start.

Some Things Learned Along the Way

• Precision is key. Natural language is fine when you're asking for a cooking recipe, but when referring to a component, if you say things like "create" instead of "add", you'll probably end up with a whole new set of components instead of additional variants of an existing one.
• The "Frame" is the context: MCPs can read everything inside the frame you select. This is a game-changer. It means the "naming conventions" debate might be shifting. If the AI understands the context visually and structurally, will we still spend hours discussing nomenclature in 2027?
• No matter what happens, you can always roll back in less than 5 minutes and start over.
• Work closely with a developer: they can help you understand MCPs and clear up any code-related doubts. Once you start to grasp their logic, you'll learn very quickly how to prompt in ways that AI actually understands.
• There's nothing to lose by asking the AI to follow a specific naming convention for the code. It keeps everything clean and readable, and it takes no extra effort.
• The AI covers roughly 80% of the work (generation, variations, exploration...), but the remaining 20% is where quality lives, and that part is not delegable. The AI executes. The judgment is still yours. And if you skip the review, you're not saving time: you'll spend it later.
• Context matters more than tooling. What you don't define, the AI will invent. Small components may be resolved well, but large interfaces require more definition from the start. A well-defined system scales. An undefined one generates inconsistencies faster than you can fix them.
• Figma is no longer the mandatory starting point. It's useful as a visual reference, a QA space, or a consolidation layer. But the AI doesn't need it. We still do.
• There's no single right workflow yet. What you do depends on the project. We're in a transition moment where the tools change faster than the standards. The best thing you can do right now is experiment.

What AI Still Can’t Replace

Through all of this, a few things became very clear. These are the parts that didn’t change:

• Knowing when something looks off. The AI generates, but it doesn't notice when the result doesn't feel right. That eye is yours.
• Direction and supervision. The AI used the components we gave it, but without someone supervising it, it invents solutions where there is no context to work from.
• The definition of done is still a human call, whether it's a conversation with a PO, a stakeholder, or just the designer's criteria. There's no prompt for that.
• The context: knowing why certain decisions matter, what a component should communicate, what the user will actually feel. Business knowledge, stakeholder dynamics, unwritten rules, empathy for the end user. These take years to build and live in the people doing the work, not in the tools they use.

My Two Cents

The tools changed, and that gave me the chills, but throughout this experience I found that the designer's role is more alive than ever.

What once took a team weeks can now be prototyped in hours. That’s not a threat; it’s an invitation to get curious.

I'm still figuring a lot of this out, and I suspect most of us are. There's no right workflow yet, and honestly, that's fine. We are in a transition where tools change faster than standards. The best thing you can do is experiment. Don't wait for a "definitive" workflow, it might be obsolete by next month.

Go ahead, try prompting your way through a component. You might be surprised how fast the system starts to take shape.

‍

Applying changes across microservices is difficult because business logic is distributed across multiple services, each with its own data, contracts, and responsibilities.

In our experiment at Kaizen Softworks, we tested whether an AI system could safely apply coordinated changes across a microservices architecture using only minimal input.

Short answer: Yes, but only when the AI has enough architectural context.

Why are coordinated changes in microservices so hard?

In distributed systems, a single business change rarely affects just one service.

It often requires:

• Updating multiple microservices
• Modifying message contracts
• Keeping DTOs (Data Transfer Objects) consistent
• Respecting domain boundaries defined by Domain-Driven Design (DDD)

Key entities in this system:

• Microservice: An independently deployable service responsible for a specific domain
• Aggregate (DDD): A cluster of domain objects treated as a single unit
• DTO (Data Transfer Object): A structured format used to transfer data between services
• Message/Event: A communication mechanism between services

The complexity is not in the code, it’s in the relationships between components.

‍

The experiment: Can AI reason across services with minimal input?

We designed a controlled experiment to test whether an AI model could apply system-wide changes with limited information.

Input given to the AI:

• Message definitions (events between services)
• DTOs (data contracts)

Tasks the AI had to perform:

1. Identify affected aggregates
2. Determine service ownership
3. Apply coordinated changes across services
4. Maintain consistency in messages and DTOs

In other words, the AI had to behave like a software architect, not just a code generator.

What was the biggest obstacle?

The biggest challenge was not technical, it was contextual.

Problem: unclear service naming

Instead of descriptive names like:

• order-service
• billing-service

Our services were named:

• john
• sally
• roger

This removed any semantic clues about responsibility.

Result: The AI could not infer which service owned which domain logic.

‍

The missing piece: aggregate ownership mapping

To solve this, we introduced a simple but powerful structure:

Aggregate → Service mapping

• Order → john
• Shipment → sally
• Invoice → roger

This created a clear relationship between domain concepts and system components.

Once ownership was explicit, the architecture became understandable.

How we used AI to generate architectural context

Instead of building this mapping manually, we used AI to analyze the codebase and extract:

• Where each aggregate was defined
• Which microservice implemented it
• The relationship between domain and infrastructure

The result was a machine-readable architecture map.

In practice, we used AI to generate the context that AI itself needed.

Results: Can AI safely apply distributed changes?

With the architecture map in place, the AI was able to:

• Trace message flows across services
• Identify affected aggregates
• Locate the correct microservices
• Apply coordinated updates
• Maintain consistency between DTOs and messages

While not perfect, the system worked reliably as a proof of concept.

What is the real limitation of AI in microservices?

The main limitation of AI is not code generation, it’s architectural understanding.

Without knowing:

• Which components exist
• How they relate
• Who owns what

AI cannot safely modify a distributed system.

AI performance depends more on context quality than model capability.

When can AI safely modify microservices?

AI works well when:

• Aggregate ownership is clearly defined
• Message contracts are explicit
• Architecture is structured and consistent

AI struggles when:

• Naming is ambiguous
• Relationships are implicit
• Context is incomplete

Simple rule: If the architecture is clear, AI can reason. If not, it guesses.

Final thoughts

This experiment revealed something important:

AI doesn’t fail because it can’t write code.
It fails because it can’t see the system.

As teams move toward AI-assisted development, the focus will likely shift from:

Writing better code to Designing better systems for machines to understand

At Kaizen Softworks, we see this as a foundational shift.

Because when AI can understand architecture, it doesn’t just generate code, it helps evolve systems.

‍

In 2026, building AI-powered features has become relatively easy. While working on AI initiatives within the Innovation Hub at Kaizen Softworks, we kept running into the same pattern: PoCs worked, demos looked impressive, and stakeholders were happy. But production hit red flags.

When you move from an internal prototype to production, uncomfortable questions start showing:

• What happens if a user pastes personal data?
• What if the bot answers about topics unrelated to our business?
• What if it confidently makes something up?
• What if it mentions a competitor?
• What happens when users actively try to break it?

AI guardrails and evaluations have shifted from "extra safety work" to core product concerns.

What are AI Guardrails?

AI Guardrails are secondary checks that sit between the user and the Large Language Model (LLM). They act as a validation checkpoint, monitoring, filtering, and validating both the input (prompts) and the output (responses) to ensure they meet safety, accuracy, and brand standards.

Instead of trusting the model blindly, you are defining the boundaries of "valid behavior, which usually means:

1. Grounding: Ensuring no hallucinations.
2. Privacy: No PII (Personally Identifiable Information) leakage.
3. Relevance: Staying strictly on-topic.
4. Brand Safety: Avoiding reputational risks or competitor mentions.

‍

Why Guardrails Matter for Production AI Systems

We’ve already seen public cases of large AI-powered products responding to almost any topic-not because the models were bad, but because clear boundaries weren’t defined. As systems become more agentic (taking actions on behalf of users), these risks only grow. 

The value of these patterns, which are covered in the DeepLearning.ai "Safe and Reliable AI" course, is that they provide a model for building responsible AI.

Guardrails aren't a silver bullet, but they are the difference between a prototype that "looks cool" and a system you can actually trust with your brand and your users' data. At Kaizen Softworks, this way of thinking is becoming increasingly important as we explore and ship AI-driven solutions. 

4 AI Guardrails for LLMs Systems

To move beyond the demo, we recommend implementing these four technical validation layers:

1- Hallucination detection via grounding (NLI)

In a RAG (Retrieval-Augmented Generation) system, a hallucination is usually a lack of grounding. A way to verify that every statement is explicitly supported by trusted source text is through Natural Language Inference (NLI).

Instead of asking "Does this answer look right?", we use a secondary, smaller model to ask if the output is logically entailed by the source context. This makes hallucinations something you can programmatically reason about and block in real-time.

2- Topic restriction and off-topic detection

Another common problem is the "Everything Bot"—that answers questions about your business, but also gives recipes or writes poetry if asked.

While you can try to "prompt" an LLM to stay on topic, it’s expensive and slow. We prefer Zero-Shot Classification. It’s a dedicated layer that categorizes the intent before it even hits the expensive LLM. It’s:

• Faster & Cheaper: You aren't burning tokens on a 175B parameter model just to say "no."
• Deterministic: You get a clear Pass/Fail for the topic.
• Local Execution: Can be run locally to reduce latency.

3- PII detection and anonymization

Data privacy is the #1 reason AI projects stall in legal. PII (Personally Identifiable Information) handling is easy to ignore in demos but is a dealbreaker in production.

Tools like Microsoft Presidio allow you to:

1. Detect sensitive data in user inputs.
2. Anonymize or Mask it before it ever reaches a third-party LLM provider.
3. De-mask it on the way back to the user if necessary.

This makes data privacy risks very tangible, especially when working with third-party LLM providers.

4- Reputation and sensitive-content guardrails

There are also examples of guardrails for:

• Competitor name detection
• Sensitive topics
• Brand-related risks

Again, the focus is not on theory, but on patterns you can actually apply.

My Take on DeepLearning.ai’s "Safe and Reliable AI via Guardrails"

To dig deeper into this topic, I took the short course “Safe and Reliable AI via Guardrails” by DeepLearning.ai. 

This course is not about training models or prompt engineering. It’s about everything that surrounds the LLM when you want to ship an AI feature safely and reliably.

You won’t leave this course as a “guardrails expert”. What you will get:

• A solid mental model
• Clear examples of common failure modes
• Practical patterns you can adapt to your own systems
• A better sense of what questions to ask before going to production

It’s a very good entry point, especially for engineers who are starting to ship AI features beyond PoCs.

For me, the biggest takeaway was a mindset shift. When you think in PoC mode, many questions don’t even come up:

• Should we store raw user inputs?
• Do we really want the bot answering unrelated topics?
• What’s the fallback when something goes wrong?

In production, those questions stop being theoretical. The course reinforces the idea that once an AI feature goes to prod, “it works” is not enough.

You start designing:

• Validation layers
• Limits
• Fallback paths
• Failure scenarios

And once you start thinking this way, you don’t really go back.

Learning AI engineering is about developing judgment: knowing when to use models, how to control them, and where they actually add value.

At our Innovation Hub, we’ve been actively experimenting, building, breaking, and refining AI-powered systems in real-world environments. Based on that hands-on experience, we curated this list of AI engineering courses we’d confidently recommend to our own team.

This list is for software engineers, tech leads, and AI practitioners who already ship production code and want to learn how to build AI systems that are reliable, maintainable, and usable. 

Which AI Engineering course should you choose?

TABLA

Retrieval-Augmented Generation (RAG) for Production AI Systems

• Platform: DeepLearning.AI 
• Level: Beginner
• Focus: Production-grade RAG pipelines
• Duration: 24 hours 33 mins
• Cost: $50

Standard LLMs are constrained by static training data and context limits. In real products, that’s a deal-breaker. Retrieval-Augmented Generation (RAG) has become the industry standard for connecting AI systems to private, real-time, and domain-specific data.

What You’ll Learn:

• Core Architecture: How retrieval and generation interact to ground LLMs in real data.
• Advanced Retrieval: Hybrid search, query rewriting, and chunking strategies using vector databases like Weaviate.
• Evaluation in Practice: Measure latency, cost, and answer quality using tools like Phoenix from Arize.
• System Hardening: Handle hallucinations, noisy data, and context window constraints.

How to test or evaluate Gen AI, LLM, RAG, Agentic AI

• Platform: Udemy 
• Level: Intermediate
• Focus: AI-powered quality assurance
• Duration: 3 hours
• Cost: $20

How do you test a system that doesn’t always give the same answer? Traditional unit tests break down when applied to LLMs. TestGenAI tackles that problem head-on by showing how AI can be used to test AI systems themselves, across UI, APIs, databases, and workflows.

What You’ll Learn:

• Automated Test Generation: Use GenAI to create test cases, scenarios, and synthetic data.
• Modern Tooling: Integrate LLMs with Selenium, Playwright, and GitHub Copilot.
• Full-Stack QA: Apply AI agents to SQL testing, API payload validation, and automated bug reporting.
• Prompting for QA: Zero-shot and few-shot techniques specifically for validation and consistency checks.

Safe and Reliable AI: Guardrails in Practice

• ​​Platform: DeepLearning.AI 
• Level: Beginner
• Focus: AI safety, compliance, and control
• Duration: 2 hours
• Cost: Free

As AI systems become user-facing, safety is no longer optional. Guardrails are programmable layers that sit between users and LLMs to prevent harmful, non-compliant, or simply incorrect outputs.

What You’ll Learn:

• Input & Output Validation: Intercept unsafe prompts and filter risky model responses.
  
  
• PII Protection: Automatically detect and redact sensitive information.
  
  
• Hallucination Detection: Use Natural Language Inference (NLI) to verify grounding.
  
  
• Practical Use Case: Build a customer-facing chatbot that enforces real business constraints.

Microsoft Certified: Azure AI Engineer Associate 

• Platform: Microsoft Learn 
• Level: Intermediate
• Focus: Enterprise AI architecture
• Duration: 34 hours
• Cost: $160

For engineers working in larger organizations, this certification is one of the most complete overviews of how AI systems live inside real enterprise infrastructure.

It goes beyond models and into architecture, governance, and deployment constraints.

What You’ll Learn:

• Solution Architecture: Design scalable systems with Azure AI Services, Azure AI Search, and Azure OpenAI.
  Multimodal AI: Combine NLP, computer vision, and knowledge mining.
• Responsible AI: Apply governance and safety controls within Azure.
• Certification Prep: Structured preparation for the AI-102 exam.

Build Apps with Windsurf’s AI Coding Agents

• Platform: DeepLearning.AI 
• Level: Beginner 
• Focus: Agent-based development workflows
• Duration: 1 hour 10 mins
• Cost: Free

We’re moving from copilots to agents. 

Windsurf is an AI-native IDE that allows agents to autonomously refactor, search, debug, and modify code across an entire codebase. This course shows how to work with those agents instead of fighting them.

What You’ll Learn:

• Agentic Workflows: Let AI handle multi-step tasks across files and services.
• Deep Context Awareness: How agent-based IDEs maintain whole-project context.
• Speed with Control: Build full-stack apps faster without losing architectural clarity.
• Human-in-the-Loop Patterns: When to guide the agent, and when to step back.

Claude Code in Action

• Platform: Anthropic
•  Level: Beginner
• Focus: CLI-based AI development & Model Context Protocol (MCP)
• Duration: 1 hour 3 mins
• Cost: Free

Claude Code brings AI directly into your terminal, allowing it to read, reason about, and modify your local codebase. It’s one of the most practical examples of LLMs as real development tools, not chatbots.

What You’ll Learn:

• CLI Mastery: Control your development environment using natural language.
• Context Management: Feed precise files and directories into the model.
• Model Context Protocol (MCP): Extend Claude with local tools, databases, and services.
• Automation: Set up GitHub workflows for PR reviews and issue handling.

How we’d choose between these courses

There’s no single “best” path. The right course depends on what you’re building, who your users are, and how close you are to production.

If you’re deciding where to start:

• Customer-facing AI products: RAG + Guardrails
• AI inside existing platforms: Testing + Azure AI
• Developer productivity: Windsurf + Claude Code

Karen has no patience.

If a button is disabled without explanation, she gets annoyed.
If an empty state looks like an error, she assumes the system is broken.
If a loading spinner doesn’t explain what’s happening, she asks for the manager.

Karen isn’t a real person.
She’s a synthetic user.

And she might be one of the most useful ways I’ve found to stress-test a design before putting it in front of real users.

What Is a Synthetic User?

A synthetic user is a constrained AI decision agent embedded in a controlled simulation framework.

It is not just a profile. It is a structured behavioral model with:

• Identity (role + expertise)
• Intent (clear objective)
• Limits (constraints + forbidden assumptions)
• Logic (behavioral and abandonment rules)
• Boundaries (strict evaluation scope)
• Accountability (structured output requirements)

It operates only within what is defined and cannot compensate for ambiguity, missing signals, or structural gaps in the interface.

A synthetic user is not:

• A fictional persona or a storytelling device
• A predictive AI that guesses user preferences
• An intelligent assistant that fixes unclear design

A synthetic user interacts strictly with what is visible in the interface and nothing more. It does not infer intent, fill gaps, or compensate for ambiguity. When the path forward is unclear, it hesitates. That hesitation is not failure. It is the signal that reveals structural friction.

What a Synthetic User Needs to Work

If you want this to be more than “ChatGPT pretending to be someone,” you need structure. You must define:

1. Functional Role: Who this user is in operational terms (Operations Manager reviewing trip segments).
2. Domain Expertise Level: How much they understand the subject matter (6 months in logistics, still learning edge cases).
3. Technical Proficiency: How comfortable they are with software (Uses dashboards daily, avoids advanced filters).
4. Explicit Objective: What they must accomplish in this session (Confirm whether a trip contains excursions).
5. Success Criteria: What level of certainty is required to consider the task complete (Needs explicit confirmation, not inference from a map).
6. Motivations: What they prioritize when making decisions (Speed over exploration).
7. Constraints: Operational limits that shape behavior (Low tolerance for ambiguity, under time pressure).
8. Behavioral Rules: How they interpret and act on information (If unclear after 3 seconds, move to another visible option).
9. Abandonment Rules: When they stop the flow (If the same friction appears twice, they exit).
10. Forbidden Assumptions: What they cannot infer or mentally “fix” (Cannot assume disabled filters require prior calculation unless explicitly stated).
11. Evaluation Scope: What part of the experience they are allowed to simulate (Only the “Segments” tab, not the full dashboard).
12. Structured Output Format: How the simulation must report results (Step → Action → Clarity → Doubt → Reason → Highest friction).

What I Learned About Using Synthetic Users

Synthetic users don’t validate whether something “works.” What they actually do is expose where a design forces users to interpret instead of confirming things explicitly. They surface structural ambiguity that often goes unnoticed in internal reviews and help distinguish between friction that affects everyone and friction that only impacts less experienced users.

In practice, they make design discussions more concrete because you’re no longer debating opinions, you’re observing constrained behavior. They don’t replace usability testing, but they significantly improve how prepared you are before running it.

How to Start Using Synthetic Users 

If you want to try it today:

1. Define a synthetic user with strict rules
2. Write a clear objective
3. Declare your "forbidden assumptions"
4. Provide the flow step-by-step
5. Force a structured output 

If the synthetic user never hesitates, your constraints are too weak

I’ve pulled together the exact resources I use:

• Download the template and prompts
• Read the full framework, with a step-by-step explanation of how to apply it in practice

This Is Still Early

Agent-based simulation is not a new idea.

What is still underdeveloped is how to apply it in a structured, practical way inside UX workflows. There is no widely adopted standard yet. No clear implementation pattern most teams follow.

What I’m sharing here is not an academic breakthrough. It’s a working implementation.

It can evolve. It can scale into automation.

But even in its current form, it has helped me detect structural friction before running formal usability testing, that alone makes it worth exploring.