Kaizen Teams

Dropdown

Table of Contents

Time to read

·

12

Published on

·

July 16, 2026

Last updated on

·

August 24, 2026

Vera Gonzalez, Frontend Developer at Kaizen Softworks

Vera Gonzalez

Too young to quit

Frontend Developer

AI

AI

AI is already reading your website. Do you know what it's finding?

Published on

·

August 27, 2026

Last updated on

·

August 24, 2026

Time to read

·

12

Vera Gonzalez, Frontend Developer at Kaizen Softworks

Vera Gonzalez

Frontend Developer

Somewhere between a prospect Googling your company and a prospect never visiting your site at all, a new kind of visitor showed up.

It doesn't click. It doesn't scroll. It doesn't show up in Google Analytics. But it scans your website, decides what matters, and quietly influences whether your business gets mentioned the next time someone asks ChatGPT, Perplexity, or Google's AI Overviews for a recommendation.

We had no real way to know what these AI bots were finding on our own site. So, before telling anyone else what to do about it, we built something to find out for ourselves.

The blind spot in your analytics

Google Analytics tracks human sessions, not server-side crawler activity. That's the blind spot. A person searches, sees a list of links, clicks one, lands on your site; that's the journey it was designed to track.

That journey is changing. Fewer people start their research by typing a query into Google and scanning ten blue links. Most of them are asking an AI assistant directly: "who are good software partners for X," "what's the best tool for Y," and trusting the shortlist it hands back. To build that answer, the AI first sent something to read the web on its behalf: a bot with a name like GPTBot, PerplexityBot, or ClaudeBot, crawling pages much like search engines have for decades.

None of that shows up in your dashboards. Those bot visits don't count as sessions, don't trigger conversion tracking, and don't appear anywhere you're already looking. If your site is hard for those bots to read, poorly structured, or quietly blocking them without anyone realizing it, you're not losing a ranking position. You're being left out of a conversation you never knew was happening. It's a new kind of competitive risk. Not "we got outranked," but "we were never in the running, and nothing told us."

That's the gap we set out to close, starting with our own site.

Are AI bots even visiting our site? We stopped guessing.

Inside our Innovation Hub, the group that experiments with new tools and workflows before we bring them into client work, someone asked a simple question: are AI bots even visiting our site? And if they are, what are they actually able to see?

Nobody could answer that with confidence. Not because it's a hard problem to reason about, but because the tool to answer it didn't exist among the tools we already had. So instead of guessing, or buying something built for someone else's website, we built a small internal dashboard for our own.

What we built: a dashboard that tracks AI bot visits

The idea is simple, even if getting there wasn't: a small piece of code sits quietly in front of our website and notes every time a known AI bot stops by. It records which one it was, which page it looked at, whether it got a clean response or hit an error, and how deep into the site it went.

Right now we're tracking bots from OpenAI (the ones behind ChatGPT), Anthropic (Claude), Perplexity, Google, Microsoft's Bing, Meta, and Apple. That list will keep growing. New AI crawlers show up faster than anyone can keep a definitive catalog.

All of that gets pulled into a dashboard the team can check the same way we'd check any other business metric: how much of the site is actually getting crawled, where bots are hitting dead ends, whether they're respecting the instructions we leave for them, and how that changes over time.

Screenshot of an AI Visibility Dashboard showing traffic metrics and a crawl coverage table for AI bots like OpenAI, Anthropic, and Microsoft, tracking hits, unique paths, and service page visits by company.

What the dashboard caught in the first two weeks

We didn't have to wait long to see the point of building this. Two things came up in the first few weeks alone.

The file we thought was working

An llms.txt is a simple file some AI models look for to understand what a site is about. Like a lot of sites getting ready for an AI-driven web, we added one, checked it was live, and moved on, assuming that box was checked.

The dashboard said otherwise. Weeks in, not a single bot had requested it.

So we went digging, and read that crawlers rely on robots.txt to know an llms.txt file exists in the first place, and ours didn't reference it. We added the missing line. Bots still weren't picking it up.

Third attempt: we added plain, visible links to the file in the site's header and footer, the same way we'd link to any other page. That's what did it. Two weeks of zero requests, and on the exact day we shipped that change, the file got six requests from five different AI companies.

Before and after adding links to llms.txt.

The detail we only noticed because the dashboard breaks bots down by type: those six requests were all from indexer and training bots, the ones that crawl the web to build a general picture of it, not yet from retrieval bots, the ones that fetch a page in real time to answer someone's specific question right now. That's a useful distinction. It's the difference between "we're now on the map" and "we're being pulled up live," and it tells us what to check for next.

None of that would have surfaced anywhere else. Not in Analytics, not in Search Console. We would have gone on believing the file was doing its job, simply because we remembered adding it.

The high-value pages AI bots were quietly skipping

The second finding was less comforting: several of our most important pages, the ones describing what we actually do, were barely being crawled at all. Not blocked, not broken. Just quietly skipped by many bots.

We built a graphic on the dashboard specifically for this: crawl coverage per bot, broken down page by page. Now, instead of assuming coverage is even across the site, we can see exactly which high-value pages each AI bot is actually reading, and which ones it's ignoring.

The Crawl Coverage table breaks down how thoroughly each AI bot is reading the site: total hits, unique paths crawled, and whether key service pages are being reached.

We're still working on closing that gap. The first fix we tried didn't move things the way we expected, so for now the coverage graphic itself is doing the real work: telling us, page by page and bot by bot, whether the next attempt actually helps instead of just hoping it does.

Neither of these was something we could have reasoned our way into. We only found them because we were finally looking.

Before you optimize, measure

It's tempting to jump straight to fixes: restructure content, add an llms.txt file, rewrite pages to be more "AI-friendly." We did some of that too. But our own llms.txt sat unused for weeks and we had no idea, because we had nothing telling us otherwise. Without a baseline, you can do all the "right" things and still have no idea whether any of them worked.

Our approach here mirrors how we tend to approach any technology problem: understand what's actually happening before deciding what to change. It's a small dashboard, built quickly, answering one honest question. It's already paid for itself twice over, and we're still early.

We'll keep sharing what we find as the picture gets clearer. If you're curious what your own numbers might look like, that's a conversation we're happy to have.

Somewhere between a prospect Googling your company and a prospect never visiting your site at all, a new kind of visitor showed up.

It doesn't click. It doesn't scroll. It doesn't show up in Google Analytics. But it scans your website, decides what matters, and quietly influences whether your business gets mentioned the next time someone asks ChatGPT, Perplexity, or Google's AI Overviews for a recommendation.

We had no real way to know what these AI bots were finding on our own site. So, before telling anyone else what to do about it, we built something to find out for ourselves.

The blind spot in your analytics

Google Analytics tracks human sessions, not server-side crawler activity. That's the blind spot. A person searches, sees a list of links, clicks one, lands on your site; that's the journey it was designed to track.

That journey is changing. Fewer people start their research by typing a query into Google and scanning ten blue links. Most of them are asking an AI assistant directly: "who are good software partners for X," "what's the best tool for Y," and trusting the shortlist it hands back. To build that answer, the AI first sent something to read the web on its behalf: a bot with a name like GPTBot, PerplexityBot, or ClaudeBot, crawling pages much like search engines have for decades.

None of that shows up in your dashboards. Those bot visits don't count as sessions, don't trigger conversion tracking, and don't appear anywhere you're already looking. If your site is hard for those bots to read, poorly structured, or quietly blocking them without anyone realizing it, you're not losing a ranking position. You're being left out of a conversation you never knew was happening. It's a new kind of competitive risk. Not "we got outranked," but "we were never in the running, and nothing told us."

That's the gap we set out to close, starting with our own site.

Are AI bots even visiting our site? We stopped guessing.

Inside our Innovation Hub, the group that experiments with new tools and workflows before we bring them into client work, someone asked a simple question: are AI bots even visiting our site? And if they are, what are they actually able to see?

Nobody could answer that with confidence. Not because it's a hard problem to reason about, but because the tool to answer it didn't exist among the tools we already had. So instead of guessing, or buying something built for someone else's website, we built a small internal dashboard for our own.

What we built: a dashboard that tracks AI bot visits

The idea is simple, even if getting there wasn't: a small piece of code sits quietly in front of our website and notes every time a known AI bot stops by. It records which one it was, which page it looked at, whether it got a clean response or hit an error, and how deep into the site it went.

Right now we're tracking bots from OpenAI (the ones behind ChatGPT), Anthropic (Claude), Perplexity, Google, Microsoft's Bing, Meta, and Apple. That list will keep growing. New AI crawlers show up faster than anyone can keep a definitive catalog.

All of that gets pulled into a dashboard the team can check the same way we'd check any other business metric: how much of the site is actually getting crawled, where bots are hitting dead ends, whether they're respecting the instructions we leave for them, and how that changes over time.

Screenshot of an AI Visibility Dashboard showing traffic metrics and a crawl coverage table for AI bots like OpenAI, Anthropic, and Microsoft, tracking hits, unique paths, and service page visits by company.

What the dashboard caught in the first two weeks

We didn't have to wait long to see the point of building this. Two things came up in the first few weeks alone.

The file we thought was working

An llms.txt is a simple file some AI models look for to understand what a site is about. Like a lot of sites getting ready for an AI-driven web, we added one, checked it was live, and moved on, assuming that box was checked.

The dashboard said otherwise. Weeks in, not a single bot had requested it.

So we went digging, and read that crawlers rely on robots.txt to know an llms.txt file exists in the first place, and ours didn't reference it. We added the missing line. Bots still weren't picking it up.

Third attempt: we added plain, visible links to the file in the site's header and footer, the same way we'd link to any other page. That's what did it. Two weeks of zero requests, and on the exact day we shipped that change, the file got six requests from five different AI companies.

Before and after adding links to llms.txt.

The detail we only noticed because the dashboard breaks bots down by type: those six requests were all from indexer and training bots, the ones that crawl the web to build a general picture of it, not yet from retrieval bots, the ones that fetch a page in real time to answer someone's specific question right now. That's a useful distinction. It's the difference between "we're now on the map" and "we're being pulled up live," and it tells us what to check for next.

None of that would have surfaced anywhere else. Not in Analytics, not in Search Console. We would have gone on believing the file was doing its job, simply because we remembered adding it.

The high-value pages AI bots were quietly skipping

The second finding was less comforting: several of our most important pages, the ones describing what we actually do, were barely being crawled at all. Not blocked, not broken. Just quietly skipped by many bots.

We built a graphic on the dashboard specifically for this: crawl coverage per bot, broken down page by page. Now, instead of assuming coverage is even across the site, we can see exactly which high-value pages each AI bot is actually reading, and which ones it's ignoring.

The Crawl Coverage table breaks down how thoroughly each AI bot is reading the site: total hits, unique paths crawled, and whether key service pages are being reached.

We're still working on closing that gap. The first fix we tried didn't move things the way we expected, so for now the coverage graphic itself is doing the real work: telling us, page by page and bot by bot, whether the next attempt actually helps instead of just hoping it does.

Neither of these was something we could have reasoned our way into. We only found them because we were finally looking.

Before you optimize, measure

It's tempting to jump straight to fixes: restructure content, add an llms.txt file, rewrite pages to be more "AI-friendly." We did some of that too. But our own llms.txt sat unused for weeks and we had no idea, because we had nothing telling us otherwise. Without a baseline, you can do all the "right" things and still have no idea whether any of them worked.

Our approach here mirrors how we tend to approach any technology problem: understand what's actually happening before deciding what to change. It's a small dashboard, built quickly, answering one honest question. It's already paid for itself twice over, and we're still early.

We'll keep sharing what we find as the picture gets clearer. If you're curious what your own numbers might look like, that's a conversation we're happy to have.

Common questions

Do AI bots crawl my website?

Most likely yes, but you can't see it in standard analytics. Bots like GPTBot, ClaudeBot, and PerplexityBot crawl sites the way search engines have for decades. The only way to know for sure is to check your server logs or a dedicated tracker.

Do AI crawler visits show up in Google Analytics?

No. Google Analytics tracks human sessions, not server-side crawler activity. AI bot visits don't count as sessions, don't trigger conversion tracking, and don't appear in your dashboards.

Why isn't my llms.txt file being read?‍

Adding the file isn't enough. Crawlers rely on robots.txt to know an llms.txt exists, so it needs to be referenced there. In our case, even that wasn't enough. Bots only started requesting it once we added plain, visible links to it in the site's header and footer.

What's the difference between indexer bots and retrieval bots?‍

Indexer and training bots crawl the web to build a general picture of it. Retrieval bots fetch a page in real time to answer someone's specific question. Being crawled by indexers means you're on the map. Being hit by retrieval bots means you're being pulled up live to answer someone right now.

How can I see which AI bots visit my site?‍

You need something that watches server-side requests and identifies known AI user agents, recording which bot visited, which page it read, whether it got a clean response, and how deep it went. Standard analytics won't show any of it.

1
What is the difference between working with Kaizen versus a single designer?

Related Articles

View all articles

·

Aug 28, 2026

Catalyst 2026 recap: inside Partnership Leaders’ 2026 NYC conference

Everything to know about Catalyst 26: dates, price, who attends, both keynote recaps, and when the next Catalyst event is.

12 read time

Read more

Catalyst 26 was Partnership Leaders' fifth annual conference for partnership, ecosystem, and go-to-market professionals. It took place August 25 and 26, 2026, at the Marriott Hotel at the Brooklyn Bridge in New York, with more than 1,000 attendees and 70-plus speakers from companies including Anthropic, OpenAI, Google, Microsoft, IBM, BCG, and Siemens.

Dates August 25–26, 2026
Location Marriott Hotel at the Brooklyn Bridge, Brooklyn, NY
Edition 5th annual
Attendees 1,000+ partnership, ecosystem, and GTM professionals
Speakers 70+, including people from Anthropic, OpenAI, Google, Microsoft, IBM, BCG, and Siemens.
Price $849 early bird, rising to $999, then $1,999

Who Catalyst events are for

Catalyst brought together people building and running partner programs across SaaS, AI, consulting, systems integration, agencies, and major cloud platforms.

Attendees included executives leading partnership organizations, and people working directly in alliances, partner sales, marketing, operations, strategy, and enablement.

What Catalyst 26 is like

You can look at the agenda before a conference and have a pretty good idea of what you'll find. Being there is different.

This year's theme was "Navigating Frontier Ecosystems". Anthropic's Head of Partnerships and one of OpenAI's partner program leads appeared on the same agenda as people from Oracle, Siemens, IBM, and BCG, companies that have run formal partner programs for two decades.

That mix was one of the most interesting parts of the conference. Newer AI companies were discussing partner tiers, co-selling, and joint delivery alongside companies where those models have been part of their business for years.

What Catalyst 26 covered

Catalyst 26 split its sessions into eight pillars:

  • Advancing Organizational Maturity: turning partnerships into something measured and repeatable instead of one founder doing favors for another.
  • Become a Strategic Partner: getting partnerships involved when product and business decisions are made, not told about them afterward.
  • Frontier Partner Experience: adapting partner programs as AI changes how companies build and integrate products.
  • Path to CPO: career sessions for people aiming to lead partnerships at the executive level.
  • Co-Build: two companies building something together.
  • Co-Market: two companies running a campaign together.
  • Co-Sell: two sales teams working the same deal.
  • Co-Serve: two companies delivering the same engagement to a client.

Catalyst 26 sessions

Day 1 Keynote

The Day 1 keynote brought together Partnership Leaders’ CEO Asher Mathew, Tribe AI’s Co-founder & CEO Jaclyn Rice Nelson, Anthropic’s Head of Partnerships Phil Samenuk, and Boomi’s Chairman & CEO Steve Lucas.

Their discussion focused on how companies are relying on partners to build, sell, and deliver products across AI, cloud, and enterprise software. A few points stood out:

  • More companies have dedicated partner teams now, which means a generic, one-size-fits-all partner program doesn't cut it anymore. Partners show up when the program fits how they work.
  • New AI products and cloud services are shipping so fast that a partner program can't just get set once and left alone. Incentives, support, and how you work together need regular updates.
  • Partnerships also came up as a way to access data a company couldn’t reach on its own, whether that meant getting access to it, combining it, or putting it to use.
  • AI doesn't change the basics of a good partnership. Account planning, clear ownership, and relationships built over time still matter most.

Day 2 Keynote

The Day 2 keynote featured Ramp’s Lead Economist Ara Kharazian, Eliza’s Founder Brian Benedict, Siemens’ EVP Global Partner Ecosystem Dion Smith, and Oracle’s SVP, Partner Sales & Operations Strategy Leah Yomtovian.

A few points stood out:

  • The spending data told a slower story than expected: AI adoption is mostly going toward productivity gains and task automation, not some overnight shift.
  • Siemens is in the middle of folding more than 68,000 partners and roughly 200 separate programs into a single global one, mainly to make it easier to coordinate across IT and operational technology.
  • Oracle's approach is a running "listening tour": every partner gets the same baseline benefits, then incentives and credits get layered based on the type of partner and how they work with Oracle.
  • There was also talk of a newer kind of service team: bring in engineers, turn AI requirements into working products, and reuse delivery methods that already work instead of starting from scratch each time.

Next Catalyst events

The date and location of Catalyst 27 hasn’t been announced yet. In the meantime, you can check out the half-day Catalyst Summits in different cities:

  • October 20, 2026 - Seattle
  • October 27, 2026 - Chicago
  • October 2026 - Los Angeles
  • December 2026 - Singapore

Check Partnership Leaders’ events page for updates.

·

Aug 26, 2026

Why adding people doesn't always fix a struggling team

Learn when a software team should hire, wait, reorganize, or build skills internally, and how to tell which option will actually help.

12 read time

Read more

When a client asks to hire someone new, a common reaction is to open a search. There's more work, more pressure, and new features to build. It seems like the obvious thing to do.

But in our experience working with software development teams, the problem often isn't a lack of people. The problem is knowledge concentrated in too few people, unclear team roles, slow onboarding, or temporary demand.

The question worth asking isn't who can fill the position, but what would help the team work better. That points to one of three answers: hire, don't hire, or build the capability from within. Figuring out which one applies, and why, is the real work before opening a search.

What you should ask before assuming you need someone new

Hiring works when three conditions are met: the need will last, no one on the team has the capacity to take it on, and the team can onboard someone well. That last condition is easy to overlook. A team can have a real, lasting gap and still not be ready to bring someone in if no one has the time to guide them.

The risk comes from jumping straight from "there's more work" to "we need someone" without checking what's causing the pressure. It's easy to turn a request into a list of requirements (X years of experience, a specific technology, advanced English) and start the search. The real cause is often something else: a project that grew too fast, a tech lead with no time to onboard new hires, processes that stopped scaling, or a team that lost key people and needs to recover knowledge before adding headcount.

That's why, before thinking about who could fill the role, we ask these questions:

  • What outcome is the client trying to achieve?
  • What's happening on that team today?
  • What specific problem is this hire meant to solve?
  • Does adding a person solve that problem?
  • Is there someone on the team who could take this on?
  • Are there other, less obvious alternatives?

When the answers confirm the need will last, the current team can't cover it, and the team has the capacity to onboard someone, hiring is the right call: opening the search fills a gap the team can't close internally.

Does the problem need someone new to fix it?

Not hiring is the right call when the problem behind the request is temporary, or when it will resolve before the new hire finishes onboarding. Recommending against a hire may sound unusual for a company that offers staff augmentation, but our job as a strategic partner is not to maximize every opportunity but to recommend the best decision for the client. Depending on what's actually going on, the fix can look like:

  • An internal rotation: moving someone with spare capacity into the gap.
  • Reorganizing responsibilities across the team instead of adding a seat.
  • Hiring a different profile than the one originally requested.
  • Combining two roles into one instead of opening two searches.
  • Waiting a few weeks, when the project context is about to change on its own.

Is a temporary increase in workload a good reason to hire?

This happened on a project with a long onboarding period. The initial request seemed clear: hire a mid-level developer. There was work and budget available. But when we spoke with the team, we found that the workload increased because one team member had been temporarily reassigned to another sub-team. Before moving forward, we considered what would happen when that person came back.

The client's system was complex: any new hire needed several months to understand the business, the architecture, and the platform before they could contribute independently.

The problem justifying the hire was going to disappear, but the new hire wouldn't. By the time that person had enough context, the need that started the search would no longer exist.

We recommended against moving forward, even though there was budget to add someone. The client avoided an unnecessary hire and months of onboarding for a problem that was already resolving itself. Sometimes the best answer is to wait a few weeks; other times, it's reorganizing the team or developing internal talent.

How can you build team capability without hiring?

Build capability internally when the team already has product context but lacks a specific skill. Developing that skill internally can be faster than waiting for someone new to reach the same level of context.

More people doesn't always mean more capacity. Onboarding a new hire takes time from the people already on the team: explaining the business and the architecture, reviewing their work, and building trust. That's why, during the first few weeks, a team can become less productive while it onboards someone new. Complex projects can include years of technical decisions and undocumented knowledge. New hires still need time to learn that context.

Should you hire a specialist or train someone on your team?

A client needed a senior SQL Server specialist. That niche skill set made the role difficult and expensive to fill. We started the search and interviewed candidates, but the deeper issue became clear quickly: the real challenge on the project wasn't SQL Server. It was understanding a product shaped by years of evolution, multiple applications, and complex business logic.

The right person to develop that expertise was already on the team. Instead of hiring someone with deep SQL Server expertise, the client supported that team member in building the SQL Server skills the project needed. That person had business knowledge, motivation, and a much shorter learning curve than an external hire would have had. An outside specialist provided targeted support when needed.

The team gained SQL Server expertise without losing months waiting for a new hire to learn the product first. The person who took on SQL Server gained a valuable new skill without stepping away from the other work they were doing on the project.

A team's capacity depends on how its people complement each other, what knowledge they share, and what autonomy they've developed, not just on headcount. A team of ten people who are aligned, with shared context and autonomy, can generate more value than a team of fifteen where much of the time goes into onboarding new hires.

Should you hire, wait, or develop the skill internally?

Scenario Signal What to do
Hire The need will last, no one on the team can cover it, and the team can onboard someone well Open the search for a clearly defined role
Don't hire The problem is temporary or resolves before onboarding finishes Wait, reorganize the team, or cover the gap another way
Build internal capability Missing specific expertise, not people; someone already has the business context Develop the skill internally, with targeted outside support if needed

What questions do we ask first?

  1. What specific problem are we trying to solve?
  2. Will the need still exist after the person has been hired and onboarded?
  3. Is there someone on the team who could cover it?
  4. Do we have the capacity to onboard someone well?
  5. Is the problem a lack of people, or is it caused by unclear roles, missing product knowledge, slow onboarding, or a temporary increase in workload?
  6. What impact will this hire have six months from now?
  7. If we couldn't hire today, what other option would we explore?
  8. What higher-priority work would someone on the team have to stop doing to cover this need?

Wait to open a search when the team can't define the problem, confirm the need will last, or support onboarding. Clarify those points first.

If you're weighing this decision with your own team, let's talk about whether to hire, reorganize, or develop someone already on the team.

llms.txt