Adding dynamic llms.txt

TLDR:

I implemented llms.txt on my site to improve AI discoverability. It connects markdown endpoints to large language models. I’ve also included ideas on how WordPress users can do the same.

I've added llms.txt support to my website as part of a broader rebuild of carmelosantana.com. This is the first public post since migrating my site from WordPress to a partially static markdown-based Next.js system, and it feels great.

For now, this update is about laying the groundwork for better AI indexing. Specifically, I’ve added an llms.txt file to the root of the domain to support structured content discovery by large language models like ChatGPT.

---

What is it?

The llms.txt specification defines a simple format that allows a website to declare its content structure directly to large language models. Think of it like robots.txt—but instead of guiding search engine crawlers, llms.txt is meant to help AI systems locate structured, meaningful, and machine-readable content.

The file itself is just a well-formatted markdown document. It includes a heading, a short bio, and links to your site’s public content. These links point not just to human-readable pages but also to raw content or APIs—whatever helps AI systems index and understand your site.

Why add more stuff?

I’m currently experimenting with how well AI tools index my content. My goal is to check back in a few months, search for some keywords on ChatGPT, and see if my content surfaces.

How it's implemented

On my rebuilt Next.js site, I migrated all blog posts to markdown. That means no database, no block editor—just clean, version-controlled .md files. With a bit of custom middleware, I now serve raw markdown at endpoints like:

• blog.md
• projects.md
• services.md

These .md routes internally resolve to /blog/md, /projects/md, and so on. But from the outside, it looks clean and direct.

The llms.txt file is automatically generated at build time. It pulls from these markdown files, validates the links, and produces a structured summary of the site. All of this happens in the build process—no manual edits are needed.

Here’s a sample of the file:

# Carmelo Santana - Technical Lead & WordPress Architect

> Technical lead empowering creators through mindful technology. WordPress architect, automation expert, and advocate for purposeful digital craftsmanship with over 15 years of engineering experience.

## About
- [Professional Experience](https://carmelosantana.com/about.md)
- [Services Overview](https://carmelosantana.com/services.md)

## Digital Experiences
- [Featured Projects](https://carmelosantana.com/projects.md)
- [Project Portfolio](https://carmelosantana.com/portfolio.md)

## Blog & Technical Writing
- [Blog Posts](https://carmelosantana.com/blog.md)

Adding your llms.txt to WordPress

Yes. If you're using WordPress and want to experiment with llms.txt, you'd likely need a plugin to:

1. Intercept any requests ending in .md
2. Convert the underlying post content to markdown
3. Return that as text/markdown for AI systems to consume

Here’s a ChatGPT prompt you can try to help build that:

Prompt:

I'm running a WordPress site and want to implement llms.txt and markdown endpoint discovery for AI systems.

Please help me create a WordPress plugin that:
- Adds llms.txt support at the root of my site
- Intercepts requests ending in .md (like /blog.md, /about.md)
- Converts matching post and page content to clean markdown format
- Returns proper content-type headers for AI consumption
- Automatically generates the llms.txt file from site content

Context for implementation:
- The llms.txt should include site description, key content sections, and direct links to markdown endpoints
- Ensure the plugin handles custom post types if needed
- Include proper error handling for non-existent content

Please provide the complete plugin structure and code.

You can even create a script to automatically generate the llms.txt file based on your site's content. This can be done using a combination of WP CLI commands and custom PHP scripts.You can update this via WP CLI or a scheduled task to keep it up to date.

// Dynamic WP CLI command to generate llms.txt
add_action('cli_init', function() {
    WP_CLI::add_command('generate_llms_txt', function() {
        $site_title = get_bloginfo('name');
        $site_description = get_bloginfo('description');
        $site_url = get_site_url();
        
        $content = "# {$site_title}\n\n";
        $content .= "> {$site_description}\n\n";
        
        // Get dynamic content sections
        $content .= "## Key Content Sections\n";
        
        // Get published pages
        $pages = get_posts([
            'post_type' => 'page',
            'post_status' => 'publish',
            'numberposts' => -1,
            'meta_query' => [
                [
                    'key' => '_visibility',
                    'value' => 'public',
                    'compare' => '!='
                ]
            ]
        ]);
        
        foreach($pages as $page) {
            $slug = $page->post_name;
            $title = $page->post_title;
            $content .= "- [{$title}]({$site_url}/{$slug}.md)\n";
        }
        
        $content .= "\n## Blog Posts\n";
        $content .= "- [All Posts]({$site_url}/blog.md)\n";
        
        // Get recent posts for individual linking
        $recent_posts = get_posts([
            'post_type' => 'post',
            'post_status' => 'publish',
            'numberposts' => 10
        ]);
        
        foreach($recent_posts as $post) {
            $slug = $post->post_name;
            $title = $post->post_title;
            $content .= "- [{$title}]({$site_url}/blog/{$slug}.md)\n";
        }
        
        // Get custom post types
        $post_types = get_post_types(['public' => true, '_builtin' => false]);
        if (!empty($post_types)) {
            $content .= "\n## Additional Content\n";
            foreach($post_types as $post_type) {
                $type_obj = get_post_type_object($post_type);
                $content .= "- [{$type_obj->labels->name}]({$site_url}/{$post_type}.md)\n";
            }
        }

        file_put_contents(ABSPATH . 'llms.txt', $content);
        WP_CLI::success('llms.txt file generated successfully with dynamic content.');
    });
});

These approaches can help bridge the gap between WordPress's database driven content and the markdown based discovery that generative AI thrives on.

---

What's next

I’ll be publishing more soon about how and why I left WordPress, and how the switch to markdown-first content has changed my workflow. For now, this llms.txt implementation is a foundational step toward making carmelosantana.com more discoverable by generative tools.

If you’re curious about building this into your own site—WordPress or not—I’m available for consultation. Reach out if you’d like to explore what’s possible.