feat: Add dynamic markdown routing for llms.txt

[artimath]

⚠️ Stacked PR: This builds on top of #8130 - review that one first, then review only the incremental changes here.

Adds dynamic markdown routing, enabling AI agents to fetch individual doc pages as markdown via clean URLs.

Summary

• Adds /api/raw-md/[[...slug]] API route to serve markdown files dynamically
• Implements rewrite rules so /:path*.md routes to the API endpoint
• Refactors llms.txt generation to use shared utilities for markdown processing
• Note: llms.txt links still point to HTML URLs; will update to .md format in follow-up commit

Changes Beyond Base PR (#8130)

Dynamic markdown API (src/pages/api/raw-md/[[...slug]].ts):

• Catch-all route handler resolving slugs to content files
• Tries multiple candidates (.md, .mdx, /index.md, /index.mdx)
• Proper cache headers (60s public, 10m CDN, 24h SWR)

Refactored generation (src/lib/llms-utils.ts):

• Extracts reusable utilities from base PR's generation script
• Markdown processing pipeline using remark/mdx
• Custom MDX component stripping (removes <Intro>, <Sandpack>, etc)
• Doc discovery, deduplication, section ordering

Updated generation (src/utils/llms.js):

• Simplified to use new shared utilities

Routing config (next.config.js):

• Rewrites /:path*.md → /api/raw-md/:path*

Examples

With this PR, AI agents can:

• Fetch /learn.md for the learn index page
• Fetch /reference/react/hooks.md for specific API docs
• Follow links from future .md format llms.txt

Incremental Changes (vs #8130)

View only this commit to see the delta: 9cd0a28

Files Changed vs Base

• next.config.js: rewrite rules
• src/lib/llms-utils.ts: shared markdown utilities (new)
• src/pages/api/raw-md/[[...slug]].ts: dynamic route handler (new)
• src/utils/llms.js: refactored to use shared utils
• yarn.lock: updated dependencies

Known Issues

• Build currently fails with Node v22 + Yarn PnP loader compatibility issue (investigating)
• Need to update llms.txt generation to output .md links instead of HTML URLs

Test Plan

• Verify /learn.md returns markdown
• Verify /reference/react/hooks.md works
• Check nested paths like /reference/react-dom/client/createRoot.md
• Verify cache headers present
• Ensure llms.txt generation still works with refactored utils
• yarn build completes successfully

[github-actions]

Size changes

📦 Next.js Bundle Analysis for react-dev

This analysis was generated by the Next.js Bundle Analysis action. 🤖

⚠️ Global Bundle Size Increased

Page	Size (compressed)
global	115.14 KB (🟡 +4.6 KB)

Details

The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!