Skip to content

Docs Search - Add script to generate openai embeddings of the markdown #2806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 28 commits into
base: main
Choose a base branch
from
Draft

Docs Search - Add script to generate openai embeddings of the markdown #2806

wants to merge 28 commits into from

Conversation

@NWylynko
Copy link
Contributor

@NWylynko NWylynko commented Nov 18, 2025 •
edited
Loading

Sibling PR
https://github.com/clerk/clerk/pull/1830

🔎 Previews:

What does this solve?

  • Embeddings is a AI powered tool that converts natural language into a set of numbers that represent that texts contextual meaning. This combined with a cosineSimilarity algorithm takes users queries and finds the most relevant information.

What changed?

  • Brings in a script that breaks up the markdown in to chunks and sends them over to openai api to create embeddings, saving them to a file called embeddings.json for clerk/clerk to pull in and use.

Checklist

  • I have clicked on "Files changed" and performed a thorough self-review
  • All existing checks pass

@NWylynko
Add vercel package for pulling project envs
8e105a9
@NWylynko
load in markdown and chunk it (still need to filter out undesired mar…
95efbec 
…kdown
@NWylynko
run the generate embeddings step
b28ece8
@NWylynko
Cut down the content being chunking by removing code examples and com…
207c920 
…ponents (we will want to refine this)
@NWylynko
Finish script, calling openai api to generate embeddings
08bd474
@NWylynko
Clean up the markdown for creating embeddings
5cb63c5
@NWylynko
Add a endpoint to query embeddings
14c72f3
@NWylynko
put the embeddings in to the api folder
e6d930b
@NWylynko
set tsconfig for api routes
1e8a2f2
@NWylynko
basic search handler
3ea5c39
@NWylynko
use type module
cf11d1a
@NWylynko
remove 'assert json'
8ef5c34
@NWylynko
use a ts file to import from
347cb26
@NWylynko
OOM during build
7251aaa
@NWylynko
just attempt to load from file system
34b3d7b
@NWylynko
list the file system
6785fbd
@NWylynko
list inside the api folder
17a9704
@NWylynko
Lets handle the search in clerk/clerk
7cdf24a
@NWylynko
clean up
f2d81dc
@NWylynko
Include more metadata for each chunk
2f3eb13
@NWylynko
move where we switch to paragraph type chunks
2a4d8ed
@NWylynko
include search ranking in chunk metadata
3c39912
@NWylynko
include search ranking
e6f2bb3
@NWylynko
Update encoding format in generate-embeddings script for optimized ne…
ee716dc 
…twork requests
@NWylynko
include both the available sdks and the active sdk for each chunk
ba6a7dd
@NWylynko
switch to using ai sdk
b2008f0
@NWylynko
drop embeddings dimensions down to 512 (1/3)
19f263e
@NWylynko
Remove vercel dependency, use ai@5 instead of beta
c5d524f
@vercel
Copy link

vercel bot commented Nov 18, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
clerk-docs Ready Ready Preview Nov 18, 2025 1:37pm

@NWylynko NWylynko mentioned this pull request Nov 21, 2025
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@NWylynko