[Docs] add docs for cuda graph v1 #24374

fhl2000 · 2025-09-06T18:35:46Z

Purpose

Add design documents for the changes in #20059.

Previews:
https://vllm--24374.org.readthedocs.build/en/24374/design/cuda_graphs.html
https://vllm--24374.org.readthedocs.build/en/24374/design/torch_compile.html

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: fhl <2410591650@qq.com>

gemini-code-assist

Code Review

This pull request adds a new design document for CUDA Graph v1, which is a great addition for understanding the new features. The document is comprehensive and well-structured. My main feedback is on the language quality of the new document, which has numerous typos and grammatical errors. I've left a single comment with a list of examples. Fixing these will significantly improve the document's quality. The update to torch_compile.md to link to this new document is appropriate.

docs/design/cuda_graphs_v1.md

Signed-off-by: fhl <2410591650@qq.com>

hmellor

Thank you for the PR! I've not looked closely at the content of the document yet but I have left some high level comments that can be actioned in the meantime.

docs/design/cuda_graphs_v1.md

docs/assets/design/cuda_graphs_v1/current_design.jpg

docs/assets/design/cuda_graphs_v1/executor_runtime.jpg

docs/assets/design/cuda_graphs_v1/previous_design.jpg

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

hmellor

With V0 effectively gone, we can drop V1 from the title

docs/design/cuda_graphs.md

docs/design/torch_compile.md

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

Signed-off-by: fhl <2410591650@qq.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor

Could you remove the Pre: and Now: labels baked into the images and use Before: and After: in the markdown content to introduce the images?
Could you also not truncate PIECEWISE to PIECE.? IKt's not explained anywhere and only saves 3 characters, it's better to be explicit.

docs/design/cuda_graphs.md

fhl2000 · 2025-09-15T15:44:39Z

Why did the pre-commit fail? @hmellor

Unordered list style [Expected: dash; Actual: asterisk]

I remember previously it asked for asterisk instead of dash.

hmellor · 2025-09-15T15:46:45Z

Why did the pre-commit fail? @hmellor
Unordered list style [Expected: dash; Actual: asterisk]
I remember previously it asked for asterisk instead of dash.

Sorry this was my mistake. pre-commit wants the style to be the same throughout the document and it chooses whichever is used first. In my formatting change I added an unordered list and used a different symbol to the one you used

docs/design/cuda_graphs.md

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

fhl2000 · 2025-09-15T16:00:49Z

Is this ok? The truncated PIECE. in some places are kept for cleanliness (not much room to add)

docs/design/cuda_graphs.md

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2025-09-15T16:09:47Z

Is this ok? The truncated PIECE. in some places are kept for cleanliness (not much room to add)

These new diagrams do look very nice. It does look like PIECEWISE should fit in the CUDAGraphWrapper boxes. For the data flows at the bottom could the PIECEWISE be expanded to the left?

Signed-off-by: fhl <2410591650@qq.com>

fhl2000 · 2025-09-15T16:53:51Z

Any thoughts on this new pre-commit error?

Error: docs/design/cuda_graphs.md:63:12 MD026/no-trailing-punctuation Trailing punctuation in heading [Punctuation: ':']

Maybe just remove : or highlight it but not a heading?

--
Edited: it's fixed.

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fhl2000 · 2025-09-20T17:27:45Z

The notes for attention ops fusion is changed, since #24281 merged!

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fhl2000 · 2025-09-27T04:30:19Z

Have had some updates since #23046 and #25444 were merged.

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

simon-mo · 2025-09-27T15:43:59Z

@codex review

chatgpt-codex-connector · 2025-09-27T15:46:28Z

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting

@codex fix this CI failure
@codex address that feedback

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

My formatting requests have been addressed

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

ProExpertProg

Thanks for this increadible writeup! A few thoughts & suggestions below

docs/design/cuda_graphs.md

Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

ProExpertProg

Great work, thanks for writing this up!

Signed-off-by: fhl <2410591650@qq.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

Signed-off-by: fhl <2410591650@qq.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Signed-off-by: fhl <2410591650@qq.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>

Signed-off-by: fhl <2410591650@qq.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

Signed-off-by: fhl <2410591650@qq.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Signed-off-by: fhl <2410591650@qq.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

drafting docs for cuda graph v1

1d3153b

Signed-off-by: fhl <2410591650@qq.com>

fhl2000 requested a review from hmellor as a code owner September 6, 2025 18:35

mergify bot added the documentation Improvements or additions to documentation label Sep 6, 2025

gemini-code-assist bot reviewed Sep 6, 2025

View reviewed changes

docs/design/cuda_graphs_v1.md Outdated Show resolved Hide resolved

fhl2000 added 2 commits September 8, 2025 00:30

fix typos and minor polish

6a830d1

Signed-off-by: fhl <2410591650@qq.com>

fix broken table

88d7346

Signed-off-by: fhl <2410591650@qq.com>

hmellor requested changes Sep 8, 2025

View reviewed changes

address comments

ccc44f6

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

hmellor reviewed Sep 8, 2025

View reviewed changes

docs/design/cuda_graphs.md Outdated Show resolved Hide resolved

docs/design/torch_compile.md Outdated Show resolved Hide resolved

fhl2000 and others added 10 commits September 8, 2025 12:17

minor

c12b82d

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fix pre-commit

35a0c54

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fix pre-commit again

631a8da

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

replace two images

e75a642

Signed-off-by: fhl <2410591650@qq.com>

replace one image

c3f8115

Signed-off-by: fhl <2410591650@qq.com>

Merge branch 'main' into cudagraph_mode_docs

614e126

small fixing of torch_compile.md

a02eb23

Signed-off-by: fhl <2410591650@qq.com>

Move assets

a582107

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Formatting and CUDA Graphs consistency

f723640

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Comment link formatting

7ef8153

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor previously requested changes Sep 15, 2025

View reviewed changes

docs/design/cuda_graphs.md Outdated Show resolved Hide resolved

hmellor reviewed Sep 15, 2025

View reviewed changes

docs/design/cuda_graphs.md Outdated Show resolved Hide resolved

Fix pre-commit

8752c24

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor reviewed Sep 15, 2025

View reviewed changes

docs/design/cuda_graphs.md Outdated Show resolved Hide resolved

pre-commit again...

0dd161d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

address comments

50a73cb

Signed-off-by: fhl <2410591650@qq.com>

fhl2000 added 2 commits September 19, 2025 16:48

fix links

8c2b392

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

modify notes for attn_ops fusion

0db6e26

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fhl2000 added 3 commits September 20, 2025 17:33

update aiter_fa cudagraph_support

08292de

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

add some recent updates

7813f6d

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

small fix

9e549c8

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fhl2000 requested a review from ProExpertProg September 27, 2025 04:31

small

9a5adf9

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

small

d53da8c

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

Merge branch 'main' into cudagraph_mode_docs

6e7e01c

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

ProExpertProg reviewed Oct 6, 2025

View reviewed changes

docs/design/cuda_graphs.md Outdated Show resolved Hide resolved

fhl2000 and others added 4 commits October 7, 2025 09:18

Update docs/design/cuda_graphs.md

99b3eb6

Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

Apply suggestions from code review

1d57323

Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

adapt from review suggestions

f8dc933

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

fix default

2f5586b

Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>

ProExpertProg approved these changes Oct 7, 2025

View reviewed changes

vllm-bot merged commit 63773a6 into vllm-project:main Oct 7, 2025
5 checks passed

Uh oh!

[Docs] add docs for cuda graph v1 #24374

[Docs] add docs for cuda graph v1 #24374

Uh oh!

Conversation

fhl2000 commented Sep 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fhl2000 commented Sep 15, 2025

Uh oh!

hmellor commented Sep 15, 2025

Uh oh!

Uh oh!

fhl2000 commented Sep 15, 2025

Uh oh!

Uh oh!

hmellor commented Sep 15, 2025

Uh oh!

fhl2000 commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fhl2000 commented Sep 20, 2025

Uh oh!

fhl2000 commented Sep 27, 2025

Uh oh!

simon-mo commented Sep 27, 2025

Uh oh!

chatgpt-codex-connector bot commented Sep 27, 2025

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fhl2000 commented Sep 6, 2025 •

edited by github-actions bot

Loading

fhl2000 commented Sep 15, 2025 •

edited

Loading