Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Commit 6eb936b

Browse files
hiento09tikikunjan-hiro-v
authored
Implement Nitro on top of TensorRT-LLM C++ repo (#2)
* Rebase to rel branch * only build release * upgrade nitro * better example * latest demo * latest demo * remove redundant include * streaming working checkpoint * latest demo * openai compatible chat * feat: add splash screen and model load * change logo * remove redundant test * remove test.cc * Add Dockerfile and update cmakelist * Add Dockerfile for github action runner to build tensorrt llm * Correct SENTENCEPIECE path nitro cmakelist * Separate dockerfile for ada and ampere arch * Add CI for nitro tensorrt-llm windows ampere * Correct build script * Install nitro_deps instead of using cache * nitro deps build using cache * Fix error Longpath on windows * Fix error build nitro deps * nitro build_deps change to use bash * nitro build_deps change to use powershell * Add remove CMakeCache file * Add update CMakeCache.txt path * Change folder git to build CMAKEList * Add CI for build ada and ampere * Add CI release * Remove debug CI * Merge pull request #14 from janhq/10-epic-add-proper-handler-for-stop-words Add naive hiding stop words case --------- Co-authored-by: Hien To <tominhhien97@gmail.com> Co-authored-by: automaticcat <daogiatuank54@gmail.com> Co-authored-by: hiro <hiro@jan.ai>
1 parent 37aee91 commit 6eb936b

21 files changed

+2531
-14
lines changed

‎.github/runners/Dockerfile.window.runner-ada‎

Lines changed: 291 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,291 @@
1+
# Use the Windows Server Core 2019 image.
2+
# https://learn.microsoft.com/en-us/visualstudio/install/build-tools-container?view=vs-2022
3+
4+
# Use the Windows Server Core 2019 image.
5+
FROM mcr.microsoft.com/windows/servercore:ltsc2019
6+
7+
# Restore the default Windows shell for correct batch processing.
8+
# (Used for VS Build Tools installation)
9+
SHELL ["cmd", "/S", "/C"]
10+
11+
# -----------------------------------------------------------------------------
12+
13+
# Install CUDA 12.2
14+
15+
RUN powershell -Command \
16+
$ErrorActionPreference = 'Stop'; \
17+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/cuda_12.2.2_537.13_windows.exe \
18+
--output "cuda_installer.exe"; \
19+
Start-Process cuda_installer.exe -Wait -ArgumentList '-s'; \
20+
Remove-Item cuda_installer.exe -Force
21+
22+
# -----------------------------------------------------------------------------
23+
24+
# Install Python 3.10.11
25+
26+
# Download and install Python
27+
RUN powershell -Command \
28+
$ErrorActionPreference = 'Stop'; \
29+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/python-3.10.11-amd64.exe --output python-3.10.11.exe ; \
30+
Start-Process python-3.10.11.exe -Wait -ArgumentList '/quiet InstallAllUsers=1 PrependPath=1' ; \
31+
Remove-Item python-3.10.11.exe -Force
32+
33+
# Add python3 command
34+
RUN powershell -Command \
35+
cp "\"C:\\\\Program Files\\\\Python310\\\\python.exe\" \"C:\\\\Program Files\\\\Python310\\\\python3.exe\""
36+
37+
# -----------------------------------------------------------------------------
38+
39+
# Install Microsoft MPI
40+
41+
# The latest version is 10.1.3, but it requires you to get a temporary download
42+
# link.
43+
# https://learn.microsoft.com/en-us/message-passing-interface/microsoft-mpi-release-notes
44+
# We use 10.1.1 which has a release on the GitHub page
45+
RUN powershell -Command \
46+
$ErrorActionPreference = 'Stop'; \
47+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/msmpisetup.exe \
48+
--output "msmpisetup.exe"; \
49+
Start-Process .\msmpisetup.exe -Wait ; \
50+
Remove-Item msmpisetup.exe -Force
51+
52+
# Add MPI binaries to Path
53+
RUN setx Path "%Path%;C:\Program Files\Microsoft MPI\Bin"
54+
55+
# Download the MSMPI SDK
56+
RUN powershell -Command \
57+
$ErrorActionPreference = 'Stop'; \
58+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/msmpisdk.msi \
59+
--output "msmpisdk.msi"; \
60+
Start-Process msiexec.exe -Wait -ArgumentList '/I msmpisdk.msi /quiet'; \
61+
Remove-Item msmpisdk.msi -Force
62+
63+
# -----------------------------------------------------------------------------
64+
65+
# Install CMake
66+
67+
RUN powershell -Command \
68+
$ErrorActionPreference = 'Stop'; \
69+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/cmake-3.27.7-windows-x86_64.msi \
70+
--output "cmake.msi"; \
71+
Start-Process msiexec.exe -Wait -ArgumentList '/I cmake.msi /quiet'; \
72+
Remove-Item cmake.msi -Force
73+
74+
# Add CMake binaries to Path
75+
RUN setx Path "%Path%;C:\Program Files\CMake\bin"
76+
77+
# -----------------------------------------------------------------------------
78+
79+
# Install VS Build Tools
80+
81+
RUN \
82+
# Download the Build Tools bootstrapper.
83+
curl -SL --output vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe \
84+
\
85+
# Install Build Tools with the Microsoft.VisualStudio.Workload.AzureBuildTools workload, excluding workloads and components with known issues.
86+
&& (start /w vs_buildtools.exe --quiet --wait --norestart --nocache \
87+
--installPath "%ProgramFiles(x86)%\Microsoft Visual Studio\2022\BuildTools" \
88+
--includeRecommended \
89+
--add Microsoft.VisualStudio.Workload.MSBuildTools \
90+
--add Microsoft.VisualStudio.Workload.VCTools \
91+
--remove Microsoft.VisualStudio.Component.Windows10SDK.10240 \
92+
--remove Microsoft.VisualStudio.Component.Windows10SDK.10586 \
93+
--remove Microsoft.VisualStudio.Component.Windows10SDK.14393 \
94+
--remove Microsoft.VisualStudio.Component.Windows81SDK \
95+
|| IF "%ERRORLEVEL%"=="3010" EXIT 0) \
96+
\
97+
# Cleanup
98+
&& del /q vs_buildtools.exe
99+
100+
# -----------------------------------------------------------------------------
101+
102+
# Install Vim (can delete this but it's nice to have)
103+
104+
RUN powershell -Command \
105+
$ErrorActionPreference = 'Stop'; \
106+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/gvim90.exe \
107+
--output "install_vim.exe"; \
108+
Start-Process install_vim.exe -Wait -ArgumentList '/S'; \
109+
Remove-Item install_vim.exe -Force
110+
111+
# Add Vim binaries to Path
112+
RUN setx Path "%Path%;C:\Program Files (x86)\Vim\vim90"
113+
114+
# -----------------------------------------------------------------------------
115+
116+
# Install Chocolatey
117+
# Chocolatey is a package manager for Windows
118+
# I probably could've used it to install some of the above, but I didn't...
119+
120+
# If you try to install Chocolatey 2.0.0, it fails on .NET Framework 4.8 installation
121+
# https://stackoverflow.com/a/76470753
122+
ENV chocolateyVersion=1.4.0
123+
124+
# https://docs.chocolatey.org/en-us/choco/setup#install-with-cmd.exe
125+
RUN powershell -Command \
126+
$ErrorActionPreference = 'Stop'; \
127+
powershell.exe -NoProfile -InputFormat None -ExecutionPolicy Bypass \
128+
-Command "[System.Net.ServicePointManager]::SecurityProtocol = 3072; \
129+
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))" && \
130+
SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
131+
132+
# -----------------------------------------------------------------------------
133+
134+
# Install Git via Chocolatey
135+
RUN powershell -Command \
136+
choco install git -y
137+
138+
# -----------------------------------------------------------------------------
139+
140+
# Install CUDA 11.8 NVTX
141+
142+
#RUN powershell -Command \
143+
# $ErrorActionPreference = 'Stop'; \
144+
# curl.exe https://developer.download.nvidia.com/compute/cuda/11.8.0/network_installers/cuda_11.8.0_windows_network.exe \
145+
# --output "cuda_11_installer.exe"; \
146+
# Start-Process cuda_11_installer.exe -Wait -ArgumentList '-s nvtx_11.8'; \
147+
# Remove-Item cuda_11_installer.exe -Force
148+
149+
# The above command-line installation method installs NVTX headers at
150+
# C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include\nvtx3\
151+
# CMake can't find this location for some reason.
152+
# Instead, we just copy the older NvToolsExt version to where CMake expects.
153+
# This assumes NvToolsExt was installed on the host machine using the
154+
# CUDA 11.8 GUI installer and copied to the build context
155+
156+
# COPY ["NvToolsExt", "C:\\\\Program Files\\\\NVIDIA Corporation\\\\NvToolsExt"]
157+
RUN powershell -Command \
158+
$ErrorActionPreference = 'Stop'; \
159+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/NvToolsExt.zip \
160+
--output NvToolsExt.zip; \
161+
Expand-Archive .\NvToolsExt.zip -DestinationPath 'C:\Program Files\NVIDIA Corporation\'; \
162+
Remove-Item NvToolsExt.zip -Force
163+
164+
# -----------------------------------------------------------------------------
165+
166+
# Create a working directory
167+
WORKDIR "C:\\\\workspace"
168+
169+
# -----------------------------------------------------------------------------
170+
171+
# Download and unzip TensorrRT 9.2.0.5 for TensorRT-LLM
172+
RUN powershell -Command \
173+
$ErrorActionPreference = 'Stop'; \
174+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/TensorRT-9.2.0.5.Windows10.x86_64.cuda-12.2.llm.beta.zip \
175+
--output TensorRT-9.2.0.5.zip; \
176+
Expand-Archive .\TensorRT-9.2.0.5.zip -DestinationPath .; \
177+
Remove-Item TensorRT-9.2.0.5.zip -Force
178+
179+
# Add TensorRT libs to Path
180+
RUN setx Path "%Path%;C:\workspace\TensorRT-9.2.0.5\lib"
181+
182+
# Install TensorRT Python wheel
183+
RUN powershell -Command \
184+
$ErrorActionPreference = 'Stop'; \
185+
pip install TensorRT-9.2.0.5\python\tensorrt-9.2.0.post12.dev5-cp310-none-win_amd64.whl
186+
187+
# -----------------------------------------------------------------------------
188+
189+
# Copy cuDNN into the working directory
190+
# This assumes cuDNN exists on the host machine in the build context
191+
# COPY ["cuDNN", "cuDNN"]
192+
RUN powershell -Command \
193+
$ErrorActionPreference = 'Stop'; \
194+
curl.exe https://delta.jan.ai/dist/windows-container-dependencies/1/cuDNN.zip \
195+
--output cuDNN.zip; \
196+
Expand-Archive .\cuDNN.zip -DestinationPath .; \
197+
Remove-Item cuDNN.zip -Force
198+
199+
# Add cuDNN libs and bin to Path.
200+
RUN setx Path "%Path%;C:\workspace\cuDNN\lib;C:\workspace\cuDNN\bin;"
201+
202+
# -----------------------------------------------------------------------------
203+
204+
# Define the entry point for the docker container.
205+
# This entry point launches the 64-bit PowerShell developer shell.
206+
# We need to launch with amd64 arch otherwise Powershell defaults to x86 32-bit build commands which don't jive with CUDA
207+
ENTRYPOINT ["C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\Common7\\Tools\\VsDevCmd.bat", "-arch=amd64", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]
208+
209+
# -----------------------------------------------------------------------------
210+
211+
# Additional dependencies to build Nitro
212+
213+
# This bellow command lt MSVC recognize cuda compiler
214+
RUN powershell -Command \
215+
$ErrorActionPreference = 'Stop'; \
216+
Copy-Item -Path 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\extras\visual_studio_integration\MSBuildExtensions\*' -Destination 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations'
217+
218+
RUN powershell -Command \
219+
$ErrorActionPreference = 'Stop'; \
220+
Copy-Item -Path 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\extras\visual_studio_integration\MSBuildExtensions\*' -Destination 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v160\BuildCustomizations'
221+
222+
223+
# Set git safe directory for nitro clone dependencies
224+
RUN powershell -Command \
225+
git config --global --add safe.directory '*'
226+
227+
# Package for nitro compile
228+
RUN powershell -Command \
229+
choco install pkgconfiglite --allow-empty-checksums -y
230+
231+
RUN powershell -Command \
232+
choco install Ninja -y
233+
234+
RUN choco install 7zip -y; \
235+
7z --help
236+
237+
# Requirements to build tensorrt-llm on windows
238+
# COPY ./requirements-windows.txt ./tensorrt-llm-nitro/requirements-windows.txt
239+
# COPY ./requirements-dev-windows.txt ./tensorrt-llm-nitro/requirements-dev-windows.txt
240+
# RUN powershell -Command \
241+
# cd tensorrt-llm-nitro; \
242+
# pip install --no-cache-dir -r .\requirements-dev-windows.txt
243+
244+
# COPY ./.git ./tensorrt-llm-nitro/.git
245+
246+
# COPY ./3rdparty ./tensorrt-llm-nitro/3rdparty
247+
248+
# COPY ./cpp ./tensorrt-llm-nitro/cpp
249+
250+
RUN powershell -Command \
251+
$ErrorActionPreference = 'Stop'; \
252+
git clone https://github.com/janhq/nitro-tensorrt-llm.git; \
253+
cd nitro-tensorrt-llm; \
254+
git checkout tensorrt-llm-nitro-rel; \
255+
git submodule update --init --recursive; \
256+
pip install --no-cache-dir -r .\requirements-dev-windows.txt; \
257+
cd cpp/tensorrt_llm/nitro; \
258+
cmake -S ./nitro_deps -B ./build_deps/nitro_deps; \
259+
cmake --build ./build_deps/nitro_deps --config Release
260+
261+
RUN setx Path "%Path%;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools"
262+
263+
RUN VsDevCmd.bat -arch=amd64 && \
264+
powershell.exe -NoLogo -ExecutionPolicy Bypass "cd nitro-tensorrt-llm; python .\scripts\build_wheel.py -a '89-real' --trt_root 'C:\workspace\TensorRT-9.2.0.5\'"
265+
266+
# # -----------------------------------------------------------------------------
267+
268+
# Requirements to build tensorrt-llm on windows
269+
ARG RUNNER_VERSION=2.314.1
270+
271+
# Define the entry point for the docker container.
272+
# This entry point launches the 64-bit PowerShell developer shell.
273+
# We need to launch with amd64 arch otherwise Powershell defaults to x86 32-bit build commands which don't jive with CUDA
274+
# ENTRYPOINT ["C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\Common7\\Tools\\VsDevCmd.bat", "-arch=amd64", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]
275+
276+
RUN powershell -Command \
277+
$ErrorActionPreference = 'Stop'; \
278+
Invoke-WebRequest \
279+
-Uri https://github.com/actions/runner/releases/download/v$env:RUNNER_VERSION/actions-runner-win-x64-$env:RUNNER_VERSION.zip \
280+
-OutFile runner.zip; \
281+
Expand-Archive -Path ./runner.zip -DestinationPath ./actions-runner; \
282+
Remove-Item -Path .\runner.zip; \
283+
setx /M PATH $(${Env:PATH} + \";${Env:ProgramFiles}\Git\bin\")
284+
285+
ADD runner.ps1 ./runner.ps1
286+
287+
RUN powershell -Command New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force
288+
289+
RUN powershell -Command icacls 'C:\workspace\nitro-tensorrt-llm' /grant 'Everyone:F' /T
290+
291+
CMD ["powershell.exe", "-ExecutionPolicy", "Unrestricted", "-File", ".\\runner.ps1"]

0 commit comments

Comments
 (0)