C and C++ bindings to Tokenizers #1888
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
very open to this! let's make sure we have a big compat with expectations in terms of the funcs we bind
| } | ||
|
|
||
| #[no_mangle] | ||
| pub extern "C" fn tokenizers_encode( |
There was a problem hiding this comment.
think we need decode, batch encode batch encode fast etc and asyc? I don't know how best to define these all but basically the same surface as python!
There was a problem hiding this comment.
Thanks for the quick review! Happy to know you're interested/open to this effort.
I will get back after surfacing more functionality to C++.
CPP bindings coverage improvement
fixed benchmarks
- Introduced a new submodule for Jinja2Cpp to handle chat template rendering. - Enhanced the C++ bindings to load and apply chat templates from a configuration file. - Added methods to retrieve special tokens and their IDs from the tokenizer configuration. - Updated the CMake configuration to include Jinja2Cpp and link it with the tokenizers_cpp library. - Refactored tests to validate the new chat template functionality and special token handling.
Support chat template in c++ api
|
Hi @ArthurZucker I've made some progress with c++ adding more APIs. I tried integrating https://github.com/jinja2cpp/Jinja2Cpp/ at c++ bindings layer, but some features are limited and lead to crash (IIRC, negative index like Any tips/recommendation for handling templates natively would be great. |
4 participants
Adding in bindings for two more languages!
bindings/cppbindings/cC is an intermediate step to bind C++ and Rust: i.e., C++ <--> C <--> Rust.
--