Skip to content

Conversation

@b00f
Copy link
Contributor

@b00f b00f commented Aug 6, 2025

This PR fixes #1825 by introducing WhitespaceOptimized struct.

@b00f b00f force-pushed the feat/whitespace-optimize branch from 971be68 to f1bd5b0 Compare August 6, 2025 09:03
@b00f b00f changed the title Feat/whitespace optimize feat: whitespace optimize Aug 6, 2025
Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is 10-30% speed boost very nice but needs to be tested for exactitude -> xnli is a good dataset for that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: Replace regex in whitespace.rs with manual code for speed improvements

2 participants