-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
Bump picomatch from 2.3.1 to 2.3.2 in /bindings/node
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#1980
opened Mar 26, 2026 by
dependabot
bot
Loading…
Bump picomatch from 2.3.1 to 2.3.2 in /tokenizers/examples/unstable_wasm/www
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#1979
opened Mar 25, 2026 by
dependabot
bot
Loading…
Bump flatted from 3.3.1 to 3.4.2 in /bindings/node
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#1972
opened Mar 20, 2026 by
dependabot
bot
Loading…
Fix node-release: all platforms, zig cross-compilation, universal macOS
#1970
opened Mar 19, 2026 by
MayCXC
Loading…
feat: performance, adding pcre2 backend + regex-shards (5-15% speedup)
#1968
opened Mar 19, 2026 by
michaelfeil
Loading…
feat: Optimize BPE tokenization: sharded cache, packed merge keys, FxHash (10-15% speedup)
#1967
opened Mar 19, 2026 by
michaelfeil
Loading…
feat: add async decode batch type annotations in decoding
#1966
opened Mar 19, 2026 by
michaelfeil
Loading…
Fix type_ids not applied to overflow encodings
#1965
opened Mar 17, 2026 by
joaquinhuigomez
Loading…
Bump minimatch from 3.1.3 to 3.1.5 in /bindings/node
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#1956
opened Feb 28, 2026 by
dependabot
bot
Loading…
Add riscv64 build, make Linux wheel build matrix more explicit
#1951
opened Feb 19, 2026 by
threexc
Loading…
Add get_special_tokens and is_special_token methods
#1945
opened Feb 5, 2026 by
ArthurZucker
Loading…
2 tasks done
Add post_process_tokens and post_process_ids methods
#1944
opened Feb 5, 2026 by
ArthurZucker
Loading…
3 tasks done
feat: add unk_token property to Unigram model
#1943
opened Feb 5, 2026 by
ArthurZucker
Loading…
4 tasks done
🚨 feat: add role_to_token field for special token metadata
#1942
opened Feb 5, 2026 by
ArthurZucker
Loading…
feat: add progress_format option for machine-readable JSON output
#1921
opened Dec 26, 2025 by
podarok
Loading…
6 tasks done
Use
unicode-normalization instead of unicode-normalization-alignments
#1912
opened Dec 14, 2025 by
IvanIsCoding
Loading…
Previous Next
ProTip!
no:milestone will show everything without a milestone.