Your message dated Thu, 29 Jan 2026 21:25:48 +0000
with message-id <[email protected]>
and subject line Bug#1109641: fixed in tokenizers 0.20.3+dfsg-1
has caused the Debian Bug report #1109641,
regarding ITP: tokenizers -- Provides an implementation of today's most used
tokenizers, with a focus on performance and versatility.
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1109641: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1109641
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: wnpp
Severity: wishlist
Owner: Kohei Sendai <[email protected]>
X-Debbugs-Cc: [email protected], [email protected]
* Package name : tokenizers
Version : 0.20.3
Upstream Contact: Nicolas Patry <[email protected]>
* URL : https://github.com/huggingface/tokenizers
* License : Apache-2.0
Programming Lang: Python
Description : Provides an implementation of today's most used tokenizers,
with a focus on performance and versatility.
Train new vocabularies and tokenize, using today's most used tokenizers.
Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text on a
server's CPU.
Easy to use, but also extremely versatile.
Designed for research and production.
Normalization comes with alignments tracking. It's always possible to get the
part of the original sentence that corresponds to a given token.
Does all the pre-processing: Truncate, Pad, add the special tokens your model
needs.
- Important package for vllm and used in many place with llm.
--- End Message ---
--- Begin Message ---
Source: tokenizers
Source-Version: 0.20.3+dfsg-1
Done: Kohei Sendai <[email protected]>
We believe that the bug you reported is fixed in the latest version of
tokenizers, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Kohei Sendai <[email protected]> (supplier of updated tokenizers package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Fri, 09 May 2025 14:01:31 +0000
Source: tokenizers
Binary: python3-tokenizers python3-tokenizers-dbgsym
Architecture: source amd64
Version: 0.20.3+dfsg-1
Distribution: experimental
Urgency: medium
Maintainer: Debian Deep Learning Team <[email protected]>
Changed-By: Kohei Sendai <[email protected]>
Description:
python3-tokenizers - Fast State-of-the-Art Tokenizers for Research and
Production
Closes: 1109641
Changes:
tokenizers (0.20.3+dfsg-1) experimental; urgency=medium
.
* Initial release. (Closes: #1109641)
Checksums-Sha1:
defba0f9c0c5978e043cd29c9f9a7cca05d6642c 2986 tokenizers_0.20.3+dfsg-1.dsc
82ac49bd47a2045467e9933a57af62ddfe880f76 1399061
tokenizers_0.20.3+dfsg.orig.tar.gz
c366982a72aa73919bafeaf937e999ba5d042689 4400
tokenizers_0.20.3+dfsg-1.debian.tar.xz
6de556f72c124993a9e303f481c535ef63bc1bc6 12156832
python3-tokenizers-dbgsym_0.20.3+dfsg-1_amd64.deb
62317a96fd2126552f3cfa1cc148f97a8450de85 1647224
python3-tokenizers_0.20.3+dfsg-1_amd64.deb
c89f772b09fb342b8cf71cdb6f11b8a13641ff1a 26340
tokenizers_0.20.3+dfsg-1_amd64.buildinfo
Checksums-Sha256:
5f41a5de3160dd89f0f2fa2799e4e9beef7479c68357c954f524b8e5a0a9e29a 2986
tokenizers_0.20.3+dfsg-1.dsc
1563753f7d5c8b945156444a9fdd0840345a2e820d2724acf1ea87996deb18eb 1399061
tokenizers_0.20.3+dfsg.orig.tar.gz
18fe1abadcd384ce79f9e2abeabb525bbf7087ca4a6c4d38fa97ca160136b55e 4400
tokenizers_0.20.3+dfsg-1.debian.tar.xz
0fc91073b4f2cb5218f110fa217d4adf042c340e318c6571daba09db9de85bd3 12156832
python3-tokenizers-dbgsym_0.20.3+dfsg-1_amd64.deb
21d393c3600988cbdb204c6a6a61c20abc410fd25ff5b2c92b45ffae0e6903be 1647224
python3-tokenizers_0.20.3+dfsg-1_amd64.deb
fd734f1ea41e078019460345061596448b07bad6a86d78e4eded0a5854c2d666 26340
tokenizers_0.20.3+dfsg-1_amd64.buildinfo
Files:
f04d31b9a65d1e8fa6ec899b35184074 2986 python optional
tokenizers_0.20.3+dfsg-1.dsc
469ab9768f74bc6614add272dcce2dab 1399061 python optional
tokenizers_0.20.3+dfsg.orig.tar.gz
f0ed573bdea549183ca5acabe2ca234c 4400 python optional
tokenizers_0.20.3+dfsg-1.debian.tar.xz
56584ec25a84d995debcd58ca03a7aac 12156832 debug optional
python3-tokenizers-dbgsym_0.20.3+dfsg-1_amd64.deb
144951e1fd8eb7ddadeab88ee8013483 1647224 python optional
python3-tokenizers_0.20.3+dfsg-1_amd64.deb
2676f63c2cc2468760193659a16a3fbf 26340 python optional
tokenizers_0.20.3+dfsg-1_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJFBAEBCgAvFiEEY4vHXsHlxYkGfjXeYmRes19oaooFAmh/nkwRHGx1bWluQGRl
Ymlhbi5vcmcACgkQYmRes19oaooHPA/8CUupADzzog6BDcEHw7OL0rIItMaRTw8S
ggbJZLTGApIo7k4G84rcB+pJF2okU9lZXIPBny+1V9rLzN0GMi+h2BTLrILKqXG6
EYasDjF9zJRIIfsejxyawqiIxPO6dV+RaH5uFFrtCUCi4Jd7SHp+sUUcaBOJvC8t
k99DgA/AsEW3pIrn/fDAqfraqSiI1AUWtF6tBUv13ifhwpiFYzwDdnLK3+FxrdaC
ATwVjmTloQgujhD8IzJpUmSzaFGdQcAGBUOvSoJ2Vz/a2XxECw+u21kWUd+0isUZ
Zsn6SP8NUDg5v0EttsvEWANCq0N3yOJTQlDReyX7f5Drt5vm/0aRp05d1Z3CwoS+
VorGVSOGncdg77r5KOuUKtVg++IlZvuCIG09N8LDdy63cfBZjz+hzl+rP1B1DSKt
dMbyKpdh3NisAeZb0Xmk0Qc/pDMpznPQuFXpRKgY53xiDsMS1+lZ887EIc8ToKpf
wNAIbV8ahVllA6OlOGK2H/LK2Bo0mVbjswhA9GKwptZ7YyO/QIfbpA6AyG5Ldagx
GGJt7ySEXb6B3TdFx5NfsHWO1PRb0lyYWZQi/nJWqOhkYg97hTI5QfmKiWVegpU9
aitAUSxxcmSCWhOhAQw0cZlgsz7f9oQ1bDQriHOa+ezlkk/j6BY8GAJkHnYvXeA0
LMLiiIeFJFk=
=e75Q
-----END PGP SIGNATURE-----
pgphQ5U6hxsy7.pgp
Description: PGP signature
--- End Message ---
