Skip to content

Files

Latest commit

c21bfe4 · Mar 24, 2025

History

History

09_extending-tokenizers

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Jan 22, 2025
Mar 24, 2025

Extending the Tiktoken BPE Tokenizer with New Tokens

  • extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly