tiktoken

About this project

⏳ tiktoken tiktoken is a fast BPE tokeniser for use with OpenAI's models. The open source version of tiktoken can be installed from PyPI: The tokeniser API is documented in tiktoken/core.py. Example code using tiktoken can be found in the OpenAI Cookbook. Performance tiktoken is between 3-6x faster than a comparable open source tokeniser: Performance measured on 1GB of text using the GPT-2 tokeniser, using GPT2TokenizerFast from tokenizers==0.13.2, transformers==4.24.0 and tiktoken==0.2.0. Getting help Please post questions in the issue tracker. If you work at OpenAI, make sure to check the internal documentation or feel free to contact @shantanu. What is BPE…

GitHub

Reviews

Built by

openai

Imported from GitHub · not yet claimed on GitPalace

View developer pageSign in with GitHub to claim

Maintain openai/tiktoken? Claiming verifies admin access through your GitHub account and gives you control of this listing.

You might also like

build-your-own-x

Developer Tools

Master programming by recreating your favorite technologies from scratch.

★ New545.6K stars

by codecrafters-io

awesome

Developer Tools

😎 Awesome lists about all kinds of interesting topics [NOTE: Pull requests are temporarily disabled until I have a chan

★ New503.7K stars

by sindresorhus

public-apis

Developer Tools

A collective list of free APIs

★ New476.5K stars

by public-apis

free-programming-books

Developer Tools

:books: Freely available programming books

★ New396.1K stars

by EbookFoundation