:tocdepth: 2 .. _module-pw_tokenizer-api: ============= API reference ============= .. pigweed-module-subpage:: :name: pw_tokenizer ----- C/C++ ----- Moved: :doxylink:`pw_tokenizer` ---- Rust ---- See `Crate pw_tokenizer `_. ------ Python ------ Tokenization ============ .. autofunction:: pw_tokenizer.encode.encode_token_and_args .. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash Detokenization ============== .. automodule:: pw_tokenizer.detokenize :members: .. automodule:: pw_tokenizer.proto :members: