Configuration Parsing Warning:Config file tokenizer_config.json cannot be fetched (too big)
Cantonese Tokenizer
This is a Cantonese sentence tokenizer based on BART Chinese. It can be used along with our CCPC Parallel Corpus dataset.
- Token Count: 161,278 (including 110,007 multi-character Chinese vocabularies)
Usage
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("raptorkwok/pyctokenizer")
print(tokenizer.tokenize("我哋去咗尖沙咀睇醫生呀！"))
# Output: ['我哋', '去', '咗', '尖沙咀', '睇醫生', '呀', '！']
- Downloads last month
- -
Model tree for raptorkwok/pyctokenizer
Base model
OpenMOSS-Team/bart-base-chinese