Skip to content
This repository was archived by the owner on Feb 25, 2022. It is now read-only.

Commit 306af79

Browse files
authored
Update encoders.py
1 parent d8ee63c commit 306af79

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

encoders.py

+1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ def fetch_encoder(params):
2222

2323
# GPT2Tokenizer and Tokenizer have different ways of fetching token ids
2424
def encode(encoder, text, gpt=True):
25+
result = encoder.encode(text)
2526
if isinstance(result, list):
2627
return result
2728
return result.ids

0 commit comments

Comments
 (0)