Skip to content
This repository was archived by the owner on Feb 25, 2022. It is now read-only.

Commit d8ee63c

Browse files
authored
Update encoders.py
1 parent b2d82d3 commit d8ee63c

File tree

1 file changed

+0
-4
lines changed

1 file changed

+0
-4
lines changed

encoders.py

-4
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,6 @@ def fetch_encoder(params):
2222

2323
# GPT2Tokenizer and Tokenizer have different ways of fetching token ids
2424
def encode(encoder, text, gpt=True):
25-
if gpt:
26-
result = encoder.encode(text)
27-
else:
28-
result = encoder.encode(text)
2925
if isinstance(result, list):
3026
return result
3127
return result.ids

0 commit comments

Comments
 (0)