运行召回任务tokenizer_config.json出现报错

[ypJiangz]

我是把hf上的https://huggingface.co/linear-moe-hub/MoM-Gated-Deltanet-340M/tree/main
手动下载上传，然后运行单个任务进行测试，
modelpath下面的模型权重文件
ls -la
total 1068206
drwxrwxr-x 4 ranwh27 ranwh27 10 Dec 28 12:45 .
drwxrwxr-x 18 ranwh27 ranwh27 24 Dec 28 12:08 ..
drwxrwxr-x 2 ranwh27 ranwh27 1 Dec 28 13:40 .ipynb_checkpoints
-rw-rw-rw- 1 ranwh27 ranwh27 310 Dec 27 13:09 README.md
drwxrwxr-x 2 ranwh27 ranwh27 0 Dec 28 12:45 based_squad
-rw-rw-rw- 1 ranwh27 ranwh27 927 Dec 27 13:08 config.json
-rw-rw-r-- 1 ranwh27 ranwh27 111 Dec 27 13:08 generation_config.json
-rw-rw-rw- 1 ranwh27 ranwh27 1519 Dec 27 13:09 gitattributes
-rw-rw-r-- 1 ranwh27 ranwh27 1090330976 Dec 27 13:28 model.safetensors
-rw-rw-r-- 1 ranwh27 ranwh27 437 Dec 27 13:08 special_tokens_map.json
-rw-rw-r-- 1 ranwh27 ranwh27 3505751 Dec 27 13:08 tokenizer.json
-rw-rw-r-- 1 ranwh27 ranwh27 1027 Dec 27 13:08 tokenizer_config.json
指令和报错如下
CUDA_VISIBLE_DEVICES=0 python launch_local.py
--batch-size 64
-t based_squad
-m /modepath
--context_length 512
--answer_length 48
--cutting_context
--limit 64
Running sweep with 1 configs
2025-12-28:13:50:14,312 INFO [main.py:241] Verbosity set to INFO
2025-12-28:13:50:14,312 INFO [init.py:358] lm_eval.tasks.initialize_tasks() is deprecated and no longer necessary. It will be removed in v0.4.2 release. TaskManager will instead be used.
2025-12-28:13:50:14,318 WARNING [main.py:253] --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2025-12-28:13:50:14,318 INFO [main.py:333] Selected Tasks: ['based_squad']
2025-12-28:13:50:14,318 INFO [main.py:334] Loading selected tasks...
2025-12-28:13:50:14,318 INFO [evaluator.py:105] Setting random seed to 0
2025-12-28:13:50:14,318 INFO [evaluator.py:109] Setting numpy seed to 1234
2025-12-28:13:50:14,319 INFO [evaluator.py:113] Setting torch manual seed to 1234
Traceback (most recent call last):
File "/home/ranwh27/.local/bin/lm_eval", line 8, in
sys.exit(cli_evaluate())
^^^^^^^^^^^^^^
File "/home/ranwh27/MoM/lm-eval-harness/lm_eval/main.py", line 336, in cli_evaluate
results = evaluator.simple_evaluate(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/MoM/lm-eval-harness/lm_eval/utils.py", line 288, in _wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/MoM/lm-eval-harness/lm_eval/evaluator.py", line 133, in simple_evaluate
lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/MoM/lm-eval-harness/lm_eval/api/model.py", line 134, in create_from_arg_string
return cls(**args, **args2)
^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/MoM/lm-eval-harness/lm_eval/models/local_lm.py", line 17, in init
tokenizer = load_tokenizer(checkpoint_name, is_hf=is_hf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/MoM/lm-eval-harness/lm_eval/models/local_utils/loading.py", line 105, in load_tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 154, in init
super().init(
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 133, in init
slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama.py", line 171, in init
self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama.py", line 198, in get_spm_processor
tokenizer.Load(self.vocab_file)
File "/home/ranwh27/.local/lib/python3.11/site-packages/sentencepiece/init.py", line 961, in Load
return self.LoadFromFile(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ranwh27/.local/lib/python3.11/site-packages/sentencepiece/init.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: not a string
Decoded with mode: default

[ypJiangz]

再次检查了代码，大概问题是loading.py文件的
def load_tokenizer(model_name: str, is_hf: bool=False) -> nn.Module:
if not is_hf:
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.model_max_length = 2048
else:
if "mamba" in model_name or "mpt" in model_name:
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
else:
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad_token_id = tokenizer.eos_token_id
return tokenizer
读取tokenizer，使用的是AutoTokenizer，应该识别到的就是slow，模型权重数据中，tokenizer_config.json设置了"tokenizer_class": "LlamaTokenizer", 但是模型权重数据给出的是safetensor文件，这个应该是适用于fast的tokenizer，目前对代码做了一些简单的修改，还没有生效，想请问下是我的复现步骤出现了什么问题吗

[JusenD]

Hi~感谢关注，这个问题似乎是因为没有tokenizer.model导致的，请您尝试下载使用https://huggingface.co/fla-hub/gsa-1.3B-100B/blob/main/tokenizer.model

[ypJiangz]

好滴感谢，我来尝试一下