ValueError: The checkpoint you are trying to load has model type `vargpt_qwen2_vl` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[LucasShi1101]

[you selected Infinity with model_kwargs={'depth': 32, 'embed_dim': 2048, 'num_heads': 16, 'drop_path_rate': 0.1, 'mlp_ratio': 4, 'block_chunks': 8}] model size: 2.20B, bf16=1
New model embedding size before resize: 152064
Original tokenizer size before adding tokens: 151660
Traceback (most recent call last):
File "/home/cshi0459/miniconda3/envs/VARGPT2/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1034, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/home/cshi0459/miniconda3/envs/VARGPT2/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 736, in getitem
raise KeyError(key)
KeyError: 'vargpt_qwen2_vl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cshi0459/3D/VARGPT-v1.1/inference_v1_1/generation_vargpt_v1_1.py", line 19, in
prepare_vargpt_qwen2vl_v1_1(model_id)
File "/home/cshi0459/3D/VARGPT-v1.1/vargpt_qwen_v1_1/prepare_vargpt_v1_1.py", line 126, in prepare_vargpt_qwen2vl_v1_1
original_model = AutoModelForVision2Seq.from_pretrained(
File "/home/cshi0459/miniconda3/envs/VARGPT2/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/home/cshi0459/miniconda3/envs/VARGPT2/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1036, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type vargpt_qwen2_vl but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[Fangyi-Chen]

Just move the registration block above the AutoModelForVision2Seq.from_pretrained() call.

Here’s exactly how your function should look:

def prepare_vargpt_qwen2vl_v1_1(save_path=vargpt_save_path, prepared_modules=["model", "tokenizer", "processor", "image_processor"], device=None):
if device is None:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

existsed = check_file_exists(save_path, "config.json")

if existsed:
    vargpt_qwen2vl_config = VARGPTQwen2VLConfig.from_pretrained(save_path)
else:
    vargpt_qwen2vl_config = VARGPTQwen2VLConfig(**cfg)

# Register early so AutoModelForVision2Seq can recognize your model
AutoConfig.register(vargpt_qwen2vl_config.model_type, VARGPTQwen2VLConfig)
AutoModelForVision2Seq.register(VARGPTQwen2VLConfig, VARGPTQwen2VLForConditionalGeneration)
AutoImageProcessor.register(VARGPTQwen2VLConfig, image_processor_class=VARGPTQwen2VLImageProcessor)
AutoProcessor.register(VARGPTQwen2VLConfig, processor_class=VARGPTQwen2VLProcessor)