`Model.from_pretrained` breaks when using SinusoidalEmbedding

[ZhiyuanChen]

System Info

• transformers version: 4.51.0
• Platform: macOS-15.3.1-arm64-arm-64bit
• Python version: 3.12.9
• Huggingface_hub version: 0.30.2
• Safetensors version: 0.5.3
• Accelerate version: 1.6.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (GPU?): 2.6.0 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed

Who can help?

SinusoidalEmbedding does not require state_dict, and since there are some bugs related to loading/saving its states (#31387), a work around is to override its related functions:

    def state_dict(self, destination=None, prefix="", keep_vars=False):
        return {}

    def load_state_dict(self, *args, state_dict, strict=True):
        return

    def _load_from_state_dict(
        self, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs
    ):
        return

This used to works, but a recent update breaks it (Going back to transformers 4.50 worked fine).

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

pip install multimolecule

Failure:

from transformers import AutoConfig, AutoModel, AutoTokenizer, pipeline
from multimolecule.models import ErnieRnaForSecondaryStructurePrediction as Model

model = Model.from_pretrained("multimolecule/ernierna-ss")

model.to("cuda")

NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Works:

from transformers import AutoConfig, AutoModel, AutoTokenizer, pipeline
from multimolecule.models import ErnieRnaForSecondaryStructurePrediction as Model

model = Model(AutoConfig.from_pretrained("multimolecule/ernierna-ss"))

model.to("cuda")

Expected behavior

No Error

[ZhiyuanChen]

@ArthurZucker Any thoughts?

[Rocketknight1]

Hi @ZhiyuanChen, it seems like you want the SinusoidalEmbedding class to have temporary weights that are not saved. Have you tried using register_buffer(persistent=False)? If you do that, our methods should hopefully stop being confused about what to do with those parameters

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.