fix: cleanup modifications in trying fix HF bug for GH actions

poetry.lock

pyproject.toml

tests/unit/encoders/test_clip.py

tests/unit/encoders/test_huggingface.py

tests/unit/encoders/test_vit.py