Accessing GPU in Google Colab with udocker

[yimingsun12138]

Hi udocker Team,

Firstly, I'd like to express my appreciation for your remarkable efforts on the udocker project. It has significantly streamlined my workflows as a data scientist, especially for running various containers on Google Colab.

I am currently exploring the possibility of leveraging Google Colab's GPU within a udocker container. I attempted to enable GPU access by using the udocker setup with Nvidia support via the following steps:

!apt install screen
!pip install udocker
!udocker --allow-root install
!udocker --allow-root pull nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04

!screen -d -m udocker --allow-root run nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 /bin/bash
!sleep 120

!udocker --allow-root ps

After retrieving the container ID, I proceeded with the Nvidia setup:

!udocker --allow-root setup --nvidia 62d47422-512d-3856-b1e5-55619651bbfd

However, I encountered some errors during the setup:

Info: Cont, has files from previous nvidia install
Error: creating nvidia dir /root/.udocker/containers/62d47422-512d-3856-b1e5-55619651bbfd/ROOT//etc/vulkan/icd.d
Warn: nvidia file in config not found /etc/vulkan/icd.d/nvidia_icd.json
Error: creating nvidia dir /root/.udocker/containers/62d47422-512d-3856-b1e5-55619651bbfd/ROOT//etc/OpenCL/vendors
Warn: nvidia file in config not found /etc/OpenCL/vendors/nvidia.icd
Info: nvidia mode set

I have no idea how to resolve these errors. Could you provide any guidance or suggestions to help address this issue? Any assistance would be greatly appreciated.

Thank you for your support and for continuing to improve this amazing tool.

[LYUYork]

how's it going?

[yimingsun12138]

how's it going?

No idea

[ghltshubh]

Any update on this @yimingsun12138

[SomeBottle]

Error: creating nvidia dir /root/.udocker/containers/62d47422-512d-3856-b1e5-55619651bbfd/ROOT//etc/vulkan/icd.d
Error: creating nvidia dir /root/.udocker/containers/62d47422-512d-3856-b1e5-55619651bbfd/ROOT//etc/OpenCL/vendors

In fact, the errors above are simply due to the absence of some directories on the host machine. The relevant Traceback is as follows:

Vulkan and OpenCL are not pre-installed in Google Colab, so this doesn't matter.

The real issue stems from the copying logic of the nvidia-related files. I've attempted to fix this in #438. Before it is merged, you can try to replace the udocker/engine/nvidia.py file with the one in the pull request. I hope the fix will work.

@yimingsun12138

---

Also you can try to use apptainer, it works in Google Colab, but you need to add nvidia library path to ldconfig manually to access GPU:

# /usr/lib64-nvidia from $LD_LIBRARY_PATH
# add it to ld conf
echo "/usr/lib64-nvidia" >> /etc/ld.so.conf.d/nvidia.conf
# update cache
ldconfig

To run apptainer by root user, you need to use unshare -r to gain capabilities.