[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

n-poulsen · 2025-02-28T10:58:21Z

This pull requests updates DeepLabCut-Live for models exported with DeepLabCut 3.0. TensorFlow models can still be used, and the code is siloed so that only the engine used to run the code is required as a package (i.e. no need to install TensorFlow if you want to run live pose estimation with PyTorch models).

If you want to give this PR a try, you can install the code in your local conda environment by running:

pip install "git+https://github.com/DeepLabCut/DeepLabCut-live.git@dlclive3"

…in code

…NX inference

…erence

This commit corrects the parameter names and order and removes unsupported arguments from `benchmark_videos` call: - device - precision - draw_keypoint_names - get_sys_info

- pyproject.toml is made PEP 517 / 518 / 621–compliant, which makes it easy to inherit dependencies - it is more tool-agnostic (no poetry required & facilitates uv packaging). - Python versions 3.10 up to 3.12 - tensorflow installation is restricted to python < 3.12 for windows

Co-authored-by: Copilot <[email protected]>

run with uv instead of poetry and for python 3.10 up to 3.12

- Remove upper bound for tensorflow version is python version >=3.11 - add pip as required dependency

remove tkinter as required dependency.

(remove conditional tf dependency install for for windows and python >= 3.11, since never reached)

…liant *Update pyproject.toml and CI* - pyproject.toml is made PEP 517 / 518 / 621–compliant, which makes it easy to inherit dependencies - it is more tool-agnostic (no poetry required & facilitates uv packaging). - Python versions 3.10 up to 3.12 - Tensorflow installation is restricted to python < 3.11 for windows - CI is updated to test more Python versions and to use uv instead of poetry.

Added a more robust downloading function in `utils.py` that handles errors more gracefully. The download is skipped if the file already exists. In previous implementation, the benchmarking video was downloaded to the home directory, which in windows could result in permission issues when reading the file. This is now changed to the check_install directory. OpenCV was silently failing to read the video file, resulting in a frame count of zero. Now a ValueError is raised when the video cannot be read.

- removed poetry instructions - added optional instructions for installation with uv - added notes on tensorflow support for windows

deruyter92

Looked trough most of the changes and addressed the merge conflicts. I think it would be great if we could merge this WIP as soon as possible, so that we can cleanly contribute aditional PRs

What I have worked on:

Merged with main (accepting combination of both, but mostly dlclive3 branch for updated code)
Complete update of pyproject.toml
- more tool-agnostic (no poetry required)
- uv-friendly, so the user can choose their own preferred packaging method
- updated missing dependencies
- Python versions 3.10, 3.11, 3.12 (except for tensorflow users on windows)
- restrict windows users to python 3.10 for tensorflow
Changed the CI accordingly to test for multiple python versions.
CI uses uv now for installation.
Added some basic testing
Adressed copilot and other comments

@sneakers-the-rat, @MMathisLab, @C-Achard please let me know what you think and if you'd like me to incorporate additional changes.

deruyter92 · 2026-01-14T15:37:10Z

.github/workflows/testing.yml

+      - name: Install and test
        shell: bash -el {0}  # Important: activates the conda environment
        run: |
          conda install pytables==3.8.0 "numpy<2"


Thanks. This CI workflow is updated now. Let me know what you think!

deruyter92 · 2026-01-14T15:38:24Z

pyproject.toml


 [tool.poetry.dependencies]
-python = ">=3.9,<3.12"
+python = ">=3.10,<3.12"


I've been testing with 3.12 recently. It seems to work fine, except on windows with tensorflow. I've updated the pyproject.toml. Let me know if you have suggestions!

deruyter92 · 2026-01-14T15:38:39Z

pyproject.toml

-tables = "^3.6"
+pandas = ">=1.0.1,!=1.5.0"
+tables = "^3.8"
 pytest = "^8.0"


C-Achard · 2026-01-14T17:48:19Z

@deruyter92 Anything I can focus on/help with ?

sneakers-the-rat · 2026-01-14T22:43:47Z

I wonder if we could remove the package-level upper bound on python version and just throw an error if someone tries to use tensorflow with python > 3.12 (or, probably, just throw an error if it cant be imported, in case someone builds their own tensorflow for >3.12). I am assiming torch works with >3.12, and it would be nice to not have that limit for downstream use, future compatibility, etc.

Edit: otherwise i still have same reservations re: maintainability duplicating the dlc code here, would prefer that code is split out in such a way that the main deeplabcut and vendored dlc-live versions cant go out of sync. Its a real shame we cant just use the same serialized model strategy as before, but if torch doesnt have model serialization, thats over our heads. But otherwise as far as i can tell looks good. The changes to the infra are good, thanks for handling that - poetry made sense at a time when it was the only thing of its kind, but the packaging ecosystem has changed a lot (for the better) since then.

.github/workflows/testing.yml

sneakers-the-rat · 2026-01-14T23:00:31Z

tests/test_benchmark_script.py

+        print(f"Running dog model: {model_path}")
+        benchmark_videos(
+            model_path=model_path,
+            model_type="base" if Engine.from_model_path(model_path) == Engine.TENSORFLOW else "pytorch",


I believe this should continue to test tensorflow models, as long as the tensorflow models remain in the benchmarking data. Is that right? There is the possibility that the benchmarking data becomes updated and we silently stop testing tensorflow, so i think it would be nice to parameterize this test to explicitly run with pytorch and tensorflow, and fail test if no tensorflow models are found.

sneakers-the-rat · 2026-01-14T23:03:13Z

dlclive/engine.py

+            if has_cfg and has_pb:
+                return cls.TENSORFLOW
+        elif path.is_file():
+            if path.suffix == ".pt":


Oh wait are there serialized pytorch models now??? That would dramatically simplify things and avoid possibility for model drift - we could take out all the model code here (keeping the great work making the switchable backends!)

I agree on your reservations regarding the duplications of the DeepLabCut code. Will try to address that.

Regarding the models: no we are not using serialized models currently. PyTorch models are exported by saving both the state dict and the model configuration in the .pt. file. In DeepLabCut-live the models are initialized using the configuration and the weights are loaded both using this exported .pt file.

(i.e. the model architecture is serialized, but the class definition must still be available in the environment when loading to reconstruct it).

Ah thats a shame. It seems like it should be possible to pickle the model classes and etc., but if the torch serialization doesnt do that, then yes lets not try and invent a file format here. Carry on

Co-authored-by: Jonny Saunders <[email protected]>

C-Achard · 2026-01-15T09:59:57Z

dlclive/check_install/check_install.py

+from dlclive.engine import Engine

 MODEL_NAME = "superanimal_quadruped"
 SNAPSHOT_NAME = "snapshot-700000.pb"


It seems to me that this always assumes that tf is installed - should this be updated dynamically ?

has_tf = find_spec("tensorflow") is not None has_torch = find_spec("torch") is not None if has_tf: model_type = "base" MODEL_NAME = "superanimal_quadruped" SNAPSHOT_NAME = "snapshot-700000.pb" elif has_torch: model_type = "pytorch" MODEL_NAME = # "pt model" SNAPSHOT_NAME = # "model.pth" else: raise RuntimeError("Neither TensorFlow nor PyTorch is installed. Please install one using `tf` or `pytorch` optional dependencies.")

Seems like this should have thia exception be raised in one place in the package - if having neither should be considered an exception, it should get raised predictably

deruyter92 · 2026-01-15T13:54:38Z

One thing I'd like to add before merging is improved modelzoo functionality where PyTorch models are exported in DeepLabCut-live format. This requires finding the corresponding model configurations & storing them inside the .pt file.

MMathisLab · 2026-01-15T15:20:43Z

Let's leave that for a follow up PR; this is already too large

Copilot

Pull request overview

Copilot reviewed 84 out of 89 changed files in this pull request and generated 17 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-15T15:25:55Z

dlclive/pose_estimation_pytorch/models/registry.py

+            self._register_module(module=module, module_name=name, force=force)
+            return module
+
+        return


The register_module decorator function returns None when used as a decorator. It should return the _register function to work properly as a decorator. Change line 330 from 'return' to 'return _register'.

Copilot · 2026-01-15T15:25:55Z

dlclive/pose_estimation_pytorch/models/predictors/dekr_predictor.py

+        pool1 = torch.nn.MaxPool2d(3, 1, 1) # TODO JR 01/2026: Are these unused variables informative?
+        pool2 = torch.nn.MaxPool2d(5, 1, 2)
+        pool3 = torch.nn.MaxPool2d(7, 1, 3)


Variables pool1 and pool3 are defined but never used. Only pool2 is used on line 270. Either remove the unused variables or clarify their purpose if they're placeholders for future use.

Copilot · 2026-01-15T15:25:56Z

dlclive/utils.py


    SK_IM = True
-except Exception:
+except ImportError as e:


The exception variable 'e' is caught but never used. Either remove the variable binding (use 'except ImportError:') or use it for logging/debugging purposes.

Copilot · 2026-01-15T15:25:56Z

dlclive/utils.py


    OPEN_CV = True
-except Exception:
+except ImportError as e:


The exception variable 'e' is caught but never used. Either remove the variable binding (use 'except ImportError:') or use it for logging/debugging purposes.

These exceptions are strange to me in a different way - why are we catching import errors for packages listed in the dependencies or trying to import packages that are not listed in the deps?

Copilot · 2026-01-15T15:25:56Z

dlclive/utils.py

    Returns
    -------
-    :class:`numpy.ndarray`
+    :class: `numpy.ndarray`


Corrected spacing in reStructuredText class reference from ':class: numpy.ndarray' to ':class:numpy.ndarray'.

Copilot · 2026-01-15T15:25:59Z

tests/test_engine.py

+Tests for the Engine class - engine detection and model type handling
+"""
+import pytest
+from pathlib import Path


Import of 'Path' is not used.

Copilot · 2026-01-15T15:26:00Z

tests/test_factory.py

+Tests for the Factory module - runner building
+"""
+import pytest
+from pathlib import Path


Import of 'Path' is not used.

Copilot · 2026-01-15T15:26:00Z

tests/test_factory.py

+from pathlib import Path
+from unittest.mock import Mock, patch
+from dlclive import factory
+from dlclive.engine import Engine


Import of 'Engine' is not used.

Copilot · 2026-01-15T15:26:01Z

tests/test_processor.py

+"""
+Tests for Processor base class
+"""
+import pytest


Import of 'pytest' is not used.

Copilot · 2026-01-15T15:26:01Z

tests/test_utils.py

+"""
+import pytest
+import numpy as np
+from pathlib import Path


Import of 'Path' is not used.

dikraMasrour and others added 24 commits February 24, 2025 15:30

Remove TensorFlow references

b11743c

Add comments on TF questions for changing dlc-live pipeline to Pytorch

c8f5839

Vanilla pytorch inference done; Commenting out TensorFlow references …

030c42c

…in code

change testing directory for Anna and allow to run on CPU

d75339a

Fix display, clean code, adapt frame processing, onnx inference

4de5829

add screenshots

40a9dd4

Fix CPU inference crash + GPU (cuda) & CPU support for Pytorch and ON…

2fc48f1

…NX inference

Video analysis feature

125a072

Improvements on benchmark_pytorch.py

9d2350e

Implement TensorRT optimisation on ONNX models and FP16 precision inf…

13b8fae

…erence

Avs live feed

8b41df6

Tutorial notebook in progress

d1e7df9

bug fixing h5 saving for live video feed

2fb39fc

add code to save numbers in csv and h5 as numbers, not tensor(number)

98b3a13

add timestamp suffix to videos and csv/h5 files

e6a914a

fix live inference and display, black and isort

29c8122

cleaning out unused files

9508a79

update docstrings, clean dlclive script

d61f892

Continued DeepLabCut-Live implementation for DeepLabCut 3.0

f527d38

working on README

4c07e04

improved docs

82daf43

improved docs for PyTorch code

40005ae

improved readme

c6a1f69

fix default top down dynamic cropping parameters

d738a2c

n-poulsen added the DLC 3.0 🔥 label Feb 28, 2025

n-poulsen mentioned this pull request Feb 28, 2025

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #120

Closed

n-poulsen requested review from AlexEMG, MMathisLab and maximpavliv February 28, 2025 10:58

n-poulsen mentioned this pull request Feb 28, 2025

MobileNets DeepLabCutAIResidency/DLC_AI2024#24

Open

deruyter92 and others added 17 commits January 14, 2026 08:51

Fix argument alignment in benchmark_videos call

cbc23ed

This commit corrects the parameter names and order and removes unsupported arguments from `benchmark_videos` call: - device - precision - draw_keypoint_names - get_sys_info

fix python version <3.13

cd4649e

Update pyproject.toml: remove duplicate line

e0e900a

Co-authored-by: Copilot <[email protected]>

update CI

f5ce1a1

run with uv instead of poetry and for python 3.10 up to 3.12

Update pyproject.toml

690f285

- Remove upper bound for tensorflow version is python version >=3.11 - add pip as required dependency

update display.py

8f1e844

remove tkinter as required dependency.

incorporate copilot suggestions

c6f7813

(remove conditional tf dependency install for for windows and python >= 3.11, since never reached)

merge branch 'main' into PR #121

9db0752

Update readme and installation docs

ecb3673

- removed poetry instructions - added optional instructions for installation with uv - added notes on tensorflow support for windows

update .gitignore (ignore .lock files)

7e9b4b4

Minor refactors (comments, unused imports, unused variables)

11690ed

remove unused script fix_deeplabcut_imports

6e2ad64

add basic tests

72d073b

update .gitignore

681c373

deruyter92 approved these changes Jan 14, 2026

View reviewed changes

sneakers-the-rat reviewed Jan 14, 2026

View reviewed changes

.github/workflows/testing.yml Outdated Show resolved Hide resolved

sneakers-the-rat reviewed Jan 14, 2026

View reviewed changes

Update CI workflow: include all tests

83390f0

Co-authored-by: Jonny Saunders <[email protected]>

C-Achard reviewed Jan 15, 2026

View reviewed changes

MMathisLab requested a review from Copilot January 15, 2026 15:21

Copilot started reviewing on behalf of MMathisLab January 15, 2026 15:22 View session

Copilot AI reviewed Jan 15, 2026

View reviewed changes

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

Are you sure you want to change the base?

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

Conversation

n-poulsen commented Feb 28, 2025

Uh oh!

deruyter92 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

C-Achard commented Jan 14, 2026

Uh oh!

sneakers-the-rat commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deruyter92 Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deruyter92 commented Jan 15, 2026

Uh oh!

MMathisLab commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

sneakers-the-rat commented Jan 14, 2026 •

edited

Loading

deruyter92 Jan 15, 2026 •

edited

Loading