backends: have separate exceptions for invalid url and dependency missing

[f-froehlich]

Currently if boto3 is not installed no repository is found. So an error like Error: Invalid or unsupported Backend Storage URL: s3:ACCESS_KEY:SECRET@https://URL/REPO/PATH is thrown. This is a bit confusing because the URL is valid but boto3 is not installed. We will now throw DependencyMissing with a proper message if using s3/b2 URL but boto3 is not installed.

[ThomasWaldmann]

Well, it says "or unsupported", so the error msg is correct.

But I agree, it would be more helpful to differentiate these 2 cases for all backends that need an extra dependency.

[ThomasWaldmann]

I think the algorithm in get_xyz_backend should be like:

• check first if the scheme in the url matches the backend implementation, if not return None. we are lucky that the scheme is always present in borgstore urls, so a simple "startswith" check should be ok.
• check if the extra requirements (if any) are installed, if not, raise an exception with clear instructions
• if the url is not successfully matched, raise an exception like "invalid url". the exception msg must contain the invalid url.
• otherwise construct and return the backend

Extras see there:

https://github.com/borgbackup/borgstore/blob/master/pyproject.toml#L29

pip install borgstore[extra]  # command to install borgstore with extra
pip install borgstore[extra1,extra2]  # ... for multiple extras

Note: doing it that way makes it impossible to have multiple separate backend implementations for the same scheme, but with different extra requirements. but I guess we can live with that.

[ThomasWaldmann]

@f-froehlich Can you please update all the backends to solve the problem completely?

[f-froehlich]

I updated the PR accordingly and add install instruction to the Error message in S3&SFTP urls. Other backends do not have requirements. So this should be fine now?

[ThomasWaldmann]

The rclone backend does not have a python library requirement, but it requires the rclone executable in a specific minimum version, so that could be handled in the same way.

[f-froehlich]

The rclone backend does not have a python library requirement, but it requires the rclone executable in a specific minimum version, so that could be handled in the same way.

The rclone backend has currently this check in place:

# Check rclone is on the path
        try:
            info = json.loads(subprocess.check_output([RCLONE, "rc", "--loopback", "core/version"]))
        except Exception:
            raise BackendDoesNotExist("rclone binary not found on the path or not working properly")
        if info["decomposed"] < [1, 57, 0]:
            raise BackendDoesNotExist(f"rclone version must be at least v1.57.0 - found {info['version']}")
        return Rclone(path=m["path"])

Should I update the BackendDoesNotExist to DependencyMissing? The Check seems fine for me

[ThomasWaldmann]

I think it would be easier (== not require changes in borg) to change DependencyMissing to BackendDoesNotExist.

[ThomasWaldmann]

And the checks in rclone backend must be in same order / style as in sftp/s3.

[f-froehlich]

I think it would be easier (== not require changes in borg) to change DependencyMissing to BackendDoesNotExist.

Done

[f-froehlich]

And the checks in rclone backend must be in same order / style as in sftp/s3.

Done

[ThomasWaldmann]

LGTM!

[ThomasWaldmann]

Thanks!