Add wcurl in Rust

[ericcurtin]

To have a compiled binary free of dependancies

[ericcurtin]

Perf results, the first invocation of the shell script version is always gonna be slow, needs to execute multiple binaries, a shell, etc. from scratch. This is all cached in following invocations. wcurl-rs is just one small binary that calls curl, so not really an issue.

$ hyperfine "./wcurl-rs/target/release/wcurl --dry-run example.com"
Benchmark 1: ./wcurl-rs/target/release/wcurl --dry-run example.com
  Time (mean ± σ):      12.9 ms ±   1.9 ms    [User: 2.9 ms, System: 3.6 ms]
  Range (min … max):     9.8 ms …  18.4 ms    125 runs
$ hyperfine "./wcurl --dry-run example.com"
Benchmark 1: ./wcurl --dry-run example.com
  Time (mean ± σ):      83.2 ms ± 128.3 ms    [User: 9.1 ms, System: 18.4 ms]
  Range (min … max):    32.7 ms … 447.0 ms    10 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (447.0 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
$ hyperfine "./wcurl --dry-run example.com"
Benchmark 1: ./wcurl --dry-run example.com
  Time (mean ± σ):      35.3 ms ±   4.5 ms    [User: 7.6 ms, System: 15.0 ms]
  Range (min … max):    29.8 ms …  47.2 ms    57 runs

[ericcurtin]

Optimized even further:

$ hyperfine "./wcurl-rs/target/release/wcurl --dry-run example.com"
Benchmark 1: ./wcurl-rs/target/release/wcurl --dry-run example.com
  Time (mean ± σ):      12.0 ms ±   2.1 ms    [User: 2.7 ms, System: 3.6 ms]
  Range (min … max):     8.4 ms …  18.2 ms    110 runs

[ericcurtin]

Even more optimized:

$ hyperfine "./wcurl-rs/target/release/wcurl --dry-run example.com"
Benchmark 1: ./wcurl-rs/target/release/wcurl --dry-run example.com
  Time (mean ± σ):      11.6 ms ±   1.8 ms    [User: 2.7 ms, System: 3.4 ms]
  Range (min … max):     8.3 ms …  18.1 ms    131 runs

[samueloph]

I've been a bit too busy lately but will reply to this PR by the weekend, sorry about the delay.

[samueloph]

I like the idea of experimenting with a Rust-based wcurl, but I don't think it makes sense to merge it.

The point of wcurl is to have a wget alternative based on curl, and for this to work, it needs to be a solution that comes shipped with curl, otherwise if it's something the user needs extra steps to install then it's no different than not using curl, this new tool might as well have any other name.

wcurl was initially shipped with the curl package in Debian, and then later it became shipped with curl upstream.

A compiled wcurl implementation would have to be something which can still be shipped with curl, supporting as many architectures as curl does. The only way we can achieve this would be with C and using libcurl.

Then for a compiled wcurl, we also need to think what we gain from it, based on recent work from @vszakats it sounds like it would make its usage easier on Windows environments, but the biggest difference would be that wcurl would be able to support more features with higher complexity (e.g.: we will never implement recursive downloads in shell).

Myself and @sergiodj were recently looking into libcurl and it seems that a libcurl wcurl will require careful planning around how to implement some of the curl CLI features without duplicating code, this should be done closely with the curl developers.

Thank you for the proposal though, @ericcurtin.

[vszakats]

From a Windows POV, specifically from the curl-for-win angle where the
official Windows binaries are coming from:

I like the size improvements (and fixing the info-leak) and found the
Rust experiment interesting. At 6 deps, I still find that somewhat high,
given the task at hand.

(The new size is comparable to the trurl binary (on macOS), which also
includes a minimal libcurl linked statically. On Windows and Linux the
size is half of this.)

But, adding a Rust-based wcurl to the Windows package would mean a large
development work to make Rust (cross-)compilation work alongside the C
(cross-)builds. Rust would be adding many dependencies, which would need
to be rolled and maintained. I expect this to be a considerable and sustained
work. It'd be also adding to the heft of the tools to be downloaded and
installed in CI, increasing build times, etc.

This amount of resources is IMO not practical to replace 300 lines of
standard POSIX script, to run natively on Windows.

(Also the script can be run with the help of a ~650 KB busybox.exe.)

Memory safety in this particular case doesn't add an advantage compared
to a C implementation, which would likely be thinner, and need relatively
little resources to integrate.

[ericcurtin]

From a Windows POV, specifically from the curl-for-win angle where the official Windows binaries are coming from:

I like the size improvements (and fixing the info-leak) and found the Rust experiment interesting. At 6 deps, I still find that somewhat high, given the task at hand.

6 dependancies? This had zero it used stdlib Rust.

The PR is closed anyway.

(The new size is comparable to the trurl binary (on macOS), which also includes a minimal libcurl linked statically. On Windows and Linux the size is half of this.)

But, adding a Rust-based wcurl to the Windows package would mean a large development work to make Rust (cross-)compilation work alongside the C (cross-)builds. Rust would be adding many dependencies, which would need to be rolled and maintained. I expect this to be a considerable and sustained work. It'd be also adding to the heft of the tools to be downloaded and installed in CI, increasing build times, etc.

This amount of resources is IMO not practical to replace 300 lines of standard POSIX script, to run natively on Windows.

(Also the script can be run with the help of a ~650 KB busybox.exe.)

Memory safety in this particular case doesn't add an advantage compared to a C implementation, which would likely be thinner, and need relatively little resources to integrate.

[vszakats]

It's more than 6:

   Compiling proc-macro2 v1.0.103
   Compiling quote v1.0.41
   Compiling unicode-ident v1.0.22
   Compiling clap_lex v0.7.6
   Compiling anstyle v1.0.13
   Compiling heck v0.5.0
   Compiling percent-encoding v2.3.2
   Compiling clap_builder v4.5.51
   Compiling syn v2.0.108
   Compiling clap_derive v4.5.49
   Compiling clap v4.5.51

I see it's closed, sorry for giving you feedback!