Download files in parallel during recursive get

[neon-ninja]

Try it: compiled Linux binaries

Built by the CI run for the current head (7be0ec9). Artifacts are zip files; unzip and chmod +x.

• dbxcli-linux-binaries — dbxcli-linux-amd64, dbxcli-linux-arm64, dbxcli-linux-arm (about 9.9 MB, expires in 30 days)
• dbxcli-release-archives — packaged .tar.gz/.zip archives for every release target plus SHA256SUMS
• Workflow run: https://github.com/UoA-eResearch/dbxcli/actions/runs/35662911699

Every push to this PR re-uploads fresh artifacts; the latest are always on the newest CI run in the Checks tab.

unzip dbxcli-linux-binaries.zip && chmod +x dbxcli-linux-amd64
./dbxcli-linux-amd64 get -r /some/folder ./local        # auto-tuned parallel download
./dbxcli-linux-amd64 get -r -w 8 /some/folder ./local   # fixed 8 workers

What changed

dbxcli get --recursive now downloads several files at once instead of one after another.

• Automatic concurrency by default. A hill-climbing controller starts at 4 concurrent downloads and raises the count in steps (4 → 6 → 9 → 13 → 16) while each step improves the measured aggregate throughput by at least 10%. As soon as a step stops paying off, which means the available bandwidth is saturated, it falls back to the last level that helped and settles there. Throughput is sampled over 2-second windows, skipping the first window after each change, and only windows where the limit was actually binding count.
• --workers / -w flag pins a fixed number of concurrent downloads. 0 (the default) selects automatic tuning; -w 1 restores the previous sequential behaviour. Negative values are rejected as invalid arguments.
• tqdm-style progress with throughput and ETA. Per-file progress for get and share-link download now reads:

  Downloading  45%|=========           | 45 MiB/100 MiB [00:12<00:15, 3.7 MiB/s]
  

  The throughput is measured over a 5-second sliding window, and redraws are throttled to 10 per second. In parallel mode, when stderr is a terminal, a single live status line shows aggregate progress plus the current worker count:

  Downloading 12/40 files  45%|=========           | 1.2 GiB/2.7 GiB [00:12<00:15, 98 MiB/s, 6 workers]
  

  and every parallel run ends with a summary line:

  Downloaded 40/40 files, 2.7 GiB in 00:28 (98 MiB/s), up to 9 workers
  

  Pipes and logs only receive complete lines (one Downloading src -> dst line per file plus the summary).
• Stable output order. Results and errors are collected per listing entry, so JSON results and error messages come out in listing order regardless of completion order.
• The workers flag is registered in the JSON help manifest, and the generated command docs, README, and changelog are updated.

CI changes

• The release build job now uploads the Linux binaries and packaged release archives as workflow artifacts (linked above).
• govulncheck@latest is pinned to v1.7.0: latest now resolves to x/vuln v1.8.0, which requires Go 1.26, so every job that installs it was failing under the pinned Go 1.25.13 toolchain before running any tests. This is the same failure currently red on master. v1.7.0 is the newest release supporting Go 1.25 and reports no vulnerabilities for this module.

Why

Single-stream downloads from Dropbox rarely saturate a fast link. Downloading multiple files concurrently uses the available bandwidth, and tuning the worker count from measured throughput avoids both under-using fast links and hammering the API on slow ones.

Validation

• CI is green on the current head (all three OS test matrices, race test, golangci-lint, actionlint, release build, generated docs, plugin sync, Chocolatey).
• Locally: go build ./..., go vet ./..., go test -race ./..., golangci-lint run ./..., and govulncheck ./... under Go 1.25.13 all pass; gen-docs and gen-json-schemas produce no further diff.
• New tests cover the tuner (climb, warm-up, plateau, settle at max, clamping), the resizable limiter, the pool (fixed limit, single worker, cancelled context, auto mode), the status writer, the progress formatter (bar, clock, ETA, rate), the sliding-window rate meter, draw throttling, flag parsing, and end-to-end recursive downloads (parallelism by default, -w N bounds, -w 1 sequential progress, listing-order JSON results and errors, export-only files).

Notes

• share-link download --recursive still downloads files sequentially; it got the new progress format but has its own loop and was left out of the parallel change.
• Single-file downloads are unchanged apart from the new progress line. The SDK supports ranged parallel downloads for one large file, which could be a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AwSTGMWogdVigTYWL9fwgN

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}