Storage CLI

A unified command-line tool for interacting with multiple cloud storage providers through a single binary. The CLI supports five blob-storage providers (Azure Blob Storage, AWS S3, Google Cloud Storage, Alibaba Cloud OSS, and WebDAV), each with its own client implementation while sharing a common command interface.

Note: This CLI works with existing storage resources (buckets, containers, etc.) that are already created and configured in your cloud provider. The storage bucket/container name and credentials must be specified in the provider-specific configuration file.

Key points

• Single binary with provider selection via -s flag.

• Each provider has its own directory (azurebs/, s3/, gcs/, alioss/, dav/) containing client implementations and configurations.

• All providers support the same core commands (put, get, delete, exists, list, copy, etc.).

• Provider-specific configurations are passed via JSON config files.

Providers

• Alioss
• Azurebs
• Dav
  • additional endpoints needed by CAPI still missing
• Gcs
• S3

Build

Build the unified storage CLI binary:

go build -o storage-cli

Or with version information:

go build -ldflags "-X main.version=1.0.0" -o storage-cli

Usage

The CLI uses a unified command structure across all providers:

storage-cli -s <provider> -c <config-file> <command> [arguments]

Flags:

• -s: Storage provider type (azurebs|s3|gcs|alioss|dav)
• -c: Path to provider-specific configuration file
• -v: Show version
• -log-file: Path to log file (optional, logs to stderr by default)
• -log-level: Logging level: debug, info, warn, error (default: warn)

Common commands:

• put <path/to/file> <remote-object> - Upload a local file to remote storage
• get <remote-object> <path/to/file> - Download a remote object to local file
• delete <remote-object> - Delete a remote object
• delete-recursive [prefix] - Delete objects recursively. If prefix is omitted, deletes all objects
• exists <remote-object> - Check if a remote object exists (exits with code 3 if not found)
• list [prefix] - List remote objects. If prefix is omitted, lists all objects
• copy <source-object> <destination-object> - Copy object within the same storage
• sign <object> <action> <duration_as_second> - Generate signed URL (action: get|put, duration: e.g., 60s)
• properties <remote-object> - Display properties/metadata of a remote object
• ensure-storage-exists - Ensure the storage container/bucket exists, if not create the storage(bucket,container etc)

Examples:

# Upload file to S3
storage-cli -s s3 -c s3-config.json put local-file.txt remote-object.txt

# List GCS objects with prefix
storage-cli -s gcs -c gcs-config.json list my-prefix

# Check if Azure blob exists
storage-cli -s azurebs -c azure-config.json exists my-blob.txt

# Get properties of an object
storage-cli -s azurebs -c azure-config.json properties my-blob.txt

# Sign object for 'get' in alioss for 60 seconds
storage-cli -s alioss -c alioss-config.json sign object.txt get 60s

# Upload file with debug logging to file
storage-cli -s s3 -c s3-config.json -log-level debug -log-file storage.log put local-file.txt remote-object.txt

# List objects with error-level logging only
storage-cli -s gcs -c gcs-config.json -log-level error list my-prefix

Contributing

Follow these steps to make a contribution to the project:

• Fork this repository
• Create a feature branch based upon the main branch (pull requests must be made against this branch)

  git checkout -b feature-name origin/main

• Run tests to check your development environment setup

  ginkgo --race --skip-package=integration --cover -v -r ./...

• Make your changes (be sure to add/update tests)
• Run tests to check your changes

  ginkgo --race --skip-package=integration --cover -v -r ./...

• If you added or modified integration tests, to run them locally, follow the instructions in the provider-specific README (see Providers section)
• Note: Integration tests require access to cloud provider credentials and cannot run on PRs from forks. They will run automatically when a maintainer merges your PR to main.
• Push changes to your fork

  git add .
  git commit -m "Commit message"
  git push origin feature-name

• Create a GitHub pull request, selecting main as the target branch

Dependency Updates

This project uses Dependabot to keep dependencies up to date. Dependencies are grouped (e.g., AWS SDK, Azure SDK, Google Cloud, testing tools) to reduce PR noise.

Integration tests on Dependabot PRs: Integration tests are skipped for Dependabot PRs since they don’t have access to secrets. If needed, maintainers can manually trigger the integration tests via workflow_dispatch before merging. Integration tests will also run automatically after merging to main.

Releases

Manual Release

Releases must be triggered manually by an approver. This can be done either via GitHub Actions (workflow dispatch) or through the GitHub Releases page using the Draft a new release option. The Release Manual workflow is responsible for creating and completing the release.

Option 1: Release via Workflow Dispatch

• Go to Actions and select the Release Manual workflow.

• Click Run workflow.

• Enter the next incremented version number with the v prefix (for example, v1.2.3).

• The workflow will create the release and upload the build artifacts once completed.

Option 2: Release via Draft Release

• Go to Releases and click Draft a new release.

• Create a new tag using the next incremented version with the v prefix.

• Fill in the release title and description.

• Click Publish release.

• The release will appear immediately on the Releases page. This action will also trigger the Release Manual workflow, which will build the artifacts and upload them to the published release once the workflow finishes.

Notes

These commit IDs represent the last migration checkpoint from each provider’s original repository, marking the final commit that was copied during the consolidation process.

• alioss -> c303a62679ff467ba5012cc1a7ecfb7b6be47ea0
• azurebs -> 18667d2a0b5237c38d053238906b4500cfb82ce8
• dav -> c64e57857539d0173d46e79093c2e998ec71ab63
• gcs -> d4ab2040f37415a559942feb7e264c6b28950f77
• s3 -> 9b75c9e3598fcd87924789ddf0035e5597a3df63