This is an automated email from the ASF dual-hosted git repository. steveloughran pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/hadoop-cloudstore.git
commit 68e6f01450c1eda69624783975ce52db0d8ea7af Author: Steve Loughran <[email protected]> AuthorDate: Fri Jun 26 17:25:57 2026 +0100 HADOOP-19880. commit docs before a release --- AGENTS.md | 4 ++-- BUILDING.md | 21 ++++++++++++++++++--- README.md | 10 +++++----- pom.xml | 4 ++-- src/site/markdown/bandwidth.md | 6 +++--- src/site/markdown/bucketmetadata.md | 6 +++--- src/site/markdown/bulkdelete.md | 4 ++-- src/site/markdown/cloudup.md | 22 +++++++++++----------- src/site/markdown/committerinfo.md | 8 ++++---- src/site/markdown/constval.md | 6 +++--- src/site/markdown/etag.md | 14 +++++++------- src/site/markdown/fetchdt.md | 6 +++--- src/site/markdown/index.md | 2 +- src/site/markdown/locatefiles.md | 4 ++-- src/site/markdown/mkbucket.md | 2 +- src/site/markdown/mkcsv.md | 2 +- src/site/markdown/put.md | 2 +- src/site/markdown/sdk.md | 10 +++++----- src/site/markdown/storediag.md | 14 +++++++------- src/site/markdown/tlsinfo.md | 8 ++++---- src/site/markdown/versioned-objects.md | 14 +++++++------- 21 files changed, 92 insertions(+), 77 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8a8e1c5..e8b56f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,7 @@ Releasing is manual and tag-driven — see `BUILDING.md`. Bump the version with The tools are executed via the `hadoop jar` command. The general syntax is: ```sh -hadoop jar target/cloudstore-1.4.jar <command> [options] <arguments...> +hadoop jar target/cloudstore-1.5.jar <command> [options] <arguments...> ``` - `<command>`: The name of the tool to run (e.g., `list`, `dux`, `storediag`). @@ -78,7 +78,7 @@ hadoop jar target/cloudstore-1.4.jar <command> [options] <arguments...> **Example: Listing files in an S3 bucket** ```sh -hadoop jar target/cloudstore-1.4.jar list -limit 10 s3a://my-bucket/path/ +hadoop jar target/cloudstore-1.5.jar list -limit 10 s3a://my-bucket/path/ ``` diff --git a/BUILDING.md b/BUILDING.md index d4e12d8..da04c96 100644 --- a/BUILDING.md +++ b/BUILDING.md @@ -137,11 +137,26 @@ cloudstore-<version>.jar.sha256`) independent of the gpg signature. It also generates a build version, with the buildnumber plugin. On release builds, this will fail the build if there are uncommitted changes. -Commit all changes before starting a release build. +** You must commit all changes before starting a release build.** -Commands (for fish) +The build will fail if there are uncommitted changes. + +``` +[INFO] --- buildnumber:3.3.0:create (default) @ cloudstore --- +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD FAILURE +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 4.344 s +[INFO] Finished at: 2026-06-26T17:23:21+01:00 +[INFO] ------------------------------------------------------------------------ +[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:3.3.0:create (default) on project cloudstore: Cannot create the build number because you have local modifications : +``` + +### Release Commands (for fish) ```bash -set -gx ver 1.4 # last released version; moved only on a release bump by dev-support/bump-version.sh +set -gx ver 1.5 # last released version; moved only on a release bump by dev-support/bump-version.sh + +# now the release build mvn clean install -Prelease,sign -DskipTests set -gx now (date '+%Y-%m-%d-%H.%M'); echo [$now] git commit -S --allow-empty -m "release $now"; git push diff --git a/README.md b/README.md index 3cc4b54..3faa0b6 100644 --- a/README.md +++ b/README.md @@ -215,7 +215,7 @@ Also prints the time to execute each operation (including instantiating the stor and with the `-verbose` option, the store statistics. ``` -hadoop jar cloudstore-1.4.jar \ +hadoop jar cloudstore-1.5.jar \ filestatus \ s3a://guarded-table/example @@ -241,7 +241,7 @@ This is by design, as encoding of the private key when stored in hadoop configur ``` -hadoop jar cloudstore-1.4.jar gcscreds gs://bucket/ +hadoop jar cloudstore-1.5.jar gcscreds gs://bucket/ key uses \n for separator -gs connector must convert to line endings 2022-01-19 17:55:51,016 [main] INFO gs.PemReader (PemReader.java:readNextSection(86)) - title match at line 1 @@ -267,7 +267,7 @@ Usage: list Example: list some of the AWS public landsat store. ```bash -> bin/hadoop jar cloudstore-1.4.jar list -limit 10 s3a://landsat-pds/ +> bin/hadoop jar cloudstore-1.5.jar list -limit 10 s3a://landsat-pds/ Listing up to 10 files under s3a://landsat-pds/ 2019-04-05 21:32:14,523 [main] INFO tools.ListFiles (StoreDurationInfo.java:<init>(53)) - Starting: Directory list @@ -313,7 +313,7 @@ See [mkcsv](src/site/markdown/mkcsv.md) Probes a filesystem for offering a specific named capability on the given path. ```bash -bin/hadoop jar cloudstore-1.4.jar pathcapability +bin/hadoop jar cloudstore-1.5.jar pathcapability Usage: pathcapability [options] <capability> <path> -D <key=value> Define a property -tokenfile <file> Hadoop token file to load @@ -322,7 +322,7 @@ Usage: pathcapability [options] <capability> <path> ``` ```bash -hadoop jar cloudstore-1.4.jar pathcapability fs.s3a.capability.select.sql s3a://landsat-pds/ +hadoop jar cloudstore-1.5.jar pathcapability fs.s3a.capability.select.sql s3a://landsat-pds/ Using filesystem s3a://landsat-pds Path s3a://landsat-pds/ has capability fs.s3a.capability.select.sql diff --git a/pom.xml b/pom.xml index 3fb0c77..48e973e 100644 --- a/pom.xml +++ b/pom.xml @@ -19,7 +19,7 @@ <groupId>org.apache.hadoop</groupId> <artifactId>cloudstore</artifactId> - <version>1.5-SNAPSHOT</version> + <version>1.5</version> <packaging>jar</packaging> <name>hadoop cloudstore</name> @@ -55,7 +55,7 @@ dev-support/update-site-docs.sh in lockstep with the markdown references to cloudstore-X.Y.jar; the verify-phase check-doc-versions execution below uses it to detect drift. --> - <cloudstore.docs.version>1.4</cloudstore.docs.version> + <cloudstore.docs.version>1.5</cloudstore.docs.version> <!-- GPG signing key id; override on the command line with -Dgpg.keyName=<keyid>. Empty default lets gpg pick the default diff --git a/src/site/markdown/bandwidth.md b/src/site/markdown/bandwidth.md index 178d2c9..d9fc659 100644 --- a/src/site/markdown/bandwidth.md +++ b/src/site/markdown/bandwidth.md @@ -19,7 +19,7 @@ Measure upload/download bandwidth; support different read policies, and optionally save the output to a CSV file. ``` -> hadoop jar cloudstore-1.4.jar bandwidth +> hadoop jar cloudstore-1.5.jar bandwidth Usage: bandwidth [options] size <path> -D <key=value> Define a single configuration option @@ -44,7 +44,7 @@ Upload 128M of data to s3 with a block size of 8 megabytes; use `-verbose` outpu statistics. Save the summary to a CSV file for review. ``` -> hadoop jar cloudstore-1.4.jar bandwidth -csv tmp/s3a128m.csv -block 8 -verbose -policy whole-file 128m s3a://example-london/tmp +> hadoop jar cloudstore-1.5.jar bandwidth -csv tmp/s3a128m.csv -block 8 -verbose -policy whole-file 128m s3a://example-london/tmp Bandwidth test against s3a://example-london/tmp with data size 128m ================================================================== @@ -287,7 +287,7 @@ Here's the experiment repeated with prefetching enabled and changes to block upl ``` ```bash -bin/hadoop jar cloudstore-1.4.jar bandwidth -D fs.s3a.prefetch.enabled=true -csv tmp/s3a128mp.csv -block 8 -verbose -policy whole-file 128m s3a://example-london +bin/hadoop jar cloudstore-1.5.jar bandwidth -D fs.s3a.prefetch.enabled=true -csv tmp/s3a128mp.csv -block 8 -verbose -policy whole-file 128m s3a://example-london Upload Summary ============== diff --git a/src/site/markdown/bucketmetadata.md b/src/site/markdown/bucketmetadata.md index c57e96e..d0e812c 100644 --- a/src/site/markdown/bucketmetadata.md +++ b/src/site/markdown/bucketmetadata.md @@ -54,7 +54,7 @@ to remap the request to the Access Point. The response indicates that this is mapped to an S3 Standard bucket. ``` -bin/hadoop jar cloudstore-1.4.jar bucketmetadata s3a://example-ap +bin/hadoop jar cloudstore-1.5.jar bucketmetadata s3a://example-ap Getting bucket info for s3a://example-ap ======================================== @@ -72,7 +72,7 @@ Location Type null When probing an S3 Express bucket, the location type and name is returned. ``` -hadoop jar cloudstore-1.4.jar bucketmetadata s3a://example--usw2-az1--x-s3 +hadoop jar cloudstore-1.5.jar bucketmetadata s3a://example--usw2-az1--x-s3 Getting bucket info for s3a://example--usw2-az1--x-s3 ===================================================== @@ -90,7 +90,7 @@ The result of the probe against third party stores is undefined, and will vary w Here is an example response from a probe of a Dell ECS store: ``` -bin/hadoop jar cloudstore-1.4.jar bucketmetadata s3a://ecsbucket/ +bin/hadoop jar cloudstore-1.5.jar bucketmetadata s3a://ecsbucket/ Getting bucket info for s3a://ecsbucket/ ========================================= diff --git a/src/site/markdown/bulkdelete.md b/src/site/markdown/bulkdelete.md index 11b0165..4b4f0ca 100644 --- a/src/site/markdown/bulkdelete.md +++ b/src/site/markdown/bulkdelete.md @@ -63,7 +63,7 @@ It will only delete files at the target path; if there is a directory there it will be ignored. ``` - bin/hadoop jar cloudstore-1.4.jar bulkdelete -page 4 -verbose s3a://example-london/ delete.txt + bin/hadoop jar cloudstore-1.5.jar bulkdelete -page 4 -verbose s3a://example-london/ delete.txt Bulk delete under s3a://example-london/ ====================================== @@ -135,7 +135,7 @@ deleted per batch ``` ``` - > hadoop jar cloudstore-1.4.jar bulkdelete -Dfs.s3a.multiobjectdelete.enable=false -page 4 -verbose s3a://example-london/ delete.txt + > hadoop jar cloudstore-1.5.jar bulkdelete -Dfs.s3a.multiobjectdelete.enable=false -page 4 -verbose s3a://example-london/ delete.txt Bulk delete under s3a://example-london/ diff --git a/src/site/markdown/cloudup.md b/src/site/markdown/cloudup.md index fe6935d..3c46c64 100644 --- a/src/site/markdown/cloudup.md +++ b/src/site/markdown/cloudup.md @@ -86,7 +86,7 @@ All those limits could be addressed, along with some other optional features Download logs from `s3a://bucket/qelogs` and save to the local (relative) path `localquelogs` ```bash -hadoop jar cloudstore-1.4.jar cloudup \ +hadoop jar cloudstore-1.5.jar cloudup \ -threads 32 -update s3a://bucket/qelogs/ localqelogs ``` @@ -94,7 +94,7 @@ And the other way, only updating new files, and uploading the largest 16 files f Use a block size of 16 MB. ```bash -hadoop jar cloudstore-1.4.jar cloudup \ +hadoop jar cloudstore-1.5.jar cloudup \ -threads 32 -update -largest 16 -block 8 localqelogs s3a://bucket/qelogs/ ``` @@ -102,14 +102,14 @@ From HDFS to google GCS. All data is downloaded from HDFS and then uploaded to G there is no buffering to local disk. ```bash - hadoop jar cloudstore-1.4.jar cloudup -threads 60 -largest 16 -update -hflush -verbose \ + hadoop jar cloudstore-1.5.jar cloudup -threads 60 -largest 16 -update -hflush -verbose \ hdfs://tmp/share/hadoop/common gs://example-london/share/common ``` Then download to the local fs ``` -hadoop jar cloudstore-1.4.jar cloudup -threads 60 -largest 16 -update -block 32 gs://example-london/share/common ./scratch/gcs +hadoop jar cloudstore-1.5.jar cloudup -threads 60 -largest 16 -update -block 32 gs://example-london/share/common ./scratch/gcs ... @@ -127,7 +127,7 @@ Seconds per file: 0.377s ```bash -hadoop jar cloudstore-1.4.jar cloudup -threads 60 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common +hadoop jar cloudstore-1.5.jar cloudup -threads 60 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common Summary of copy from file:/Users/alice/Projects/Releases/cdp-7.1.9/share/hadoop/common to s3a://example-london/share/common =========================================================================================================================== @@ -193,7 +193,7 @@ editing the `core-site.xml` settings of your deployment ``` ```bash -hadoop jar cloudstore-1.4.jar cloudup -threads 60 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common +hadoop jar cloudstore-1.5.jar cloudup -threads 60 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common Copying from file:/Users/alice/Projects/Releases/cdp-7.1.9/share/hadoop/common to s3a://example-london/share/common; threads=60; large files=32; block size=67108864; overwrite=false; update=true verbose=false; ignore failures=false Listing source files under file:/Users/alice/Projects/Releases/cdp-7.1.9/share/hadoop/common @@ -255,7 +255,7 @@ Here, no copies took place at all. The IOStatistics report 151 HEAD requests aga And download it ```bash -hadoop jar cloudstore-1.4.jar cloudup -threads 60 -largest 16 -update -block 32 s3a://example-london/share/common ./scratch/s3a +hadoop jar cloudstore-1.5.jar cloudup -threads 60 -largest 16 -update -block 32 s3a://example-london/share/common ./scratch/s3a Copying from s3a://example-london/share/common to file:/Users/alice/Projects/Releases/cdp-7.1.9/scratch/s3a; threads=60; large files=16; block size=33554432; overwrite=false; update=true verbose=false; ignore failures=false Listing source files under s3a://example-london/share/common @@ -332,7 +332,7 @@ A larger block size `-block` can be more efficient when reading data, especially This is because it avoids the need of the store to guess how much data to prefetch. ```bash -hadoop jar cloudstore-1.4.jar cloudup \ +hadoop jar cloudstore-1.5.jar cloudup \ -threads 32 -update -largest 16 -block 32 s3a://bucket/qelogs/ ./scratch ``` @@ -351,7 +351,7 @@ Smaller files are slower to upload. Increasing the number of threads copying files with the `-threads` offers speedup when there are many small files ```bash - hadoop jar cloudstore-1.4.jar cloudup -threads 100 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common + hadoop jar cloudstore-1.5.jar cloudup -threads 100 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common ... @@ -373,7 +373,7 @@ Seconds per file: 0.155s And with 120 threads ```bash - hadoop jar cloudstore-1.4.jar cloudup -threads 120 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common + hadoop jar cloudstore-1.5.jar cloudup -threads 120 -largest 32 -block 64 -update share/hadoop/common s3a://example-london/share/common ... @@ -429,7 +429,7 @@ If this occurs, set the property `fs.s3a.connection.request.timeout` to a larger We recommend doing this on the command line itself, rather than set it site-wide: ```bash -bin/hadoop jar cloudstore-1.4.jar cloudup -D fs.s3a.connection.request.timeout=15m -overwrite \ +bin/hadoop jar cloudstore-1.5.jar cloudup -D fs.s3a.connection.request.timeout=15m -overwrite \ ../releases/hadoop-3.4.1/hadoop-3.4.1.tar.gz \ s3a://example-london/ ``` \ No newline at end of file diff --git a/src/site/markdown/committerinfo.md b/src/site/markdown/committerinfo.md index 8a4e3a6..397e348 100644 --- a/src/site/markdown/committerinfo.md +++ b/src/site/markdown/committerinfo.md @@ -25,7 +25,7 @@ run against doesn't have this new API. The committer is therefore explicitly the ``` -hadoop jar cloudstore-1.4.jar committerinfo abfs://[email protected]/ +hadoop jar cloudstore-1.5.jar committerinfo abfs://[email protected]/ 2021-09-16 19:42:59,731 [main] INFO commands.CommitterInfo (StoreDurationInfo.java:<init>(53)) - Starting: Create committer Committer factory for path abfs://[email protected]/ is org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory@3315d2d7 (classname org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory) @@ -53,7 +53,7 @@ supports this committer, do try it. *Danger*: S3A Bucket with the classic `FileOutputCommitter` ``` -> hadoop jar cloudstore-1.4.jar committerinfo s3a://landsat-pds/ +> hadoop jar cloudstore-1.5.jar committerinfo s3a://landsat-pds/ 2019-08-05 17:38:38,213 [main] INFO commands.CommitterInfo (DurationInfo.java:<init>(53)) - Starting: Create committer 2019-08-05 17:38:40,968 [main] WARN commit.AbstractS3ACommitterFactory (S3ACommitterFactory.java:createTaskCommitter(90)) - Using standard FileOutputCommitter to commit work. This is slow and potentially unsafe. 2019-08-05 17:38:40,968 [main] INFO output.FileOutputCommitter (FileOutputCommitter.java:<init>(141)) - File Output Committer Algorithm version is 2 @@ -67,7 +67,7 @@ supports this committer, do try it. *Good* : S3A bucket with a staging committer: ``` -> hadoop jar cloudstore-1.4.jar committerinfo s3a://example-ireland-ireland-new/ +> hadoop jar cloudstore-1.5.jar committerinfo s3a://example-ireland-ireland-new/ 2019-08-05 17:42:53,563 [main] INFO commands.CommitterInfo (DurationInfo.java:<init>(53)) - Starting: Create committer Committer factory for path s3a://example-ireland-ireland-new/ is org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory@3088660d (classname org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory) 2019-08-05 17:42:55,433 [main] INFO output.FileOutputCommitter (FileOutputCommitter.java:<init>(141)) - File Output Committer Algorithm version is 1 @@ -86,7 +86,7 @@ The classic filesystem committer v1 is used because it works well here: the file *Good* : S3A bucket with a magic committer: ``` -> hadoop jar cloudstore-1.4.jar committerinfo s3a://example-ireland-ireland-new/ +> hadoop jar cloudstore-1.5.jar committerinfo s3a://example-ireland-ireland-new/ 2019-08-05 17:37:42,615 [main] INFO commands.CommitterInfo (DurationInfo.java:<init>(53)) - Starting: Create committer 2019-08-05 17:37:44,462 [main] INFO commit.AbstractS3ACommitterFactory (S3ACommitterFactory.java:createTaskCommitter(83)) - Using committer magic to output data to s3a://example-ireland-ireland-new/ diff --git a/src/site/markdown/constval.md b/src/site/markdown/constval.md index d2445ce..115483d 100644 --- a/src/site/markdown/constval.md +++ b/src/site/markdown/constval.md @@ -19,13 +19,13 @@ Loads a class, resolves a constant/static final field and prints its value. ```bash -hadoop jar cloudstore-1.4.jar constval org.apache.hadoop.fs.s3a.Constants REQUEST_TIMEOUT +hadoop jar cloudstore-1.5.jar constval org.apache.hadoop.fs.s3a.Constants REQUEST_TIMEOUT Value of org.apache.hadoop.fs.s3a.Constants.REQUEST_TIMEOUT = "fs.s3a.connection.request.timeout" -hadoop jar cloudstore-1.4.jar constval org.apache.hadoop.fs.s3a.Constants DEFAULT_REQUEST_TIMEOUT_DURATION +hadoop jar cloudstore-1.5.jar constval org.apache.hadoop.fs.s3a.Constants DEFAULT_REQUEST_TIMEOUT_DURATION Value of org.apache.hadoop.fs.s3a.Constants.DEFAULT_REQUEST_TIMEOUT_DURATION = "PT0S" -hadoop jar cloudstore-1.4.jar constval org.apache.hadoop.fs.s3a.Constants DEFAULT_REQUEST_TIMEOUT +hadoop jar cloudstore-1.5.jar constval org.apache.hadoop.fs.s3a.Constants DEFAULT_REQUEST_TIMEOUT Value of org.apache.hadoop.fs.s3a.Constants.DEFAULT_REQUEST_TIMEOUT = "0" ``` diff --git a/src/site/markdown/etag.md b/src/site/markdown/etag.md index f4f542a..00bf1b4 100644 --- a/src/site/markdown/etag.md +++ b/src/site/markdown/etag.md @@ -29,7 +29,7 @@ In ASF Hadoop, it is available on 3.3.2+ for S3A and ABFS stores. Zero byte file, S3 standard, no default encryption. ``` -> bin/hadoop jar cloudstore-1.4.jar etag s3a://example-london/file +> bin/hadoop jar cloudstore-1.5.jar etag s3a://example-london/file 2025-10-21 12:40:02,204 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for s3a://example-london/file 2025-10-21 12:40:02,580 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for s3a://example-london/file: 00:00:00.378 @@ -40,7 +40,7 @@ Etag of s3a://example-london/file = "d41d8cd98f00b204e9800998ecf8427e" ### Example: S3 Express ``` -> bin/hadoop jar cloudstore-1.4.jar etag s3a://alice--usw2-az1--x-s3/file +> bin/hadoop jar cloudstore-1.5.jar etag s3a://alice--usw2-az1--x-s3/file 2025-10-21 12:28:22,878 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for s3a://alice--usw2-az1--x-s3/file 2025-10-21 12:28:23,863 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for s3a://alice--usw2-az1--x-s3/file: 00:00:00.986 @@ -56,7 +56,7 @@ The result of the probe against third party stores is undefined, and will vary w Here is an example response from a probe of a Dell ECS store: ``` -> bin/hadoop jar cloudstore-1.4.jar etag s3a://ecsstore/file +> bin/hadoop jar cloudstore-1.5.jar etag s3a://ecsstore/file 2025-10-21 12:18:54,817 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for s3a://ecsstore/file 2025-10-21 12:18:55,523 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for s3a://ecsstore/file: 00:00:00.707 @@ -74,7 +74,7 @@ the etag returned is `null`, which is considered an error." ``` > bin/hadoop fs -mkdir -p s3a://example-london/dir/subdir -> bin/hadoop jar cloudstore-1.4.jar etag s3a://example-london/dir/subdir +> bin/hadoop jar cloudstore-1.5.jar etag s3a://example-london/dir/subdir 2025-10-21 13:00:59,697 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for s3a://example-london/dir/subdir 2025-10-21 13:01:00,160 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for s3a://example-london/dir/subdir: 00:00:00.464 @@ -90,7 +90,7 @@ This holds for parent directories too, which may or may not have a directory mar ## Azure ABFS ``` -> bin/hadoop jar cloudstore-1.4.jar etag abfs://[email protected]/file +> bin/hadoop jar cloudstore-1.5.jar etag abfs://[email protected]/file 2025-10-21 12:42:45,257 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for abfs://[email protected]/file 2025-10-21 12:42:45,456 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for abfs://[email protected]/file: 00:00:00.200 @@ -100,7 +100,7 @@ Etag of abfs://[email protected]/file = 0x8DE1096EF7 ## Local FileSystem ``` -> bin/hadoop jar cloudstore-1.4.jar etag file +> bin/hadoop jar cloudstore-1.5.jar etag file 2025-10-21 12:46:30,236 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for file 2025-10-21 12:46:30,240 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for file: 00:00:00.005 @@ -114,7 +114,7 @@ Error code 51: Unimplemented. And a probe of a nonexistent file ``` -> bin/hadoop jar cloudstore-1.4.jar etag file2 +> bin/hadoop jar cloudstore-1.5.jar etag file2 2025-10-21 12:51:19,515 [main] INFO commands.EtagCommand (StoreDurationInfo.java:<init>(91)) - Starting: get path status for file2 2025-10-21 12:51:19,515 [main] INFO commands.EtagCommand (StoreDurationInfo.java:close(200)) - Duration of get path status for file2: 00:00:00.001 diff --git a/src/site/markdown/fetchdt.md b/src/site/markdown/fetchdt.md index a86ede3..35c0288 100644 --- a/src/site/markdown/fetchdt.md +++ b/src/site/markdown/fetchdt.md @@ -36,7 +36,7 @@ Usage: fetchdt <file> [-renewer <renewer>] [-r] [-p] <url1> ... <url999> Successful query of an S3A session delegation token. ```bash -> bin/hadoop jar cloudstore-1.4.jar fetchdt -p -r file:/tmp/secrets.bin s3a://landsat-pds/ +> bin/hadoop jar cloudstore-1.5.jar fetchdt -p -r file:/tmp/secrets.bin s3a://landsat-pds/ Collecting tokens for 1 filesystem to to file:/tmp/secrets.bin 2018-12-05 17:50:44,276 INFO fs.FetchTokens: Starting: Fetching token for s3a://landsat-pds/ 2018-12-05 17:50:44,399 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties @@ -63,7 +63,7 @@ Successful query of an S3A session delegation token. Failure to get anything from fs, with `-r` option to require them ```bash -> hadoop jar cloudstore-1.4.jar fetchdt -p -r file:/tmm/secrets.bin file:/// +> hadoop jar cloudstore-1.5.jar fetchdt -p -r file:/tmm/secrets.bin file:/// Collecting tokens for 1 filesystem to to file:/tmm/secrets.bin 2018-12-05 17:47:00,970 INFO fs.FetchTokens: Starting: Fetching token for file:/ @@ -75,7 +75,7 @@ No token for file:/ Same command, without the -r. ```bash -> hadoop jar cloudstore-1.4.jar fetchdt -p file:/tmm/secrets.bin file:/// +> hadoop jar cloudstore-1.5.jar fetchdt -p file:/tmm/secrets.bin file:/// Collecting tokens for 1 filesystem to to file:/tmp/secrets.bin 2018-12-05 17:54:26,776 INFO fs.FetchTokens: Starting: Fetching token for file:/tmp No token for file:/tmp diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md index 9e33be2..28ab0c6 100644 --- a/src/site/markdown/index.md +++ b/src/site/markdown/index.md @@ -37,7 +37,7 @@ part of the Apache Hadoop release. The full per-command reference is in the **Commands** sidebar. Run ```bash -hadoop jar target/cloudstore-1.4.jar help +hadoop jar target/cloudstore-1.5.jar help ``` for a list of every command bundled in the jar. diff --git a/src/site/markdown/locatefiles.md b/src/site/markdown/locatefiles.md index 9a0572c..89deb56 100644 --- a/src/site/markdown/locatefiles.md +++ b/src/site/markdown/locatefiles.md @@ -28,7 +28,7 @@ jobs, set the value in the configuration option Usage: ``` -hadoop jar cloudstore-1.4.jar locatefiles +hadoop jar cloudstore-1.5.jar locatefiles Usage: locatefiles -D <key=value> Define a single configuration option -sysprop <file> Property file of system properties @@ -44,7 +44,7 @@ Usage: locatefiles Example ``` -> hadoop jar cloudstore-1.4.jar locatefiles \ +> hadoop jar cloudstore-1.5.jar locatefiles \ -threads 8 -verbose \ s3a://landsat-pds/L8/001/002/LC80010022016230LGN00/ diff --git a/src/site/markdown/mkbucket.md b/src/site/markdown/mkbucket.md index 4b56c9b..66b6d41 100644 --- a/src/site/markdown/mkbucket.md +++ b/src/site/markdown/mkbucket.md @@ -22,7 +22,7 @@ Usage: mkbucket <region> <S3A path> ``` ```bash -hadoop jar cloudstore-1.4.jar mkbucket us-east-2 s3a://new-bucket-name/ +hadoop jar cloudstore-1.5.jar mkbucket us-east-2 s3a://new-bucket-name/ ``` The per-bucket settings of the target bucket name are used to create the bucket, diff --git a/src/site/markdown/mkcsv.md b/src/site/markdown/mkcsv.md index 605c68c..d8b96ac 100644 --- a/src/site/markdown/mkcsv.md +++ b/src/site/markdown/mkcsv.md @@ -32,7 +32,7 @@ Usage: mkcsv ``` ```bash -hadoop jar cloudstore-1.4.jar mkcsv -header -quote -verbose 10000 s3a://bucket/file.csv +hadoop jar cloudstore-1.5.jar mkcsv -header -quote -verbose 10000 s3a://bucket/file.csv ``` The format is a variable width sequence, with entries cross referencing each other for validation. diff --git a/src/site/markdown/put.md b/src/site/markdown/put.md index b79d6a8..7d3a3c7 100644 --- a/src/site/markdown/put.md +++ b/src/site/markdown/put.md @@ -23,7 +23,7 @@ Put a file from a source path (local or remote), to a destination. * Supports the `-debug` mode for a low level trace of the network IO. ```bash -bin/hadoop jar cloudstore-1.4.jar put -options options -debug bin/hadoop s3a://target-london/hadoop +bin/hadoop jar cloudstore-1.5.jar put -options options -debug bin/hadoop s3a://target-london/hadoop ``` Here the options file adds a custom header to the object, and forces multipart IO on hadoop releases with diff --git a/src/site/markdown/sdk.md b/src/site/markdown/sdk.md index 815c6ad..32eafe2 100644 --- a/src/site/markdown/sdk.md +++ b/src/site/markdown/sdk.md @@ -23,7 +23,7 @@ Prints some of the low level diagnostics information about an S3 bucket which can be obtained via the AWS APIs. ``` -hadoop jar cloudstore-1.4.jar \ +hadoop jar cloudstore-1.5.jar \ bucketstate \ s3a://mybucket/ @@ -37,7 +37,7 @@ NONE If you don't have the permissions to read the bucket policy, you get a stack trace. ``` -hadoop jar cloudstore-1.4.jar \ +hadoop jar cloudstore-1.5.jar \ bucketstate \ s3a://mybucket/ @@ -54,7 +54,7 @@ com.amazonaws.services.s3.model.AmazonS3Exception: The specified method is not a Generate AWS IAM policy for a given bucket ``` -hadoop jar cloudstore-1.4.jar iampolicy s3a://example-bucket/ +hadoop jar cloudstore-1.5.jar iampolicy s3a://example-bucket/ { "Version" : "2012-10-17", @@ -107,7 +107,7 @@ Usage: listobjects <path> Example ``` -> hadoop jar cloudstore-1.4.jar listobjects -limit 500 s3a://aws-public-blockchain/v1.0/eth/ +> hadoop jar cloudstore-1.5.jar listobjects -limit 500 s3a://aws-public-blockchain/v1.0/eth/ 2026-05-28 19:42:27,734 [main] INFO sdk.ListObjects (StoreDurationInfo.java:<init>(84)) - Starting: listobjects @@ -153,7 +153,7 @@ This is how all AWS service clients determine the region for sending/signing req not explicitly set. ```bash -hadoop jar cloudstore-1.4.jar regions +hadoop jar cloudstore-1.5.jar regions Determining AWS region for SDK clients ====================================== diff --git a/src/site/markdown/storediag.md b/src/site/markdown/storediag.md index 8f02648..3b2c674 100644 --- a/src/site/markdown/storediag.md +++ b/src/site/markdown/storediag.md @@ -42,10 +42,10 @@ This is to support safer reporting of issues in bug reports within an organisati ```bash -hadoop jar cloudstore-1.4.jar storediag s3a://noaa-cors-pds/raw/2023/017/ -hadoop jar cloudstore-1.4.jar storediag -w --tokenfile mytokens.bin s3a://my-readwrite-bucket/subdirectory -hadoop jar cloudstore-1.4.jar storediag -w --tokenfile mytokens.bin hdfs://namenode/user/alice/subdir -hadoop jar cloudstore-1.4.jar storediag -reveal abfs://container@user/ +hadoop jar cloudstore-1.5.jar storediag s3a://noaa-cors-pds/raw/2023/017/ +hadoop jar cloudstore-1.5.jar storediag -w --tokenfile mytokens.bin s3a://my-readwrite-bucket/subdirectory +hadoop jar cloudstore-1.5.jar storediag -w --tokenfile mytokens.bin hdfs://namenode/user/alice/subdir +hadoop jar cloudstore-1.5.jar storediag -reveal abfs://container@user/ ``` The remote store is required to grant read access to the caller. @@ -90,7 +90,7 @@ a #-prefixed comment, a blank line, a classname, a resource (with "/" in). These are all loaded ```bash -hadoop jar cloudstore-1.4.jar storediag -5 -required required.txt s3a://something/ +hadoop jar cloudstore-1.5.jar storediag -5 -required required.txt s3a://something/ ``` and with a `required.txt` listing extra classes which must be on the classpath @@ -116,7 +116,7 @@ The comments are printed too. This means you can use them in the reports. ## Example: ``` -bin/hadoop jar cloudstore-1.4.jar storediag -w s3a://example-london/temp/subdir > output 2>&1 +bin/hadoop jar cloudstore-1.5.jar storediag -w s3a://example-london/temp/subdir > output 2>&1 ``` This does diagnostics test including a write against the bucket @@ -1275,7 +1275,7 @@ JVM: memory=203162280 ``` -> hadoop jar cloudstore-1.4.jar storediag abfs://[email protected]/ +> hadoop jar cloudstore-1.5.jar storediag abfs://[email protected]/ 1. Store Diagnostics for alice (auth:SIMPLE) on VXM63P4JG2/192.168.50.139 ========================================================================== diff --git a/src/site/markdown/tlsinfo.md b/src/site/markdown/tlsinfo.md index 9acfdf1..d01b9f3 100644 --- a/src/site/markdown/tlsinfo.md +++ b/src/site/markdown/tlsinfo.md @@ -23,7 +23,7 @@ Usage: tlsinfo [-verbose] [-debug] [<match>] * If a string is passed in, only certificates whose name contains that string (case independent) are logged. ``` -> hadoop jar cloudstore-1.4.jar tlsinfo +> hadoop jar cloudstore-1.5.jar tlsinfo 1. TLS System Properties ======================== @@ -223,7 +223,7 @@ See https://www.java.com/en/configure_crypto.html Print certificates containing a specific string, here everything with the string "amazon" in it. ``` -> hadoop jar cloudstore-1.4.jar tlsinfo amazon +> hadoop jar cloudstore-1.5.jar tlsinfo amazon 1. TLS System Properties ======================== @@ -269,7 +269,7 @@ Number of certificates matching the string "amazon" :4 A full dump of the details of matching certificates ``` -> hadoop jar cloudstore-1.4.jar tlsinfo -verbose amazon +> hadoop jar cloudstore-1.5.jar tlsinfo -verbose amazon 1. TLS System Properties @@ -543,7 +543,7 @@ KeyIdentifier [ Failure to find any match returns an exit code of -1 ``` -hadoop jar cloudstore-1.4.jar tlsinfo unknown +hadoop jar cloudstore-1.5.jar tlsinfo unknown 1. TLS System Properties ======================== diff --git a/src/site/markdown/versioned-objects.md b/src/site/markdown/versioned-objects.md index 1a4118a..04cc008 100644 --- a/src/site/markdown/versioned-objects.md +++ b/src/site/markdown/versioned-objects.md @@ -36,7 +36,7 @@ Lists all versions of files under a path. ## Usage ``` -hadoop jar cloudstore-1.4.jar listversions +hadoop jar cloudstore-1.5.jar listversions Usage: listversions <path> -D <key=value> Define a property -deleted include delete markers @@ -57,7 +57,7 @@ Usage: listversions <path> ## list to a file ```bash -hadoop jar cloudstore-1.4.jar listversions -out out.tsv -limit 10 s3a://example-london/ +hadoop jar cloudstore-1.5.jar listversions -out out.tsv -limit 10 s3a://example-london/ ``` Exports a version listing as a tab separated file. Any hadoop filesystem URI is supported as a destination. @@ -100,7 +100,7 @@ The `-q` option scans a bucket but only prints the summary ``` -hadoop jar cloudstore-1.4.jar listversions -q s3a://example-london/ +hadoop jar cloudstore-1.5.jar listversions -q s3a://example-london/ 2023-03-01 17:37:43,931 [main] INFO extra.ListVersions (StoreDurationInfo.java:<init>(56)) - Starting: listversions 2023-03-01 17:37:44,622 [main] WARN s3a.S3AFileSystem (S3AFileSystem.java:getAmazonS3ClientForTesting(1221)) - Access to S3A client requested, reason listversions @@ -139,7 +139,7 @@ A single file can be restored using the `restore` command, which will take a fil and create a new copy in a different location in the same bucket. ```bash -> hadoop jar cloudstore-1.4.jar restore s3a://example-london/FileSystemContractBaseTest "zn8c734o2CaH2ZfsS.I7_mpDl6kgh63O" /restored.txt +> hadoop jar cloudstore-1.5.jar restore s3a://example-london/FileSystemContractBaseTest "zn8c734o2CaH2ZfsS.I7_mpDl6kgh63O" /restored.txt restoring s3a://example-london/FileSystemContractBaseTest @ zn8c734o2CaH2ZfsS.I7_mpDl6kgh63O to s3a://example-london/restored.txt Starting: restore @@ -182,7 +182,7 @@ Preparation: create then delete a file Deleted s3a://example-london/undelete.txt # list versions including tombstones -bin/hadoop jar cloudstore-1.4.jar listversions -deleted s3a://example-london/undelete.txt +bin/hadoop jar cloudstore-1.5.jar listversions -deleted s3a://example-london/undelete.txt Starting: listversions 2023-03-03 15:28:16,552 [main] WARN s3a.S3AFileSystem (S3AFileSystem.java:getAmazonS3ClientForTesting(1216)) - Access to S3A client requested, reason listversions @@ -207,7 +207,7 @@ The object `s3a://example-london/undelete.txt` at version "ZJMKH.5PrEwiy9z1HbDqP Calling the undelete command will remove the tombstone ```bash -hadoop jar cloudstore-1.4.jar undelete s3a://example-london/undelete.txt +hadoop jar cloudstore-1.5.jar undelete s3a://example-london/undelete.txt 2023-03-03 15:30:59,755 [main] WARN s3a.S3AFileSystem (S3AFileSystem.java:getAmazonS3ClientForTesting(1216)) - Access to S3A client requested, reason undelete Starting: undelete @@ -244,7 +244,7 @@ must have been created on or after (hint: there are web pages to calculate this) of the command. ```bash -> hadoop jar cloudstore-1.4.jar undelete -age 24000 -limit 10 s3a://example-london/ +> hadoop jar cloudstore-1.5.jar undelete -age 24000 -limit 10 s3a://example-london/ 2023-03-03 15:39:18,964 [main] WARN s3a.S3AFileSystem (S3AFileSystem.java:getAmazonS3ClientForTesting(1216)) - Access to S3A client requested, reason undelete Starting: undelete --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
