>
> +1 binding.
>
> One caveat: warn people that guava is now at 27.0 -and that if you run
> with an older version of Guava things will inevitably break.
>
>
> steps to validate
> ==================
>
> * downloaded src and binary artifacts
> * after import of KEYS and trusting Rohith's key, validate GPG signatures
> * test basic hadoop fs commands against s3a with s3guard and abfds
>
>
>
>
> Validating S3A connector
> ========================
>
> * grabbed the latest build of my cloudstore diagnostics JAR
> https://github.com/steveloughran/cloudstore/releases/tag/tag_2019-09-13
> * and set an env var to it:
> set -gx CLOUDSTORE cloudstore/target/cloudstore-0.1-SNAPSHOT.jar
>
> bin/hadoop jar $CLOUDSTORE storediag s3a://hwdev-steve-ireland-new
>
>
>
> Diagnostics for filesystem s3a://hwdev-steve-ireland-new/
> =========================================================
>
> S3A FileSystem connector
> ASF Filesystem Connector to Amazon S3 Storage and compatible stores
>
> https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
>
> Hadoop information
> ==================
>
> Hadoop 3.2.1
> Compiled by rohithsharmaks on 2019-09-10T15:56Z
> Compiled with protoc 2.5.0
> From source with checksum 776eaf9eee9c0ffc370bcbc1888737
>
> Required Classes
> ================
>
> All these classes must be on the classpath
>
> class: org.apache.hadoop.fs.s3a.S3AFileSystem
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar
> class: com.amazonaws.services.s3.AmazonS3
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar
> class: com.amazonaws.ClientConfiguration
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar
>
> Optional Classes
> ================
>
> These classes are needed in some versions of Hadoop.
> And/or for optional features to work.
>
> class: com.amazonaws.services.dynamodbv2.AmazonDynamoDB
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar
> class: com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar
> class: com.fasterxml.jackson.annotation.JacksonAnnotation
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-annotations-2.9.8.jar
> class: com.fasterxml.jackson.core.JsonParseException
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-2.9.8.jar
> class: com.fasterxml.jackson.databind.ObjectMapper
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-databind-2.9.8.jar
> class: org.joda.time.Interval
> Not found on classpath: org.joda.time.Interval
> class: org.apache.hadoop.fs.s3a.s3guard.S3Guard
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar
> class: org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar
> class: org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar
> class: org.apache.hadoop.fs.s3a.Invoker
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar
> class: org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar
>
> then some classes which aren't in 3.2 and so which I didn't expect to
> see.
>
> class: org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens
> Not found on classpath:
> org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens
> class: com.amazonaws.services.s3.model.SelectObjectContentRequest
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar
> class: org.apache.hadoop.fs.s3a.select.SelectInputStream
> Not found on classpath:
> org.apache.hadoop.fs.s3a.select.SelectInputStream
> class: org.apache.hadoop.fs.s3a.impl.RenameOperation
> Not found on classpath:
> org.apache.hadoop.fs.s3a.impl.RenameOperation
>
>
>
> + the command then executed basic list/read/write operations; all good.
>
>
>
> Validating abfs connector
> =========================
>
> * set -gx HADOOP_OPTIONAL_TOOLS hadoop-azure
>
>
>
>
> Diagnostics for filesystem abfs://contai...@someone.dfs.core.windows.net/
>
> ============================================================================
>
> Azure Abfs connector
> ASF Filesystem Connector to Microsoft Azure ABFS Storage
> https://hadoop.apache.org/docs/current/hadoop-azure/index.html
>
> Hadoop information
> ==================
>
> Hadoop 3.2.1
> Compiled by rohithsharmaks on 2019-09-10T15:56Z
> Compiled with protoc 2.5.0
> From source with checksum 776eaf9eee9c0ffc370bcbc1888737
>
> Environment Variables
> =====================
>
> HADOOP_HOME = "/Users/stevel/hadoop/candidates/hadoop-3.2.1"
> HADOOP_CONF_DIR = "/Users/stevel/hadoop-3.2.1/etc/hadoop"
> HADOOP_OPTIONAL_TOOLS =
> "hadoop-azure,hadoop-aws,hadoop-azure-datalake,hadoop-openstack"
> HADOOP_SHELL_SCRIPT_DEBUG = (unset)
> HADOOP_TOOLS_HOME = (unset)
> HADOOP_TOOLS_OPTIONS = (unset)
> HDP_VERSION = (unset)
> SPARK_HOME = (unset)
> SPARK_CONF_DIR = (unset)
> PYSPARK_DRIVER_PYTHON = (unset)
> SPARK_SCALA_VERSION = (unset)
> YARN_CONF_DIR = (unset)
> HADOOP_TOKEN_FILE_LOCATION = (unset)
> HADOOP_TOKEN = (unset)
>
>
>
> Hadoop Options
> ==============
>
> fs.defaultFS = "file:///" [core-default.xml]
> fs.default.name = "file:///"
> hdp.version = (unset)
> yarn.resourcemanager.address = "0.0.0.0:8032" [yarn-default.xml]
> yarn.resourcemanager.webapp.address = "0.0.0.0:8088" [yarn-default.xml]
> yarn.resourcemanager.webapp.https.address = "0.0.0.0:8090"
> [yarn-default.xml]
> mapreduce.input.fileinputformat.list-status.num-threads = "1"
> [mapred-default.xml]
>
>
> Required Classes
> ================
>
> All these classes must be on the classpath
>
> class: com.fasterxml.jackson.annotation.JsonProperty
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-annotations-2.9.8.jar
> class: com.google.common.base.Preconditions
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar
> class: com.fasterxml.jackson.core.JsonFactory
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-2.9.8.jar
> class: com.fasterxml.jackson.databind.ObjectReader
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-databind-2.9.8.jar
> class: com.microsoft.azure.storage.StorageErrorCode
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/azure-storage-7.0.0.jar
> class: org.apache.http.client.utils.URIBuilder
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/httpclient-4.5.6.jar
> class: org.wildfly.openssl.OpenSSLProvider
>
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/wildfly-openssl-1.0.7.Final.jar
> class: org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
>
> file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-azure-3.2.1.jar
>
> -again a full filesystem list/read/write/delete sequence was executed;
> again all good.
>
>
> -------------------------------------------
>
> spark
> =====
>
>
> (branch master; version 3.0.0-SNAPSHOT)
>
> After editing my maven settings to avoid our in-house nexus server, I did
> a clean build with the profiles:
>
> -Dhadoop.version=3.2.1 -Phive -Phive-thriftserver -Pyarn -Phadoop-3.2
> -Phadoop-cloud -Psnapshots-and-staging
>
>
> Downloading from ASF Staging:
> https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/3.2.1/hadoop-project-3.2.1.pom
> Downloaded from ASF Staging:
> https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/3.2.1/hadoop-project-3.2.1.pom
> (70 kB at 72 kB/s)
>
>
> For the record, three issues surfaced there
> * spark now demands mvn 3.6.2, so upgrade time there
> * for some reason the build decided to download all the other artifacts
> that maven central has ever hosted
> * spark's build hangs if you pass in the -T 1C argument to maven. Worth
> remembering.
> * and of couse you have to rebuild the next day.
>
> Spark compilation completed: I did not run its test suites.
>
> spark cloud integration tests
> =============================
>
> Once spark was built, I went with my downstream test suite.
> https://github.com/hortonworks-spark/cloud-integration
>
> Here here I got burned at runtime by the guava change; the module was set
> up to use guava 0.19, and Configuration.set() failed.
>
> S3ACommitBulkDataSuite:
> com.cloudera.spark.cloud.s3.commit.S3ACommitBulkDataSuite *** ABORTED ***
> java.lang.RuntimeException: Unable to load a Suite class that was
> discovered in the runpath:
>
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
> at
> com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$2(CloudSuiteTrait.scala:145)
> at
> com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$2$adapted(CloudSuiteTrait.scala:144)
> at scala.Option.foreach(Option.scala:274)
> at
> com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$1(CloudSuiteTrait.scala:144)
> at
> com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$1$adapted(CloudSuiteTrait.scala:143)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> com.cloudera.spark.cloud.common.CloudSuiteTrait.overlayConfiguration(CloudSuiteTrait.scala:143)
> at
> com.cloudera.spark.cloud.common.CloudSuiteTrait.overlayConfiguration$(CloudSuiteTrait.scala:142)
> at
> com.cloudera.spark.cloud.common.CloudSuite$.overlayConfiguration(CloudSuite.scala:41)
> ...
>
> This is one of those things which was inevitable. I do think we'll need to
> warn people in the release notes.
>
> (I also discovered someone has deleted a test source data bucket of mine
> so some tests failed; not something I fixed today.)
>
> With the guava tests, other than a failure due to someone deleting a
> bucket of mine containing test source data, all tests against s3a and abfs
> were happy.
>
>
>
