Hi, Stephen and Steve.
Apache Spark community starts to publish it as a snapshot and Apache Spark
3.2.0 will be the first release has it.
-
https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hadoop-cloud_2.12/3.2.0-SNAPSHOT/
Please check the snapshot artifacts and
off the record: Really irritates me too, as it forces me to do local builds
even though I shouldn't have to. Sometimes I do that for other reasons, but
still.
Getting the cloud-storage module in was hard enough at the time that I
wasn't going to push harder; I essentially stopped trying to get
I have been building Apache Spark from source just so I can get this dependency.
1. git checkout v3.1.1
2. dev/make-distribution.sh --name hadoop-cloud-3.2 --tgz -Pyarn
-Phadoop-3.2 -Pyarn -Phadoop-cloud -Phive-thriftserver -Dhadoop.version=3.2.0
It is kind of a nuisance having to do
I know it's not enabled by default when the binary artifacts are built, but
not exactly sure why it's not built separately at all. It's almost a
dependencies-only pom artifact, but there are two source files. Steve do
you have an angle on that?
On Mon, May 31, 2021 at 5:37 AM Erik Torres wrote:
Hi,
I'm following this documentation to configure my Spark-based application to
interact with Amazon S3. However, I cannot find the spark-hadoop-cloud module
in Maven central for the non-commercial distribution of Apache Spark. From the
documentation I would expect that I can get this module
