Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/12004
sean: there's two things: tests and packaging.
1. The packaging has to go in as probably the only way to get whatever
spark is built with to be consistent. That includes excluding the stuff you
don't need.
1. If built against Hadoop <=2.5, spark includes the s3n client. With the
move to 2.6, the s3n client went to hadoop-aws JAR, where it's left out, even
when the jets3t JAR ends up on the CP
1. As for the docs, they cover the core setup problems, code issues and set
up expectations properly (no speculation, performance settings). There's
already `docs/storage-openstack-swift.md` a dated doc on openstack, so this
isn't any fundamental change.
1. The tests, I can move them out; put them somewhere our respective QE
teams can use for jenkins and integration test runs.
1. the unit ones to check classpath/init are still handy to keep in, for a
failfast on classpath inconsistencies.
That would leave a more minimal patch of package: a module, some docs, a
few tests that don't need any blobstore credentials, And, for others in the
future, provide somewhere to put object store-specific code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
