Github user srowen commented on the issue: https://github.com/apache/spark/pull/12004 I have the impression that you can't really use Spark with S3 and only S3, not as an intermediate store, because it's too eventually-consistent. Does the presence of additional integration libraries alone change that, or am I mistaken? that is, I'm wondering whether this really does what it appears to say on the tin, which is to make Spark usable with just S3. My other question was indeed whether we need a different module if we're just about to only support 2.7, or 2.6/2.7. That's more of a detail of implementation. Does a build of Spark + Hadoop 2.7 right now have no ability at all to read from S3 out of the box, or just not full / ideal support? Finally, is the Spark build the best thing to provide these dependencies? well, it provides the core Hadoop FS support already, so yes. But on the other hand, Hadoop is farming this out as optional itself, it can be added by a user app (right?), it would already be present if running in the presence of a cluster. It looks like there's a little more to it than just adding one new dependency, but not much more. I'm working out just how much the module is needed for users to get it right, vs complexity. The docs are valuable.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org