[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14646 thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14646 OK for the sake of moving it forward I'm going to merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14646 Has anyone had a chance to review this. It's nicely self-contained, makes it easier to use Spark as regression testing for ASF prerelease binaries of any dependent project. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66019/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14646 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14646 **[Test build #66019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66019/consoleFull)** for PR 14646 at commit [`09d96be`](https://github.com/apache/spark/commit/09d96bec29c9ac9a58966c603c2c6b3b642eb9d2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14646 **[Test build #66019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66019/consoleFull)** for PR 14646 at commit [`09d96be`](https://github.com/apache/spark/commit/09d96bec29c9ac9a58966c603c2c6b3b642eb9d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14646 I'm adding the ability to test against staged releases, such as Hadoop 2.7.3 RC1. Add this profile and testing that spark runs with the new RC is a matter of setting the version with a -D and ask for staging artifacts -there's no need to edit the POMs at all: ``` dev/make-distribution.sh -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 ``` If all I wanted to do was test with locally built stuff, I wouldn't need the profile; just do the `mvn install` in Hadoop then build spark with `-Dhadoop.version=2.8.0-SNAPSHOT`; this works perfectly well. What this patch adds is the ability to test against the real ASF RC artifacts, so do regression testing against them. I used this as part of the review of the RC; it'll need to be repeated when the 2.8.x RCs are out. ``` +1 binding 1. built and tested apache slider (incubating) against the Hadoop 2.7.3 artifacts 2. did a build & test of Apache Spark master branch iwth 2.7.3 JARs, For that I had to tweak spark's build to support the staging repo; hopefully that will get into Spark https://issues.apache.org/jira/browse/SPARK-17058 3. did a test run of my WiP SPARK-7481 spark-cloud module; after fixing a couple of things on the test setup side related to HADOOP-13058, mvn test --pl cloud -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 -Dcloud.test.configuration.file=../conf/cloud-tests.xml all was well âalbeit measurably slower than Hadoop 2.8. That's proof that the 2.8 version of s3a really does deliver measurable speedup for those tests (currently just file input/seek; more to come). I had originally thought things were broken as s3 init was failing -but that's because the s3 bucket was in frankfurt, and the AWS library used can't talk to that endpoint (v4 auth protocol, see). 4. did a full spark distribution build of that SPARK-7481 branch dev/make-distribution.sh -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 ran command line test to do read of s3a data: bin/spark-submit --class org.apache.spark.cloud.s3.examples.S3LineCount \ --conf spark.hadoop.fs.s3a.access.key=$AWS_KEY \ --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET \ examples/jars/spark-examples_2.11-2.1.0-SNAPSHOT.jar 5. Pulled out the microsoft Azure JAR azure-storage-2.0.0.jar and repeated step 4 -this showed that the 2.7.x branch does handle the failure to load a filesystem due to dependency or other classloading problems âthis was proving a big problem in adding the aws & azure stuff to the spark build, as it'd stop spark from starting up if the dependencies were absent. I've not done any of the .tar.gz diligence; I've just looked at the staged JARs and how they worked with downstream apps âthat being a key way that Hadoop artifacts are adopted. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14646 Are you thinking about testing Hadoop versions that are unpublished? But if that's the case, you'd need to modify the version anyway. Why not just add this when you need to do those tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14646 I'd be against making it default for a few reasons 1. You don't want to accidentally pick up some staging artifact or upstream snapshot. 2. I don't know how SBT/Ivy handles remote staging artifact location; having looked through existing JIRAs related to differences between SBT and maven repositories, I didn't want to diverge things. 3. If you are doing local dev of upstream code, it's actually a real pain when a build suddenly decides to get a remote snapshot of an artifact which you haven't built locally that day. You generally do prefer the build to halt than for it to pick up some remote snapshot which doesn't have your code in. Best bit: maven will do this inside a project itself if the build spans midnight. Keeping it isolated avoids that. What I could do is add a section on this to the building-spark doc, to avoid people having to read through the POM to find it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14646 How about adding this to the default set of repos rather than putting it behind a profile -- does that cause a problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14646 Note that Jenkins, being SBT-based, isn't going to explore the codepath here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63782/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14646 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14646 **[Test build #63782 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63782/consoleFull)** for PR 14646 at commit [`09d96be`](https://github.com/apache/spark/commit/09d96bec29c9ac9a58966c603c2c6b3b642eb9d2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14646: [SPARK-17058] [build] Add maven snapshots-and-staging pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14646 **[Test build #63782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63782/consoleFull)** for PR 14646 at commit [`09d96be`](https://github.com/apache/spark/commit/09d96bec29c9ac9a58966c603c2c6b3b642eb9d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
