[GitHub] spark pull request: SPARK-1668: Add implicit preference as an opti...

2014-05-03 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/597#issuecomment-42125991 On this note, recall there was a change a while back to handle the case of negative confidence levels. 0 still means "don't know" and positive values mean "confident that t

[GitHub] spark pull request: SPARK-1693: Most of the tests throw a java.lan...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/628#issuecomment-42125947 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: SPARK-1693: Most of the tests throw a java.lan...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/628#issuecomment-42125950 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1693: Most of the tests throw a java.lan...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/628#issuecomment-42125871 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1693: Most of the tests throw a java.lan...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/628#issuecomment-42125857 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark pull request: SPARK-1693: Most of the tests throw a java.lan...

2014-05-03 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/628 SPARK-1693: Most of the tests throw a java.lang.SecurityException when s... ...park built for hadoop 2.3.0 , 2.4.0 You can merge this pull request into a Git repository by running: $ git pull http

[GitHub] spark pull request: SPARK-1668: Add implicit preference as an opti...

2014-05-03 Thread MLnick
Github user MLnick commented on the pull request: https://github.com/apache/spark/pull/597#issuecomment-42125746 It is true that implicit prefs predict 0/1 (ie a "preference" matrix rather than a "rating" matrix), but the ratings are taken as confidence levels indicating preference (o

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/626#issuecomment-42125591 Great - thanks for paring this down. I can merge it. Let's look at cleaning this up once we ship 1.0. --- If your project is set up for it, you can reply to this email a

[GitHub] spark pull request: [SQL] Whitelist Hive Tests

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/596 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: [SQL] Whitelist Hive Tests

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/596#issuecomment-42125301 Thanks - I merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/627#issuecomment-42124758 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/627#issuecomment-42124759 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14639/ --- If your project

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259904 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- I tested the current enviro

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259852 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Yes, I'm sorry, my englis

[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/627#issuecomment-42124219 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/627#issuecomment-42124220 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...

2014-05-03 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/627 SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7. This add some guards and good warning messages if users hit this issue. /cc @aarondav with whom I discussed parts of the design.

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259828 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Ah I see. I didn't under

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259715 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- When `hadop.version` is 1.

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259635 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Leaving the dependency decl

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259629 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Yes, I understand, but l

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259611 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- In `mvn -DskipTests clean p

[GitHub] spark pull request: [SQL] Whitelist Hive Tests

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/596#issuecomment-42123156 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14638/ --- If your project

[GitHub] spark pull request: [SQL] Whitelist Hive Tests

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/596#issuecomment-42123155 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259452 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- I don't mind requiring t

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259437 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Could you explain what y

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-05-03 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/577#issuecomment-42122270 > will be good to enforce proper resource lifecycle. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...

2014-05-03 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/549#issuecomment-42122245 Is there any further suggestion about this one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SQL] Whitelist Hive Tests

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/596#issuecomment-42122046 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SQL] Whitelist Hive Tests

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/596#issuecomment-42122047 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259340 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- |maven| hadoop.version | y

[GitHub] spark pull request: [SQL] Better logging when applying rules.

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/616 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259307 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- You're right. but in `mvn

[GitHub] spark pull request: [SQL] Better logging when applying rules.

2014-05-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/616#issuecomment-42121390 Thanks. Merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark pull request: EC2 configurable workers

2014-05-03 Thread douglaz
Github user douglaz commented on the pull request: https://github.com/apache/spark/pull/612#issuecomment-42121342 Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [WIP][Spark-SQL] Optimize the Constant Folding...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/482#issuecomment-42121295 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [WIP][Spark-SQL] Optimize the Constant Folding...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/482#issuecomment-42121296 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14637/ --- If your project

[GitHub] spark pull request: [SQL] Better logging when applying rules.

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/616#issuecomment-42121275 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SQL] Better logging when applying rules.

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/616#issuecomment-42121276 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14636/ --- If your project

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259085 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- What's the problem exact

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12259068 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- This is not very good, othe

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12258922 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Bascially what I want he

[GitHub] spark pull request: The default version of yarn is equal to the ha...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/626#discussion_r12258918 --- Diff: pom.xml --- @@ -558,65 +560,8 @@ jets3t 0.7.1 - --- End diff -- Could you also remove th

[GitHub] spark pull request: Fix SPARK-1629: Spark should inline use of com...

2014-05-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/569#issuecomment-42120375 Maven dependency still shows org.apache.commons:commons-lang3:jar - am I missing something here ? Btw, we do depend on it for repl tests too ... --- If your proj

[GitHub] spark pull request: Improve build configuration � �

2014-05-03 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/590#issuecomment-42120300 @pwendell I did not notice here, has been modified --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If yo

[GitHub] spark pull request: EC2 configurable workers

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/612 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: [WIP][Spark-SQL] Optimize the Constant Folding...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/482#issuecomment-42120166 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SQL] Better logging when applying rules.

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/616#issuecomment-42120164 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [WIP][Spark-SQL] Optimize the Constant Folding...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/482#issuecomment-42120168 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SQL] Better logging when applying rules.

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/616#issuecomment-42120167 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP][Spark-SQL] Optimize the Constant Folding...

2014-05-03 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/482#issuecomment-42120157 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [WIP][Spark-SQL] Optimize the Constant Folding...

2014-05-03 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/482#issuecomment-42120149 Seems the test didn't restart. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: EC2 configurable workers

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/612#issuecomment-42119736 I merged this. I also opened up a new JIRA to just allow us to launch multiple executors for a given app on the same node: https://issues.apache.org/jira/browse/S

[GitHub] spark pull request: EC2 configurable workers

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/612#issuecomment-42119635 Thanks - this is a nice feature. I played with this locally and it worked. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-1620] Handle uncaught exceptions in fun...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/622#issuecomment-42118495 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14635/ --- If your project

[GitHub] spark pull request: [SPARK-1620] Handle uncaught exceptions in fun...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/622#issuecomment-42118494 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-1620] Handle uncaught exceptions in fun...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/622#issuecomment-42117741 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1620] Handle uncaught exceptions in fun...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/622#issuecomment-42117738 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: EC2 configurable workers

2014-05-03 Thread douglaz
Github user douglaz commented on the pull request: https://github.com/apache/spark/pull/612#issuecomment-42117075 Done: https://issues.apache.org/jira/browse/SPARK-1705 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42116990 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14634/ --- If your project is set up for it, you can r

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42116989 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: SPARK-1689 AppClient should indicate app is de...

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/605 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: [Bugfix] Tachyon file cleanup logical error

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/575 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: Remove hardcoded Spark version string to use S...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/600#issuecomment-42116524 @pierre-borckmans Appreciate you looking into this, but I really don't think it's worth doing anything complex here - we just just leave it hard coded. We already

[GitHub] spark pull request: Fix SPARK-1629: Spark should inline use of com...

2014-05-03 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/569#issuecomment-42116492 The problem was that the project was not depending on commons lang already. The code is the same as is contained in commons lang. The property being referenced is a s

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/580#issuecomment-42116271 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14633/ --- If your project

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/580#issuecomment-42116270 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: Remove hardcoded Spark version string to use S...

2014-05-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/600#issuecomment-42115749 Please ensure it works in both sbt and maven case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Fix SPARK-1629: Spark should inline use of com...

2014-05-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/569#issuecomment-42115677 I am still catching up on PR's and bugs. Why was this changed ? Hacky solutions based on string parsing of properties lead to fragility in case of changes in future

[GitHub] spark pull request: SPARK-1689 AppClient should indicate app is de...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/605#issuecomment-42115523 Looks good, thanks Aaron. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [Bugfix] Tachyon file cleanup logical error

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/575#issuecomment-42115458 Thanks - merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42115441 Added Python API changes. Skipped filter() as I don't see a way to translate Python functions into Java land. Py4j supports implementing Java interfaces in Python callbac

[GitHub] spark pull request: SPARK-1663. Corrections for several compile er...

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/589 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/580#issuecomment-42115310 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42115311 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42115307 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/580#issuecomment-42115306 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/580#issuecomment-42115272 Good catch! Made some small comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/580#issuecomment-42115276 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/580#discussion_r12257998 --- Diff: make-distribution.sh --- @@ -43,12 +43,13 @@ FWDIR="$(cd `dirname $0`; pwd)" DISTDIR="$FWDIR/dist" -VERSION=$(mvn help:evaluate -

[GitHub] spark pull request: SPARK-1658: Correctly identify if maven is ins...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/580#discussion_r12257989 --- Diff: make-distribution.sh --- @@ -43,12 +43,13 @@ FWDIR="$(cd `dirname $0`; pwd)" DISTDIR="$FWDIR/dist" -VERSION=$(mvn help:evaluate -

[GitHub] spark pull request: EC2 configurable workers

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/612#issuecomment-42114608 Do you mind creating a JIRA for this on the spark issue tracker? https://issues.apache.org/jira/browse/SPARK It would be good to call it something like: "Allow

[GitHub] spark pull request: SPARK-1663. Corrections for several compile er...

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/589#issuecomment-42114207 @srowen these are all great - thanks! I'm going to merge it. For the reduceByKeyAndWindow issue, I think that's just an oversight that occurred during the Java 8

[GitHub] spark pull request: SPARK-1663. Corrections for several compile er...

2014-05-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/589#discussion_r12257762 --- Diff: docs/streaming-programming-guide.md --- @@ -684,7 +694,7 @@ operation `reduceByKeyAndWindow`. {% highlight scala %} // Reduce last 30

[GitHub] spark pull request: Improve build configuration � �

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/590#issuecomment-42113629 @witgo - would you mind isolating (1) and (3) and putting them in a separate pull request? Those I think should go in ASAP. #626 still has a bunch of other changes. I mig

[GitHub] spark pull request: SPARK-1556: bump jets3t version to 0.9.0

2014-05-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/468#issuecomment-42113320 @srowen YARN version does need to be separate from hadoop version. Downstream consumers of our build sometimes do this. For instance, if they want to build against a cust

[GitHub] spark pull request: [WIP] SPARK-1676: Cache Hadoop UGIs by default...

2014-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/621 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: SPARK-1556: bump jets3t version to 0.9.0

2014-05-03 Thread berngp
Github user berngp commented on the pull request: https://github.com/apache/spark/pull/468#issuecomment-42112284 I think in general is an edge case but there are folks still using hdfs 1.0.x with a different version of YARN, that said it is not my case. I like what you sugg

[GitHub] spark pull request: [WIP] SPARK-1676: Cache Hadoop UGIs by default...

2014-05-03 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/621#issuecomment-42111962 LGTM too. Thanks for the clarifications, guys. Merging into master, branch-1.0, and branch-0.9. --- If your project is set up for it, you can reply to this email and hav

[GitHub] spark pull request: SPARK-1667 re-fetch fails occasionally

2014-05-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/604#issuecomment-4256 Can you add a testcase to verify this ? Where None is returned and validated. I suspect we have also observed in the past too - but I never got around to naili

[GitHub] spark pull request: SPARK-1667 re-fetch fails occasionally

2014-05-03 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/604#discussion_r12257245 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -71,13 +71,13 @@ private[spark] class DiskBlockManager(shuffleManager: Sh

[GitHub] spark pull request: [WIP] SPARK-1676: Cache Hadoop UGIs by default...

2014-05-03 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/621#issuecomment-42111008 This does look good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-1667 re-fetch fails occasionally

2014-05-03 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/604#discussion_r12257233 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -203,14 +203,22 @@ object BlockFetcherIterator { // these a

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-05-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/577#issuecomment-42110570 Might be good idea to abstract out the try/finally idiom out. @mateiz, any thoughts ? We have a bunch of places where resource cleanup does not happen properly - which

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-05-03 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/577#discussion_r12257197 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -77,7 +77,12 @@ private class DiskStore(blockManager: BlockManager, diskManager:

[GitHub] spark pull request: SPARK-1623. Broadcast cleaner should use getCa...

2014-05-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/546#issuecomment-42110226 It is not about a few uses here or there - either spark codebase as a whole moves to a) canonical path always; or always sticks to b) paths relative to cwd and/or what is

[GitHub] spark pull request: SPARK-1544 Add support for deep decision trees...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/475#issuecomment-42110111 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14632/ --- If your project

[GitHub] spark pull request: SPARK-1544 Add support for deep decision trees...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/475#issuecomment-42110110 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-1556: bump jets3t version to 0.9.0

2014-05-03 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/468#issuecomment-42110042 @srowen Related discussion in [PR 502](https://github.com/apache/spark/pull/502). @berngp Can you explain the reason of not using the same version of HDFS vs YARN ?

[GitHub] spark pull request: SPARK-1556: bump jets3t version to 0.9.0

2014-05-03 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/468#issuecomment-42109781 @witgo Hm, is there an example that comes up repeatedly? Is it ever intentional, or just some accident of someone's legacy deployment? I don't know of a case of this, and

  1   2   >