[GitHub] spark pull request: [SPARK-4237][BUILD] Fix MANIFEST.MF in maven a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3103#issuecomment-62295132
  
  [Test build #23116 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23116/consoleFull)
 for   PR 3103 at commit 
[`8332304`](https://github.com/apache/spark/commit/8332304f00130c4a7ff429d3892c55e02494a0c0).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4237][BUILD] Fix MANIFEST.MF in maven a...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/3103#issuecomment-62295066
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4250] [SQL] Fix bug of constant null va...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/3114#issuecomment-62294967
  
@marmbrus The problem @chenghao-intel pointed out can be illustrated by 
this Scala shell snippet:

```
scala> def foo(v: Any) = v match {
 |   case _: String => "string"
 |   case _ => "other"
 | }
foo: (v: Any)String

scala> val s: String = null
s: String = null

scala> foo(s)
res0: String = other
```

That's why type annotations in `toInspector` are removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62293983
  
  [Test build #23115 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23115/consoleFull)
 for   PR 3159 at commit 
[`5dcd602`](https://github.com/apache/spark/commit/5dcd602ca04d90d80066d6405920a684749aeea4).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62293906
  
  [Test build #23114 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23114/consoleFull)
 for   PR 3159 at commit 
[`5dcd602`](https://github.com/apache/spark/commit/5dcd602ca04d90d80066d6405920a684749aeea4).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62293843
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3173#issuecomment-62293224
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23113/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3173#issuecomment-62293221
  
  [Test build #23113 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23113/consoleFull)
 for   PR 3173 at commit 
[`cc4647d`](https://github.com/apache/spark/commit/cc4647d2332806026fe592c5798f0900ba001744).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class ClusteredOrderedDistribution(clustering: Seq[Expression]) 
extends Distribution `
  * `case class HashSortedPartitioning(expressions: Seq[Expression], 
numPartitions: Int)`
  * `case class MergeJoin(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3173#issuecomment-62293022
  
  [Test build #23113 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23113/consoleFull)
 for   PR 3173 at commit 
[`cc4647d`](https://github.com/apache/spark/commit/cc4647d2332806026fe592c5798f0900ba001744).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[Ishiihara]

Github user Ishiihara commented on a diff in the pull request:

https://github.com/apache/spark/pull/3173#discussion_r20056877
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -84,6 +84,10 @@ private[sql] abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
   makeBroadcastHashJoin(leftKeys, rightKeys, left, right, 
condition, joins.BuildLeft)
 
   case ExtractEquiJoinKeys(Inner, leftKeys, rightKeys, condition, 
left, right) =>
+val mergeJoin = joins.MergeJoin(leftKeys, rightKeys, 
planLater(left), planLater(right))
+condition.map(Filter(_, mergeJoin)).getOrElse(mergeJoin) :: Nil
+  
+  case ExtractEquiJoinKeys(Inner, leftKeys, rightKeys, condition, 
left, right) =>
--- End diff --

You are right, the hash inner join will not be matched. I am trying to 
resolve to merge join on inner join. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[aarondav]

Github user aarondav commented on a diff in the pull request:

https://github.com/apache/spark/pull/3173#discussion_r20056856
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -84,6 +84,10 @@ private[sql] abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
   makeBroadcastHashJoin(leftKeys, rightKeys, left, right, 
condition, joins.BuildLeft)
 
   case ExtractEquiJoinKeys(Inner, leftKeys, rightKeys, condition, 
left, right) =>
+val mergeJoin = joins.MergeJoin(leftKeys, rightKeys, 
planLater(left), planLater(right))
+condition.map(Filter(_, mergeJoin)).getOrElse(mergeJoin) :: Nil
+  
+  case ExtractEquiJoinKeys(Inner, leftKeys, rightKeys, condition, 
left, right) =>
--- End diff --

just passing through, but is this right? It appears to be the same case as 
above, so I'm not sure when it would be matched.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3173#issuecomment-62292538
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23112/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3173#issuecomment-62292537
  
  [Test build #23112 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23112/consoleFull)
 for   PR 3173 at commit 
[`5cb98c3`](https://github.com/apache/spark/commit/5cb98c306f76183e4148d9b0a6b0a8ce4d58368e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class ClusteredOrderedDistribution(clustering: Seq[Expression]) 
extends Distribution `
  * `case class HashSortedPartitioning(expressions: Seq[Expression], 
numPartitions: Int)`
  * `case class MergeJoin(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3173#issuecomment-62292327
  
  [Test build #23112 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23112/consoleFull)
 for   PR 3173 at commit 
[`5cb98c3`](https://github.com/apache/spark/commit/5cb98c306f76183e4148d9b0a6b0a8ce4d58368e).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213][SQL] Sort Merge Join

[Ishiihara]

GitHub user Ishiihara opened a pull request:

https://github.com/apache/spark/pull/3173

[SPARK-2213][SQL] Sort Merge Join 

This PR adds MergeJoin operator to Spark SQL. The semantics of MergeJoin 
operator is similar to Hive's Sort merge bucket join.  

MergeJoin operator relies on SortBasedShuffle to create partitions that 
sorted by the join key. In each partition, we merge the two child iterators. 
The tricky part in merge step is handling duplicate join keys. To handle 
duplicate keys, we use a buffer to store all matching elements in right 
iterator for a certain join key. The buffer is used for generating join tuples 
when the join key of the next left element is the same as the current join key. 

MergeJoin reduces extra memory consumption, in the current implementation, 
MergeJoin only needs memory that can hold elements of the key that has the most 
duplicates in right iterator. 

For query optimization,  we may resolve to MergeJoin when both relations 
are large and neither of the two can fit in memory. Currently, this heuristic 
is not added to optimizer. I would appreciate if you can add comments on how to 
resolve to MergeJoin in optimizer. 
 
Currently, MergeJoin only supports inner join. However, it can be extended 
to support outer join. Will handle outer join in separate PRs. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ishiihara/spark SparkSQL-merge

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3173


commit 1c41f6f248f1145c7d730129795e50bdd8a53f2b
Author: Liquan Pei 
Date:   2014-10-28T23:47:35Z

initial commit

commit dc6a6840e2d2b1681e70a6a3eeb10d7a9e6437ce
Author: Liquan Pei 
Date:   2014-10-29T00:17:59Z

add MergeJoin.scala

commit f5ef4624aea5304ffdcc8daf5fbebc20943c3cf4
Author: Liquan Pei 
Date:   2014-11-09T04:05:56Z

Merge join working

commit b13cc4526f0098386b64bce50b2f983f95709f23
Author: Liquan Pei 
Date:   2014-11-09T04:09:11Z

Merge remote-tracking branch 'upstream/master' into SparkSQL-merge

Conflicts:
sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala

commit d6b6e7b8194682c713400823a4fd17e0419d89e4
Author: Liquan Pei 
Date:   2014-11-09T04:51:04Z

add inline comments for merge join

commit 837eb081e6382a23b4fd67a5265188aab1c7e305
Author: Liquan Pei 
Date:   2014-11-09T05:02:17Z

use merge join as inner join operator in JoinSuite

commit 5cb98c306f76183e4148d9b0a6b0a8ce4d58368e
Author: Liquan Pei 
Date:   2014-11-09T05:30:52Z

improve inline comments




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3172#issuecomment-62291934
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23111/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3172#issuecomment-62291933
  
  [Test build #23111 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23111/consoleFull)
 for   PR 3172 at commit 
[`04cddc8`](https://github.com/apache/spark/commit/04cddc82343a698b4b128782c2982d646afd0672).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `public final class LazyFileRegion extends AbstractReferenceCounted 
implements FileRegion `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62291931
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23109/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Minor fix for TreeNode

[scwf]

Github user scwf closed the pull request at:

https://github.com/apache/spark/pull/3171


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62291927
  
  [Test build #23109 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23109/consoleFull)
 for   PR 3159 at commit 
[`0b9f0f4`](https://github.com/apache/spark/commit/0b9f0f4a0d87a5bb3b2fe39bcb1fc568c552f6e7).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[aarondav]

Github user aarondav commented on a diff in the pull request:

https://github.com/apache/spark/pull/3172#discussion_r20056740
  
--- Diff: 
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockManager.java
 ---
@@ -56,16 +58,24 @@
   // Single-threaded Java executor used to perform expensive recursive 
directory deletion.
   private final Executor directoryCleaner;
 
-  public ExternalShuffleBlockManager() {
+  private final TransportConf conf;
+
+  public ExternalShuffleBlockManager(TransportConf conf) {
 // TODO: Give this thread a name.
-this(Executors.newSingleThreadExecutor());
+this(Executors.newSingleThreadExecutor(), conf);
   }
 
   // Allows tests to have more control over when directories are cleaned 
up.
   @VisibleForTesting
-  ExternalShuffleBlockManager(Executor directoryCleaner) {
+  ExternalShuffleBlockManager(Executor directoryCleaner, TransportConf 
conf) {
--- End diff --

nit: please make conf first parameter to be consistent with other classes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Minor fix for TreeNode

[scwf]

Github user scwf commented on the pull request:

https://github.com/apache/spark/pull/3171#issuecomment-62291925
  
Hmm, remove ```case null``` leads to  code generation failed, so close this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[aarondav]

Github user aarondav commented on the pull request:

https://github.com/apache/spark/pull/3172#issuecomment-62291922
  
LGTM, just a few minor style nits.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[aarondav]

Github user aarondav commented on a diff in the pull request:

https://github.com/apache/spark/pull/3172#discussion_r20056737
  
--- Diff: 
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockManager.java
 ---
@@ -56,16 +58,24 @@
   // Single-threaded Java executor used to perform expensive recursive 
directory deletion.
   private final Executor directoryCleaner;
 
-  public ExternalShuffleBlockManager() {
+  private final TransportConf conf;
+
+  public ExternalShuffleBlockManager(TransportConf conf) {
 // TODO: Give this thread a name.
-this(Executors.newSingleThreadExecutor());
+this(Executors.newSingleThreadExecutor(), conf);
   }
 
   // Allows tests to have more control over when directories are cleaned 
up.
   @VisibleForTesting
-  ExternalShuffleBlockManager(Executor directoryCleaner) {
+  ExternalShuffleBlockManager(Executor directoryCleaner, TransportConf 
conf) {
 this.executors = Maps.newConcurrentMap();
 this.directoryCleaner = directoryCleaner;
+this.conf = conf;
+  }
+
+  @VisibleForTesting
--- End diff --

we probably don't need 2 VisibleForTesting constructors, do we? If someone 
wants to provide an explicit Executor, they can damn well create their own 
TransportConf. This is only done in a single unit test anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3172#issuecomment-62291803
  
  [Test build #23111 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23111/consoleFull)
 for   PR 3172 at commit 
[`04cddc8`](https://github.com/apache/spark/commit/04cddc82343a698b4b128782c2982d646afd0672).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[aarondav]

Github user aarondav commented on a diff in the pull request:

https://github.com/apache/spark/pull/3172#discussion_r20056709
  
--- Diff: 
network/common/src/main/java/org/apache/spark/network/buffer/FileSegmentManagedBuffer.java
 ---
@@ -31,27 +31,27 @@
 
 import org.apache.spark.network.util.JavaUtils;
 import org.apache.spark.network.util.LimitedInputStream;
+import org.apache.spark.network.util.SystemPropertyConfigProvider;
+import org.apache.spark.network.util.TransportConf;
 
 /**
  * A {@link ManagedBuffer} backed by a segment in a file.
  */
 public final class FileSegmentManagedBuffer extends ManagedBuffer {
-
-  /**
-   * Memory mapping is expensive and can destabilize the JVM (SPARK-1145, 
SPARK-3889).
-   * Avoid unless there's a good reason not to.
-   */
-  // TODO: Make this configurable
-  private static final long MIN_MEMORY_MAP_BYTES = 2 * 1024 * 1024;
-
   private final File file;
   private final long offset;
   private final long length;
+  private final TransportConf conf;
 
-  public FileSegmentManagedBuffer(File file, long offset, long length) {
+  public FileSegmentManagedBuffer(File file, long offset, long length, 
TransportConf conf) {
 this.file = file;
 this.offset = offset;
 this.length = length;
+this.conf = conf;
+  }
+
+  public FileSegmentManagedBuffer(File file, long offset, long length) {
--- End diff --

Please add a comment that the other constructor is preferred (this creates 
an extra object, for instance, and gets configuration from a random place).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[aarondav]

Github user aarondav commented on a diff in the pull request:

https://github.com/apache/spark/pull/3172#discussion_r20056703
  
--- Diff: 
core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala ---
@@ -22,6 +22,9 @@ import java.nio.ByteBuffer
 import java.util.concurrent.ConcurrentLinkedQueue
 import java.util.concurrent.atomic.AtomicInteger
 
+import org.apache.spark.network.netty.SparkTransportConf
--- End diff --

nit: import order


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4307] Initialize FileDescriptor lazily ...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/3172

[SPARK-4307] Initialize FileDescriptor lazily in FileRegion.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark lazyFD

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3172


commit 04cddc82343a698b4b128782c2982d646afd0672
Author: Reynold Xin 
Date:   2014-11-09T05:04:10Z

[SPARK-4307] Initialize FileDescriptor lazily in FileRegion.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Minor fix for TreeNode

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3171#issuecomment-62291100
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23110/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Minor fix for TreeNode

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3171#issuecomment-62291098
  
  [Test build #23110 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23110/consoleFull)
 for   PR 3171 at commit 
[`c2b09e3`](https://github.com/apache/spark/commit/c2b09e3febd1752098837f8760c42df9a6b54d54).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Minor fix for TreeNode

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3171#issuecomment-62290939
  
  [Test build #23110 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23110/consoleFull)
 for   PR 3171 at commit 
[`c2b09e3`](https://github.com/apache/spark/commit/c2b09e3febd1752098837f8760c42df9a6b54d54).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Minor fix for TreeNode

[scwf]

GitHub user scwf opened a pull request:

https://github.com/apache/spark/pull/3171

[SQL] Minor fix for TreeNode

1 remove `case` never reached
2 add transformUp for test("transform works on nodes with Option children") 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scwf/spark treenode

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3171.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3171


commit c2b09e3febd1752098837f8760c42df9a6b54d54
Author: wangfei 
Date:   2014-11-09T04:19:39Z

minor fix for TreeNode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62289795
  
  [Test build #23109 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23109/consoleFull)
 for   PR 3159 at commit 
[`0b9f0f4`](https://github.com/apache/spark/commit/0b9f0f4a0d87a5bb3b2fe39bcb1fc568c552f6e7).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Scala 2.11 Support in Spark

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/2986#issuecomment-62289760
  
@JoshRosen yes I will add docs - this was just running some tests through 
jenkins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Scala 2.11 Support in Spark

[pwendell]

Github user pwendell closed the pull request at:

https://github.com/apache/spark/pull/2986


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62289725
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23108/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Support cross building for Scala 2.11

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3159#issuecomment-62289504
  
  [Test build #23108 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23108/consoleFull)
 for   PR 3159 at commit 
[`125ba46`](https://github.com/apache/spark/commit/125ba4644af90c23040453069e0bac1a77e9dba8).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3648: Provide a script for fetching remo...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/3165#issuecomment-62288954
  
I'm actually happy to just drop this though if we can update the 
documentation in our wiki to suggest people use hub. @JoshRosen or @rxin would 
one of you guys be able to put a few lines in 
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools with 
the process you use? I can then verify and if it's all good I can just drip 
this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3648: Provide a script for fetching remo...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/3165#issuecomment-62288892
  
I didn't use hub - but I preferred this since I can just create a temporary 
ref that gets thrown away and I never have to worry about cleaning it up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1

[adampingel]

Github user adampingel commented on the pull request:

https://github.com/apache/spark/pull/2947#issuecomment-62288693
  
Algebird 0.8.1 for Scala 2.11 is on the central repo: 
http://search.maven.org/#artifactdetails%7Ccom.twitter%7Calgebird_2.11%7C0.8.1%7Cjar


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4301] StreamingContext should not allow...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/3160


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4301] StreamingContext should not allow...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/3160#issuecomment-62287744
  
merging this. thanks josh for this PR!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3179. Add task OutputMetrics.

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2968#issuecomment-62285457
  
  [Test build #23107 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23107/consoleFull)
 for   PR 2968 at commit 
[`dce4784`](https://github.com/apache/spark/commit/dce4784fd428bd3b0df55080a8a94b60a153b094).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class OutputMetrics(writeMethod: DataWriteMethod.Value) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3179. Add task OutputMetrics.

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2968#issuecomment-62285463
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23107/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3179. Add task OutputMetrics.

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2968#issuecomment-62282883
  
  [Test build #23107 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23107/consoleFull)
 for   PR 2968 at commit 
[`dce4784`](https://github.com/apache/spark/commit/dce4784fd428bd3b0df55080a8a94b60a153b094).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3179. Add task OutputMetrics.

[sryza]

Github user sryza commented on a diff in the pull request:

https://github.com/apache/spark/pull/2968#discussion_r20055135
  
--- Diff: 
core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala ---
@@ -73,4 +78,32 @@ class InputMetricsSuite extends FunSuite with 
SharedSparkContext {
 assert(taskBytesRead.length == 2)
 assert(taskBytesRead.sum >= file.length())
   }
+
+  test("output metrics when writing text file") {
+val fs = FileSystem.getLocal(new Configuration())
+val outPath = new Path(fs.getWorkingDirectory, "outdir")
+
+if (SparkHadoopUtil.get.getFSBytesWrittenOnThreadCallback(outPath, 
fs.getConf).isDefined) {
--- End diff --

Yeah, I had this same thought, but I can't think of any good way to prevent 
this.  Note that the other tests in this class have the same problem. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62281021
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23106/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62281019
  
  [Test build #23106 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23106/consoleFull)
 for   PR 3158 at commit 
[`97d00a8`](https://github.com/apache/spark/commit/97d00a89dc5e804eafd9e10c722e8394f0c39a26).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class LhsLiteral(x: Any) `
  * `final class MutableDate extends MutableValue `
  * `final class MutableTimestamp extends MutableValue `
  * `class RichDate(milliseconds: Long) extends Date(milliseconds) `
  * `class RichTimestamp(milliseconds: Long) extends 
Timestamp(milliseconds) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3648: Provide a script for fetching remo...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/3165#issuecomment-62279935
  
Just realized Josh also pointed this out earlier. I've been using hub and 
it's pretty good. It does require a local branch, but that's pretty easy to do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3648: Provide a script for fetching remo...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/3165#issuecomment-62279046
  
Why not using hub? It is super easy and we don't need to duplicate the 
work. 

https://github.com/github/hub


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62278867
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23105/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62278862
  
  [Test build #23105 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23105/consoleFull)
 for   PR 3158 at commit 
[`e1ba56e`](https://github.com/apache/spark/commit/e1ba56e5bc516a5b1b4a94b6c3134e14c5547095).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class LhsLiteral(x: Any) `
  * `final class MutableDate extends MutableValue `
  * `final class MutableTimestamp extends MutableValue `
  * `class RichDate(milliseconds: Long) extends Date(milliseconds) `
  * `class RichTimestamp(milliseconds: Long) extends 
Timestamp(milliseconds) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] [Core] Don't NPE on closeQuietly(null)

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/3166


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] [Core] Don't NPE on closeQuietly(null)

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/3166#issuecomment-62278736
  
Merged in master & branch-1.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62278701
  
  [Test build #23106 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23106/consoleFull)
 for   PR 3158 at commit 
[`97d00a8`](https://github.com/apache/spark/commit/97d00a89dc5e804eafd9e10c722e8394f0c39a26).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62278623
  
  [Test build #23105 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23105/consoleFull)
 for   PR 3158 at commit 
[`e1ba56e`](https://github.com/apache/spark/commit/e1ba56e5bc516a5b1b4a94b6c3134e14c5547095).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62276463
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23104/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62276461
  
  [Test build #23104 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23104/consoleFull)
 for   PR 3158 at commit 
[`b61540e`](https://github.com/apache/spark/commit/b61540ea1febf9c8425767c0f82177ec84051487).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class LhsLiteral(x: Any) `
  * `final class MutableDate extends MutableValue `
  * `final class MutableTimestamp extends MutableValue `
  * `class RichDate(milliseconds: Long) extends Date(milliseconds) `
  * `class RichTimestamp(milliseconds: Long) extends 
Timestamp(milliseconds) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date with comp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3158#issuecomment-62275092
  
  [Test build #23104 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23104/consoleFull)
 for   PR 3158 at commit 
[`b61540e`](https://github.com/apache/spark/commit/b61540ea1febf9c8425767c0f82177ec84051487).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4261][BUILD][SQL] Make right version in...

[scwf]

Github user scwf commented on the pull request:

https://github.com/apache/spark/pull/3124#issuecomment-62269619
  
cc @pwendell @marmbrus 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4274] [SQL] Print informative message w...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/3139#discussion_r20053092
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
 ---
@@ -341,12 +341,21 @@ abstract class HiveComparisonTest
   val query = new TestHive.HiveQLQueryExecution(queryString)
   try { (query, prepareAnswer(query, query.stringResult())) } 
catch {
 case e: Throwable =>
+  val logicalQueryInString = try {
+query.toString
--- End diff --

I don't think thats correct, stringOrError takes a closure so the actual 
calculation is done inside of the function and should be correctly caught:

```scala
scala> def stringOrError[A](f: => A): String =
 |   try f.toString catch { case e: Throwable => e.toString }
stringOrError: [A](f: => A)String

scala> stringOrError(sys.error("test"))
res0: String = java.lang.RuntimeException: test

scala> def error = sys.error("test")
error: Nothing

scala> stringOrError(error)
res1: String = java.lang.RuntimeException: test
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4001][MLlib] adding apriori algorithm f...

[varadharajan]

Github user varadharajan commented on the pull request:

https://github.com/apache/spark/pull/2847#issuecomment-62263230
  
As mentioned in comments of SPARK-2432. I was wondering how the PFP 
versions compared with YAFIM 
(http://pasa-bigdata.nju.edu.cn/people/ronggu/pub/YAFIM_ParLearning.pdf). 
Probably i will do a bit more reading on this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3131#issuecomment-62262792
  
  [Test build #23103 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23103/consoleFull)
 for   PR 3131 at commit 
[`8a56fd6`](https://github.com/apache/spark/commit/8a56fd602dc3e6b0c91f428e40b2580387987280).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3131#issuecomment-62262797
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23103/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2894#issuecomment-62262246
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23102/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2894#issuecomment-62262242
  
  [Test build #23102 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23102/consoleFull)
 for   PR 2894 at commit 
[`5f9406b`](https://github.com/apache/spark/commit/5f9406b858ae1c80f228bd77d22bb1eebc6b1200).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4250] [SQL] Fix bug of constant null va...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3114#issuecomment-62261556
  
  [Test build #23101 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23101/consoleFull)
 for   PR 3114 at commit 
[`e603bda`](https://github.com/apache/spark/commit/e603bdaa46e89f3cc67ebf593ccdeb8a2fde8647).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4250] [SQL] Fix bug of constant null va...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3114#issuecomment-62261557
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23101/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP] [SPARK-4273] [SQL] Providing ExternalSet...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/3137#issuecomment-62260611
  
Probably a better way is providing the Sort-Merge-Aggregation not just for 
`DistinctCount`. I am working on a POC, hope we can discuss that soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[adrian-wang]

Github user adrian-wang commented on a diff in the pull request:

https://github.com/apache/spark/pull/3131#discussion_r20051975
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: 
HiveContext) extends Catalog with
   tableName: String,
   alias: Option[String]): LogicalPlan = synchronized {
 val (databaseName, tblName) = processDatabaseAndTableName(
-
db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
 val table = client.getTable(databaseName, tblName)
-val partitions: Seq[Partition] =
-  if (table.isPartitioned) {
-HiveShim.getAllPartitionsOf(client, table).toSeq
-  } else {
-Nil
-  }
+if (table.isView) {
+  // if the unresolved relation is from hive view
+  // parse the text into logic node.
+  HiveQl.createPlanForView(table, alias)
--- End diff --

I don't think we have to lowercase names. In resolving reference we will 
decide whether we should ignore case, and will get all names resolved 
accordingly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3131#issuecomment-62260293
  
  [Test build #23103 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23103/consoleFull)
 for   PR 3131 at commit 
[`8a56fd6`](https://github.com/apache/spark/commit/8a56fd602dc3e6b0c91f428e40b2580387987280).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[adrian-wang]

Github user adrian-wang commented on a diff in the pull request:

https://github.com/apache/spark/pull/3131#discussion_r20051957
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: 
HiveContext) extends Catalog with
   tableName: String,
   alias: Option[String]): LogicalPlan = synchronized {
 val (databaseName, tblName) = processDatabaseAndTableName(
-
db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
--- End diff --

Sure, that's much clearer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[adrian-wang]

Github user adrian-wang commented on a diff in the pull request:

https://github.com/apache/spark/pull/3131#discussion_r20051950
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -428,7 +432,7 @@ private[hive] case class MetastoreRelation
 }
   )
 
-  val tableDesc = HiveShim.getTableDesc(
+  lazy val tableDesc = HiveShim.getTableDesc(
--- End diff --

I tried to change this first to locate where we call `tableDesc`, it is not 
a necessary change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4250] [SQL] Fix bug of constant null va...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/3114#issuecomment-62260018
  
Thank you @marmbrus to review this. 
All of primitive constant null value actually will not find a match, and 
will be considered as normal ObjectInspector, this won't bring the incorrect 
result. But the `ConstantObjectInspector` is definitely a way of optimization, 
at least we can avoid `wrap`ing the Literal arguments for UDFs. I have plan to 
do this with follow ups.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4012] call tryOrExit instead of logUnca...

[CodingCat]

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/2864#issuecomment-62259510
  
Hi, @andrewor14 , Sorry about the misunderstanding and thanks for your 
patient explanation 

I will resubmit it and hopefully will get more feedbacks about this, thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...

[varadharajan]

Github user varadharajan commented on the pull request:

https://github.com/apache/spark/pull/2894#issuecomment-62259405
  
Also i think it would help users if we can document in the LR section of 
the MLlib guide, which algorithm should be preferred in which scenarios.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4250] [SQL] Fix bug of constant null va...

[chenghao-intel]

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/3114#discussion_r20051871
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ---
@@ -254,46 +255,59 @@ private[hive] trait HiveInspectors {
   }
 
   def toInspector(expr: Expression): ObjectInspector = expr match {
-case Literal(value: String, StringType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Int, IntegerType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Double, DoubleType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Boolean, BooleanType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Long, LongType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Float, FloatType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Short, ShortType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Byte, ByteType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Array[Byte], BinaryType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: java.sql.Date, DateType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: java.sql.Timestamp, TimestampType) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: BigDecimal, DecimalType()) =>
-  HiveShim.getPrimitiveWritableConstantObjectInspector(value)
-case Literal(value: Decimal, DecimalType()) =>
-  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.toBigDecimal)
+case Literal(value, StringType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[String])
+case Literal(value, IntegerType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Int])
+case Literal(value, DoubleType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Double])
+case Literal(value, BooleanType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Boolean])
+case Literal(value, LongType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Long])
+case Literal(value, FloatType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Float])
+case Literal(value, ShortType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Short])
+case Literal(value, ByteType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Byte])
+case Literal(value, BinaryType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[Array[Byte]])
+case Literal(value, DateType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[java.sql.Date])
+case Literal(value, TimestampType) =>
+  
HiveShim.getPrimitiveWritableConstantObjectInspector(value.asInstanceOf[java.sql.Timestamp])
+case Literal(value, DecimalType()) =>
+  if (null == value) {
+HiveShim.getPrimitiveWritableConstantObjectInspector(
+  null.asInstanceOf[BigDecimal])
+  } else {
+HiveShim.getPrimitiveWritableConstantObjectInspector(
+  value.asInstanceOf[Decimal].toBigDecimal)
+  }
 case Literal(_, NullType) =>
   HiveShim.getPrimitiveNullWritableConstantObjectInspector
-case Literal(value: Seq[_], ArrayType(dt, _)) =>
--- End diff --

We don't bind a type here, just to keep consistent with pattern match 
above, and it is also helpful in debugging, you know, sometimes we just 
confused what type for catalyst array (`Array` or `Seq`), the `.asInstanceOf` 
will be a strong check. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2894#issuecomment-62259296
  
  [Test build #23102 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23102/consoleFull)
 for   PR 2894 at commit 
[`5f9406b`](https://github.com/apache/spark/commit/5f9406b858ae1c80f228bd77d22bb1eebc6b1200).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...

[varadharajan]

Github user varadharajan commented on the pull request:

https://github.com/apache/spark/pull/2894#issuecomment-62259256
  
@jkbradley Makes sense. I've updated the warnings, please let me know if 
wordings can be improved. Also i just noticed that pyspark classification model 
does not have LR-LBFGS implementation. I'll probably create a new issue and 
work on it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4250] [SQL] Fix bug of constant null va...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3114#issuecomment-62259067
  
  [Test build #23101 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23101/consoleFull)
 for   PR 3114 at commit 
[`e603bda`](https://github.com/apache/spark/commit/e603bdaa46e89f3cc67ebf593ccdeb8a2fde8647).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl

[adrian-wang]

Github user adrian-wang commented on a diff in the pull request:

https://github.com/apache/spark/pull/3131#discussion_r20051793
  
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -21,6 +21,7 @@ import java.sql.Date
 import org.apache.hadoop.hive.conf.HiveConf
 import org.apache.hadoop.hive.ql.Context
 import org.apache.hadoop.hive.ql.lib.Node
+import org.apache.hadoop.hive.ql.metadata.Table
--- End diff --

line 263, declare parameter type of `createPlanForView()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4274] [SQL] Print informative message w...

[chenghao-intel]

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/3139#discussion_r20051763
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
 ---
@@ -341,12 +341,21 @@ abstract class HiveComparisonTest
   val query = new TestHive.HiveQLQueryExecution(queryString)
   try { (query, prepareAnswer(query, query.stringResult())) } 
catch {
 case e: Throwable =>
+  val logicalQueryInString = try {
+query.toString
--- End diff --

See
```
protected abstract class QueryExecution {
...
override def toString: String =
  s"""== Parsed Logical Plan ==
 |${stringOrError(logical)}
 |== Analyzed Logical Plan ==
 |${stringOrError(analyzed)}
 |== Optimized Logical Plan ==
 |${stringOrError(optimizedPlan)}
 |== Physical Plan ==
 |${stringOrError(executedPlan)}
 |Code Generation: ${executedPlan.codegenEnabled}
 |== RDD ==
  """.stripMargin.trim
  }
```
`logical`, `analyzed` etc. will be computed(which probably cause exception) 
before passing into `stringOrError`, not happen inside of `stringOrError`.

Or you think we'd better to fix the `toString` here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...

[culler]

Github user culler commented on the pull request:

https://github.com/apache/spark/pull/3066#issuecomment-62257326
  
Thanks!  I'm proud to be a member of the club. :^)

On Fri, Nov 7, 2014 at 11:16 PM, Cheng Lian 
wrote:

> @culler  Don't worry, I guess every
> contributor had screwed up rebasing at least once :)
>
> —
> Reply to this email directly or view it on GitHub
> .
>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-948 [WEB UI] Move "Classpath Entries" in...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3170#issuecomment-62257227
  
  [Test build #23100 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23100/consoleFull)
 for   PR 3170 at commit 
[`865fb5a`](https://github.com/apache/spark/commit/865fb5a23cc27c1dd3031a9f970605560ad6c35a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-948 [WEB UI] Move "Classpath Entries" in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3170#issuecomment-62257228
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23100/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-4156 [MLLIB] EM algorithm for GMMs

[tgaloppo]

Github user tgaloppo commented on the pull request:

https://github.com/apache/spark/pull/3022#issuecomment-62256712
  
This test appeared to fail due to some form of timeout during the pull; is 
there any action I need to take?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-971 [DOCS] Link to Confluence wiki from ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3169#issuecomment-62255849
  
  [Test build #23099 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23099/consoleFull)
 for   PR 3169 at commit 
[`dcb84d0`](https://github.com/apache/spark/commit/dcb84d070713136495e094cda7f0d7b01b3e9b10).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-971 [DOCS] Link to Confluence wiki from ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3169#issuecomment-62255850
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23099/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-1344 [DOCS] Scala API docs for top metho...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3168#issuecomment-62255695
  
  [Test build #23098 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23098/consoleFull)
 for   PR 3168 at commit 
[`6963fcc`](https://github.com/apache/spark/commit/6963fccee667fe45a476c878d1433b7d8e835c5e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-1344 [DOCS] Scala API docs for top metho...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3168#issuecomment-62255698
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23098/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

[witgo]

Github user witgo commented on the pull request:

https://github.com/apache/spark/pull/1290#issuecomment-62254979
  
I agree with what @debasish83 said. We should find a suitable solution to 
weight matrix distributed storage. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-948 [WEB UI] Move "Classpath Entries" in...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3170#issuecomment-62254881
  
  [Test build #23100 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23100/consoleFull)
 for   PR 3170 at commit 
[`865fb5a`](https://github.com/apache/spark/commit/865fb5a23cc27c1dd3031a9f970605560ad6c35a).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-948 [WEB UI] Move "Classpath Entries" in...

[srowen]

GitHub user srowen opened a pull request:

https://github.com/apache/spark/pull/3170

SPARK-948 [WEB UI] Move "Classpath Entries" in WebUI

This is a trivial change that resolves SPARK-948. Feel free to reject it 
too if there's not agreement on it, and then the JIRA can be closed. Just 
proposing it to resolve the question.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srowen/spark SPARK-948

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3170


commit 865fb5a23cc27c1dd3031a9f970605560ad6c35a
Author: Sean Owen 
Date:   2014-11-08T11:34:04Z

Move up Classpath Entries in Environment tab




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Update RecoverableNetworkWordCount.scala

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2735#issuecomment-62254523
  
  [Test build #23097 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23097/consoleFull)
 for   PR 2735 at commit 
[`b6d8001`](https://github.com/apache/spark/commit/b6d800148cb80c2bd2c43dca5b78af962a89fcb5).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Update RecoverableNetworkWordCount.scala

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2735#issuecomment-62254525
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23097/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-4305 [BUILD] yarn-alpha profile won't bu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3167#issuecomment-62253946
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23094/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-4305 [BUILD] yarn-alpha profile won't bu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3167#issuecomment-62253944
  
  [Test build #23094 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23094/consoleFull)
 for   PR 3167 at commit 
[`88938cb`](https://github.com/apache/spark/commit/88938cbdcbddf7aece088243cce8cfab0d6ca2f6).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4293][SQL] Make Cast be able to handle ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3150#issuecomment-62253732
  
  [Test build #23096 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23096/consoleFull)
 for   PR 3150 at commit 
[`8999868`](https://github.com/apache/spark/commit/89998684122af72482e8c1c2d22198dfc66aa4d4).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-971 [DOCS] Link to Confluence wiki from ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3169#issuecomment-62253730
  
  [Test build #23099 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23099/consoleFull)
 for   PR 3169 at commit 
[`dcb84d0`](https://github.com/apache/spark/commit/dcb84d070713136495e094cda7f0d7b01b3e9b10).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4293][SQL] Make Cast be able to handle ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3150#issuecomment-62253734
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23096/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org