[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9059#discussion_r41837372
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -289,13 +290,15 @@ private[ui] class AllJobsPage(parent: JobsTab) 
extends WebUIPage("") {
   val summary: NodeSeq =
 
   
-{if (parent.sc.isDefined) {
-  // Total duration is not meaningful unless the UI is live
-  
-Total Uptime: 
-{UIUtils.formatDuration(System.currentTimeMillis() - 
startTime)}
-  
-}}
+
+  Total Uptime:
+  {
+if (endTime < 0) {
+  val endTime = System.currentTimeMillis();
+}
+UIUtils.formatDuration(endTime - startTime)
--- End diff --

Right I missed that totally. You are declaring a local that shadows the var 
you mean to set. Remove val.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11072][SQL] simplify self join handling

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9081#issuecomment-147639157
  
  [Test build #43635 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43635/consoleFull)
 for   PR 9081 at commit 
[`c9645bf`](https://github.com/apache/spark/commit/c9645bf7f7d03f8cabb852d3d5e7d09074228b8b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10983] Unified memory manager

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9084#issuecomment-147642601
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43622/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147642587
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43633/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147642538
  
  [Test build #43633 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43633/console)
 for   PR 9089 at commit 
[`55bb135`](https://github.com/apache/spark/commit/55bb1354efcef98944caf96f8d59dc2f4a6459c0).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class KeyHint(newKeys: Seq[Key], child: LogicalPlan) extends 
UnaryNode `
  * `sealed abstract class Key `
  * `case class UniqueKey(attr: Attribute) extends Key `
  * `case class ForeignKey(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10983] Unified memory manager

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9084#issuecomment-147642518
  
  [Test build #43622 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43622/console)
 for   PR 9084 at commit 
[`24a391c`](https://github.com/apache/spark/commit/24a391cbba36812d555d32153daa5c132b3e608d).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10983] Unified memory manager

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9084#issuecomment-147642599
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147642585
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11000][YARN]Bug fix: Derby have booted ...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9026#issuecomment-147644840
  
Makes sense. Is it a problem that we actually have two metastores? Maybe 
not. That's my only question, looking at this from the outside.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10921] [YARN] Completely remove the use...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/8976#issuecomment-147645261
  
Yeah you'll still have to filter this spurious binary compatibility check 
warning:

```
[error]  * method preferredNodeLocationData_=(scala.collection.Map)Unit in 
class org.apache.spark.SparkContext does not have a correspondent in new version
[error]filter with: 
ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.SparkContext.preferredNodeLocationData_=")
[info] Done updating.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147624248
  
  [Test build #43621 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43621/console)
 for   PR 9050 at commit 
[`89c1401`](https://github.com/apache/spark/commit/89c140104ff931f4d5c154f754d9f43891cd600f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147624308
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43621/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10104][SQL] Consolidate different forms...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8453#issuecomment-147624407
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147624305
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10829][SQL]Filter combine partition key...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/8916#issuecomment-147625179
  
LGTM, cc @liancheng to take another look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11079] Review Netty-based RPC implement...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9091#discussion_r41833362
  
--- Diff: core/src/main/scala/org/apache/spark/rpc/RpcAddress.scala ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.rpc
+
+import org.apache.spark.util.Utils
+
+
+/**
+ * Address for an RPC environment, with hostname and port.
+ */
+private[spark] case class RpcAddress(host: String, port: Int) {
+
+  def hostPort: String = host + ":" + port
+
+  /** Returns a string in the form of "spark://host:port". */
+  def toSparkURL: String = "spark://" + hostPort
+
+  override def toString: String = hostPort
+}
+
+
+private[spark] object RpcAddress {
--- End diff --

i removed one method that was unused.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11079] Review Netty-based RPC implement...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9091#issuecomment-147626977
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147630724
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[ankurdave]

Github user ankurdave commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147636061
  
@rxin Thanks, I added the Experimental tags.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[ankurdave]

Github user ankurdave commented on a diff in the pull request:

https://github.com/apache/spark/pull/9089#discussion_r41836429
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -362,14 +362,35 @@ class Analyzer(
 j
   case Some((oldRelation, newRelation)) =>
 val attributeRewrites = 
AttributeMap(oldRelation.output.zip(newRelation.output))
-val newRight = right transformUp {
-  case r if r == oldRelation => newRelation
-} transformUp {
-  case other => other transformExpressions {
-case a: Attribute => attributeRewrites.get(a).getOrElse(a)
+def applyRewrites(plan: LogicalPlan): LogicalPlan =
+  plan transformUp {
+case r if r == oldRelation => newRelation
+  } transformUp {
+case other => other transformExpressions {
+  case a: Attribute => 
attributeRewrites.get(a).getOrElse(a)
+}
   }
-}
-j.copy(right = newRight)
+val newRight = applyRewrites(right)
+// Also apply the rewrites to foreign keys on the left side, 
because these are meant to
+// reference the right side.
+val newLeft =
+  if (left.keys.nonEmpty) {
+left.transform {
+  case KeyHint(keys, child) =>
+val newKeys = keys.collect {
+  case ForeignKey(attr, referencedRelation, 
referencedAttr) =>
+ForeignKey(
+  attr,
+  applyRewrites(referencedRelation),
+  
attributeRewrites.get(referencedAttr).getOrElse(referencedAttr))
+  case other => other
+}
+KeyHint((keys ++ newKeys).distinct, child)
--- End diff --

Good eye! This is to accommodate future self-joins. If we got rid of the 
old foreign keys, a future self-join would not recognize that the new keys 
applied to it, because the attributes would have been rewritten.  I just added 
a comment noting this.

There's [a unit 
test](https://github.com/apache/spark/pull/9089/files#diff-09ca3beb9c48d89b5fcf248e48d888ddR261)
 that covers this (fails if you remove the old keys).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147642051
  
  [Test build #43637 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43637/consoleFull)
 for   PR 9078 at commit 
[`e43c955`](https://github.com/apache/spark/commit/e43c955c43f197f82f883e2c460b58fc34125571).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147641313
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10104][SQL] Consolidate different forms...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8453#issuecomment-147651856
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43627/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10104][SQL] Consolidate different forms...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8453#issuecomment-147651853
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Core]Remove useless if branch

[zhichao-li]

GitHub user zhichao-li opened a pull request:

https://github.com/apache/spark/pull/9096

[Core]Remove useless if branch

We don't need the if checking here. it's redundant. The final result would 
always use `hstr`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhichao-li/spark uselessBranch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9096.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9096


commit 16b4614c49c8a2345b45fbdb7966a74a9a450d45
Author: zhichao.li 
Date:   2015-10-13T08:58:26Z

remove useless if branch




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10515] When killing executor, the pendi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8945#issuecomment-147626144
  
  [Test build #43628 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43628/consoleFull)
 for   PR 8945 at commit 
[`da13040`](https://github.com/apache/spark/commit/da13040aca20f0c739be8958675f5b39dac4d82c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147627183
  
We can tag them as Experimental (even though the entire DataFrame API is 
experimental!)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/9089#discussion_r41833942
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -362,14 +362,35 @@ class Analyzer(
 j
   case Some((oldRelation, newRelation)) =>
 val attributeRewrites = 
AttributeMap(oldRelation.output.zip(newRelation.output))
-val newRight = right transformUp {
-  case r if r == oldRelation => newRelation
-} transformUp {
-  case other => other transformExpressions {
-case a: Attribute => attributeRewrites.get(a).getOrElse(a)
+def applyRewrites(plan: LogicalPlan): LogicalPlan =
+  plan transformUp {
+case r if r == oldRelation => newRelation
+  } transformUp {
+case other => other transformExpressions {
+  case a: Attribute => 
attributeRewrites.get(a).getOrElse(a)
+}
   }
-}
-j.copy(right = newRight)
+val newRight = applyRewrites(right)
+// Also apply the rewrites to foreign keys on the left side, 
because these are meant to
+// reference the right side.
+val newLeft =
+  if (left.keys.nonEmpty) {
+left.transform {
+  case KeyHint(keys, child) =>
+val newKeys = keys.collect {
+  case ForeignKey(attr, referencedRelation, 
referencedAttr) =>
+ForeignKey(
+  attr,
+  applyRewrites(referencedRelation),
+  
attributeRewrites.get(referencedAttr).getOrElse(referencedAttr))
+  case other => other
+}
+KeyHint((keys ++ newKeys).distinct, child)
--- End diff --

Can't we just use `newKeys` here? Why do we need to keep old keys?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147637636
  
  [Test build #43633 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43633/consoleFull)
 for   PR 9089 at commit 
[`55bb135`](https://github.com/apache/spark/commit/55bb1354efcef98944caf96f8d59dc2f4a6459c0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11080] [SQL] Throw exception when Named...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9093#issuecomment-147641225
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43632/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11080] [SQL] Throw exception when Named...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9093#issuecomment-147641224
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11080] [SQL] Throw exception when Named...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9093#issuecomment-147641155
  
  [Test build #43632 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43632/console)
 for   PR 9093 at commit 
[`48e3d1c`](https://github.com/apache/spark/commit/48e3d1c8fe5e2a58114c225334eaa8affd2a2f45).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147640719
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147643826
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147643952
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11072][SQL] simplify self join handling

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9081#issuecomment-147647068
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43635/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11072][SQL] simplify self join handling

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9081#issuecomment-147647067
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9095#issuecomment-147649000
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[jerryshao]

GitHub user jerryshao opened a pull request:

https://github.com/apache/spark/pull/9095

[SPARK-11082][YARN] Fix wrong core number when response vcore is less than 
requested vcore

This should be guarded out and use response vcore number, this will be 
happened when use `DefaultResourceCalculator` in capacity scheduler by default.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/apache-spark SPARK-11082

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9095.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9095


commit 5fb7413b503a97141776a76413a8d7020f97e027
Author: jerryshao 
Date:   2015-10-13T08:06:20Z

fix wrong vcore number




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9095#issuecomment-147649014
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9095#issuecomment-147656903
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Core]Remove useless if branch

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9096#issuecomment-147656764
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9095#issuecomment-147656497
  
  [Test build #43639 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43639/console)
 for   PR 9095 at commit 
[`5fb7413`](https://github.com/apache/spark/commit/5fb7413b503a97141776a76413a8d7020f97e027).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Core]Remove useless if branch

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9096#issuecomment-147656720
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9095#issuecomment-147656906
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43639/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9776]Another instance of Derby may have...

[KaiXinXiaoLei]

Github user KaiXinXiaoLei closed the pull request at:

https://github.com/apache/spark/pull/8947


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10104][SQL] Consolidate different forms...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8453#issuecomment-147624706
  
  [Test build #43627 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43627/consoleFull)
 for   PR 8453 at commit 
[`2714fc0`](https://github.com/apache/spark/commit/2714fc05299562c5add6d87644e297f7a5551bf8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[ankurdave]

Github user ankurdave commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147625512
  
@marmbrus I addressed your comments from the review about a month ago:

1. Foreign key references now store the referenced relation directly as a 
logical plan rather than requiring a catalog lookup.
2. We now use `semanticEquals` and `AttributeSet` for attributes instead of 
normal equality.

There were a few comments that didn't make sense on second thought:

1. Move the attribute equivalence check in `ForeignKeyFinder` to a method 
on `LogicalPlan`. We thought this would simplify the logic, but it turned out 
not to (still need to maintain the disjoint-set data structure, and the logic 
gets split between `LogicalPlan` and `Project`).
2. Move foreign key attribute resolution to its own rule that runs at the 
end of analysis. This would work fine, but it seems to fit well within 
`ResolveReferences`.

Finally, the new DataFrame methods should probably be marked as alpha 
somehow, but I'm not sure of the best way. Maybe a new ScalaDoc group?

cc @rxin, @jkbradley



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147628251
  
  [Test build #43626 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43626/console)
 for   PR 9078 at commit 
[`e43c955`](https://github.com/apache/spark/commit/e43c955c43f197f82f883e2c460b58fc34125571).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `trait QueryExecutionListener `
  * `class ExecutionListenerManager extends Logging `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147628290
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43626/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147628289
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11069] [ML] Add RegexTokenizer option t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9092#issuecomment-147628224
  
  [Test build #43630 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43630/consoleFull)
 for   PR 9092 at commit 
[`ce09ef5`](https://github.com/apache/spark/commit/ce09ef532f2ec633e508840097fd0ac1b5285284).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11036][SQL] AttributeReference should n...

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/9094

[SPARK-11036][SQL] AttributeReference should not be assigned new expression 
id inside tasks

JIRA: https://issues.apache.org/jira/browse/SPARK-11036

Related to #9093, SPARK-11036 is proposed to not allow AttributeReference 
to be created in executors. #9093 solves part of SPARK-11036. But the method 
`withExprId` will possibly assign new expression id to AttributeReference too. 
This patch tries to prevent that by using the same mechanism of #9093.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 not-newexprid-on-executor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9094


commit 3df9212991659003b09c97f79aa73212bf411ecd
Author: Liang-Chi Hsieh 
Date:   2015-10-13T08:04:56Z

We should not assign new expression id to AttributeReference in tasks.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11069] [ML] Add RegexTokenizer option t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9092#issuecomment-147640440
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11069] [ML] Add RegexTokenizer option t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9092#issuecomment-147640443
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43630/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[ankurdave]

Github user ankurdave commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147642886
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11076][SQL]Add decimal support for floo...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/9086#issuecomment-147624010
  
cc @rxin @davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11069] [ML] Add RegexTokenizer option t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9092#issuecomment-147627765
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11069] [ML] Add RegexTokenizer option t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9092#issuecomment-147627734
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11017] [SQL] Support ImperativeAggregat...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/9038#issuecomment-147631386
  
See #9093 for an exploration of some sketchy-looking `newExprId` calls 
which might be involved in the problems that we're experiencing here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147639405
  
  [Test build #1887 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1887/consoleFull)
 for   PR 9050 at commit 
[`3aec389`](https://github.com/apache/spark/commit/3aec389d82363975fc4e8a17e8bf69474a70c988).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11069] [ML] Add RegexTokenizer option t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9092#issuecomment-147639962
  
  [Test build #43630 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43630/console)
 for   PR 9092 at commit 
[`ce09ef5`](https://github.com/apache/spark/commit/ce09ef532f2ec633e508840097fd0ac1b5285284).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147646844
  
  [Test build #43638 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43638/consoleFull)
 for   PR 9089 at commit 
[`55bb135`](https://github.com/apache/spark/commit/55bb1354efcef98944caf96f8d59dc2f4a6459c0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11072][SQL] simplify self join handling

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9081#issuecomment-147647003
  
  [Test build #43635 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43635/console)
 for   PR 9081 at commit 
[`c9645bf`](https://github.com/apache/spark/commit/c9645bf7f7d03f8cabb852d3d5e7d09074228b8b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...

[jbonofre]

Github user jbonofre commented on a diff in the pull request:

https://github.com/apache/spark/pull/9059#discussion_r41839587
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -289,13 +290,15 @@ private[ui] class AllJobsPage(parent: JobsTab) 
extends WebUIPage("") {
   val summary: NodeSeq =
 
   
-{if (parent.sc.isDefined) {
-  // Total duration is not meaningful unless the UI is live
-  
-Total Uptime: 
-{UIUtils.formatDuration(System.currentTimeMillis() - 
startTime)}
-  
-}}
+
+  Total Uptime:
+  {
+if (endTime < 0) {
+  val endTime = System.currentTimeMillis();
+}
+UIUtils.formatDuration(endTime - startTime)
--- End diff --

My bad, early wake up mistake ;)
Let me fix. Sorry about that guys.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147654556
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9090#issuecomment-147623285
  
  [Test build #43623 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43623/consoleFull)
 for   PR 9090 at commit 
[`697bd24`](https://github.com/apache/spark/commit/697bd241944c9769bf8cf74ca1ca581f1fc3dffa).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147622583
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147623169
  
  [Test build #43624 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43624/consoleFull)
 for   PR 9050 at commit 
[`3aec389`](https://github.com/apache/spark/commit/3aec389d82363975fc4e8a17e8bf69474a70c988).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147624909
  
  [Test build #43625 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43625/consoleFull)
 for   PR 9085 at commit 
[`b73b67e`](https://github.com/apache/spark/commit/b73b67ec2a9245b1e1e5b27bdd6019f4baf0e049).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10983] Unified memory manager

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/9084#discussion_r41833851
  
--- Diff: core/src/main/scala/org/apache/spark/memory/MemoryManager.scala 
---
@@ -72,46 +92,62 @@ private[spark] abstract class MemoryManager {
   def acquireUnrollMemory(
--- End diff --

actually, it's more than a synonym. In `StaticMemoryManager` it's required 
to preserve existing behavior where unrolling doesn't evict all the blocks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11080] Throw exception when NamedExpres...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9093#issuecomment-147631990
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11080] Throw exception when NamedExpres...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9093#issuecomment-147631968
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147634492
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147634471
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147634491
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11077] [SQL] Join elimination in Cataly...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9089#issuecomment-147634467
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11079] Post-hoc review Netty-based RPC ...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9091#issuecomment-147634406
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...

[jbonofre]

Github user jbonofre commented on a diff in the pull request:

https://github.com/apache/spark/pull/9059#discussion_r41837043
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -289,13 +290,15 @@ private[ui] class AllJobsPage(parent: JobsTab) 
extends WebUIPage("") {
   val summary: NodeSeq =
 
   
-{if (parent.sc.isDefined) {
-  // Total duration is not meaningful unless the UI is live
-  
-Total Uptime: 
-{UIUtils.formatDuration(System.currentTimeMillis() - 
startTime)}
-  
-}}
+
+  Total Uptime:
+  {
+if (endTime < 0) {
+  val endTime = System.currentTimeMillis();
+}
+UIUtils.formatDuration(endTime - startTime)
--- End diff --

Hi @sarutak 
endTime is already set earlier in the AllJobPage. So it should be ok, no ? 
I think I'm missing something.
Please, can you explain what you mean be "alive" ?
Thanks a lot guys!
Regards
JB


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11036][SQL] AttributeReference should n...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9094#issuecomment-147641277
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11068][SQL] add callback to query execu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9078#issuecomment-147641283
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11036][SQL] AttributeReference should n...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9094#issuecomment-147641314
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10104][SQL] Consolidate different forms...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8453#issuecomment-147651548
  
  [Test build #43627 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43627/console)
 for   PR 8453 at commit 
[`2714fc0`](https://github.com/apache/spark/commit/2714fc05299562c5add6d87644e297f7a5551bf8).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class SimpleCatalog(val conf: CatalystConf) extends Catalog `
  * `trait ExtendedCatalog extends SimpleCatalog `
  * `  case class QualifiedTableName(database: String, name: String)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11082][YARN] Fix wrong core number when...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9095#issuecomment-147651353
  
  [Test build #43639 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43639/consoleFull)
 for   PR 9095 at commit 
[`5fb7413`](https://github.com/apache/spark/commit/5fb7413b503a97141776a76413a8d7020f97e027).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[mengxr]

GitHub user mengxr opened a pull request:

https://github.com/apache/spark/pull/9090

[SPARK-7402][ML] JSON SerDe for standard param types

This PR implements the JSON SerDe for the following param types: `Boolean`, 
`Int`, `Long`, `Float`, `Double`, `String`, `Array[Int]`, `Array[Double]`, and 
`Array[String]`. The implementation of `Float`, `Double`, and `Array[Double]` 
are specialized to handle `NaN` and `Inf`s. This will be used in pipeline 
persistence. @jkbradley 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mengxr/spark SPARK-7402

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9090.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9090


commit 697bd241944c9769bf8cf74ca1ca581f1fc3dffa
Author: Xiangrui Meng 
Date:   2015-10-13T06:34:06Z

JSON SerDe for standard param types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147620479
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/9090#discussion_r41831432
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala ---
@@ -287,6 +422,16 @@ class StringArrayParam(parent: Params, name: String, 
doc: String, isValid: Array
 
   /** Creates a param pair with a [[java.util.List]] of values (for Java 
and Python). */
   def w(value: java.util.List[String]): ParamPair[Array[String]] = 
w(value.asScala.toArray)
+
+  override def jsonEncode(value: Array[String]): String = {
+import org.json4s.JsonDSL._
--- End diff --

This is not imported globally because it shadows `++`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11009] [SQL] fix wrong result of Window...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9050#issuecomment-147620505
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9090#issuecomment-147620477
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9090#issuecomment-147620504
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10829][SQL]Filter combine partition key...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/8916#issuecomment-147623900
  
Thank you @cloud-fan for the reviewing, updated, and passed the unit test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11079] Review Netty-based RPC implement...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9091#issuecomment-147626999
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9090#issuecomment-147628837
  
  [Test build #43623 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43623/console)
 for   PR 9090 at commit 
[`697bd24`](https://github.com/apache/spark/commit/697bd241944c9769bf8cf74ca1ca581f1fc3dffa).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9090#issuecomment-147628990
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43623/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7402][ML] JSON SerDe for standard param...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9090#issuecomment-147628985
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11080] Throw exception when NamedExpres...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/9093

[SPARK-11080] Throw exception when NamedExpression.newExprId is called 
inside tasks

My understanding of `NamedExpression.newExprId` is that it is only intended 
to be called on the driver. If it is called on executors, then this may lead to 
scenarios where the same expression id is re-used in two different 
NamedExpressions.

More generally, I think that calling `NamedExpression.newExprId` within 
tasks may be an indicator of potential attribute binding bugs. Therefore, I 
think that we should throw an exception when this method is called inside of 
tasks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark SPARK-11080

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9093.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9093


commit 48e3d1c8fe5e2a58114c225334eaa8affd2a2f45
Author: Josh Rosen 
Date:   2015-10-13T07:37:21Z

Throw exception when NamedExpression.newExprId is called from task.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147638012
  
  [Test build #43634 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43634/consoleFull)
 for   PR 9085 at commit 
[`c0e8cb5`](https://github.com/apache/spark/commit/c0e8cb5f7ca716ea49b8015f50a4573752895c26).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11036][SQL] AttributeReference should n...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9094#issuecomment-147642286
  
  [Test build #43636 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43636/consoleFull)
 for   PR 9094 at commit 
[`3df9212`](https://github.com/apache/spark/commit/3df9212991659003b09c97f79aa73212bf411ecd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147654388
  
  [Test build #43625 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43625/console)
 for   PR 9085 at commit 
[`b73b67e`](https://github.com/apache/spark/commit/b73b67ec2a9245b1e1e5b27bdd6019f4baf0e049).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  case class DecimalLit(chars: String) extends Token `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10533] [SQL] handle scientific notation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9085#issuecomment-147654560
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43625/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org