[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11984#issuecomment-203768142
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203767874
  
**[Test build #54591 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54591/consoleFull)**
 for PR 12077 at commit 
[`cec6902`](https://github.com/apache/spark/commit/cec6902fc6bb5b501aa5b1c536b034508f7a5c75).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11984#issuecomment-203766948
  
**[Test build #54599 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54599/consoleFull)**
 for PR 11984 at commit 
[`16cf602`](https://github.com/apache/spark/commit/16cf602b506c37c9e5cd9cbb48d577b7e66f14d4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203764881
  
**[Test build #54598 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54598/consoleFull)**
 for PR 11987 at commit 
[`48aec92`](https://github.com/apache/spark/commit/48aec92480ec59ed4a965941d56126d9222cb853).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Fix Expand operator constra...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11995#discussion_r58003817
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -514,12 +524,13 @@ private[sql] object Expand {
  * @param projections to apply
  * @param output of all projections.
  * @param child operator.
+ * @param groupByAttrs the attributes used in group by.
  */
 case class Expand(
 projections: Seq[Seq[Expression]],
 output: Seq[Attribute],
-child: LogicalPlan) extends UnaryNode {
-
+child: LogicalPlan,
+groupByAttrs: Seq[Attribute]) extends UnaryNode {
--- End diff --

Ok. Let me remove the constraints first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-13784][ML][WIP] Model export/import for...

[GayathriMurali]

Github user GayathriMurali commented on the pull request:

https://github.com/apache/spark/pull/12023#issuecomment-203763228
  
@jkbradley I am sorry, I am afraid I will not be able to complete tonight. 
Can you please help me with reusing Splitdata/build code from DecisionTrees in 
RandomForests? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203761313
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54589/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203761312
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12077#discussion_r58003342
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
 ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.aggregate
+
+import org.apache.spark.sql.expressions.Aggregator
+

+
+// This file defines internal implementations for aggregators.

+
+
+
+class TypedSum[IN, OUT : Numeric](f: IN => OUT) extends Aggregator[IN, 
OUT, OUT] {
+  val numeric = implicitly[Numeric[OUT]]
+  override def zero: OUT = numeric.zero
+  override def reduce(b: OUT, a: IN): OUT = numeric.plus(b, f(a))
+  override def merge(b1: OUT, b2: OUT): OUT = numeric.plus(b1, b2)
+  override def finish(reduction: OUT): OUT = reduction
+}
+
+
+class TypedSumDouble[IN](f: IN => Double) extends Aggregator[IN, Double, 
Double] {
--- End diff --

in the future we should just turn these into aggregate expressions without 
aggregator


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203761179
  
**[Test build #54589 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54589/consoleFull)**
 for PR 12080 at commit 
[`7dd88a3`](https://github.com/apache/spark/commit/7dd88a369c375009715de4d0c604d956dab69002).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [DO_NOT_MERGE]Try to reproduce StateStoreRDDSu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12005#issuecomment-203757779
  
**[Test build #54597 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54597/consoleFull)**
 for PR 12005 at commit 
[`9a4d8b3`](https://github.com/apache/spark/commit/9a4d8b3f9a388505f156115919a13949e48d0bc6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12343][YARN] Simplify Yarn client and c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11603#issuecomment-203756317
  
**[Test build #54596 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54596/consoleFull)**
 for PR 11603 at commit 
[`3bb44b4`](https://github.com/apache/spark/commit/3bb44b4b1b84f9a972ad8ea4876b70369ba07d0c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203753133
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54584/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203753127
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Fix Expand operator constra...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/11995#discussion_r58001590
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -514,12 +524,13 @@ private[sql] object Expand {
  * @param projections to apply
  * @param output of all projections.
  * @param child operator.
+ * @param groupByAttrs the attributes used in group by.
  */
 case class Expand(
 projections: Seq[Seq[Expression]],
 output: Seq[Attribute],
-child: LogicalPlan) extends UnaryNode {
-
+child: LogicalPlan,
+groupByAttrs: Seq[Attribute]) extends UnaryNode {
--- End diff --

Yeah, I do think it would be good if it just took a 
`Seq[Seq[NamedExpression]]` (or at least I can't come up with anything better). 
 I'd still consider breaking this into two PRs.  Simple fix for now that just 
removes invalid constraints and a refactoring that add back in valid ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203752798
  
**[Test build #54595 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54595/consoleFull)**
 for PR 12060 at commit 
[`f1407c0`](https://github.com/apache/spark/commit/f1407c0bb302355f7f06aad9ece00541063bde6e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203752828
  
**[Test build #54584 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54584/consoleFull)**
 for PR 12057 at commit 
[`dd71ba9`](https://github.com/apache/spark/commit/dd71ba90cd41722c8fd3811d2c698da344aeb690).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[ueshin]

Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203752562
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11987#discussion_r58001426
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -74,12 +75,16 @@ statement
 SET SKEWED LOCATION skewedLocationList 
#setTableSkewLocations
 | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
 partitionSpecLocation+ 
#addTablePartition
+| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
+partitionSpec+ 
#addViewPartition
--- End diff --

@viirya Yeah, I am doing it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11987#discussion_r58001378
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -74,12 +75,16 @@ statement
 SET SKEWED LOCATION skewedLocationList 
#setTableSkewLocations
 | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
 partitionSpecLocation+ 
#addTablePartition
+| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
+partitionSpec+ 
#addViewPartition
--- End diff --

re-use label?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11987#discussion_r58001379
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -74,12 +75,16 @@ statement
 SET SKEWED LOCATION skewedLocationList 
#setTableSkewLocations
 | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
 partitionSpecLocation+ 
#addTablePartition
+| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
+partitionSpec+ 
#addViewPartition
--- End diff --

@hvanhovell Still trying to combine `addViewPartition` and 
`addTablePartition`. Will keep you posted. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203751599
  
**[Test build #54594 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54594/consoleFull)**
 for PR 11987 at commit 
[`dd34529`](https://github.com/apache/spark/commit/dd34529ea4f15b627cdf4dd921bf5a726d543b6d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14270][SQL] whole stage codegen support...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12061#issuecomment-203751600
  
**[Test build #54593 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54593/consoleFull)**
 for PR 12061 at commit 
[`aa95fd6`](https://github.com/apache/spark/commit/aa95fd6fd0a6146cd9f41cd59dbfb7f58b6c17da).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Fix Expand operator constra...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11995#discussion_r58000861
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -514,12 +524,13 @@ private[sql] object Expand {
  * @param projections to apply
  * @param output of all projections.
  * @param child operator.
+ * @param groupByAttrs the attributes used in group by.
  */
 case class Expand(
 projections: Seq[Seq[Expression]],
 output: Seq[Attribute],
-child: LogicalPlan) extends UnaryNode {
-
+child: LogicalPlan,
+groupByAttrs: Seq[Attribute]) extends UnaryNode {
--- End diff --

Thanks for pointing the problem. Agreed after re-thinking about it. As 
separating `projections` and `output` causes the problem. How about we get the 
output from `projections`?

As there are more than one projection, we can just get the output from the 
first projection and verify its consistency with other projections.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203750876
  
**[Test build #54592 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54592/consoleFull)**
 for PR 11987 at commit 
[`38ea348`](https://github.com/apache/spark/commit/38ea348cab7ffcf9a107bb56fb905bab7a6f2d00).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203750523
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54582/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203750521
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203750120
  
**[Test build #54582 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54582/consoleFull)**
 for PR 12060 at commit 
[`f1407c0`](https://github.com/apache/spark/commit/f1407c0bb302355f7f06aad9ece00541063bde6e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203749671
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54580/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203749668
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14133][SQL] Throws exception for unsupp...

[sureshthalamati]

Github user sureshthalamati commented on the pull request:

https://github.com/apache/spark/pull/12069#issuecomment-203749355
  
Thanks for the input, Herman.  Addressed the review comments , and fixed 
the test failure in my latest commit. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203749214
  
**[Test build #54580 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54580/consoleFull)**
 for PR 12038 at commit 
[`80f7573`](https://github.com/apache/spark/commit/80f75737d6dd25fa859b06e876f16f21d5b6b247).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14277] UnsafeSorterSpillReader should d...

[xerial]

Github user xerial commented on the pull request:

https://github.com/apache/spark/pull/12074#issuecomment-203748806
  
Released snappy-java-1.1.2.4 with this fix. Thanks for letting me know. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14133][SQL] Throws exception for unsupp...

[sureshthalamati]

Github user sureshthalamati commented on a diff in the pull request:

https://github.com/apache/spark/pull/12069#discussion_r58000282
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -167,6 +167,11 @@ unsupportedHiveNativeCommands
 | kw1=SHOW kw2=TRANSACTIONS
 | kw1=SHOW kw2=INDEXES
 | kw1=SHOW kw2=LOCKS
+| kw1=CREATE kw2=INDEX
+| kw1=DROP kw2=INDEX
+| kw1=LOCK kw2=TABLE
--- End diff --

Add lock/unlock database also.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203748254
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54581/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203748302
  
**[Test build #54591 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54591/consoleFull)**
 for PR 12077 at commit 
[`cec6902`](https://github.com/apache/spark/commit/cec6902fc6bb5b501aa5b1c536b034508f7a5c75).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203748250
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203748158
  
**[Test build #54581 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54581/consoleFull)**
 for PR 12057 at commit 
[`8dc1adf`](https://github.com/apache/spark/commit/8dc1adfb12a35280a01b4c8ab95b5aed346d8f0f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14133][SQL] Throws exception for unsupp...

[sureshthalamati]

Github user sureshthalamati commented on a diff in the pull request:

https://github.com/apache/spark/pull/12069#discussion_r5740
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -861,6 +866,7 @@ ROLES: 'ROLES';
 COMPACTIONS: 'COMPACTIONS';
 PRINCIPALS: 'PRINCIPALS';
 TRANSACTIONS: 'TRANSACTIONS';
+INDEX: 'INDEX';
--- End diff --

Thank you for reviewing the PR, Herman.  Added it to the non-Reserved list. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747707
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747708
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54590/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747701
  
**[Test build #54590 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54590/consoleFull)**
 for PR 12077 at commit 
[`6fb03e5`](https://github.com/apache/spark/commit/6fb03e5163637e612bc6366583915ec612e7f0a0).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747344
  
**[Test build #54590 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54590/consoleFull)**
 for PR 12077 at commit 
[`6fb03e5`](https://github.com/apache/spark/commit/6fb03e5163637e612bc6366583915ec612e7f0a0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203746656
  
**[Test build #54589 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54589/consoleFull)**
 for PR 12080 at commit 
[`7dd88a3`](https://github.com/apache/spark/commit/7dd88a369c375009715de4d0c604d956dab69002).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203744494
  
**[Test build #54587 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54587/consoleFull)**
 for PR 12052 at commit 
[`4dc9c79`](https://github.com/apache/spark/commit/4dc9c79a2d1d101226b415689663f0572adccb2d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203744481
  
**[Test build #54588 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54588/consoleFull)**
 for PR 11888 at commit 
[`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203744495
  
**[Test build #54586 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54586/consoleFull)**
 for PR 12078 at commit 
[`1b04630`](https://github.com/apache/spark/commit/1b046304313c7663015667ab9cc8fe4201d17eb2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203744234
  
add to whitelist @zsxwing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203744275
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54579/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203744274
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203744186
  
**[Test build #54579 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54579/consoleFull)**
 for PR 12057 at commit 
[`87f4bb4`](https://github.com/apache/spark/commit/87f4bb41096bbfbb8de49dbe4898e688ca3ccd6c).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203744206
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203744084
  
**[Test build #54585 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54585/consoleFull)**
 for PR 12080 at commit 
[`7459a3c`](https://github.com/apache/spark/commit/7459a3c7293e2659aaf87485d4a937bae9fdd384).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203744088
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54585/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203744087
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203744003
  
LGTM retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203743978
  
**[Test build #54585 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54585/consoleFull)**
 for PR 12080 at commit 
[`7459a3c`](https://github.com/apache/spark/commit/7459a3c7293e2659aaf87485d4a937bae9fdd384).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[brkyvz]

GitHub user brkyvz opened a pull request:

https://github.com/apache/spark/pull/12080

[SPARK-14287] isStreaming method for Dataset

With the addition of StreamExecution (ContinuousQuery) to Datasets, data 
will become unbounded. With unbounded data, the execution of some methods and 
operations will not make sense, e.g. `Dataset.count()`.

A simple API is required to check whether the data in a Dataset is bounded 
or unbounded. This will allow users to check whether their Dataset is in 
streaming mode or not. ML algorithms may check if the data is unbounded and 
throw an exception for example.

The implementation of this method is simple, however naming it is the 
challenge. Some possible names for this method are:
 - isStreaming
 - isContinuous
 - isBounded
 - isUnbounded

I've gone with `isStreaming` for now. We can change it before Spark 2.0 if 
we decide to come up with a different name. For that reason I've marked it as 
`@Experimental`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brkyvz/spark is-streaming

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12080


commit 7459a3c7293e2659aaf87485d4a937bae9fdd384
Author: Burak Yavuz 
Date:   2016-03-31T03:56:29Z

added isStreaming method to Dataset




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12079#issuecomment-203741023
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

[yongtang]

GitHub user yongtang opened a pull request:

https://github.com/apache/spark/pull/12079

[SPARK-14238][ML][MLLIB][PYSPARK] Add binary toggle Param to PySpark 
HashingTF in ML & MLlib

## What changes were proposed in this pull request?

This fix tries to add binary toggle Param to PySpark HashingTF in ML & 
MLlib. If this toggle is set, then all non-zero counts will be set to 1.

## How was this patch tested?

This fix adds two tests to cover the code changes. One for HashingTF in 
PySpark's ML and one for HashingTF in PySpark's MLLib.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yongtang/spark SPARK-14238

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12079


commit e58d1a279aaded9045c9e7a7a161500163b81fd6
Author: Yong Tang 
Date:   2016-03-31T03:49:33Z

[SPARK-14238][ML][MLLIB][PYSPARK] Add binary toggle Param to PySpark 
HashingTF in ML & MLlib

This fix tries to add binary toggle Param to PySpark HashingTF in ML & 
MLlib.
If this toggle is set, then all non-zero counts will be set to 1.

This fix adds two tests to cover the code changes. One for HashingTF in 
PySpark's ML
and one for HashingTF in PySpark's MLLib.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14098][SQL][WIP] Generate Java code tha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11956#issuecomment-203739715
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14098][SQL][WIP] Generate Java code tha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11956#issuecomment-203739717
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54583/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14098][SQL][WIP] Generate Java code tha...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11956#issuecomment-203739675
  
**[Test build #54583 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54583/consoleFull)**
 for PR 11956 at commit 
[`eb6088b`](https://github.com/apache/spark/commit/eb6088bc7192548896e81e68617333c5547ff55f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203739189
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54577/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203739188
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203739078
  
**[Test build #54577 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54577/consoleFull)**
 for PR 11888 at commit 
[`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-20373
  
**[Test build #54584 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54584/consoleFull)**
 for PR 12057 at commit 
[`dd71ba9`](https://github.com/apache/spark/commit/dd71ba90cd41722c8fd3811d2c698da344aeb690).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203734295
  
@JoshRosen I had addressed you comments also pushed some changes to improve 
the performance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SCHEDULER] Make DAGScheduler.get...

[ueshin]

Github user ueshin closed the pull request at:

https://github.com/apache/spark/pull/11720


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14269][SCHEDULER] Eliminate unnecessary...

[ueshin]

Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203734186
  
@maropu I see. I'll close #11720.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14277] UnsafeSorterSpillReader should d...

[xerial]

Github user xerial commented on the pull request:

https://github.com/apache/spark/pull/12074#issuecomment-203734165
  
@sitalkedia Sure. I'll do that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SCHEDULER] Make DAGScheduler.get...

[ueshin]

Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/11720#issuecomment-203734197
  
I'm going to close this in favor of #12060.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14098][SQL][WIP] Generate Java code tha...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11956#issuecomment-203732220
  
**[Test build #54583 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54583/consoleFull)**
 for PR 11956 at commit 
[`eb6088b`](https://github.com/apache/spark/commit/eb6088bc7192548896e81e68617333c5547ff55f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/12076#issuecomment-203731423
  
Huh, weird. It looks like this somehow _is_ a legitimate failure. I'll 
investigate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11984#discussion_r57995703
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala
 ---
@@ -88,16 +92,21 @@ object GenerateColumnAccessor extends 
CodeGenerator[Seq[DataType], ColumnarItera
 case array: ArrayType => classOf[ArrayColumnAccessor].getName
 case t: MapType => classOf[MapColumnAccessor].getName
   }
-  ctx.addMutableState(accessorCls, accessorName, s"$accessorName = 
null;")
-
-  val createCode = dt match {
-case t if ctx.isPrimitiveType(dt) =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case NullType | StringType | BinaryType =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case other =>
-  s"""$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder),
- (${dt.getClass.getName}) columnTypes[$index]);"""
+  ctx.addMutableState(accessorCls, accessorName, "")
+
+  val createCode = {
+val shortCls = accessorCls.substring(accessorCls.lastIndexOf(".") 
+ 1)
+dt match {
+  case t if ctx.isPrimitiveType(dt) =>
+s"$accessorName = 
get${accessorClasses.getOrElseUpdate(accessorCls, shortCls)}($index);"
--- End diff --

When you directly call `ColumnAccessor.apply`, I think we don't need 
`getXXXColumnAccessor` anymore?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14269][SCHEDULER] Eliminate unnecessary...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203727839
  
**[Test build #54582 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54582/consoleFull)**
 for PR 12060 at commit 
[`f1407c0`](https://github.com/apache/spark/commit/f1407c0bb302355f7f06aad9ece00541063bde6e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14277] UnsafeSorterSpillReader should d...

[sitalkedia]

Github user sitalkedia commented on the pull request:

https://github.com/apache/spark/pull/12074#issuecomment-203727586
  
@xerial - I am seeing similar issue for snappy write as well. Can we fix 
the write code path as well? 

Stack trace - 


org.xerial.snappy.SnappyNative.arrayCopy(Native Method)
org.xerial.snappy.Snappy.arrayCopy(Snappy.java:85)
org.xerial.snappy.SnappyOutputStream.rawWrite(SnappyOutputStream.java:273)
org.xerial.snappy.SnappyOutputStream.write(SnappyOutputStream.java:115)

org.apache.spark.io.SnappyOutputStreamWrapper.write(CompressionCodec.scala:202)

org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:220)

org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.write(UnsafeSorterSpillWriter.java:126)

org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:192)

org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:175)

org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:249)
org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:83)

org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.growPointerArrayIfNecessary(UnsafeExternalSorter.java:298)

org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:338)

org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:93)

org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:179)
org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90)
org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)

org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)

org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
org.apache.spark.scheduler.Task.run(Task.scala:89)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14269][SCHEDULER] Eliminate unnecessary...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12060#discussion_r57995458
  
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
---
@@ -1247,7 +1252,7 @@ class DAGScheduler(
 }
   }
 
-  // Note: newly runnable stages will be submitted below when 
we submit waiting stages
+  submitWaitingChildStages(shuffleStage)
--- End diff --

@markhamstra Thank you for your review.
Definitely we can move this into else branch.
I'll modify it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/11984#discussion_r57995435
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala
 ---
@@ -88,16 +92,21 @@ object GenerateColumnAccessor extends 
CodeGenerator[Seq[DataType], ColumnarItera
 case array: ArrayType => classOf[ArrayColumnAccessor].getName
 case t: MapType => classOf[MapColumnAccessor].getName
   }
-  ctx.addMutableState(accessorCls, accessorName, s"$accessorName = 
null;")
-
-  val createCode = dt match {
-case t if ctx.isPrimitiveType(dt) =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case NullType | StringType | BinaryType =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case other =>
-  s"""$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder),
- (${dt.getClass.getName}) columnTypes[$index]);"""
+  ctx.addMutableState(accessorCls, accessorName, "")
+
+  val createCode = {
+val shortCls = accessorCls.substring(accessorCls.lastIndexOf(".") 
+ 1)
+dt match {
+  case t if ctx.isPrimitiveType(dt) =>
+s"$accessorName = 
get${accessorClasses.getOrElseUpdate(accessorCls, shortCls)}($index);"
--- End diff --

I understand your motivation. I will revert my changes to avoid these 
complicity for reducing bytecode size.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203727221
  
**[Test build #54581 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54581/consoleFull)**
 for PR 12057 at commit 
[`8dc1adf`](https://github.com/apache/spark/commit/8dc1adfb12a35280a01b4c8ab95b5aed346d8f0f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12857][STREAMING] Standardize "records"...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/12032#issuecomment-203726152
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203726024
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/11984#discussion_r57994632
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala
 ---
@@ -88,16 +92,21 @@ object GenerateColumnAccessor extends 
CodeGenerator[Seq[DataType], ColumnarItera
 case array: ArrayType => classOf[ArrayColumnAccessor].getName
 case t: MapType => classOf[MapColumnAccessor].getName
   }
-  ctx.addMutableState(accessorCls, accessorName, s"$accessorName = 
null;")
-
-  val createCode = dt match {
-case t if ctx.isPrimitiveType(dt) =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case NullType | StringType | BinaryType =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case other =>
-  s"""$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder),
- (${dt.getClass.getName}) columnTypes[$index]);"""
+  ctx.addMutableState(accessorCls, accessorName, "")
+
+  val createCode = {
+val shortCls = accessorCls.substring(accessorCls.lastIndexOf(".") 
+ 1)
+dt match {
+  case t if ctx.isPrimitiveType(dt) =>
+s"$accessorName = 
get${accessorClasses.getOrElseUpdate(accessorCls, shortCls)}($index);"
--- End diff --

It's better to call `ColumnAccessor.apply()` to avoid these complicity.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14160] Time Windowing functions for Dat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12008#issuecomment-203724873
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54575/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203724822
  
**[Test build #54580 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54580/consoleFull)**
 for PR 12038 at commit 
[`80f7573`](https://github.com/apache/spark/commit/80f75737d6dd25fa859b06e876f16f21d5b6b247).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[viper-kun]

GitHub user viper-kun opened a pull request:

https://github.com/apache/spark/pull/12078

[SPARK-13112]CoarsedExecutorBackend register to driver should wait Executor 
was ready

## What changes were proposed in this pull request?

When CoarseGrainedExecutorBackend receives RegisterExecutorResponse slow 
after LaunchTask, it will occurs the problem. 

## How was this patch tested?
Executor host IO Busy

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viper-kun/spark patch-3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12078


commit 1b046304313c7663015667ab9cc8fe4201d17eb2
Author: xukun 
Date:   2016-03-31T02:39:40Z

CoarsedExecutorBackend register to driver should wait Executor was ready




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14160] Time Windowing functions for Dat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12008#issuecomment-203724870
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14160] Time Windowing functions for Dat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12008#issuecomment-203724577
  
**[Test build #54575 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54575/consoleFull)**
 for PR 12008 at commit 
[`8bc9799`](https://github.com/apache/spark/commit/8bc9799db4e252f334e8239765732ded6dbe32c4).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[liyezhang556520]

Github user liyezhang556520 commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203724131
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203723752
  
**[Test build #54579 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54579/consoleFull)**
 for PR 12057 at commit 
[`87f4bb4`](https://github.com/apache/spark/commit/87f4bb41096bbfbb8de49dbe4898e688ca3ccd6c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [sql] Modify SQLConf to use new co...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11570#issuecomment-203723176
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [sql] Modify SQLConf to use new co...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11570#issuecomment-203723177
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54567/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203723039
  
**[Test build #54578 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54578/consoleFull)**
 for PR 12038 at commit 
[`80f7573`](https://github.com/apache/spark/commit/80f75737d6dd25fa859b06e876f16f21d5b6b247).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203723066
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54578/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/11984#discussion_r57994067
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala
 ---
@@ -88,16 +92,21 @@ object GenerateColumnAccessor extends 
CodeGenerator[Seq[DataType], ColumnarItera
 case array: ArrayType => classOf[ArrayColumnAccessor].getName
 case t: MapType => classOf[MapColumnAccessor].getName
   }
-  ctx.addMutableState(accessorCls, accessorName, s"$accessorName = 
null;")
-
-  val createCode = dt match {
-case t if ctx.isPrimitiveType(dt) =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case NullType | StringType | BinaryType =>
-  s"$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder));"
-case other =>
-  s"""$accessorName = new 
$accessorCls(ByteBuffer.wrap(buffers[$index]).order(nativeOrder),
- (${dt.getClass.getName}) columnTypes[$index]);"""
+  ctx.addMutableState(accessorCls, accessorName, "")
+
+  val createCode = {
+val shortCls = accessorCls.substring(accessorCls.lastIndexOf(".") 
+ 1)
+dt match {
+  case t if ctx.isPrimitiveType(dt) =>
+s"$accessorName = 
get${accessorClasses.getOrElseUpdate(accessorCls, shortCls)}($index);"
--- End diff --

For example, a generated method `getIntColumnAccessor()` still calls 
```ColumnAccessor.apply()``.

Do you want to directly call  ```ColumnAccessor.apply()`` from a method 
```hasNext()``` instead of calling it thru `getIntColumnAccessor`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203723065
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [sql] Modify SQLConf to use new co...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11570#issuecomment-203723004
  
**[Test build #54567 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54567/consoleFull)**
 for PR 11570 at commit 
[`4588c7e`](https://github.com/apache/spark/commit/4588c7e2003add6abe59832b6c2529144f1ccbf3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class MultilayerPerceptronClassificationModel(JavaModel, 
JavaMLWritable, JavaMLReadable):`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-13784][ML][WIP] Model export/import for...

[GayathriMurali]

Github user GayathriMurali commented on a diff in the pull request:

https://github.com/apache/spark/pull/12023#discussion_r57993953
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala 
---
@@ -199,21 +210,71 @@ final class RandomForestRegressionModel private[ml] (
   private[ml] def toOld: OldRandomForestModel = {
 new OldRandomForestModel(OldAlgo.Regression, _trees.map(_.toOld))
   }
+
+  @Since("2.0.0")
+  override def write: MLWriter =
+new RandomForestRegressionModel.RandomForestRegressionModelWriter(this)
+
+  @Since("2.0.0")
+  override def read: MLReader[RandomForestRegressionModel] =
+new RandomForestRegressionModel.RandomForestRegressionModelReader(this)
 }
 
-private[ml] object RandomForestRegressionModel {
-
-  /** (private[ml]) Convert a model from the old API */
-  def fromOld(
-  oldModel: OldRandomForestModel,
-  parent: RandomForestRegressor,
-  categoricalFeatures: Map[Int, Int],
-  numFeatures: Int = -1): RandomForestRegressionModel = {
-require(oldModel.algo == OldAlgo.Regression, "Cannot convert 
RandomForestModel" +
-  s" with algo=${oldModel.algo} (old API) to 
RandomForestRegressionModel (new API).")
-val newTrees = oldModel.trees.map { tree =>
-  // parent for each tree is null since there is no good way to set 
this.
-  DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures)
+@Since("2.0.0")
+object RandomForestRegressionModel extends 
MLReadable[RandomForestRegressionModel] {
+
+@Since("2.0.0")
+override def load(path: String): RandomForestRegressionModel = 
super.load(path)
+
+private[RandomForestRegressionModel]
+class RandomForestRegressionModelWriter(instance: 
RandomForestRegressionModel)
+  extends MLWriter {
+
+  override protected def saveImpl(path: String): Unit = {
+val extraMetadata: JObject = Map(
+"numFeatures" -> instance.numFeatures)
+DefaultParamsWriter.saveMetadata(instance, path, sc, 
Some(extraMetadata))
+for ( treeIndex <- 1 to instance.getNumTrees) {
--- End diff --

@jkbradley Should saveImpl and load methods in RandomForestClassifier and 
Regressor over ride this method? I assume loadImpl will also have same 
signature. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12076#issuecomment-203722283
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54571/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12076#issuecomment-203722281
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org