[GitHub] spark pull request: [SPARK-5733] Error Link in Pagination of Histr...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/4523#issuecomment-73844774
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5712] [SQL] fix comment with semicolon ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4500#issuecomment-73843947
  
  [Test build #27280 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27280/consoleFull)
 for   PR 4500 at commit 
[`90068a2`](https://github.com/apache/spark/commit/90068a2c09d55cf4c79c91d0861c556a0854ded2).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73843764
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27277/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73843759
  
  [Test build #27277 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27277/consoleFull)
 for   PR 4467 at commit 
[`355f9ce`](https://github.com/apache/spark/commit/355f9ce34cb0d01048b15c96d5028fb291604286).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class TaskCommitDenied(`
  * `class CommitDeniedException(`
  * `  class OutputCommitCoordinatorActor(outputCommitCoordinator: 
OutputCommitCoordinator)`
  * `case class StreamingListenerReceiverRegistered(receiverInfo: 
ReceiverInfo)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5712] [SQL] fix comment with semicolon ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4500#issuecomment-73843625
  
  [Test build #27279 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27279/consoleFull)
 for   PR 4500 at commit 
[`e6ebcf5`](https://github.com/apache/spark/commit/e6ebcf5f5a436397d2ec5e45ea93de50bdbbc203).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5454] More robust handling of self join...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4520#issuecomment-73843441
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27276/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5454] More robust handling of self join...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4520#issuecomment-73843431
  
  [Test build #27276 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27276/consoleFull)
 for   PR 4520 at commit 
[`49c8e26`](https://github.com/apache/spark/commit/49c8e26868ac2e7c8a1e935f5929924e0cc64a02).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5727 [BUILD] Deprecate Debian packaging

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/4516#discussion_r24478002
  
--- Diff: docs/building-spark.md ---
@@ -161,6 +161,8 @@ For help in setting up IntelliJ IDEA or Eclipse for 
Spark development, and troub
 
 # Building Spark Debian Packages
 
+_NOTE: Debian packaging is deprecated and will be removed in a future 
release._
--- End diff --

Yes, will do. That sounds like a good idea.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5727 [BUILD] Deprecate Debian packaging

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/4516#discussion_r24477671
  
--- Diff: docs/building-spark.md ---
@@ -161,6 +161,8 @@ For help in setting up IntelliJ IDEA or Eclipse for 
Spark development, and troub
 
 # Building Spark Debian Packages
 
+_NOTE: Debian packaging is deprecated and will be removed in a future 
release._
--- End diff --

Should we maybe just say "in Spark 1.4"? It might be good to set 
expectations conservatively... no harm if we decide to keep it longer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4524#issuecomment-73841997
  
  [Test build #27278 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27278/consoleFull)
 for   PR 4524 at commit 
[`237a256`](https://github.com/apache/spark/commit/237a256c9f037b7be70ebc5972ce62305323c67e).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/4524#issuecomment-73841717
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73840846
  
@dondrake This looks good to me, just one comment, thanks!

In Python, we infer the schema only based on the first few rows, so it's 
possible to get IntegerType for large Python int after this patch. In this 
case, user could provide a correct schema to workaround it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5522] Accelerate the Histroty Server st...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4525#issuecomment-73840730
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/4521#discussion_r24477026
  
--- Diff: python/pyspark/tests.py ---
@@ -923,6 +923,20 @@ def test_infer_schema(self):
 result = self.sqlCtx.sql("SELECT l[0].a from test2 where 
d['key'].d = '2'")
 self.assertEqual(1, result.first()[0])
 
+def test_infer_long_type(self):
+longrow = [Row(f1='a', f2=100)]
+lrdd = self.sc.parallelize(longrow)
+slrdd = self.sqlCtx.inferSchema(lrdd)
--- End diff --

inferSchema() is deprecated, use createDataFrame() please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5522] Accelerate the Histroty Server st...

[marsishandsome]

GitHub user marsishandsome opened a pull request:

https://github.com/apache/spark/pull/4525

[SPARK-5522] Accelerate the Histroty Server start

When starting the history server, all the log files will be fetched and 
parsed in order to get the applications' meta data e.g. App Name, Start Time, 
Duration, etc. In our production cluster, there exist 2600 log files (160G) in 
HDFS and it costs 3 hours to restart the history server, which is a little bit 
too long for us.

It would be better, if the history server can show logs with missing 
information during start-up and fill the missing information after fetching and 
parsing a log file.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/marsishandsome/spark Spark5522

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4525.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4525


commit be5670c937d163b3c8238248c80bcf472333678f
Author: guliangliang 
Date:   2015-02-11T06:45:01Z

[SPARK-5522] Accelerate the Histroty Server start




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73840357
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27275/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73840354
  
  [Test build #27275 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27275/consoleFull)
 for   PR 4521 at commit 
[`f53b94c`](https://github.com/apache/spark/commit/f53b94c6d6a30b3c3b4a27b5e51ca7670ccddb75).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5658][SQL] Finalize DDL and write suppo...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/4446#discussion_r24476792
  
--- Diff: 
sql/core/src/test/java/org/apache/spark/sql/sources/JavaSaveLoadSuite.java ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.sources;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.test.TestSQLContext$;
+import org.apache.spark.sql.*;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.util.Utils;
+
+public class JavaSaveLoadSuite {
+
+  private transient JavaSparkContext sc;
+  private transient SQLContext sqlContext;
+
+  String originalDefaultSource;
+  File path;
+  DataFrame df;
+
+  private void checkAnswer(DataFrame actual, List expected) {
+String errorMessage = QueryTest$.MODULE$.checkAnswer(actual, expected);
+if (errorMessage != null) {
+  Assert.fail(errorMessage);
+}
+  }
+
+  @Before
+  public void setUp() throws IOException {
+sqlContext = TestSQLContext$.MODULE$;
+sc = new JavaSparkContext(sqlContext.sparkContext());
+
+originalDefaultSource = sqlContext.conf().defaultDataSourceName();
+path =
+  Utils.createTempDir(System.getProperty("java.io.tmpdir"), 
"datasource").getCanonicalFile();
+if (path.exists()) {
+  path.delete();
+}
+
+List jsonObjects = new ArrayList(10);
+for (int i = 0; i < 10; i++) {
+  jsonObjects.add("{\"a\":" + i + ", \"b\":\"str" + i + "\"}");
+}
+JavaRDD rdd = sc.parallelize(jsonObjects);
+df = sqlContext.jsonRDD(rdd);
+df.registerTempTable("jsonTable");
+  }
+
+  @Test
+  public void saveAndLoad() {
+Map options = new HashMap();
+options.put("path", path.toString());
+df.save("org.apache.spark.sql.json", SaveMode.ErrorIfExists, options);
+
+DataFrame loadedDF = sqlContext.load("org.apache.spark.sql.json", 
options);
+
+checkAnswer(loadedDF, df.collectAsList());
+  }
+
+  @Test
+  public void saveAndLoadWithSchema() {
+Map options = new HashMap();
+options.put("path", path.toString());
+df.save("org.apache.spark.sql.json", SaveMode.ErrorIfExists, options);
+
+List fields = new ArrayList<>();
--- End diff --

We have fixed it. Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5732][CORE]:Add an option to print the ...

[uncleGen]

Github user uncleGen commented on the pull request:

https://github.com/apache/spark/pull/4522#issuecomment-73839883
  
An irrelevant test failure in `DirectKafkaStreamSuite` introduced by 
[PR](4384)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5684][SQL]: Pass in partition name alon...

[saucam]

Github user saucam commented on the pull request:

https://github.com/apache/spark/pull/4469#issuecomment-73839872
  
Hi @liancheng , thanks for the comments. We are using spark-1.2.1 and the 
old parquet support is being used. Can this be merged so that we have proper 
partitioning with different locations as well. I tried partitioning on 2 
columns and it worked fine (Also applied this patch for specifying a different 
location) 

On a different note, When I create a parquet table with smallint type in 
spark, the schema being used in parquet shows 'int32 type', is that by design 
in spark or its a parquet limitation ?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73838853
  
  [Test build #27277 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27277/consoleFull)
 for   PR 4467 at commit 
[`355f9ce`](https://github.com/apache/spark/commit/355f9ce34cb0d01048b15c96d5028fb291604286).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5706] [SQL] Add json schema inferring A...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/4492#issuecomment-73838797
  
Thank you @cjnolet for letting me know this, I will review the code when 
it's ready.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5732][CORE]:Add an option to print the ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4522#issuecomment-73838664
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27274/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5732][CORE]:Add an option to print the ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4522#issuecomment-73838659
  
  [Test build #27274 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27274/consoleFull)
 for   PR 4522 at commit 
[`015ddee`](https://github.com/apache/spark/commit/015ddeed95df5dc35be4ed179c26186a9713a49b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5706] [SQL] Add json schema inferring A...

[cjnolet]

Github user cjnolet commented on the pull request:

https://github.com/apache/spark/pull/4492#issuecomment-73838447
  
I'm actively working on a PR for SPARK-5260. I have moved  a few of the 
utility functions into an object called JsonSchema. I'll post it soon. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

[guowei2]

Github user guowei2 commented on the pull request:

https://github.com/apache/spark/pull/3821#issuecomment-73838255
  
@rxin  yeah, still exist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5498][SQL]fix query exception when part...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/4289#issuecomment-73838104
  
@jeanlyn The HiveThriftServer unit test was disable previously before #4486 
merged. From the log it's hard to say the failure reason, can you try it in you 
local?
```
build/sbt -Phive-0.13.1 -Phive-thriftserver assembly
build/sbt -Phive-0.13.1 -Phive-thriftserver 'test-only 
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2Suite'
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

[tianyi]

Github user tianyi closed the pull request at:

https://github.com/apache/spark/pull/2542


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

[tianyi]

Github user tianyi commented on the pull request:

https://github.com/apache/spark/pull/2542#issuecomment-73837719
  
It's hard to rebase this branch due to so much changes in the latest master 
branch. 
So I had opened a new PR #4524 for this Issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4524#issuecomment-73837743
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/3821#issuecomment-73837755
  
@guowei2 does this problem still exist with the latest master branch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

[tianyi]

GitHub user tianyi opened a pull request:

https://github.com/apache/spark/pull/4524

[SPARK-3688][SQL]LogicalPlan can't resolve column correctlly

This PR fixed the resolving problem described in 
https://issues.apache.org/jira/browse/SPARK-3688
```
CREATE TABLE t1(x INT);
CREATE TABLE t2(a STRUCT, k INT);
SELECT a.x FROM t1 a JOIN t2 b ON a.x = b.k;
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tianyi/spark SPARK-3688

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4524


commit 09bc3f2c09ee93008c6dbfb0f38b9050dbec72e0
Author: tianyi 
Date:   2015-02-10T08:32:27Z

resolve an name with table.column pattern first.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5733] Error Link in Pagination of Histr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4523#issuecomment-73837491
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5498][SQL]fix query exception when part...

[jeanlyn]

Github user jeanlyn commented on the pull request:

https://github.com/apache/spark/pull/4289#issuecomment-73837462
  
Hi,@marmbrus , @chenghao-intel I have no idea why `SPARK-4407 regression: 
Complex type support` this test failed after i resolved the merge conflicts.It 
seems that not my problems,because i had passed this unit tests before.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5733] Error Link in Pagination of Histr...

[marsishandsome]

GitHub user marsishandsome opened a pull request:

https://github.com/apache/spark/pull/4523

[SPARK-5733] Error Link in Pagination of HistroyPage when showing 
Incomplete Applications

The links in pagination of HistroyPage is wrong when showing Incomplete 
Applications.

If "2" is click on the following page 
"http://history-server:18080/?page=1&showIncomplete=true";, it will go to 
"http://history-server:18080/?page=2"; instead of 
"http://history-server:18080/?page=2&showIncomplete=true";.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/marsishandsome/spark Spark5733

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4523.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4523


commit 9d7b5931f6f802892de6d86c6728c4d31edfe105
Author: guliangliang 
Date:   2015-02-11T05:39:07Z

[SPARK-5733] Error Link in Pagination of HistroyPage when showing 
Incomplete Applications




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5454] More robust handling of self join...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4520#issuecomment-73837250
  
  [Test build #27276 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27276/consoleFull)
 for   PR 4520 at commit 
[`49c8e26`](https://github.com/apache/spark/commit/49c8e26868ac2e7c8a1e935f5929924e0cc64a02).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5714][Mllib] Refactor initial step of L...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/4501#issuecomment-73837186
  
LGTM. Merged into master and branch-1.3. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5714][Mllib] Refactor initial step of L...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4501


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5498][SQL]fix query exception when part...

[jeanlyn]

Github user jeanlyn commented on the pull request:

https://github.com/apache/spark/pull/4289#issuecomment-73836885
  
Retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[scwf]

Github user scwf commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-73836812
  
Updated, @marmbrus any comment here? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73835887
  
Do you mind adding [SQL] to the pull request title, after [SPARK-5722]? 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73835941
  
  [Test build #27275 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27275/consoleFull)
 for   PR 4521 at commit 
[`f53b94c`](https://github.com/apache/spark/commit/f53b94c6d6a30b3c3b4a27b5e51ca7670ccddb75).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73835869
  
  [Test build #27273 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27273/consoleFull)
 for   PR 4467 at commit 
[`3d568e8`](https://github.com/apache/spark/commit/3d568e8ecac5ef53a96ae463e496cf51813eedfd).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class TaskCommitDenied(`
  * `class CommitDeniedException(`
  * `  class OutputCommitCoordinatorActor(outputCommitCoordinator: 
OutputCommitCoordinator)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73835891
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73835874
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27273/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5732][CORE]:Add an option to print the ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4522#issuecomment-73835711
  
  [Test build #27274 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27274/consoleFull)
 for   PR 4522 at commit 
[`015ddee`](https://github.com/apache/spark/commit/015ddeed95df5dc35be4ed179c26186a9713a49b).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-73835676
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27272/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-73835672
  
  [Test build #27272 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27272/consoleFull)
 for   PR 4425 at commit 
[`db86dc4`](https://github.com/apache/spark/commit/db86dc47c27c31532889ed14611c4edfdc26fd3b).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5732][CORE]:Add an option to print the ...

[uncleGen]

GitHub user uncleGen opened a pull request:

https://github.com/apache/spark/pull/4522

[SPARK-5732][CORE]:Add an option to print the spark version in spark script.

Naturally, we need to add an option to print the spark version in spark 
script. It is pretty common in script tool. There are some little changes in 
`verbose`. Its abbreviation, i.e. "-v", is removed and used by option 
"--version". Maybe, it will influence the established habits. Any ideas? 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uncleGen/spark master-clean-150211

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4522.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4522


commit 463f02c1f5e4c7b0a6f3cd12a817472c6b19bce0
Author: uncleGen 
Date:   2015-02-11T05:00:18Z

minor changes

commit 015ddeed95df5dc35be4ed179c26186a9713a49b
Author: uncleGen 
Date:   2015-02-11T05:01:50Z

minor changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[CodingCat]

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73835223
  
sure, thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73834940
  
I agree, the external actor system would need the same configuration, I
will let you take the call.

Prashant Sharma



On Wed, Feb 11, 2015 at 10:31 AM, Nan Zhu  wrote:

> @ScrapCodes  , I just realized that we may
> not be able to provide an empty class for failure detector
>
> because in some cases, the user needs to connect external akka system with
> spark's actor system, if an empty class does not send any heartbeat...the
> external akka system will always disassociate with it...
>
> if we send heartbeat in the empty class...how about the frequency?
>
> So...I think leaving the large interval for sending heartbeat might be
> better...at least it provides some interface for the external system to
> integrate...
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[CodingCat]

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73834654
  
@ScrapCodes , I just realized that we may not be able to provide an empty 
class for failure detector

because in some cases, the user needs to connect external akka system with 
spark's actor system, if an empty class does not send any heartbeat...the 
external akka system will always disassociate with it...

if we send heartbeat in the empty class...how about the frequency?

So...I think leaving the large interval for sending heartbeat might be 
better...at least it provides some interface for the external system to 
integrate...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4521#issuecomment-73834442
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5706] [SQL] Add json schema inferring A...

[chenghao-intel]

Github user chenghao-intel closed the pull request at:

https://github.com/apache/spark/pull/4492


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5722] fix for infer long type in python...

[dondrake]

GitHub user dondrake opened a pull request:

https://github.com/apache/spark/pull/4521

[SPARK-5722] fix for infer long type in python similar to Java long

Please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dondrake/spark branch-1.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4521


commit f53b94c6d6a30b3c3b4a27b5e51ca7670ccddb75
Author: Don Drake 
Date:   2015-02-11T04:45:51Z

fix for SPARK-5722 infer long type in python similar to Java long




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[CodingCat]

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73834047
  
no, the change is not that big at all, just adding a new class and one 
configuration line...

just asking to ensure that  the change will be along with the roadmap...

sure, I will handle it 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73833899
  
yes, I think in the meantime it is worth doing it - considering we would be 
saving on a useless thread. And it is not a lot of effort either (hopefully ?). 
What do you think ? is it a bigger undertaking ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/4512#discussion_r24474315
  
--- Diff: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala ---
@@ -106,7 +104,6 @@ private[spark] object AkkaUtils extends Logging {
   |akka.remote.secure-cookie = "$secureCookie"
   |akka.remote.transport-failure-detector.heartbeat-interval = 
$akkaHeartBeatInterval s
   |akka.remote.transport-failure-detector.acceptable-heartbeat-pause = 
$akkaHeartBeatPauses s
-  |akka.remote.transport-failure-detector.threshold = 
$akkaFailureDetector
--- End diff --

Just checked looks like the implementation class for the Failure Detector 
is changed, and so it does not need threshold property. The most idiomatic 
solution is like I have already said - provide an empty implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4964] [Streaming] refactor createRDD to...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4511#issuecomment-73833756
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27269/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4964] [Streaming] refactor createRDD to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4511#issuecomment-73833754
  
  [Test build #27269 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27269/consoleFull)
 for   PR 4511 at commit 
[`f7151d4`](https://github.com/apache/spark/commit/f7151d4e80c9bfe72d0240902d448df4412dc172).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[CodingCat]

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73833709
  
I see. you mean provide an empty class in our code base?

But, since they are discussing about removing Akka from Spark, is it still 
worth doing this (instead of just providing large values)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5641] Allow spark_ec2.py to copy a wide...

[florianverhein]

Github user florianverhein commented on the pull request:

https://github.com/apache/spark/pull/4487#issuecomment-73833605
  
... Actually that `extra` example should technically be able to be made to 
work without substitution as I source the `/root/extra/*.sh` scripts in 
`/root/spark-ec2/extra/*.sh`... so I think bash variables (e.g. `$MASTERS` 
rather than `{{master_list}}`) should therefore be fine in that case... ( ? )

But that doesn't affect the point re: separation of binary and 
configuration - just whether template substitution is strictly necessary. On 
the latter, I'm sure there's a case for having it (or at least no strong 
argument against allowing it)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5702][SQL] Allow short names for built-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4489


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4512#issuecomment-73833260
  
Ahh looks like the threshold property is gone, I think the most appropriate 
solution is to provide empty implementation under this property 
`implementation-class = "akka.remote.DeadlineFailureDetector"`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73833008
  
  [Test build #27273 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27273/consoleFull)
 for   PR 4467 at commit 
[`3d568e8`](https://github.com/apache/spark/commit/3d568e8ecac5ef53a96ae463e496cf51813eedfd).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5714][Mllib] Refactor initial step of L...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4501#issuecomment-73832862
  
  [Test build #27268 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27268/consoleFull)
 for   PR 4501 at commit 
[`4870fe4`](https://github.com/apache/spark/commit/4870fe4a0ebefad8074052857bbf9fee6a505464).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5714][Mllib] Refactor initial step of L...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4501#issuecomment-73832866
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27268/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5724] fix the misconfiguration in AkkaU...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/4512#discussion_r24473919
  
--- Diff: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala ---
@@ -106,7 +104,6 @@ private[spark] object AkkaUtils extends Logging {
   |akka.remote.secure-cookie = "$secureCookie"
   |akka.remote.transport-failure-detector.heartbeat-interval = 
$akkaHeartBeatInterval s
   |akka.remote.transport-failure-detector.acceptable-heartbeat-pause = 
$akkaHeartBeatPauses s
-  |akka.remote.transport-failure-detector.threshold = 
$akkaFailureDetector
--- End diff --

Failure detector threshold was set to a high value to disable it. Since 
there is not property in akka that let's us do that more idiomatically. By 
removing the above line we are simply asking akka to pick up default which is 
very low and will cause workers to disassociate during GC pauses.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4519#issuecomment-73832523
  
  [Test build #27263 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27263/consoleFull)
 for   PR 4519 at commit 
[`87ff763`](https://github.com/apache/spark/commit/87ff763847783edb98b65c3b7b9cc62a3ae1420b).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4519#issuecomment-73832527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27263/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5729] Potential NPE in standalone REST ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4518


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73832125
  

It should accept addblock even it is stopping because there might be 
receivers processing data.

Modified state behavior should be:

* If state = Initialized, then disallow all actions. (actor is not 
initialized yet)
* If state = Started, then allow all actions {register, addblock, report, 
deregister}.
* If state = Stopping, then disallow {register}, allow {addblock, report, 
deregister}.
* If state = Stopped, then disallow all actions. (actor is destroyed)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use driver to coordinate Hadoop o...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/4066#issuecomment-73831899
  
I only merged this into master and 1.3 for now because there are 
significant merge conflicts. Thanks @JoshRosen and @mccheah.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5706] [SQL] Add json schema inferring A...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/4492#discussion_r24473563
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -380,6 +380,11 @@ class SQLContext(@transient val sparkContext: 
SparkContext)
 jsonRDD(json.rdd, schema)
   }
 
+  @Experimental
+  def inferJsonSchema(json: String): StructType = {
--- End diff --

How about we close it for now? Also, feel free to leave comment to that 
JIRA. Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use driver to coordinate Hadoop o...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4066


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4155


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5706] [SQL] Add json schema inferring A...

[chenghao-intel]

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/4492#discussion_r24473491
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -380,6 +380,11 @@ class SQLContext(@transient val sparkContext: 
SparkContext)
 jsonRDD(json.rdd, schema)
   }
 
+  @Experimental
+  def inferJsonSchema(json: String): StructType = {
--- End diff --

Thanks for pointing this out! I agree we should provide a utility for the 
general purpose, let me know if I can offer help, or should I close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5681][Streaming] Add tracker status and...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/4467#issuecomment-73831487
  

The state behavior should be:

* If state = Initialized, then disallow all actions. (actor is not 
initialized yet)
* If state = Started, then allow all actions {register, addblock, report, 
deregister}.
* If state = Stopping, then disallow {register, addblock}, allow {report, 
deregister}.
* If state = Stopped, then disallow all actions. (actor is destroyed)

For the receivers that are still starting and have not registered yet, we 
have two options.

1. As you said, we have to wait for them to all be started.
2. We ignore their register messages and let them timeout. So they simply 
fail their starting process and don't process any data.

I think both options guarantee no data would be lost. I was thinking using 
option 2 in this pr. Because it  should be more simple and, semantically we 
should not allow receivers to register and then process data after stop is 
called.

I just realized that the current implementation of `ReceiverSupervisor` 
calls `receiver.onStart` before sending `RegisterReceiver` message. Thus it is 
possible the receiver are already started and process data. I think it is 
incorrect behavior. Correct one should be getting registered first and then 
begin to start the receiver. Otherwise, the receiver may process data and store 
data before it has registered successfully. It might cause some problems hard 
to detect. We should change the order so it register with tracer first then 
begin starting procedure. 

The important reason I think we don't choose option 1 to wait for receivers 
to all be started is, from the tracker's aspect, it has not idea what receivers 
are started or not. It just asynchronously waits for them to register and 
deregister. The receivers are visible to the tracker only when they are 
registered with it. When it is going to stop, because it doesn't know if there 
are receivers started but not registered yet, so it doesn't know how longer it 
should wait for them. Thus it is safer to make sure that the receivers must 
register before they start.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5021] [MLlib] Gaussian Mixture now supp...

[MechCoder]

Github user MechCoder commented on the pull request:

https://github.com/apache/spark/pull/4459#issuecomment-73831296
  
Thanks @tgaloppo and @mengxr . Any idea what to touch in GaussianMixture 
next? The parallelized Gaussian initialization.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-7383
  
  [Test build #27272 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27272/consoleFull)
 for   PR 4425 at commit 
[`db86dc4`](https://github.com/apache/spark/commit/db86dc47c27c31532889ed14611c4edfdc26fd3b).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4964] [Streaming] refactor createRDD to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4511#issuecomment-73831038
  
  [Test build #597 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/597/consoleFull)
 for   PR 4511 at commit 
[`6f8680b`](https://github.com/apache/spark/commit/6f8680bc910a197d0cb13d279aa62e5798ed7ac3).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use driver to coordinate Hadoop o...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4066#issuecomment-73830997
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27264/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4879] Use driver to coordinate Hadoop o...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4066#issuecomment-73830994
  
  [Test build #27264 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27264/consoleFull)
 for   PR 4066 at commit 
[`658116b`](https://github.com/apache/spark/commit/658116b57c1eeb785d9f3d4577b0a7631ffcfaf9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[scwf]

Github user scwf commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-73830984
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5454] More robust handling of self join...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4520#issuecomment-73830800
  
  [Test build #27271 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27271/consoleFull)
 for   PR 4520 at commit 
[`6fc38de`](https://github.com/apache/spark/commit/6fc38dec76044d97ed8621fabbac0f890a987b1c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5454] More robust handling of self join...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4520#issuecomment-73830805
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27271/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-73830680
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27266/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5649][SQL] added a rule to check dataty...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4425#issuecomment-73830672
  
  [Test build #27266 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27266/consoleFull)
 for   PR 4425 at commit 
[`db86dc4`](https://github.com/apache/spark/commit/db86dc47c27c31532889ed14611c4edfdc26fd3b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5498][SQL]fix query exception when part...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4289#issuecomment-73830591
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27267/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5498][SQL]fix query exception when part...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4289#issuecomment-73830587
  
  [Test build #27267 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27267/consoleFull)
 for   PR 4289 at commit 
[`535b0b6`](https://github.com/apache/spark/commit/535b0b6896bae2c0540ba29991a3f083545d52b2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4519


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5706] [SQL] Add json schema inferring A...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/4492#discussion_r24473025
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala 
---
@@ -127,7 +135,7 @@ private[sql] object JsonRDD extends Logging {
   StructType((topLevelFields ++ structFields).sortBy(_.name))
 }
 
-makeStruct(resolved.keySet.toSeq, Nil)
+nullTypeToStringType(makeStruct(resolved.keySet.toSeq, Nil))
--- End diff --

Oh, sorry, I was not clear. I meant merging two `StructType`s in future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/4519#issuecomment-73830345
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/4519#issuecomment-73830319
  
This passed catalyst/sql/hive tests.  The core tests are know flakey.  
Merging after talking with @pwendell so this can be included in the snapshot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5454] More robust handling of self join...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4520#issuecomment-73830143
  
  [Test build #27271 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27271/consoleFull)
 for   PR 4520 at commit 
[`6fc38de`](https://github.com/apache/spark/commit/6fc38dec76044d97ed8621fabbac0f890a987b1c).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5712] [SQL] fix comment with semicolon ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4500#issuecomment-73829962
  
  [Test build #27265 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27265/consoleFull)
 for   PR 4500 at commit 
[`b90a4f4`](https://github.com/apache/spark/commit/b90a4f4a0fe289c13ae5aeb22b6fcd594334fa9f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5712] [SQL] fix comment with semicolon ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4500#issuecomment-73829965
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27265/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4519#issuecomment-73829953
  
  [Test build #27261 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27261/consoleFull)
 for   PR 4519 at commit 
[`015c4fc`](https://github.com/apache/spark/commit/015c4fcb3e4250268cd0c1933d6c0c02a30ce0cc).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][DataFrame] Fix column computability bug.

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4519#issuecomment-73829959
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27261/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5729] Potential NPE in standalone REST ...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/4518#issuecomment-73829920
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org