[GitHub] spark issue #22583: [SPARK-10816][SS] SessionWindow support for Structure St...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22583
  
**[Test build #96753 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96753/testReport)**
 for PR 22583 at commit 
[`d65015e`](https://github.com/apache/spark/commit/d65015e970a126e6f128aab63a47f436f272028c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22583: [SPARK-10816][SS] SessionWindow support for Structure St...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22583
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22583: [SPARK-10816][SS] SessionWindow support for Structure St...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22583
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96753/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/22577
  
Can you create a jira ticket and put more description? This is a bug fix 
and strictly speaking it's nothing about scala 2.12. It's just the scala 2.12.7 
compiler helps you to find this bug.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/22574#discussion_r221277340
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
 ---
@@ -488,26 +494,27 @@ private[parquet] class ParquetFilters(
   .map(_(nameToParquetField(name).fieldName, value))
 
   case sources.And(lhs, rhs) =>
-// At here, it is not safe to just convert one side if we do not 
understand the
-// other side. Here is an example used to explain the reason.
-// Let's say we have NOT(a = 2 AND b in ('1')) and we do not 
understand how to
-// convert b in ('1'). If we only convert a = 2, we will end up 
with a filter
-// NOT(a = 2), which will generate wrong results.
-// Pushing one side of AND down is only safe to do at the top 
level.
-// You can see ParquetRelation's initializeLocalJobFunc method as 
an example.
--- End diff --

+1


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...

[bersprockets]

Github user bersprockets commented on the issue:

https://github.com/apache/spark/pull/22504
  
The Py4JJavaError StackOverflow happens pretty reliably. I am guessing its 
related to the change.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22577
  
I think we can consider this more of the same in SPARK-25398, and attach it 
to that JIRA. If we end up passing the current 2.4 RC I'll untangle it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21157: [SPARK-22674][PYTHON] Removed the namedtuple pickling pa...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21157
  
**[Test build #96760 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96760/testReport)**
 for PR 21157 at commit 
[`765362e`](https://github.com/apache/spark/commit/765362efb1f40fde3820704fee1628c90d6c0566).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21157: [SPARK-22674][PYTHON] Removed the namedtuple pickling pa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21157
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21157: [SPARK-22674][PYTHON] Removed the namedtuple pickling pa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21157
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96760/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...

[ankuriitg]

Github user ankuriitg commented on the issue:

https://github.com/apache/spark/pull/22504
  
Yes, I will try to reproduce it locally


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22582: [SPARK-25505][SQL][FOLLOWUP] Fix for attributes cosmetic...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22582
  
**[Test build #96752 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96752/testReport)**
 for PR 22582 at commit 
[`2ed74f3`](https://github.com/apache/spark/commit/2ed74f3752ff178723c260c4d3c8d7d37ae3c99a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22582: [SPARK-25505][SQL][FOLLOWUP] Fix for attributes cosmetic...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22582
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22582: [SPARK-25505][SQL][FOLLOWUP] Fix for attributes cosmetic...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22582
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96752/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21809
  
**[Test build #96762 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96762/testReport)**
 for PR 21809 at commit 
[`3b73840`](https://github.com/apache/spark/commit/3b7384011c0d4e41ecd5c4b797f08c1c62088fbd).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/22581
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21809: [SPARK-24851][UI] Map a Stage ID to it's Associat...

[pgandhi999]

Github user pgandhi999 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21809#discussion_r221295304
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -105,15 +105,30 @@ private[ui] class StagePage(parent: StagesTab, store: 
AppStatusStore) extends We
 val stageAttemptId = parameterAttempt.toInt
 
 val stageHeader = s"Details for Stage $stageId (Attempt 
$stageAttemptId)"
-val stageData = parent.store
-  .asOption(parent.store.stageAttempt(stageId, stageAttemptId, details 
= false))
-  .getOrElse {
-val content =
-  
-No information to display for Stage {stageId} (Attempt 
{stageAttemptId})
-  
-return UIUtils.headerSparkPage(request, stageHeader, content, 
parent)
-  }
+var stageDataTuple: Option[Tuple2[StageData, Seq[Int]]] = try {
--- End diff --

Makes sense, simplified the code and tested it. Looks good. Thank you.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21809: [SPARK-24851][UI] Map a Stage ID to it's Associat...

[pgandhi999]

Github user pgandhi999 commented on a diff in the pull request:

https://github.com/apache/spark/pull/21809#discussion_r221295745
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -182,6 +197,15 @@ private[ui] class StagePage(parent: StagesTab, store: 
AppStatusStore) extends We
   {Utils.bytesToString(stageData.diskBytesSpilled)}
 
   }}
+  {if (!stageJobIds.isEmpty) {
--- End diff --

Since, we are returning from the code if it is None, it should never reach 
there.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22581
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3572/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22581
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22579: [SPARK-25429][SQL] Use Set instead of Array to improve l...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22579
  
**[Test build #96764 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96764/testReport)**
 for PR 22579 at commit 
[`9b3b164`](https://github.com/apache/spark/commit/9b3b164af3e8e43caec7e1cedfcaf3a8b1c6d8f2).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check

[maryannxue]

Github user maryannxue commented on the issue:

https://github.com/apache/spark/pull/22060
  
Sorry for the late reply. The purpose of this is to find out the rules that 
violate the once-policy assumption and also tests that can reproduce the 
issues. I think we should eventually turn this check on after we've fixed all 
those rules and extend this check to optimizer too.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22581
  
**[Test build #96763 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96763/testReport)**
 for PR 22581 at commit 
[`adda54f`](https://github.com/apache/spark/commit/adda54f76033a6eae160b3cfb80a67c22dc2a190).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22579: [SPARK-25429][SQL] Use Set instead of Array to improve l...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22579
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3573/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22579: [SPARK-25429][SQL] Use Set instead of Array to improve l...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22579
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22379
  
**[Test build #96751 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96751/testReport)**
 for PR 22379 at commit 
[`b36d96a`](https://github.com/apache/spark/commit/b36d96af3853701690297e5bd89a37b7209a446e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22579: [SPARK-25429][SQL] Use Set instead of Array to im...

[wangyum]

Github user wangyum commented on a diff in the pull request:

https://github.com/apache/spark/pull/22579#discussion_r221296477
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
 ---
@@ -83,7 +83,7 @@ class SQLAppStatusListener(
 // track of the metrics knows which accumulators to look at.
 val accumIds = exec.metrics.map(_.accumulatorId).sorted.toList
--- End diff --

Thanks @HeartSaVioR @kiszk


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96751/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22574#discussion_r221304857
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
 ---
@@ -488,26 +494,27 @@ private[parquet] class ParquetFilters(
   .map(_(nameToParquetField(name).fieldName, value))
 
   case sources.And(lhs, rhs) =>
-// At here, it is not safe to just convert one side if we do not 
understand the
-// other side. Here is an example used to explain the reason.
-// Let's say we have NOT(a = 2 AND b in ('1')) and we do not 
understand how to
-// convert b in ('1'). If we only convert a = 2, we will end up 
with a filter
-// NOT(a = 2), which will generate wrong results.
-// Pushing one side of AND down is only safe to do at the top 
level.
-// You can see ParquetRelation's initializeLocalJobFunc method as 
an example.
--- End diff --

+1


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22572: [SPARK-25521][SQL]Job id showing null in the logs...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/22572#discussion_r221305617
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
 ---
@@ -183,15 +183,16 @@ object FileFormatWriter extends Logging {
   val commitMsgs = ret.map(_.commitMsg)
 
   committer.commitJob(job, commitMsgs)
-  logInfo(s"Job ${job.getJobID} committed.")
+  logInfo(s"Job ${job.getConfiguration.get("mapreduce.job.id")} 
committed.")
--- End diff --

SparkHadoopWriter needs a similar change, then, BTW


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22466: [SPARK-25464][SQL]On dropping the Database it wil...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/22466#discussion_r221307039
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -207,6 +207,16 @@ class SessionCatalog(
   "you cannot create a database with this name.")
 }
 validateName(dbName)
+// SPARK-25464 fail if DB location exists and is not empty
+val dbPath = new Path(dbDefinition.locationUri)
+val fs = dbPath.getFileSystem(hadoopConf)
+if (fs.exists(dbPath)) {
+  val fileStatus = fs.listStatus(dbPath)
+  if (0 != fileStatus.length) {
--- End diff --

`if (fileStatus.nonEmpty)`? at least, flip the comparison. You can add the 
location to the exception message, maybe.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18906
  
**[Test build #96765 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96765/testReport)**
 for PR 18906 at commit 
[`fac6f1e`](https://github.com/apache/spark/commit/fac6f1e729a152540b4a408f9efef96ba9df965d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

[dhruve]

Github user dhruve commented on the issue:

https://github.com/apache/spark/pull/22288
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22576
  
**[Test build #96757 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96757/testReport)**
 for PR 22576 at commit 
[`450dd73`](https://github.com/apache/spark/commit/450dd7311a323e61190e991bf15d253f8bc676c7).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22288
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22288
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3574/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

[dhruve]

Github user dhruve commented on the issue:

https://github.com/apache/spark/pull/22288
  
the failures seem to be unrelated. I wasn't able to reproduce them.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22576
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22576
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96757/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22288
  
**[Test build #96767 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96767/testReport)**
 for PR 22288 at commit 
[`4c88168`](https://github.com/apache/spark/commit/4c881680fdde32244030b54b44125ac217dacb0d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22466
  
**[Test build #96766 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96766/testReport)**
 for PR 22466 at commit 
[`86e4d50`](https://github.com/apache/spark/commit/86e4d50d27ecec914ef534e84dcc75965456a32b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22466
  
**[Test build #96768 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96768/testReport)**
 for PR 22466 at commit 
[`3c2831b`](https://github.com/apache/spark/commit/3c2831bf70d4ed83414625fd3e4b7607624d09fc).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22473: [SPARK-25449][CORE] Heartbeat shouldn't include a...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/22473#discussion_r221313411
  
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -609,13 +609,13 @@ class SparkConf(loadDefaults: Boolean) extends 
Cloneable with Logging with Seria
 require(!encryptionEnabled || get(NETWORK_AUTH_ENABLED),
   s"${NETWORK_AUTH_ENABLED.key} must be enabled when enabling 
encryption.")
 
-val executorTimeoutThreshold = 
getTimeAsSeconds("spark.network.timeout", "120s")
-val executorHeartbeatInterval = 
getTimeAsSeconds("spark.executor.heartbeatInterval", "10s")
+val executorTimeoutThreshold = getTimeAsMs("spark.network.timeout", 
"120s")
--- End diff --

Could you use `getTimeAsSeconds` and manually convert it to `ms`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

[RussellSpitzer]

Github user RussellSpitzer commented on the issue:

https://github.com/apache/spark/pull/22576
  
Looks like the session with extensions from the test suite is leaking to
other suites ... Investigating

On Fri, Sep 28, 2018 at 11:25 AM UCB AMPLab 
wrote:

> Test FAILed.
> Refer to this link for build results (access rights to CI server needed):
>
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96757/
> Test FAILed.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/22473
  
Looks like `org.apache.spark.deploy.history.HistoryServerSuite.executor 
list with executor metrics json` and 
`org.apache.spark.util.JsonProtocolSuite.SparkListenerEvent` are broken by the 
changes. Could you also fix them?




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/22237
  
@MaxGekk is it difficult to add a conf?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/22237
  
> @MaxGekk is it difficult to add a conf?

@HyukjinKwon I will try.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22339: [SPARK-17159][STREAM] Significant speed up for ru...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/22339#discussion_r22134
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -196,29 +191,29 @@ class FileInputDStream[K, V, F <: NewInputFormat[K, 
V]](
   logDebug(s"Getting new files for time $currentTime, " +
 s"ignoring files older than $modTimeIgnoreThreshold")
 
-  val newFileFilter = new PathFilter {
-def accept(path: Path): Boolean = isNewFile(path, currentTime, 
modTimeIgnoreThreshold)
-  }
-  val directoryFilter = new PathFilter {
-override def accept(path: Path): Boolean = 
fs.getFileStatus(path).isDirectory
-  }
-  val directories = fs.globStatus(directoryPath, 
directoryFilter).map(_.getPath)
+  val directories = 
Option(fs.globStatus(directoryPath)).getOrElse(Array.empty[FileStatus])
+  .filter(_.isDirectory)
+  .map(_.getPath)
   val newFiles = directories.flatMap(dir =>
-fs.listStatus(dir, newFileFilter).map(_.getPath.toString))
+fs.listStatus(dir)
+.filter(isNewFile(_, currentTime, modTimeIgnoreThreshold))
+.map(_.getPath.toString))
   val timeTaken = clock.getTimeMillis() - lastNewFileFindingTime
-  logInfo("Finding new files took " + timeTaken + " ms")
-  logDebug("# cached file times = " + fileToModTime.size)
+  logInfo(s"Finding new files took $timeTaken ms")
--- End diff --

depends on how big it grows over time


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22339: [SPARK-17159][STREAM] Significant speed up for running s...

[steveloughran]

Github user steveloughran commented on the issue:

https://github.com/apache/spark/pull/22339
  
Why the speedups? Comes from that glob filter calling getFileStatus() on 
every entry, which is is 1-3 HTTP requests and a few hundred millis per call, 
when instead that can be handled later. As a result, the more files you have in 
a path, the more time the scan takes, until eventually the scan time > window 
interval at which point your code is dead.

The other stuff is simply associated optimisations.

Now, I'm obviously happy with this, especially as I seem I getting credit. 
And it will help speedup working with any store. But I need to warn people: it 
is not sufficient

The key problem here is: files uploaded by S3 multipart upload get a 
timestamp on when the upload began, not finished —yet only become visible at 
the end of the upload. If a caller starts up an upload in window t, and doesn't 
complete it until window t+1, then it may get missed.

There's not much which can be done here, except in documenting the risk.

What is a good solution? It'd be to use the cloud-infra-providers own event 
notification mechanism and subscribe to changes in a store. AWS, Azure and GCS 
all offer something like this. 

There's a home for the S3 one of those in aws-kinesis, perhaps


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22339: [SPARK-17159][STREAM] Significant speed up for ru...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/22339#discussion_r221326673
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -196,29 +191,29 @@ class FileInputDStream[K, V, F <: NewInputFormat[K, 
V]](
   logDebug(s"Getting new files for time $currentTime, " +
 s"ignoring files older than $modTimeIgnoreThreshold")
 
-  val newFileFilter = new PathFilter {
-def accept(path: Path): Boolean = isNewFile(path, currentTime, 
modTimeIgnoreThreshold)
-  }
-  val directoryFilter = new PathFilter {
-override def accept(path: Path): Boolean = 
fs.getFileStatus(path).isDirectory
-  }
-  val directories = fs.globStatus(directoryPath, 
directoryFilter).map(_.getPath)
+  val directories = 
Option(fs.globStatus(directoryPath)).getOrElse(Array.empty[FileStatus])
+  .filter(_.isDirectory)
+  .map(_.getPath)
   val newFiles = directories.flatMap(dir =>
-fs.listStatus(dir, newFileFilter).map(_.getPath.toString))
+fs.listStatus(dir)
+.filter(isNewFile(_, currentTime, modTimeIgnoreThreshold))
+.map(_.getPath.toString))
   val timeTaken = clock.getTimeMillis() - lastNewFileFindingTime
-  logInfo("Finding new files took " + timeTaken + " ms")
-  logDebug("# cached file times = " + fileToModTime.size)
+  logInfo(s"Finding new files took $timeTaken ms")
--- End diff --

It was originally @ info, so if it it filled up logs *too much* there'd be 
complaints. What's important is that the time to scan is printed, either @ info 
or debug, so someone can see what's happening. Probably what does need logging 
@ warn is when the time to scan is greater than the window, or just getting 
close to it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

[RussellSpitzer]

Github user RussellSpitzer commented on the issue:

https://github.com/apache/spark/pull/22576
  
Ah I was registering functions with the built-in registry which is not 
reset. I've changed it to register only with a clone of the built-in registry. 
This would allow multiple extensions, or some sessions to use extensions and 
others not to 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22576
  
**[Test build #96769 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96769/testReport)**
 for PR 22576 at commit 
[`718c8ec`](https://github.com/apache/spark/commit/718c8ecf48b22c6ad3e93db4318e631e75cb49f7).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22339: [SPARK-17159][STREAM] Significant speed up for running s...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22339
  
Yeah I agree, I was saying I do think it will speed things up. If it's a 
non-trivial win it's worthwhile even if it isn't the last optimization here. Is 
there any downside to this?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22316
  
**[Test build #96759 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96759/testReport)**
 for PR 22316 at commit 
[`d645d06`](https://github.com/apache/spark/commit/d645d06df4520522c6eae9dad2d75c9ed73a3e39).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22316
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96759/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22316
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22581: [SPARK-25565][BUILD] Add scalastyle rule to check...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/22581#discussion_r221333480
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -975,7 +975,7 @@ object SQLConf {
 "checkpoint location.")
   .stringConf
--- End diff --

not quite relevant with what the PR targets but okay.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22581
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22581
  
**[Test build #96770 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96770/testReport)**
 for PR 22581 at commit 
[`e011d5f`](https://github.com/apache/spark/commit/e011d5fb10a08f2c400836ecbf8e11d253e86270).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22581
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3575/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22339: [SPARK-17159][STREAM] Significant speed up for running s...

[steveloughran]

Github user steveloughran commented on the issue:

https://github.com/apache/spark/pull/22339
  
no, no cost penalties. Slightly lower namenode load too. If you had many, 
many spark streaming clients scanning directories, HDFS ops teams would 
eventually get upset. This will postpone the day


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22566
  
**[Test build #96761 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96761/testReport)**
 for PR 22566 at commit 
[`bf18765`](https://github.com/apache/spark/commit/bf187657c4281d04ee2a34d079689b1e599ef9f9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22566
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22566
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96761/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22473
  
**[Test build #96771 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96771/testReport)**
 for PR 22473 at commit 
[`f6fa337`](https://github.com/apache/spark/commit/f6fa33790769c14d9dde6f56e07233c2887d80a6).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18906
  
**[Test build #96765 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96765/testReport)**
 for PR 18906 at commit 
[`fac6f1e`](https://github.com/apache/spark/commit/fac6f1e729a152540b4a408f9efef96ba9df965d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18906
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96765/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18906
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22579: [SPARK-25429][SQL] Use Set instead of Array to improve l...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22579
  
**[Test build #96764 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96764/testReport)**
 for PR 22579 at commit 
[`9b3b164`](https://github.com/apache/spark/commit/9b3b164af3e8e43caec7e1cedfcaf3a8b1c6d8f2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22579: [SPARK-25429][SQL] Use Set instead of Array to improve l...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22579
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22579: [SPARK-25429][SQL] Use Set instead of Array to improve l...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22579
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96764/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22237
  
**[Test build #96772 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96772/testReport)**
 for PR 22237 at commit 
[`1f052c4`](https://github.com/apache/spark/commit/1f052c47276193280bfc08028b65e32c81f0c022).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22581
  
**[Test build #96763 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96763/testReport)**
 for PR 22581 at commit 
[`adda54f`](https://github.com/apache/spark/commit/adda54f76033a6eae160b3cfb80a67c22dc2a190).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22581
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22503: [SPARK-25493][SQL] Use auto-detection for CRLF in CSV da...

[justinuang]

Github user justinuang commented on the issue:

https://github.com/apache/spark/pull/22503
  
What does it take to get this to be merged in?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22581: [SPARK-25565][BUILD] Add scalastyle rule to check add Lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22581
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96763/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/22473
  
LGTM pending tests


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21809
  
**[Test build #96762 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96762/testReport)**
 for PR 21809 at commit 
[`3b73840`](https://github.com/apache/spark/commit/3b7384011c0d4e41ecd5c4b797f08c1c62088fbd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21809
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21809
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96762/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup t...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/22584

[SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

## What changes were proposed in this pull request?

This adds a missing end markup tag.

## How was this patch tested?

This is a doc-only change. Manual via `SKIP_API=1 jekyll build`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-25262

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22584.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22584


commit 14bf049e1b171cb1808670c180e032f15b82fde5
Author: Dongjoon Hyun 
Date:   2018-09-28T19:52:49Z

[SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22584
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22584
  
**[Test build #96773 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96773/testReport)**
 for PR 22584 at commit 
[`14bf049`](https://github.com/apache/spark/commit/14bf049e1b171cb1808670c180e032f15b82fde5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22584
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3576/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup t...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22584#discussion_r221365570
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -799,7 +799,7 @@ specific to Spark on Kubernetes.
 
 
   spark.kubernetes.local.dirs.tmpfs
-  false
+  false
--- End diff --

cc @rvesse @mccheah


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22466
  
**[Test build #96768 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96768/testReport)**
 for PR 22466 at commit 
[`3c2831b`](https://github.com/apache/spark/commit/3c2831bf70d4ed83414625fd3e4b7607624d09fc).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22466
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22466
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96768/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/spark/pull/22584
  
LGTM


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22585: [SPARK-23285][DOC][FOLLOWUP] Fix missing markup t...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/22585

[SPARK-23285][DOC][FOLLOWUP] Fix missing markup tag

## What changes were proposed in this pull request?

This adds a missing markup tag. This should go to `master/branch-2.4`.

## How was this patch tested?

Manual via `SKIP_API=1 jekyll build`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-23285

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22585


commit 44b80f05cde07fd62064cbaa1f36940764f59b26
Author: Dongjoon Hyun 
Date:   2018-09-28T20:03:48Z

[SPARK-23285][DOC][FOLLOWUP] Fix missing markup tag




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22585: [SPARK-23285][DOC][FOLLOWUP] Fix missing markup t...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22585#discussion_r221368258
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -691,6 +691,7 @@ specific to Spark on Kubernetes.
 Example values include 0.1, 500m, 1.5, 5, etc., with the definition of 
cpu units documented in [CPU 
units](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units).
   
 This is distinct from spark.executor.cores: it is only 
used and takes precedence over spark.executor.cores for specifying 
the executor pod cpu request if set. Task 
 parallelism, e.g., number of tasks an executor can run concurrently is 
not affected by this.
+  
--- End diff --

cc @liyinan926 @foxish


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22584
  
Thank you for review, @mgaido91 !


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22466
  
**[Test build #96766 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96766/testReport)**
 for PR 22466 at commit 
[`86e4d50`](https://github.com/apache/spark/commit/86e4d50d27ecec914ef534e84dcc75965456a32b).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22585: [SPARK-23285][DOC][FOLLOWUP] Fix missing markup t...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/22585#discussion_r221368693
  
--- Diff: docs/running-on-kubernetes.md ---
@@ -691,6 +691,7 @@ specific to Spark on Kubernetes.
 Example values include 0.1, 500m, 1.5, 5, etc., with the definition of 
cpu units documented in [CPU 
units](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units).
   
 This is distinct from spark.executor.cores: it is only 
used and takes precedence over spark.executor.cores for specifying 
the executor pod cpu request if set. Task 
 parallelism, e.g., number of tasks an executor can run concurrently is 
not affected by this.
+  
--- End diff --

Thanks for catching it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22585: [SPARK-23285][DOC][FOLLOWUP] Fix missing markup tag

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22585
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22585: [SPARK-23285][DOC][FOLLOWUP] Fix missing markup tag

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22585
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3577/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22584: [SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22584
  
**[Test build #96773 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96773/testReport)**
 for PR 22584 at commit 
[`14bf049`](https://github.com/apache/spark/commit/14bf049e1b171cb1808670c180e032f15b82fde5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22466
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96766/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22585: [SPARK-23285][DOC][FOLLOWUP] Fix missing markup tag

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22585
  
**[Test build #96774 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96774/testReport)**
 for PR 22585 at commit 
[`44b80f0`](https://github.com/apache/spark/commit/44b80f05cde07fd62064cbaa1f36940764f59b26).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org