[GitHub] spark pull request #17581: [SPARK-20248][ SQL]Spark SQL add limit parameter ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17581#discussion_r111069701
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -359,6 +359,16 @@ object SQLConf {
   .booleanConf
   .createWithDefault(false)
 
+  val THRIFTSERVER_RESULT_LIMIT =
+buildConf("spark.sql.thriftserver.retainedResults")
--- End diff --

How about Hive thrift server? Does it has the similar parameters? The 
parameter name does not look straightforward to me. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17581: [SPARK-20248][ SQL]Spark SQL add limit parameter ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17581#discussion_r111069617
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -359,6 +359,16 @@ object SQLConf {
   .booleanConf
   .createWithDefault(false)
 
+  val THRIFTSERVER_RESULT_LIMIT =
+buildConf("spark.sql.thriftserver.retainedResults")
+  .internal()
+  .doc("The maximum number of rows returned by Thrift Server when 
running a query " +
+"without a limit, and when a query with a limit or this is set to 
0, " +
--- End diff --

`without a limit, and when` -> `without a limit. When`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...

[wangmiao1981]

Github user wangmiao1981 commented on the issue:

https://github.com/apache/spark/pull/17611
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...

[wangmiao1981]

Github user wangmiao1981 commented on the issue:

https://github.com/apache/spark/pull/17611
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17524: [SPARK-19235] [SQL] [TEST] [FOLLOW-UP] Enable Test Cases...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17524
  
**[Test build #75727 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75727/testReport)**
 for PR 17524 at commit 
[`427741f`](https://github.com/apache/spark/commit/427741f548ff4469d62906546655f7ec96564ced).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17524: [SPARK-19235] [SQL] [TEST] [FOLLOW-UP] Enable Test Cases...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17524
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17613
  
**[Test build #75726 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75726/testReport)**
 for PR 17613 at commit 
[`4d6e3cb`](https://github.com/apache/spark/commit/4d6e3cb957e5c08a0ba2b62d7a4445cc218f5e83).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17615: [SPARK-20303][SQL] Rename createTempFunction to r...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17615#discussion_r111068193
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala 
---
@@ -59,17 +60,13 @@ case class CreateFunctionCommand(
   // We first load resources and then put the builder in the function 
registry.
   // Please note that it is allowed to overwrite an existing temp 
function.
   catalog.loadFunctionResources(resources)
-  val info = new ExpressionInfo(className, functionName)
-  val builder = catalog.makeFunctionBuilder(functionName, className)
-  catalog.createTempFunction(functionName, info, builder, 
ignoreIfExists = false)
+  catalog.registerFunction(func, ignoreIfExists = false)
 } else {
   // For a permanent, we will store the metadata into underlying 
external catalog.
   // This function will be loaded into the FunctionRegistry when a 
query uses it.
   // We do not load it into FunctionRegistry right now.
   // TODO: should we also parse "IF NOT EXISTS"?
--- End diff --

Should we support it? @cloud-fan @yhuai @rxin 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17615: [SPARK-20303][SQL] Rename createTempFunction to r...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17615#discussion_r111068131
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -1050,7 +1050,7 @@ class SessionCatalog(
*
* This performs reflection to decide what type of [[Expression]] to 
return in the builder.
*/
-  def makeFunctionBuilder(name: String, functionClassName: String): 
FunctionBuilder = {
+  protected def makeFunctionBuilder(name: String, functionClassName: 
String): FunctionBuilder = {
--- End diff --

`registerFunction` is the only caller of `makeFunctionBuilder` after this 
PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17615
  
**[Test build #75725 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75725/testReport)**
 for PR 17615 at commit 
[`e876af1`](https://github.com/apache/spark/commit/e876af1882a53fcd5569594e9ea486dba66850b4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17615: [SPARK-20303][SQL] Rename createTempFunction to r...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17615#discussion_r111068037
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -124,13 +124,6 @@ private[sql] class HiveSessionCatalog(
   }
 
   private def lookupFunction0(name: FunctionIdentifier, children: 
Seq[Expression]): Expression = {
-// TODO: Once lookupFunction accepts a FunctionIdentifier, we should 
refactor this method to
-// if (super.functionExists(name)) {
-//   super.lookupFunction(name, children)
-// } else {
-//   // This function is a Hive builtin function.
-//   ...
-// }
--- End diff --

`LookupFunction` already accepts `FunctionIdentifier`, but we are unable to 
do it using the above way because `functionExists` does not consider the 
difference among Hive built-in, Spark temporary and permanent functions. More 
following clean-ups are needed. Will try to do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoo...

[brkyvz]

Github user brkyvz commented on a diff in the pull request:

https://github.com/apache/spark/pull/17613#discussion_r111067826
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -284,42 +284,38 @@ class StreamExecution(
 triggerExecutor.execute(() => {
   startTrigger()
 
-  val continueToRun =
-if (isActive) {
-  reportTimeTaken("triggerExecution") {
-if (currentBatchId < 0) {
-  // We'll do this initialization only once
-  populateStartOffsets(sparkSessionToRunBatches)
-  logDebug(s"Stream running from $committedOffsets to 
$availableOffsets")
-} else {
-  constructNextBatch()
-}
-if (dataAvailable) {
-  currentStatus = currentStatus.copy(isDataAvailable = 
true)
-  updateStatusMessage("Processing new data")
-  runBatch(sparkSessionToRunBatches)
-}
+  if (isActive) {
+reportTimeTaken("triggerExecution") {
+  if (currentBatchId < 0) {
+// We'll do this initialization only once
+populateStartOffsets(sparkSessionToRunBatches)
+logDebug(s"Stream running from $committedOffsets to 
$availableOffsets")
+  } else {
+constructNextBatch()
   }
-  // Report trigger as finished and construct progress object.
-  finishTrigger(dataAvailable)
   if (dataAvailable) {
-// Update committed offsets.
-batchCommitLog.add(currentBatchId)
-committedOffsets ++= availableOffsets
-logDebug(s"batch ${currentBatchId} committed")
-// We'll increase currentBatchId after we complete 
processing current batch's data
-currentBatchId += 1
-  } else {
-currentStatus = currentStatus.copy(isDataAvailable = false)
-updateStatusMessage("Waiting for data to arrive")
-Thread.sleep(pollingDelayMs)
+currentStatus = currentStatus.copy(isDataAvailable = true)
+updateStatusMessage("Processing new data")
+runBatch(sparkSessionToRunBatches)
   }
-  true
+}
+// Report trigger as finished and construct progress object.
+finishTrigger(dataAvailable)
--- End diff --

I don't think I moved it out. Is the diff and whitespace confusing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17615: [SPARK-20303][SQL] Rename createTempFunction to r...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/17615

[SPARK-20303][SQL] Rename createTempFunction to registerFunction

### What changes were proposed in this pull request?
Session catalog API `createTempFunction` is being used by Hive build-in 
functions, persistent functions, and temporary functions. Thus, the name is 
confusing. This PR is to rename it by `registerFunction`. Also we can move 
construction of `FunctionBuilder` and `ExpressionInfo` into the new 
`registerFunction`, instead of duplicating the logics everywhere.

In the next PRs, the remaining Function-related APIs also need cleanups. 

### How was this patch tested?
Existing test cases.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark cleanupCreateTempFunction

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17615


commit cc164813d7d775f233f62901e09885fc322bc150
Author: Xiao Li 
Date:   2017-04-12T05:20:07Z

fix.

commit e876af1882a53fcd5569594e9ea486dba66850b4
Author: Xiao Li 
Date:   2017-04-12T05:29:02Z

fix.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17613
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75720/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17613
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17613
  
**[Test build #75720 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75720/testReport)**
 for PR 17613 at commit 
[`c060e6b`](https://github.com/apache/spark/commit/c060e6b1b811f1e55d4ac0becf38683cfc1fe536).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17614: [SPARK-20302][SQL] Short circuit cast when from a...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/17614#discussion_r111064001
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala ---
@@ -288,4 +288,30 @@ object DataType {
   case (fromDataType, toDataType) => fromDataType == toDataType
 }
   }
+
+  /**
+   * Returns true if the two data types share the same "shape", i.e. the 
types (including
+   * nullability) are the same, but the field names don't need to be the 
same.
+   */
+  def equalsStructurally(from: DataType, to: DataType): Boolean = {
+(from, to) match {
+  case (left: ArrayType, right: ArrayType) =>
+equalsStructurally(left.elementType, right.elementType) &&
+  left.containsNull == right.containsNull
--- End diff --

That's not symmetric. equalsStructurally should be symmetric, unless we 
rename this something else (e.g. structurallyCastable)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17546: [SPARK-20233] [SQL] Apply star-join filter heuris...

[ioana-delaney]

Github user ioana-delaney commented on a diff in the pull request:

https://github.com/apache/spark/pull/17546#discussion_r111063912
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala
 ---
@@ -54,8 +54,6 @@ case class CostBasedJoinReorder(conf: SQLConf) extends 
Rule[LogicalPlan] with Pr
 
   private def reorder(plan: LogicalPlan, output: Seq[Attribute]): 
LogicalPlan = {
 val (items, conditions) = extractInnerJoins(plan)
-// TODO: Compute the set of star-joins and use them in the join 
enumeration
-// algorithm to prune un-optimal plan choices.
--- End diff --

@cloud-fan Once CBO is enabled by default, I can remove the call from 
```ReorderJoin```.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17614: [SPARK-20302][SQL] Short circuit cast when from and to t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17614
  
**[Test build #75724 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75724/testReport)**
 for PR 17614 at commit 
[`b97b46e`](https://github.com/apache/spark/commit/b97b46e412d3e56ad5dee038e69cdeac5623b411).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17614: [SPARK-20302][SQL] Short circuit cast when from a...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17614#discussion_r111063773
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala ---
@@ -288,4 +288,30 @@ object DataType {
   case (fromDataType, toDataType) => fromDataType == toDataType
 }
   }
+
+  /**
+   * Returns true if the two data types share the same "shape", i.e. the 
types (including
+   * nullability) are the same, but the field names don't need to be the 
same.
+   */
+  def equalsStructurally(from: DataType, to: DataType): Boolean = {
+(from, to) match {
+  case (left: ArrayType, right: ArrayType) =>
+equalsStructurally(left.elementType, right.elementType) &&
+  left.containsNull == right.containsNull
--- End diff --

shall we be more flexible here? i.e. `!left.containsNull || 
right.containsNull`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17590: [SPARK-20278][R] Disable 'multiple_dots_linter' lint rul...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17590
  
I think it is because it only checks multiple dots and the case I described 
above was finally found now (in my previous PR related with `from_json` 
function).

I think multiple dots are still valid per ...

> The preferred form for variable names is all lower case letters and words 
separated with dots

Yea, maybe. I am fine with leaving this open for some days more to see if 
there is any objection. Probably, let me cc who I saw made many contributions. 
cc @shivaram, @yanboliang, @wangmiao1981 and @actuaryzhang here. Please let me 
know if there is any concern.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17550: [SPARK-20240][SQL] SparkSQL support limitations o...

[zenglinxi0615]

Github user zenglinxi0615 closed the pull request at:

https://github.com/apache/spark/pull/17550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17550: [SPARK-20240][SQL] SparkSQL support limitations of max d...

[zenglinxi0615]

Github user zenglinxi0615 commented on the issue:

https://github.com/apache/spark/pull/17550
  
ok，going to close this PR and open a new PR using the master branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17586
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17330: [SPARK-19993][SQL] Caching logical plans containing subq...

[dilipbiswal]

Github user dilipbiswal commented on the issue:

https://github.com/apache/spark/pull/17330
  
@cloud-fan Thanks a lot!!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17586
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75721/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17586
  
**[Test build #75721 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75721/testReport)**
 for PR 17586 at commit 
[`6d1e5fa`](https://github.com/apache/spark/commit/6d1e5fa9828670be1c9bc5b5e1bdf175d94f0f85).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17436#discussion_r111062746
  
--- Diff: 
core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala ---
@@ -210,7 +210,7 @@ object UnifiedMemoryManager {
   private def getMaxMemory(conf: SparkConf): Long = {
 val systemMemory = conf.getLong("spark.testing.memory", 
Runtime.getRuntime.maxMemory)
 val reservedMemory = conf.getLong("spark.testing.reservedMemory",
-  if (conf.contains("spark.testing")) 0 else 
RESERVED_SYSTEM_MEMORY_BYTES)
+  if (conf.contains("spark.testing") || true) 0 else 
RESERVED_SYSTEM_MEMORY_BYTES)
--- End diff --

?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17330: [SPARK-19993][SQL] Caching logical plans containi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17330


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17330: [SPARK-19993][SQL] Caching logical plans containing subq...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/17330
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17590: [SPARK-20278][R] Disable 'multiple_dots_linter' lint rul...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17590
  
Changes look fine. I think we should get some feedback on style/guideline 
changes like this though.

Also do you know why we don't ever see these types of error in Jenkins? Is 
it because it's running an older lintr?



R/functions.R:2462:31: style: Words within variable and function names 
should be separated by '_' rather than '.'.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17469
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17469
  
**[Test build #75723 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75723/testReport)**
 for PR 17469 at commit 
[`b157bc3`](https://github.com/apache/spark/commit/b157bc3288c326fac847aca3fecb8d0f79592f42).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17469
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75723/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17375: [SPARK-19019][PYTHON][BRANCH-1.6] Fix hijacked `collecti...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17375
  
@joshrosen, do you mind if I ask a quick look here? I know you know PySpark 
well. I think this backport got a sign-off and a positive comment from both 
committers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17477
  
Looks there is another break.

```
[error] 
/home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/catalog/Catalog.java:453:
 error: reference not found
[error]* Invalidates and refreshes all the cached data (and the 
associated metadata) for any {@link Dataset}
[error]
```

Let me clean up this and address comments. Thank you @JoshRosen.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17477
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75722/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17477
  
**[Test build #75722 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75722/testReport)**
 for PR 17477 at commit 
[`4d39544`](https://github.com/apache/spark/commit/4d39544cc4f8075242370f65e6849d2abd2562f9).
 * This patch **fails to generate documentation**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17477
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17469
  
**[Test build #75723 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75723/testReport)**
 for PR 17469 at commit 
[`b157bc3`](https://github.com/apache/spark/commit/b157bc3288c326fac847aca3fecb8d0f79592f42).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/17469
  
jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/17477
  
Let a few nitpicky comments. 

@srowen @jkbradley, could you take a look and merge it after changes if it 
looks okay to you? Overall build change structure looks okay to me if we're 
fine with failing PR build on doc build breaks. I did a somewhat cursory 
examination of the actual doc changes, so additional review there is welcome if 
you have time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17477#discussion_r111059960
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala ---
@@ -74,7 +74,7 @@ abstract class Classifier[
* and features (`Vector`).
* @param numClasses  Number of classes label can take.  Labels must be 
integers in the range
*[0, numClasses).
-   * @throws SparkException  if any label is not an integer >= 0
+   * @note Throws `SparkException` if any label is not an integer is 
greater than or equal to 0
--- End diff --

`is not a nonnegative integer`? 
http://mathworld.wolfram.com/NonnegativeInteger.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17477#discussion_r111059991
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala ---
@@ -74,7 +74,7 @@ abstract class Classifier[
* and features (`Vector`).
* @param numClasses  Number of classes label can take.  Labels must be 
integers in the range
*[0, numClasses).
-   * @throws SparkException  if any label is not an integer >= 0
+   * @note Throws `SparkException` if any label is not an integer is 
greater than or equal to 0
--- End diff --

Or `is a non-integer or is negative`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17477#discussion_r111059834
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -704,12 +704,12 @@ private[spark] object TaskSchedulerImpl {
* Used to balance containers across hosts.
*
* Accepts a map of hosts to resource offers for that host, and returns 
a prioritized list of
-   * resource offers representing the order in which the offers should be 
used.  The resource
+   * resource offers representing the order in which the offers should be 
used. The resource
* offers are ordered such that we'll allocate one container on each 
host before allocating a
* second container on any host, and so on, in order to reduce the 
damage if a host fails.
*
-   * For example, given , , , 
returns
-   * [o1, o5, o4, 02, o6, o3]
+   * For example, given a map consisting of h1 to [o1, o2, o3], h2 to [o4] 
and h3 to [o5, o6],
+   * returns a list, [o1, o5, o4, o2, o6, o3].
--- End diff --

Can we also wrap this in code or otherwise escape it or use a different 
symbol?

```
{h1: [o1, o2, o3], h2: [o4], ...}
```

is clearer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17477
  
**[Test build #75722 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75722/testReport)**
 for PR 17477 at commit 
[`4d39544`](https://github.com/apache/spark/commit/4d39544cc4f8075242370f65e6849d2abd2562f9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17477#discussion_r111059727
  
--- Diff: core/src/main/scala/org/apache/spark/rpc/RpcEndpoint.scala ---
@@ -33,9 +33,9 @@ private[spark] trait RpcEnvFactory {
  *
  * It is guaranteed that `onStart`, `receive` and `onStop` will be called 
in sequence.
  *
- * The life-cycle of an endpoint is:
+ * The life-cycle of an endpoint is as below in an order:
--- End diff --

Can we just wrap this block as code? The rewording is confusing and doesn't 
read as clearly to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

[yssharma]

Github user yssharma commented on the issue:

https://github.com/apache/spark/pull/17467
  
Waiting for review @brkyvz . Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/17477
  
jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

[yssharma]

Github user yssharma commented on the issue:

https://github.com/apache/spark/pull/17506
  
@srowen do you feel this patch could be merged now ?
Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/17612
  
Oh, thank you, @gatorsmile !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoo...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/17613#discussion_r111058990
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala ---
@@ -277,6 +277,11 @@ trait StreamTest extends QueryTest with 
SharedSQLContext with Timeouts {
 
 def threadState =
   if (currentStream != null && currentStream.microBatchThread.isAlive) 
"alive" else "dead"
+def threadStackTrace = if (currentStream != null && 
currentStream.microBatchThread.isAlive) {
--- End diff --

+1 on keeping this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17586
  
**[Test build #75721 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75721/testReport)**
 for PR 17586 at commit 
[`6d1e5fa`](https://github.com/apache/spark/commit/6d1e5fa9828670be1c9bc5b5e1bdf175d94f0f85).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoo...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/17613#discussion_r111058917
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
 ---
@@ -284,42 +284,38 @@ class StreamExecution(
 triggerExecutor.execute(() => {
   startTrigger()
 
-  val continueToRun =
-if (isActive) {
-  reportTimeTaken("triggerExecution") {
-if (currentBatchId < 0) {
-  // We'll do this initialization only once
-  populateStartOffsets(sparkSessionToRunBatches)
-  logDebug(s"Stream running from $committedOffsets to 
$availableOffsets")
-} else {
-  constructNextBatch()
-}
-if (dataAvailable) {
-  currentStatus = currentStatus.copy(isDataAvailable = 
true)
-  updateStatusMessage("Processing new data")
-  runBatch(sparkSessionToRunBatches)
-}
+  if (isActive) {
+reportTimeTaken("triggerExecution") {
+  if (currentBatchId < 0) {
+// We'll do this initialization only once
+populateStartOffsets(sparkSessionToRunBatches)
+logDebug(s"Stream running from $committedOffsets to 
$availableOffsets")
+  } else {
+constructNextBatch()
   }
-  // Report trigger as finished and construct progress object.
-  finishTrigger(dataAvailable)
   if (dataAvailable) {
-// Update committed offsets.
-batchCommitLog.add(currentBatchId)
-committedOffsets ++= availableOffsets
-logDebug(s"batch ${currentBatchId} committed")
-// We'll increase currentBatchId after we complete 
processing current batch's data
-currentBatchId += 1
-  } else {
-currentStatus = currentStatus.copy(isDataAvailable = false)
-updateStatusMessage("Waiting for data to arrive")
-Thread.sleep(pollingDelayMs)
+currentStatus = currentStatus.copy(isDataAvailable = true)
+updateStatusMessage("Processing new data")
+runBatch(sparkSessionToRunBatches)
   }
-  true
+}
+// Report trigger as finished and construct progress object.
+finishTrigger(dataAvailable)
--- End diff --

why did you move this out of the `reportTimeTaken { ... }`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17606: [SPARK-20291][SQL] NaNvl(FloatType, NullType) sho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17606


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17606: [SPARK-20291][SQL] NaNvl(FloatType, NullType) should not...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/17606
  
LGTM, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17546: [SPARK-20233] [SQL] Apply star-join filter heuris...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17546#discussion_r111058094
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala
 ---
@@ -54,8 +54,6 @@ case class CostBasedJoinReorder(conf: SQLConf) extends 
Rule[LogicalPlan] with Pr
 
   private def reorder(plan: LogicalPlan, output: Seq[Attribute]): 
LogicalPlan = {
 val (items, conditions) = extractInnerJoins(plan)
-// TODO: Compute the set of star-joins and use them in the join 
enumeration
-// algorithm to prune un-optimal plan choices.
--- End diff --

do we have a plan to completely merge these 2 rules?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17613
  
**[Test build #75720 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75720/testReport)**
 for PR 17613 at commit 
[`c060e6b`](https://github.com/apache/spark/commit/c060e6b1b811f1e55d4ac0becf38683cfc1fe536).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFuncti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17222
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75719/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFuncti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17222
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoo...

[brkyvz]

GitHub user brkyvz opened a pull request:

https://github.com/apache/spark/pull/17613

[SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell.runCommand 
flakiness in Structured Streaming tests

## What changes were proposed in this pull request?

Some Structured Streaming tests show flakiness

## How was this patch tested?

Thousand retries locally and Jenkins of the flaky tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brkyvz/spark flaky-stream-agg

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17613.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17613


commit c060e6b1b811f1e55d4ac0becf38683cfc1fe536
Author: Burak Yavuz 
Date:   2017-04-12T02:48:39Z

ready for jenkins




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFuncti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17222
  
**[Test build #75719 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75719/testReport)**
 for PR 17222 at commit 
[`6aa5d85`](https://github.com/apache/spark/commit/6aa5d85c91c33fd771a01e3b1370597b106d650e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17612: [MINOR][DOCS] Update supported versions for Hive ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17612


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17612
  
Thanks! Merging to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17608: [SPARK-20293][WEB UI][History]In the page of 'jobs' or '...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/17608
  
I just downloaded the latest spark master code to compile and install , 
test the problem, there are still bugs, the page is wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast ArrayType(InternalType)

[ashokblend]

Github user ashokblend commented on the issue:

https://github.com/apache/spark/pull/16057
  
Any Reason why its not merged in branch2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast ArrayType(InternalType)

[ashokblend]

Github user ashokblend commented on the issue:

https://github.com/apache/spark/pull/16057
  
Hi Guys


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17608: [SPARK-20293][WEB UI][History]In the page of 'jobs' or '...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/17608
  
So how do I deal with this PR?@ajbozarth


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17608: [SPARK-20293][WEB UI][History]In the page of 'jobs' or '...

[ajbozarth]

Github user ajbozarth commented on the issue:

https://github.com/apache/spark/pull/17608
  
On a quick git blame this is my code that seems to be broken. I'll take a 
more detailed look when I can in the next couple days.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16774
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75716/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16774
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16774
  
**[Test build #75716 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75716/testReport)**
 for PR 16774 at commit 
[`5e8a086`](https://github.com/apache/spark/commit/5e8a0869dcefaa5febf6cc354a7840225268acf9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `trait ExecutorServiceFactory `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17109: [SPARK-19740][MESOS]Add support in Spark to pass arbitra...

[tnachen]

Github user tnachen commented on the issue:

https://github.com/apache/spark/pull/17109
  
@srowen Appreciate the help you're doing, I think we're doing what we can 
to help review these patches and making sure Mesos support is still being 
maintained and improved over time. 
If you trust our judgement and also us still around fixing issues when 
arises, then we really just need someone like you to help merge patches. 
Ensuring someone else or if anyone that's been contributing to this area 
can become a committer definitely is a ever ongoing problem that we're still 
hoping one day can be addressed. Another parallel effort that I think is very 
worth investigating is to decouple the cluster manager intergation from Spark, 
which I believe is becoming more relevant now as we have more integration 
coming.

Long story short, if you can still help in the mean time will be greatly 
appreciated as we can still make sure improvements around Mesos integration can 
still happen


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17608: [SPARK-20293][WEB UI][History]In the page of 'jobs' or '...

[ajbozarth]

Github user ajbozarth commented on the issue:

https://github.com/apache/spark/pull/17608
  
After taking another look I was mixing this bug up with another when I 
asked that


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17608: [SPARK-20293][WEB UI][History]In the page of 'jobs' or '...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/17608
  
@ajbozarth I am using the  latest  spark version.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17612
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75717/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFuncti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17222
  
**[Test build #75719 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75719/testReport)**
 for PR 17222 at commit 
[`6aa5d85`](https://github.com/apache/spark/commit/6aa5d85c91c33fd771a01e3b1370597b106d650e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17612
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17612
  
**[Test build #75717 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75717/testReport)**
 for PR 17612 at commit 
[`d6792c2`](https://github.com/apache/spark/commit/d6792c2fe60d52f3a2931a7a32458159d3f28e2d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFuncti...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/spark/pull/17222
  
Good catch ! @holdenk `return` is removed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17586
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17586
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75718/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17586
  
**[Test build #75718 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75718/testReport)**
 for PR 17586 at commit 
[`b34603c`](https://github.com/apache/spark/commit/b34603c6706f7f90ef319be26164c4932824d252).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17109: [SPARK-19740][MESOS]Add support in Spark to pass ...

[skonto]

Github user skonto commented on a diff in the pull request:

https://github.com/apache/spark/pull/17109#discussion_r111045370
  
--- Diff: 
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala
 ---
@@ -99,6 +99,26 @@ private[mesos] object MesosSchedulerBackendUtil extends 
Logging {
 .toList
   }
 
+  /**
+   * Parse a list of docker parameters, each of which
+   * takes the form key=value
+   */
+  private def parseParamsSpec(params: String): List[Parameter] = {
+params.split(",").map(_.split("=")).flatMap { spec: Array[String] =>
--- End diff --

It should be quoted: https://github.com/docker/docker/issues/12763


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17469
  
@map222, thanks, I don't have a permission to retrigger the build and know 
why it does not automatically from the newly pushed commit. Would be great if 
@srowen or @holdenk are able to retrigger the build.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17586: [SPARK-20249][ML][PYSPARK] Add summary for LinearSVCMode...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17586
  
**[Test build #75718 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75718/testReport)**
 for PR 17586 at commit 
[`b34603c`](https://github.com/apache/spark/commit/b34603c6706f7f90ef319be26164c4932824d252).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17469: [SPARK-20132][Docs] Add documentation for column string ...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17469
  
@map222, thanks, I don't have a permission to retrigger the build and know 
why it does not. Would be great if @srowen or @holdenk are able to retrigger 
the build.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17477
  
@joshrosen, I am fine with closing it for now if you are currently not sure 
of it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17612
  
**[Test build #75717 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75717/testReport)**
 for PR 17612 at commit 
[`d6792c2`](https://github.com/apache/spark/commit/d6792c2fe60d52f3a2931a7a32458159d3f28e2d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17612: [MINOR][DOCS] Update supported versions for Hive Metasto...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/17612
  
Hi, @gatorsmile .
If possible, can we update the docs consistently with the code, too?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17612: [MINOR][DOCS] Update supported versions for Hive ...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/17612

[MINOR][DOCS] Update supported versions for Hive Metastore

## What changes were proposed in this pull request?

Since SPARK-18112 and SPARK-13446, Apache Spark starts to support reading 
Hive metastore 2.0 ~ 2.1.1. This updates the docs.

## How was this patch tested?

N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark metastore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17612.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17612


commit d6792c2fe60d52f3a2931a7a32458159d3f28e2d
Author: Dongjoon Hyun 
Date:   2017-04-12T00:28:33Z

[MINOR][DOCS] Update supported versions for Hive Metastore




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16774: [SPARK-19357][ML] Adding parallel model evaluatio...

[BryanCutler]

Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/16774#discussion_r111042302
  
--- Diff: 
examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaTrainValidationSplitExample.scala
 ---
@@ -65,6 +65,8 @@ object ModelSelectionViaTrainValidationSplitExample {
   .setEstimatorParamMaps(paramGrid)
   // 80% of the data will be used for training and the remaining 20% 
for validation.
   .setTrainRatio(0.8)
+  // Evaluate up to 2 parameter settings in parallel
+  .setNumParallelEval(2)
--- End diff --

TODO: I should probably set this in Java too, to be consistent


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17586: [SPARK-20249][ML][PYSPARK] Add summary for Linear...

[zjffdu]

Github user zjffdu commented on a diff in the pull request:

https://github.com/apache/spark/pull/17586#discussion_r111042227
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala ---
@@ -355,6 +368,19 @@ object LinearSVCModel extends 
MLReadable[LinearSVCModel] {
 }
 
 /**
+ * Abstraction for Linear SVC Training results.
+ * Currently, the training summary ignores the training weights except
+ * for the objective trace.
+ */
+case class LinearSVCTrainingSummary(
--- End diff --

The classes below `LinearSVCTrainingSummary` are private classes, so I 
think it would better to keep LinearSVCTrainingSummary there (above the private 
classes)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16774
  
**[Test build #75716 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75716/testReport)**
 for PR 16774 at commit 
[`5e8a086`](https://github.com/apache/spark/commit/5e8a0869dcefaa5febf6cc354a7840225268acf9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/16774
  
Thanks for the review @MLnick!  I changed `setExecutorService` to use a 
trait instead of just a function, which can be implemented in Java.  Works the 
same, but does add the public trait if that is ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17586: [SPARK-20249][ML][PYSPARK] Add summary for Linear...

[zjffdu]

Github user zjffdu commented on a diff in the pull request:

https://github.com/apache/spark/pull/17586#discussion_r111042049
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala ---
@@ -355,6 +368,19 @@ object LinearSVCModel extends 
MLReadable[LinearSVCModel] {
 }
 
 /**
+ * Abstraction for Linear SVC Training results.
+ * Currently, the training summary ignores the training weights except
+ * for the objective trace.
+ */
--- End diff --

weight column also is not included in `LogisticRegressionTrainingSummary`, 
should I add that as well ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17610: [SPARK-20131][Core]Use a separate lock for StandaloneSch...

[mridulm]

Github user mridulm commented on the issue:

https://github.com/apache/spark/pull/17610
  
bq. I don't get it. But I think the stack trace shows why this dead-lock 
happens.
Based on your description/stacktrace, I get why the deadlock happens - what 
I meant was, do any of the super.* methods invoked in the `stop` call tree 
assume they are invoked with `this` already locked ?

If not, then a narrow lock on `this` just to flip the state of `stopped` 
might be better. An `AtomicBoolean` will introduce a new lock (which is not 
required here I think).
The deadlock occurs because we are calling rpc with the lock held already 
(which is probably be a pattern we should somehow catch since it will 
invariably cause deadlocks !)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17527: [SPARK-20156][CORE][SQL][STREAMING][MLLIB] Java String t...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17527
  
Do you mind if I ask a example case? I just would like to look into this to 
help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17527: [SPARK-20156][CORE][SQL][STREAMING][MLLIB] Java String t...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17527
  
Thank you for your explanation. Just did a few run in our DDL support. We 
still have a few bugs in the locale support. If we use Turkish locale, a few 
test cases failed.

Do you know what is the existing locale support for Hive and Hive 
metastore? 

Also cc @cloud-fan 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16906: [SPARK-19570][PYSPARK] Allow to disable hive in pyspark ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/spark/pull/16906
  
Kindly ping @holdenk 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org