[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125567012
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,26 +46,47 @@ case class CreateFunctionCommand(
functionName: String,
className: String,
resources: Seq[FunctionResource],
-isTemp: Boolean)
+isTemp: Boolean,
+ifNotExists: Boolean,
+replace: Boolean)
extends RunnableCommand {
+ if (ifNotExists && replace) {
+throw new AnalysisException("CREATE FUNCTION with both IF NOT EXISTS
and REPLACE" +
+ " is not allowed.")
+ }
+
+ // Disallows 'CREATE TEMPORARY FUNCTION IF NOT EXISTS' to be consistent
+ // with 'CREATE TEMPORARY FUNCTION'
--- End diff --
@cloud-fan
The logic of the synax `CREATE TEMPORAY FUNCTION IF NOT EXISTS` like this
1. If the function is already exists, we do nothing not override.
2. If hte function is not exists , we create a new one.
The logic of the synax `CREATE OR REPLACE TEMPORAY FUNCTION` like this
1. If the function is already exists, override it.
2. If hte function is not exists , we create a new one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18527: [SPARK-21101][SQL] Catch IllegalStateException when CREA...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18527 **[Test build #79199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79199/testReport)** for PR 18527 at commit [`479fd44`](https://github.com/apache/spark/commit/479fd4439fed92d2daadce6dd344c68cff00ce2a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13873: [SPARK-16167][SQL] RowEncoder should preserve array/map ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13873 **[Test build #79198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79198/testReport)** for PR 13873 at commit [`1cc58a6`](https://github.com/apache/spark/commit/1cc58a6be2a63a6c575e4a707efc4fa5c7e84a0c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125564468
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,26 +46,47 @@ case class CreateFunctionCommand(
functionName: String,
className: String,
resources: Seq[FunctionResource],
-isTemp: Boolean)
+isTemp: Boolean,
+ifNotExists: Boolean,
+replace: Boolean)
extends RunnableCommand {
+ if (ifNotExists && replace) {
+throw new AnalysisException("CREATE FUNCTION with both IF NOT EXISTS
and REPLACE" +
+ " is not allowed.")
+ }
+
+ // Disallows 'CREATE TEMPORARY FUNCTION IF NOT EXISTS' to be consistent
+ // with 'CREATE TEMPORARY FUNCTION'
--- End diff --
Ah, do you mean that the value of the `overrideIfExists `parameter can be
`ifnotexists `or `replace` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17758: [SPARK-20460][SQL] Make it more consistent to han...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/17758#discussion_r125564439 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -304,7 +329,13 @@ case class DataSource( case (dataSource: SchemaRelationProvider, Some(schema)) => dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions, schema) case (dataSource: RelationProvider, None) => -dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions) +val baseRelation = + dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions) +SchemaUtils.checkColumnNameDuplication( + baseRelation.schema.map(_.name), + "in the relation schema", + equality) +baseRelation --- End diff -- ok, I'll check we can do so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17758: [SPARK-20460][SQL] Make it more consistent to han...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17758#discussion_r125564070
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -304,7 +329,13 @@ case class DataSource(
case (dataSource: SchemaRelationProvider, Some(schema)) =>
dataSource.createRelation(sparkSession.sqlContext,
caseInsensitiveOptions, schema)
case (dataSource: RelationProvider, None) =>
-dataSource.createRelation(sparkSession.sqlContext,
caseInsensitiveOptions)
+val baseRelation =
+ dataSource.createRelation(sparkSession.sqlContext,
caseInsensitiveOptions)
+SchemaUtils.checkColumnNameDuplication(
+ baseRelation.schema.map(_.name),
+ "in the relation schema",
+ equality)
+baseRelation
--- End diff --
an idea to unify the validation for both user-specified schema and inferred
schema: we can put the validation at the end of `resolveRelation`:
```
relation match {
case h: HadoopFsRelation =>
// validate data schema
// validate partition schema
// validate bucket
case _ => // validate data schema
}
relation
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125563990
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
---
@@ -1316,8 +1316,10 @@ abstract class SessionCatalogSuite extends
AnalysisTest {
val funcMeta2 = newFunc("yes_me", None)
val tempFunc1 = (e: Seq[Expression]) => e.head
val tempFunc2 = (e: Seq[Expression]) => e.last
- catalog.createFunction(newFunc("func2", Some("db2")), ignoreIfExists
= false)
- catalog.createFunction(newFunc("not_me", Some("db2")),
ignoreIfExists = false)
--- End diff --
Sorry about it , I mistake it when i resolving the conflicts. I'll revert
this later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16422: [SPARK-17642] [SQL] support DESC EXTENDED/FORMATT...
Github user wzhfy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16422#discussion_r125563599
--- Diff:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -262,7 +262,7 @@ describeFuncName
;
describeColName
-: identifier ('.' (identifier | STRING))*
+: identifier
--- End diff --
@cloud-fan I changed the syntax here so that the command supports quoted
column name but not nested columns. Could you take another look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17681 **[Test build #79196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79196/testReport)** for PR 17681 at commit [`5eb9379`](https://github.com/apache/spark/commit/5eb9379108a52824fd30ad529defa03ec984d285). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16422: [SPARK-17642] [SQL] support DESC EXTENDED/FORMATTED tabl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16422 **[Test build #79197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79197/testReport)** for PR 16422 at commit [`4b51baf`](https://github.com/apache/spark/commit/4b51baf80c3172e388a6c71ad98e9df0c8725860). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125563099
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,26 +46,47 @@ case class CreateFunctionCommand(
functionName: String,
className: String,
resources: Seq[FunctionResource],
-isTemp: Boolean)
+isTemp: Boolean,
+ifNotExists: Boolean,
+replace: Boolean)
extends RunnableCommand {
+ if (ifNotExists && replace) {
+throw new AnalysisException("CREATE FUNCTION with both IF NOT EXISTS
and REPLACE" +
+ " is not allowed.")
+ }
+
+ // Disallows 'CREATE TEMPORARY FUNCTION IF NOT EXISTS' to be consistent
+ // with 'CREATE TEMPORARY FUNCTION'
--- End diff --
`CREATE TEMPORARY VIEW`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125563038
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
---
@@ -1316,8 +1316,10 @@ abstract class SessionCatalogSuite extends
AnalysisTest {
val funcMeta2 = newFunc("yes_me", None)
val tempFunc1 = (e: Seq[Expression]) => e.head
val tempFunc2 = (e: Seq[Expression]) => e.last
- catalog.createFunction(newFunc("func2", Some("db2")), ignoreIfExists
= false)
- catalog.createFunction(newFunc("not_me", Some("db2")),
ignoreIfExists = false)
--- End diff --
why this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16056: [SPARK-18623][SQL] Add `returnNullable` to `Stati...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16056 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125562694
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -590,6 +590,13 @@ class InMemoryCatalog(
catalog(db).functions.remove(funcName)
}
+ override protected def doAlterFunction(db: String, func:
CatalogFunction): Unit = synchronized {
+requireDbExists(db)
+requireFunctionExists(db, func.identifier.funcName)
+catalog(db).functions.remove(func.identifier.funcName)
+catalog(db).functions.put(func.identifier.funcName, func)
--- End diff --
we can call `put` directly, which will update the old entry.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18174 **[Test build #79195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79195/testReport)** for PR 18174 at commit [`dc3997a`](https://github.com/apache/spark/commit/dc3997ab6d6546fac35d0b0f4044d351f7d03847). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18174 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16056 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/18174 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18416: [SPARK-21204][SQL] Add support for Scala Set collection ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18416: [SPARK-21204][SQL] Add support for Scala Set collection ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79184/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/18174 thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18416: [SPARK-21204][SQL] Add support for Scala Set collection ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #79184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79184/testReport)** for PR 18416 at commit [`83db461`](https://github.com/apache/spark/commit/83db461d84765703e32b93dd2630eeab94a05908). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18529: [SPARK-21304][SQL] remove unnecessary isNull vari...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18529 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18529 thanks for review, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17681 **[Test build #79194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79194/testReport)** for PR 17681 at commit [`1cc5cac`](https://github.com/apache/spark/commit/1cc5cacf857ae1b97aade0a2005df8701455f7ef). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16056 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16056 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79183/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16056 **[Test build #79183 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79183/testReport)** for PR 16056 at commit [`b849b59`](https://github.com/apache/spark/commit/b849b59f03c824be0530565032154f12e5001c66). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18534 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79191/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18534 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18534 **[Test build #79191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79191/testReport)** for PR 18534 at commit [`0c47cee`](https://github.com/apache/spark/commit/0c47ceef2cefb6f39ee8e040297329d8169da7df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/18529 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18529 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79182/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18460: [SPARK-21247][SQL] Allow case-insensitive type equality ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18460 Hi, @cloud-fan and @gatorsmile . Could you review this PR again? I simplified this PR with `SQLConf.get`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18529 **[Test build #79182 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79182/testReport)** for PR 18529 at commit [`50126bd`](https://github.com/apache/spark/commit/50126bdd941588337ea1b4b25974935edc6c2d7b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18516: [SPARK-21281][SQL] Use string types by default if array ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18516 **[Test build #79193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79193/testReport)** for PR 18516 at commit [`04cbf78`](https://github.com/apache/spark/commit/04cbf784944aff0c9a1c195b4facb5cc97d0fe8a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18531: [SPARK-21307] [SQL] Remove SQLConf parameters fro...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18531#discussion_r125559881
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/VariableSubstitutionSuite.scala
---
@@ -35,26 +34,29 @@ class VariableSubstitutionSuite extends SparkFunSuite {
}
test("Spark configuration variable") {
-conf.setConfString("some-random-string-abcd", "1234abcd")
-assert(sub.substitute("${hiveconf:some-random-string-abcd}") ==
"1234abcd")
-assert(sub.substitute("${sparkconf:some-random-string-abcd}") ==
"1234abcd")
-assert(sub.substitute("${spark:some-random-string-abcd}") ==
"1234abcd")
-assert(sub.substitute("${some-random-string-abcd}") == "1234abcd")
+withSQLConf("some-random-string-abcd" -> "1234abcd") {
+ assert(sub.substitute("${hiveconf:some-random-string-abcd}") ==
"1234abcd")
+ assert(sub.substitute("${sparkconf:some-random-string-abcd}") ==
"1234abcd")
+ assert(sub.substitute("${spark:some-random-string-abcd}") ==
"1234abcd")
+ assert(sub.substitute("${some-random-string-abcd}") == "1234abcd")
+}
}
test("multiple substitutes") {
val q = "select ${bar} ${foo} ${doo} this is great"
conf.setConfString("bar", "1")
conf.setConfString("foo", "2")
conf.setConfString("doo", "3")
--- End diff --
@gatorsmile .
Let's remove line 47 ~ 49.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18529: [SPARK-21304][SQL] remove unnecessary isNull vari...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18529#discussion_r125559540
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -588,12 +591,14 @@ case class MapObjects private(
case _ => genFunction.value
}
-val loopNullCheck = inputDataType match {
- case _: ArrayType => s"$loopIsNull =
${genInputData.value}.isNullAt($loopIndex);"
- // The element of primitive array will never be null.
- case ObjectType(cls) if cls.isArray &&
cls.getComponentType.isPrimitive =>
-s"$loopIsNull = false"
- case _ => s"$loopIsNull = $loopValue == null;"
+val loopNullCheck = if (loopIsNull != "false") {
+ ctx.addMutableState("boolean", loopIsNull, "")
+ inputDataType match {
+case _: ArrayType => s"$loopIsNull =
${genInputData.value}.isNullAt($loopIndex);"
+case _ => s"$loopIsNull = $loopValue == null;"
--- End diff --
for `Array[Int]`, because we calculate the `elementNullable` based on type,
so `elementNullable` will be false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18460: [SPARK-21247][SQL] Allow case-insensitive type equality ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18460 **[Test build #79192 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79192/testReport)** for PR 18460 at commit [`b41a6b4`](https://github.com/apache/spark/commit/b41a6b4acdde6ddbb2d1a016d5ef20ef867e1bcd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18534 **[Test build #79191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79191/testReport)** for PR 18534 at commit [`0c47cee`](https://github.com/apache/spark/commit/0c47ceef2cefb6f39ee8e040297329d8169da7df). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18534 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18534 **[Test build #79190 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79190/testReport)** for PR 18534 at commit [`f523149`](https://github.com/apache/spark/commit/f523149709f33c9bd805f24589f6651675cc6359). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18534 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79190/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18534 **[Test build #79190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79190/testReport)** for PR 18534 at commit [`f523149`](https://github.com/apache/spark/commit/f523149709f33c9bd805f24589f6651675cc6359). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17865 **[Test build #79189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79189/testReport)** for PR 17865 at commit [`9cc34c8`](https://github.com/apache/spark/commit/9cc34c895eaad5ba4366308a0324c8adb2f9510e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpark
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/18534 @yanboliang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18534: [SPARK-21310][ML][PySpark] Expose offset in PySpa...
GitHub user actuaryzhang opened a pull request: https://github.com/apache/spark/pull/18534 [SPARK-21310][ML][PySpark] Expose offset in PySpark ## What changes were proposed in this pull request? Add offset to PySpark in GLM as in #16699. ## How was this patch tested? Python test You can merge this pull request into a Git repository by running: $ git pull https://github.com/actuaryzhang/spark pythonOffset Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18534.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18534 commit f523149709f33c9bd805f24589f6651675cc6359 Author: actuaryzhang Date: 2017-07-05T05:33:02Z add offset to pyspark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17383: [SPARK-3165][MLlib][WIP] DecisionTree does not us...
Github user facaiy closed the pull request at: https://github.com/apache/spark/pull/17383 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18533: [SPARK-21308] [SQL] Remove SQLConf parameters from the o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18533 **[Test build #79188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79188/testReport)** for PR 18533 at commit [`092b414`](https://github.com/apache/spark/commit/092b41449a3fb12e32539f5733be49b2bdb9049f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18531: [SPARK-21307] [SQL] Remove SQLConf parameters from the p...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18531 **[Test build #79187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79187/testReport)** for PR 18531 at commit [`dda5e70`](https://github.com/apache/spark/commit/dda5e70bd9660e0bd853f24b72090900d7790d0b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18516: [SPARK-21281][SQL] Use string types by default if array ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18516 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79180/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18516: [SPARK-21281][SQL] Use string types by default if array ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18516 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18516: [SPARK-21281][SQL] Use string types by default if array ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18516 **[Test build #79180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79180/testReport)** for PR 18516 at commit [`d9c05a6`](https://github.com/apache/spark/commit/d9c05a6625d904113500fb55039ed9adae568e4e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18531: [SPARK-21307] [SQL] Remove SQLConf parameters from the p...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18531 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18531: [SPARK-21307] [SQL] Remove SQLConf parameters from the p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18531 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79185/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18416: [SPARK-21204][SQL] Add support for Scala Set coll...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18416#discussion_r125556076
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
---
@@ -498,6 +520,19 @@ object ScalaReflection extends ScalaReflection {
serializerFor(_, valueType, valuePath, seenTypeSet),
valueNullable = !valueType.typeSymbol.asClass.isPrimitive)
+ case t if t <:< localTypeOf[scala.collection.Set[_]] =>
+val TypeRef(_, _, Seq(elementType)) = t
+
+// There's no corresponding Catalyst type for `Set`, we serialize
a `Set` to Catalyst array.
+// Note that the property of `Set` is only kept when manipulating
the data as domain object.
+val newInput =
+ Invoke(
+ inputObject,
+ "toSeq",
+ ObjectType(classOf[Seq[_]]))
--- End diff --
For primitive, calling `toArray` can directly construct `UnsafeArrayData`
in `toCatalystArray`. However, `toArray` requires a `ClassTag`. To generate it
with a `StaticInvoke` might be hacky. So for now I simply use `toSeq`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18416: [SPARK-21204][SQL] Add support for Scala Set collection ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #79186 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79186/testReport)** for PR 18416 at commit [`4602689`](https://github.com/apache/spark/commit/46026890fb8831fcec741b9ca2df89dcbf622be4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18531: [SPARK-21307] [SQL] Remove SQLConf parameters from the p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18531 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18531: [SPARK-21307] [SQL] Remove SQLConf parameters from the p...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18531 **[Test build #79185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79185/testReport)** for PR 18531 at commit [`dda5e70`](https://github.com/apache/spark/commit/dda5e70bd9660e0bd853f24b72090900d7790d0b). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18529: [SPARK-21304][SQL] remove unnecessary isNull vari...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18529#discussion_r12875
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -588,12 +591,14 @@ case class MapObjects private(
case _ => genFunction.value
}
-val loopNullCheck = inputDataType match {
- case _: ArrayType => s"$loopIsNull =
${genInputData.value}.isNullAt($loopIndex);"
- // The element of primitive array will never be null.
- case ObjectType(cls) if cls.isArray &&
cls.getComponentType.isPrimitive =>
-s"$loopIsNull = false"
- case _ => s"$loopIsNull = $loopValue == null;"
+val loopNullCheck = if (loopIsNull != "false") {
+ ctx.addMutableState("boolean", loopIsNull, "")
+ inputDataType match {
+case _: ArrayType => s"$loopIsNull =
${genInputData.value}.isNullAt($loopIndex);"
+case _ => s"$loopIsNull = $loopValue == null;"
--- End diff --
Hmm, it depends on if we correctly set `elementNullable` when calling
`MapObjects.apply` for all the cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18529: [SPARK-21304][SQL] remove unnecessary isNull vari...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18529#discussion_r12516
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -588,12 +591,14 @@ case class MapObjects private(
case _ => genFunction.value
}
-val loopNullCheck = inputDataType match {
- case _: ArrayType => s"$loopIsNull =
${genInputData.value}.isNullAt($loopIndex);"
- // The element of primitive array will never be null.
- case ObjectType(cls) if cls.isArray &&
cls.getComponentType.isPrimitive =>
-s"$loopIsNull = false"
- case _ => s"$loopIsNull = $loopValue == null;"
+val loopNullCheck = if (loopIsNull != "false") {
+ ctx.addMutableState("boolean", loopIsNull, "")
+ inputDataType match {
+case _: ArrayType => s"$loopIsNull =
${genInputData.value}.isNullAt($loopIndex);"
+case _ => s"$loopIsNull = $loopValue == null;"
--- End diff --
Actually, I chose the example intentionally to be `loopIsNull != "false"`
(because `elementNullable` is `true` here), and we can hit the branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18416: [SPARK-21204][SQL] Add support for Scala Set coll...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18416#discussion_r12460
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
---
@@ -498,6 +520,31 @@ object ScalaReflection extends ScalaReflection {
serializerFor(_, valueType, valuePath, seenTypeSet),
valueNullable = !valueType.typeSymbol.asClass.isPrimitive)
+ case t if t <:< localTypeOf[scala.collection.Set[_]] =>
+val TypeRef(_, _, Seq(elementType)) = t
+val dt = dataTypeFor(elementType)
+
+// There's no corresponding Catalyst type for `Set`, we serialize
a `Set` to Catalyst array.
+// Note that the property of `Set` is only kept when manipulating
the data as domain object.
+val newInput =
+ Invoke(
+ inputObject,
+ "toSeq",
+ ObjectType(classOf[Seq[_]]))
+
+dt match {
--- End diff --
yap.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18416: [SPARK-21204][SQL] Add support for Scala Set coll...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18416#discussion_r12071
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -834,6 +834,153 @@ case class CollectObjectsToMap private(
}
}
+object CollectObjectsToSet {
+ private val curId = new java.util.concurrent.atomic.AtomicInteger()
+
+ /**
+ * Construct an instance of CollectObjectsToSet case class.
+ *
+ * @param function The function applied on the collection elements.
+ * @param inputData An expression that when evaluated returns a
collection object.
+ * @param collClass The type of the resulting collection.
+ */
+ def apply(
+ function: Expression => Expression,
+ inputData: Expression,
+ collClass: Class[_]): CollectObjectsToSet = {
+val id = curId.getAndIncrement()
+val loopValue = s"CollectObjectsToSet_loopValue$id"
+val loopIsNull = s"CollectObjectsToSet_loopIsNull$id"
+val arrayType = inputData.dataType.asInstanceOf[ArrayType]
+val loopVar = LambdaVariable(loopValue, loopIsNull,
arrayType.elementType)
+CollectObjectsToSet(
+ loopValue, loopIsNull, function(loopVar), inputData, collClass)
+ }
+}
+
+/**
+ * Expression used to convert a Catalyst Array to an external Scala `Set`.
+ * The collection is constructed using the associated builder, obtained by
calling `newBuilder`
+ * on the collection's companion object.
+ *
+ * Notice that when we convert a Catalyst array which contains duplicated
elements to an external
+ * Scala `Set`, the elements will be de-duplicated.
+ *
+ * @param loopValue the name of the loop variable that is used when
iterating over the value
+ * collection, and which is used as input for the
`lambdaFunction`
+ * @param loopIsNull the nullability of the loop variable that is used
when iterating over
+ *the value collection, and which is used as input
for the
+ *`lambdaFunction`
+ * @param lmbdaFunction A function that takes the `loopValue` as input,
and is used as
+ *a lambda function to handle collection
elements.
+ * @param inputData An expression that when evaluated returns an array
object.
+ * @param collClass The type of the resulting collection.
+ */
+case class CollectObjectsToSet private(
+loopValue: String,
+loopIsNull: String,
+lambdaFunction: Expression,
+inputData: Expression,
+collClass: Class[_]) extends Expression with NonSQLExpression {
+
+ override def nullable: Boolean = inputData.nullable
+
+ override def children: Seq[Expression] = lambdaFunction :: inputData ::
Nil
+
+ override def eval(input: InternalRow): Any =
+throw new UnsupportedOperationException("Only code-generated
evaluation is supported")
+
+ override def dataType: DataType = ObjectType(collClass)
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+// The data with PythonUserDefinedType are actually stored with the
data type of its sqlType.
+def inputDataType(dataType: DataType) = dataType match {
+ case p: PythonUserDefinedType => p.sqlType
+ case _ => dataType
+}
+
+val arrayType =
inputDataType(inputData.dataType).asInstanceOf[ArrayType]
+val loopValueJavaType = ctx.javaType(arrayType.elementType)
+if (arrayType.containsNull) {
+ ctx.addMutableState("boolean", loopIsNull, "")
+} else {
+ ctx.addMutableState("boolean", loopIsNull, s"$loopIsNull = false;")
--- End diff --
Now we reuse `MapObjects` for it and avoid the two new expressions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18531: [SPARK-21307] [SQL] Remove SQLConf parameters from the p...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18531 **[Test build #79185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79185/testReport)** for PR 18531 at commit [`dda5e70`](https://github.com/apache/spark/commit/dda5e70bd9660e0bd853f24b72090900d7790d0b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18531: [SPARK-21307] [SQL] Remove SQLConf parameters fro...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18531#discussion_r125554810
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -52,9 +53,11 @@ class SparkSqlParser(conf: SQLConf) extends
AbstractSqlParser {
/**
* Builder that converts an ANTLR ParseTree into a
LogicalPlan/Expression/TableIdentifier.
*/
-class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf) {
+class SparkSqlAstBuilder extends AstBuilder {
import org.apache.spark.sql.catalyst.parser.ParserUtils._
+ val conf = SQLConf.get
--- End diff --
Yes. That is the cause.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18416: [SPARK-21204][SQL] Add support for Scala Set coll...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18416#discussion_r125554796
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
---
@@ -498,6 +520,31 @@ object ScalaReflection extends ScalaReflection {
serializerFor(_, valueType, valuePath, seenTypeSet),
valueNullable = !valueType.typeSymbol.asClass.isPrimitive)
+ case t if t <:< localTypeOf[scala.collection.Set[_]] =>
+val TypeRef(_, _, Seq(elementType)) = t
+val dt = dataTypeFor(elementType)
+
+// There's no corresponding Catalyst type for `Set`, we serialize
a `Set` to Catalyst array.
+// Note that the property of `Set` is only kept when manipulating
the data as domain object.
+val newInput =
+ Invoke(
+ inputObject,
+ "toSeq",
+ ObjectType(classOf[Seq[_]]))
+
+dt match {
--- End diff --
shall we call `toCatalystArray` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16056 LGTM, pending test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79176/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18529 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18529 **[Test build #79176 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79176/testReport)** for PR 18529 at commit [`568fd76`](https://github.com/apache/spark/commit/568fd76b1d9f98613b304e5eedaeec63aeda3163). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18416: [SPARK-21204][SQL] Add support for Scala Set collection ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18416 **[Test build #79184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79184/testReport)** for PR 18416 at commit [`83db461`](https://github.com/apache/spark/commit/83db461d84765703e32b93dd2630eeab94a05908). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18533: [SPARK-21308] [SQL] Remove SQLConf parameters from the o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79178/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18533: [SPARK-21308] [SQL] Remove SQLConf parameters from the o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18533: [SPARK-21308] [SQL] Remove SQLConf parameters from the o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18533 **[Test build #79178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79178/testReport)** for PR 18533 at commit [`0a48ba3`](https://github.com/apache/spark/commit/0a48ba3a75519bf40eaf3c2c3e039e6f4264b323). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18511: [SPARK-21286][Test] Modified StorageTabSuite unit test
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79172/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18511: [SPARK-21286][Test] Modified StorageTabSuite unit test
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18511 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18511: [SPARK-21286][Test] Modified StorageTabSuite unit test
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18511 **[Test build #79172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79172/testReport)** for PR 18511 at commit [`301662b`](https://github.com/apache/spark/commit/301662b758bc808d2c0b84e3df0302c1e624b20d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18493: [SPARK-20889][SparkR][Followup] Clean up grouped doc for...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/18493 thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18493: [SPARK-20889][SparkR][Followup] Clean up grouped ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18493 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16056 **[Test build #79183 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79183/testReport)** for PR 16056 at commit [`b849b59`](https://github.com/apache/spark/commit/b849b59f03c824be0530565032154f12e5001c66). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18493: [SPARK-20889][SparkR][Followup] Clean up grouped doc for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79179/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18493: [SPARK-20889][SparkR][Followup] Clean up grouped doc for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18493 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18493: [SPARK-20889][SparkR][Followup] Clean up grouped doc for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18493 **[Test build #79179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79179/testReport)** for PR 18493 at commit [`821be85`](https://github.com/apache/spark/commit/821be85a40fc8cc6cb3eacfaeac8024a9fe676f7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/16056 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18529: [SPARK-21304][SQL] remove unnecessary isNull variable fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18529 **[Test build #79182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79182/testReport)** for PR 18529 at commit [`50126bd`](https://github.com/apache/spark/commit/50126bdd941588337ea1b4b25974935edc6c2d7b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #79181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79181/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). * This patch **fails Python style tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79181/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18516: [SPARK-21281][SQL] Use string types by default if array ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18516 **[Test build #79180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79180/testReport)** for PR 18516 at commit [`d9c05a6`](https://github.com/apache/spark/commit/d9c05a6625d904113500fb55039ed9adae568e4e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #79181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79181/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18530: [SPARK-20256][SQL][BRANCH-2.1] SessionState shoul...
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/18530 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18524: [SPARK-21300][SQL] ExternalMapToCatalyst should n...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18524 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18524: [SPARK-21300][SQL] ExternalMapToCatalyst should null-che...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18524 thanks, merging to master/2.2! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18174 LGTM, pending test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18524: [SPARK-21300][SQL] ExternalMapToCatalyst should null-che...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79171/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125548280
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle
writer.")
--- End diff --
OK,
I have modify it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18524: [SPARK-21300][SQL] ExternalMapToCatalyst should null-che...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18524 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18524: [SPARK-21300][SQL] ExternalMapToCatalyst should null-che...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18524 **[Test build #79171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79171/testReport)** for PR 18524 at commit [`0e8bf6e`](https://github.com/apache/spark/commit/0e8bf6e9f50616dd7b32ce51131718cc7bd30f16). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18174#discussion_r125548173
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -335,4 +335,29 @@ package object config {
"spark.")
.booleanConf
.createWithDefault(false)
+
+ private[spark] val SHUFFLE_FILE_BUFFER_SIZE =
+ConfigBuilder("spark.shuffle.file.buffer")
+ .doc("Used to set the buffer size of outputstream in shuffle
writer.")
+ .bytesConf(ByteUnit.KiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / 1024,
+s"The file buffer size must be greater than 0 to ${Int.MaxValue /
1024}.")
--- End diff --
thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
