[GitHub] spark pull request #17663: [SPARK-17647][SQL][FOLLOWUP][MINOR] fix typo
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17663 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17342 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17342 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75887/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #75887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75887/testReport)** for PR 17342 at commit [`0d30271`](https://github.com/apache/spark/commit/0d302717b85cdf2d4c35eebcb97795a456fd1bed). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17666 **[Test build #75888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75888/testReport)** for PR 17666 at commit [`539a9e8`](https://github.com/apache/spark/commit/539a9e82aa306b0f4a5fa0cd32121732e9e3ef15). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16989 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16989 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75883/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16989 **[Test build #75883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75883/testReport)** for PR 16989 at commit [`c63f39b`](https://github.com/apache/spark/commit/c63f39b755b0f9b66c61d96fb84368ddd6b0a6fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 @tgravescs , with the changes of history UI, REST API and web UI are now mixed. The base URL to list all the apps is through REST API. The key problem here is that in History Server we could have configured two ACLs (spark.acls.enable [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala#L299) and spark.history.ui.acls.enable [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L248)), this two ACLs checks different URLs, for example: ``` /api/v1/applications /api/v1/applications/ /api/v1/applications//logs ``` are controlled by `spark.acls.enable`. And web UI will use this API to list and show apps on the UI. If this is enabled, then only SHS owner, admin/view user groups could see the apps list on the SHS's base web UI. And all the other URLs (web UI and REST API) for application details are controlled by `spark.history.ui.acls.enable`. If we configured differently for this two ACLs, then we will get some unexpected behaviors. So what I here fixed is to unify the ACL and offer the right behavior. >No, there shouldn't be sensitive information there and many times a user is looking for a job run by say a headless user or other user. I guess you could filter only the jobs that user has acls to but that makes it more complicated. Do you have a concrete reason it should be protected? Note that this follow how other Hadoop UI's work. We worked with customers and they wish to filter and list apps based on the login users. But I don't have strong opinion it. I could change to what you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...
Github user witgo commented on the issue: https://github.com/apache/spark/pull/17480 @vanzin Sorry, I do not understand what you mean. Do you submit a new PR to your own ideas? If you can, I will close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on columns w...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17667 I just filed in https://issues.apache.org/jira/browse/SPARK-20364 and I cc'ed you both (I hope the ids were correct). We could close this for now maybe :). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17298 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75882/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17298 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17298 **[Test build #75882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75882/testReport)** for PR 17298 at commit [`2bfaa16`](https://github.com/apache/spark/commit/2bfaa16176be0d8a61062480fd3472a7c345e4b0). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17298 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75881/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17298 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17298 **[Test build #75881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75881/testReport)** for PR 17298 at commit [`955f163`](https://github.com/apache/spark/commit/955f16359bb5111fb2f41d0b29cbb7915f7a2bbf). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on columns w...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17667 Sure. Thank you for confirmation. Let me cc you in the JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17667 @HyukjinKwon thanks for looking at this! Please feel free to open a Jira so we can begin discussing a fix. I haven't started working on a patch yet, only have the test case at this point. So if you come up with a solution I'd love to see it! cc @robert3005 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17568: [SPARK-20254][SQL] Remove unnecessary data conver...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/17568#discussion_r111874759 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala --- @@ -96,3 +98,22 @@ object CombineTypedFilters extends Rule[LogicalPlan] { } } } + +/** + * Removes MapObjects when the following conditions are satisfied + * 1. Mapobject(e) where e is lambdavariable(), which means types for input output + * are primitive types + * 2. no custom collection class specified + * representation of data item. For example back to back map operations. + */ +object EliminateMapObjects extends Rule[LogicalPlan] { + def apply(plan: LogicalPlan): LogicalPlan = plan transform { +case DeserializeToObject(Invoke( +MapObjects(_, _, _, _ : LambdaVariable, inputData, None), --- End diff -- @cloud-fan Unfortunately, this change caused two test failures. `"checkAnswer should compare map correctly"` and `"SPARK-18717: code generation works for both scala.collection.Map and scala.collection.imutable.Map"` in `DatasetSuite` I will check what's happen very soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLowerCase ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17655 There is another issue in locale support in Avro serde. It is out of our controls. In the QueryTest, we already set the Locale to US. Thus, it covers DDLSuite and HiveDDLSuite. Thus, I do not think we should do any change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLow...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17655#discussion_r111873856 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -114,14 +114,14 @@ class SessionCatalog( * Format table name, taking into account case sensitivity. */ protected[this] def formatTableName(name: String): String = { -if (conf.caseSensitiveAnalysis) name else name.toLowerCase +if (conf.caseSensitiveAnalysis) name else name.toLowerCase(Locale.ROOT) --- End diff -- Yup, so, it seems the column names don't have such restrictions. I added the case you mentioned above in my comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLow...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17655#discussion_r111873718 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -114,14 +114,14 @@ class SessionCatalog( * Format table name, taking into account case sensitivity. */ protected[this] def formatTableName(name: String): String = { -if (conf.caseSensitiveAnalysis) name else name.toLowerCase +if (conf.caseSensitiveAnalysis) name else name.toLowerCase(Locale.ROOT) --- End diff -- Then, it works as expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLow...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17655#discussion_r111872932 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -114,14 +114,14 @@ class SessionCatalog( * Format table name, taking into account case sensitivity. */ protected[this] def formatTableName(name: String): String = { -if (conf.caseSensitiveAnalysis) name else name.toLowerCase +if (conf.caseSensitiveAnalysis) name else name.toLowerCase(Locale.ROOT) --- End diff -- Do you mean `` selectExpr(`ì`) `` via the parser? Yes, it seems throwing an exception as below: ```scala scala> Seq(1).toDF("ì").selectExpr("ì") org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'ì' expecting {'(', 'SELECT', 'FROM', 'ADD', 'AS', 'ALL', 'DISTINCT', 'WHERE', 'GROUP', 'BY', 'GROUPING', 'SETS', 'CUBE', 'ROLLUP', 'ORDER', 'HAVING', 'LIMIT', 'AT', 'OR', 'AND', 'IN', NOT, 'NO', 'EXISTS', 'BETWEEN', 'LIKE', RLIKE, 'IS', 'NULL', 'TRUE', 'FALSE', 'NULLS', 'ASC', 'DESC', 'FOR', 'INTERVAL', 'CASE', 'WHEN', 'THEN', 'ELSE', 'END', 'JOIN', 'CROSS', 'OUTER', 'INNER', 'LEFT', 'SEMI', 'RIGHT', 'FULL', 'NATURAL', 'ON', 'LATERAL', 'WINDOW', 'OVER', 'PARTITION', 'RANGE', 'ROWS', 'UNBOUNDED', 'PRECEDING', 'FOLLOWING', 'CURRENT', 'FIRST', 'AFTER', 'LAST', 'ROW', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'VIEW', 'REPLACE', 'INSERT', 'DELETE', 'INTO', 'DESCRIBE', 'EXPLAIN', 'FORMAT', 'LOGICAL', 'CODEGEN', 'COST', 'CAST', 'SHOW', 'TABLES', 'COLUMNS', 'COLUMN', 'USE', 'PARTITIONS', 'FUNCTIONS', 'DROP', 'UNION', 'EXCEPT', 'MINUS', 'INTERSECT', 'TO', 'TABLESAMPLE', 'STRATIFY', 'ALTER', 'RENAME', 'ARRAY', 'MAP', 'STRUCT', 'COMMENT', 'SET', 'RESET', 'DATA', 'START', 'TRA NSACTION', 'COMMIT', 'ROLLBACK', 'MACRO', 'IGNORE', 'IF', '+', '-', '*', 'DIV', '~', 'PERCENT', 'BUCKET', 'OUT', 'OF', 'SORT', 'CLUSTER', 'DISTRIBUTE', 'OVERWRITE', 'TRANSFORM', 'REDUCE', 'USING', 'SERDE', 'SERDEPROPERTIES', 'RECORDREADER', 'RECORDWRITER', 'DELIMITED', 'FIELDS', 'TERMINATED', 'COLLECTION', 'ITEMS', 'KEYS', 'ESCAPED', 'LINES', 'SEPARATED', 'FUNCTION', 'EXTENDED', 'REFRESH', 'CLEAR', 'CACHE', 'UNCACHE', 'LAZY', 'FORMATTED', 'GLOBAL', TEMPORARY, 'OPTIONS', 'UNSET', 'TBLPROPERTIES', 'DBPROPERTIES', 'BUCKETS', 'SKEWED', 'STORED', 'DIRECTORIES', 'LOCATION', 'EXCHANGE', 'ARCHIVE', 'UNARCHIVE', 'FILEFORMAT', 'TOUCH', 'COMPACT', 'CONCATENATE', 'CHANGE', 'CASCADE', 'RESTRICT', 'CLUSTERED', 'SORTED', 'PURGE', 'INPUTFORMAT', 'OUTPUTFORMAT', DATABASE, DATABASES, 'DFS', 'TRUNCATE', 'ANALYZE', 'COMPUTE', 'LIST', 'STATISTICS', 'PARTITIONED', 'EXTERNAL', 'DEFINED', 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'REPAIR', 'RECOVER', 'EXPORT', 'IMPORT', 'LOAD', 'ROLE', 'ROLES', 'COMPACT IONS', 'PRINCIPALS', 'TRANSACTIONS', 'INDEX', 'INDEXES', 'LOCKS', 'OPTION', 'ANTI', 'LOCAL', 'INPATH', 'CURRENT_DATE', 'CURRENT_TIMESTAMP', STRING, BIGINT_LITERAL, SMALLINT_LITERAL, TINYINT_LITERAL, INTEGER_VALUE, DECIMAL_VALUE, DOUBLE_LITERAL, BIGDECIMAL_LITERAL, IDENTIFIER, BACKQUOTED_IDENTIFIER}(line 1, pos 0) == SQL == ì ^^^ ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLowerCase ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17655 Are you going to then open another JIRA and PR (or a followup) to fix test cases related to Turkish locale bug? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17568: [SPARK-20254][SQL] Remove unnecessary data conver...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17568#discussion_r111872139 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala --- @@ -96,3 +98,22 @@ object CombineTypedFilters extends Rule[LogicalPlan] { } } } + +/** + * Removes MapObjects when the following conditions are satisfied + * 1. Mapobject(e) where e is lambdavariable(), which means types for input output + * are primitive types + * 2. no custom collection class specified + * representation of data item. For example back to back map operations. + */ +object EliminateMapObjects extends Rule[LogicalPlan] { + def apply(plan: LogicalPlan): LogicalPlan = plan transform { --- End diff -- shall we call `plan.transformAllExpressions`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75880/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLow...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17655#discussion_r111872008 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -114,14 +114,14 @@ class SessionCatalog( * Format table name, taking into account case sensitivity. */ protected[this] def formatTableName(name: String): String = { -if (conf.caseSensitiveAnalysis) name else name.toLowerCase +if (conf.caseSensitiveAnalysis) name else name.toLowerCase(Locale.ROOT) --- End diff -- Have you tried to use backticks to quote the column names? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17636 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLowerCase ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17655 This PR is not to fix the test cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17636 **[Test build #75880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75880/testReport)** for PR 17636 at commit [`55c64ca`](https://github.com/apache/spark/commit/55c64cad9b6691a8c2269f34e3867cc2d52dc1ed). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17660 In general, we can't decide whether a UDF is null-propagate or not, so we can't catch NPE for `Expression.eval` blindly. When we call `Expression.eval` in optimizer, we should make sure we do not pass unexpected input like `EliminateOuterJoin` did. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user weiqingy commented on a diff in the pull request: https://github.com/apache/spark/pull/17342#discussion_r111870688 --- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala --- @@ -1021,4 +1021,19 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging { secretKeys.foreach { key => assert(redactedConf(key) === Utils.REDACTION_REPLACEMENT_TEXT) } assert(redactedConf("spark.regular.property") === "not_a_secret") } + + test("SparkUrlStreamHandlerFactory") { +URL.setURLStreamHandlerFactory(new SparkUrlStreamHandlerFactory()) + +// if 'hdfs' is not supported, MalformedURLException will be thrown +new URL("hdfs://docs.oracle.com/test.jar") --- End diff -- The test also works without network connection. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #75887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75887/testReport)** for PR 17342 at commit [`0d30271`](https://github.com/apache/spark/commit/0d302717b85cdf2d4c35eebcb97795a456fd1bed). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #75886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75886/testReport)** for PR 17342 at commit [`7c2d61a`](https://github.com/apache/spark/commit/7c2d61a81d492aa1b259071e9c9af5a01320dbfb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17342: [SPARK-12868][SQL] Allow adding jars from hdfs
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #75885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75885/testReport)** for PR 17342 at commit [`be0257b`](https://github.com/apache/spark/commit/be0257b6e527e1fbe4b5f19991f20189f04ba426). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17495: [SPARK-20172][Core] Add file permission check when listi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17495 **[Test build #75884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75884/testReport)** for PR 17495 at commit [`4d3838a`](https://github.com/apache/spark/commit/4d3838a3050e849189d5cd48511f7c1bc891cd92). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17667 Do you mind if I open a JIRA and work on it if you are currently not @ash211? and also, could you confirm my investigation is correct and choose one of both ways if I was right @liancheng ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17667 I just checked the cause. It seems dot in the column names via `FilterAPI` tries to separate the field name with dot (`ColumnPath` with multiple column paths) whereas the actual column name is `col.dots`. I just tried several ways to solve this. One is simply to don't push down filters when there are dots in column names. The other way is override `FilterApi` so that we can get a field name via `ColumnPath.get` not `ColumnPath.fromDotString`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75877/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #75877 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75877/testReport)** for PR 15049 at commit [`9167eda`](https://github.com/apache/spark/commit/9167edab212a58324a65b9f700e8b09493466b25). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75878/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #75878 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75878/testReport)** for PR 15049 at commit [`e034695`](https://github.com/apache/spark/commit/e034695baabe0db5fb62b76001c53e4647a84845). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17660: [SPARK-20359][SQL] catch NPE in EliminateOuterJoin optim...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17660 I think the root problem is, in `EliminateOuterJoin.buildNewJoinType`, we always build `leftHasNonNullPredicate` and `rightHasNonNullPredicate`. If it's left join, only `rightHasNonNullPredicate` is used, and when building `leftHasNonNullPredicate`, we may pass null values to a UDF that is not supposed to run on null values. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17298 **[Test build #75882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75882/testReport)** for PR 17298 at commit [`2bfaa16`](https://github.com/apache/spark/commit/2bfaa16176be0d8a61062480fd3472a7c345e4b0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16989 **[Test build #75883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75883/testReport)** for PR 16989 at commit [`c63f39b`](https://github.com/apache/spark/commit/c63f39b755b0f9b66c61d96fb84368ddd6b0a6fd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17661: [SPARK-20349] [SQL] [REVERT-Branch2.1] ListFuncti...
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/17661 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17298 **[Test build #75881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75881/testReport)** for PR 17298 at commit [`955f163`](https://github.com/apache/spark/commit/955f16359bb5111fb2f41d0b29cbb7915f7a2bbf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17667 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75879/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17667 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17662: [SPARK-20360][PYTHON] Create _repr_html_ for SparkContex...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17662 I like it, I'm going to leave this a for a bit and see if anyone has any comments overnight :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17667 **[Test build #75879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75879/testReport)** for PR 17667 at commit [`51cb89b`](https://github.com/apache/spark/commit/51cb89b0587d06a6911e14f5a4381bed0b6da2eb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17636 **[Test build #75880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75880/testReport)** for PR 17636 at commit [`55c64ca`](https://github.com/apache/spark/commit/55c64cad9b6691a8c2269f34e3867cc2d52dc1ed). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/17636 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17588: [SPARK-20275][UI] Do not display "Completed" column for ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17588 Ping @srowen again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17666 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75876/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17666 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17666 **[Test build #75876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75876/testReport)** for PR 17666 at commit [`a611a13`](https://github.com/apache/spark/commit/a611a13ceef3b1f3431355a9caf1965bc5793e55). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedTableValuedFunction(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17663: [SPARK-17647][SQL][FOLLOWUP][MINOR] fix typo
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17663 got it. this was why I answered in details here too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17663: [SPARK-17647][SQL][FOLLOWUP][MINOR] fix typo
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17663 Yea, I just had to note this just in case someone refers this as a reference. Thanks for bearing with me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17495: [SPARK-20172][Core] Add file permission check whe...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/17495#discussion_r111860131 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -143,12 +156,26 @@ class FsHistoryProviderSuite extends SparkFunSuite with BeforeAndAfter with Matc SparkListenerApplicationStart("app1-2", Some("app1-2"), 1L, "test", None), SparkListenerApplicationEnd(2L) ) + +val path = new Path(logFile2.toURI) +val fs = path.getFileSystem(SparkHadoopUtil.get.conf) +val status = fs.getFileStatus(path) +SparkHadoopUtil.get.checkAccessPermission(status, FsAction.READ) should be (true) --- End diff -- Thanks @vanzin , I will update the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17636 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17636 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75875/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17636 **[Test build #75875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75875/testReport)** for PR 17636 at commit [`55c64ca`](https://github.com/apache/spark/commit/55c64cad9b6691a8c2269f34e3867cc2d52dc1ed). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17663: [SPARK-17647][SQL][FOLLOWUP][MINOR] fix typo
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17663 sure, I agree completely, but the reason I opened this PR was because this just went in today with this comment missed. I wouldn't suggest or try to fix individual typo otherwise --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17656: [SPARK-20354][CORE][REST-API]When I request access to th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17656 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17656: [SPARK-20354][CORE][REST-API]When I request access to th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17656 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75870/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17656: [SPARK-20354][CORE][REST-API]When I request access to th...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17656 **[Test build #75870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75870/testReport)** for PR 17656 at commit [`9ecc1a0`](https://github.com/apache/spark/commit/9ecc1a0257964acca6aabddcfc26cbcecf5086e8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17667 **[Test build #75879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75879/testReport)** for PR 17667 at commit [`51cb89b`](https://github.com/apache/spark/commit/51cb89b0587d06a6911e14f5a4381bed0b6da2eb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17667: Failing test for parquet predicate pushdown on do...
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/17667 Failing test for parquet predicate pushdown on dots with columns // checking against Jenkins to make sure this is still live on master You can merge this pull request into a Git repository by running: $ git pull https://github.com/ash211/spark test/parquet-dots-in-column2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17667.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17667 commit 51cb89b0587d06a6911e14f5a4381bed0b6da2eb Author: Andrew Ash Date: 2017-04-18T01:23:05Z Failing test for parquet predicate pushdown on dots with columns --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #75878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75878/testReport)** for PR 15049 at commit [`e034695`](https://github.com/apache/spark/commit/e034695baabe0db5fb62b76001c53e4647a84845). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15049 I simply changed the default to `true` to prevent behaviour change for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15049 I simply change the default to `true` to prevent behaviour change for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #75877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75877/testReport)** for PR 15049 at commit [`9167eda`](https://github.com/apache/spark/commit/9167edab212a58324a65b9f700e8b09493466b25). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17298 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17298 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75869/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17298: [SPARK-19094][WIP][PySpark] Plumb through logging for IJ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17298 **[Test build #75869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75869/testReport)** for PR 17298 at commit [`15d999b`](https://github.com/apache/spark/commit/15d999bb901aa0a0eef73ff50f2ba3d24c4d3f72). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support for arra...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16611 gentle ping @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17666 **[Test build #75876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75876/testReport)** for PR 17666 at commit [`a611a13`](https://github.com/apache/spark/commit/a611a13ceef3b1f3431355a9caf1965bc5793e55). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17666: [SPARK-20311][SQL] Support aliases for table valu...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/17666 [SPARK-20311][SQL] Support aliases for table value functions ## What changes were proposed in this pull request? This pr added parsing rules to support aliases in table value functions. ## How was this patch tested? Added tests in `PlanParserSuite`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark SPARK-20311 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17666.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17666 commit a611a13ceef3b1f3431355a9caf1965bc5793e55 Author: Takeshi Yamamuro Date: 2017-04-18T00:39:26Z Support aliases for table value functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17648: [SPARK-19851] Add support for EVERY and ANY (SOME) aggre...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17648 Can we just do a logical rewrite to turn them into "condA + condB + condC > 0" (for Some/Any) ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17657: [TEST][MINOR] Replace repartitionBy with distribu...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17657 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17664: Typo fix: distitrbuted -> distributed
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17664 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLowerCase ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17655 @gatorsmile, BTW, I just wonder if there test cases we should fix. If there are few, we might better fix them together here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17657: [TEST][MINOR] Replace repartitionBy with distribute in C...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17657 Merging in master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17661: [SPARK-20349] [SQL] [REVERT-Branch2.1] ListFunctions ret...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17661 Merging in branch-2.1. Can you close your PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17655: [SPARK-20156] [SQL] [FOLLOW-UP] Java String toLow...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17655#discussion_r111852553 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -114,14 +114,14 @@ class SessionCatalog( * Format table name, taking into account case sensitivity. */ protected[this] def formatTableName(name: String): String = { -if (conf.caseSensitiveAnalysis) name else name.toLowerCase +if (conf.caseSensitiveAnalysis) name else name.toLowerCase(Locale.ROOT) --- End diff -- I don't think column names have such restrictions. Assuming https://github.com/apache/spark/pull/7165, it seems we support other characters in column names. I can provide several cases that data becomes column names as below: ```scala scala> Seq("ì").toDF("a").groupBy("a").pivot("a").count().show() +---+---+ | a| ì| +---+---+ | ì| 1| +---+---+ ``` ```scala scala> import org.apache.spark.sql.functions import org.apache.spark.sql.functions scala> spark.range(1).select(functions.lit("ì")).show() +---+ | ì| +---+ | ì| +---+ ``` Seems parser does not allow such characters though. ```sql scala> sql("SELECT ì FROM tbl") org.apache.spark.sql.catalyst.parser.ParseException: no viable alternative at input 'SELECT ì'(line 1, pos 7) == SQL == SELECT ì FROM tbl ---^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:210) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:112) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:66) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:622) ... 48 elided ``` It seems we can still select ```scala scala> Seq("ì").toDF("a").groupBy("a").pivot("a").count().createOrReplaceTempView("tbl") scala> sql("SELECT *FROM tbl").show() +---+---+ | a| ì| +---+---+ | ì| 1| +---+---+ ``` If these were mistakenly supported, these should have the restrictions first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17664: Typo fix: distitrbuted -> distributed
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17664 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17665 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17665 **[Test build #75874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75874/testReport)** for PR 17665 at commit [`a47c9c0`](https://github.com/apache/spark/commit/a47c9c04f61dce38f64e291c66793742239761b7). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17665 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75874/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17636: [SPARK-20334][SQL] Return a better error message when co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17636 **[Test build #75875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75875/testReport)** for PR 17636 at commit [`55c64ca`](https://github.com/apache/spark/commit/55c64cad9b6691a8c2269f34e3867cc2d52dc1ed). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17665 **[Test build #75874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75874/testReport)** for PR 17665 at commit [`a47c9c0`](https://github.com/apache/spark/commit/a47c9c04f61dce38f64e291c66793742239761b7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17540 @rdblue You need to click the "consle output" tag on the left to see the pyspark failures. Here it is ``` ERROR: test_stream_exception (pyspark.sql.tests.SQLTests) -- Traceback (most recent call last): File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests.py", line 1438, in test_stream_exception sq.processAllAvailable() File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/streaming.py", line 149, in processAllAvailable return self._jsq.processAllAvailable() File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/utils.py", line 75, in deco raise StreamingQueryException(s.split(': ', 1)[1], stackTrace) StreamingQueryException: u'spark.sql.execution.id is already set\n=== Streaming Query ===\nIdentifier: query_explain [id = 568c91fc-0d6d-4fc8-a7d7-db94b6603c8a, runId = e646eee0-22c5-4620-9987-8a0d1c682476]\nCurrent Committed Offsets: {}\nCurrent Available Offsets: {FileStreamSource[file:/home/jenkins/workspace/SparkPullRequestBuilder/python/test_support/sql/streaming]: {"logOffset":0}}\n\nCurrent State: ACTIVE\nThread State: RUNNABLE\n\nLogical Plan:\nFileStreamSource[file:/home/jenkins/workspace/SparkPullRequestBuilder/python/test_support/sql/streaming]' ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17665 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75873/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17665 **[Test build #75873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75873/testReport)** for PR 17665 at commit [`4014b99`](https://github.com/apache/spark/commit/4014b99290829b1ba1662aff82dea514fb8fdb58). * This patch **fails Scala style tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17665 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17665: [SPARK-16742] Mesos Kerberos Support
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17665 **[Test build #75873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75873/testReport)** for PR 17665 at commit [`4014b99`](https://github.com/apache/spark/commit/4014b99290829b1ba1662aff82dea514fb8fdb58). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org