[jira] [Assigned] (SPARK-30119) Support pagination for spark streaming tab
[ https://issues.apache.org/jira/browse/SPARK-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30119: Assignee: (was: Apache Spark) > Support pagination for spark streaming tab > --- > > Key: SPARK-30119 > URL: https://issues.apache.org/jira/browse/SPARK-30119 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Support pagination for spark streaming tab -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-30119) Support pagination for spark streaming tab
[ https://issues.apache.org/jira/browse/SPARK-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30119: Assignee: Apache Spark > Support pagination for spark streaming tab > --- > > Key: SPARK-30119 > URL: https://issues.apache.org/jira/browse/SPARK-30119 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.1.0 >Reporter: jobit mathew >Assignee: Apache Spark >Priority: Minor > > Support pagination for spark streaming tab -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30119) Support pagination for spark streaming tab
[ https://issues.apache.org/jira/browse/SPARK-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098314#comment-17098314 ] Apache Spark commented on SPARK-30119: -- User 'iRakson' has created a pull request for this issue: https://github.com/apache/spark/pull/28439 > Support pagination for spark streaming tab > --- > > Key: SPARK-30119 > URL: https://issues.apache.org/jira/browse/SPARK-30119 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Support pagination for spark streaming tab -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30119) Support pagination for spark streaming tab
[ https://issues.apache.org/jira/browse/SPARK-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098315#comment-17098315 ] Apache Spark commented on SPARK-30119: -- User 'iRakson' has created a pull request for this issue: https://github.com/apache/spark/pull/28439 > Support pagination for spark streaming tab > --- > > Key: SPARK-30119 > URL: https://issues.apache.org/jira/browse/SPARK-30119 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Support pagination for spark streaming tab -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode
[ https://issues.apache.org/jira/browse/SPARK-31527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098319#comment-17098319 ] Apache Spark commented on SPARK-31527: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/28440 > date add/subtract interval only allow those day precision in ansi mode > -- > > Key: SPARK-31527 > URL: https://issues.apache.org/jira/browse/SPARK-31527 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 3.0.0 > > > Under ANSI mode, we should not allow date add interval with hours, minutes... > microseconds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31033) Support 'SHOW views' command
[ https://issues.apache.org/jira/browse/SPARK-31033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-31033. - Resolution: Duplicate > Support 'SHOW views' command > > > Key: SPARK-31033 > URL: https://issues.apache.org/jira/browse/SPARK-31033 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Priority: Major > Attachments: PowerBI.png > > > {noformat} > spark-sql> show views; > Error in query: > missing 'FUNCTIONS' at ''(line 1, pos 10) > == SQL == > show views > --^^^ > spark-sql> show views in default; > Error in query: > missing 'FUNCTIONS' at 'in'(line 1, pos 11) > == SQL == > show views in default > ---^^^ > {noformat} > Hive 3.1 support this command: > {noformat} > hive> select version(); > OK > 3.1.2 r8190d2be7b7165effa62bd21b7d60ef81fb0e4af > Time taken: 4.874 seconds, Fetched: 1 row(s) > hive> show views in default; > OK > Time taken: 0.01 seconds > hive> show views; > OK > Time taken: 0.013 seconds > hive> show views in default; > OK > Time taken: 0.013 seconds > {noformat} > *PowerBI* also need this command: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31627) Font style of Spark SQL DAG-viz is broken in Chrome on macOS
[ https://issues.apache.org/jira/browse/SPARK-31627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098389#comment-17098389 ] Kousuke Saruta commented on SPARK-31627: Thanks [~hyukjin.kwon] . I'll close this. > Font style of Spark SQL DAG-viz is broken in Chrome on macOS > > > Key: SPARK-31627 > URL: https://issues.apache.org/jira/browse/SPARK-31627 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.1.0 > Environment: * macOS > * Chrome 81 >Reporter: Kousuke Saruta >Priority: Minor > Attachments: font-weight-does-not-work.png > > > If all the following condition is true, font style of Spark SQL DAG-viz can > be broken. > More specifically, plan name will not be bold style if the plan is > WholeStageCodegen. > * macOS > * Chrome (version 81) > The current master uses Bootstrap4, which defines the default font family as > follows. > {code:java} > -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica > Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI > Emoji","Segoe UI Symbol","Noto Color Emoji" > {code} > If we use Chrome, BlinkMacSystemFont is used but font-weight property doesn't > work with the font when the font is used in SVG tags. > This issue is reported here > [here|https://bugs.chromium.org/p/chromium/issues/detail?id=1057654] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31627) Font style of Spark SQL DAG-viz is broken in Chrome on macOS
[ https://issues.apache.org/jira/browse/SPARK-31627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-31627. Resolution: Won't Fix > Font style of Spark SQL DAG-viz is broken in Chrome on macOS > > > Key: SPARK-31627 > URL: https://issues.apache.org/jira/browse/SPARK-31627 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.1.0 > Environment: * macOS > * Chrome 81 >Reporter: Kousuke Saruta >Priority: Minor > Attachments: font-weight-does-not-work.png > > > If all the following condition is true, font style of Spark SQL DAG-viz can > be broken. > More specifically, plan name will not be bold style if the plan is > WholeStageCodegen. > * macOS > * Chrome (version 81) > The current master uses Bootstrap4, which defines the default font family as > follows. > {code:java} > -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica > Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI > Emoji","Segoe UI Symbol","Noto Color Emoji" > {code} > If we use Chrome, BlinkMacSystemFont is used but font-weight property doesn't > work with the font when the font is used in SVG tags. > This issue is reported here > [here|https://bugs.chromium.org/p/chromium/issues/detail?id=1057654] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31629) "py4j.protocol.Py4JJavaError: An error occurred while calling o90.save" in pyspark 2.3.1
appleyuchi created SPARK-31629: -- Summary: "py4j.protocol.Py4JJavaError: An error occurred while calling o90.save" in pyspark 2.3.1 Key: SPARK-31629 URL: https://issues.apache.org/jira/browse/SPARK-31629 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 2.3.1 Environment: Ubuntu19.10 anaconda3-python3.6.10 scala 2.11.8 apache-hive-3.0.0-bin hadoop-2.7.7 spark-2.3.1-bin-hadoop2.7 java version "1.8.0_131" Mysql Server version: 8.0.19-0ubuntu0.19.10.3 (Ubuntu) driver:mysql-connector-java-8.0.20.jar [Driver link|[https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.20]] Reporter: appleyuchi Fix For: 1.5.0, 1.4.1 I have search the forum, [SPARK-8365|https://issues.apache.org/jira/browse/SPARK-8365] mentioned the same issue in spark 1.4.0 [SPARK-8368|https://issues.apache.org/jira/browse/SPARK-8368] fix it in spark 1.4.1 1.5.0 However,in spark 2.3.1,this bug occur again Please help me ,thanks~!!! #-- test.py [https://paste.ubuntu.com/p/HJfbcQ2zq3/] running method is: spark-submit --master yarn --deploy-mode cluster test.py then I got: Traceback (most recent call last): File "test.py", line 45, in password="appleyuchi").mode('append').save() File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/pyspark.zip/pyspark/sql/readwriter.py", line 703, in save File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/pyspark.zip/pyspark/sql/utils.py", line 63, in deco File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value *py4j.protocol.Py4JJavaError: An error occurred while calling o90.save.* : java.sql.SQLSyntaxErrorException: Unknown database 'leaf' at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:456) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197) at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:63) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654) at org.apache.spark.sql.DataFrameWriter
[jira] [Updated] (SPARK-31629) "py4j.protocol.Py4JJavaError: An error occurred while calling o90.save" in pyspark 2.3.1
[ https://issues.apache.org/jira/browse/SPARK-31629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] appleyuchi updated SPARK-31629: --- Environment: Ubuntu19.10 anaconda3-python3.6.10 scala 2.11.8 apache-hive-3.0.0-bin hadoop-2.7.7 spark-2.3.1-bin-hadoop2.7 java version "1.8.0_131" Mysql Server version: 8.0.19-0ubuntu0.19.10.3 (Ubuntu) driver:mysql-connector-java-8.0.20.jar [Driver link|[https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.20]] was: Ubuntu19.10 anaconda3-python3.6.10 scala 2.11.8 apache-hive-3.0.0-bin hadoop-2.7.7 spark-2.3.1-bin-hadoop2.7 java version "1.8.0_131" Mysql Server version: 8.0.19-0ubuntu0.19.10.3 (Ubuntu) driver:mysql-connector-java-8.0.20.jar [Driver link|[https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.20]] > "py4j.protocol.Py4JJavaError: An error occurred while calling o90.save" in > pyspark 2.3.1 > > > Key: SPARK-31629 > URL: https://issues.apache.org/jira/browse/SPARK-31629 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.3.1 > Environment: Ubuntu19.10 > > anaconda3-python3.6.10 > > scala 2.11.8 > > apache-hive-3.0.0-bin > > hadoop-2.7.7 > > spark-2.3.1-bin-hadoop2.7 > > java version "1.8.0_131" > > Mysql Server version: 8.0.19-0ubuntu0.19.10.3 (Ubuntu) > > driver:mysql-connector-java-8.0.20.jar > > [Driver > link|[https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.20]] > >Reporter: appleyuchi >Priority: Critical > Fix For: 1.4.1, 1.5.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > I have search the forum, > [SPARK-8365|https://issues.apache.org/jira/browse/SPARK-8365] > mentioned the same issue in spark 1.4.0 > [SPARK-8368|https://issues.apache.org/jira/browse/SPARK-8368] > fix it in spark 1.4.1 1.5.0 > > However,in spark 2.3.1,this bug occur again > Please help me ,thanks~!!! > #-- > test.py > [https://paste.ubuntu.com/p/HJfbcQ2zq3/] > > running method is: > spark-submit --master yarn --deploy-mode cluster test.py > > then I got: > Traceback (most recent call last): > File "test.py", line 45, in > password="appleyuchi").mode('append').save() > File > "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/pyspark.zip/pyspark/sql/readwriter.py", > line 703, in save > File > "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/py4j-0.10.7-src.zip/py4j/java_gateway.py", > line 1257, in __call__ > File > "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/pyspark.zip/pyspark/sql/utils.py", > line 63, in deco > File > "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/py4j-0.10.7-src.zip/py4j/protocol.py", > line 328, in get_return_value > *py4j.protocol.Py4JJavaError: An error occurred while calling o90.save.* > : java.sql.SQLSyntaxErrorException: Unknown database 'leaf' > at > com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) > at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) > at > com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) > at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836) > at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:456) > at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) > at > com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197) > at > org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45) > at > org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63) > at > org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54) > at > org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:63) > at > org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) > at > org.apache.spark.sql.executi
[jira] [Created] (SPARK-31630) Skip timestamp rebasing after 1900-01-01
Maxim Gekk created SPARK-31630: -- Summary: Skip timestamp rebasing after 1900-01-01 Key: SPARK-31630 URL: https://issues.apache.org/jira/browse/SPARK-31630 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.0.0, 3.1.0 Reporter: Maxim Gekk The conversions of Catalyst's DATE/TIMESTAMPS to/from Java's types java.sql.Date/java.sql.Timestamps have almost the same implementation except addition rebasing op. If we look at switch and diffs arrays of all available time zones, we can detect that there is a time point when all diffs are 0. This is 1900-01-01 00:00:00Z. So, we can compare input micros with the time point and skip conversion for modern timestamps. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31629) "py4j.protocol.Py4JJavaError: An error occurred while calling o90.save" in pyspark 2.3.1
[ https://issues.apache.org/jira/browse/SPARK-31629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] appleyuchi updated SPARK-31629: --- Description: I have search the forum, SPARK-8365 mentioned the same issue in spark 1.4.0 SPARK-8368 fix it in spark 1.4.1 1.5.0 However,in spark 2.3.1,this bug occur again Please help me ,thanks~!!! #-- test.py [https://paste.ubuntu.com/p/HJfbcQ2zq3/] running method is: ①spark-submit --master yarn --deploy-mode cluster test.py ②pyspark --master yarn( and then paste the code above) each method can replicate this error. then I got: Traceback (most recent call last): File "test.py", line 45, in password="appleyuchi").mode('append').save() File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/pyspark.zip/pyspark/sql/readwriter.py", line 703, in save File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/pyspark.zip/pyspark/sql/utils.py", line 63, in deco File "/home/appleyuchi/bigdata/hadoop_tmp/nm-local-dir/usercache/appleyuchi/appcache/application_1588504345289_0003/container_1588504345289_0003_01_01/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value {color:#FF}*py4j.protocol.Py4JJavaError: An error occurred while calling o90.save.*{color} : java.sql.SQLSyntaxErrorException: Unknown database 'leaf'*{color:#FF}(I'm sure this database exists){color}* at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:456) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197) at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:63) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:54) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:63) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngin
[jira] [Commented] (SPARK-31630) Skip timestamp rebasing after 1900-01-01
[ https://issues.apache.org/jira/browse/SPARK-31630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098414#comment-17098414 ] Apache Spark commented on SPARK-31630: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/28441 > Skip timestamp rebasing after 1900-01-01 > > > Key: SPARK-31630 > URL: https://issues.apache.org/jira/browse/SPARK-31630 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > The conversions of Catalyst's DATE/TIMESTAMPS to/from Java's types > java.sql.Date/java.sql.Timestamps have almost the same implementation except > addition rebasing op. If we look at switch and diffs arrays of all available > time zones, we can detect that there is a time point when all diffs are 0. > This is 1900-01-01 00:00:00Z. So, we can compare input micros with the time > point and skip conversion for modern timestamps. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31630) Skip timestamp rebasing after 1900-01-01
[ https://issues.apache.org/jira/browse/SPARK-31630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31630: Assignee: Apache Spark > Skip timestamp rebasing after 1900-01-01 > > > Key: SPARK-31630 > URL: https://issues.apache.org/jira/browse/SPARK-31630 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Maxim Gekk >Assignee: Apache Spark >Priority: Major > > The conversions of Catalyst's DATE/TIMESTAMPS to/from Java's types > java.sql.Date/java.sql.Timestamps have almost the same implementation except > addition rebasing op. If we look at switch and diffs arrays of all available > time zones, we can detect that there is a time point when all diffs are 0. > This is 1900-01-01 00:00:00Z. So, we can compare input micros with the time > point and skip conversion for modern timestamps. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31630) Skip timestamp rebasing after 1900-01-01
[ https://issues.apache.org/jira/browse/SPARK-31630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31630: Assignee: (was: Apache Spark) > Skip timestamp rebasing after 1900-01-01 > > > Key: SPARK-31630 > URL: https://issues.apache.org/jira/browse/SPARK-31630 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > The conversions of Catalyst's DATE/TIMESTAMPS to/from Java's types > java.sql.Date/java.sql.Timestamps have almost the same implementation except > addition rebasing op. If we look at switch and diffs arrays of all available > time zones, we can detect that there is a time point when all diffs are 0. > This is 1900-01-01 00:00:00Z. So, we can compare input micros with the time > point and skip conversion for modern timestamps. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode
[ https://issues.apache.org/jira/browse/SPARK-31527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098436#comment-17098436 ] Apache Spark commented on SPARK-31527: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28369 > date add/subtract interval only allow those day precision in ansi mode > -- > > Key: SPARK-31527 > URL: https://issues.apache.org/jira/browse/SPARK-31527 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 3.0.0 > > > Under ANSI mode, we should not allow date add interval with hours, minutes... > microseconds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13
[ https://issues.apache.org/jira/browse/SPARK-29292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098448#comment-17098448 ] Sean R. Owen commented on SPARK-29292: -- I think the issue is that if you import scala.collection.Seq, then yes you keep the type, but unless callers do the same thing, you end up not being able to write simple things in 2.13 like {{val mySeq: Seq[...] = spark.somethingThatReturnsSeq()}} because Spark is returning a collection.Seq, not an immutable Seq. It's viable, just think it looked like a lot more change for callers. There was more discussion at https://issues.apache.org/jira/browse/SPARK-27681 > Fix internal usages of mutable collection for Seq in 2.13 > - > > Key: SPARK-29292 > URL: https://issues.apache.org/jira/browse/SPARK-29292 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Sean R. Owen >Assignee: Sean R. Owen >Priority: Minor > > Kind of related to https://issues.apache.org/jira/browse/SPARK-27681, but a > simpler subset. > In 2.13, a mutable collection can't be returned as a > {{scala.collection.Seq}}. It's easy enough to call .toSeq on these as that > still works on 2.12. > {code} > [ERROR] [Error] > /Users/seanowen/Documents/spark_2.13/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:467: > type mismatch; > found : Seq[String] (in scala.collection) > required: Seq[String] (in scala.collection.immutable) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13
[ https://issues.apache.org/jira/browse/SPARK-29292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098459#comment-17098459 ] Guillaume Martres commented on SPARK-29292: --- Right, so ideally you'd change Seq to collection.Seq for parameter types but not for result type, but that might be tricky to manage and it looks like that was already considered and dismissed in the discussion you linked to. Going back to the issue with copying when using toSeq everywhere, one way to reduce that would be to use builders when possible: instead of creating an ArrayBuffer, filling it, then copying using toSeq, one can create a `Seq.newBuilder`, fill it, then calll `result` on it to get back an immutable Seq without copying. (This might be worse than using ArrayBuffer in 2.12, because the default builder will construct a List, ideally one would use immutable.ArraySeq.newBuilder to get something backed by a plain Array but that one doesn't exist in 2.12). If most of the collections being copied are small then this might not make a significant difference any way. > Fix internal usages of mutable collection for Seq in 2.13 > - > > Key: SPARK-29292 > URL: https://issues.apache.org/jira/browse/SPARK-29292 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Sean R. Owen >Assignee: Sean R. Owen >Priority: Minor > > Kind of related to https://issues.apache.org/jira/browse/SPARK-27681, but a > simpler subset. > In 2.13, a mutable collection can't be returned as a > {{scala.collection.Seq}}. It's easy enough to call .toSeq on these as that > still works on 2.12. > {code} > [ERROR] [Error] > /Users/seanowen/Documents/spark_2.13/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:467: > type mismatch; > found : Seq[String] (in scala.collection) > required: Seq[String] (in scala.collection.immutable) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31631) Fix test flakiness caused by MiniKdc which throws "address in use" BindException
Kent Yao created SPARK-31631: Summary: Fix test flakiness caused by MiniKdc which throws "address in use" BindException Key: SPARK-31631 URL: https://issues.apache.org/jira/browse/SPARK-31631 Project: Spark Issue Type: Bug Components: Tests Affects Versions: 3.0.0, 3.1.0 Reporter: Kent Yao {code:java} [info] org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite *** ABORTED *** (15 seconds, 426 milliseconds) [info] java.net.BindException: Address already in use [info] at sun.nio.ch.Net.bind0(Native Method) [info] at sun.nio.ch.Net.bind(Net.java:433) [info] at sun.nio.ch.Net.bind(Net.java:425) [info] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) [info] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) [info] at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) [info] at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) [info] at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) [info] at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) [info] at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) [info] at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [info] at java.lang.Thread.run(Thread.java:748) {code} This is an issue fixed in hadoop 2.8.0 https://issues.apache.org/jira/browse/HADOOP-12656 We may need apply the approach from HBASE first before we drop Hadoop 2.7.x https://issues.apache.org/jira/browse/HBASE-14734 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31631) Fix test flakiness caused by MiniKdc which throws "address in use" BindException
[ https://issues.apache.org/jira/browse/SPARK-31631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098488#comment-17098488 ] Apache Spark commented on SPARK-31631: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28442 > Fix test flakiness caused by MiniKdc which throws "address in use" > BindException > > > Key: SPARK-31631 > URL: https://issues.apache.org/jira/browse/SPARK-31631 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Major > > {code:java} > [info] org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite *** ABORTED > *** (15 seconds, 426 milliseconds) > [info] java.net.BindException: Address already in use > [info] at sun.nio.ch.Net.bind0(Native Method) > [info] at sun.nio.ch.Net.bind(Net.java:433) > [info] at sun.nio.ch.Net.bind(Net.java:425) > [info] at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > [info] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > [info] at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > [info] at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > [info] at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > [info] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [info] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [info] at java.lang.Thread.run(Thread.java:748) > {code} > This is an issue fixed in hadoop 2.8.0 > https://issues.apache.org/jira/browse/HADOOP-12656 > We may need apply the approach from HBASE first before we drop Hadoop 2.7.x > https://issues.apache.org/jira/browse/HBASE-14734 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31631) Fix test flakiness caused by MiniKdc which throws "address in use" BindException
[ https://issues.apache.org/jira/browse/SPARK-31631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31631: Assignee: Apache Spark > Fix test flakiness caused by MiniKdc which throws "address in use" > BindException > > > Key: SPARK-31631 > URL: https://issues.apache.org/jira/browse/SPARK-31631 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Apache Spark >Priority: Major > > {code:java} > [info] org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite *** ABORTED > *** (15 seconds, 426 milliseconds) > [info] java.net.BindException: Address already in use > [info] at sun.nio.ch.Net.bind0(Native Method) > [info] at sun.nio.ch.Net.bind(Net.java:433) > [info] at sun.nio.ch.Net.bind(Net.java:425) > [info] at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > [info] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > [info] at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > [info] at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > [info] at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > [info] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [info] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [info] at java.lang.Thread.run(Thread.java:748) > {code} > This is an issue fixed in hadoop 2.8.0 > https://issues.apache.org/jira/browse/HADOOP-12656 > We may need apply the approach from HBASE first before we drop Hadoop 2.7.x > https://issues.apache.org/jira/browse/HBASE-14734 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31631) Fix test flakiness caused by MiniKdc which throws "address in use" BindException
[ https://issues.apache.org/jira/browse/SPARK-31631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31631: Assignee: (was: Apache Spark) > Fix test flakiness caused by MiniKdc which throws "address in use" > BindException > > > Key: SPARK-31631 > URL: https://issues.apache.org/jira/browse/SPARK-31631 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Major > > {code:java} > [info] org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite *** ABORTED > *** (15 seconds, 426 milliseconds) > [info] java.net.BindException: Address already in use > [info] at sun.nio.ch.Net.bind0(Native Method) > [info] at sun.nio.ch.Net.bind(Net.java:433) > [info] at sun.nio.ch.Net.bind(Net.java:425) > [info] at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > [info] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > [info] at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > [info] at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > [info] at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > [info] at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > [info] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [info] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [info] at java.lang.Thread.run(Thread.java:748) > {code} > This is an issue fixed in hadoop 2.8.0 > https://issues.apache.org/jira/browse/HADOOP-12656 > We may need apply the approach from HBASE first before we drop Hadoop 2.7.x > https://issues.apache.org/jira/browse/HBASE-14734 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode
[ https://issues.apache.org/jira/browse/SPARK-31527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098530#comment-17098530 ] Apache Spark commented on SPARK-31527: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28310 > date add/subtract interval only allow those day precision in ansi mode > -- > > Key: SPARK-31527 > URL: https://issues.apache.org/jira/browse/SPARK-31527 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 3.0.0 > > > Under ANSI mode, we should not allow date add interval with hours, minutes... > microseconds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode
[ https://issues.apache.org/jira/browse/SPARK-31527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098531#comment-17098531 ] Apache Spark commented on SPARK-31527: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28310 > date add/subtract interval only allow those day precision in ansi mode > -- > > Key: SPARK-31527 > URL: https://issues.apache.org/jira/browse/SPARK-31527 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 3.0.0 > > > Under ANSI mode, we should not allow date add interval with hours, minutes... > microseconds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098547#comment-17098547 ] Apache Spark commented on SPARK-31212: -- User 'tianshizz' has created a pull request for this issue: https://github.com/apache/spark/pull/28443 > Failure of casting the '1000-02-29' string to the date type > --- > > Key: SPARK-31212 > URL: https://issues.apache.org/jira/browse/SPARK-31212 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 >Reporter: Maxim Gekk >Priority: Major > > The '1000-02-29' is valid date in the Julian calendar used in Spark 2.4.5 for > dates before 1582-10-15 but casting the string to the date type fails: > {code:scala} > scala> val df = > Seq("1000-02-29").toDF("dateS").select($"dateS".cast("date").as("date")) > df: org.apache.spark.sql.DataFrame = [date: date] > scala> df.show > ++ > |date| > ++ > |null| > ++ > {code} > Creating a dataset from java.sql.Date w/ the same input string works > correctly: > {code:scala} > scala> val df2 = > Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date")) > df2: org.apache.spark.sql.DataFrame = [date: date] > scala> df2.show > +--+ > | date| > +--+ > |1000-02-29| > +--+ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31212: Assignee: (was: Apache Spark) > Failure of casting the '1000-02-29' string to the date type > --- > > Key: SPARK-31212 > URL: https://issues.apache.org/jira/browse/SPARK-31212 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 >Reporter: Maxim Gekk >Priority: Major > > The '1000-02-29' is valid date in the Julian calendar used in Spark 2.4.5 for > dates before 1582-10-15 but casting the string to the date type fails: > {code:scala} > scala> val df = > Seq("1000-02-29").toDF("dateS").select($"dateS".cast("date").as("date")) > df: org.apache.spark.sql.DataFrame = [date: date] > scala> df.show > ++ > |date| > ++ > |null| > ++ > {code} > Creating a dataset from java.sql.Date w/ the same input string works > correctly: > {code:scala} > scala> val df2 = > Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date")) > df2: org.apache.spark.sql.DataFrame = [date: date] > scala> df2.show > +--+ > | date| > +--+ > |1000-02-29| > +--+ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31212: Assignee: Apache Spark > Failure of casting the '1000-02-29' string to the date type > --- > > Key: SPARK-31212 > URL: https://issues.apache.org/jira/browse/SPARK-31212 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 >Reporter: Maxim Gekk >Assignee: Apache Spark >Priority: Major > > The '1000-02-29' is valid date in the Julian calendar used in Spark 2.4.5 for > dates before 1582-10-15 but casting the string to the date type fails: > {code:scala} > scala> val df = > Seq("1000-02-29").toDF("dateS").select($"dateS".cast("date").as("date")) > df: org.apache.spark.sql.DataFrame = [date: date] > scala> df.show > ++ > |date| > ++ > |null| > ++ > {code} > Creating a dataset from java.sql.Date w/ the same input string works > correctly: > {code:scala} > scala> val df2 = > Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date")) > df2: org.apache.spark.sql.DataFrame = [date: date] > scala> df2.show > +--+ > | date| > +--+ > |1000-02-29| > +--+ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098548#comment-17098548 ] Apache Spark commented on SPARK-31212: -- User 'tianshizz' has created a pull request for this issue: https://github.com/apache/spark/pull/28443 > Failure of casting the '1000-02-29' string to the date type > --- > > Key: SPARK-31212 > URL: https://issues.apache.org/jira/browse/SPARK-31212 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 >Reporter: Maxim Gekk >Priority: Major > > The '1000-02-29' is valid date in the Julian calendar used in Spark 2.4.5 for > dates before 1582-10-15 but casting the string to the date type fails: > {code:scala} > scala> val df = > Seq("1000-02-29").toDF("dateS").select($"dateS".cast("date").as("date")) > df: org.apache.spark.sql.DataFrame = [date: date] > scala> df.show > ++ > |date| > ++ > |null| > ++ > {code} > Creating a dataset from java.sql.Date w/ the same input string works > correctly: > {code:scala} > scala> val df2 = > Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date")) > df2: org.apache.spark.sql.DataFrame = [date: date] > scala> df2.show > +--+ > | date| > +--+ > |1000-02-29| > +--+ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31632) The ApplicationInfo in KVStore may be accessed before it's prepared
Xingcan Cui created SPARK-31632: --- Summary: The ApplicationInfo in KVStore may be accessed before it's prepared Key: SPARK-31632 URL: https://issues.apache.org/jira/browse/SPARK-31632 Project: Spark Issue Type: Bug Components: Spark Core, Web UI Affects Versions: 3.0.0 Reporter: Xingcan Cui While starting some local tests, I occasionally encountered the following exceptions for Web UI. {noformat} 23:00:29.845 WARN org.eclipse.jetty.server.HttpChannel: /jobs/ java.util.NoSuchElementException at java.util.Collections$EmptyIterator.next(Collections.java:4191) at org.apache.spark.util.kvstore.InMemoryStore$InMemoryIterator.next(InMemoryStore.java:467) at org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:39) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:266) at org.apache.spark.ui.WebUI.$anonfun$attachPage$1(WebUI.scala:89) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:80) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Thread.java:748){noformat} *Reason* That is because {{AppStatusStore.applicationInfo()}} accesses an empty view (iterator) returned by {{InMemoryStore}}. AppStatusStore {code:java} def applicationInfo(): v1.ApplicationInfo = { store.view(classOf[ApplicationInfoWrapper]).max(1).iterator().next().info } {code} InMemoryStore {code:java} public KVStoreView view(Class type){ InstanceList list = inMemoryLists.get(type); return list != null ? list.view() : emptyView(); } {code} During the initialization of {{SparkContext}}, it first starts the Web UI (SparkContext: L475 _ui.foreach(_.bind())) and then setup the {{LiveListenerBus}} thread (SparkContext: L608 {{setupAndStartListenerBus()}}) for dispatching the {{SparkListenerApplicationStart}} event (which will trigger writing the requested {{ApplicationInfo}} to {{InMemoryStore}}). *Solution* Since the {{applicationInfo()}} method is expected to always return a valid {{ApplicationInfo}}, maybe we can add a while-loop-check here to guarantee the availability of {{ApplicationInfo}}. {code:java} def applicationInfo(): v1.ApplicationInfo = { var iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() while (!iterator.hasNext){ Thread.sleep(20) iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() } iterator.next().info } {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31632) The ApplicationInfo in KVStore may be accessed before it's prepared
[ https://issues.apache.org/jira/browse/SPARK-31632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31632: Assignee: (was: Apache Spark) > The ApplicationInfo in KVStore may be accessed before it's prepared > --- > > Key: SPARK-31632 > URL: https://issues.apache.org/jira/browse/SPARK-31632 > Project: Spark > Issue Type: Bug > Components: Spark Core, Web UI >Affects Versions: 3.0.0 >Reporter: Xingcan Cui >Priority: Minor > > While starting some local tests, I occasionally encountered the following > exceptions for Web UI. > {noformat} > 23:00:29.845 WARN org.eclipse.jetty.server.HttpChannel: /jobs/ > java.util.NoSuchElementException > at java.util.Collections$EmptyIterator.next(Collections.java:4191) > at > org.apache.spark.util.kvstore.InMemoryStore$InMemoryIterator.next(InMemoryStore.java:467) > at > org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:39) > at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:266) > at org.apache.spark.ui.WebUI.$anonfun$attachPage$1(WebUI.scala:89) > at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:80) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) > at > org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:505) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) > at java.lang.Thread.run(Thread.java:748){noformat} > *Reason* > That is because {{AppStatusStore.applicationInfo()}} accesses an empty view > (iterator) returned by {{InMemoryStore}}. > AppStatusStore > {code:java} > def applicationInfo(): v1.ApplicationInfo = { > store.view(classOf[ApplicationInfoWrapper]).max(1).iterator().next().info > } > {code} > InMemoryStore > {code:java} > public KVStoreView view(Class type){ > InstanceList list = inMemoryLists.get(type); > return list != null ? list.view() : emptyView(); > } > {code} > During the initialization of {{SparkContext}}, it first starts the Web UI > (SparkContext: L475 _ui.foreach(_.bind())) and then setup the > {{LiveListenerBus}} thread (SparkContext: L608 > {{setupAndStartListenerBus()}}) for dispatching the > {{SparkListenerApplicationStart}} event (which will trigger writing the > requested {{ApplicationInfo}} to {{InMemoryStore}}). > *Solution* > Since the {{applicationInfo()}} method is expected to always return a valid > {{ApplicationInfo}}, maybe we can add a while-loop-check here to guarantee > the availability of {{ApplicationInfo}}. > {code:java} > def applicationInfo(): v1.ApplicationInfo = { > var iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() > while (!iterator.hasNext){ > Thread.sleep(20) > iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() > } > iterator.next().info > } > {code} -- This message was sent by
[jira] [Commented] (SPARK-31632) The ApplicationInfo in KVStore may be accessed before it's prepared
[ https://issues.apache.org/jira/browse/SPARK-31632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098590#comment-17098590 ] Apache Spark commented on SPARK-31632: -- User 'xccui' has created a pull request for this issue: https://github.com/apache/spark/pull/28444 > The ApplicationInfo in KVStore may be accessed before it's prepared > --- > > Key: SPARK-31632 > URL: https://issues.apache.org/jira/browse/SPARK-31632 > Project: Spark > Issue Type: Bug > Components: Spark Core, Web UI >Affects Versions: 3.0.0 >Reporter: Xingcan Cui >Priority: Minor > > While starting some local tests, I occasionally encountered the following > exceptions for Web UI. > {noformat} > 23:00:29.845 WARN org.eclipse.jetty.server.HttpChannel: /jobs/ > java.util.NoSuchElementException > at java.util.Collections$EmptyIterator.next(Collections.java:4191) > at > org.apache.spark.util.kvstore.InMemoryStore$InMemoryIterator.next(InMemoryStore.java:467) > at > org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:39) > at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:266) > at org.apache.spark.ui.WebUI.$anonfun$attachPage$1(WebUI.scala:89) > at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:80) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) > at > org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:505) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) > at java.lang.Thread.run(Thread.java:748){noformat} > *Reason* > That is because {{AppStatusStore.applicationInfo()}} accesses an empty view > (iterator) returned by {{InMemoryStore}}. > AppStatusStore > {code:java} > def applicationInfo(): v1.ApplicationInfo = { > store.view(classOf[ApplicationInfoWrapper]).max(1).iterator().next().info > } > {code} > InMemoryStore > {code:java} > public KVStoreView view(Class type){ > InstanceList list = inMemoryLists.get(type); > return list != null ? list.view() : emptyView(); > } > {code} > During the initialization of {{SparkContext}}, it first starts the Web UI > (SparkContext: L475 _ui.foreach(_.bind())) and then setup the > {{LiveListenerBus}} thread (SparkContext: L608 > {{setupAndStartListenerBus()}}) for dispatching the > {{SparkListenerApplicationStart}} event (which will trigger writing the > requested {{ApplicationInfo}} to {{InMemoryStore}}). > *Solution* > Since the {{applicationInfo()}} method is expected to always return a valid > {{ApplicationInfo}}, maybe we can add a while-loop-check here to guarantee > the availability of {{ApplicationInfo}}. > {code:java} > def applicationInfo(): v1.ApplicationInfo = { > var iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() > while (!iterator.hasNext){ > Thread.sleep(20) > iterator = store.view(classOf[Applic
[jira] [Assigned] (SPARK-31632) The ApplicationInfo in KVStore may be accessed before it's prepared
[ https://issues.apache.org/jira/browse/SPARK-31632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31632: Assignee: Apache Spark > The ApplicationInfo in KVStore may be accessed before it's prepared > --- > > Key: SPARK-31632 > URL: https://issues.apache.org/jira/browse/SPARK-31632 > Project: Spark > Issue Type: Bug > Components: Spark Core, Web UI >Affects Versions: 3.0.0 >Reporter: Xingcan Cui >Assignee: Apache Spark >Priority: Minor > > While starting some local tests, I occasionally encountered the following > exceptions for Web UI. > {noformat} > 23:00:29.845 WARN org.eclipse.jetty.server.HttpChannel: /jobs/ > java.util.NoSuchElementException > at java.util.Collections$EmptyIterator.next(Collections.java:4191) > at > org.apache.spark.util.kvstore.InMemoryStore$InMemoryIterator.next(InMemoryStore.java:467) > at > org.apache.spark.status.AppStatusStore.applicationInfo(AppStatusStore.scala:39) > at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:266) > at org.apache.spark.ui.WebUI.$anonfun$attachPage$1(WebUI.scala:89) > at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:80) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) > at > org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:505) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) > at java.lang.Thread.run(Thread.java:748){noformat} > *Reason* > That is because {{AppStatusStore.applicationInfo()}} accesses an empty view > (iterator) returned by {{InMemoryStore}}. > AppStatusStore > {code:java} > def applicationInfo(): v1.ApplicationInfo = { > store.view(classOf[ApplicationInfoWrapper]).max(1).iterator().next().info > } > {code} > InMemoryStore > {code:java} > public KVStoreView view(Class type){ > InstanceList list = inMemoryLists.get(type); > return list != null ? list.view() : emptyView(); > } > {code} > During the initialization of {{SparkContext}}, it first starts the Web UI > (SparkContext: L475 _ui.foreach(_.bind())) and then setup the > {{LiveListenerBus}} thread (SparkContext: L608 > {{setupAndStartListenerBus()}}) for dispatching the > {{SparkListenerApplicationStart}} event (which will trigger writing the > requested {{ApplicationInfo}} to {{InMemoryStore}}). > *Solution* > Since the {{applicationInfo()}} method is expected to always return a valid > {{ApplicationInfo}}, maybe we can add a while-loop-check here to guarantee > the availability of {{ApplicationInfo}}. > {code:java} > def applicationInfo(): v1.ApplicationInfo = { > var iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() > while (!iterator.hasNext){ > Thread.sleep(20) > iterator = store.view(classOf[ApplicationInfoWrapper]).max(1).iterator() > } > iterator.next().info > } > {code} --
[jira] [Commented] (SPARK-31374) Returning complex types in Pandas UDF
[ https://issues.apache.org/jira/browse/SPARK-31374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098609#comment-17098609 ] Oleksii Kachaiev commented on SPARK-31374: -- [~hoeze] do you have an example of the code of {{spark_window_overlap}} function? > Returning complex types in Pandas UDF > - > > Key: SPARK-31374 > URL: https://issues.apache.org/jira/browse/SPARK-31374 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.0.0 >Reporter: F. H. >Priority: Major > Labels: features > > I would like to return a complex type in an GROUPED_AGG operation: > {code:python} > window_overlap_schema = t.StructType([ > t.StructField("counts", t.ArrayType(t.LongType())), > t.StructField("starts", t.ArrayType(t.LongType())), > t.StructField("ends", t.ArrayType(t.LongType())), > ]) > @f.pandas_udf(window_overlap_schema, f.PandasUDFType.GROUPED_AGG) > def spark_window_overlap([...]): > [...] > {code} > However, I get the following error when trying to run this: > {code:python} > NotImplementedError: Invalid returnType with grouped aggregate Pandas UDFs: > StructType(List(StructField(counts,ArrayType(LongType,true),true),StructField(starts,ArrayType(LongType,true),true),StructField(ends,ArrayType(LongType,true),true))) > is not supported > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098644#comment-17098644 ] Apache Spark commented on SPARK-31212: -- User 'tianshizz' has created a pull request for this issue: https://github.com/apache/spark/pull/28445 > Failure of casting the '1000-02-29' string to the date type > --- > > Key: SPARK-31212 > URL: https://issues.apache.org/jira/browse/SPARK-31212 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 >Reporter: Maxim Gekk >Priority: Major > > The '1000-02-29' is valid date in the Julian calendar used in Spark 2.4.5 for > dates before 1582-10-15 but casting the string to the date type fails: > {code:scala} > scala> val df = > Seq("1000-02-29").toDF("dateS").select($"dateS".cast("date").as("date")) > df: org.apache.spark.sql.DataFrame = [date: date] > scala> df.show > ++ > |date| > ++ > |null| > ++ > {code} > Creating a dataset from java.sql.Date w/ the same input string works > correctly: > {code:scala} > scala> val df2 = > Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date")) > df2: org.apache.spark.sql.DataFrame = [date: date] > scala> df2.show > +--+ > | date| > +--+ > |1000-02-29| > +--+ > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098651#comment-17098651 ] Rajeev Kumar commented on SPARK-26385: -- Sorry for coming late on this thread. I was doing some testing. I found one issue in HadoopFSDelegationTokenProvider. I might be wrong also. Please validate below. Spark does not renew the token rather it creates the token at the scheduled interval. Spark needs two HDFS_DELEGATION_TOKEN. One for resource manager and second for application user. {code:java} val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf), fsToGetTokens, creds) // Get the token renewal interval if it is not set. It will only be called once. if (tokenRenewalInterval == null) { tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } {code} At the first call to the obtainDelegationTokens it creates TWO tokens correctly. Token for resource manager is getting created by method fetchDelegationTokens. Token for application user is getting created inside getTokenRenewalInterval method. {code:java} // Code snippet private var tokenRenewalInterval: Option[Long] = null {code} {code:java} sparkConf.get(PRINCIPAL).flatMap { renewer => val creds = new Credentials() fetchDelegationTokens(renewer, filesystems, creds) {code} But after 18 hours or whatever the renewal period when scheduled thread of AMCredentialRenewer tries to create HDFS_DELEFATION_TOKEN, it creates only one token (for resource manager as result of call to fetchDelegationTokens method ). But it does not create HDFS_DELEFATION_TOKEN for application user because tokenRenewalInterval is NOT NULL this time. Hence after expiration of HDFS_DELEFATION_TOKEN (typically 24 hrs) spark fails to update the spark checkpointing directory and job dies. As part of my testing, I just called getTokenRenewalInterval in else block and job is running fine. It did not die after 24 hrs. {code:java} if (tokenRenewalInterval == null) { // I put this custom log logInfo("Token Renewal interval is null. Calling getTokenRenewalInterval " + getTokenRenewer(hadoopConf)) tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } else { // I put this custom log logInfo("Token Renewal interval is NOT null. Calling getTokenRenewalInterval " + getTokenRenewer(hadoopConf)) getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } {code} Logs are - {code:java} 20/05/01 14:36:19 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is null. Calling getTokenRenewalInterval rm/host:port 20/05/02 08:36:42 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is NOT null. Calling getTokenRenewalInterval rm/host:port 20/05/03 02:37:00 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is NOT null. Calling getTokenRenewalInterval rm/host:port 20/05/03 20:37:18 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is NOT null. Calling getTokenRenewalInterval rm/host:port {code} > YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in > cache > --- > > Key: SPARK-26385 > URL: https://issues.apache.org/jira/browse/SPARK-26385 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.4.0 > Environment: Hadoop 2.6.0, Spark 2.4.0 >Reporter: T M >Priority: Major > > > Hello, > > I have Spark Structured Streaming job which is runnning on YARN(Hadoop 2.6.0, > Spark 2.4.0). After 25-26 hours, my job stops working with following error: > {code:java} > 2018-12-16 22:35:17 ERROR > org.apache.spark.internal.Logging$class.logError(Logging.scala:91): Query > TestQuery[id = a61ce197-1d1b-4e82-a7af-60162953488b, runId = > a56878cf-dfc7-4f6a-ad48-02cf738ccc2f] terminated with error > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > token (token for REMOVED: HDFS_DELEGATION_TOKEN owner=REMOVED, renewer=yarn, > realUser=, issueDate=1544903057122, maxDate=1545507857122, > sequenceNumber=10314, masterKeyId=344) can't be found in cache at > org.apache.hadoop.ipc.Client.call(Client.java:1470) at > org.apache.hadoop.ipc.Client.call(Client.java:1401) at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) a
[jira] [Created] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
Dongjoon Hyun created SPARK-31633: - Summary: Upgrade SLF4J from 1.7.16 to 1.7.30 Key: SPARK-31633 URL: https://issues.apache.org/jira/browse/SPARK-31633 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098651#comment-17098651 ] Rajeev Kumar edited comment on SPARK-26385 at 5/4/20, 4:44 AM: --- Sorry for coming late on this thread. I was doing some testing. I found one issue in HadoopFSDelegationTokenProvider. I might be wrong also. Please validate below. Spark does not renew the token rather it creates the token at the scheduled interval. Spark needs two HDFS_DELEGATION_TOKEN. One for resource manager and second for application user. {code:java} val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf), fsToGetTokens, creds) // Get the token renewal interval if it is not set. It will only be called once. if (tokenRenewalInterval == null) { tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } {code} At the first call to the obtainDelegationTokens it creates TWO tokens correctly. Token for resource manager is getting created by method fetchDelegationTokens. Token for application user is getting created inside getTokenRenewalInterval method. {code:java} // Code snippet private var tokenRenewalInterval: Option[Long] = null {code} {code:java} sparkConf.get(PRINCIPAL).flatMap { renewer => val creds = new Credentials() fetchDelegationTokens(renewer, filesystems, creds) {code} But after 18 hours or whatever the renewal period when scheduled thread of AMCredentialRenewer tries to create HDFS_DELEFATION_TOKEN, it creates only one token (for resource manager as result of call to fetchDelegationTokens method ). But it does not create HDFS_DELEFATION_TOKEN for application user because tokenRenewalInterval is NOT NULL this time. Hence after expiration of HDFS_DELEFATION_TOKEN (typically 24 hrs) spark fails to update the spark checkpointing directory and job dies. As part of my testing, I just called getTokenRenewalInterval in else block and job is running fine. It did not die after 24 hrs. {code:java} if (tokenRenewalInterval == null) { // I put this custom log logInfo("Token Renewal interval is null. Calling getTokenRenewalInterval " + getTokenRenewer(hadoopConf)) tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } else { // I put this custom log logInfo("Token Renewal interval is NOT null. Calling getTokenRenewalInterval " + getTokenRenewer(hadoopConf)) getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } {code} Logs are - {code:java} 20/05/01 14:36:19 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is null. Calling getTokenRenewalInterval rm/host:port 20/05/02 08:36:42 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is NOT null. Calling getTokenRenewalInterval rm/host:port 20/05/03 02:37:00 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is NOT null. Calling getTokenRenewalInterval rm/host:port 20/05/03 20:37:18 INFO HadoopFSDelegationTokenProvider: Token Renewal interval is NOT null. Calling getTokenRenewalInterval rm/host:port {code} [~kabhwan] Let me know if I need to create new ticket. was (Author: rajeevkumar): Sorry for coming late on this thread. I was doing some testing. I found one issue in HadoopFSDelegationTokenProvider. I might be wrong also. Please validate below. Spark does not renew the token rather it creates the token at the scheduled interval. Spark needs two HDFS_DELEGATION_TOKEN. One for resource manager and second for application user. {code:java} val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf), fsToGetTokens, creds) // Get the token renewal interval if it is not set. It will only be called once. if (tokenRenewalInterval == null) { tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) } {code} At the first call to the obtainDelegationTokens it creates TWO tokens correctly. Token for resource manager is getting created by method fetchDelegationTokens. Token for application user is getting created inside getTokenRenewalInterval method. {code:java} // Code snippet private var tokenRenewalInterval: Option[Long] = null {code} {code:java} sparkConf.get(PRINCIPAL).flatMap { renewer => val creds = new Credentials() fetchDelegationTokens(renewer, filesystems, creds) {code} But after 18 hours or whatever the renewal period when scheduled thread of AMCredentialRenewer tries to create HDFS_DELEFATION_TOKEN, it creates only one token (for resource manager as result of call to fetchDelegationTokens method ). But it does not create HDFS_DELEFATION_TOKEN for application user because tokenRenewalInterval is NOT NULL this time. Hence after expiration of HDFS_DELEFATION_TOKEN (typically 24 hrs) spark fails to update the spark checkpointing directory and job dies. As part of my testing, I just called getTokenRenewalInterval i
[jira] [Commented] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098654#comment-17098654 ] Apache Spark commented on SPARK-31633: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/28446 > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31633: Assignee: Apache Spark > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31633: Assignee: (was: Apache Spark) > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098653#comment-17098653 ] Apache Spark commented on SPARK-31633: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/28446 > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31633: -- Description: SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run under Java 9. Also, this will bring all latest bug fixes. - http://www.slf4j.org/news.html {quote}When running under Java 9, log4j version 1.2.x is unable to correctly parse the "java.version" system property. Assuming an inccorect Java version, it proceeded to disable its MDC functionality. The slf4j-log4j12 module shipping in this release fixes the issue by tweaking MDC internals by reflection, allowing log4j to run under Java 9. See also SLF4J-393. Fixed issue EventRecodingLogger not saving marker data in the event. This issue was reported in SLF4J-379 by Manish Soni with Jonas Neukomm providing the relevant PR. The slf4j-simple module now uses the latest reference to System.out or System.err. In previous releases the reference was set at the beginning and re-used. This change fixes SLF4J-389 reported by Igor Polevoy.{quote} > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > > SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run > under Java 9. Also, this will bring all latest bug fixes. > - http://www.slf4j.org/news.html > {quote}When running under Java 9, log4j version 1.2.x is unable to correctly > parse the "java.version" system property. Assuming an inccorect Java version, > it proceeded to disable its MDC functionality. The slf4j-log4j12 module > shipping in this release fixes the issue by tweaking MDC internals by > reflection, allowing log4j to run under Java 9. See also SLF4J-393. > Fixed issue EventRecodingLogger not saving marker data in the event. This > issue was reported in SLF4J-379 by Manish Soni with Jonas Neukomm providing > the relevant PR. > The slf4j-simple module now uses the latest reference to System.out or > System.err. In previous releases the reference was set at the beginning and > re-used. This change fixes SLF4J-389 reported by Igor Polevoy.{quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31633: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/28446) > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > > SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run > under Java 9. Also, this will bring all latest bug fixes. > - http://www.slf4j.org/news.html > {quote}When running under Java 9, log4j version 1.2.x is unable to correctly > parse the "java.version" system property. Assuming an inccorect Java version, > it proceeded to disable its MDC functionality. The slf4j-log4j12 module > shipping in this release fixes the issue by tweaking MDC internals by > reflection, allowing log4j to run under Java 9. See also SLF4J-393. > Fixed issue EventRecodingLogger not saving marker data in the event. This > issue was reported in SLF4J-379 by Manish Soni with Jonas Neukomm providing > the relevant PR. > The slf4j-simple module now uses the latest reference to System.out or > System.err. In previous releases the reference was set at the beginning and > re-used. This change fixes SLF4J-389 reported by Igor Polevoy.{quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31634) "show tables like" support for SQL wildcard characters (% and _)
Yuming Wang created SPARK-31634: --- Summary: "show tables like" support for SQL wildcard characters (% and _) Key: SPARK-31634 URL: https://issues.apache.org/jira/browse/SPARK-31634 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.1.0 Reporter: Yuming Wang https://docs.snowflake.com/en/sql-reference/sql/show-tables.html https://clickhouse.tech/docs/en/sql-reference/statements/show/ https://www.mysqltutorial.org/mysql-show-tables/ https://issues.apache.org/jira/browse/HIVE-23359 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31633: -- Description: SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run under Java 9. Also, this will bring all latest bug fixes. - [http://www.slf4j.org/news.html] {quote}When running under Java 9, log4j version 1.2.x is unable to correctly parse the "java.version" system property. Assuming an inccorect Java version, it proceeded to disable its MDC functionality. The slf4j-log4j12 module shipping in this release fixes the issue by tweaking MDC internals by reflection, allowing log4j to run under Java 9. See also SLF4J-393. {quote} was: SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run under Java 9. Also, this will bring all latest bug fixes. - http://www.slf4j.org/news.html {quote}When running under Java 9, log4j version 1.2.x is unable to correctly parse the "java.version" system property. Assuming an inccorect Java version, it proceeded to disable its MDC functionality. The slf4j-log4j12 module shipping in this release fixes the issue by tweaking MDC internals by reflection, allowing log4j to run under Java 9. See also SLF4J-393. Fixed issue EventRecodingLogger not saving marker data in the event. This issue was reported in SLF4J-379 by Manish Soni with Jonas Neukomm providing the relevant PR. The slf4j-simple module now uses the latest reference to System.out or System.err. In previous releases the reference was set at the beginning and re-used. This change fixes SLF4J-389 reported by Igor Polevoy.{quote} > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > > SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run > under Java 9. Also, this will bring all latest bug fixes. > - [http://www.slf4j.org/news.html] > {quote}When running under Java 9, log4j version 1.2.x is unable to correctly > parse the "java.version" system property. Assuming an inccorect Java version, > it proceeded to disable its MDC functionality. The slf4j-log4j12 module > shipping in this release fixes the issue by tweaking MDC internals by > reflection, allowing log4j to run under Java 9. See also SLF4J-393. > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31633) Upgrade SLF4J from 1.7.16 to 1.7.30
[ https://issues.apache.org/jira/browse/SPARK-31633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31633: -- Parent: SPARK-29194 Issue Type: Sub-task (was: Bug) > Upgrade SLF4J from 1.7.16 to 1.7.30 > --- > > Key: SPARK-31633 > URL: https://issues.apache.org/jira/browse/SPARK-31633 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > > SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run > under Java 9. Also, this will bring all latest bug fixes. > - [http://www.slf4j.org/news.html] > {quote}When running under Java 9, log4j version 1.2.x is unable to correctly > parse the "java.version" system property. Assuming an inccorect Java version, > it proceeded to disable its MDC functionality. The slf4j-log4j12 module > shipping in this release fixes the issue by tweaking MDC internals by > reflection, allowing log4j to run under Java 9. See also SLF4J-393. > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31267) Flaky test: WholeStageCodegenSparkSubmitSuite.Generated code on driver should not embed platform-specific constant
[ https://issues.apache.org/jira/browse/SPARK-31267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31267. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/28438 > Flaky test: WholeStageCodegenSparkSubmitSuite.Generated code on driver should > not embed platform-specific constant > -- > > Key: SPARK-31267 > URL: https://issues.apache.org/jira/browse/SPARK-31267 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gabor Somogyi >Priority: Major > Fix For: 3.0.0 > > > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120363/testReport/ > {code} > Error Message > org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to > failAfter did not complete within 1 minute. > Stacktrace > sbt.ForkMain$ForkError: > org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to > failAfter did not complete within 1 minute. > at java.lang.Thread.getStackTrace(Thread.java:1559) > at > org.scalatest.concurrent.TimeLimits.failAfterImpl(TimeLimits.scala:234) > at > org.scalatest.concurrent.TimeLimits.failAfterImpl$(TimeLimits.scala:233) > at > org.apache.spark.deploy.SparkSubmitSuite$.failAfterImpl(SparkSubmitSuite.scala:1416) > at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:230) > at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:229) > at > org.apache.spark.deploy.SparkSubmitSuite$.failAfter(SparkSubmitSuite.scala:1416) > at > org.apache.spark.deploy.SparkSubmitSuite$.runSparkSubmit(SparkSubmitSuite.scala:1435) > at > org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite.$anonfun$new$1(WholeStageCodegenSparkSubmitSuite.scala:53) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) > at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) > at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) > at org.scalatest.Transformer.apply(Transformer.scala:22) > at org.scalatest.Transformer.apply(Transformer.scala:20) > at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) > at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:151) > at > org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184) > at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196) > at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286) > at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196) > at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178) > at > org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:58) > at > org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221) > at > org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214) > at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:58) > at > org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229) > at > org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:393) > at scala.collection.immutable.List.foreach(List.scala:392) > at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:381) > at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:376) > at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:458) > at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229) > at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228) > at org.scalatest.FunSuite.runTests(FunSuite.scala:1560) > at org.scalatest.Suite.run(Suite.scala:1124) > at org.scalatest.Suite.run$(Suite.scala:1106) > at > org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560) > at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233) > at org.scalatest.SuperEngine.runImpl(Engine.scala:518) > at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233) > at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232) > at > org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:58) > at > org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) > at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) > at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) > at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:58) > at > org.scalatest.too
[jira] [Assigned] (SPARK-31626) Port HIVE-10415: hive.start.cleanup.scratchdir configuration is not taking effect
[ https://issues.apache.org/jira/browse/SPARK-31626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31626: Assignee: Yuming Wang > Port HIVE-10415: hive.start.cleanup.scratchdir configuration is not taking > effect > - > > Key: SPARK-31626 > URL: https://issues.apache.org/jira/browse/SPARK-31626 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-10415 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31626) Port HIVE-10415: hive.start.cleanup.scratchdir configuration is not taking effect
[ https://issues.apache.org/jira/browse/SPARK-31626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31626. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28436 [https://github.com/apache/spark/pull/28436] > Port HIVE-10415: hive.start.cleanup.scratchdir configuration is not taking > effect > - > > Key: SPARK-31626 > URL: https://issues.apache.org/jira/browse/SPARK-31626 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Fix For: 3.0.0 > > > https://issues.apache.org/jira/browse/HIVE-10415 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31606) reduce the perf regression of vectorized parquet reader caused by datetime rebase
[ https://issues.apache.org/jira/browse/SPARK-31606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31606. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28406 [https://github.com/apache/spark/pull/28406] > reduce the perf regression of vectorized parquet reader caused by datetime > rebase > - > > Key: SPARK-31606 > URL: https://issues.apache.org/jira/browse/SPARK-31606 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098723#comment-17098723 ] Jungtaek Lim commented on SPARK-26385: -- [~rajeevkumar] Yes please raise a separate JIRA issue. Please make sure the new JIRA issue contains the information I described above. https://issues.apache.org/jira/browse/SPARK-26385?focusedCommentId=17091075&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17091075 Please mention explicitly where the log messages come from (driver, AM, executor) whenever you refer the log message. Otherwise it's non-trivial to track down. > YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in > cache > --- > > Key: SPARK-26385 > URL: https://issues.apache.org/jira/browse/SPARK-26385 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.4.0 > Environment: Hadoop 2.6.0, Spark 2.4.0 >Reporter: T M >Priority: Major > > > Hello, > > I have Spark Structured Streaming job which is runnning on YARN(Hadoop 2.6.0, > Spark 2.4.0). After 25-26 hours, my job stops working with following error: > {code:java} > 2018-12-16 22:35:17 ERROR > org.apache.spark.internal.Logging$class.logError(Logging.scala:91): Query > TestQuery[id = a61ce197-1d1b-4e82-a7af-60162953488b, runId = > a56878cf-dfc7-4f6a-ad48-02cf738ccc2f] terminated with error > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > token (token for REMOVED: HDFS_DELEGATION_TOKEN owner=REMOVED, renewer=yarn, > realUser=, issueDate=1544903057122, maxDate=1545507857122, > sequenceNumber=10314, masterKeyId=344) can't be found in cache at > org.apache.hadoop.ipc.Client.call(Client.java:1470) at > org.apache.hadoop.ipc.Client.call(Client.java:1401) at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at > org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1977) at > org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:133) at > org.apache.hadoop.fs.FileContext$14.next(FileContext.java:1120) at > org.apache.hadoop.fs.FileContext$14.next(FileContext.java:1116) at > org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at > org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1116) at > org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1581) at > org.apache.spark.sql.execution.streaming.FileContextBasedCheckpointFileManager.exists(CheckpointFileManager.scala:326) > at > org.apache.spark.sql.execution.streaming.HDFSMetadataLog.get(HDFSMetadataLog.scala:142) > at > org.apache.spark.sql.execution.streaming.HDFSMetadataLog.add(HDFSMetadataLog.scala:110) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$1.apply$mcV$sp(MicroBatchExecution.scala:544) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$1.apply(MicroBatchExecution.scala:542) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$1.apply(MicroBatchExecution.scala:542) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:554) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:542) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166) > at > org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.