[jira] [Created] (SPARK-30723) Executing the example on https://spark.apache.org/docs/latest/running-on-yarn.html fails

2020-02-03 Thread Reinhard Eilmsteiner (Jira)
Reinhard Eilmsteiner created SPARK-30723: Summary: Executing the example on https://spark.apache.org/docs/latest/running-on-yarn.html fails Key: SPARK-30723 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Commented] (SPARK-30701) SQL test running on Windows: hadoop chgrp warnings

2020-02-03 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029619#comment-17029619 ] Guram Savinov commented on SPARK-30701: --- Ok, let's go to Hadoop project:  https://

[jira] [Commented] (SPARK-30706) TimeZone in writing pure date type in CSV output

2020-02-03 Thread Waldemar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029604#comment-17029604 ] Waldemar commented on SPARK-30706: -- Yes please. I have attached zip with these csv spar

[jira] [Updated] (SPARK-30706) TimeZone in writing pure date type in CSV output

2020-02-03 Thread Waldemar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Waldemar updated SPARK-30706: - Attachment: DateZoneBug.zip > TimeZone in writing pure date type in CSV output > ---

[jira] [Created] (SPARK-30722) Document type hints in pandas UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-30722: Summary: Document type hints in pandas UDF Key: SPARK-30722 URL: https://issues.apache.org/jira/browse/SPARK-30722 Project: Spark Issue Type: Documentation

[jira] [Resolved] (SPARK-30717) AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`

2020-02-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30717. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27446 [https://gith

[jira] [Assigned] (SPARK-30717) AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`

2020-02-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30717: --- Assignee: Wei Xue > AQE subquery map should cache `SubqueryExec` instead of > `ExecSubquer

[jira] [Updated] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30688: - Affects Version/s: (was: 3.0.0) > Spark SQL Unix Timestamp produces incorrect result with un

[jira] [Comment Edited] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029576#comment-17029576 ] Hyukjin Kwon edited comment on SPARK-30688 at 2/4/20 4:45 AM:

[jira] [Updated] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30688: - Affects Version/s: 3.0.0 > Spark SQL Unix Timestamp produces incorrect result with unix_timestam

[jira] [Reopened] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-30688: -- > Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF > ---

[jira] [Commented] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029576#comment-17029576 ] Hyukjin Kwon commented on SPARK-30688: -- Ah, okay. I misread this: {quote} Spark-3.0

[jira] [Commented] (SPARK-30677) Spark Streaming Job stuck when Kinesis Shard is increased when the job is running

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029574#comment-17029574 ] Hyukjin Kwon commented on SPARK-30677: -- Are you able to produce reproducible steps

[jira] [Commented] (SPARK-30675) Spark Streaming Job stopped reading events from Queue upon Deregister Exception

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029575#comment-17029575 ] Hyukjin Kwon commented on SPARK-30675: -- Are you able to provide minimised reproduce

[jira] [Commented] (SPARK-30687) When reading from a file with pre-defined schema and encountering a single value that is not the same type as that of its column , Spark nullifies the entire row

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029572#comment-17029572 ] Hyukjin Kwon commented on SPARK-30687: -- [~bnguye1010], Spark 2.3.x is EOL. Can you

[jira] [Resolved] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30688. -- Resolution: Incomplete > Spark SQL Unix Timestamp produces incorrect result with unix_timestam

[jira] [Commented] (SPARK-30688) Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029570#comment-17029570 ] Hyukjin Kwon commented on SPARK-30688: -- So switching to new java.time APIs fixed th

[jira] [Resolved] (SPARK-30701) SQL test running on Windows: hadoop chgrp warnings

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30701. -- Resolution: Not A Problem I am resolving this as it's a Hadoop side problem. > SQL test runni

[jira] [Assigned] (SPARK-30701) SQL test running on Windows: hadoop chgrp warnings

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30701: Assignee: (was: Felix Cheung) > SQL test running on Windows: hadoop chgrp warnings >

[jira] [Commented] (SPARK-30706) TimeZone in writing pure date type in CSV output

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029566#comment-17029566 ] Hyukjin Kwon commented on SPARK-30706: -- Can you show your csv files? > TimeZone in

[jira] [Commented] (SPARK-30709) Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables.

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029562#comment-17029562 ] Hyukjin Kwon commented on SPARK-30709: -- Please ask questions into mailing list (htt

[jira] [Resolved] (SPARK-30709) Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables.

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30709. -- Resolution: Invalid > Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables

[jira] [Updated] (SPARK-30710) SPARK 2.4.4 - DROP TABLE and drop HDFS

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30710: - Fix Version/s: (was: 2.4.4) > SPARK 2.4.4 - DROP TABLE and drop HDFS > -

[jira] [Resolved] (SPARK-30710) SPARK 2.4.4 - DROP TABLE and drop HDFS

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30710. -- Resolution: Invalid > SPARK 2.4.4 - DROP TABLE and drop HDFS > ---

[jira] [Commented] (SPARK-30710) SPARK 2.4.4 - DROP TABLE and drop HDFS

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029561#comment-17029561 ] Hyukjin Kwon commented on SPARK-30710: -- Please ask questions into mailing list (ht

[jira] [Commented] (SPARK-30711) 64KB JVM bytecode limit - janino.InternalCompilerException

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029560#comment-17029560 ] Hyukjin Kwon commented on SPARK-30711: -- It seems passing fine in the master. [~schr

[jira] [Updated] (SPARK-30710) SPARK 2.4.4 - DROP TABLE and drop HDFS

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30710: - Target Version/s: (was: 2.4.4) > SPARK 2.4.4 - DROP TABLE and drop HDFS >

[jira] [Commented] (SPARK-30712) Estimate sizeInBytes from file metadata for parquet files

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029558#comment-17029558 ] Hyukjin Kwon commented on SPARK-30712: -- Do you have some works already done and/or

[jira] [Commented] (SPARK-30712) Estimate sizeInBytes from file metadata for parquet files

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029557#comment-17029557 ] Hyukjin Kwon commented on SPARK-30712: -- To do that, it should actually reads the fi

[jira] [Commented] (SPARK-30714) DSV2: Vectorized datasource does not have handling for ProlepticCalendar

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029555#comment-17029555 ] Hyukjin Kwon commented on SPARK-30714: -- Spark 2.3.x is EOL so there wouldn't be no

[jira] [Resolved] (SPARK-30718) Exclude jdk.tools dependency from hadoop-yarn-api

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30718. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27445 [https://

[jira] [Assigned] (SPARK-30718) Exclude jdk.tools dependency from hadoop-yarn-api

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30718: - Assignee: Dongjoon Hyun > Exclude jdk.tools dependency from hadoop-yarn-api > -

[jira] [Commented] (SPARK-30721) Turning off WSCG did not take effect in AQE query planning

2020-02-03 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029527#comment-17029527 ] Wei Xue commented on SPARK-30721: - cc [~cloud_fan], [~Jk_Self] > Turning off WSCG did n

[jira] [Created] (SPARK-30721) Turning off WSCG did not take effect in AQE query planning

2020-02-03 Thread Wei Xue (Jira)
Wei Xue created SPARK-30721: --- Summary: Turning off WSCG did not take effect in AQE query planning Key: SPARK-30721 URL: https://issues.apache.org/jira/browse/SPARK-30721 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30719) AQE should not issue a "not supported" warning for queries being by-passed

2020-02-03 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029525#comment-17029525 ] Wei Xue commented on SPARK-30719: - cc [~cloud_fan], [~Jk_Self] > AQE should not issue a

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Created] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
Andrei Stankevich created SPARK-30720: - Summary: Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service. Key: SPARK-30720 URL: https://issues.apache.org/

[jira] [Updated] (SPARK-30720) Spark framework hangs and becomes inactive on Mesos UI if executor can not connect to shuffle external service.

2020-02-03 Thread Andrei Stankevich (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Stankevich updated SPARK-30720: -- Description: We are using spark 2.4.3 with mesos and with external shuffle service. Ex

[jira] [Created] (SPARK-30719) AQE should not issue a "not supported" warning for queries being by-passed

2020-02-03 Thread Wei Xue (Jira)
Wei Xue created SPARK-30719: --- Summary: AQE should not issue a "not supported" warning for queries being by-passed Key: SPARK-30719 URL: https://issues.apache.org/jira/browse/SPARK-30719 Project: Spark

[jira] [Updated] (SPARK-30718) Exclude jdk.tools dependency from hadoop-yarn-api

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30718: -- Target Version/s: 3.0.0 > Exclude jdk.tools dependency from hadoop-yarn-api >

[jira] [Updated] (SPARK-30718) Exclude jdk.tools dependency from hadoop-yarn-api

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30718: -- Parent: SPARK-29194 Issue Type: Sub-task (was: Bug) > Exclude jdk.tools dependency fr

[jira] [Created] (SPARK-30718) Exclude jdk.tools dependency from hadoop-yarn-api

2020-02-03 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30718: - Summary: Exclude jdk.tools dependency from hadoop-yarn-api Key: SPARK-30718 URL: https://issues.apache.org/jira/browse/SPARK-30718 Project: Spark Issue Typ

[jira] [Updated] (SPARK-30717) AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression`

2020-02-03 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Xue updated SPARK-30717: Summary: AQE subquery map should cache `SubqueryExec` instead of `ExecSubqueryExpression` (was: AQE subqu

[jira] [Created] (SPARK-30717) AQE subquery map should cache `BaseSubqueryExec` instead of `ExecSubqueryExpression`

2020-02-03 Thread Wei Xue (Jira)
Wei Xue created SPARK-30717: --- Summary: AQE subquery map should cache `BaseSubqueryExec` instead of `ExecSubqueryExpression` Key: SPARK-30717 URL: https://issues.apache.org/jira/browse/SPARK-30717 Project: S

[jira] [Commented] (SPARK-30711) 64KB JVM bytecode limit - janino.InternalCompilerException

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029139#comment-17029139 ] Dongjoon Hyun commented on SPARK-30711: --- Thank you for reporting, [~schreiber]. Co

[jira] [Updated] (SPARK-30711) 64KB JVM bytecode limit - janino.InternalCompilerException

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30711: -- Summary: 64KB JVM bytecode limit - janino.InternalCompilerException (was: 64KB JBM bytecode l

[jira] [Commented] (SPARK-30715) Upgrade fabric8 to 4.7.1 to support K8s 1.17

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029136#comment-17029136 ] Dongjoon Hyun commented on SPARK-30715: --- For now, I converted this to `Improvement

[jira] [Updated] (SPARK-30715) Upgrade fabric8 to 4.7.1 to support K8s 1.17

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30715: -- Summary: Upgrade fabric8 to 4.7.1 to support K8s 1.17 (was: Upgrade fabric8 to 4.7.1) > Upgr

[jira] [Updated] (SPARK-30715) Upgrade fabric8 to 4.7.1

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30715: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Upgrade fabric8 to 4.7.1 >

[jira] [Updated] (SPARK-30715) Upgrade fabric8 to 4.7.1

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30715: -- Issue Type: Improvement (was: Dependency upgrade) > Upgrade fabric8 to 4.7.1 > --

[jira] [Commented] (SPARK-30715) Upgrade fabric8 to 4.7.1

2020-02-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029132#comment-17029132 ] Dongjoon Hyun commented on SPARK-30715: --- Hi, [~onursatici]. We still use `4.6.4`.

[jira] [Updated] (SPARK-30525) HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore

2020-02-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30525: Fix Version/s: (was: 3.0.0) 3.1.0 > HiveTableScanExec do not need to prune

[jira] [Resolved] (SPARK-30525) HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore

2020-02-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30525. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27232 [https://gith

[jira] [Assigned] (SPARK-30525) HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore

2020-02-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30525: --- Assignee: Hu Fuwang > HiveTableScanExec do not need to prune partitions again after pushing

[jira] [Created] (SPARK-30716) Change `SkewedPartitionReaderExec` into `UnaryExecNode` and replace the direct link with dependent stages with a reuse link

2020-02-03 Thread Wei Xue (Jira)
Wei Xue created SPARK-30716: --- Summary: Change `SkewedPartitionReaderExec` into `UnaryExecNode` and replace the direct link with dependent stages with a reuse link Key: SPARK-30716 URL: https://issues.apache.org/jira/bro

[jira] [Commented] (SPARK-30614) The native ALTER COLUMN syntax should change one thing at a time

2020-02-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029068#comment-17029068 ] Wenchen Fan commented on SPARK-30614: - Yea we can deduce the column type so nothing

[jira] [Created] (SPARK-30715) Upgrade fabric8 to 4.7.1

2020-02-03 Thread Onur Satici (Jira)
Onur Satici created SPARK-30715: --- Summary: Upgrade fabric8 to 4.7.1 Key: SPARK-30715 URL: https://issues.apache.org/jira/browse/SPARK-30715 Project: Spark Issue Type: Dependency upgrade

[jira] [Commented] (SPARK-30714) DSV2: Vectorized datasource does not have handling for ProlepticCalendar

2020-02-03 Thread Shubham Chaurasia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028975#comment-17028975 ] Shubham Chaurasia commented on SPARK-30714: --- Oh looks like https://issues.apac

[jira] [Updated] (SPARK-30714) DSV2: Vectorized datasource does not have handling for ProlepticCalendar

2020-02-03 Thread Shubham Chaurasia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated SPARK-30714: -- Description: Consider the following scenarios - 1) {code:scala} scala> spark.read.for

[jira] [Created] (SPARK-30714) DSV2: Vectorized datasource does not have handling for ProlepticCalendar

2020-02-03 Thread Shubham Chaurasia (Jira)
Shubham Chaurasia created SPARK-30714: - Summary: DSV2: Vectorized datasource does not have handling for ProlepticCalendar Key: SPARK-30714 URL: https://issues.apache.org/jira/browse/SPARK-30714 Pr

[jira] [Created] (SPARK-30713) Respect mapOutputSize in memory in adaptive execution

2020-02-03 Thread liupengcheng (Jira)
liupengcheng created SPARK-30713: Summary: Respect mapOutputSize in memory in adaptive execution Key: SPARK-30713 URL: https://issues.apache.org/jira/browse/SPARK-30713 Project: Spark Issue T

[jira] [Updated] (SPARK-30711) 64KB JBM bytecode limit - janino.InternalCompilerException

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederik Schreiber updated SPARK-30711: --- Environment: Windows 10 Spark 2.4.4 scalaVersion 2.11.12 JVM Oracle 1.8.0_221-b11

[jira] [Created] (SPARK-30712) Estimate sizeInBytes from file metadata for parquet files

2020-02-03 Thread liupengcheng (Jira)
liupengcheng created SPARK-30712: Summary: Estimate sizeInBytes from file metadata for parquet files Key: SPARK-30712 URL: https://issues.apache.org/jira/browse/SPARK-30712 Project: Spark Iss

[jira] [Commented] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028902#comment-17028902 ] Frederik Schreiber commented on SPARK-22510: extract my code to an example a

[jira] [Comment Edited] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028902#comment-17028902 ] Frederik Schreiber edited comment on SPARK-22510 at 2/3/20 12:35 PM: -

[jira] [Updated] (SPARK-30711) 64KB JBM bytecode limit - janino.InternalCompilerException

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederik Schreiber updated SPARK-30711: --- Description: Exception {code:java} ERROR CodeGenerator: failed to compile: org.code

[jira] [Created] (SPARK-30711) 64KB JBM bytecode limit - janino.InternalCompilerException

2020-02-03 Thread Frederik Schreiber (Jira)
Frederik Schreiber created SPARK-30711: -- Summary: 64KB JBM bytecode limit - janino.InternalCompilerException Key: SPARK-30711 URL: https://issues.apache.org/jira/browse/SPARK-30711 Project: Spark

[jira] [Created] (SPARK-30710) SPARK 2.4.4 - DROP TABLE and drop HDFS

2020-02-03 Thread Nguyen Nhanduc (Jira)
Nguyen Nhanduc created SPARK-30710: -- Summary: SPARK 2.4.4 - DROP TABLE and drop HDFS Key: SPARK-30710 URL: https://issues.apache.org/jira/browse/SPARK-30710 Project: Spark Issue Type: Questi

[jira] [Commented] (SPARK-27990) Provide a way to recursively load data from datasource

2020-02-03 Thread Jorge Machado (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028856#comment-17028856 ] Jorge Machado commented on SPARK-27990: --- [~nchammas]: Just pass this like:  {code:

[jira] [Commented] (SPARK-27990) Provide a way to recursively load data from datasource

2020-02-03 Thread Jorge Machado (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028854#comment-17028854 ] Jorge Machado commented on SPARK-27990: --- Can we backport this to 2.4.4 ? > Provid

[jira] [Resolved] (SPARK-28413) sizeInByte is Not updated for parquet datasource on Next Insert.

2020-02-03 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-28413. - Resolution: Fixed The issue fixed by https://github.com/apache/spark/commit/17881a467a1ac4224a5

[jira] [Created] (SPARK-30709) Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables.

2020-02-03 Thread Carlos Mario (Jira)
Carlos Mario created SPARK-30709: Summary: Spark 2.3 to Spark 2.4 Upgrade. Problems reading HIVE partitioned tables. Key: SPARK-30709 URL: https://issues.apache.org/jira/browse/SPARK-30709 Project: Sp

[jira] [Commented] (SPARK-29245) CCE during creating HiveMetaStoreClient

2020-02-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028804#comment-17028804 ] Hyukjin Kwon commented on SPARK-29245: -- I sent an email to Hive dev for Hive 2.3.7

[jira] [Commented] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028791#comment-17028791 ] Frederik Schreiber commented on SPARK-22510: thank you for answer i try to e

[jira] [Commented] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2020-02-03 Thread Kazuaki Ishizaki (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028784#comment-17028784 ] Kazuaki Ishizaki commented on SPARK-22510: -- [~schreiber] Thank you for reportin

[jira] [Commented] (SPARK-25094) proccesNext() failed to compile size is over 64kb

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028783#comment-17028783 ] Frederik Schreiber commented on SPARK-25094: Should this issue linked to SPA

[jira] [Comment Edited] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028763#comment-17028763 ] Frederik Schreiber edited comment on SPARK-22510 at 2/3/20 8:29 AM: --

[jira] [Commented] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2020-02-03 Thread Frederik Schreiber (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028763#comment-17028763 ] Frederik Schreiber commented on SPARK-22510: Hi [~smilegator], [~kiszk] we

[jira] [Commented] (SPARK-30708) first_value/last_value window function throws ParseException

2020-02-03 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028752#comment-17028752 ] jiaan.geng commented on SPARK-30708: I'm working! > first_value/last_value window f

[jira] [Updated] (SPARK-30707) Lead/Lag window function throws AnalysisException without ORDER BY clause

2020-02-03 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30707: --- Description:  Lead/Lag window function throws AnalysisException without ORDER BY clause: {code:java}

[jira] [Updated] (SPARK-30708) first_value/last_value window function throws ParseException

2020-02-03 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30708: --- Summary: first_value/last_value window function throws ParseException (was: first_value/last_value

[jira] [Created] (SPARK-30708) first_value/last_value throws ParseException

2020-02-03 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30708: -- Summary: first_value/last_value throws ParseException Key: SPARK-30708 URL: https://issues.apache.org/jira/browse/SPARK-30708 Project: Spark Issue Type: Sub-task