[jira] [Commented] (SPARK-31579) Replace floorDiv by / in localRebaseGregorianToJulianDays()

2020-05-05 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099579#comment-17099579 ] Maxim Gekk commented on SPARK-31579: [~suddhuASF] The replace floorDiv by / is trivial. Please,

[jira] [Created] (SPARK-31630) Skip timestamp rebasing after 1900-01-01

2020-05-03 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31630: -- Summary: Skip timestamp rebasing after 1900-01-01 Key: SPARK-31630 URL: https://issues.apache.org/jira/browse/SPARK-31630 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-31623) Benchmark rebasing of INT96 and TIMESTAMP_MILLIS timestamps in read/write

2020-05-01 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31623: -- Summary: Benchmark rebasing of INT96 and TIMESTAMP_MILLIS timestamps in read/write Key: SPARK-31623 URL: https://issues.apache.org/jira/browse/SPARK-31623 Project: Spark

[jira] [Resolved] (SPARK-31554) Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite

2020-04-30 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk resolved SPARK-31554. Resolution: Not A Problem > Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite >

[jira] [Created] (SPARK-31579) Replace floorDiv by / in localRebaseGregorianToJulianDays()

2020-04-27 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31579: -- Summary: Replace floorDiv by / in localRebaseGregorianToJulianDays() Key: SPARK-31579 URL: https://issues.apache.org/jira/browse/SPARK-31579 Project: Spark

[jira] [Commented] (SPARK-31449) Investigate the difference between JDK and Spark's time zone offset calculation

2020-04-26 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092824#comment-17092824 ] Maxim Gekk commented on SPARK-31449: [~cloud_fan] [~hyukjin.kwon] I compared results of those 2

[jira] [Commented] (SPARK-31563) Failure of InSet.sql for UTF8String collection

2020-04-25 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092168#comment-17092168 ] Maxim Gekk commented on SPARK-31563: I am working on the issue > Failure of InSet.sql for

[jira] [Created] (SPARK-31563) Failure of InSet.sql for UTF8String collection

2020-04-25 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31563: -- Summary: Failure of InSet.sql for UTF8String collection Key: SPARK-31563 URL: https://issues.apache.org/jira/browse/SPARK-31563 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-31554) Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite

2020-04-24 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091614#comment-17091614 ] Maxim Gekk commented on SPARK-31554: [~cloud_fan] [~hyukjin.kwon] Can I we disable the flaky test

[jira] [Created] (SPARK-31554) Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite

2020-04-24 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31554: -- Summary: Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite Key: SPARK-31554 URL: https://issues.apache.org/jira/browse/SPARK-31554 Project: Spark

[jira] [Commented] (SPARK-31553) Wrong result of isInCollection for large collections

2020-04-24 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091490#comment-17091490 ] Maxim Gekk commented on SPARK-31553: I am working on the issue > Wrong result of isInCollection for

[jira] [Created] (SPARK-31553) Wrong result of isInCollection for large collections

2020-04-24 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31553: -- Summary: Wrong result of isInCollection for large collections Key: SPARK-31553 URL: https://issues.apache.org/jira/browse/SPARK-31553 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31463) Enhance JsonDataSource by replacing jackson with simdjson

2020-04-24 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091389#comment-17091389 ] Maxim Gekk commented on SPARK-31463: Parsing itself takes 10-20%. JSON datasource spends significant

[jira] [Updated] (SPARK-31449) Investigate the difference between JDK and Spark's time zone offset calculation

2020-04-24 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31449: --- Summary: Investigate the difference between JDK and Spark's time zone offset calculation (was: Is

[jira] [Updated] (SPARK-31449) Investigate the difference between JDK and Spark's time zone offset calculation

2020-04-24 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31449: --- Issue Type: Improvement (was: Question) > Investigate the difference between JDK and Spark's time

[jira] [Created] (SPARK-31490) Benchmark conversions to/from Java 8 date-time types

2020-04-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31490: -- Summary: Benchmark conversions to/from Java 8 date-time types Key: SPARK-31490 URL: https://issues.apache.org/jira/browse/SPARK-31490 Project: Spark Issue Type:

[jira] [Created] (SPARK-31489) Failure on pushing down filters with java.time.LocalDate values in ORC

2020-04-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31489: -- Summary: Failure on pushing down filters with java.time.LocalDate values in ORC Key: SPARK-31489 URL: https://issues.apache.org/jira/browse/SPARK-31489 Project: Spark

[jira] [Updated] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

2020-04-19 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31488: --- Description: Currently, ParquetFilters supports only java.sql.Date values of DateType, and

[jira] [Created] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

2020-04-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31488: -- Summary: Support `java.time.LocalDate` in Parquet filter pushdown Key: SPARK-31488 URL: https://issues.apache.org/jira/browse/SPARK-31488 Project: Spark Issue

[jira] [Created] (SPARK-31471) Add a script to run multiple benchmarks

2020-04-17 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31471: -- Summary: Add a script to run multiple benchmarks Key: SPARK-31471 URL: https://issues.apache.org/jira/browse/SPARK-31471 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-15 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084308#comment-17084308 ] Maxim Gekk commented on SPARK-31423: [~bersprockets] I think we should take the next valid date for

[jira] [Created] (SPARK-31449) Is there a difference between JDK and Spark's time zone offset calculation

2020-04-15 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31449: -- Summary: Is there a difference between JDK and Spark's time zone offset calculation Key: SPARK-31449 URL: https://issues.apache.org/jira/browse/SPARK-31449 Project:

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-14 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083595#comment-17083595 ] Maxim Gekk commented on SPARK-31423: I have debugged this slightly on Spark 2.4, so, '1582-10-14'

[jira] [Resolved] (SPARK-31445) Avoid floating-point division in millisToDays

2020-04-14 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk resolved SPARK-31445. Resolution: Won't Fix > Avoid floating-point division in millisToDays >

[jira] [Created] (SPARK-31445) Avoid floating-point division in millisToDays

2020-04-14 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31445: -- Summary: Avoid floating-point division in millisToDays Key: SPARK-31445 URL: https://issues.apache.org/jira/browse/SPARK-31445 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-14 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083314#comment-17083314 ] Maxim Gekk commented on SPARK-31423: I am working on the issue. > DATES and TIMESTAMPS for a

[jira] [Comment Edited] (SPARK-31443) Perf regression of toJavaDate

2020-04-14 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083217#comment-17083217 ] Maxim Gekk edited comment on SPARK-31443 at 4/14/20, 1:21 PM: -- FYI

[jira] [Commented] (SPARK-31443) Perf regression of toJavaDate

2020-04-14 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083217#comment-17083217 ] Maxim Gekk commented on SPARK-31443: FYI [~cloud_fan] > Perf regression of toJavaDate >

[jira] [Updated] (SPARK-31443) Perf regression of toJavaDate

2020-04-14 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31443: --- Description: DateTimeBenchmark shows the regression Spark 2.4.6-SNAPSHOT at the PR

[jira] [Created] (SPARK-31443) Perf regression of toJavaDate

2020-04-14 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31443: -- Summary: Perf regression of toJavaDate Key: SPARK-31443 URL: https://issues.apache.org/jira/browse/SPARK-31443 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-31439) Perf regression of fromJavaDate

2020-04-13 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31439: -- Summary: Perf regression of fromJavaDate Key: SPARK-31439 URL: https://issues.apache.org/jira/browse/SPARK-31439 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-31426) Regression in loading/saving timestamps from/to ORC files

2020-04-13 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31426: --- Parent: SPARK-31404 Issue Type: Sub-task (was: Bug) > Regression in loading/saving

[jira] [Issue Comment Deleted] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-12 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31423: --- Comment: was deleted (was: This is intentional behavior because ORC format assumes the hybrid

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-12 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082051#comment-17082051 ] Maxim Gekk commented on SPARK-31423: This is intentional behavior because ORC format assumes the

[jira] [Created] (SPARK-31426) Regression in loading/saving timestamps from/to ORC files

2020-04-12 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31426: -- Summary: Regression in loading/saving timestamps from/to ORC files Key: SPARK-31426 URL: https://issues.apache.org/jira/browse/SPARK-31426 Project: Spark Issue

[jira] [Commented] (SPARK-28624) make_date is inconsistent when reading from table

2020-04-10 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080312#comment-17080312 ] Maxim Gekk commented on SPARK-28624: toJavaDate is implemented differently in the master 

[jira] [Created] (SPARK-31402) Incorrect rebasing of BCE dates

2020-04-09 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31402: -- Summary: Incorrect rebasing of BCE dates Key: SPARK-31402 URL: https://issues.apache.org/jira/browse/SPARK-31402 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-31398) Speed up reading dates in ORC

2020-04-09 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31398: -- Summary: Speed up reading dates in ORC Key: SPARK-31398 URL: https://issues.apache.org/jira/browse/SPARK-31398 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-31385) Results of Julian-Gregorian rebasing don't match to Gregorian-Julian rebasing

2020-04-08 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31385: -- Summary: Results of Julian-Gregorian rebasing don't match to Gregorian-Julian rebasing Key: SPARK-31385 URL: https://issues.apache.org/jira/browse/SPARK-31385 Project:

[jira] [Created] (SPARK-31359) Speed up timestamps rebasing

2020-04-06 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31359: -- Summary: Speed up timestamps rebasing Key: SPARK-31359 URL: https://issues.apache.org/jira/browse/SPARK-31359 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-31353) Set time zone in DateTimeBenchmark and DateTimeRebaseBenchmark

2020-04-05 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31353: -- Summary: Set time zone in DateTimeBenchmark and DateTimeRebaseBenchmark Key: SPARK-31353 URL: https://issues.apache.org/jira/browse/SPARK-31353 Project: Spark

[jira] [Created] (SPARK-31343) Check codegen does not fail on expressions with special characters in string parameters

2020-04-03 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31343: -- Summary: Check codegen does not fail on expressions with special characters in string parameters Key: SPARK-31343 URL: https://issues.apache.org/jira/browse/SPARK-31343

[jira] [Updated] (SPARK-31328) Incorrect timestamps rebasing on autumn daylight saving time

2020-04-02 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31328: --- Description: Run the following code in the *America/Los_Angeles* time zone: {code:scala}

[jira] [Updated] (SPARK-31328) Incorrect timestamps rebasing on autumn daylight saving time

2020-04-02 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31328: --- Description: Run the following code in the *America/Los_Angeles* time zone: {code:scala}

[jira] [Created] (SPARK-31328) Incorrect timestamps rebasing on autumn daylight saving time

2020-04-02 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31328: -- Summary: Incorrect timestamps rebasing on autumn daylight saving time Key: SPARK-31328 URL: https://issues.apache.org/jira/browse/SPARK-31328 Project: Spark

[jira] [Updated] (SPARK-31318) Split Parquet/Avro configs for rebasing dates/timestamps in read and in write

2020-03-31 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31318: --- Parent: SPARK-30951 Issue Type: Sub-task (was: Improvement) > Split Parquet/Avro configs

[jira] [Created] (SPARK-31318) Split Parquet/Avro configs for rebasing dates/timestamps in read and in write

2020-03-31 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31318: -- Summary: Split Parquet/Avro configs for rebasing dates/timestamps in read and in write Key: SPARK-31318 URL: https://issues.apache.org/jira/browse/SPARK-31318 Project:

[jira] [Updated] (SPARK-31311) Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31311: --- Description: * Benchmark saving dates/timestamps before and after 1582-10-15 * Benchmark loading

[jira] [Created] (SPARK-31311) Benchmark date-time rebasing in ORC datasource

2020-03-31 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31311: -- Summary: Benchmark date-time rebasing in ORC datasource Key: SPARK-31311 URL: https://issues.apache.org/jira/browse/SPARK-31311 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31297) Speed-up date-time rebasing

2020-03-29 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070457#comment-17070457 ] Maxim Gekk commented on SPARK-31297: The rebasing of days doesn't depend on time zone, and has just

[jira] [Commented] (SPARK-31297) Speed-up date-time rebasing

2020-03-29 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070286#comment-17070286 ] Maxim Gekk commented on SPARK-31297: [~cloud_fan] [~hyukjin.kwon] [~dongjoon] WDYT? > Speed-up

[jira] [Created] (SPARK-31297) Speed-up date-time rebasing

2020-03-29 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31297: -- Summary: Speed-up date-time rebasing Key: SPARK-31297 URL: https://issues.apache.org/jira/browse/SPARK-31297 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-31296) Benchmark date-time rebasing in Parquet datasource

2020-03-29 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31296: --- Summary: Benchmark date-time rebasing in Parquet datasource (was: Benchmark date-time rebasing

[jira] [Created] (SPARK-31296) Benchmark date-time rebasing to/from Julian calendar

2020-03-29 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31296: -- Summary: Benchmark date-time rebasing to/from Julian calendar Key: SPARK-31296 URL: https://issues.apache.org/jira/browse/SPARK-31296 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31286) Specify formats of time zone ID for JSON/CSV option and from/to_utc_timestamp

2020-03-27 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31286: --- Description: There are two distinct types of ID (see

[jira] [Created] (SPARK-31286) Specify formats of time zone ID for JSON/CSV option and from/to_utc_timestamp

2020-03-27 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31286: -- Summary: Specify formats of time zone ID for JSON/CSV option and from/to_utc_timestamp Key: SPARK-31286 URL: https://issues.apache.org/jira/browse/SPARK-31286 Project:

[jira] [Created] (SPARK-31284) Check rebasing of timestamps in ORC datasource

2020-03-27 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31284: -- Summary: Check rebasing of timestamps in ORC datasource Key: SPARK-31284 URL: https://issues.apache.org/jira/browse/SPARK-31284 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-31277) Migrate `DateTimeTestUtils` from `TimeZone` to `ZoneId`

2020-03-26 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31277: -- Summary: Migrate `DateTimeTestUtils` from `TimeZone` to `ZoneId` Key: SPARK-31277 URL: https://issues.apache.org/jira/browse/SPARK-31277 Project: Spark Issue

[jira] [Created] (SPARK-31254) `HiveResult.toHiveString` does not use the current session time zone

2020-03-25 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31254: -- Summary: `HiveResult.toHiveString` does not use the current session time zone Key: SPARK-31254 URL: https://issues.apache.org/jira/browse/SPARK-31254 Project: Spark

[jira] [Commented] (SPARK-31238) Incompatible ORC dates with Spark 2.4

2020-03-25 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066427#comment-17066427 ] Maxim Gekk commented on SPARK-31238: I am working on the issue. > Incompatible ORC dates with Spark

[jira] [Created] (SPARK-31237) Replace 3-letter time zones by zone offsets

2020-03-24 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31237: -- Summary: Replace 3-letter time zones by zone offsets Key: SPARK-31237 URL: https://issues.apache.org/jira/browse/SPARK-31237 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-31232) Specify formats of `spark.sql.session.timeZone`

2020-03-24 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31232: -- Summary: Specify formats of `spark.sql.session.timeZone` Key: SPARK-31232 URL: https://issues.apache.org/jira/browse/SPARK-31232 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type

2020-03-22 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064550#comment-17064550 ] Maxim Gekk commented on SPARK-31212: I think it would be better to use isLeapYear of

[jira] [Created] (SPARK-31221) Rebase all dates/timestamps in conversion in Java types

2020-03-22 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31221: -- Summary: Rebase all dates/timestamps in conversion in Java types Key: SPARK-31221 URL: https://issues.apache.org/jira/browse/SPARK-31221 Project: Spark Issue

[jira] [Commented] (SPARK-31183) Incompatible Avro dates/timestamps with Spark 2.4

2020-03-22 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064376#comment-17064376 ] Maxim Gekk commented on SPARK-31183: [~koert] The problem will be resolved soon, see

[jira] [Commented] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type

2020-03-21 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064017#comment-17064017 ] Maxim Gekk commented on SPARK-31212: The isLeapYear() function in 2.4 assumes Proleptic Gregorian

[jira] [Created] (SPARK-31212) Failure of casting the '1000-02-29' string to the date type

2020-03-21 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31212: -- Summary: Failure of casting the '1000-02-29' string to the date type Key: SPARK-31212 URL: https://issues.apache.org/jira/browse/SPARK-31212 Project: Spark

[jira] [Created] (SPARK-31211) Failure on loading 1000-02-29 from parquet saved by Spark 2.4.5

2020-03-21 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31211: -- Summary: Failure on loading 1000-02-29 from parquet saved by Spark 2.4.5 Key: SPARK-31211 URL: https://issues.apache.org/jira/browse/SPARK-31211 Project: Spark

[jira] [Created] (SPARK-31195) Reuse days rebase functions of DateTimeUtils in DaysWritable

2020-03-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31195: -- Summary: Reuse days rebase functions of DateTimeUtils in DaysWritable Key: SPARK-31195 URL: https://issues.apache.org/jira/browse/SPARK-31195 Project: Spark

[jira] [Commented] (SPARK-31183) Incompatible Avro dates/timestamps with Spark 2.4

2020-03-18 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061902#comment-17061902 ] Maxim Gekk commented on SPARK-31183: I am working on the issue. > Incompatible Avro

[jira] [Commented] (SPARK-31183) Incompatible Avro dates/timestamps with Spark 2.4

2020-03-18 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061903#comment-17061903 ] Maxim Gekk commented on SPARK-31183: [~cloud_fan] FYI > Incompatible Avro dates/timestamps with

[jira] [Created] (SPARK-31183) Incompatible Avro dates/timestamps with Spark 2.4

2020-03-18 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31183: -- Summary: Incompatible Avro dates/timestamps with Spark 2.4 Key: SPARK-31183 URL: https://issues.apache.org/jira/browse/SPARK-31183 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31159) Incompatible Parquet dates/timestamps with Spark 2.4

2020-03-15 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059617#comment-17059617 ] Maxim Gekk commented on SPARK-31159: [~cloud_fan] FYI > Incompatible Parquet dates/timestamps with

[jira] [Created] (SPARK-31159) Incompatible Parquet dates/timestamps with Spark 2.4

2020-03-15 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31159: -- Summary: Incompatible Parquet dates/timestamps with Spark 2.4 Key: SPARK-31159 URL: https://issues.apache.org/jira/browse/SPARK-31159 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30565) Regression in the ORC benchmark

2020-03-11 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057619#comment-17057619 ] Maxim Gekk commented on SPARK-30565: Per [~dongjoon] , default ORC reader doesn't fully cover

[jira] [Created] (SPARK-31076) Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time

2020-03-06 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31076: -- Summary: Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time Key: SPARK-31076 URL: https://issues.apache.org/jira/browse/SPARK-31076 Project:

[jira] [Created] (SPARK-31044) Support foldable input by `schema_of_json`

2020-03-04 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31044: -- Summary: Support foldable input by `schema_of_json` Key: SPARK-31044 URL: https://issues.apache.org/jira/browse/SPARK-31044 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-30563) Regressions in Join benchmarks

2020-03-04 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051076#comment-17051076 ] Maxim Gekk commented on SPARK-30563: [~petertoth] If you think it is possible to avoid some overhead

[jira] [Commented] (SPARK-30563) Regressions in Join benchmarks

2020-03-04 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051073#comment-17051073 ] Maxim Gekk commented on SPARK-30563: > we spend a lot of time in this loop even The loop just

[jira] [Created] (SPARK-31025) Support foldable input by `schema_of_csv`

2020-03-03 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31025: -- Summary: Support foldable input by `schema_of_csv` Key: SPARK-31025 URL: https://issues.apache.org/jira/browse/SPARK-31025 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-31020) Support foldable schemas by `from_csv`

2020-03-03 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31020: --- Description: Currently, Spark accepts only literals or schema_of_csv w/ literal input as the

[jira] [Created] (SPARK-31023) Support foldable schemas by `from_json`

2020-03-03 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31023: -- Summary: Support foldable schemas by `from_json` Key: SPARK-31023 URL: https://issues.apache.org/jira/browse/SPARK-31023 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-31020) Support foldable schemas by `from_csv`

2020-03-03 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31020: -- Summary: Support foldable schemas by `from_csv` Key: SPARK-31020 URL: https://issues.apache.org/jira/browse/SPARK-31020 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-31005) Support time zone ids in casting strings to timestamps

2020-03-01 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31005: -- Summary: Support time zone ids in casting strings to timestamps Key: SPARK-31005 URL: https://issues.apache.org/jira/browse/SPARK-31005 Project: Spark Issue

[jira] [Created] (SPARK-30988) Add more edge-case exercising values to stats tests

2020-02-28 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30988: -- Summary: Add more edge-case exercising values to stats tests Key: SPARK-30988 URL: https://issues.apache.org/jira/browse/SPARK-30988 Project: Spark Issue Type:

[jira] [Created] (SPARK-30925) Overflow/round errors in conversions of milliseconds to/from microseconds

2020-02-23 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30925: -- Summary: Overflow/round errors in conversions of milliseconds to/from microseconds Key: SPARK-30925 URL: https://issues.apache.org/jira/browse/SPARK-30925 Project: Spark

[jira] [Commented] (SPARK-30894) The behavior of Size function should not depend on SQLConf.get

2020-02-20 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041284#comment-17041284 ] Maxim Gekk commented on SPARK-30894: I am working on it. > The behavior of Size function should not

[jira] [Updated] (SPARK-30892) Exclude spark.sql.variable.substitute.depth from removedSQLConfigs

2020-02-20 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-30892: --- Description: The spark.sql.variable.substitute.depth SQL config is not used since Spark 2.4

[jira] [Created] (SPARK-30892) Exclude spark.sql.variable.substitute.depth from removedSQLConfigs

2020-02-20 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30892: -- Summary: Exclude spark.sql.variable.substitute.depth from removedSQLConfigs Key: SPARK-30892 URL: https://issues.apache.org/jira/browse/SPARK-30892 Project: Spark

[jira] [Comment Edited] (SPARK-30858) IntegralDivide's dataType should not depend on SQLConf.get

2020-02-18 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039433#comment-17039433 ] Maxim Gekk edited comment on SPARK-30858 at 2/18/20 8:29 PM: - The *div*

[jira] [Commented] (SPARK-30858) IntegralDivide's dataType should not depend on SQLConf.get

2020-02-18 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039433#comment-17039433 ] Maxim Gekk commented on SPARK-30858: The *div* function binds on this particular expressions 

[jira] [Commented] (SPARK-30858) IntegralDivide's dataType should not depend on SQLConf.get

2020-02-18 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039410#comment-17039410 ] Maxim Gekk commented on SPARK-30858: > This is a problem because the configuration can change

[jira] [Created] (SPARK-30869) Convert dates to/from timestamps in microseconds precision

2020-02-18 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30869: -- Summary: Convert dates to/from timestamps in microseconds precision Key: SPARK-30869 URL: https://issues.apache.org/jira/browse/SPARK-30869 Project: Spark Issue

[jira] [Created] (SPARK-30865) Refactor DateTimeUtils

2020-02-17 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30865: -- Summary: Refactor DateTimeUtils Key: SPARK-30865 URL: https://issues.apache.org/jira/browse/SPARK-30865 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-30857) Wrong truncations of timestamps before the epoch to hours and days

2020-02-17 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-30857: --- Description: Truncations to seconds and minutes of timestamps after the epoch are correct:

[jira] [Created] (SPARK-30857) Wrong truncations of timestamps before the epoch to hours and days

2020-02-17 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30857: -- Summary: Wrong truncations of timestamps before the epoch to hours and days Key: SPARK-30857 URL: https://issues.apache.org/jira/browse/SPARK-30857 Project: Spark

[jira] [Updated] (SPARK-30793) Wrong truncations of timestamps before the epoch to minutes and seconds

2020-02-17 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-30793: --- Affects Version/s: 2.4.5 > Wrong truncations of timestamps before the epoch to minutes and seconds

[jira] [Resolved] (SPARK-29669) Refactor IntervalUtils.fromDayTimeString()

2020-02-16 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk resolved SPARK-29669. Resolution: Won't Fix > Refactor IntervalUtils.fromDayTimeString() >

[jira] [Resolved] (SPARK-28643) Use Java 8 time API in Dataset.show

2020-02-16 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk resolved SPARK-28643. Resolution: Won't Fix > Use Java 8 time API in Dataset.show > ---

[jira] [Created] (SPARK-30843) Wrong results of getting time components before 1582 year

2020-02-15 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30843: -- Summary: Wrong results of getting time components before 1582 year Key: SPARK-30843 URL: https://issues.apache.org/jira/browse/SPARK-30843 Project: Spark Issue

<    1   2   3   4   5   6   7   8   9   10   >