[jira] [Updated] (SPARK-30400) Test failure in SQL module on ppc64le

2020-01-09 Thread AK97 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] AK97 updated SPARK-30400: - Shepherd: Yin Huai > Test failure in SQL module on ppc64le > - > >

[jira] [Updated] (SPARK-30400) Test failure in SQL module on ppc64le

2020-01-09 Thread AK97 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] AK97 updated SPARK-30400: - Shepherd: (was: Yin Huai) > Test failure in SQL module on ppc64le > - > >

[jira] [Updated] (SPARK-30400) Test failure in SQL module on ppc64le

2020-01-09 Thread AK97 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] AK97 updated SPARK-30400: - Shepherd: Yin Huai Environment: os: rhel 7.6 arch: ppc64le was: os: rhel 7.6 arch: ppc64le > Test f

[jira] [Commented] (SPARK-30400) Test failure in SQL module on ppc64le

2020-01-09 Thread AK97 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012542#comment-17012542 ] AK97 commented on SPARK-30400: -- Any Leads will be appreciated. > Test failure in SQL modu

[jira] [Created] (SPARK-30481) Integrate event log compactor into Spark History Server

2020-01-09 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-30481: Summary: Integrate event log compactor into Spark History Server Key: SPARK-30481 URL: https://issues.apache.org/jira/browse/SPARK-30481 Project: Spark Issue

[jira] [Resolved] (SPARK-30480) Pyspark test "test_memory_limit" fails consistently

2020-01-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30480. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed in [https://github.com/apache/spark/pull/2

[jira] [Closed] (SPARK-29776) rpad and lpad should return NULL when padstring parameter is empty

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-29776. - > rpad and lpad should return NULL when padstring parameter is empty > -

[jira] [Created] (SPARK-30480) Pyspark test "test_memory_limit" fails consistently

2020-01-09 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-30480: Summary: Pyspark test "test_memory_limit" fails consistently Key: SPARK-30480 URL: https://issues.apache.org/jira/browse/SPARK-30480 Project: Spark Issue Typ

[jira] [Updated] (SPARK-27686) Update migration guide

2020-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27686: Attachment: hive-1.2.1-lib.tgz > Update migration guide > --- > >

[jira] [Created] (SPARK-30479) Apply compaction of event log to SQL events

2020-01-09 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-30479: Summary: Apply compaction of event log to SQL events Key: SPARK-30479 URL: https://issues.apache.org/jira/browse/SPARK-30479 Project: Spark Issue Type: Sub-t

[jira] [Updated] (SPARK-29779) Compact old event log files and clean up

2020-01-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-29779: - Description: This issue is to track the effort on compacting old event logs (and cleaning up af

[jira] [Updated] (SPARK-30477) More KeyValueGroupedDataset methods should be composable

2020-01-09 Thread Paul Jones (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Jones updated SPARK-30477: --- Description: Right now many `KeyValueGroupedDataset` do not return a  `KeyValueGroupedDataset`. In s

[jira] [Updated] (SPARK-27686) Update migration guide

2020-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27686: Description: The built-in Hive 2.3 fixes the following issues: * HIVE-6727: Table level stats for

[jira] [Updated] (SPARK-27686) Update migration guide

2020-01-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27686: Parent Issue: SPARK-30034 (was: SPARK-23710) > Update migration guide > ---

[jira] [Updated] (SPARK-30474) Writing data to parquet with dynamic partitionOverwriteMode should not do the folder rename in commitjob stage

2020-01-09 Thread Zaisheng Dai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zaisheng Dai updated SPARK-30474: - Description: In the current spark implementation if you set, {code:java} spark.sql.sources.parti

[jira] [Updated] (SPARK-30474) Writing data to parquet with dynamic partitionOverwriteMode should not do the folder rename in commitjob stage

2020-01-09 Thread Zaisheng Dai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zaisheng Dai updated SPARK-30474: - Summary: Writing data to parquet with dynamic partitionOverwriteMode should not do the folder re

[jira] [Commented] (SPARK-27686) Update migration guide

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012434#comment-17012434 ] Dongjoon Hyun commented on SPARK-27686: --- Hi, [~yumwang]. Can we have this document

[jira] [Commented] (SPARK-30441) Improve the memory usage in StronglyConnectedComponents

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012432#comment-17012432 ] Dongjoon Hyun commented on SPARK-30441: --- Hi, [~jmzhou]. Please don't set `Fixed Ve

[jira] [Updated] (SPARK-30441) Improve the memory usage in StronglyConnectedComponents

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30441: -- Target Version/s: (was: 3.0.0) > Improve the memory usage in StronglyConnectedComponents > -

[jira] [Updated] (SPARK-30441) Improve the memory usage in StronglyConnectedComponents

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30441: -- Flags: (was: Important) > Improve the memory usage in StronglyConnectedComponents >

[jira] [Updated] (SPARK-30441) Improve the memory usage in StronglyConnectedComponents

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30441: -- Affects Version/s: (was: 2.4.4) (was: 2.4.0)

[jira] [Updated] (SPARK-30441) Improve the memory usage in StronglyConnectedComponents

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30441: -- Fix Version/s: (was: 3.0.0) > Improve the memory usage in StronglyConnectedComponents > --

[jira] [Commented] (SPARK-30296) Dataset diffing transformation

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012430#comment-17012430 ] Dongjoon Hyun commented on SPARK-30296: --- Hi, [~EnricoMi]. Please don't set `Fixed

[jira] [Updated] (SPARK-30296) Dataset diffing transformation

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30296: -- Affects Version/s: (was: 2.4.4) 3.0.0 > Dataset diffing transformat

[jira] [Updated] (SPARK-30296) Dataset diffing transformation

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30296: -- Fix Version/s: (was: 3.0.0) > Dataset diffing transformation > ---

[jira] [Updated] (SPARK-25017) Add test suite for ContextBarrierState

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25017: -- Target Version/s: (was: 3.0.0) > Add test suite for ContextBarrierState > --

[jira] [Created] (SPARK-30478) update memory package doc

2020-01-09 Thread SongXun (Jira)
SongXun created SPARK-30478: --- Summary: update memory package doc Key: SPARK-30478 URL: https://issues.apache.org/jira/browse/SPARK-30478 Project: Spark Issue Type: Improvement Components

[jira] [Updated] (SPARK-30131) Add array_median function

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30131: -- Fix Version/s: (was: 3.0.0) > Add array_median function > - > >

[jira] [Updated] (SPARK-30131) Add array_median function

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30131: -- Target Version/s: (was: 2.4.4) > Add array_median function > - > >

[jira] [Resolved] (SPARK-30034) Use Apache Hive 2.3 dependency by default

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30034. --- Fix Version/s: 3.0.0 Resolution: Done > Use Apache Hive 2.3 dependency by default > -

[jira] [Resolved] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2020-01-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29988. --- Fix Version/s: 3.0.0 Resolution: Fixed Thank you. It looks working. I'll monitor them

[jira] [Created] (SPARK-30477) More KeyValueGroupedDataset methods should be composable

2020-01-09 Thread Paul Jones (Jira)
Paul Jones created SPARK-30477: -- Summary: More KeyValueGroupedDataset methods should be composable Key: SPARK-30477 URL: https://issues.apache.org/jira/browse/SPARK-30477 Project: Spark Issue Ty

[jira] [Commented] (SPARK-28396) Add PathCatalog for data source V2

2020-01-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012381#comment-17012381 ] Gengliang Wang commented on SPARK-28396: [~jerrychenhf] they are still handled b

[jira] [Created] (SPARK-30476) NullPointException when Insert data to hive mongo external table by spark-sql

2020-01-09 Thread XiongCheng (Jira)
XiongCheng created SPARK-30476: -- Summary: NullPointException when Insert data to hive mongo external table by spark-sql Key: SPARK-30476 URL: https://issues.apache.org/jira/browse/SPARK-30476 Project: Sp

[jira] [Resolved] (SPARK-30439) support NOT NULL in column data type

2020-01-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30439. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27110 [https://gi

[jira] [Resolved] (SPARK-30416) Log a warning for deprecated SQL config in `set()` and `unset()`

2020-01-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30416. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27092 [https://gi

[jira] [Assigned] (SPARK-30416) Log a warning for deprecated SQL config in `set()` and `unset()`

2020-01-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30416: Assignee: Maxim Gekk > Log a warning for deprecated SQL config in `set()` and `unset()` >

[jira] [Updated] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30468: - Description: Currently data columns are displayed in one line for show create table command, wh

[jira] [Commented] (SPARK-28396) Add PathCatalog for data source V2

2020-01-09 Thread Haifeng Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012336#comment-17012336 ] Haifeng Chen commented on SPARK-28396: -- [~Gengliang.Wang] Gengliang, I am trying to

[jira] [Resolved] (SPARK-24714) AnalysisSuite should use ClassTag to check the runtime instance

2020-01-09 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24714. -- Resolution: Won't Fix > AnalysisSuite should use ClassTag to check the runtime instanc

[jira] [Commented] (SPARK-24714) AnalysisSuite should use ClassTag to check the runtime instance

2020-01-09 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012323#comment-17012323 ] Takeshi Yamamuro commented on SPARK-24714: -- I'll close this because the corresp

[jira] [Updated] (SPARK-30475) File source V2: Push data filters for file listing

2020-01-09 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-30475: --- Description: Follow up on [SPARK-30428|https://github.com/apache/spark/pull/27112] which added supp

[jira] [Commented] (SPARK-30475) File source V2: Push data filters for file listing

2020-01-09 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012298#comment-17012298 ] Guy Khazma commented on SPARK-30475: PR https://github.com/apache/spark/pull/27157

[jira] [Updated] (SPARK-30475) File source V2: Push data filters for file listing

2020-01-09 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-30475: --- External issue URL: https://github.com/apache/spark/pull/27157 > File source V2: Push data filters f

[jira] [Updated] (SPARK-30475) File source V2: Push data filters for file listing

2020-01-09 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-30475: --- External issue URL: (was: https://github.com/apache/spark/pull/27157) > File source V2: Push data

[jira] [Created] (SPARK-30475) File source V2: Push data filters for file listing

2020-01-09 Thread Guy Khazma (Jira)
Guy Khazma created SPARK-30475: -- Summary: File source V2: Push data filters for file listing Key: SPARK-30475 URL: https://issues.apache.org/jira/browse/SPARK-30475 Project: Spark Issue Type: Im

[jira] [Updated] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2020-01-09 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29988: Attachment: Screen Shot 2020-01-09 at 1.59.25 PM.png > Adjust Jenkins jobs for `hive-1.2/2.3` comb

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2020-01-09 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012261#comment-17012261 ] Shane Knapp commented on SPARK-29988: - it's hard to tell but i disabled the old jobs

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2020-01-09 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012260#comment-17012260 ] Shane Knapp commented on SPARK-29988: - done!   !Screen Shot 2020-01-09 at 1.59.25

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2020-01-09 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012246#comment-17012246 ] Shane Knapp commented on SPARK-29988: - ok, after banging my head against jenkins job

[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2020-01-09 Thread Everett Rush (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012229#comment-17012229 ] Everett Rush commented on SPARK-27249: -- [~nafshartous]  Hi Nick,   I would like

[jira] [Comment Edited] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2020-01-09 Thread Nick Afshartous (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010756#comment-17010756 ] Nick Afshartous edited comment on SPARK-27249 at 1/9/20 7:49 PM: -

[jira] [Resolved] (SPARK-30459) Fix ignoreMissingFiles/ignoreCorruptFiles in DSv2

2020-01-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-30459. Resolution: Fixed This issue is resolved in https://github.com/apache/spark/pull/27136 >

[jira] [Updated] (SPARK-30459) Fix ignoreMissingFiles/ignoreCorruptFiles in DSv2

2020-01-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-30459: --- Issue Type: Bug (was: Improvement) > Fix ignoreMissingFiles/ignoreCorruptFiles in DSv2 > --

[jira] [Assigned] (SPARK-30459) Fix ignoreMissingFiles/ignoreCorruptFiles in DSv2

2020-01-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-30459: -- Assignee: wuyi > Fix ignoreMissingFiles/ignoreCorruptFiles in DSv2 >

[jira] [Resolved] (SPARK-29219) DataSourceV2: Support all SaveModes in DataFrameWriter.save

2020-01-09 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29219. - Fix Version/s: 3.0.0 Resolution: Done Resolved by [https://github.com/apache/spark/pull/2

[jira] [Assigned] (SPARK-29219) DataSourceV2: Support all SaveModes in DataFrameWriter.save

2020-01-09 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-29219: --- Assignee: Burak Yavuz > DataSourceV2: Support all SaveModes in DataFrameWriter.save > -

[jira] [Updated] (SPARK-30474) Writing data to parquet with dynamic partition should not be done in commit job stage

2020-01-09 Thread Zaisheng Dai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zaisheng Dai updated SPARK-30474: - Description: In the current spark implementation if you set, {code:java} spark.sql.sources.parti

[jira] [Updated] (SPARK-30474) Writing data to parquet with dynamic partition should not be done in commit job stage

2020-01-09 Thread Zaisheng Dai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zaisheng Dai updated SPARK-30474: - Description: In the current spark implementation if you set spark.sql.sources.partitionOverwrit

[jira] [Created] (SPARK-30474) Writing data to parquet with dynamic partition should not be done in commit job stage

2020-01-09 Thread Zaisheng Dai (Jira)
Zaisheng Dai created SPARK-30474: Summary: Writing data to parquet with dynamic partition should not be done in commit job stage Key: SPARK-30474 URL: https://issues.apache.org/jira/browse/SPARK-30474

[jira] [Updated] (SPARK-30467) On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers are not able to connect to Remote Master.

2020-01-09 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-30467: --- Description: On _*Federal Information Processing Standard*_ (FIPS) enabled clusters, Spark

[jira] [Updated] (SPARK-30467) On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers are not able to connect to Remote Master.

2020-01-09 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-30467: --- Priority: Blocker (was: Major) > On Federal Information Processing Standard (FIPS) enabled

[jira] [Updated] (SPARK-30473) PySpark enum subclass crashes when used inside UDF

2020-01-09 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-30473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Härtwig updated SPARK-30473: Description: PySpark enum subclass crashes when used inside a UDF.   Example: {code:java} from e

[jira] [Updated] (SPARK-30473) PySpark enum subclass crashes when used inside UDF

2020-01-09 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-30473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Härtwig updated SPARK-30473: Description: PySpark enum subclass crashes when used inside a UDF.   Example: {code:java} from e

[jira] [Created] (SPARK-30473) PySpark enum subclass crashes when used inside UDF

2020-01-09 Thread Jira
Max Härtwig created SPARK-30473: --- Summary: PySpark enum subclass crashes when used inside UDF Key: SPARK-30473 URL: https://issues.apache.org/jira/browse/SPARK-30473 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30472) [SQL] ANSI SQL: Throw exception on format invalid and overflow when casting String to IntegerType.

2020-01-09 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-30472: Summary: [SQL] ANSI SQL: Throw exception on format invalid and overflow when casting String to IntegerType

[jira] [Updated] (SPARK-30467) On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers are not able to connect to Remote Master.

2020-01-09 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-30467: --- Description: On _*Federal Information Processing Standard*_ (FIPS) enabled clusters, Spark

[jira] [Updated] (SPARK-30467) On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers are not able to connect to Remote Master.

2020-01-09 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-30467: --- Description: On _*Federal Information Processing Standard*_ (FIPS) enabled clusters, Spark

[jira] [Updated] (SPARK-30467) On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers are not able to connect to Remote Master.

2020-01-09 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-30467: --- Summary: On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers ar

[jira] [Resolved] (SPARK-30452) Add predict and numFeatures in Python IsotonicRegressionModel

2020-01-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30452. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27122 [https://gi

[jira] [Assigned] (SPARK-30452) Add predict and numFeatures in Python IsotonicRegressionModel

2020-01-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30452: Assignee: Huaxin Gao > Add predict and numFeatures in Python IsotonicRegressionModel > --

[jira] [Commented] (SPARK-30421) Dropped columns still available for filtering

2020-01-09 Thread Tobias Hermann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011941#comment-17011941 ] Tobias Hermann commented on SPARK-30421: It allows you to write code that should

[jira] [Created] (SPARK-30472) ANSI SQL: Cast String to Integer Type, throw exception on format invalid and overflow.

2020-01-09 Thread feiwang (Jira)
feiwang created SPARK-30472: --- Summary: ANSI SQL: Cast String to Integer Type, throw exception on format invalid and overflow. Key: SPARK-30472 URL: https://issues.apache.org/jira/browse/SPARK-30472 Project:

[jira] [Commented] (SPARK-30421) Dropped columns still available for filtering

2020-01-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011930#comment-17011930 ] Wenchen Fan commented on SPARK-30421: - but it will not break anything, right? It jus

[jira] [Issue Comment Deleted] (SPARK-28317) Built-in Mathematical Functions: SCALE

2020-01-09 Thread Oleg Bonar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Bonar updated SPARK-28317: --- Comment: was deleted (was: HI [~shivuson...@gmail.com]! Have you made any progress on the issue?) >

[jira] [Resolved] (SPARK-30428) File source V2: support partition pruning

2020-01-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30428. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27112 [https://gith

[jira] [Updated] (SPARK-30471) Fix issue when compare string and IntegerType

2020-01-09 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-30471: Description: When we comparing a String Type and IntegerType: '2147483648'(StringType, which exceed Int.Ma

[jira] [Updated] (SPARK-30471) Fix issue when compare string and IntegerType

2020-01-09 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-30471: Description: When we comparing a String Type and IntegerType: '2147483648'(StringType, which exceed Int.Ma

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2020-01-09 Thread Rafat (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011732#comment-17011732 ] Rafat commented on SPARK-10816: --- same question above  Is there SLA for this feature ? Th

[jira] [Created] (SPARK-30471) Fix issue when compare string and IntegerType

2020-01-09 Thread feiwang (Jira)
feiwang created SPARK-30471: --- Summary: Fix issue when compare string and IntegerType Key: SPARK-30471 URL: https://issues.apache.org/jira/browse/SPARK-30471 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30470) Uncache table in tempViews if needed on session closed

2020-01-09 Thread liupengcheng (Jira)
liupengcheng created SPARK-30470: Summary: Uncache table in tempViews if needed on session closed Key: SPARK-30470 URL: https://issues.apache.org/jira/browse/SPARK-30470 Project: Spark Issue

[jira] [Updated] (SPARK-30469) Partition columns should not be involved when calculating sizeInBytes of Project logical plan

2020-01-09 Thread Hu Fuwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30469: -- Description: When getting the statistics of a Project logical plan, if CBO not enabled, Spark will ca

[jira] [Updated] (SPARK-30469) Hive Partition columns should not be involved when calculating sizeInBytes of Project logical plan

2020-01-09 Thread Hu Fuwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30469: -- Description: When getting the statistics of a Project logical plan, if CBO not enabled, Spark will ca

[jira] [Updated] (SPARK-30469) Partition columns should not be involved when calculating sizeInBytes of Project logical plan

2020-01-09 Thread Hu Fuwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30469: -- Summary: Partition columns should not be involved when calculating sizeInBytes of Project logical plan

[jira] [Created] (SPARK-30469) Hive Partition columns should not be involved when calculating sizeInBytes of Project logical plan

2020-01-09 Thread Hu Fuwang (Jira)
Hu Fuwang created SPARK-30469: - Summary: Hive Partition columns should not be involved when calculating sizeInBytes of Project logical plan Key: SPARK-30469 URL: https://issues.apache.org/jira/browse/SPARK-30469

[jira] [Updated] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30468: - Description: Currently data columns are displayed in one line for show create table command, wh

[jira] [Updated] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30468: - Description: Currently data columns are displayed in one line for show create table command, wh

[jira] [Updated] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30468: - Description: Currently data columns are displayed in one line for show create table command, wh

[jira] [Updated] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30468: - Description: Currently data columns are displayed in one line for show create table command, wh

[jira] [Updated] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30468: - Description: Currently data columns are displayed in one line for show create table command, whe

[jira] [Created] (SPARK-30468) Use multiple lines to display data columns for show create table command

2020-01-09 Thread Zhenhua Wang (Jira)
Zhenhua Wang created SPARK-30468: Summary: Use multiple lines to display data columns for show create table command Key: SPARK-30468 URL: https://issues.apache.org/jira/browse/SPARK-30468 Project: Spa

[jira] [Commented] (SPARK-28883) Fix a flaky test: ThriftServerQueryTestSuite

2020-01-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011533#comment-17011533 ] Jungtaek Lim commented on SPARK-28883: -- Would SPARK-30345 be a complement of this?