[jira] [Assigned] (SPARK-31597) extracting day from intervals should be interval.days + days in interval.microsecond

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31597: --- Assignee: Kent Yao > extracting day from intervals should be interval.days + days in > int

[jira] [Resolved] (SPARK-31597) extracting day from intervals should be interval.days + days in interval.microsecond

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31597. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28396 [https://gith

[jira] [Resolved] (SPARK-29492) SparkThriftServer can't support jar class as table serde class when executestatement in sync mode

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29492. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26141 [https://gith

[jira] [Assigned] (SPARK-29492) SparkThriftServer can't support jar class as table serde class when executestatement in sync mode

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29492: --- Assignee: angerszhu > SparkThriftServer can't support jar class as table serde class when

[jira] [Resolved] (SPARK-31596) Generate SQL Configurations from hive module to configuration doc

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-31596. -- Fix Version/s: 3.0.0 Assignee: Kent Yao Resolution: Fixed Resolved by

[jira] [Commented] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095124#comment-17095124 ] angerszhu commented on SPARK-31602: --- cc [~cloud_fan]  > memory leak of JobConf >

[jira] [Updated] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-31602: -- Description: !image-2020-04-29-14-34-39-496.png! !image-2020-04-29-14-35-55-986.png! was: !image-2

[jira] [Updated] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-31602: -- Description: !image-2020-04-29-14-34-39-496.png! Screen Shot 2020-04-29 at 2.08.28 PM was: !image-

[jira] [Updated] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-31602: -- Attachment: image-2020-04-29-14-35-55-986.png > memory leak of JobConf > -- > >

[jira] [Updated] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-31602: -- Attachment: image-2020-04-29-14-34-39-496.png > memory leak of JobConf > -- > >

[jira] [Commented] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095122#comment-17095122 ] angerszhu commented on SPARK-31602: --- In HadoopRDD , if you don't set  spark.hadoop.clo

[jira] [Created] (SPARK-31602) memory leak of JobConf

2020-04-28 Thread angerszhu (Jira)
angerszhu created SPARK-31602: - Summary: memory leak of JobConf Key: SPARK-31602 URL: https://issues.apache.org/jira/browse/SPARK-31602 Project: Spark Issue Type: Bug Components: Spark

[jira] [Created] (SPARK-31601) Fix spark.kubernetes.executor.podNamePrefix to work

2020-04-28 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-31601: - Summary: Fix spark.kubernetes.executor.podNamePrefix to work Key: SPARK-31601 URL: https://issues.apache.org/jira/browse/SPARK-31601 Project: Spark Issue T

[jira] [Commented] (SPARK-31480) Improve the EXPLAIN FORMATTED's output for DSV2's Scan Node

2020-04-28 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095093#comment-17095093 ] Xiao Li commented on SPARK-31480: - let me assign it to [~dkbiswal]  > Improve the EXPLA

[jira] [Assigned] (SPARK-31567) Update AppVeyor Rtools to 4.0.0

2020-04-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31567: Assignee: Dongjoon Hyun > Update AppVeyor Rtools to 4.0.0 > -

[jira] [Resolved] (SPARK-31567) Update AppVeyor Rtools to 4.0.0

2020-04-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31567. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28358 [https://gi

[jira] [Updated] (SPARK-31567) Update AppVeyor Rtools to 4.0.0

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31567: -- Summary: Update AppVeyor Rtools to 4.0.0 (was: Update AppVeyor R version to 4.0.0) > Update

[jira] [Updated] (SPARK-31567) Update AppVeyor Rtools to 4.0.0

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31567: -- Description: This is a preparation for upgrade to R. > Update AppVeyor Rtools to 4.0.0 > -

[jira] [Closed] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-31591. - > namePrefix could be null in Utils.createDirectory > --

[jira] [Resolved] (SPARK-31182) PairRDD support aggregateByKeyWithinPartitions

2020-04-28 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-31182. -- Resolution: Not A Problem > PairRDD support aggregateByKeyWithinPartitions > -

[jira] [Created] (SPARK-31600) Error message from DataFrame creation is misleading.

2020-04-28 Thread Olexiy Oryeshko (Jira)
Olexiy Oryeshko created SPARK-31600: --- Summary: Error message from DataFrame creation is misleading. Key: SPARK-31600 URL: https://issues.apache.org/jira/browse/SPARK-31600 Project: Spark Is

[jira] [Resolved] (SPARK-30261) Should not change owner of hive table for some commands like 'alter' operation

2020-04-28 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30261. -- Target Version/s: 2.4.3, 2.3.0 (was: 2.3.0, 2.4.3) Resolution: Duplicate > Should n

[jira] [Resolved] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-31591. -- Resolution: Not A Problem See: https://github.com/apache/spark/pull/28385#issuecomment

[jira] [Commented] (SPARK-31595) Spark sql cli should allow unescaped quote mark in quoted string

2020-04-28 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094980#comment-17094980 ] Adrian Wang commented on SPARK-31595: - [~Ankitraj] Thanks, I have already created a

[jira] [Resolved] (SPARK-31584) NullPointerException when parsing event log with InMemoryStore

2020-04-28 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-31584. Target Version/s: 3.0.0, 3.0.1, 3.1.0 (was: 3.0.1) Assignee: Baohe Zhang

[jira] [Resolved] (SPARK-31556) Document LIKE clause in SQL Reference

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-31556. -- Fix Version/s: 3.0.0 Assignee: Huaxin Gao Resolution: Fixed Resolved b

[jira] [Commented] (SPARK-26365) spark-submit for k8s cluster doesn't propagate exit code

2020-04-28 Thread Lorenzo Pisani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094932#comment-17094932 ] Lorenzo Pisani commented on SPARK-26365: I'm also seeing this behavior specifica

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

2020-04-28 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094925#comment-17094925 ] Jungtaek Lim commented on SPARK-31599: -- Oh sorry I should guide to user@ mailing li

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

2020-04-28 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094917#comment-17094917 ] Felix Kizhakkel Jose commented on SPARK-31599: -- How do I do that?  > Readi

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

2020-04-28 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094915#comment-17094915 ] Jungtaek Lim commented on SPARK-31599: -- Please post a mail thread on dev@ mailing l

[jira] [Commented] (SPARK-8333) Spark failed to delete temp directory created by HiveContext

2020-04-28 Thread Sunil Kumar Chakrapani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094893#comment-17094893 ] Sunil Kumar Chakrapani commented on SPARK-8333: --- Any plans to fix this issu

[jira] [Updated] (SPARK-31549) Pyspark SparkContext.cancelJobGroup do not work correctly

2020-04-28 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-31549: -- Target Version/s: 3.0.0 > Pyspark SparkContext.cancelJobGroup do not work correctly >

[jira] [Commented] (SPARK-31595) Spark sql cli should allow unescaped quote mark in quoted string

2020-04-28 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094830#comment-17094830 ] Ankit Raj Boudh commented on SPARK-31595: - [~adrian-wang], can i start working o

[jira] [Commented] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094829#comment-17094829 ] Ankit Raj Boudh commented on SPARK-31591: - [~cltlfcjin], It's ok Thank you for r

[jira] [Updated] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

2020-04-28 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31599: - Description: I have a S3 bucket which has data streamed (Parquet format) to it b

[jira] [Created] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

2020-04-28 Thread Felix Kizhakkel Jose (Jira)
Felix Kizhakkel Jose created SPARK-31599: Summary: Reading from S3 (Structured Streaming Bucket) Fails after Compaction Key: SPARK-31599 URL: https://issues.apache.org/jira/browse/SPARK-31599

[jira] [Updated] (SPARK-30741) The data returned from SAS using JDBC reader contains column label

2020-04-28 Thread Gary Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Liu updated SPARK-30741: - Attachment: ExamplesFromSASSupport.png > The data returned from SAS using JDBC reader contains column la

[jira] [Updated] (SPARK-30741) The data returned from SAS using JDBC reader contains column label

2020-04-28 Thread Gary Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Liu updated SPARK-30741: - Attachment: ReplyFromSASSupport.png > The data returned from SAS using JDBC reader contains column label

[jira] [Reopened] (SPARK-30741) The data returned from SAS using JDBC reader contains column label

2020-04-28 Thread Gary Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Liu reopened SPARK-30741: -- *Problem:* The spark JDBC reader read SAS data incorrectly, and returned the data with the column names a

[jira] [Resolved] (SPARK-31339) Changed PipelineModel(...) to self.cls(...) in pyspark.ml.pipeline.PipelineModelReader.load()

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31339. -- Resolution: Not A Problem > Changed PipelineModel(...) to self.cls(...) in > pyspark.ml.pipel

[jira] [Resolved] (SPARK-31165) Multiple wrong references in Dockerfile for k8s

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31165. -- Resolution: Not A Problem > Multiple wrong references in Dockerfile for k8s > ---

[jira] [Resolved] (SPARK-31149) PySpark job not killing Spark Daemon processes after the executor is killed due to OOM

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31149. -- Resolution: Won't Fix > PySpark job not killing Spark Daemon processes after the executor is k

[jira] [Resolved] (SPARK-30133) Support DELETE Jar and DELETE File functionality in spark

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30133. -- Resolution: Won't Fix > Support DELETE Jar and DELETE File functionality in spark > --

[jira] [Resolved] (SPARK-30135) Add documentation for DELETE JAR and DELETE File command

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30135. -- Resolution: Won't Fix > Add documentation for DELETE JAR and DELETE File command > ---

[jira] [Resolved] (SPARK-30134) DELETE JAR should remove from addedJars list and from classpath

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30134. -- Resolution: Won't Fix > DELETE JAR should remove from addedJars list and from classpath > ---

[jira] [Resolved] (SPARK-30137) Support DELETE file

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30137. -- Resolution: Won't Fix > Support DELETE file > > > Key: S

[jira] [Resolved] (SPARK-30136) DELETE JAR should also remove the jar from executor classPath

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30136. -- Resolution: Won't Fix > DELETE JAR should also remove the jar from executor classPath > --

[jira] [Created] (SPARK-31598) LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps

2020-04-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-31598: - Summary: LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps Key: SPARK-31598 URL: https://issues.apache.org/jira/browse/SPARK-31598 Proj

[jira] [Commented] (SPARK-31592) bufferPoolsBySize in HeapMemoryAllocator should be thread safe

2020-04-28 Thread Yunbo Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094669#comment-17094669 ] Yunbo Fan commented on SPARK-31592: --- I checked my executor log again and I find the ex

[jira] [Updated] (SPARK-31592) bufferPoolsBySize in HeapMemoryAllocator should be thread safe

2020-04-28 Thread Yunbo Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunbo Fan updated SPARK-31592: -- Affects Version/s: (was: 2.4.5) 2.4.3 > bufferPoolsBySize in HeapMemoryAllo

[jira] [Resolved] (SPARK-29458) Document scalar functions usage in APIs in SQL getting started.

2020-04-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29458. -- Fix Version/s: 3.0.0 Assignee: Huaxin Gao Resolution: Fixed Resolved by https:

[jira] [Comment Edited] (SPARK-31583) grouping_id calculation should be improved

2020-04-28 Thread Costas Piliotis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094651#comment-17094651 ] Costas Piliotis edited comment on SPARK-31583 at 4/28/20, 4:16 PM: ---

[jira] [Commented] (SPARK-31583) grouping_id calculation should be improved

2020-04-28 Thread Costas Piliotis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094651#comment-17094651 ] Costas Piliotis commented on SPARK-31583: - [~maropu] I'm trying to avoid referen

[jira] [Updated] (SPARK-31519) Cast in having aggregate expressions returns the wrong result

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31519: -- Labels: correctness (was: ) > Cast in having aggregate expressions returns the wrong result >

[jira] [Updated] (SPARK-31534) Text for tooltip should be escaped

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31534: -- Fix Version/s: 3.0.0 > Text for tooltip should be escaped > --

[jira] [Commented] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094627#comment-17094627 ] Dongjoon Hyun commented on SPARK-29048: --- This is reverted via https://github.com/

[jira] [Updated] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29048: -- Fix Version/s: (was: 3.0.0) > Query optimizer slow when using Column.isInCollection() with

[jira] [Reopened] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-29048: --- Assignee: (was: Weichen Xu) > Query optimizer slow when using Column.isInCollection()

[jira] [Updated] (SPARK-31404) file source backward compatibility after calendar switch

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31404: Summary: file source backward compatibility after calendar switch (was: file source backward comp

[jira] [Updated] (SPARK-31404) file source backward compatibility issues after switching to Proleptic Gregorian calendar

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31404: Summary: file source backward compatibility issues after switching to Proleptic Gregorian calendar

[jira] [Commented] (SPARK-31597) extracting day from intervals should be interval.days + days in interval.microsecond

2020-04-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094560#comment-17094560 ] Kent Yao commented on SPARK-31597: -- work log manually [https://github.com/apache/spark/

[jira] [Comment Edited] (SPARK-31597) extracting day from intervals should be interval.days + days in interval.microsecond

2020-04-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094560#comment-17094560 ] Kent Yao edited comment on SPARK-31597 at 4/28/20, 2:30 PM:

[jira] [Assigned] (SPARK-31553) Wrong result of isInCollection for large collections

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31553: --- Assignee: Maxim Gekk > Wrong result of isInCollection for large collections > -

[jira] [Resolved] (SPARK-31553) Wrong result of isInCollection for large collections

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31553. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28388 [https://gith

[jira] [Resolved] (SPARK-31586) Replace expression TimeSub(l, r) with TimeAdd(l -r)

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31586. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28381 [https://gith

[jira] [Assigned] (SPARK-31586) Replace expression TimeSub(l, r) with TimeAdd(l -r)

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31586: --- Assignee: Kent Yao > Replace expression TimeSub(l, r) with TimeAdd(l -r) >

[jira] [Created] (SPARK-31597) extracting day from intervals should be interval.days + days in interval.microsecond

2020-04-28 Thread Kent Yao (Jira)
Kent Yao created SPARK-31597: Summary: extracting day from intervals should be interval.days + days in interval.microsecond Key: SPARK-31597 URL: https://issues.apache.org/jira/browse/SPARK-31597 Project:

[jira] [Updated] (SPARK-31596) Generate SQL Configurations from hive module to configuration doc

2020-04-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-31596: - Description: ATT > Generate SQL Configurations from hive module to configuration doc > -

[jira] [Created] (SPARK-31596) Generate SQL Configurations from hive module to configuration doc

2020-04-28 Thread Kent Yao (Jira)
Kent Yao created SPARK-31596: Summary: Generate SQL Configurations from hive module to configuration doc Key: SPARK-31596 URL: https://issues.apache.org/jira/browse/SPARK-31596 Project: Spark Is

[jira] [Created] (SPARK-31595) Spark sql cli should allow unescaped quote mark in quoted string

2020-04-28 Thread Adrian Wang (Jira)
Adrian Wang created SPARK-31595: --- Summary: Spark sql cli should allow unescaped quote mark in quoted string Key: SPARK-31595 URL: https://issues.apache.org/jira/browse/SPARK-31595 Project: Spark

[jira] [Commented] (SPARK-31594) Do not display rand/randn seed numbers in schema

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094362#comment-17094362 ] Takeshi Yamamuro commented on SPARK-31594: -- I'm working on this https://github.

[jira] [Updated] (SPARK-31594) Do not display rand/randn seed numbers in schema

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-31594: - Summary: Do not display rand/randn seed numbers in schema (was: Do not display rand/ran

[jira] [Created] (SPARK-31594) Do not display rand/randn seed in schema

2020-04-28 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-31594: Summary: Do not display rand/randn seed in schema Key: SPARK-31594 URL: https://issues.apache.org/jira/browse/SPARK-31594 Project: Spark Issue Type:

[jira] [Created] (SPARK-31593) Remove unnecessary streaming query progress update

2020-04-28 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-31593: - Summary: Remove unnecessary streaming query progress update Key: SPARK-31593 URL: https://issues.apache.org/jira/browse/SPARK-31593 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-31592) bufferPoolsBySize in HeapMemoryAllocator should be thread safe

2020-04-28 Thread Yunbo Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunbo Fan updated SPARK-31592: -- Description: Currently, bufferPoolsBySize in HeapMemoryAllocator uses a Map type whose value type is 

[jira] [Created] (SPARK-31592) bufferPoolsBySize in HeapMemoryAllocator should be thread safe

2020-04-28 Thread Yunbo Fan (Jira)
Yunbo Fan created SPARK-31592: - Summary: bufferPoolsBySize in HeapMemoryAllocator should be thread safe Key: SPARK-31592 URL: https://issues.apache.org/jira/browse/SPARK-31592 Project: Spark Iss

[jira] [Resolved] (SPARK-26924) Fix CRAN hack as soon as Arrow is available on CRAN

2020-04-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26924. -- Resolution: Duplicate > Fix CRAN hack as soon as Arrow is available on CRAN >

[jira] [Commented] (SPARK-31583) grouping_id calculation should be improved

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094284#comment-17094284 ] Takeshi Yamamuro commented on SPARK-31583: -- [~cpiliotis] Hi, thanks for your re

[jira] [Comment Edited] (SPARK-31583) grouping_id calculation should be improved

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094284#comment-17094284 ] Takeshi Yamamuro edited comment on SPARK-31583 at 4/28/20, 8:28 AM: --

[jira] [Updated] (SPARK-31573) Use fixed=TRUE where possible for internal efficiency

2020-04-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31573: - Issue Type: Bug (was: Documentation) > Use fixed=TRUE where possible for internal efficiency >

[jira] [Resolved] (SPARK-31573) Use fixed=TRUE where possible for internal efficiency

2020-04-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31573. -- Assignee: Michael Chirico Resolution: Fixed Fixed in https://github.com/apache/spark/pul

[jira] [Updated] (SPARK-31573) Use fixed=TRUE where possible for internal efficiency

2020-04-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31573: - Fix Version/s: 3.0.0 > Use fixed=TRUE where possible for internal efficiency > -

[jira] [Resolved] (SPARK-31519) Cast in having aggregate expressions returns the wrong result

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31519. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28294 [https://gith

[jira] [Assigned] (SPARK-31519) Cast in having aggregate expressions returns the wrong result

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31519: --- Assignee: Yuanjian Li > Cast in having aggregate expressions returns the wrong result > ---

[jira] [Commented] (SPARK-30868) Throw Exception if runHive(sql) failed

2020-04-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094274#comment-17094274 ] Yuming Wang commented on SPARK-30868: - Issue resolved by pull request 27644 https://

[jira] [Resolved] (SPARK-30868) Throw Exception if runHive(sql) failed

2020-04-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-30868. - Fix Version/s: 3.0.0 Resolution: Fixed > Throw Exception if runHive(sql) failed > ---

[jira] [Assigned] (SPARK-30868) Throw Exception if runHive(sql) failed

2020-04-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-30868: --- Assignee: Jackey Lee > Throw Exception if runHive(sql) failed > ---

[jira] [Resolved] (SPARK-31524) Add metric to the split number for skew partition when enable AQE

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31524. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28109 [https://gith

[jira] [Assigned] (SPARK-31524) Add metric to the split number for skew partition when enable AQE

2020-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31524: --- Assignee: Ke Jia > Add metric to the split number for skew partition when enable AQE > ---

[jira] [Commented] (SPARK-26199) Long expressions cause mutate to fail

2020-04-28 Thread Michael Chirico (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094232#comment-17094232 ] Michael Chirico commented on SPARK-26199: - Just saw this. https://issues.apach

[jira] [Commented] (SPARK-31586) Replace expression TimeSub(l, r) with TimeAdd(l -r)

2020-04-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094231#comment-17094231 ] Kent Yao commented on SPARK-31586: -- Hi [~Ankitraj],  PR is ready [https://github.com/ap

[jira] [Updated] (SPARK-31589) Use `r-lib/actions/setup-r` in GitHub Action

2020-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31589: -- Fix Version/s: 2.4.6 > Use `r-lib/actions/setup-r` in GitHub Action >

[jira] [Updated] (SPARK-31583) grouping_id calculation should be improved

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-31583: - Component/s: (was: Spark Core) SQL > grouping_id calculation should

[jira] [Updated] (SPARK-31583) grouping_id calculation should be improved

2020-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-31583: - Affects Version/s: (was: 2.4.5) 3.1.0 > grouping_id calculati

[jira] [Commented] (SPARK-31586) Replace expression TimeSub(l, r) with TimeAdd(l -r)

2020-04-28 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094217#comment-17094217 ] Ankit Raj Boudh commented on SPARK-31586: - Hi Kent Yao, are you working on this

[jira] [Commented] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094215#comment-17094215 ] Lantao Jin commented on SPARK-31591: https://github.com/apache/spark/pull/28385 > n

[jira] [Commented] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094214#comment-17094214 ] Lantao Jin commented on SPARK-31591: [~Ankitraj] I have already filed a PR. > nameP