[jira] [Commented] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2019-04-02 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808388#comment-16808388 ] vaquar khan commented on SPARK-21797: - {quote}Issue is related to AWS storage class not Apache Spark

[jira] [Resolved] (SPARK-27255) Aggregate functions should not be allowed in WHERE

2019-04-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27255. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24209

[jira] [Assigned] (SPARK-27255) Aggregate functions should not be allowed in WHERE

2019-04-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27255: --- Assignee: Dilip Biswal > Aggregate functions should not be allowed in WHERE >

[jira] [Commented] (SPARK-27342) Optimize limit 0 queries

2019-04-02 Thread Aayushmaan Jain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808369#comment-16808369 ] Aayushmaan Jain commented on SPARK-27342: - [~Francis47] Since the right child of Left Outer Join

[jira] [Updated] (SPARK-27349) Dealing with TimeVars removed in Hive 2.x

2019-04-02 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27349: Description: {{hive.stats.jdbc.timeout}} and {{hive.stats.retries.wait}}  were removed by 

[jira] [Updated] (SPARK-27349) Dealing with TimeVars removed in Hive 2.x

2019-04-02 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27349: Description: {{hive.stats.jdbc.timeout}} and {{hive.stats.retries.wait}}  were removed by 

[jira] [Commented] (SPARK-27345) Expose approx_percentile in python api

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808353#comment-16808353 ] Hyukjin Kwon commented on SPARK-27345: -- you can easily use it via {{expr(...)}}. > Expose

[jira] [Commented] (SPARK-27349) Dealing with TimeVars removed in Hive 2.x

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808352#comment-16808352 ] Hyukjin Kwon commented on SPARK-27349: -- [~yumwang], I know it's tedious to write up the description

[jira] [Created] (SPARK-27350) Support create table on data source V2

2019-04-02 Thread Xianyang Liu (JIRA)
Xianyang Liu created SPARK-27350: Summary: Support create table on data source V2 Key: SPARK-27350 URL: https://issues.apache.org/jira/browse/SPARK-27350 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-25496) Deprecate from_utc_timestamp and to_utc_timestamp

2019-04-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25496. - Resolution: Fixed Issue resolved by pull request 24195

[jira] [Created] (SPARK-27349) Dealing with TimeVars removed in Hive 2.x

2019-04-02 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27349: --- Summary: Dealing with TimeVars removed in Hive 2.x Key: SPARK-27349 URL: https://issues.apache.org/jira/browse/SPARK-27349 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-27346) Loosen the newline assert condition on 'examples' field in ExpressionInfo

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27346: Assignee: Hyukjin Kwon > Loosen the newline assert condition on 'examples' field in

[jira] [Resolved] (SPARK-27346) Loosen the newline assert condition on 'examples' field in ExpressionInfo

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27346. -- Resolution: Fixed Fix Version/s: 2.4.2 3.0.0 Issue resolved by pull

[jira] [Commented] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808194#comment-16808194 ] Hyukjin Kwon commented on SPARK-25003: -- This session stuff logic is a bit convoluted and many

[jira] [Resolved] (SPARK-11702) Guava ClassLoading Issue When Using Different Hive Metastore Version

2019-04-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11702. Resolution: Cannot Reproduce Pretty sure this does not happen in the 2.x line. > Guava

[jira] [Resolved] (SPARK-25762) Upgrade guava version in spark dependency lists due to CVE issue

2019-04-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25762. Resolution: Duplicate Target Version/s: (was: 3.0.0) > Upgrade guava version

[jira] [Assigned] (SPARK-27180) Fix testing issues with yarn module in Hadoop-3

2019-04-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27180: - Assignee: Yuming Wang > Fix testing issues with yarn module in Hadoop-3 >

[jira] [Resolved] (SPARK-27180) Fix testing issues with yarn module in Hadoop-3

2019-04-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27180. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24115

[jira] [Comment Edited] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-04-02 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808011#comment-16808011 ] Jacek Lewandowski edited comment on SPARK-27220 at 4/2/19 7:52 PM: ---

[jira] [Updated] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27348: - Description: When a heartbeat timeout happens in HeartbeatReceiver, it doesn't remove lost

[jira] [Updated] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27348: - Description: When a heartbeat timeout happens in HeartbeatReceiver, it doesn't remove lost

[jira] [Created] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-02 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27348: Summary: HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend Key: SPARK-27348 URL: https://issues.apache.org/jira/browse/SPARK-27348

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-04-02 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808011#comment-16808011 ] Jacek Lewandowski commented on SPARK-27220: --- Thank you [~irashid]. It is not a custom fork

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-04-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808000#comment-16808000 ] Imran Rashid commented on SPARK-27220: -- {quote} because we add something to executor id and it ends

[jira] [Commented] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

2019-04-02 Thread Sunitha Kambhampati (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807961#comment-16807961 ] Sunitha Kambhampati commented on SPARK-25003: - Thanks [~rspitzer].  That is good to know. 

[jira] [Closed] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2019-04-02 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp closed SPARK-23874. --- > Upgrade apache/arrow to 0.10.0 > -- > > Key: SPARK-23874

[jira] [Resolved] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests

2019-04-02 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-26775. - Resolution: Fixed this is done > Update Jenkins nodes to support local volumes for K8s

[jira] [Resolved] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-04-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26998. Resolution: Fixed Fix Version/s: 2.3.4 2.4.2

[jira] [Assigned] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-04-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26998: -- Assignee: Gabor Somogyi > spark.ssl.keyStorePassword in plaintext on 'ps -ef' output

[jira] [Updated] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-02 Thread Sam Tran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tran updated SPARK-27347: - Description: Ran into scenarios where {{--supervised}} Spark jobs were retried multiple times when an

[jira] [Updated] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-02 Thread Sam Tran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tran updated SPARK-27347: - Description: Ran into scenarios where {{--supervised}} Spark jobs were retried multiple times when an

[jira] [Updated] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-02 Thread Sam Tran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tran updated SPARK-27347: - Description: Ran into scenarios where {{--supervised}} Spark jobs were retried multiple times when an

[jira] [Updated] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-02 Thread Sam Tran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tran updated SPARK-27347: - Description: Ran into scenarios where {{--supervised}} Spark jobs were retried multiple times when an

[jira] [Created] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-02 Thread Sam Tran (JIRA)
Sam Tran created SPARK-27347: Summary: [MESOS] Fix supervised driver retry logic when agent crashes/restarts Key: SPARK-27347 URL: https://issues.apache.org/jira/browse/SPARK-27347 Project: Spark

[jira] [Commented] (SPARK-24839) Incorrect drop of lit() column results in cross join

2019-04-02 Thread Cheng Yichao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807840#comment-16807840 ] Cheng Yichao commented on SPARK-24839: -- Any updates to this issue? > Incorrect drop of lit()

[jira] [Resolved] (SPARK-27323) Use Single-Abstract-Method support in Scala 2.12 to simplify code

2019-04-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27323. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via 

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-02 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807780#comment-16807780 ] Marco Gaido commented on SPARK-27287: - I think the problem here is that the configuration are copied

[jira] [Commented] (SPARK-21454) Decimal up cast to higher scale fails while reading parquet to Dataset

2019-04-02 Thread Bill Schneider (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807772#comment-16807772 ] Bill Schneider commented on SPARK-21454: I have the same issue with Spark 2.4.0. The argument

[jira] [Commented] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

2019-04-02 Thread Bill Schneider (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807768#comment-16807768 ] Bill Schneider commented on SPARK-27339: I was wary of commenting on a resolved/closed issue.  

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807749#comment-16807749 ] Imran Rashid commented on SPARK-27216: -- I think its probably fine to upgrade it, I just wanted to

[jira] [Issue Comment Deleted] (SPARK-27259) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" fr

2019-04-02 Thread Chris Thistlethwaite (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Thistlethwaite updated SPARK-27259: - Comment: was deleted (was: Fix this please for my) > Processing Compressed

[jira] [Commented] (SPARK-27342) Optimize limit 0 queries

2019-04-02 Thread Mingcong Han (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807734#comment-16807734 ] Mingcong Han commented on SPARK-27342: -- I'm wondering what kind of optimization is expected? For

[jira] [Updated] (SPARK-27346) Loosen the newline assert condition on 'examples' field in ExpressionInfo

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27346: - Description: I haven't tested by myself on Windows and I am not 100% sure if this is going to

[jira] [Updated] (SPARK-27346) Loosen the newline assert condition on 'examples' field in ExpressionInfo

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27346: - Description: I haven't tested by myself on Windows and I am not 100% sure if this is going to

[jira] [Updated] (SPARK-27346) Loosen the newline assert condition on 'examples' field in ExpressionInfo

2019-04-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27346: - Affects Version/s: 2.4.1 > Loosen the newline assert condition on 'examples' field in

[jira] [Created] (SPARK-27346) Loosen the newline assert condition on 'examples' field in ExpressionInfo

2019-04-02 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-27346: Summary: Loosen the newline assert condition on 'examples' field in ExpressionInfo Key: SPARK-27346 URL: https://issues.apache.org/jira/browse/SPARK-27346 Project:

[jira] [Reopened] (SPARK-25496) Deprecate from_utc_timestamp and to_utc_timestamp

2019-04-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-25496: --- This is reverted due to UT failure. Could you make another PR please, [~maxgekk]? > Deprecate

[jira] [Commented] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql

2019-04-02 Thread Henryk Cesnolovic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807438#comment-16807438 ] Henryk Cesnolovic commented on SPARK-27017: --- Hello, I have checked this one. Still for version 

[jira] [Comment Edited] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql

2019-04-02 Thread Henryk Cesnolovic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807438#comment-16807438 ] Henryk Cesnolovic edited comment on SPARK-27017 at 4/2/19 6:43 AM: ---

[jira] [Created] (SPARK-27345) Expose approx_percentile in python api

2019-04-02 Thread Roi Zaig (JIRA)
Roi Zaig created SPARK-27345: Summary: Expose approx_percentile in python api Key: SPARK-27345 URL: https://issues.apache.org/jira/browse/SPARK-27345 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-04-02 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807434#comment-16807434 ] Jacek Lewandowski commented on SPARK-27220: --- I know it looks very minor, but in our case it is

[jira] [Created] (SPARK-27344) Support the LocalDate and Instant classes in Java Bean encoders

2019-04-02 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27344: -- Summary: Support the LocalDate and Instant classes in Java Bean encoders Key: SPARK-27344 URL: https://issues.apache.org/jira/browse/SPARK-27344 Project: Spark