[jira] [Created] (SPARK-27225) Implement join strategy hints

2019-03-20 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-27225: --- Summary: Implement join strategy hints Key: SPARK-27225 URL: https://issues.apache.org/jira/browse/SPARK-27225 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-27214) Upgrading locality level when lots of pending tasks have been waiting more than locality.wait

2019-03-20 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-27214: - Description: Currently, Spark locality wait mechanism is not friendly for large job, when

[jira] [Resolved] (SPARK-26894) Fix Alias handling in AggregateEstimation

2019-03-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-26894. -- Resolution: Fixed Fix Version/s: 3.0.0 Resolved by

[jira] [Resolved] (SPARK-27223) Remove private methods that skip conversion when passing user schemas for constructing a DataFrame

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27223. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24162

[jira] [Resolved] (SPARK-27221) Improve the assert error message in TreeNode.parseToJson

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27221. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24159

[jira] [Assigned] (SPARK-27223) Remove private methods that skip conversion when passing user schemas for constructing a DataFrame

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27223: Assignee: Maryann Xue > Remove private methods that skip conversion when passing user

[jira] [Comment Edited] (SPARK-24105) Spark 2.3.0 on kubernetes

2019-03-20 Thread Kevin Hogeland (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797719#comment-16797719 ] Kevin Hogeland edited comment on SPARK-24105 at 3/21/19 1:07 AM: -

[jira] [Comment Edited] (SPARK-24105) Spark 2.3.0 on kubernetes

2019-03-20 Thread Kevin Hogeland (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797719#comment-16797719 ] Kevin Hogeland edited comment on SPARK-24105 at 3/21/19 1:09 AM: -

[jira] [Comment Edited] (SPARK-24105) Spark 2.3.0 on kubernetes

2019-03-20 Thread Kevin Hogeland (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797719#comment-16797719 ] Kevin Hogeland edited comment on SPARK-24105 at 3/21/19 1:08 AM: -

[jira] [Commented] (SPARK-24105) Spark 2.3.0 on kubernetes

2019-03-20 Thread Kevin Hogeland (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797719#comment-16797719 ] Kevin Hogeland commented on SPARK-24105: [~vanzin] Why was this marked "Won't Fix"? This is a

[jira] [Commented] (SPARK-27224) Spark to_json parses UTC timestamp incorrectly

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797711#comment-16797711 ] Hyukjin Kwon commented on SPARK-27224: -- This was an unexpected behaviour from SimpleDateFormat if I

[jira] [Commented] (SPARK-27177) Update jenkins locale to en_US.UTF-8

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797709#comment-16797709 ] Hyukjin Kwon commented on SPARK-27177: -- Yes, I think we can unblock

[jira] [Resolved] (SPARK-27224) Spark to_json parses UTC timestamp incorrectly

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27224. -- Resolution: Cannot Reproduce > Spark to_json parses UTC timestamp incorrectly >

[jira] [Commented] (SPARK-27006) SPIP: .NET bindings for Apache Spark

2019-03-20 Thread Markus Weimer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797688#comment-16797688 ] Markus Weimer commented on SPARK-27006: --- +1 on a .NET API for Spark. Over in

[jira] [Created] (SPARK-27224) Spark to_json parses UTC timestamp incorrectly

2019-03-20 Thread Jeff Xu (JIRA)
Jeff Xu created SPARK-27224: --- Summary: Spark to_json parses UTC timestamp incorrectly Key: SPARK-27224 URL: https://issues.apache.org/jira/browse/SPARK-27224 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

2019-03-20 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27194: -- Affects Version/s: 2.3.3 > Job failures when task attempts do not clean up spark-staging

[jira] [Commented] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

2019-03-20 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797633#comment-16797633 ] Dongjoon Hyun commented on SPARK-27194: --- Thank you for checking that, [~ajithshetty]. > Job

[jira] [Assigned] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

2019-03-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27205: - Assignee: Jungtaek Lim > spark-shell with packages option fails to load transitive

[jira] [Resolved] (SPARK-27205) spark-shell with packages option fails to load transitive dependencies even ivy successfully pulls jars

2019-03-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27205. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24147

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-03-20 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797630#comment-16797630 ] Dongjoon Hyun commented on SPARK-27220: --- +1 for [~srowen]'s advice~ > Remove Yarn specific

[jira] [Resolved] (SPARK-26729) Spark on Kubernetes tooling hardcodes default image names

2019-03-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26729. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23846

[jira] [Assigned] (SPARK-26729) Spark on Kubernetes tooling hardcodes default image names

2019-03-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26729: -- Assignee: Rob Vesse > Spark on Kubernetes tooling hardcodes default image names >

[jira] [Resolved] (SPARK-27215) Correct the kryo configurations

2019-03-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27215. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24156

[jira] [Assigned] (SPARK-27215) Correct the kryo configurations

2019-03-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27215: -- Assignee: Lantao Jin > Correct the kryo configurations >

[jira] [Updated] (SPARK-27223) Remove private methods that skip conversion when passing user schemas for constructing a DataFrame

2019-03-20 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-27223: Summary: Remove private methods that skip conversion when passing user schemas for constructing a

[jira] [Created] (SPARK-27223) Remove private methods that allow no conversion when passing user schemas for constructing a DataFrame

2019-03-20 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-27223: --- Summary: Remove private methods that allow no conversion when passing user schemas for constructing a DataFrame Key: SPARK-27223 URL:

[jira] [Created] (SPARK-27222) Support Instant and LocalDate in Literal.apply

2019-03-20 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27222: -- Summary: Support Instant and LocalDate in Literal.apply Key: SPARK-27222 URL: https://issues.apache.org/jira/browse/SPARK-27222 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

2019-03-20 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27160. --- Resolution: Fixed Assignee: Darcy Shen Fix Version/s: 3.0.0 > Incorrect

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-03-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797460#comment-16797460 ] Sean Owen commented on SPARK-27220: --- Probably [~irashid] would know better > Remove Yarn specific

[jira] [Resolved] (SPARK-27094) Thread interrupt being swallowed while launching executors in YarnAllocator

2019-03-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27094. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24017

[jira] [Assigned] (SPARK-27094) Thread interrupt being swallowed while launching executors in YarnAllocator

2019-03-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27094: -- Assignee: Marcelo Vanzin > Thread interrupt being swallowed while launching

[jira] [Commented] (SPARK-27219) Misleading exceptions in transport code's SASL fallback path

2019-03-20 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797435#comment-16797435 ] Ajith S commented on SPARK-27219: - So do we just log a simple warn with one line message and print the

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-03-20 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797428#comment-16797428 ] Ajith S commented on SPARK-27220: - # About making *currentExecutorIdCounter* datatype consistent, Yes, 

[jira] [Updated] (SPARK-27221) Improve the assert error message in TreeNode.parseToJson

2019-03-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27221: - Summary: Improve the assert error message in TreeNode.parseToJson (was: Improve the assert

[jira] [Updated] (SPARK-27221) Improve the assert error message in TreeNode

2019-03-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27221: - Description: When TreeNode.parseToJson may throw an assert error without any error message

[jira] [Created] (SPARK-27221) Improve the assert error message in TreeNode

2019-03-20 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27221: Summary: Improve the assert error message in TreeNode Key: SPARK-27221 URL: https://issues.apache.org/jira/browse/SPARK-27221 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

2019-03-20 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Safi updated SPARK-27194: -- Issue Type: Bug (was: Improvement) > Job failures when task attempts do not clean up spark-staging

[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

2019-03-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797321#comment-16797321 ] Imran Rashid commented on SPARK-27112: -- [~Dhruve Ashar] -- rc8 is already defined, there is nothing

[jira] [Updated] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-03-20 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Lewandowski updated SPARK-27220: -- Description: {{CoarseGrainedSchedulerBackend}} has the following field: {code:scala}

[jira] [Created] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-03-20 Thread Jacek Lewandowski (JIRA)
Jacek Lewandowski created SPARK-27220: - Summary: Remove Yarn specific leftover from CoarseGrainedSchedulerBackend Key: SPARK-27220 URL: https://issues.apache.org/jira/browse/SPARK-27220 Project:

[jira] [Created] (SPARK-27219) Misleading exceptions in transport code's SASL fallback path

2019-03-20 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-27219: -- Summary: Misleading exceptions in transport code's SASL fallback path Key: SPARK-27219 URL: https://issues.apache.org/jira/browse/SPARK-27219 Project: Spark

[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

2019-03-20 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797253#comment-16797253 ] Dhruve Ashar commented on SPARK-27112: -- [~irashid] - I think this is a critical bug and since it is

[jira] [Commented] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

2019-03-20 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797241#comment-16797241 ] Ajith S commented on SPARK-27194: - Hi [~dongjoon] , i have some analysis here :

[jira] [Comment Edited] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

2019-03-20 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797216#comment-16797216 ] Ajith S edited comment on SPARK-27194 at 3/20/19 2:38 PM: -- [~dongjoon] Yes i

[jira] [Commented] (SPARK-27194) Job failures when task attempts do not clean up spark-staging parquet files

2019-03-20 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797216#comment-16797216 ] Ajith S commented on SPARK-27194: - [~dongjoon] Yes i tried with spark 2.3.3, and the issue persist. Here

[jira] [Created] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-20 Thread Emanuele Sabellico (JIRA)
Emanuele Sabellico created SPARK-27218: -- Summary: spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming Key: SPARK-27218 URL: https://issues.apache.org/jira/browse/SPARK-27218

[jira] [Updated] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-20 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuele Sabellico updated SPARK-27218: --- Description: Hi, I'm trying to stream a kafka topic with 

[jira] [Updated] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-20 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuele Sabellico updated SPARK-27218: --- Description: Hi, I'm trying to stream a kafka topic with 

[jira] [Created] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-03-20 Thread colin fang (JIRA)
colin fang created SPARK-27217: -- Summary: Nested schema pruning doesn't work for aggregation e.g. `sum`. Key: SPARK-27217 URL: https://issues.apache.org/jira/browse/SPARK-27217 Project: Spark

[jira] [Resolved] (SPARK-27206) Using slice method with streaming api's Interval on DStream

2019-03-20 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-27206. --- Resolution: Invalid > Using slice method with streaming api's Interval on DStream >

[jira] [Assigned] (SPARK-27216) Kryo serialization with RoaringBitmap

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27216: Assignee: (was: Apache Spark) > Kryo serialization with RoaringBitmap >

[jira] [Assigned] (SPARK-27216) Kryo serialization with RoaringBitmap

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27216: Assignee: Apache Spark > Kryo serialization with RoaringBitmap >

[jira] [Commented] (SPARK-27206) Using slice method with streaming api's Interval on DStream

2019-03-20 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797153#comment-16797153 ] Gabor Somogyi commented on SPARK-27206: --- [~aarthipa] Can you please write a mail to the dev list?

[jira] [Updated] (SPARK-27216) Kryo serialization with RoaringBitmap

2019-03-20 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Description: HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But

[jira] [Created] (SPARK-27216) Kryo serialization with RoaringBitmap

2019-03-20 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-27216: -- Summary: Kryo serialization with RoaringBitmap Key: SPARK-27216 URL: https://issues.apache.org/jira/browse/SPARK-27216 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-27215) Correct the kryo configurations

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27215: Assignee: (was: Apache Spark) > Correct the kryo configurations >

[jira] [Assigned] (SPARK-27215) Correct the kryo configurations

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27215: Assignee: Apache Spark > Correct the kryo configurations >

[jira] [Created] (SPARK-27215) Correct the kryo configurations

2019-03-20 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-27215: -- Summary: Correct the kryo configurations Key: SPARK-27215 URL: https://issues.apache.org/jira/browse/SPARK-27215 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-27199) Replace TimeZone by ZoneId in TimestampFormatter API

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27199. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24141

[jira] [Assigned] (SPARK-27201) Show full job description on click

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27201: Assignee: Gengliang Wang > Show full job description on click >

[jira] [Resolved] (SPARK-27201) Show full job description on click

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27201. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24145

[jira] [Commented] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Udbhav Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797112#comment-16797112 ] Udbhav Agrawal commented on SPARK-27213: i will try to check this issue > Unexpected results

[jira] [Assigned] (SPARK-27199) Replace TimeZone by ZoneId in TimestampFormatter API

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27199: Assignee: Maxim Gekk > Replace TimeZone by ZoneId in TimestampFormatter API >

[jira] [Commented] (SPARK-27089) Loss of precision during decimal division

2019-03-20 Thread Jared Leable (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797111#comment-16797111 ] Jared Leable commented on SPARK-27089: -- I validated that setting 

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rinaz Belhaj updated SPARK-27213: - Priority: Major (was: Minor) > Unexpected results when filter is used after distinct >

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

2019-03-20 Thread acupple (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797070#comment-16797070 ] acupple commented on SPARK-27169: - Thanks for you suggestion, and I will try increment the queue size

[jira] [Commented] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797079#comment-16797079 ] Hyukjin Kwon commented on SPARK-27100: -- Would you be interested in narrowing down the problem, so

[jira] [Updated] (SPARK-27214) Upgrading locality level when lots of pending tasks have been waiting more than locality.wait

2019-03-20 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-27214: - Description: Currently, Spark locality wait mechanism is not friendly for large job, when

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

2019-03-20 Thread acupple (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797059#comment-16797059 ] acupple commented on SPARK-27169: - Can not find any "Dropping event" log,  but some warn that "Dropped

[jira] [Created] (SPARK-27214) Upgrading locality level when lots of pending tasks have been waiting more than locality.wait

2019-03-20 Thread liupengcheng (JIRA)
liupengcheng created SPARK-27214: Summary: Upgrading locality level when lots of pending tasks have been waiting more than locality.wait Key: SPARK-27214 URL: https://issues.apache.org/jira/browse/SPARK-27214

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

2019-03-20 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797066#comment-16797066 ] shahid commented on SPARK-27169: Yes. that means many event drops happens. Can you try increasing the

[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page

2019-03-20 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797066#comment-16797066 ] shahid edited comment on SPARK-27169 at 3/20/19 11:26 AM: -- Yes. that means many

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rinaz Belhaj updated SPARK-27213: - Description: The following code gives unexpected output due to the filter not getting pushed

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rinaz Belhaj updated SPARK-27213: - Description: The following code gives unexpected output due to the filter not getting pushed

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rinaz Belhaj updated SPARK-27213: - Description: The following code gives unexpected output due to the filter not getting pushed

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rinaz Belhaj updated SPARK-27213: - Description: The following code gives unexpected output due to the filter not getting pushed

[jira] [Assigned] (SPARK-27200) History Environment tab must sort Configurations/Properties by default

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27200: Assignee: Ajith S > History Environment tab must sort Configurations/Properties by

[jira] [Resolved] (SPARK-27200) History Environment tab must sort Configurations/Properties by default

2019-03-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27200. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24143

[jira] [Created] (SPARK-27213) Unexpected results due when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
Rinaz Belhaj created SPARK-27213: Summary: Unexpected results due when filter is used after distinct Key: SPARK-27213 URL: https://issues.apache.org/jira/browse/SPARK-27213 Project: Spark

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-03-20 Thread Rinaz Belhaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rinaz Belhaj updated SPARK-27213: - Summary: Unexpected results when filter is used after distinct (was: Unexpected results due

[jira] [Assigned] (SPARK-27212) Eliminate TimeZone to ZoneId conversion in stringToTimestamp

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27212: Assignee: (was: Apache Spark) > Eliminate TimeZone to ZoneId conversion in

[jira] [Assigned] (SPARK-27212) Eliminate TimeZone to ZoneId conversion in stringToTimestamp

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27212: Assignee: Apache Spark > Eliminate TimeZone to ZoneId conversion in stringToTimestamp >

[jira] [Created] (SPARK-27212) Eliminate TimeZone to ZoneId conversion in stringToTimestamp

2019-03-20 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27212: -- Summary: Eliminate TimeZone to ZoneId conversion in stringToTimestamp Key: SPARK-27212 URL: https://issues.apache.org/jira/browse/SPARK-27212 Project: Spark

[jira] [Commented] (SPARK-27169) number of active tasks is negative on executors page

2019-03-20 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797040#comment-16797040 ] shahid commented on SPARK-27169: Hi, It seems, from the above log we can't say that event drop has

[jira] [Commented] (SPARK-27203) Spark Fails to read a view using CTE (WITH clause) and created via beeline

2019-03-20 Thread Igor Ngouagna (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796958#comment-16796958 ] Igor Ngouagna commented on SPARK-27203: --- I thought "UNION ALL" was part of the issue here, but

[jira] [Commented] (SPARK-27203) Spark Fails to read a view using CTE (WITH clause) and created via beeline

2019-03-20 Thread Igor Ngouagna (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796946#comment-16796946 ] Igor Ngouagna commented on SPARK-27203: --- Furthermore, I notice something weird when i tried to

[jira] [Commented] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-03-20 Thread Mateusz Kaczor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796924#comment-16796924 ] Mateusz Kaczor commented on SPARK-26606: I think I've come across similar issue (at least with

[jira] [Resolved] (SPARK-27099) Expose xxHash64 as a flexible 64-bit column hash like `hash`

2019-03-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27099. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24019

[jira] [Assigned] (SPARK-27099) Expose xxHash64 as a flexible 64-bit column hash like `hash`

2019-03-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27099: --- Assignee: Huon Wilson > Expose xxHash64 as a flexible 64-bit column hash like `hash` >

[jira] [Created] (SPARK-27211) cast error when select column from Row

2019-03-20 Thread Guiju Zhang (JIRA)
Guiju Zhang created SPARK-27211: --- Summary: cast error when select column from Row Key: SPARK-27211 URL: https://issues.apache.org/jira/browse/SPARK-27211 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-27211) cast error when select column from Row

2019-03-20 Thread Guiju Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guiju Zhang updated SPARK-27211: Description: 1.First, I have an object RawLogPlayload which has an field: long timestamp 2.Then

[jira] [Updated] (SPARK-27211) cast error when select column from Row

2019-03-20 Thread Guiju Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guiju Zhang updated SPARK-27211: Description: First, I have an object RawLogPlayload which has an field: long timestamp Then I

[jira] [Assigned] (SPARK-27210) Cleanup incomplete output files in ManifestFileCommitProtocol if task is aborted

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27210: Assignee: Apache Spark > Cleanup incomplete output files in ManifestFileCommitProtocol

[jira] [Assigned] (SPARK-27210) Cleanup incomplete output files in ManifestFileCommitProtocol if task is aborted

2019-03-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27210: Assignee: (was: Apache Spark) > Cleanup incomplete output files in

[jira] [Commented] (SPARK-27208) RestSubmissionClient only supports http

2019-03-20 Thread Jorge Machado (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796871#comment-16796871 ] Jorge Machado commented on SPARK-27208: --- Furthermore I'm getting the next error: 

[jira] [Commented] (SPARK-27208) RestSubmissionClient only supports http

2019-03-20 Thread Jorge Machado (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796860#comment-16796860 ] Jorge Machado commented on SPARK-27208: --- I would like to take this. > RestSubmissionClient only

[jira] [Updated] (SPARK-27208) RestSubmissionClient only supports http

2019-03-20 Thread Jorge Machado (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jorge Machado updated SPARK-27208: -- Shepherd: Sean Owen > RestSubmissionClient only supports http >

[jira] [Created] (SPARK-27210) Cleanup incomplete output files in ManifestFileCommitProtocol if task is aborted

2019-03-20 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-27210: Summary: Cleanup incomplete output files in ManifestFileCommitProtocol if task is aborted Key: SPARK-27210 URL: https://issues.apache.org/jira/browse/SPARK-27210

[jira] [Updated] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

2019-03-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27160: -- Fix Version/s: (was: 3.0.0) > Incorrect Literal Casting of DecimalType in OrcFilters >

[jira] [Updated] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

2019-03-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27160: -- Priority: Major (was: Blocker) > Incorrect Literal Casting of DecimalType in OrcFilters >

  1   2   >