[jira] [Commented] (SPARK-19811) sparksql 2.1 can not prune hive partition

2019-10-10 Thread Vasu Bajaj (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948299#comment-16948299 ] Vasu Bajaj commented on SPARK-19811: It's too late, but the issue arises when you us

[jira] [Comment Edited] (SPARK-19811) sparksql 2.1 can not prune hive partition

2019-10-10 Thread Vasu Bajaj (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948299#comment-16948299 ] Vasu Bajaj edited comment on SPARK-19811 at 10/10/19 8:00 AM:

[jira] [Created] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-10 Thread pin_zhang (Jira)
pin_zhang created SPARK-29423: - Summary: leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus Key: SPARK-29423 URL: https://issues.apache.org/jira/browse/SPARK-29423 Project: Spark

[jira] [Created] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
angerszhu created SPARK-29424: - Summary: Prevent Spark to committing stage of too much Task Key: SPARK-29424 URL: https://issues.apache.org/jira/browse/SPARK-29424 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29424: -- Description: Our user always submit bad SQL in query platform, Such as : # write wrong join condition

[jira] [Updated] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29424: -- Description: Our user always submit bad SQL in query platform, Such as : # write wrong join condition

[jira] [Updated] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29424: -- Description: Our user always submit bad SQL in query platform, Such as : # write wrong join condition

[jira] [Updated] (SPARK-29409) spark drop partition always throws Exception

2019-10-10 Thread ant_nebula (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ant_nebula updated SPARK-29409: --- Description: The table is: {code:java} CREATE TABLE `test_spark.test_drop_partition`( `platform` st

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

2019-10-10 Thread Jatin Puri (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948361#comment-16948361 ] Jatin Puri commented on SPARK-10848: This issue still exists in `2.4.4`. Should a ne

[jira] [Commented] (SPARK-29288) Spark SQL add jar can't support HTTP path.

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948364#comment-16948364 ] angerszhu commented on SPARK-29288: --- [~dongjoon] Sorry for later reply , the hive Jir

[jira] [Commented] (SPARK-29409) spark drop partition always throws Exception

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948368#comment-16948368 ] angerszhu commented on SPARK-29409: --- Thanks, I will check this problem. > spark drop

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948236#comment-16948236 ] Yuming Wang edited comment on SPARK-29354 at 10/10/19 10:02 AM: --

[jira] [Created] (SPARK-29425) Alter database statement erases hive database's ownership

2019-10-10 Thread Kent Yao (Jira)
Kent Yao created SPARK-29425: Summary: Alter database statement erases hive database's ownership Key: SPARK-29425 URL: https://issues.apache.org/jira/browse/SPARK-29425 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29425) Alter database statement erases hive database's ownership

2019-10-10 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-29425: - Description: Commands like `ALTER DATABASE kyuubi SET DBPROPERTIES ('in'='out')` will erase a hive datab

[jira] [Created] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
jingshanglu created SPARK-29426: --- Summary: Watermark does not take effect Key: SPARK-29426 URL: https://issues.apache.org/jira/browse/SPARK-29426 Project: Spark Issue Type: Bug Compon

[jira] [Updated] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jingshanglu updated SPARK-29426: Environment: (was: my kafka mes like this: {code:java} // code placeholder [kafka@HC-25-28-36

[jira] [Commented] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948393#comment-16948393 ] Sean R. Owen commented on SPARK-29424: -- I doubt we want to throw yet another limit/

[jira] [Created] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset

2019-10-10 Thread Alexander Hagerf (Jira)
Alexander Hagerf created SPARK-29427: Summary: Create KeyValueGroupedDataset from RelationalGroupedDataset Key: SPARK-29427 URL: https://issues.apache.org/jira/browse/SPARK-29427 Project: Spark

[jira] [Commented] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948399#comment-16948399 ] angerszhu commented on SPARK-29424: --- [~srowen] Since resource limit is  established,

[jira] [Updated] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset

2019-10-10 Thread Alexander Hagerf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Hagerf updated SPARK-29427: - Description: The scenario I'm having is that I'm reading two huge bucketed tables and si

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948410#comment-16948410 ] angerszhu commented on SPARK-29354: --- [~Elixir Kook] [~yumwang] Jline is brought by hi

[jira] [Created] (SPARK-29428) Can't persist/set None-valued param

2019-10-10 Thread Borys Biletskyy (Jira)
Borys Biletskyy created SPARK-29428: --- Summary: Can't persist/set None-valued param Key: SPARK-29428 URL: https://issues.apache.org/jira/browse/SPARK-29428 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-29428) Can't persist/set None-valued param

2019-10-10 Thread Borys Biletskyy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Borys Biletskyy updated SPARK-29428: Description: {code:java} import pytest from pyspark import keyword_only from pyspark.ml im

[jira] [Commented] (SPARK-13346) Using DataFrames iteratively leads to slow query planning

2019-10-10 Thread Izek Greenfield (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948443#comment-16948443 ] Izek Greenfield commented on SPARK-13346: - [~davies] why this issue gets closed?

[jira] [Comment Edited] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948379#comment-16948379 ] jingshanglu edited comment on SPARK-29426 at 10/10/19 11:45 AM: --

[jira] [Comment Edited] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948379#comment-16948379 ] jingshanglu edited comment on SPARK-29426 at 10/10/19 11:46 AM: --

[jira] [Comment Edited] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948506#comment-16948506 ] Lantao Jin edited comment on SPARK-29421 at 10/10/19 12:03 PM: ---

[jira] [Commented] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948506#comment-16948506 ] Lantao Jin commented on SPARK-29421: [~cloud_fan] Yes, Hive support the similar comm

[jira] [Comment Edited] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948506#comment-16948506 ] Lantao Jin edited comment on SPARK-29421 at 10/10/19 12:07 PM: ---

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948575#comment-16948575 ] Yuming Wang commented on SPARK-29354: - Does {{bin/spark-shell}} need jline? > Spark

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948610#comment-16948610 ] Sungpeo Kook commented on SPARK-29354: -- [~yumwang] I meant spark binary distributio

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:12 PM: --

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:14 PM: --

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:17 PM: --

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:17 PM: --

[jira] [Commented] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

2019-10-10 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948652#comment-16948652 ] Thomas Graves commented on SPARK-28859: --- I wouldn't expect users to specify the si

[jira] [Resolved] (SPARK-29400) Improve PrometheusResource to use labels

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29400. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26060 [https://

[jira] [Assigned] (SPARK-29400) Improve PrometheusResource to use labels

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29400: - Assignee: Dongjoon Hyun > Improve PrometheusResource to use labels > --

[jira] [Updated] (SPARK-29400) Improve PrometheusResource to use labels

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29400: -- Parent: SPARK-29429 Issue Type: Sub-task (was: Improvement) > Improve PrometheusResou

[jira] [Created] (SPARK-29429) Support Prometheus monitoring

2019-10-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29429: - Summary: Support Prometheus monitoring Key: SPARK-29429 URL: https://issues.apache.org/jira/browse/SPARK-29429 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-29064) Add PrometheusResource to export Executor metrics

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29064: -- Parent: SPARK-29429 Issue Type: Sub-task (was: Improvement) > Add PrometheusResource

[jira] [Updated] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29032: -- Parent: SPARK-29429 Issue Type: Sub-task (was: Improvement) > Simplify Prometheus sup

[jira] [Updated] (SPARK-29429) Support Prometheus monitoring

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29429: -- Target Version/s: 3.0.0 > Support Prometheus monitoring > - > >

[jira] [Assigned] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29032: - Assignee: Dongjoon Hyun > Simplify Prometheus support by adding PrometheusServlet > ---

[jira] [Updated] (SPARK-29429) Support Prometheus monitoring natively

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29429: -- Summary: Support Prometheus monitoring natively (was: Support Prometheus monitoring) > Suppo

[jira] [Assigned] (SPARK-29064) Add PrometheusResource to export Executor metrics

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29064: - Assignee: Dongjoon Hyun > Add PrometheusResource to export Executor metrics > -

[jira] [Created] (SPARK-29430) Document new metric endpoints for Prometheus

2019-10-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29430: - Summary: Document new metric endpoints for Prometheus Key: SPARK-29430 URL: https://issues.apache.org/jira/browse/SPARK-29430 Project: Spark Issue Type: Su

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-10 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948720#comment-16948720 ] Imran Rashid commented on SPARK-29396: -- My hack to get around this in the past was

[jira] [Created] (SPARK-29431) Improve Web UI / Sql tab visualization with cached dataframes.

2019-10-10 Thread Pablo Langa Blanco (Jira)
Pablo Langa Blanco created SPARK-29431: -- Summary: Improve Web UI / Sql tab visualization with cached dataframes. Key: SPARK-29431 URL: https://issues.apache.org/jira/browse/SPARK-29431 Project: S

[jira] [Assigned] (SPARK-29430) Document new metric endpoints for Prometheus

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29430: - Assignee: Dongjoon Hyun > Document new metric endpoints for Prometheus > --

[jira] [Commented] (SPARK-29358) Make unionByName optionally fill missing columns with nulls

2019-10-10 Thread Mukul Murthy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948800#comment-16948800 ] Mukul Murthy commented on SPARK-29358: -- That would be a start to make us not have t

[jira] [Updated] (SPARK-20629) Copy shuffle data when nodes are being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20629: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Reopened] (SPARK-20629) Copy shuffle data when nodes are being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20629: --- > Copy shuffle data when nodes are being shut down > ---

[jira] [Reopened] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20624: --- > Add better handling for node shutdown > - > >

[jira] [Updated] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20624: -- Labels: (was: bulk-closed) > Add better handling for node shutdown > ---

[jira] [Updated] (SPARK-20629) Copy shuffle data when nodes are being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20629: -- Labels: (was: bulk-closed) > Copy shuffle data when nodes are being shut down >

[jira] [Updated] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20624: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Updated] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20624: -- Priority: Major (was: Minor) > Add better handling for node shutdown > --

[jira] [Updated] (SPARK-20732) Copy cache data when node is being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20732: -- Labels: (was: bulk-closed) > Copy cache data when node is being shut down >

[jira] [Updated] (SPARK-20732) Copy cache data when node is being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20732: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Reopened] (SPARK-20732) Copy cache data when node is being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20732: --- > Copy cache data when node is being shut down > >

[jira] [Reopened] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-21040: --- > On executor/worker decommission consider speculatively re-launching current > tasks > ---

[jira] [Updated] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21040: -- Labels: (was: bulk-closed) > On executor/worker decommission consider speculatively re-launc

[jira] [Updated] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21040: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-10-10 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948832#comment-16948832 ] Nasir Ali commented on SPARK-28502: --- [~bryanc] I tested it and it works fine with mast

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-10-10 Thread Michael Albert (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948860#comment-16948860 ] Michael Albert commented on SPARK-28921: Is there a timeline for this fix being

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-10-10 Thread Paul Schweigert (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948896#comment-16948896 ] Paul Schweigert commented on SPARK-28921: - [~albertmichaelj] You can replace the

[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-10 Thread antonkulaga (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948897#comment-16948897 ] antonkulaga commented on SPARK-28547: - [~hyukjin.kwon] what is not clear for you? I

[jira] [Comment Edited] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-10 Thread antonkulaga (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948897#comment-16948897 ] antonkulaga edited comment on SPARK-28547 at 10/10/19 7:24 PM: ---

[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-10 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948907#comment-16948907 ] Sean R. Owen commented on SPARK-28547: -- I agree, this is too open-ended. It's not c

[jira] [Created] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
Prasanna Saraswathi Krishnan created SPARK-29432: Summary: nullable flag of new column changes when persisting a pyspark dataframe Key: SPARK-29432 URL: https://issues.apache.org/jira/browse/SPARK-

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with {{withC

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with {{withC

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with {{withC

[jira] [Commented] (SPARK-29116) Refactor py classes related to DecisionTree

2019-10-10 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948942#comment-16948942 ] Huaxin Gao commented on SPARK-29116: I will submit a PR after DecisionTree refactor

[jira] [Created] (SPARK-29433) Web UI Stages table tooltip correction

2019-10-10 Thread Pablo Langa Blanco (Jira)
Pablo Langa Blanco created SPARK-29433: -- Summary: Web UI Stages table tooltip correction Key: SPARK-29433 URL: https://issues.apache.org/jira/browse/SPARK-29433 Project: Spark Issue Type

[jira] [Commented] (SPARK-26651) Use Proleptic Gregorian calendar

2019-10-10 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948957#comment-16948957 ] Maxim Gekk commented on SPARK-26651: [~jiangxb] Could you consider this for includin

[jira] [Created] (SPARK-29434) Improve the MapStatuses serialization performance

2019-10-10 Thread DB Tsai (Jira)
DB Tsai created SPARK-29434: --- Summary: Improve the MapStatuses serialization performance Key: SPARK-29434 URL: https://issues.apache.org/jira/browse/SPARK-29434 Project: Spark Issue Type: New Featu

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948979#comment-16948979 ] koert kuipers commented on SPARK-27665: --- i tried using spark.shuffle.useOldFetchPr

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-10-10 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948981#comment-16948981 ] Bryan Cutler commented on SPARK-28502: -- Thanks for testing it out [~nasirali]! It's

[jira] [Created] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
koert kuipers created SPARK-29435: - Summary: Spark 3 doesnt work with older shuffle service Key: SPARK-29435 URL: https://issues.apache.org/jira/browse/SPARK-29435 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948987#comment-16948987 ] Marcelo Masiero Vanzin commented on SPARK-29435: I think you have to set

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948989#comment-16948989 ] koert kuipers commented on SPARK-29435: --- [~vanzin] sorry i should have been more c

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

2019-10-10 Thread Cheng Lian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-26806: --- Description: Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will make "avg

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

2019-10-10 Thread Cheng Lian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-26806: --- Reporter: Cheng Lian (was: liancheng) > EventTimeStats.merge doesn't handle "zero.merge(zero)" corr

[jira] [Resolved] (SPARK-29367) pandas udf not working with latest pyarrow release (0.15.0)

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29367. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26045 [https://gi

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949050#comment-16949050 ] angerszhu commented on SPARK-29354: --- [~Elixir Kook]  i download spark-2.4.4-bin-hadoo

[jira] [Resolved] (SPARK-29284) df.distinct.count throw NoSuchElementException when enabled daptive executor

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29284. -- Resolution: Cannot Reproduce > df.distinct.count throw NoSuchElementException when enabled dap

[jira] [Updated] (SPARK-28636) Thriftserver can not support decimal type with negative scale

2019-10-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28636: Issue Type: Bug (was: Improvement) > Thriftserver can not support decimal type with negative scal

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29432: - Description: When I add a new column to a dataframe with {{withColumn}} function, by default, t

[jira] [Resolved] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29432. -- Resolution: Cannot Reproduce Can't fine {{withcolTest}} table. Also, please ask questions into

[jira] [Updated] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29423: - Component/s: (was: SQL) Structured Streaming > leak on org.apache.spark.sq

[jira] [Commented] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949060#comment-16949060 ] Hyukjin Kwon commented on SPARK-29423: -- 2.3.x is EOL releases. Can you try in highe

[jira] [Commented] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949068#comment-16949068 ] Prasanna Saraswathi Krishnan commented on SPARK-29432: -- My bad. Whe

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24266: - Affects Version/s: 3.0.0 > Spark client terminates while driver is still running > -

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24266: - Labels: (was: bulk-closed) > Spark client terminates while driver is still running > -

[jira] [Reopened] (SPARK-24266) Spark client terminates while driver is still running

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-24266: -- > Spark client terminates while driver is still running >

[jira] [Resolved] (SPARK-29337) How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29337. -- Resolution: Invalid Please see [https://spark.apache.org/community.html] > How to Cache Table

[jira] [Resolved] (SPARK-29335) Cost Based Optimizer stats are not used while evaluating query plans in Spark Sql

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29335. -- Resolution: Invalid Please see [https://spark.apache.org/community.html] > Cost Based Optimiz

[jira] [Commented] (SPARK-29222) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_parameter_convergence

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949086#comment-16949086 ] Hyukjin Kwon commented on SPARK-29222: -- Shall we increase the time a bit more if it

  1   2   >