[jira] [Comment Edited] (SPARK-28008) Default values & column comments in AVRO schema converters

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949140#comment-16949140 ] Hyukjin Kwon edited comment on SPARK-28008 at 10/11/19 5:16 AM: If the

[jira] [Commented] (SPARK-28008) Default values & column comments in AVRO schema converters

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949140#comment-16949140 ] Hyukjin Kwon commented on SPARK-28008: -- If the change is pretty small and clean, I suspect it's

[jira] [Commented] (SPARK-28008) Default values & column comments in AVRO schema converters

2019-10-10 Thread Terry Moschou (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949125#comment-16949125 ] Terry Moschou commented on SPARK-28008: --- We also have a use case for propagating application

[jira] [Comment Edited] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949095#comment-16949095 ] koert kuipers edited comment on SPARK-29435 at 10/11/19 3:20 AM: -

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949095#comment-16949095 ] koert kuipers commented on SPARK-29435: --- actually, it doesnt matter if i use spark 2 or spark 3

[jira] [Commented] (SPARK-29280) DataFrameReader should support a compression option

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949087#comment-16949087 ] Hyukjin Kwon commented on SPARK-29280: -- yup, sounds reasonable to me too. > DataFrameReader should

[jira] [Commented] (SPARK-29222) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_parameter_convergence

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949086#comment-16949086 ] Hyukjin Kwon commented on SPARK-29222: -- Shall we increase the time a bit more if it was verified

[jira] [Resolved] (SPARK-29335) Cost Based Optimizer stats are not used while evaluating query plans in Spark Sql

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29335. -- Resolution: Invalid Please see [https://spark.apache.org/community.html] > Cost Based

[jira] [Resolved] (SPARK-29337) How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29337. -- Resolution: Invalid Please see [https://spark.apache.org/community.html] > How to Cache

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24266: - Affects Version/s: 3.0.0 > Spark client terminates while driver is still running >

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24266: - Labels: (was: bulk-closed) > Spark client terminates while driver is still running >

[jira] [Reopened] (SPARK-24266) Spark client terminates while driver is still running

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-24266: -- > Spark client terminates while driver is still running >

[jira] [Commented] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949068#comment-16949068 ] Prasanna Saraswathi Krishnan commented on SPARK-29432: -- My bad. When I formatted

[jira] [Commented] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949060#comment-16949060 ] Hyukjin Kwon commented on SPARK-29423: -- 2.3.x is EOL releases. Can you try in higher versions? >

[jira] [Updated] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29423: - Component/s: (was: SQL) Structured Streaming > leak on

[jira] [Resolved] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29432. -- Resolution: Cannot Reproduce Can't fine {{withcolTest}} table. Also, please ask questions

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29432: - Description: When I add a new column to a dataframe with {{withColumn}} function, by default,

[jira] [Updated] (SPARK-28636) Thriftserver can not support decimal type with negative scale

2019-10-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28636: Issue Type: Bug (was: Improvement) > Thriftserver can not support decimal type with negative

[jira] [Resolved] (SPARK-29284) df.distinct.count throw NoSuchElementException when enabled daptive executor

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29284. -- Resolution: Cannot Reproduce > df.distinct.count throw NoSuchElementException when enabled

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949050#comment-16949050 ] angerszhu commented on SPARK-29354: --- [~Elixir Kook]  i download spark-2.4.4-bin-hadoop2.7  in your

[jira] [Resolved] (SPARK-29367) pandas udf not working with latest pyarrow release (0.15.0)

2019-10-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29367. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26045

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

2019-10-10 Thread Cheng Lian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-26806: --- Reporter: Cheng Lian (was: liancheng) > EventTimeStats.merge doesn't handle "zero.merge(zero)"

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

2019-10-10 Thread Cheng Lian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-26806: --- Description: Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will make

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948989#comment-16948989 ] koert kuipers commented on SPARK-29435: --- [~vanzin] sorry i should have been more clear, i did set

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948987#comment-16948987 ] Marcelo Masiero Vanzin commented on SPARK-29435: I think you have to set

[jira] [Created] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
koert kuipers created SPARK-29435: - Summary: Spark 3 doesnt work with older shuffle service Key: SPARK-29435 URL: https://issues.apache.org/jira/browse/SPARK-29435 Project: Spark Issue Type:

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-10-10 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948981#comment-16948981 ] Bryan Cutler commented on SPARK-28502: -- Thanks for testing it out [~nasirali]! It's unlikely that

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948979#comment-16948979 ] koert kuipers commented on SPARK-27665: --- i tried using spark.shuffle.useOldFetchProtocol=true

[jira] [Created] (SPARK-29434) Improve the MapStatuses serialization performance

2019-10-10 Thread DB Tsai (Jira)
DB Tsai created SPARK-29434: --- Summary: Improve the MapStatuses serialization performance Key: SPARK-29434 URL: https://issues.apache.org/jira/browse/SPARK-29434 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-26651) Use Proleptic Gregorian calendar

2019-10-10 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948957#comment-16948957 ] Maxim Gekk commented on SPARK-26651: [~jiangxb] Could you consider this for including to the major

[jira] [Created] (SPARK-29433) Web UI Stages table tooltip correction

2019-10-10 Thread Pablo Langa Blanco (Jira)
Pablo Langa Blanco created SPARK-29433: -- Summary: Web UI Stages table tooltip correction Key: SPARK-29433 URL: https://issues.apache.org/jira/browse/SPARK-29433 Project: Spark Issue

[jira] [Commented] (SPARK-29116) Refactor py classes related to DecisionTree

2019-10-10 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948942#comment-16948942 ] Huaxin Gao commented on SPARK-29116: I will submit a PR after DecisionTree refactor is in, so I

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with

[jira] [Created] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-10 Thread Prasanna Saraswathi Krishnan (Jira)
Prasanna Saraswathi Krishnan created SPARK-29432: Summary: nullable flag of new column changes when persisting a pyspark dataframe Key: SPARK-29432 URL:

[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-10 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948907#comment-16948907 ] Sean R. Owen commented on SPARK-28547: -- I agree, this is too open-ended. It's not clear whether

[jira] [Comment Edited] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-10 Thread antonkulaga (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948897#comment-16948897 ] antonkulaga edited comment on SPARK-28547 at 10/10/19 7:24 PM: ---

[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-10 Thread antonkulaga (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948897#comment-16948897 ] antonkulaga commented on SPARK-28547: - [~hyukjin.kwon] what is not clear for you? I think it is

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-10-10 Thread Paul Schweigert (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948896#comment-16948896 ] Paul Schweigert commented on SPARK-28921: - [~albertmichaelj] You can replace the

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-10-10 Thread Michael Albert (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948860#comment-16948860 ] Michael Albert commented on SPARK-28921: Is there a timeline for this fix being present in a

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-10-10 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948832#comment-16948832 ] Nasir Ali commented on SPARK-28502: --- [~bryanc] I tested it and it works fine with master branch. Is

[jira] [Reopened] (SPARK-20732) Copy cache data when node is being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20732: --- > Copy cache data when node is being shut down > >

[jira] [Reopened] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-21040: --- > On executor/worker decommission consider speculatively re-launching current > tasks >

[jira] [Updated] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21040: -- Labels: (was: bulk-closed) > On executor/worker decommission consider speculatively

[jira] [Updated] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20624: -- Priority: Major (was: Minor) > Add better handling for node shutdown >

[jira] [Updated] (SPARK-20732) Copy cache data when node is being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20732: -- Labels: (was: bulk-closed) > Copy cache data when node is being shut down >

[jira] [Updated] (SPARK-20732) Copy cache data when node is being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20732: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Updated] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21040: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Updated] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20624: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Reopened] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20624: --- > Add better handling for node shutdown > - > >

[jira] [Updated] (SPARK-20624) Add better handling for node shutdown

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20624: -- Labels: (was: bulk-closed) > Add better handling for node shutdown >

[jira] [Updated] (SPARK-20629) Copy shuffle data when nodes are being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20629: -- Labels: (was: bulk-closed) > Copy shuffle data when nodes are being shut down >

[jira] [Updated] (SPARK-20629) Copy shuffle data when nodes are being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20629: -- Affects Version/s: (was: 2.3.0) (was: 2.2.0)

[jira] [Reopened] (SPARK-20629) Copy shuffle data when nodes are being shut down

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20629: --- > Copy shuffle data when nodes are being shut down >

[jira] [Commented] (SPARK-29358) Make unionByName optionally fill missing columns with nulls

2019-10-10 Thread Mukul Murthy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948800#comment-16948800 ] Mukul Murthy commented on SPARK-29358: -- That would be a start to make us not have to do #1, but #2

[jira] [Assigned] (SPARK-29430) Document new metric endpoints for Prometheus

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29430: - Assignee: Dongjoon Hyun > Document new metric endpoints for Prometheus >

[jira] [Created] (SPARK-29431) Improve Web UI / Sql tab visualization with cached dataframes.

2019-10-10 Thread Pablo Langa Blanco (Jira)
Pablo Langa Blanco created SPARK-29431: -- Summary: Improve Web UI / Sql tab visualization with cached dataframes. Key: SPARK-29431 URL: https://issues.apache.org/jira/browse/SPARK-29431 Project:

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-10 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948720#comment-16948720 ] Imran Rashid commented on SPARK-29396: -- My hack to get around this in the past was to create a

[jira] [Created] (SPARK-29430) Document new metric endpoints for Prometheus

2019-10-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29430: - Summary: Document new metric endpoints for Prometheus Key: SPARK-29430 URL: https://issues.apache.org/jira/browse/SPARK-29430 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29032: - Assignee: Dongjoon Hyun > Simplify Prometheus support by adding PrometheusServlet >

[jira] [Updated] (SPARK-29429) Support Prometheus monitoring natively

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29429: -- Summary: Support Prometheus monitoring natively (was: Support Prometheus monitoring) >

[jira] [Assigned] (SPARK-29064) Add PrometheusResource to export Executor metrics

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29064: - Assignee: Dongjoon Hyun > Add PrometheusResource to export Executor metrics >

[jira] [Updated] (SPARK-29429) Support Prometheus monitoring

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29429: -- Target Version/s: 3.0.0 > Support Prometheus monitoring > - > >

[jira] [Updated] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29032: -- Parent: SPARK-29429 Issue Type: Sub-task (was: Improvement) > Simplify Prometheus

[jira] [Updated] (SPARK-29064) Add PrometheusResource to export Executor metrics

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29064: -- Parent: SPARK-29429 Issue Type: Sub-task (was: Improvement) > Add PrometheusResource

[jira] [Updated] (SPARK-29400) Improve PrometheusResource to use labels

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29400: -- Parent: SPARK-29429 Issue Type: Sub-task (was: Improvement) > Improve

[jira] [Created] (SPARK-29429) Support Prometheus monitoring

2019-10-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29429: - Summary: Support Prometheus monitoring Key: SPARK-29429 URL: https://issues.apache.org/jira/browse/SPARK-29429 Project: Spark Issue Type: Umbrella

[jira] [Resolved] (SPARK-29400) Improve PrometheusResource to use labels

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29400. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26060

[jira] [Assigned] (SPARK-29400) Improve PrometheusResource to use labels

2019-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29400: - Assignee: Dongjoon Hyun > Improve PrometheusResource to use labels >

[jira] [Commented] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

2019-10-10 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948652#comment-16948652 ] Thomas Graves commented on SPARK-28859: --- I wouldn't expect users to specify the size when enabled

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:17 PM:

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:17 PM:

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:14 PM:

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948610#comment-16948610 ] Sungpeo Kook edited comment on SPARK-29354 at 10/10/19 2:12 PM:

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Sungpeo Kook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948610#comment-16948610 ] Sungpeo Kook commented on SPARK-29354: -- [~yumwang] I meant spark binary distributions which are

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948575#comment-16948575 ] Yuming Wang commented on SPARK-29354: - Does {{bin/spark-shell}} need jline? > Spark has direct

[jira] [Comment Edited] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948506#comment-16948506 ] Lantao Jin edited comment on SPARK-29421 at 10/10/19 12:07 PM: ---

[jira] [Commented] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948506#comment-16948506 ] Lantao Jin commented on SPARK-29421: [~cloud_fan] Yes, Hive support the similar command with

[jira] [Comment Edited] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948506#comment-16948506 ] Lantao Jin edited comment on SPARK-29421 at 10/10/19 12:03 PM: ---

[jira] [Comment Edited] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948379#comment-16948379 ] jingshanglu edited comment on SPARK-29426 at 10/10/19 11:46 AM: my kafka

[jira] [Comment Edited] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948379#comment-16948379 ] jingshanglu edited comment on SPARK-29426 at 10/10/19 11:45 AM: my kafka

[jira] [Commented] (SPARK-13346) Using DataFrames iteratively leads to slow query planning

2019-10-10 Thread Izek Greenfield (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948443#comment-16948443 ] Izek Greenfield commented on SPARK-13346: - [~davies] why this issue gets closed? > Using

[jira] [Updated] (SPARK-29428) Can't persist/set None-valued param

2019-10-10 Thread Borys Biletskyy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Borys Biletskyy updated SPARK-29428: Description: {code:java} import pytest from pyspark import keyword_only from pyspark.ml

[jira] [Created] (SPARK-29428) Can't persist/set None-valued param

2019-10-10 Thread Borys Biletskyy (Jira)
Borys Biletskyy created SPARK-29428: --- Summary: Can't persist/set None-valued param Key: SPARK-29428 URL: https://issues.apache.org/jira/browse/SPARK-29428 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948410#comment-16948410 ] angerszhu commented on SPARK-29354: --- [~Elixir Kook] [~yumwang] Jline is brought by hive-beeline

[jira] [Updated] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset

2019-10-10 Thread Alexander Hagerf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Hagerf updated SPARK-29427: - Description: The scenario I'm having is that I'm reading two huge bucketed tables and

[jira] [Created] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset

2019-10-10 Thread Alexander Hagerf (Jira)
Alexander Hagerf created SPARK-29427: Summary: Create KeyValueGroupedDataset from RelationalGroupedDataset Key: SPARK-29427 URL: https://issues.apache.org/jira/browse/SPARK-29427 Project: Spark

[jira] [Commented] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948399#comment-16948399 ] angerszhu commented on SPARK-29424: --- [~srowen] Since resource limit is  established, these bad

[jira] [Commented] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948393#comment-16948393 ] Sean R. Owen commented on SPARK-29424: -- I doubt we want to throw yet another limit/config at this.

[jira] [Updated] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jingshanglu updated SPARK-29426: Environment: (was: my kafka mes like this: {code:java} // code placeholder [kafka@HC-25-28-36

[jira] [Created] (SPARK-29426) Watermark does not take effect

2019-10-10 Thread jingshanglu (Jira)
jingshanglu created SPARK-29426: --- Summary: Watermark does not take effect Key: SPARK-29426 URL: https://issues.apache.org/jira/browse/SPARK-29426 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-29425) Alter database statement erases hive database's ownership

2019-10-10 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-29425: - Description: Commands like `ALTER DATABASE kyuubi SET DBPROPERTIES ('in'='out')` will erase a hive

[jira] [Created] (SPARK-29425) Alter database statement erases hive database's ownership

2019-10-10 Thread Kent Yao (Jira)
Kent Yao created SPARK-29425: Summary: Alter database statement erases hive database's ownership Key: SPARK-29425 URL: https://issues.apache.org/jira/browse/SPARK-29425 Project: Spark Issue

[jira] [Comment Edited] (SPARK-29354) Spark has direct dependency on jline, but binaries for 'without hadoop' don't have a jline jar file.

2019-10-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948236#comment-16948236 ] Yuming Wang edited comment on SPARK-29354 at 10/10/19 10:02 AM: Hi

[jira] [Commented] (SPARK-29409) spark drop partition always throws Exception

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948368#comment-16948368 ] angerszhu commented on SPARK-29409: --- Thanks, I will check this problem. > spark drop partition always

[jira] [Commented] (SPARK-29288) Spark SQL add jar can't support HTTP path.

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948364#comment-16948364 ] angerszhu commented on SPARK-29288: --- [~dongjoon] Sorry for later reply , the hive Jira is 

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

2019-10-10 Thread Jatin Puri (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948361#comment-16948361 ] Jatin Puri commented on SPARK-10848: This issue still exists in `2.4.4`. Should a new issue be

[jira] [Updated] (SPARK-29409) spark drop partition always throws Exception

2019-10-10 Thread ant_nebula (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ant_nebula updated SPARK-29409: --- Description: The table is: {code:java} CREATE TABLE `test_spark.test_drop_partition`( `platform`

[jira] [Updated] (SPARK-29424) Prevent Spark to committing stage of too much Task

2019-10-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29424: -- Description: Our user always submit bad SQL in query platform, Such as : # write wrong join condition

  1   2   >