[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

2019-09-11 Thread zhoukang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928204#comment-16928204 ] zhoukang commented on SPARK-21492: -- Any progress of this issue? [~jiangxb1987] We also encountered this

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928180#comment-16928180 ] feiwang commented on SPARK-29037: - [~cloud_fan] > [Core] Spark gives duplicate result when an

[jira] [Issue Comment Deleted] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Comment: was deleted (was: If we have several applications, which insert overwrite a partition of same

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: When we insert overwrite a partition of table. For a stage, whose tasks commit output, a

[jira] [Created] (SPARK-29064) Add PrometheusResource to export Executor metrics

2019-09-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29064: - Summary: Add PrometheusResource to export Executor metrics Key: SPARK-29064 URL: https://issues.apache.org/jira/browse/SPARK-29064 Project: Spark Issue

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: When we insert overwrite a partition of table. For a stage, whose tasks commit output, a

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Affects Version/s: 2.3.3 > [Core] Spark gives duplicate result when an application was killed and rerun >

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928170#comment-16928170 ] feiwang commented on SPARK-29037: - If we have several applications, which insert overwrite a partition

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928168#comment-16928168 ] feiwang commented on SPARK-29037: - This committedTaskPath is hard coded in FileOutputCommitter class. >

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928165#comment-16928165 ] feiwang commented on SPARK-29037: - This is the unit test log. !screenshot-1.png! We can see that, the

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Attachment: screenshot-1.png > [Core] Spark gives duplicate result when an application was killed and

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: When we insert overwrite a partition of table. For a stage, whose tasks commit output, a

[jira] [Updated] (SPARK-29063) fillna support for joined table

2019-09-11 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-29063: Description: When you have a joined table that has the same field name from both original table,

[jira] [Created] (SPARK-29063) fillna support for joined table

2019-09-11 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-29063: --- Summary: fillna support for joined table Key: SPARK-29063 URL: https://issues.apache.org/jira/browse/SPARK-29063 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-11 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928144#comment-16928144 ] Lantao Jin commented on SPARK-29038: [~smilegator] Yes. It's physically stored. I will create a

[jira] [Resolved] (SPARK-29046) Possible NPE on SQLConf.get when SparkContext is stopping in another thread

2019-09-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29046. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25753

[jira] [Assigned] (SPARK-29046) Possible NPE on SQLConf.get when SparkContext is stopping in another thread

2019-09-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29046: Assignee: Jungtaek Lim > Possible NPE on SQLConf.get when SparkContext is stopping in

[jira] [Updated] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread dengziming (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dengziming updated SPARK-29050: --- Issue Type: Improvement (was: Bug) > Fix typo in some docs > - > >

[jira] [Commented] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread dengziming (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928121#comment-16928121 ] dengziming commented on SPARK-29050: [~srowen] thank you! > Fix typo in some docs >

[jira] [Updated] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread dengziming (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dengziming updated SPARK-29050: --- Issue Type: Bug (was: Improvement) > Fix typo in some docs > - > >

[jira] [Created] (SPARK-29062) Add V1_BATCH_WRITE to the TableCapabilityChecks in the Analyzer

2019-09-11 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29062: --- Summary: Add V1_BATCH_WRITE to the TableCapabilityChecks in the Analyzer Key: SPARK-29062 URL: https://issues.apache.org/jira/browse/SPARK-29062 Project: Spark

[jira] [Created] (SPARK-29061) Prints bytecode statistics in debugCodegen

2019-09-11 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29061: Summary: Prints bytecode statistics in debugCodegen Key: SPARK-29061 URL: https://issues.apache.org/jira/browse/SPARK-29061 Project: Spark Issue

[jira] [Resolved] (SPARK-29057) remove InsertIntoTable

2019-09-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29057. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25763

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-11 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928100#comment-16928100 ] Adrian Wang commented on SPARK-29038: - This seems duplicates with our proposal of SPARK-26764 . We

[jira] [Resolved] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-09-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29041. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25749

[jira] [Assigned] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-09-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29041: Assignee: Hyukjin Kwon > Allow createDataFrame to accept bytes as binary type >

[jira] [Assigned] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24663: -- Assignee: Jungtaek Lim > Flaky test: StreamingContextSuite "stop slow receiver

[jira] [Resolved] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24663. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25725

[jira] [Comment Edited] (SPARK-27781) Tried to access method org.apache.avro.specific.SpecificData.()V

2019-09-11 Thread Michael Heuer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921700#comment-16921700 ] Michael Heuer edited comment on SPARK-27781 at 9/11/19 7:47 PM: -This is

[jira] [Resolved] (SPARK-27781) Tried to access method org.apache.avro.specific.SpecificData.()V

2019-09-11 Thread Michael Heuer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Heuer resolved SPARK-27781. --- Fix Version/s: 2.4.4 Resolution: Fixed > Tried to access method

[jira] [Comment Edited] (SPARK-27781) Tried to access method org.apache.avro.specific.SpecificData.()V

2019-09-11 Thread Michael Heuer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921700#comment-16921700 ] Michael Heuer edited comment on SPARK-27781 at 9/11/19 7:46 PM: -This is

[jira] [Commented] (SPARK-27781) Tried to access method org.apache.avro.specific.SpecificData.()V

2019-09-11 Thread Michael Heuer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927953#comment-16927953 ] Michael Heuer commented on SPARK-27781: --- This issue has been fixed in Spark 2.4.4, and fixed in

[jira] [Updated] (SPARK-27781) Tried to access method org.apache.avro.specific.SpecificData.()V

2019-09-11 Thread Michael Heuer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Heuer updated SPARK-27781: -- Affects Version/s: (was: 2.4.4) > Tried to access method

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927912#comment-16927912 ] koert kuipers commented on SPARK-29027: --- [~gsomogyi] if you email me at koert at tresata dot com i

[jira] [Created] (SPARK-29060) Add tree traversal helper for adaptive spark plans

2019-09-11 Thread Maryann Xue (Jira)
Maryann Xue created SPARK-29060: --- Summary: Add tree traversal helper for adaptive spark plans Key: SPARK-29060 URL: https://issues.apache.org/jira/browse/SPARK-29060 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29059) Support for Hive Materialized Views in Spark SQL.

2019-09-11 Thread Amogh Margoor (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amogh Margoor updated SPARK-29059: -- Description: Materialized view was introduced in Apache Hive 3.0.0. Currently, Spark

[jira] [Updated] (SPARK-29059) Support for Hive Materialized Views in Spark SQL.

2019-09-11 Thread Amogh Margoor (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amogh Margoor updated SPARK-29059: -- Summary: Support for Hive Materialized Views in Spark SQL. (was: Support for Hive

[jira] [Created] (SPARK-29059) Support for Hive Materialized Views for Spark SQL.

2019-09-11 Thread Amogh Margoor (Jira)
Amogh Margoor created SPARK-29059: - Summary: Support for Hive Materialized Views for Spark SQL. Key: SPARK-29059 URL: https://issues.apache.org/jira/browse/SPARK-29059 Project: Spark Issue

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927889#comment-16927889 ] koert kuipers commented on SPARK-29027: --- just for this one test debug logs is 62mb of kerberos and

[jira] [Created] (SPARK-29058) Reading csv file with DROPMALFORMED showing incorrect record count

2019-09-11 Thread Suchintak Patnaik (Jira)
Suchintak Patnaik created SPARK-29058: - Summary: Reading csv file with DROPMALFORMED showing incorrect record count Key: SPARK-29058 URL: https://issues.apache.org/jira/browse/SPARK-29058

[jira] [Resolved] (SPARK-29007) Possible leak of SparkContext in tests / test suites initializing StreamingContext

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29007. Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed > Possible

[jira] [Resolved] (SPARK-26989) Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26989. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25706

[jira] [Assigned] (SPARK-26989) Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26989: -- Assignee: Jungtaek Lim > Flaky test:DAGSchedulerSuite.Barrier task failures from the

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927822#comment-16927822 ] Gabor Somogyi commented on SPARK-29027: --- You can remove the sensitive parts or if you only trust

[jira] [Created] (SPARK-29057) remove InsertIntoTable

2019-09-11 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-29057: --- Summary: remove InsertIntoTable Key: SPARK-29057 URL: https://issues.apache.org/jira/browse/SPARK-29057 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-29014) DataSourceV2: Clean up current, default, and session catalog uses

2019-09-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927796#comment-16927796 ] Wenchen Fan commented on SPARK-29014: - It doesn't require a major refactor but it's easier and

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-11 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927784#comment-16927784 ] Xiao Li commented on SPARK-29038: - So far, the doc does not contain enough details. It requires

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927785#comment-16927785 ] koert kuipers commented on SPARK-29027: --- [~gsomogyi] i can email you debug log file directly if

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927783#comment-16927783 ] koert kuipers commented on SPARK-29027: --- i get same error in sbt i think, plus i find sbt a lot

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927715#comment-16927715 ] koert kuipers commented on SPARK-29027: --- i renamed /etc/krb5.conf and it did not change anything.

[jira] [Updated] (SPARK-29056) ThriftServerSessionPage displays 1970/01/01 for queries that are not finished and not closed

2019-09-11 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski updated SPARK-29056: -- Issue Type: Bug (was: Improvement) > ThriftServerSessionPage displays 1970/01/01 for

[jira] [Created] (SPARK-29056) ThriftServerSessionPage displays 1970/01/01 for queries that are not finished and not closed

2019-09-11 Thread Juliusz Sompolski (Jira)
Juliusz Sompolski created SPARK-29056: - Summary: ThriftServerSessionPage displays 1970/01/01 for queries that are not finished and not closed Key: SPARK-29056 URL:

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927646#comment-16927646 ] koert kuipers commented on SPARK-29027: --- let me try to get debug logs > KafkaDelegationTokenSuite

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927640#comment-16927640 ] Gabor Somogyi commented on SPARK-29027: --- Can you give for example the output of this cmd:

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927637#comment-16927637 ] Gabor Somogyi commented on SPARK-29027: --- I've tried to create a krb5.conf file which contains

[jira] [Commented] (SPARK-28985) Pyspark ClassificationModel and RegressionModel support column setters/getters/predict

2019-09-11 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927612#comment-16927612 ] Huaxin Gao commented on SPARK-28985: Thanks [~podongfeng] I will work on this. > Pyspark

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Affects Version/s: (was: 2.4.2) (was: 2.4.1)

[jira] [Updated] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-29050: -- Issue Type: Improvement (was: Bug) Priority: Trivial (was: Major) This can't be considered a

[jira] [Resolved] (SPARK-27492) GPU scheduling - High level user documentation

2019-09-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27492. --- Fix Version/s: 3.0.0 Resolution: Fixed > GPU scheduling - High level user

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927558#comment-16927558 ] Thomas Graves commented on SPARK-27495: --- [~felixcheung]  [~jiangxb1987]  I put this up for vote on

[jira] [Updated] (SPARK-28987) DiskBlockManager#createTempShuffleBlock should skip directory which is read-only

2019-09-11 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28987: -- Priority: Minor (was: Major) > DiskBlockManager#createTempShuffleBlock should skip directory which

[jira] [Resolved] (SPARK-28987) DiskBlockManager#createTempShuffleBlock should skip directory which is read-only

2019-09-11 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28987. --- Resolution: Won't Fix > DiskBlockManager#createTempShuffleBlock should skip directory which is >

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: image-2019-09-11-16-14-34-963.png > Memory leak in Spark Driver >

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: image-2019-09-11-16-14-26-765.png > Memory leak in Spark Driver >

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Description: In Spark 2.3.3+ the driver memory is increasing continuously. I don't have this

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: (was: image-2019-09-11-16-13-20-588.png) > Memory leak in Spark Driver >

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Description: In Spark 2.3.3+ the driver memory is increasing continuously. I don't have this

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Description: In Spark 2.3.3+ the driver memory is increasing continuously. I don't have this

[jira] [Resolved] (SPARK-28906) `bin/spark-submit --version` shows incorrect info

2019-09-11 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28906. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: image-2019-09-11-16-13-20-588.png > Memory leak in Spark Driver >

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: (was: image-2019-09-11-16-13-32-650.png) > Memory leak in Spark Driver >

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: image-2019-09-11-16-13-32-650.png > Memory leak in Spark Driver >

[jira] [Assigned] (SPARK-28906) `bin/spark-submit --version` shows incorrect info

2019-09-11 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28906: - Assignee: Kazuaki Ishizaki > `bin/spark-submit --version` shows incorrect info >

[jira] [Created] (SPARK-29055) Memory leak in Spark Driver

2019-09-11 Thread George Papa (Jira)
George Papa created SPARK-29055: --- Summary: Memory leak in Spark Driver Key: SPARK-29055 URL: https://issues.apache.org/jira/browse/SPARK-29055 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927533#comment-16927533 ] Jungtaek Lim commented on SPARK-29043: -- 5+! I'm very surprised to hear that, as it means 5+

[jira] [Created] (SPARK-29054) Invalidate Kafka consumer when new delegation token available

2019-09-11 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-29054: - Summary: Invalidate Kafka consumer when new delegation token available Key: SPARK-29054 URL: https://issues.apache.org/jira/browse/SPARK-29054 Project: Spark

[jira] [Commented] (SPARK-28985) Pyspark ClassificationModel and RegressionModel support column setters/getters/predict

2019-09-11 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927498#comment-16927498 ] zhengruifeng commented on SPARK-28985: -- [~huaxingao] You can refer to my old prs

[jira] [Updated] (SPARK-29053) Spark Thrift JDBC/ODBC Server application UI, Sorting is not working for Duration field

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29053: - Description: Spark Thrift JDBC/ODBC Server application UI, *Sorting* is not working for

[jira] [Updated] (SPARK-29053) Spark Thrift JDBC/ODBC Server application UI, Sorting is not working for Duration field

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29053: - Attachment: Sort Icon.png > Spark Thrift JDBC/ODBC Server application UI, Sorting is not

[jira] [Commented] (SPARK-29053) Spark Thrift JDBC/ODBC Server application UI, Sorting is not working for Duration field

2019-09-11 Thread Rakesh Raushan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927467#comment-16927467 ] Rakesh Raushan commented on SPARK-29053: I will work on this one. > Spark Thrift JDBC/ODBC

[jira] [Created] (SPARK-29053) Spark Thrift JDBC/ODBC Server application UI, Sorting is not working for Duration field

2019-09-11 Thread jobit mathew (Jira)
jobit mathew created SPARK-29053: Summary: Spark Thrift JDBC/ODBC Server application UI, Sorting is not working for Duration field Key: SPARK-29053 URL: https://issues.apache.org/jira/browse/SPARK-29053

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-11 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927374#comment-16927374 ] Lantao Jin edited comment on SPARK-29038 at 9/11/19 10:12 AM: --

[jira] [Created] (SPARK-29052) Create a Migration Guide tap in Spark documentation

2019-09-11 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-29052: Summary: Create a Migration Guide tap in Spark documentation Key: SPARK-29052 URL: https://issues.apache.org/jira/browse/SPARK-29052 Project: Spark Issue

[jira] [Updated] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29051: - Description: Spark Application UI *Search is not working* for some fields in *Spark Web UI

[jira] [Updated] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29051: - Attachment: Duration Search1.png > Spark Application UI search is not working for some fields >

[jira] [Updated] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29051: - Attachment: Duration Search.png > Spark Application UI search is not working for some fields >

[jira] [Updated] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29051: - Attachment: Search Missing.png > Spark Application UI search is not working for some fields >

[jira] [Updated] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-29051: - Attachment: Search Missing.png > Spark Application UI search is not working for some fields >

[jira] [Updated] (SPARK-28882) Memory leak when stopping spark session

2019-09-11 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-28882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Łukasz Pińkowski updated SPARK-28882: - Description: When calling stop() method on spark session underlying SparkContext is

[jira] [Commented] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927434#comment-16927434 ] Aman Omer commented on SPARK-29051: --- I would like to handle this. > Spark Application UI search is

[jira] [Created] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-11 Thread jobit mathew (Jira)
jobit mathew created SPARK-29051: Summary: Spark Application UI search is not working for some fields Key: SPARK-29051 URL: https://issues.apache.org/jira/browse/SPARK-29051 Project: Spark

[jira] [Comment Edited] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread dengziming (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927421#comment-16927421 ] dengziming edited comment on SPARK-29050 at 9/11/19 9:17 AM: - Hi, I have

[jira] [Commented] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread dengziming (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927421#comment-16927421 ] dengziming commented on SPARK-29050: [https://github.com/apache/spark/pull/25756] > Fix typo in

[jira] [Created] (SPARK-29050) Fix typo in some docs

2019-09-11 Thread dengziming (Jira)
dengziming created SPARK-29050: -- Summary: Fix typo in some docs Key: SPARK-29050 URL: https://issues.apache.org/jira/browse/SPARK-29050 Project: Spark Issue Type: Bug Components:

[jira] [Comment Edited] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927402#comment-16927402 ] feiwang edited comment on SPARK-29043 at 9/11/19 8:41 AM: -- [~kabhwan] * How

[jira] [Commented] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927402#comment-16927402 ] feiwang commented on SPARK-29043: - * How long "spark.history.fs.update.interval" has been set?20s *

[jira] [Created] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-11 Thread Xianyin Xin (Jira)
Xianyin Xin created SPARK-29049: --- Summary: Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames Key: SPARK-29049 URL: https://issues.apache.org/jira/browse/SPARK-29049

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-11 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927374#comment-16927374 ] Lantao Jin commented on SPARK-29038: [~smilegator] Sure, we will totally fellow ANSI SQL when commit

[jira] [Assigned] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

2019-09-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-29048: -- Assignee: Weichen Xu > Query optimizer slow when using Column.isInCollection() with a large

[jira] [Created] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

2019-09-11 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-29048: -- Summary: Query optimizer slow when using Column.isInCollection() with a large size collection Key: SPARK-29048 URL: https://issues.apache.org/jira/browse/SPARK-29048

  1   2   >