[jira] [Assigned] (SPARK-29509) Deduplicate code blocks in Kafka data source

2019-10-28 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-29509: -- Assignee: Jungtaek Lim > Deduplicate code blocks in Kafka d

[jira] [Resolved] (SPARK-29472) Mechanism for Excluding Jars at Launch for YARN

2019-10-24 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29472. Resolution: Won't Fix > Mechanism for Excluding Jars at Launch for Y

[jira] [Commented] (SPARK-29472) Mechanism for Excluding Jars at Launch for YARN

2019-10-24 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959179#comment-16959179 ] Marcelo Masiero Vanzin commented on SPARK-29472: bq. customers sometimes need to debug

[jira] [Commented] (SPARK-29593) Enhance Cluster Managers to be Pluggable

2019-10-24 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959078#comment-16959078 ] Marcelo Masiero Vanzin commented on SPARK-29593: bq. Is there any plans to make it more

[jira] [Commented] (SPARK-29593) Enhance Cluster Managers to be Pluggable

2019-10-24 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959063#comment-16959063 ] Marcelo Masiero Vanzin commented on SPARK-29593: This already exists

[jira] [Created] (SPARK-29562) SQLAppStatusListener metrics aggregation is slow and memory hungry

2019-10-22 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29562: -- Summary: SQLAppStatusListener metrics aggregation is slow and memory hungry Key: SPARK-29562 URL: https://issues.apache.org/jira/browse/SPARK-29562

[spark] branch master updated (78bdcfa -> 00347a3)

2019-10-17 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 78bdcfa [SPARK-27812][K8S] Bump K8S client version to 4.6.1 add 00347a3 [SPARK-28762][CORE] Read JAR main

[jira] [Assigned] (SPARK-28762) Read JAR main class if JAR is not located in local file system

2019-10-17 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-28762: -- Assignee: Ivan Gozali > Read JAR main class if JAR is not loca

[jira] [Resolved] (SPARK-28762) Read JAR main class if JAR is not located in local file system

2019-10-17 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-28762. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-17 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954053#comment-16954053 ] Marcelo Masiero Vanzin commented on SPARK-29396: Wow, sorry, I missed all the comments

[jira] [Resolved] (SPARK-28869) Roll over event log files

2019-10-17 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-28869. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-28869) Roll over event log files

2019-10-17 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-28869: -- Assignee: Jungtaek Lim > Roll over event log fi

[spark] branch master updated (2f0a38c -> 100fc58)

2019-10-17 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2f0a38c [SPARK-29398][CORE] Support dedicated thread pools for RPC endpoints add 100fc58 [SPARK-28869][CORE

[jira] [Assigned] (SPARK-28947) Status logging occurs on every state change but not at an interval for liveness.

2019-10-15 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-28947: -- Assignee: Kent Yao > Status logging occurs on every state cha

[jira] [Resolved] (SPARK-28947) Status logging occurs on every state change but not at an interval for liveness.

2019-10-15 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-28947. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[spark] branch master updated: [SPARK-28947][K8S] Status logging not happens at an interval for liveness

2019-10-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 02c5b4f [SPARK-28947][K8S] Status logging

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948987#comment-16948987 ] Marcelo Masiero Vanzin commented on SPARK-29435: I think you have to set

[jira] [Created] (SPARK-29399) Remove old ExecutorPlugin API (or wrap it using new API)

2019-10-08 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29399: -- Summary: Remove old ExecutorPlugin API (or wrap it using new API) Key: SPARK-29399 URL: https://issues.apache.org/jira/browse/SPARK-29399 Project

[jira] [Created] (SPARK-29398) Allow RPC endpoints to use dedicated thread pools

2019-10-08 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29398: -- Summary: Allow RPC endpoints to use dedicated thread pools Key: SPARK-29398 URL: https://issues.apache.org/jira/browse/SPARK-29398 Project: Spark

[jira] [Created] (SPARK-29397) Create new plugin interface for driver and executor plugins

2019-10-08 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29397: -- Summary: Create new plugin interface for driver and executor plugins Key: SPARK-29397 URL: https://issues.apache.org/jira/browse/SPARK-29397

[jira] [Created] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-08 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29396: -- Summary: Extend Spark plugin interface to driver Key: SPARK-29396 URL: https://issues.apache.org/jira/browse/SPARK-29396 Project: Spark

[spark] branch branch-2.4 updated: [SPARK-28917][CORE] Synchronize access to RDD mutable state

2019-10-08 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 4f46e8f [SPARK-28917][CORE] Synchronize

[jira] [Assigned] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

2019-10-08 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-28917: -- Assignee: Imran Rashid > Jobs can hang because of r

[jira] [Resolved] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

2019-10-08 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-28917. Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed

[spark] branch master updated (de360e9 -> 0da667d)

2019-10-08 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from de360e9 [SPARK-29336][SQL] Fix the implementation of QuantileSummaries.merge (guarantee that the relativeError

[jira] [Resolved] (SPARK-29054) Invalidate Kafka consumer when new delegation token available

2019-10-03 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29054. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[spark] branch master updated (4e0e4e5 -> 6b5e0e2)

2019-10-03 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e0e4e5 [MINOR][TESTS] Rename JSONBenchmark to JsonBenchmark add 6b5e0e2 [SPARK-29054][SS] Invalidate Kafka

[jira] [Assigned] (SPARK-29054) Invalidate Kafka consumer when new delegation token available

2019-10-03 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-29054: -- Assignee: Gabor Somogyi > Invalidate Kafka consumer when

[jira] [Resolved] (SPARK-29273) Spark peakExecutionMemory metrics is zero

2019-10-02 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29273. Fix Version/s: 3.0.0 Assignee: huangweiyi Resolution

[spark] branch master updated (f2ead4d -> 85dafab)

2019-10-02 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f2ead4d [SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2 add 85dafab [SPARK-29273][CORE

[jira] [Assigned] (SPARK-29321) Possible memory leak in Spark

2019-10-01 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-29321: -- Assignee: (was: Jungtaek Lim) > Possible memory leak in Sp

[jira] [Resolved] (SPARK-29055) Memory leak in Spark

2019-10-01 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29055. Fix Version/s: 3.0.0 2.4.5 Assignee: Jungtaek

[spark] branch branch-2.4 updated: [SPARK-29055][CORE] Update driver/executors' storage memory when block is removed from BlockManager

2019-10-01 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 3173439 [SPARK-29055][CORE] Update

[spark] branch master updated (0cf2f48 -> a4601cb)

2019-10-01 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0cf2f48 [SPARK-29022][SQL] Fix SparkSQLCLI can not add jars by AddJarCommand add a4601cb [SPARK-29055][CORE

[jira] [Created] (SPARK-29319) Memory and disk usage not accurate when blocks are evicted and re-loaded in memory

2019-10-01 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29319: -- Summary: Memory and disk usage not accurate when blocks are evicted and re-loaded in memory Key: SPARK-29319 URL: https://issues.apache.org/jira/browse/SPARK

[spark] branch branch-2.4 updated: [SPARK-29263][CORE][TEST][FOLLOWUP][2.4] Fix build failure of `TaskSchedulerImplSuite`

2019-09-27 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 9ae7393 [SPARK-29263][CORE][TEST

[jira] [Resolved] (SPARK-29070) Make SparkLauncher log full spark-submit command line

2019-09-27 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29070. Fix Version/s: 3.0.0 Assignee: Jeff Evans Resolution

[spark] branch master updated (420abb4 -> 233c214)

2019-09-27 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 420abb4 [SPARK-29263][SCHEDULER] Update `availableSlots` in `resourceOffers()` before checking available slots

[spark] branch master updated (8beb736 -> d3679a9)

2019-09-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8beb736 [SPARK-29256][DOCS] Fix typo in building document add d3679a9 [SPARK-27748][SS][FOLLOWUP] Correct

[jira] [Assigned] (SPARK-29082) Spark driver cannot start with only delegation tokens

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-29082: -- Assignee: Marcelo Vanzin > Spark driver cannot start with only delegation tok

[spark] branch master updated (db9e0fd -> f32f16f)

2019-09-18 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from db9e0fd [SPARK-22796][PYTHON][ML] Add multiple columns support to PySpark QuantileDiscretizer add f32f16f

[jira] [Resolved] (SPARK-29082) Spark driver cannot start with only delegation tokens

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29082. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25805 [https

[spark] branch master updated (dfb0a8b -> cd48177)

2019-09-18 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dfb0a8b [SPARK-28208][BUILD][SQL] Upgrade to ORC 1.5.6 including closing the ORC readers add cd48177 [SPARK

[jira] [Assigned] (SPARK-28091) Extend Spark metrics system with user-defined metrics using executor plugins

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28091: -- Assignee: Luca Canali > Extend Spark metrics system with user-defined metrics us

[jira] [Resolved] (SPARK-28091) Extend Spark metrics system with user-defined metrics using executor plugins

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28091. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 24901 [https

[jira] [Assigned] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-29105: -- Assignee: Marcelo Vanzin > SHS may delete driver log file of in progress applicat

[jira] [Resolved] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29105. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25819 [https

[spark] branch master updated: [SPARK-29105][CORE] Keep driver log file size up to date in HDFS

2019-09-18 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 276 [SPARK-29105][CORE] Keep driver log

[jira] [Resolved] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-17 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29027. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25803 [https

[jira] [Assigned] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-17 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-29027: -- Assignee: Gabor Somogyi > KafkaDelegationTokenSuite fa

[spark] branch master updated (02db706 -> 71e7516)

2019-09-17 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 02db706 [SPARK-29115][SQL][TEST] Add benchmarks for make_date() and make_timestamp() add 71e7516 [SPARK-29027

[jira] [Created] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-16 Thread Marcelo Vanzin (Jira)
Marcelo Vanzin created SPARK-29105: -- Summary: SHS may delete driver log file of in progress application Key: SPARK-29105 URL: https://issues.apache.org/jira/browse/SPARK-29105 Project: Spark

[jira] [Assigned] (SPARK-26929) table owner should use user instead of principal when use kerberos

2019-09-16 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26929: -- Assignee: hong dongdong > table owner should use user instead of principal when

[jira] [Resolved] (SPARK-26929) table owner should use user instead of principal when use kerberos

2019-09-16 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26929. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 23952 [https

[jira] [Created] (SPARK-29082) Spark driver cannot start with only delegation tokens

2019-09-13 Thread Marcelo Vanzin (Jira)
Marcelo Vanzin created SPARK-29082: -- Summary: Spark driver cannot start with only delegation tokens Key: SPARK-29082 URL: https://issues.apache.org/jira/browse/SPARK-29082 Project: Spark

[jira] [Assigned] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24663: -- Assignee: Jungtaek Lim > Flaky test: StreamingContextSuite "stop slow

[jira] [Resolved] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24663. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25725 [https

[spark] branch master updated (b62ef8f -> c18f849)

2019-09-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b62ef8f [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new

[jira] [Resolved] (SPARK-29007) Possible leak of SparkContext in tests / test suites initializing StreamingContext

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29007. Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed > Possi

[spark] branch master updated: [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext

2019-09-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b62ef8f [SPARK-29007][STREAMING][MLLIB][TESTS

[jira] [Resolved] (SPARK-26989) Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26989. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25706 [https

[jira] [Assigned] (SPARK-26989) Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries

2019-09-11 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26989: -- Assignee: Jungtaek Lim > Flaky test:DAGSchedulerSuite.Barrier task failures f

[spark] branch master updated (f263909 -> 2736efa)

2019-09-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f263909 [SPARK-23243][CORE][FOLLOWUP] Remove todo added by SPARK-23207 add 2736efa [SPARK-26989][CORE][TEST

[jira] [Resolved] (SPARK-28570) Shuffle Storage API: Use writer API in UnsafeShuffleWriter

2019-09-10 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28570. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25304 [https

[jira] [Assigned] (SPARK-28570) Shuffle Storage API: Use writer API in UnsafeShuffleWriter

2019-09-10 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28570: -- Assignee: Matt Cheah > Shuffle Storage API: Use writer API in UnsafeShuffleWri

[spark] branch master updated (fa75db2 -> 7f36cd2)

2019-09-10 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from fa75db2 [SPARK-29026][SQL] Improve error message in `schemaFor` in trait without companion object constructor

[jira] [Assigned] (SPARK-28928) Use Kafka delegation token protocol on sources/sinks

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28928: -- Assignee: Gabor Somogyi > Use Kafka delegation token protocol on sources/si

[jira] [Resolved] (SPARK-28928) Use Kafka delegation token protocol on sources/sinks

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28928. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25631 [https

[spark] branch master updated (8018ded -> e516f7e)

2019-09-09 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8018ded [SPARK-28214][STREAMING][TESTS] CheckpointSuite: wait for batch to be fully processed before accessing

[jira] [Resolved] (SPARK-28214) Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28214. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25731 [https

[jira] [Assigned] (SPARK-28214) Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28214: -- Assignee: Jungtaek Lim > Flaky t

[spark] branch master updated (125af78d -> 8018ded)

2019-09-09 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 125af78d [SPARK-28831][DOC][SQL] Document CLEAR CACHE statement in SQL Reference add 8018ded [SPARK-28214

[jira] [Assigned] (SPARK-25151) Apply Apache Commons Pool to KafkaDataConsumer

2019-09-04 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25151: -- Assignee: Jungtaek Lim > Apply Apache Commons Pool to KafkaDataConsu

[jira] [Resolved] (SPARK-25151) Apply Apache Commons Pool to KafkaDataConsumer

2019-09-04 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25151. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 22138 [https

[spark] branch master updated (712874f -> 594c9c5)

2019-09-04 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 712874f [SPARK-28931][CORE][TESTS] Fix couple of bugs in FsHistoryProviderSuite add 594c9c5 [SPARK-25151][SS

[jira] [Resolved] (SPARK-28931) Fix couple of bugs in FsHistoryProviderSuite

2019-09-04 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28931. Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed >

[spark] branch master updated (9f478a6 -> 712874f)

2019-09-04 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9f478a6 [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI add 712874f [SPARK

[jira] [Assigned] (SPARK-28571) Shuffle storage API: Use API in SortShuffleWriter

2019-08-30 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28571: -- Assignee: Matt Cheah > Shuffle storage API: Use API in SortShuffleWri

[jira] [Resolved] (SPARK-28571) Shuffle storage API: Use API in SortShuffleWriter

2019-08-30 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28571. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25342 [https

[spark] branch master updated: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-30 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ea90ea6 [SPARK-28571][CORE][SHUFFLE] Use

[spark] branch master updated (fb1053d -> 7d72c07)

2019-08-29 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from fb1053d [SPARK-28807][DOCS][SQL] Document SHOW DATABASES in SQL Reference add 7d72c07 [SPARK-28760][SS][TESTS

[jira] [Resolved] (SPARK-28760) Add end-to-end Kafka delegation token test

2019-08-29 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28760. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25477 [https

[jira] [Assigned] (SPARK-28760) Add end-to-end Kafka delegation token test

2019-08-29 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28760: -- Assignee: Gabor Somogyi > Add end-to-end Kafka delegation token t

[jira] [Commented] (SPARK-25874) Simplify abstractions in the K8S backend

2019-08-29 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918813#comment-16918813 ] Marcelo Vanzin commented on SPARK-25874: I'm going to close this one; the only remaining task

[jira] [Resolved] (SPARK-25874) Simplify abstractions in the K8S backend

2019-08-29 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25874. Fix Version/s: 3.0.0 Resolution: Done > Simplify abstractions in the K8S back

[jira] [Commented] (SPARK-28906) `bin/spark-submit --version` shows incorrect info

2019-08-28 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918204#comment-16918204 ] Marcelo Vanzin commented on SPARK-28906: The code is fine. This is a problem in the release

Re: [VOTE] Release Apache Spark 2.3.4 (RC1)

2019-08-28 Thread Marcelo Vanzin
+1 On Mon, Aug 26, 2019 at 1:28 PM Kazuaki Ishizaki wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.3.4. > > The vote is open until August 29th 2PM PST and passes if a majority +1 PMC > votes are cast, with > a minimum of 3 +1 votes. > > [ ] +1 Release

Re: [VOTE] Release Apache Spark 2.3.4 (RC1)

2019-08-28 Thread Marcelo Vanzin
(Ah, and the 2.4 RC has the same issue.) On Wed, Aug 28, 2019 at 2:23 PM Marcelo Vanzin wrote: > > Just noticed something before I started to run some tests. The output > of "spark-submit --version" is a little weird, in that it's missing > information (see end of e-mail).

Re: [VOTE] Release Apache Spark 2.3.4 (RC1)

2019-08-28 Thread Marcelo Vanzin
Just noticed something before I started to run some tests. The output of "spark-submit --version" is a little weird, in that it's missing information (see end of e-mail). Personally I don't think a lot of that output is super useful (like "Compiled by" or the repo URL), but the branch and

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

2019-08-28 Thread Marcelo Vanzin
+1 On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.4. > > The vote is open until August 30th 5PM PST and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this

[commons-crypto] branch master updated: CRYPTO-137: Fix compilation + support OpenSSL 1.1 on Windows. (#94)

2019-08-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/commons-crypto.git The following commit(s) were added to refs/heads/master by this push: new 83f1a72 CRYPTO-137: Fix compilation

[spark] branch master updated (b205269 -> 64032cb)

2019-08-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b205269 [SPARK-28875][DSTREAMS][SS][TESTS] Add Task retry tests to make sure new consumer used add 64032cb

[spark] branch master updated: [SPARK-28875][DSTREAMS][SS][TESTS] Add Task retry tests to make sure new consumer used

2019-08-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b205269 [SPARK-28875][DSTREAMS][SS][TESTS] Add

[jira] [Resolved] (SPARK-28875) Cover Task retry scenario with test in Kafka connector

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28875. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25582 [https

[jira] [Assigned] (SPARK-28875) Cover Task retry scenario with test in Kafka connector

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28875: -- Assignee: Gabor Somogyi > Cover Task retry scenario with test in Kafka connec

[jira] [Resolved] (SPARK-28679) Spark Yarn ResourceRequestHelper shouldn't lookup setResourceInformation is no resources specified

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28679. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25403 [https

[jira] [Assigned] (SPARK-28679) Spark Yarn ResourceRequestHelper shouldn't lookup setResourceInformation is no resources specified

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28679: -- Assignee: Alessandro Bellina > Spark Yarn ResourceRequestHelper shouldn't loo

[spark] branch master updated: [SPARK-28679][YARN] changes to setResourceInformation to handle empty resources and reflection error handling

2019-08-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new dd0725d [SPARK-28679][YARN] changes

[jira] [Resolved] (SPARK-28607) Don't hold a reference to two partitionLengths arrays

2019-08-26 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28607. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25341 [https

[spark] branch master updated (13fd32c -> 2efa6f5)

2019-08-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 13fd32c [SPARK-28701][TEST-HADOOP3.2][TEST-JAVA11][K8S] adding java11 support for pull request builds add

<    1   2   3   4   5   6   7   8   9   10   >