[jira] [Created] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2

2019-02-26 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26997: -- Summary: k8s integration tests failing after client upgraded to 4.1.2 Key: SPARK-26997 URL: https://issues.apache.org/jira/browse/SPARK-26997 Project: Spark

[jira] [Resolved] (SPARK-26674) Consolidate CompositeByteBuf when reading large frame

2019-02-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26674. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23602 [https

[jira] [Assigned] (SPARK-26674) Consolidate CompositeByteBuf when reading large frame

2019-02-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26674: -- Assignee: liupengcheng > Consolidate CompositeByteBuf when reading large fr

[spark] branch master updated: [SPARK-26674][CORE] Consolidate CompositeByteBuf when reading large frame

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 52a180f [SPARK-26674][CORE] Consolidate

[spark] branch master updated: [SPARK-26674][CORE] Consolidate CompositeByteBuf when reading large frame

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 52a180f [SPARK-26674][CORE] Consolidate

[jira] [Created] (SPARK-26989) Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries

2019-02-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26989: -- Summary: Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries Key: SPARK-26989 URL: https://issues.apache.org

[spark] branch master updated: [SPARK-25035][CORE] Avoiding memory mapping at disk-stored blocks replication

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0ac516b [SPARK-25035][CORE] Avoiding memory

[jira] [Resolved] (SPARK-25035) Replicating disk-stored blocks should avoid memory mapping

2019-02-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25035. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23688 [https

[jira] [Assigned] (SPARK-25035) Replicating disk-stored blocks should avoid memory mapping

2019-02-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25035: -- Assignee: Attila Zsolt Piros > Replicating disk-stored blocks should avoid mem

[spark] branch master updated: [SPARK-25035][CORE] Avoiding memory mapping at disk-stored blocks replication

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0ac516b [SPARK-25035][CORE] Avoiding memory

[spark] branch branch-2.3 updated: [MINOR][BUILD] Update all checkstyle dtd to use "https://checkstyle.org"

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 3ece965 [MINOR][BUILD] Update all

[spark] branch branch-2.4 updated: [MINOR][BUILD] Update all checkstyle dtd to use "https://checkstyle.org"

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new b031f4a [MINOR][BUILD] Update all

[spark] branch branch-2.3 updated: [MINOR][BUILD] Update all checkstyle dtd to use "https://checkstyle.org"

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 3ece965 [MINOR][BUILD] Update all

[spark] branch master updated: [MINOR][BUILD] Update all checkstyle dtd to use "https://checkstyle.org"

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c5de804 [MINOR][BUILD] Update all checkstyle

[spark] branch master updated: [MINOR][BUILD] Update all checkstyle dtd to use "https://checkstyle.org"

2019-02-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c5de804 [MINOR][BUILD] Update all checkstyle

[jira] [Assigned] (SPARK-26895) When running spark 2.3 as a proxy user (--proxy-user), SparkSubmit fails to resolve globs owned by target user

2019-02-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26895: -- Assignee: Alessandro Bellina > When running spark 2.3 as a proxy user (--proxy-u

[spark] branch master updated: [SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users

2019-02-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 79a6504 [SPARK-26895][CORE

[spark] branch master updated: [SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users

2019-02-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 79a6504 [SPARK-26895][CORE

[jira] [Resolved] (SPARK-26895) When running spark 2.3 as a proxy user (--proxy-user), SparkSubmit fails to resolve globs owned by target user

2019-02-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26895. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23806 [https

Re: [VOTE] Redirect github notifications to issues@

2019-02-22 Thread Marcelo Vanzin
Jira link: https://issues.apache.org/jira/browse/INFRA-17892 On Fri, Feb 22, 2019 at 9:44 AM Marcelo Vanzin wrote: > > Thanks all, vote passes with 11 +1s, no -1s. > > I'll file the INFRA ticket and link it back here soon. > > On Tue, Feb 19, 2019 at 1:35 PM Marcelo Vanzin

Re: [VOTE] Redirect github notifications to issues@

2019-02-22 Thread Marcelo Vanzin
Thanks all, vote passes with 11 +1s, no -1s. I'll file the INFRA ticket and link it back here soon. On Tue, Feb 19, 2019 at 1:35 PM Marcelo Vanzin wrote: > > I'm opening a vote based on recent discussions about the extra noise > generated by github updates going to dev@. So pl

[jira] [Resolved] (SPARK-26954) Do not attemp when user code throws exception

2019-02-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26954. Resolution: Not A Problem See discussion in the PR for why this is not a problem

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-02-20 Thread Marcelo Vanzin
Just wanted to point out that https://issues.apache.org/jira/browse/SPARK-26859 is not in this RC, and is marked as a correctness bug. (The fix is in the 2.4 branch, just not in rc2.) On Wed, Feb 20, 2019 at 12:07 PM DB Tsai wrote: > > Please vote on releasing the following candidate as Apache

[jira] [Resolved] (SPARK-26877) Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified

2019-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26877. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23786 [https

[jira] [Assigned] (SPARK-26877) Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified

2019-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26877: -- Assignee: liupengcheng > Support user-level app staging directory in yarn mode w

[spark] branch master updated: [SPARK-26877][YARN] Support user-level app staging directory in yarn mode when spark.yarn…

2019-02-20 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new eb6fd7e [SPARK-26877][YARN] Support user-level

Re: Github notification and Jira Issues (was: Re: [VOTE] Redirect github notifications to issues@)

2019-02-20 Thread Marcelo Vanzin
On Wed, Feb 20, 2019 at 11:13 AM Pascal Schumacher wrote: > Am 20.02.2019 um 19:39 schrieb Marcelo Vanzin: > > Rob: > >> I almost think that we should have Pull Requests generate jiras. > > I've seen this set up in a couple of projects and jira becomes > > unrea

Re: [VOTE] Redirect github notifications to issues@

2019-02-20 Thread Marcelo Vanzin
On Wed, Feb 20, 2019 at 5:41 AM Gary Gregory wrote: > Is this a LAZY VOTE? Sorry, but not familiar with the semantics of when to call a lazy vs. non-lazy vote. Given the current number of votes, does it matter? Rob: > I almost think that we should have Pull Requests generate jiras. I've seen

[jira] [Resolved] (CRYPTO-138) failed to run on openssl 1.1.0g

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/CRYPTO-138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved CRYPTO-138. --- Resolution: Fixed Fix Version/s: 1.1.0 Should be fixed now: https://github.com/apache

[jira] [Resolved] (SPARK-26933) spark-submit does not make zip files provided with --py-files visible to pyspark

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26933. Resolution: Not A Problem Did something wrong when testing this in the morning

[jira] [Resolved] (SPARK-24894) Invalid DNS name due to hostname truncation

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24894. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 3.0.0 > Inva

[spark] branch master updated: [SPARK-24894][K8S] Make sure valid host names are created for executors.

2019-02-19 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 61c3cdc [SPARK-24894][K8S] Make sure valid host

[jira] [Resolved] (SPARK-26882) lint-scala script does not check all components

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26882. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23792 [https

[spark] branch master updated: [SPARK-26882] Check the Kubernetes integration tests scalatyle

2019-02-19 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6b3c832 [SPARK-26882] Check the Kubernetes

[jira] [Assigned] (SPARK-26891) Flaky test:YarnSchedulerBackendSuite."RequestExecutors reflects node blacklist and is serializable"

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26891: -- Assignee: Attila Zsolt Piros > Flaky test:YarnSchedulerBackendSu

[jira] [Resolved] (SPARK-26891) Flaky test:YarnSchedulerBackendSuite."RequestExecutors reflects node blacklist and is serializable"

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26891. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23801 [https

[spark] branch master updated: [SPARK-26891][YARN] Fixing flaky test in YarnSchedulerBackendSuite

2019-02-19 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e4e4e2b [SPARK-26891][YARN] Fixing flaky test

[VOTE] Redirect github notifications to issues@

2019-02-19 Thread Marcelo Vanzin
I'm opening a vote based on recent discussions about the extra noise generated by github updates going to dev@. So please vote: - +1 to redirect github updates of all commons repos to the issues@ list - -1 to keep things as is If the vote passes, I'll take care of opening an infra ticket

[jira] [Created] (SPARK-26934) python dependencies with "local:" URIs are not visible to executors

2019-02-19 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26934: -- Summary: python dependencies with "local:" URIs are not visible to executors Key: SPARK-26934 URL: https://issues.apache.org/jira/browse/SPARK-26934

[jira] [Created] (SPARK-26933) spark-submit does not make zip files provided with --py-files visible to pyspark

2019-02-19 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26933: -- Summary: spark-submit does not make zip files provided with --py-files visible to pyspark Key: SPARK-26933 URL: https://issues.apache.org/jira/browse/SPARK-26933

[jira] [Commented] (SPARK-24736) --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772337#comment-16772337 ] Marcelo Vanzin commented on SPARK-24736: I'm going to fork this issue into 2: - I'll leave

[commons-crypto] branch master updated: OpenSSL 1.1.0 updates with backward compatibility for OpenSSL 1.0.2 and 1.0.1 (#92)

2019-02-19 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/commons-crypto.git The following commit(s) were added to refs/heads/master by this push: new 2875340 OpenSSL 1.1.0 updates

[jira] [Updated] (SPARK-26873) FileFormatWriter creates inconsistent MR job IDs

2019-02-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26873: --- Fix Version/s: 2.3.4 > FileFormatWriter creates inconsistent MR job

[spark] branch branch-2.3 updated: [SPARK-26873][SQL] Use a consistent timestamp to build Hadoop Job IDs.

2019-02-19 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 41df43f [SPARK-26873][SQL] Use

Re: merge script stopped working; Python 2/3 input() issue?

2019-02-15 Thread Marcelo Vanzin
BTW the main script has this that the website script does not: if sys.version < '3': input = raw_input # noqa On Fri, Feb 15, 2019 at 3:55 PM Sean Owen wrote: > > I'm seriously confused on this one. The spark-website merge script > just stopped working for me. It fails on the call to

Re: merge script stopped working; Python 2/3 input() issue?

2019-02-15 Thread Marcelo Vanzin
You're talking about the spark-website script, right? The main repo's script has been working for me, the website one is broken. I think it was caused by this dude changing raw_input to input recently: commit 8b6e7dceaf5d73de3f92907ceeab8925a2586685 Author: Sean Owen Date: Sat Jan 19 19:02:30

[jira] [Resolved] (LIVY-556) HearbeatExpired is not stubbed correctly in test cases

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-556. - Resolution: Fixed Issue resolved by pull request 143 [https://github.com/apache/incubator-livy

[incubator-livy] branch master updated: [LIVY-556] HearbeatExpired is not stubbed correctly in test cases

2019-02-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new 7d9b453 [LIVY-556] HearbeatExpired

[jira] [Resolved] (LIVY-557) Make Travis-CI logs less verbose

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-557. - Resolution: Fixed Issue resolved by pull request 144 [https://github.com/apache/incubator-livy

[incubator-livy] branch master updated: [LIVY-557] Make Travis-CI logs less verbose

2019-02-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new 2240c71 [LIVY-557] Make Travis-CI logs

svn commit: r32523 - /dev/spark/v2.3.3-rc2-bin/ /release/spark/spark-2.3.3/

2019-02-15 Thread vanzin
Author: vanzin Date: Fri Feb 15 23:02:24 2019 New Revision: 32523 Log: Release Spark 2.3.3. Added: release/spark/spark-2.3.3/ - copied from r32522, dev/spark/v2.3.3-rc2-bin/ Removed: dev/spark/v2.3.3-rc2-bin

[jira] [Resolved] (SPARK-26772) Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26772. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23686 [https

[jira] [Assigned] (SPARK-26772) Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26772: -- Assignee: Gabor Somogyi > Delete ServiceCredentialProvider and m

[spark] branch master updated: [SPARK-26772][YARN] Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 28ced38 [SPARK-26772][YARN] Delete

[jira] [Commented] (SPARK-26888) Upgrade to Log4j 2.x

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769740#comment-16769740 ] Marcelo Vanzin commented on SPARK-26888: SPARK-6305? > Upgrade to Log4j

[jira] [Resolved] (SPARK-26790) Yarn executor to self-retrieve log urls and attributes

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26790. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23706 [https

[jira] [Assigned] (SPARK-26790) Yarn executor to self-retrieve log urls and attributes

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26790: -- Assignee: Jungtaek Lim > Yarn executor to self-retrieve log urls and attribu

[jira] [Resolved] (SPARK-23891) Debian based Dockerfile

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23891. Resolution: Duplicate I'm consolidating all docker image-related bugs under SPARK-24655

[jira] [Resolved] (SPARK-26398) Support building GPU docker images

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26398. Resolution: Duplicate I'm consolidating all docker image-related bugs under SPARK-24655

[jira] [Resolved] (SPARK-26597) Support using images with different entrypoints on Kubernetes

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26597. Resolution: Duplicate I'm consolidating all docker image-related bugs under SPARK-24655

[jira] [Resolved] (SPARK-26773) Consider alternative base images for Kubernetes

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26773. Resolution: Duplicate I'm consolidating all docker image-related bugs under SPARK-24655

[spark] branch master updated: [SPARK-26790][CORE] Change approach for retrieving executor logs and attributes: self-retrieve

2019-02-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b6c6875 [SPARK-26790][CORE] Change approach

[DISCUSS] Change github notifications for all commons sub-projects

2019-02-15 Thread Marcelo Vanzin
Hey all, There was a recent thread ([1]) with a brief discussion about the number of github updates that are currently ending up in the dev@ mailing list. Personally I find that a little too noisy (especially since I get 2 e-mails for repos that I'm subscribed to), and it seems others also don't

[jira] [Assigned] (SPARK-25922) [K8] Spark Driver/Executor "spark-app-selector" label mismatch

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25922: -- Assignee: Wang, Xinglong > [K8] Spark Driver/Executor "spark-app-selecto

[jira] [Resolved] (SPARK-25922) [K8] Spark Driver/Executor "spark-app-selector" label mismatch

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25922. Resolution: Fixed Fix Version/s: 2.4.1 Issue resolved by pull request 23779 [https

[jira] [Updated] (SPARK-25922) [K8] Spark Driver/Executor "spark-app-selector" label mismatch

2019-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25922: --- Fix Version/s: 3.0.0 > [K8] Spark Driver/Executor "spark-app-selector" l

[spark] branch branch-2.4 updated: [SPARK-25922][K8S] Spark Driver/Executor "spark-app-selector" label mismatch (branch-2.4)

2019-02-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new fccc6d3 [SPARK-25922][K8S] Spark Driver

[jira] [Resolved] (SPARK-23082) Allow separate node selectors for driver and executors in Kubernetes

2019-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23082. Resolution: Won't Fix Pretty sure this is covered by pod templates. > Allow separ

[jira] [Resolved] (SPARK-24353) Add support for pod affinity/anti-affinity

2019-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24353. Resolution: Won't Fix Covered by pod templates, I'm almost sure. > Add support for

[jira] [Resolved] (SPARK-24599) SPARK_MOUNTED_CLASSPATH contains incorrect semicolon on Windows

2019-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24599. Resolution: Won't Fix SPARK_MOUNTED_CLASSPATH doesn't exist anymore, so I'm assuming

[jira] [Commented] (SPARK-24736) --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.

2019-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768538#comment-16768538 ] Marcelo Vanzin commented on SPARK-24736: This isn't about staging local files. This is about

[jira] [Resolved] (SPARK-26873) FileFormatWriter creates inconsistent MR job IDs

2019-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26873. Resolution: Fixed Assignee: Ryan Blue Fix Version/s: 3.0.0

[spark] branch branch-2.4 updated: [SPARK-26873][SQL] Use a consistent timestamp to build Hadoop Job IDs.

2019-02-14 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new bc1e960 [SPARK-26873][SQL] Use

[spark] branch master updated: [SPARK-26873][SQL] Use a consistent timestamp to build Hadoop Job IDs.

2019-02-14 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 4e2 [SPARK-26873][SQL] Use a consistent

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767724#comment-16767724 ] Marcelo Vanzin commented on SPARK-25355: Doesn't that work already? I don't see any checks

[jira] [Assigned] (SPARK-25261) Standardize the default units of spark.driver|executor.memory

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25261: -- Assignee: Marcelo Vanzin > Standardize the default units of spark.dri

[jira] [Assigned] (SPARK-25261) Standardize the default units of spark.driver|executor.memory

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25261: -- Assignee: (was: Marcelo Vanzin) > Standardize the default units of spark.dri

[jira] [Commented] (SPARK-26150) __spark_conf__XXX.zip doesn't exist

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767617#comment-16767617 ] Marcelo Vanzin commented on SPARK-26150: If you can provide the full YARN logs for your

[jira] [Resolved] (SPARK-25766) AMCredentialRenewer can leak FS clients

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25766. Resolution: Not A Problem I took a look at the new code (and some of the {{FileSystem

[jira] [Resolved] (SPARK-9209) Using executor allocation, a executor is removed but it exists in ExecutorsPage of the web ui

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-9209. --- Resolution: Not A Problem Dead executors are now explicitly kept by Spark, so

[jira] [Resolved] (SPARK-8622) Spark 1.3.1 and 1.4.0 doesn't put executor working directory on executor classpath

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-8622. --- Resolution: Not A Problem This works as designed. {{--jars}} are added to the Spark class

Re: [OT][GIT help] How to restore a branch in a github fork

2019-02-12 Thread Marcelo Vanzin
view. IMO it doesn't really make a whole lot of sense to have the upstream repo branches in your fork, since they're not kept in sync after the fork. e.g. https://github.com/vanzin/spark/tree/master is a few years behind the current master... I never actually use my own fork's master, it's just

Re: [OT][GIT help] How to restore a branch in a github fork

2019-02-12 Thread Marcelo Vanzin
On Tue, Feb 12, 2019 at 4:44 PM sebb wrote: > > On Wed, 13 Feb 2019 at 00:31, Marcelo Vanzin > wrote: > > > > On Tue, Feb 12, 2019 at 4:25 PM sebb wrote: > > > It's shown as one of 'my' branches, but I guess I can live with that. > > > > Not su

Re: [OT][GIT help] How to restore a branch in a github fork

2019-02-12 Thread Marcelo Vanzin
On Tue, Feb 12, 2019 at 4:25 PM sebb wrote: > It's shown as one of 'my' branches, but I guess I can live with that. Not sure what you mean by that? The branches in your fork don't really have any relation to the original repo. They may have the same name and even reference the same commit, but

Re: [OT][GIT help] How to restore a branch in a github fork

2019-02-12 Thread Marcelo Vanzin
On Tue, Feb 12, 2019 at 3:58 PM sebb wrote: > Deleted from local and remote. > But it still exists in the upstream from which the Github repo was forked > > It is still in the upstream source - that is a 3rd party repo > How can I restore it from there? It's easy then: - make sure you have the

[jira] [Commented] (SPARK-26395) Spark Thrift server memory leak

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766547#comment-16766547 ] Marcelo Vanzin commented on SPARK-26395: The code that cleans up stages does clean up the RDD

[jira] [Resolved] (SPARK-26588) Idle executor should properly be killed when no job is submitted

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26588. Resolution: Duplicate > Idle executor should properly be killed when no job is submit

Re: [OT][GIT help] How to restore a branch in a github fork

2019-02-12 Thread Marcelo Vanzin
If: - you deleted the branch from the remote Just push your local branch to the remote. - you deleted the branch from the local repo only Just checkout the remote branch again. If you had local changes that are not in the remote, continue below. - you deleted the branch from local and it did

[jira] [Updated] (SPARK-26770) Misleading/unhelpful error message when wrapping a null in an Option

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26770: --- Component/s: (was: Spark Core) SQL > Misleading/unhelpful er

[jira] [Resolved] (SPARK-25917) Spark UI's executors page loads forever when memoryMetrics in None. Fix is to JSON ignore memorymetrics when it is None.

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25917. Resolution: Cannot Reproduce > Spark UI's executors page loads forever when memoryMetr

[jira] [Updated] (SPARK-26631) Issue while reading Parquet data from Hadoop Archive files (.har)

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26631: --- Component/s: (was: Spark Core) SQL > Issue while reading Parquet d

[jira] [Updated] (SPARK-25987) StackOverflowError when executing many operations on a table with many columns

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25987: --- Component/s: (was: Spark Core) SQL > StackOverflowError when execut

[jira] [Updated] (SPARK-26150) __spark_conf__XXX.zip doesn't exist

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26150: --- Component/s: (was: Spark Submit) (was: Spark Core

[jira] [Updated] (SPARK-26240) [pyspark] Updating illegal column names with withColumnRenamed does not change schema changes, causing pyspark.sql.utils.AnalysisException

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26240: --- Component/s: (was: Spark Core) SQL > [pyspark] Updating illegal col

[jira] [Updated] (SPARK-26325) Interpret timestamp fields in Spark while reading json (timestampFormat)

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26325: --- Component/s: (was: Spark Core) SQL > Interpret timestamp fie

[jira] [Updated] (SPARK-26320) udf with multiple arrays as input

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26320: --- Component/s: (was: Spark Core) SQL > udf with multiple arrays as in

[jira] [Resolved] (SPARK-26279) Remove unused method in Logging

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26279. Resolution: Won't Fix > Remove unused method in Logg

[jira] [Resolved] (SPARK-26417) Make comments for states available for logging

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26417. Resolution: Won't Fix If you're talking about the constants in {{SparkAppHandle.State

[jira] [Updated] (SPARK-26436) Dataframe resulting from a GroupByKey and flatMapGroups operation throws java.lang.UnsupportedException when groupByKey is applied on it.

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26436: --- Component/s: (was: Spark Core) SQL > Dataframe resulting f

[jira] [Updated] (SPARK-26509) Parquet DELTA_BYTE_ARRAY is not supported in Spark 2.x's Vectorized Reader

2019-02-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26509: --- Component/s: (was: Spark Core) > Parquet DELTA_BYTE_ARRAY is not supported in Spar

<    3   4   5   6   7   8   9   10   11   12   >