Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-28 Thread Marcelo Vanzin
Alright, uploaded the missing packages. I'll send a PR to update the release scripts just in case... On Thu, Jun 28, 2018 at 10:08 AM, Sean Owen wrote: > If it's easy enough to produce them, I agree you can just add them to the RC > dir. > > On Thu, Jun 28, 2018 at 11:56 AM Ma

Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-28 Thread Marcelo Vanzin
a new RC. On Tue, Jun 26, 2018 at 1:25 PM, Marcelo Vanzin wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.1.3. > > The vote is open until Fri, June 29th @ 9PM UTC (2PM PDT) and passes if a > majority +1 PMC votes are cast, with a minimu

Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-28 Thread Marcelo Vanzin
BTW that would be a great fix in the docs now that we'll have a 2.3.2 being prepared. On Thu, Jun 28, 2018 at 9:17 AM, Felix Cheung wrote: > Exactly... > > > From: Marcelo Vanzin > Sent: Thursday, June 28, 2018 9:16:08 AM > To: Tom Graves >

Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-28 Thread Marcelo Vanzin
http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Spark-2-1-2-RC2-tt22540.html#a22555 > > Since it isn’t a regression I’d say +1 from me. > > > ________ > From: Tom Graves > Sent: Thursday, June 28, 2018 6:56:16 AM > To: Marcelo Vanzin

Re: Time for 2.3.2?

2018-06-28 Thread Marcelo Vanzin
28, 2018 at 12:56 PM Saisai Shao >>>>> wrote: >>>>> >>>>>> +1, like mentioned by Marcelo, these issues seems quite severe. >>>>>> >>>>>> I can work on the release if short of hands :). >>>>>> >>>>

Re: Time for 2.3.2?

2018-06-27 Thread Marcelo Vanzin
+1. SPARK-24589 / SPARK-24552 are kinda nasty and we should get fixes for those out. (Those are what delayed 2.2.2 and 2.1.3 for those watching...) On Wed, Jun 27, 2018 at 7:59 PM, Wenchen Fan wrote: > Hi all, > > Spark 2.3.1 was released just a while ago, but unfortunately we discovered > and

Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-27 Thread Marcelo Vanzin
lakes: https://amplab.cs.berkeley.edu/jenkins/user/vanzin/my-views/view/Spark/ (Look for the 2.1 branch jobs.) > ____ > From: Marcelo Vanzin > Sent: Wednesday, June 27, 2018 6:55 PM > To: Felix Cheung > Cc: Marcelo Vanzin; Tom Graves; dev > > Sub

Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-27 Thread Marcelo Vanzin
mes in code not in docs: > singular.ok > Mismatches in argument names: > Position: 16 Code: singular.ok Docs: contrasts > Position: 17 Code: contrasts Docs: ... > > > From: Sean Owen > Sent: Wednesday, June 27, 2018 5:02:37 AM > To: Marcelo Van

Re: [VOTE] Spark 2.2.2 (RC2)

2018-06-27 Thread Marcelo Vanzin
+1 Checked sigs + ran a bunch of tests on the hadoop-2.7 binary package. On Wed, Jun 27, 2018 at 1:30 PM, Tom Graves wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.2.2. > > The vote is open until Mon, July 2nd @ 9PM UTC (2PM PDT) and passes if a > majority

[jira] [Resolved] (SPARK-24533) typesafe has rebranded to lightbend. change the build/mvn endpoint from downloads.typesafe.com to downloads.lightbend.com

2018-06-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24533. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21636 [https

[jira] [Assigned] (SPARK-24533) typesafe has rebranded to lightbend. change the build/mvn endpoint from downloads.typesafe.com to downloads.lightbend.com

2018-06-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24533: -- Assignee: Sanket Reddy > typesafe has rebranded to lightbend. change the build/

[jira] [Resolved] (SPARK-24660) SHS is not showing properly errors when downloading logs

2018-06-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24660. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21644 [https

[jira] [Assigned] (SPARK-24660) SHS is not showing properly errors when downloading logs

2018-06-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24660: -- Assignee: Marco Gaido > SHS is not showing properly errors when downloading l

[jira] [Resolved] (SPARK-24446) Library path with special characters breaks Spark on YARN

2018-06-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24446. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0 > Libr

[jira] [Created] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

2018-06-26 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24663: -- Summary: Flaky test: StreamingContextSuite "stop slow receiver gracefully" Key: SPARK-24663 URL: https://issues.apache.org/jira/browse/SPARK-24663 Proj

[jira] [Assigned] (SPARK-6237) Support uploading blocks > 2GB as a stream

2018-06-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-6237: - Assignee: Imran Rashid > Support uploading blocks > 2GB as a

[jira] [Resolved] (SPARK-6237) Support uploading blocks > 2GB as a stream

2018-06-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-6237. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21346 [https

[VOTE] Spark 2.1.3 (RC2)

2018-06-26 Thread Marcelo Vanzin
Please vote on releasing the following candidate as Apache Spark version 2.1.3. The vote is open until Fri, June 29th @ 9PM UTC (2PM PDT) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.1.3 [ ] -1 Do not release this

Re: [VOTE] Spark 2.1.3 (RC2)

2018-06-26 Thread Marcelo Vanzin
Starting with my own +1. On Tue, Jun 26, 2018 at 1:25 PM, Marcelo Vanzin wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.1.3. > > The vote is open until Fri, June 29th @ 9PM UTC (2PM PDT) and passes if a > majority +1 PMC votes are cast, with

[jira] [Issue Comment Deleted] (SPARK-24631) Cannot up cast column from bigint to smallint as it may truncate

2018-06-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24631: --- Comment: was deleted (was: User 'vanzin' has created a pull request for this issue: https

[jira] [Commented] (SPARK-24631) Cannot up cast column from bigint to smallint as it may truncate

2018-06-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523918#comment-16523918 ] Marcelo Vanzin commented on SPARK-24631: Sorry for the noise, pasted the wrong bug number in my

[jira] [Assigned] (SPARK-24631) Cannot up cast column from bigint to smallint as it may truncate

2018-06-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24631: -- Assignee: Marcelo Vanzin > Cannot up cast column from bigint to smallint as it

[jira] [Assigned] (SPARK-24631) Cannot up cast column from bigint to smallint as it may truncate

2018-06-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24631: -- Assignee: (was: Marcelo Vanzin) > Cannot up cast column from bigint to small

Re: A new link request to my project and one question

2018-06-25 Thread Marcelo Vanzin
.superusers". Here in this case, the sql > server user should be added as a superuser, who can impersonate other > different users. > > Marcelo Vanzin 于2018年6月26日周二 上午9:12写道: > >> You're talking about another service between the user and the application. >> >> In th

Re: A new link request to my project and one question

2018-06-25 Thread Marcelo Vanzin
Sorry, but I'm not familiar with Livy internal logic. > > > On Tue, Jun 26, 2018 at 9:14 AM Marcelo Vanzin > wrote: > >> On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro >> wrote: >> > In that case, I think Livy is useful; the application can pass proxyUs

Re: A new link request to my project and one question

2018-06-25 Thread Marcelo Vanzin
On Mon, Jun 25, 2018 at 5:09 PM, Takeshi Yamamuro wrote: > In that case, I think Livy is useful; the application can pass proxyUser to > build LivyClient for each user > and run spark queries as each user authorization. But Livy already supports impersonation. It can impersonate the

[jira] [Resolved] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24552. Resolution: Fixed Assignee: Ryan Blue Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-24646) Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes

2018-06-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522946#comment-16522946 ] Marcelo Vanzin commented on SPARK-24646: I'm not sure I understand what is the problem here. Can

[jira] [Created] (SPARK-24653) Flaky test "JoinSuite.test SortMergeJoin (with spill)"

2018-06-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24653: -- Summary: Flaky test "JoinSuite.test SortMergeJoin (with spill)" Key: SPARK-24653 URL: https://issues.apache.org/jira/browse/SPARK-24653 Proj

[jira] [Resolved] (SPARK-24532) HiveExternalCatalogVersionSuite should be resilient to missing versions

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24532. Resolution: Won't Fix I've added documentation on the release docs about this test, so

[jira] [Updated] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-22897: --- Fix Version/s: 2.1.3 > Expose stageAttemptId in TaskCont

[jira] [Updated] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24589: --- Fix Version/s: 2.1.3 > OutputCommitCoordinator may allow duplicate comm

[jira] [Resolved] (SPARK-24518) Using Hadoop credential provider API to store password

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24518. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21548 [https

[jira] [Assigned] (SPARK-24518) Using Hadoop credential provider API to store password

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24518: -- Assignee: Saisai Shao > Using Hadoop credential provider API to store passw

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520578#comment-16520578 ] Marcelo Vanzin commented on SPARK-24611: One more: adjust the test so that it ensures that state

[jira] [Commented] (SPARK-23710) Upgrade Hive to 2.3.2

2018-06-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520522#comment-16520522 ] Marcelo Vanzin commented on SPARK-23710: There are a few places in Spark that are affected

[jira] [Resolved] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24620. Resolution: Not A Bug Like "spark.ui.killEnabled"? > [security] Disable

[jira] [Created] (SPARK-24611) Clean up OutputCommitCoordinator

2018-06-20 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24611: -- Summary: Clean up OutputCommitCoordinator Key: SPARK-24611 URL: https://issues.apache.org/jira/browse/SPARK-24611 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24578) Reading remote cache block behavior changes and causes timeout issue

2018-06-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517461#comment-16517461 ] Marcelo Vanzin commented on SPARK-24578: Ah, I see. That makes sense. (I actually took at look

[jira] [Commented] (SPARK-24578) Reading remote cache block behavior changes and causes timeout issue

2018-06-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517449#comment-16517449 ] Marcelo Vanzin commented on SPARK-24578: Attila's suggestion looks good, but I wonder what

[jira] [Commented] (HIVE-19888) Misleading "METASTORE_FILTER_HOOK will be ignored" warning from SessionState

2018-06-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517443#comment-16517443 ] Marcelo Vanzin commented on HIVE-19888: --- Is there anything I should look at here? I can't even find

Re: Time for 2.1.3

2018-06-19 Thread Marcelo Vanzin
12, 2018 at 4:27 PM, Marcelo Vanzin wrote: > Hey all, > > There are some fixes that went into 2.1.3 recently that probably > deserve a release. So as usual, please take a look if there's anything > else you'd like on that release, otherwise I'd like to start with the > process

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24552: --- Target Version/s: 2.1.3, 2.2.2, 2.3.2 Added some target versions. We should take the chance

[jira] [Issue Comment Deleted] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24552: --- Comment: was deleted (was: User 'vanzin' has created a pull request for this issue: https

[jira] [Commented] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516523#comment-16516523 ] Marcelo Vanzin commented on SPARK-24589: [~tgraves] fyi > OutputCommitCoordinator may al

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516522#comment-16516522 ] Marcelo Vanzin commented on SPARK-24552: I forked the output commiter issue into SPARK-24589 so

[jira] [Created] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-18 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24589: -- Summary: OutputCommitCoordinator may allow duplicate commits Key: SPARK-24589 URL: https://issues.apache.org/jira/browse/SPARK-24589 Project: Spark

[jira] [Commented] (SPARK-19084) conditional function: field

2018-06-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516482#comment-16516482 ] Marcelo Vanzin commented on SPARK-19084: (Please ignore my PR above - it should have tagged

[jira] [Resolved] (SPARK-24490) Use WebUI.addStaticHandler in web UIs

2018-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24490. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21510 [https

[jira] [Assigned] (SPARK-24490) Use WebUI.addStaticHandler in web UIs

2018-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24490: -- Assignee: Jacek Laskowski > Use WebUI.addStaticHandler in web

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Fix Version/s: 2.3.2 > HiveExternalCatalogVersionsSuite failing due to missing 2.

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Fix Version/s: 2.2.2 > HiveExternalCatalogVersionsSuite failing due to missing 2.

Re: Issue upgrading to Spark 2.3.1 (Maintenance Release)

2018-06-15 Thread Marcelo Vanzin
I'm not familiar with PyCharm. But if you can run "pyspark" from the command line and not hit this, then this might be an issue with PyCharm or your environment - e.g. having an old version of the pyspark code around, or maybe PyCharm itself might need to be updated. On Thu, Jun 14, 2018 at 10:01

Re: A new link request to my project and one question

2018-06-15 Thread Marcelo Vanzin
re: proxy user, you have to be extremely careful with that. Livy currently supports proxy user, but for the server only. It allows the server to impersonate anyone, so that sessions can run as the requesting user. If you let the user decide who the session will be run as, you'll need to add

[jira] [Resolved] (SPARK-24319) run-example can not print usage

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24319. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21450 [https

[jira] [Assigned] (SPARK-24319) run-example can not print usage

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24319: -- Assignee: Gabor Somogyi > run-example can not print us

[jira] [Moved] (YARN-8430) Some zip files passed with spark-submit --archives causing "invalid CEN header" error

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin moved SPARK-24559 to YARN-8430: -- Affects Version/s: (was: 2.2.0) Component/s

[jira] [Commented] (SPARK-24559) Some zip files passed with spark-submit --archives causing "invalid CEN header" error

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513036#comment-16513036 ] Marcelo Vanzin commented on SPARK-24559: {{\-\-archives}} is completely handled by YARN, so

Re: Spark user classpath setting

2018-06-14 Thread Marcelo Vanzin
I only know of a way to do that with YARN. You can distribute the jar files using "--files" and add just their names (not the full path) to the "extraClassPath" configs. You don't need "userClassPathFirst" in that case. On Thu, Jun 14, 2018 at 1:28 PM, Arjun kr wrote: > Hi All, > > > I am

[jira] [Commented] (HIVE-19888) Misleading "METASTORE_FILTER_HOOK will be ignored" warning from SessionState

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512937#comment-16512937 ] Marcelo Vanzin commented on HIVE-19888: --- (Pinging [~stakiar] for help.) > Mislead

[jira] [Resolved] (SPARK-24563) Allow running PySpark shell without Hive

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24563. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21569 [https

[jira] [Assigned] (SPARK-24563) Allow running PySpark shell without Hive

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24563: -- Assignee: Li Jin > Allow running PySpark shell without H

Re: [ANNOUNCE] Announcing Apache Spark 2.3.1

2018-06-14 Thread Marcelo Vanzin
tured-streaming > Mastering Kafka Streams https://bit.ly/mastering-kafka-streams > Follow me at https://twitter.com/jaceklaskowski > > On Mon, Jun 11, 2018 at 9:47 PM, Marcelo Vanzin wrote: >> >> We are happy to announce the availability of Spark 2.3.1! >> >>

Re: Missing HiveConf when starting PySpark from head

2018-06-14 Thread Marcelo Vanzin
Yes, my bad. The code in session.py needs to also catch TypeError like before. On Thu, Jun 14, 2018 at 11:03 AM, Li Jin wrote: > Sounds good. Thanks all for the quick reply. > > https://issues.apache.org/jira/browse/SPARK-24563 > > > On Thu, Jun 14, 2018 at 12:19 PM, Xiao Li wrote: >> >> Thanks

[jira] [Commented] (HIVE-19888) Misleading "METASTORE_FILTER_HOOK will be ignored" warning from SessionState

2018-06-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512777#comment-16512777 ] Marcelo Vanzin commented on HIVE-19888: --- Tests haven't run, did I name the patch wrongly

[jira] [Updated] (SPARK-23732) Broken link to scala source code in Spark Scala api Scaladoc

2018-06-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23732: --- Fix Version/s: 2.2.2 2.1.3 > Broken link to scala source code in Sp

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511747#comment-16511747 ] Marcelo Vanzin commented on HIVE-16391: --- It would be good to get comments from people on the Hive

[jira] [Updated] (HIVE-19888) Misleading "METASTORE_FILTER_HOOK will be ignored" warning from SessionState

2018-06-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated HIVE-19888: -- Attachment: HIVE-19888.1.patch Status: Patch Available (was: Open) Been a while since

[jira] [Created] (HIVE-19888) Misleading "METASTORE_FILTER_HOOK will be ignored" warning from SessionState

2018-06-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created HIVE-19888: - Summary: Misleading "METASTORE_FILTER_HOOK will be ignored" warning from SessionState Key: HIVE-19888 URL: https://issues.apache.org/jira/browse/HIVE-19888

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

2018-06-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511365#comment-16511365 ] Marcelo Vanzin commented on SPARK-24539: >From a previous chat with [~tgraves] this sounds l

[jira] [Resolved] (SPARK-24506) Spark.ui.filters not applied to /sqlserver/ url

2018-06-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24506. Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0

Time for 2.1.3

2018-06-12 Thread Marcelo Vanzin
Hey all, There are some fixes that went into 2.1.3 recently that probably deserve a release. So as usual, please take a look if there's anything else you'd like on that release, otherwise I'd like to start with the process by early next week. I'll go through jira to see what's the status of

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Target Version/s: 2.2.2, 2.3.2 > HiveExternalCatalogVersionsSuite failing due to miss

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Priority: Blocker (was: Major) > HiveExternalCatalogVersionsSuite failing due to miss

[jira] [Created] (SPARK-24532) HiveExternalCatalogVersionSuite should be resilient to missing versions

2018-06-12 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24532: -- Summary: HiveExternalCatalogVersionSuite should be resilient to missing versions Key: SPARK-24532 URL: https://issues.apache.org/jira/browse/SPARK-24532 Project

[jira] [Commented] (SPARK-24522) Centralize code to deal with security-related HTTP features

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508727#comment-16508727 ] Marcelo Vanzin commented on SPARK-24522: FYI I have a prototype for this that I'll clean up

[jira] [Created] (SPARK-24522) Centralize code to deal with security-related HTTP features

2018-06-11 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24522: -- Summary: Centralize code to deal with security-related HTTP features Key: SPARK-24522 URL: https://issues.apache.org/jira/browse/SPARK-24522 Project: Spark

[ANNOUNCE] Announcing Apache Spark 2.3.1

2018-06-11 Thread Marcelo Vanzin
We are happy to announce the availability of Spark 2.3.1! Apache Spark 2.3.1 is a maintenance release, based on the branch-2.3 maintenance branch of Spark. We strongly recommend all 2.3.x users to upgrade to this stable release. To download Spark 2.3.1, head over to the download page:

[ANNOUNCE] Announcing Apache Spark 2.3.1

2018-06-11 Thread Marcelo Vanzin
We are happy to announce the availability of Spark 2.3.1! Apache Spark 2.3.1 is a maintenance release, based on the branch-2.3 maintenance branch of Spark. We strongly recommend all 2.3.x users to upgrade to this stable release. To download Spark 2.3.1, head over to the download page:

[jira] [Resolved] (SPARK-24511) Spark WebUI allows Weak TLS Protocols

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24511. Resolution: Not A Problem The default in jdk8 is 1.2. If you configure your application

[jira] [Resolved] (SPARK-24508) Spark WebUIs [Security] - Inadequate Cache Directive Headers

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24508. Resolution: Won't Fix > Spark WebUIs [Security] - Inadequate Cache Directive Head

[jira] [Reopened] (SPARK-24508) Spark WebUIs [Security] - Inadequate Cache Directive Headers

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-24508: > Spark WebUIs [Security] - Inadequate Cache Directive Head

[jira] [Resolved] (SPARK-24508) Spark WebUIs [Security] - Inadequate Cache Directive Headers

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24508. Resolution: Fixed The "impact" doesn't really explain anything that would make

[jira] [Resolved] (SPARK-24509) Spark WebUI [security] - Web Server Version Disclosure

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24509. Resolution: Won't Fix The version of the http server Spark uses isn't really a secret

[jira] [Resolved] (SPARK-24510) Spark WebUI filters use Basic Authentication [security]

2018-06-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24510. Resolution: Not A Bug Please read the documentation. Spark doesn't provide any

[jira] [Updated] (SPARK-24106) Spark Structure Streaming with RF model taking long time in processing probability for each mini batch

2018-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24106: --- Fix Version/s: (was: 2.4.0) (was: 2.3.0) > Spark Struct

[jira] [Updated] (SPARK-24106) Spark Structure Streaming with RF model taking long time in processing probability for each mini batch

2018-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24106: --- Target Version/s: (was: 2.2.1, 2.3.0) > Spark Structure Streaming with RF model tak

[VOTE] [RESULT] Spark 2.3.1 (RC4)

2018-06-08 Thread Marcelo Vanzin
The vote passes. Thanks to all who helped with the release! I'll follow up later with a release announcement once everything is published. +1 (* = binding): - Marcelo Vanzin * - Reynold Xin * - Sean Owen * - Denny Lee - Dongjoon Hyun - Ricardo Almeida - Hyukjin Kwon - John Zhuge - Mark Hamstra

Re: Time for 2.2.2 release

2018-06-07 Thread Marcelo Vanzin
Took a look at our branch and most of the stuff that is not already in 2.2 are flaky test fixes, so +1. On Wed, Jun 6, 2018 at 7:54 AM, Tom Graves wrote: > Hello all, > > I think its time for another 2.2 release. > I took a look at Jira and I don't see anything explicitly targeted for 2.2.2 >

Re: [SparkLauncher] stateChanged event not received in standalone cluster mode

2018-06-06 Thread Marcelo Vanzin
That feature has not been implemented yet. https://issues.apache.org/jira/browse/SPARK-11033 On Wed, Jun 6, 2018 at 5:18 AM, Behroz Sikander wrote: > I have a client application which launches multiple jobs in Spark Cluster > using SparkLauncher. I am using Standalone cluster mode. Launching

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502035#comment-16502035 ] Marcelo Vanzin commented on HIVE-16391: --- bq. I'm not sure if there's a way to publish two pom files

Re: [VOTE] Spark 2.3.1 (RC4)

2018-06-02 Thread Marcelo Vanzin
ork for me > either (even building with -Phadoop-2.7). I guess I’ve been relying on an > unsupported pattern and will need to figure something else out going forward > in order to use s3a://. > > > On Fri, Jun 1, 2018 at 9:09 PM Marcelo Vanzin wrote: >> >> I have

Re: [VOTE] Spark 2.3.1 (RC4)

2018-06-01 Thread Marcelo Vanzin
ct) and figure > out what I need to change (as due diligence for Flintrock’s users). > > Nick > > > On Fri, Jun 1, 2018 at 8:21 PM Marcelo Vanzin wrote: >> >> Using the hadoop-aws package is probably going to be a little more >> complicated than that. The best

Re: [VOTE] Spark 2.3.1 (RC4)

2018-06-01 Thread Marcelo Vanzin
= local-m2-cache: tried > > file:/home/ec2-user/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar > > I’d guess I’m probably using the wrong version of hadoop-aws, but I called > make-distribution.sh with -Phadoop-2.8 so I’m not sure what else to try. > >

Re: [VOTE] Spark 2.3.1 (RC4)

2018-06-01 Thread Marcelo Vanzin
Starting with my own +1 (binding). On Fri, Jun 1, 2018 at 3:28 PM, Marcelo Vanzin wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.3.1. > > Given that I expect at least a few people to be busy with Spark Summit next > week, I'm taking the lib

[VOTE] Spark 2.3.1 (RC4)

2018-06-01 Thread Marcelo Vanzin
Please vote on releasing the following candidate as Apache Spark version 2.3.1. Given that I expect at least a few people to be busy with Spark Summit next week, I'm taking the liberty of setting an extended voting period. The vote will be open until Friday, June 8th, at 19:00 UTC (that's 12:00

[jira] [Updated] (SPARK-24369) A bug when having multiple distinct aggregations

2018-06-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24369: --- Fix Version/s: (was: 2.3.1) > A bug when having multiple distinct aggregati

Re: [VOTE] Spark 2.3.1 (RC3)

2018-06-01 Thread Marcelo Vanzin
. On Fri, Jun 1, 2018 at 1:20 PM, Xiao Li wrote: > Sorry, I need to say -1 > > This morning, just found a regression in 2.3.1 and reverted > https://github.com/apache/spark/pull/21443 > > Xiao > > 2018-06-01 13:09 GMT-07:00 Marcelo Vanzin : >> >> Please vote

[VOTE] Spark 2.3.1 (RC3)

2018-06-01 Thread Marcelo Vanzin
Please vote on releasing the following candidate as Apache Spark version 2.3.1. Given that I expect at least a few people to be busy with Spark Summit next week, I'm taking the liberty of setting an extended voting period. The vote will be open until Friday, June 8th, at 19:00 UTC (that's 12:00

[jira] [Resolved] (SPARK-24451) Spark-Streaming-Kafka-1.6.3- KafkaUtils.createStream function uses the old Logging class

2018-06-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24451. Resolution: Invalid You're trying to use a library built for Spark 1.6.3 on top of Spark

<    6   7   8   9   10   11   12   13   14   15   >