Re: The process of Livy 0.4.0-incubating release

will write a doc as well as release script for the next release. Sorry about it. Best regards, Jerry On Wed, Aug 30, 2017 at 2:10 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > Hi Luciano, > > I'm trying to push maven artifacts to maven staging repository at > repository.apache.

Re: The process of Livy 0.4.0-incubating release

is profile for Livy? Also how can I create this profile? Thanks Jerry On Mon, Aug 28, 2017 at 3:39 PM, Luciano Resende <luckbr1...@gmail.com> wrote: > On Sun, Aug 27, 2017 at 11:53 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > > > Hi mentors, > > > > Wou

[jira] [Commented] (SPARK-21829) Enable config to permanently blacklist a list of nodes

[ https://issues.apache.org/jira/browse/SPARK-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144889#comment-16144889 ] Saisai Shao commented on SPARK-21829: - I understand you concern, since it is much easier for you

Re: Update ASF GitHub Bot Settings?

b.com/ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details for Saisai Shao ---08/28/2017 06:53:28 > PM---Hi Alex, Do you want to achieve what Spark did in the JIRA? A]Saisai > Shao ---08/28/2017 06:53:28

Re: Update ASF GitHub Bot Settings?

Hi Alex, Do you want to achieve what Spark did in the JIRA? AFAIK Spark uses a script to sync between github and jira, it doesn't enable gihub robot to mirror everything from github. Thanks Jerry On Tue, Aug 29, 2017 at 5:00 AM, Alex Bozarth wrote: > Opened an INFRA JIRA

The process of Livy 0.4.0-incubating release

Hi mentors, Would you please guide us on how to release the Livy 0.4.0-incubating, including artifact publish. Also regarding push artifacts to repo, are we inclining to change to Maven central repo, or we still use Cloudera repo, any suggestion? Besides, for Python package release, do we need

[RESULT][VOTE] Apache Livy (incubating) 0.4.0 release RC2

Hi All, The vote for releasing Apache Livy 0.4.0-incubating passed with 4 binding +1s, 3 non-binding +1s, and no 0 or -1. Binding +1s: Bikas Saha Jean-Baptiste Onofré Brock Noland Luciano Resende The votes were ( https://www.mail-archive.com/general@incubator.apache.org/msg60990.html) Thanks

[jira] [Commented] (SPARK-21829) Enable config to permanently blacklist a list of nodes

[ https://issues.apache.org/jira/browse/SPARK-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141081#comment-16141081 ] Saisai Shao commented on SPARK-21829: - Cross post the comment here. Since you're running Spark

[jira] [Closed] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-21819. --- Resolution: Not A Problem > UserGroupInformation initialization in SparkHadoopUtilwill overwr

[jira] [Commented] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139590#comment-16139590 ] Saisai Shao commented on SPARK-21819: - Then I think there should no issue in Spark, right

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139502#comment-16139502 ] Saisai Shao commented on SPARK-17321: - 1. if NM recovery is enabled, then yarn will provide

Please vote for Apache Livy 0.4.0-incubating release

Hi teams, Would you please help to vote for the Apache Livy release in general incubator mail list. Thanks a lot. http://mail-archives.apache.org/mod_mbox/incubator-general/201708.mbox/%3CCANvfmP8sAoofvRcs9AT5S-VW4LfiWeGfRXP2rhvEP4zyTag%2BYQ%40mail.gmail.com%3E Thanks

Re: Livy with Spark package

You could set "spark.jars.packages" in `conf` field of session post API ( https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md#post-sessions). This is equal to --package in spark-submit. BTW you'd better ask livy question in u...@livy.incubator.apache.org. Thanks Jerry On Thu,

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138320#comment-16138320 ] Saisai Shao commented on SPARK-17321: - We're facing the same issue. I think YARN shuffle service

[jira] [Commented] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138308#comment-16138308 ] Saisai Shao commented on SPARK-21819: - I'm not sure if Spark expose the user API to set

[jira] [Commented] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138300#comment-16138300 ] Saisai Shao commented on SPARK-21819: - I think here because `Configuration` object created

[jira] [Commented] (SPARK-21660) Yarn ShuffleService failed to start when the chosen directory become read-only

[ https://issues.apache.org/jira/browse/SPARK-21660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137842#comment-16137842 ] Saisai Shao commented on SPARK-21660: - Yes [~hyukjin.kwon] this is a dup JIRA. > Yarn ShuffleServ

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

Nan, I think Meisam already had a PR about this this, maybe you can discuss with him on the github based on the proposed code. Sorry I didn't follow the long discussion thread, but I think Paypal's solution sounds simpler. On Wed, Aug 23, 2017 at 12:07 AM, Nan Zhu wrote:

Re: [VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

OK, sure, I will remove RC1 from the directory. Thanks Jerry On Tue, Aug 22, 2017 at 7:24 PM, John D. Ament wrote: > Hi, > > Looking at your release, it's confusing what we are voting on. If RC2 is > under vote, please remove RC1 from this directory. > > John > > On

Re: [VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

OK, sure, I will remove RC1 from the directory. Thanks Jerry On Tue, Aug 22, 2017 at 7:24 PM, John D. Ament wrote: > Hi, > > Looking at your release, it's confusing what we are voting on. If RC2 is > under vote, please remove RC1 from this directory. > > John > > On

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136454#comment-16136454 ] Saisai Shao commented on SPARK-21733: - [~1028344...@qq.com], I'm really not sure what are you trying

[jira] [Resolved] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21733. - Resolution: Not A Problem > ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL T

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136334#comment-16136334 ] Saisai Shao commented on SPARK-21733: - I'm going to close this issue again because the behavior

[jira] [Comment Edited] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136326#comment-16136326 ] Saisai Shao edited comment on SPARK-21733 at 8/22/17 5:41 AM

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136326#comment-16136326 ] Saisai Shao commented on SPARK-21733: - This is because executor is killed by NM with SIGTERM

[jira] [Commented] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136196#comment-16136196 ] Saisai Shao commented on SPARK-21798: - I think this one could be used, looks like there's no other

Re: Spark Web UI SSL Encryption

Can you please post the specific problem you met? Thanks Jerry On Sat, Aug 19, 2017 at 1:49 AM, Anshuman Kumar wrote: > Hello, > > I have recently installed Sparks 2.2.0, and trying to use it for some big > data processing. Spark is installed on a server that I

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130078#comment-16130078 ] Saisai Shao commented on SPARK-21752: - For configurations like "spark.jars.packages&qu

Help to verify Apache Livy 0.4.0-incubating release

Hi all, We're under progress to make a first Apache release of Livy (0.4.0-incubating), we really hope you could verify the RC2[1] release (binary and source) locally and return us the feedbacks. We will call for an incubation vote next week if everything is fine. Thanks a lot for your help.

[jira] [Commented] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

[ https://issues.apache.org/jira/browse/SPARK-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127199#comment-16127199 ] Saisai Shao commented on SPARK-21714: - Let me take a crack on this if no one is working

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126974#comment-16126974 ] Saisai Shao commented on SPARK-21733: - Should the resolution of this JIRA be "invalid&quo

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126894#comment-16126894 ] Saisai Shao commented on SPARK-21733: - Is there any issue here? > ER

[jira] [Commented] (SPARK-9104) expose network layer memory usage in shuffle part

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125331#comment-16125331 ] Saisai Shao commented on SPARK-9104: I think it is quite useful to get the memory details of Netty, so

[jira] [Commented] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

[ https://issues.apache.org/jira/browse/SPARK-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125115#comment-16125115 ] Saisai Shao commented on SPARK-21714: - I noticed this issue before and tried to fix

Re: Spark 2.1.x client with 2.2.0 cluster

As I remembered using Spark 2.1 Driver to communicate with Spark 2.2 executors will throw some RPC exceptions (I don't remember the details of exception). On Thu, Aug 10, 2017 at 4:23 PM, Ted Yu wrote: > Hi, > Has anyone used Spark 2.1.x client with Spark 2.2.0 cluster ? >

[jira] [Commented] (SPARK-21660) Yarn ShuffleService failed to start when the chosen directory become read-only

[ https://issues.apache.org/jira/browse/SPARK-21660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121125#comment-16121125 ] Saisai Shao commented on SPARK-21660: - Will yarn NM handle this bad disk problem and return a good

Re: To release a first Apache version Livy

cense file (see below as an example) > > http://svn.apache.org/repos/asf/tuscany/sca-java-2.x/ > trunk/distribution/all/src/main/release/bin/LICENSE > > > > But note that, the source release artifact, which should not include any > > dependency jars, should have th

Re: To release a first Apache version Livy

what is in the > root of git today. > > > > > > On Mon, Aug 7, 2017 at 2:16 AM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > > > Hi Team, > > > > Today I cut a RC1 release based on branch-0.4, here is the link ( > > https://github.c

Re: To release a first Apache version Livy

an Apache release to transition to. > > > Given that its still a new project, a shorter cadence would help make bug > fix releases available. > > > Btw, does anyone know whats holding up the Apache jira process? If not, I > can follow up on that. > > > Bikas >

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114291#comment-16114291 ] Saisai Shao commented on SPARK-21618: - Hi Steve, I'm not quite following your comments. You mean

Re: Question about folder in https://dist.apache.org/repos/dist/dev/incubator/

this up on your dev list? Mentors are > responsible for creating the dist area. > > I went ahead and created one for you, in both /dist/dev and /dist/release. > > John > > On Fri, Aug 4, 2017 at 5:05 AM Saisai Shao <sai.sai.s...@gmail.com> wrote: > > > Hi Team

Question about folder in https://dist.apache.org/repos/dist/dev/incubator/

Hi Team, We're working on the first Apache release of incubator-livy project, as the document mentioned that source release should be staged under this folder, but I don't find a folder named "livy", how do I create this folder, please suggest, thanks! The Incubator PMC expects the source

Re: To release a first Apache version Livy

eleases come out when major > new functionality has added. My preference is a time based release cadence > because it provides regular bug and security related fixes available in a > released form for end users. > > > Thanks! > > Bikas > > > >

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113806#comment-16113806 ] Saisai Shao commented on SPARK-21618: - [~benmayne] If you try the master branch of Spark with SPARK

Re: To release a first Apache version Livy

Hi Team, We're planning to release a first Apache version of Livy. Would you please guide us the process of Apache incubating release, is there any doc to follow. Thanks a lot! Best regards Saisai On Tue, Aug 1, 2017 at 11:34 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > OK, than

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112036#comment-16112036 ] Saisai Shao commented on SPARK-21570: - Sorry I'm not familiar with NFS/Lustre FS, does this kind

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110617#comment-16110617 ] Saisai Shao commented on SPARK-21570: - This __spark_libs_xxx.zip is created by Spark on yarn to zip

Re: To release a first Apache version Livy

.@us.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details for Saisai Shao ---07/31/2017 07:12:11 > PM---Hi Ale

Re: To release a first Apache version Livy

;ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > Jeff Zhang ---07/28/2017 01:12:24 AM---+1 for making the firs

Re: Need input: Location of Livy Documentation

I'm in favor of store the docs in main repo, while put seldom changed frameworks in website repo. I think you already mentioned the reason here (as the Documentation grows and changes from version to version it would be better to store it in the more regularly updated repo). What I saw is that

[jira] [Commented] (SPARK-21521) History service requires user is in any group

[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099629#comment-16099629 ] Saisai Shao commented on SPARK-21521: - I I think we should have a special logics to treat special

[jira] [Commented] (SPARK-21521) History service requires user is in any group

[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099392#comment-16099392 ] Saisai Shao commented on SPARK-21521: - [~vanzin], I guess so, in the current logics

Re: Input file as an argument og a Spark code

I think you have to make this csv file accessible from Spark cluster, putting to HDFS is one possible solution. On Tue, Jul 25, 2017 at 1:26 AM, Joaquín Silva wrote: > Hello, > > > > I'm building a BASH program (using Curl) that should run a Spark code > remotely using

Re: When Livy JIRA system will be ready?

gt; Spark Technology Center > -- > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095447#comment-16095447 ] Saisai Shao commented on SPARK-21460: - I think this is basically a ListenerBus issue, not a dynamic

Re: Commit style best practices, was Re: incubator-livy-website git commit: Fix bug in merge_livy_pr.py because incubator-livy-website repo has no branch it's name started with "branch-"

Sorry about it. Basically because I don't know whether JIRA is necessary for incubator-livy-website repo, also where to create JIRA. On Thu, Jul 20, 2017 at 1:20 PM, Alex Bozarth wrote: > +1. I think we've been using close to this format, but with "LIVY-XXX." > instead of

[jira] [Created] (SPARK-21475) Change the usage of FileInputStream/OutputStream to Files.newInput/OutputStream in the critical path

Saisai Shao created SPARK-21475: --- Summary: Change the usage of FileInputStream/OutputStream to Files.newInput/OutputStream in the critical path Key: SPARK-21475 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Commented] (SPARK-21407) Support spnego for ThriftServer thrift/http auth

[ https://issues.apache.org/jira/browse/SPARK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088151#comment-16088151 ] Saisai Shao commented on SPARK-21407: - Sorry Marcelo, haven't noticed your comment. Yes

[jira] [Commented] (SPARK-21411) Failed to get new HDFS delegation tokens in AMCredentialRenewer

[ https://issues.apache.org/jira/browse/SPARK-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086777#comment-16086777 ] Saisai Shao commented on SPARK-21411: - This issue is introduced by SPARK-20434. > Failed to get

[jira] [Created] (SPARK-21411) Failed to get new HDFS delegation tokens in AMCredentialRenewer

Saisai Shao created SPARK-21411: --- Summary: Failed to get new HDFS delegation tokens in AMCredentialRenewer Key: SPARK-21411 URL: https://issues.apache.org/jira/browse/SPARK-21411 Project: Spark

[jira] [Commented] (SPARK-21398) Data on Rest end point is not updating after first fetch

[ https://issues.apache.org/jira/browse/SPARK-21398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086637#comment-16086637 ] Saisai Shao commented on SPARK-21398: - Can you please elaborate more about the issue, which rest api

[jira] [Updated] (SPARK-21407) Support spnego for ThriftServer thrift/http auth

[ https://issues.apache.org/jira/browse/SPARK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21407: Description: Spark ThriftServer doesn't support spnego auth for thrift/http protocol, this mainly

[jira] [Updated] (SPARK-21407) Support spnego for ThriftServer thrift/http auth

[ https://issues.apache.org/jira/browse/SPARK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21407: Summary: Support spnego for ThriftServer thrift/http auth (was: Support spnego for ThriftServer

[jira] [Created] (SPARK-21407) Support spnego for ThriftServer http auth

Saisai Shao created SPARK-21407: --- Summary: Support spnego for ThriftServer http auth Key: SPARK-21407 URL: https://issues.apache.org/jira/browse/SPARK-21407 Project: Spark Issue Type

[jira] [Updated] (SPARK-21377) Jars specified with --jars or --packages are not added into AM's system classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Summary: Jars specified with --jars or --packages are not added into AM's system classpath

[jira] [Comment Edited] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084460#comment-16084460 ] Saisai Shao edited comment on SPARK-21376 at 7/12/17 6:31 PM: -- I'm

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084460#comment-16084460 ] Saisai Shao commented on SPARK-21376: - I'm referrring to o.a.s.deploy.yarn.Client this class

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084270#comment-16084270 ] Saisai Shao commented on SPARK-21376: - Hi [~tgraves], it is the local yarn launcher process which

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Affects Version/s: (was: 2.1.0) 2.2.0 > Jars pulled from "--

[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083081#comment-16083081 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 10:13 PM: --- My original

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083081#comment-16083081 ] Saisai Shao commented on SPARK-21377: - My original purpose is to add jars uploaded by distributed

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083060#comment-16083060 ] Saisai Shao commented on SPARK-21377: - Thanks [~vanzin] for your comment. Your comment is correct

[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083060#comment-16083060 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 9:54 PM: -- Thanks

[jira] [Updated] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Description: In this issue we have a long running Spark application with secure HBase, which

[jira] [Updated] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Summary: Add a new configuration to extend AM classpath in yarn client mode (was: Jars pulled

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Description: STR: * Set below config in spark-default.conf {code

[jira] [Commented] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082864#comment-16082864 ] Saisai Shao commented on SPARK-21377: - [~srowen] this is a separate issue to SPARK-21376

[jira] [Commented] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082873#comment-16082873 ] Saisai Shao commented on SPARK-21377: - SPARK-21376 and here are both security issues, but the root

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Priority: Minor (was: Major) > Jars pulled from "--packages" are not added into

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Priority: Minor (was: Major) > Token is not renewed in yarn client process in cluster m

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Component/s: (was: Spark Core) YARN > Jars pulled from "--

[jira] [Comment Edited] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082864#comment-16082864 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 7:58 PM: -- [~srowen

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Affects Version/s: (was: 2.1.0) 2.2.0 2.1.1

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082857#comment-16082857 ] Saisai Shao commented on SPARK-21376: - I will work on this, thanks [~yeshavora]. > To

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Component/s: (was: Spark Core) YARN > Token is not renewed in yarn cli

[jira] [Comment Edited] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082508#comment-16082508 ] Saisai Shao edited comment on SPARK-14151 at 7/11/17 5:00 PM: -- Thanks

[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082508#comment-16082508 ] Saisai Shao commented on SPARK-14151: - Thanks [~jiangxb] for your response. I'm not sure

[jira] [Comment Edited] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079407#comment-16079407 ] Saisai Shao edited comment on SPARK-21346 at 7/9/17 1:29 AM: - Jars, files can

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079407#comment-16079407 ] Saisai Shao commented on SPARK-21346: - Jars, files can be downloaded from HTTPS if you configured

[jira] [Comment Edited] (SPARK-21334) Fix metrics for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079389#comment-16079389 ] Saisai Shao edited comment on SPARK-21334 at 7/9/17 12:43 AM: -- Are you using

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079389#comment-16079389 ] Saisai Shao commented on SPARK-21334: - Are you using external shuffle service with YARN? If so I

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079385#comment-16079385 ] Saisai Shao commented on SPARK-21346: - SSL is still used for fetch remote resources from HTTPS

[jira] [Commented] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079381#comment-16079381 ] Saisai Shao commented on SPARK-21348: - SPARK-18085 is trying to address this issue. [~duyanghao

Re: Kafka 0.10 with PySpark

Please see the reason in this thread ( https://github.com/apache/spark/pull/14340). It would better to use structured streaming instead. So I would like to -1 this patch. I think it's been a mistake to support > dstream in Python -- yes it satisfies a checkbox and Spark could claim > there's

Re: Scala 2.11 and Hive

Hi Mobin, Livy supports Scala 2.11 as well as Spark 2.0+. Can you please elaborate your problem? Thanks Jerry On Sun, Jul 2, 2017 at 9:19 PM, Mobin Ranjbar wrote: > Hi there, > > > I have a problem around using Livy and Apache Spark(Scala 2.10). I have > Apache Hive

[jira] [Created] (ZEPPELIN-2716) Change default of zeppelin.livy.displayAppInfo to true

Saisai Shao created ZEPPELIN-2716: - Summary: Change default of zeppelin.livy.displayAppInfo to true Key: ZEPPELIN-2716 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2716 Project: Zeppelin

Re: Question about standalone Spark cluster reading from Kerberosed hadoop

Spark running with standalone cluster manager currently doesn't support accessing security Hadoop. Basically the problem is that standalone mode Spark doesn't have the facility to distribute delegation tokens. Currently only Spark on YARN or local mode supports security Hadoop. Thanks Jerry On

[jira] [Comment Edited] (SPARK-21080) Workaround for HDFS delegation token expiry broken with some Hadoop versions

[ https://issues.apache.org/jira/browse/SPARK-21080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058843#comment-16058843 ] Saisai Shao edited comment on SPARK-21080 at 6/22/17 6:39 AM: -- That PR

[jira] [Commented] (SPARK-21080) Workaround for HDFS delegation token expiry broken with some Hadoop versions

[ https://issues.apache.org/jira/browse/SPARK-21080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058843#comment-16058843 ] Saisai Shao commented on SPARK-21080: - That PR should be worked, we applied that one to our internal

[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050016#comment-16050016 ] Saisai Shao commented on SPARK-21082: - That's fine if the storage memory is not enough to cache all

<    4   5   6   7   8   9   10   11   12   13   >