Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

2017-08-22 Thread Saisai Shao 
Nan, I think Meisam already had a PR about this this, maybe you can discuss
with him on the github based on the proposed code.

Sorry I didn't follow the long discussion thread, but I think Paypal's
solution sounds simpler.

On Wed, Aug 23, 2017 at 12:07 AM, Nan Zhu  wrote:

> based on this result, I think we should follow the bulk operation pattern
>
> Shall we move forward with the PR from Paypal?
>
> Best,
>
> Nan
>
> On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi 
> wrote:
>
> > Bottom line up front:
> > 1. The cost of calling 1 individual REST calls is about two order of
> > magnitude higher than calling a single batch REST call (1 * 0.05
> > seconds vs. 1.4 seconds)
> > 2. Time to complete a batch REST call plateaus at about 10,000
> application
> > reports per call.
> >
> > Full story:
> > I experimented and measure how long it takes to fetch Application Reports
> > from YARN with the REST API. My objective was to compare doing a batch
> REST
> > call to get all ApplicationReports vs doing individual REST calls for
> each
> > Application Report.
> >
> > I did the tests on 4 different cluster: 1) a test cluster, 2) a
> moderately
> > used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> > used production cluster. For each cluster I made 7 REST call to get 1,
> 10,
> > 100, 1000, 1, 10, 100 application reports respectively. I
> > repeated each call 200 times to count for variations and I reported the
> > median time.
> > To measure the time, I used the following curl command:
> >
> > $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> > $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> > applicationTypes=$limit"
> >
> > The attached charts show the results. In all the charts, the x axis show
> > the number of results that were request in the call.
> > The bar chart show the time it takes to complete a REST call on each
> > cluster.
> > The first line plot also shows the same results as the bar chart on a log
> > scale (it is easier to see that the time to complete the REST call
> plateaus
> > at 10,000
> > The last chart shows the size of data that is being downloaded on each
> > REST call, which explains why the time plateaus  at 10,000.
> >
> >
> > [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][
> image:
> > size_downloaded_line_plot.png]
> >
> >>
> >>
> > Thanks,
> > Meisam
> >
>

Re: Help to verify Apache Livy 0.4.0-incubating release

2017-08-22 Thread Alex Bozarth 

Nice catch, it must not be a big issue though because it's been that way
since the original Livy 0.2.0 release.

@jerry do you think we should update this with a minor and open a rc3 or
just fix it in the next release?

   
 Alex Bozarth   
   
 Software Engineer  
   
 Spark Technology Center
   

   

 

 

 
 E-mail: ajboz...@us.ibm.com
 
 GitHub: github.com/ajbozarth   
 
   505 Howard 
Street 
 San Francisco, CA 
94105 
   United 
States 

 








From:   Meisam Fathi 
To: dev@livy.incubator.apache.org
Date:   08/22/2017 10:00 AM
Subject:Re: Help to verify Apache Livy 0.4.0-incubating release



The version of org.apache.httpcomponents:httpclient is different in
/pom.xml from the version in /client-http/pom.xml

/pom.xml --->  ${httpclient.version} ---> 4.5.2
/client-http/pom.xml --->  4.5.1

Is this intended?

Thanks,
Meisam

On Thu, Aug 17, 2017 at 12:34 AM, Saisai Shao 

> Hi all,
>
>
> We're under progress to make a first Apache release of Livy
>
> (0.4.0-incubating), we really hope you could verify the RC2[1] release
>
> (binary and source) locally and return us the feedbacks.
>
[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.apache.org_repos_dist_dev_incubator_livy_0=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=Kb-p_IVOP_NdMjLDuaQ3n9M8JuXyOyZoJMW1js8FH1c=LgN1J8mfdzW3GoDhuYQeFGOVag4Ncz_WBKBTcVau970=
 .
>

Re: Help to verify Apache Livy 0.4.0-incubating release

2017-08-22 Thread Meisam Fathi 
The version of org.apache.httpcomponents:httpclient is different in
/pom.xml from the version in /client-http/pom.xml

/pom.xml --->  ${httpclient.version} ---> 4.5.2
/client-http/pom.xml --->  4.5.1

Is this intended?

Thanks,
Meisam

On Thu, Aug 17, 2017 at 12:34 AM, Saisai Shao 

> Hi all,
>
>
> We're under progress to make a first Apache release of Livy
>
> (0.4.0-incubating), we really hope you could verify the RC2[1] release
>
> (binary and source) locally and return us the feedbacks.
>
[1]https://dist.apache.org/repos/dist/dev/incubator/livy/0.
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

2017-08-22 Thread Nan Zhu 
based on this result, I think we should follow the bulk operation pattern

Shall we move forward with the PR from Paypal?

Best,

Nan

On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi 
wrote:

> Bottom line up front:
> 1. The cost of calling 1 individual REST calls is about two order of
> magnitude higher than calling a single batch REST call (1 * 0.05
> seconds vs. 1.4 seconds)
> 2. Time to complete a batch REST call plateaus at about 10,000 application
> reports per call.
>
> Full story:
> I experimented and measure how long it takes to fetch Application Reports
> from YARN with the REST API. My objective was to compare doing a batch REST
> call to get all ApplicationReports vs doing individual REST calls for each
> Application Report.
>
> I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
> used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> used production cluster. For each cluster I made 7 REST call to get 1, 10,
> 100, 1000, 1, 10, 100 application reports respectively. I
> repeated each call 200 times to count for variations and I reported the
> median time.
> To measure the time, I used the following curl command:
>
> $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> applicationTypes=$limit"
>
> The attached charts show the results. In all the charts, the x axis show
> the number of results that were request in the call.
> The bar chart show the time it takes to complete a REST call on each
> cluster.
> The first line plot also shows the same results as the bar chart on a log
> scale (it is easier to see that the time to complete the REST call plateaus
> at 10,000
> The last chart shows the size of data that is being downloaded on each
> REST call, which explains why the time plateaus  at 10,000.
>
>
> [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
> size_downloaded_line_plot.png]
>
>>
>>
> Thanks,
> Meisam
>

Re: [VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

2017-08-22 Thread Saisai Shao 
OK, sure, I will remove RC1 from the directory.

Thanks
Jerry

On Tue, Aug 22, 2017 at 7:24 PM, John D. Ament 
wrote:

> Hi,
>
> Looking at your release, it's confusing what we are voting on.  If RC2 is
> under vote, please remove RC1 from this directory.
>
> John
>
> On Tue, Aug 22, 2017 at 3:33 AM Jerry Shao  wrote:
>
> > Hello Incubator PMC’ers,
> >
> > The Apache Livy community has decided to release Apache Livy
> > 0.4.0-incubating based on 0.4.0-incubating Release Candidate 2. We now
> > kindly request the Incubator PMC members to review and vote on this
> > incubator
> > release.
> >
> > Livy is web service that exposes a REST interface for managing long
> running
> > Apache Spark contexts in your cluster. With Livy, new applications can be
> > built on top of Apache Spark that require fine grained interaction with
> > many Spark contexts.
> >
> > Artifacts are available at
> > https://dist.apache.org/repos/dist/dev/incubator/livy/, public keys are
> > available at https://dist.apache.org/repos/dist/dev/incubator/livy/KEYS.
> >
> > livy-0.4.0-incubating-src.zip <
> >
> > https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-
> incubating/livy-0.4.0-incubating-src-RC2.zip
> > > is a source release. Along with it, for convenience, please find the
> > binary release as livy-0.4.0-incubating-bin-RC2.zip <
> >
> > https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-
> incubating/livy-0.4.0-incubating-bin-RC2.zip
> > >.
> >
> >
> > Git tag:
> > *
> > https://github.com/apache/incubator-livy/releases/tag/
> v0.4.0-incubating-rc2
> > <
> > https://github.com/apache/incubator-livy/releases/tag/
> v0.4.0-incubating-rc2
> > >*
> >
> > The vote will be open for at least 72 hours or until necessary number of
> > votes are reached.
> >
> > Members please be sure to indicate "(Binding)" with your vote which will
> > help in tallying the vote(s).
> >
> > * Here is my +1 (non-binding) *
> >
> > Cheers,
> > Jerry
> >
>

Re: [VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

2017-08-22 Thread John D. Ament 
Hi,

Looking at your release, it's confusing what we are voting on.  If RC2 is
under vote, please remove RC1 from this directory.

John

On Tue, Aug 22, 2017 at 3:33 AM Jerry Shao  wrote:

> Hello Incubator PMC’ers,
>
> The Apache Livy community has decided to release Apache Livy
> 0.4.0-incubating based on 0.4.0-incubating Release Candidate 2. We now
> kindly request the Incubator PMC members to review and vote on this
> incubator
> release.
>
> Livy is web service that exposes a REST interface for managing long running
> Apache Spark contexts in your cluster. With Livy, new applications can be
> built on top of Apache Spark that require fine grained interaction with
> many Spark contexts.
>
> Artifacts are available at
> https://dist.apache.org/repos/dist/dev/incubator/livy/, public keys are
> available at https://dist.apache.org/repos/dist/dev/incubator/livy/KEYS.
>
> livy-0.4.0-incubating-src.zip <
>
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/livy-0.4.0-incubating-src-RC2.zip
> > is a source release. Along with it, for convenience, please find the
> binary release as livy-0.4.0-incubating-bin-RC2.zip <
>
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/livy-0.4.0-incubating-bin-RC2.zip
> >.
>
>
> Git tag:
> *
> https://github.com/apache/incubator-livy/releases/tag/v0.4.0-incubating-rc2
> <
> https://github.com/apache/incubator-livy/releases/tag/v0.4.0-incubating-rc2
> >*
>
> The vote will be open for at least 72 hours or until necessary number of
> votes are reached.
>
> Members please be sure to indicate "(Binding)" with your vote which will
> help in tallying the vote(s).
>
> * Here is my +1 (non-binding) *
>
> Cheers,
> Jerry
>

[VOTE] Release Livy 0.4.0-incubating based on Livy 0.4.0 RC2

2017-08-22 Thread Jerry Shao 
Hello Incubator PMC’ers,

The Apache Livy community has decided to release Apache Livy
0.4.0-incubating based on 0.4.0-incubating Release Candidate 2. We now
kindly request the Incubator PMC members to review and vote on this incubator
release.

Livy is web service that exposes a REST interface for managing long running
Apache Spark contexts in your cluster. With Livy, new applications can be
built on top of Apache Spark that require fine grained interaction with
many Spark contexts.

Artifacts are available at
https://dist.apache.org/repos/dist/dev/incubator/livy/, public keys are
available at https://dist.apache.org/repos/dist/dev/incubator/livy/KEYS.

livy-0.4.0-incubating-src.zip <
https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/livy-0.4.0-incubating-src-RC2.zip
> is a source release. Along with it, for convenience, please find the
binary release as livy-0.4.0-incubating-bin-RC2.zip <
https://dist.apache.org/repos/dist/dev/incubator/livy/0.4.0-incubating/livy-0.4.0-incubating-bin-RC2.zip
>.


Git tag:
*https://github.com/apache/incubator-livy/releases/tag/v0.4.0-incubating-rc2
*

The vote will be open for at least 72 hours or until necessary number of
votes are reached.

Members please be sure to indicate "(Binding)" with your vote which will
help in tallying the vote(s).

* Here is my +1 (non-binding) *

Cheers,
Jerry