Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Saisai Shao]

Nan, I think Meisam already had a PR about this this, maybe you can discuss
with him on the github based on the proposed code.

Sorry I didn't follow the long discussion thread, but I think Paypal's
solution sounds simpler.

On Wed, Aug 23, 2017 at 12:07 AM, Nan Zhu  wrote:

> based on this result, I think we should follow the bulk operation pattern
>
> Shall we move forward with the PR from Paypal?
>
> Best,
>
> Nan
>
> On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi 
> wrote:
>
> > Bottom line up front:
> > 1. The cost of calling 1 individual REST calls is about two order of
> > magnitude higher than calling a single batch REST call (1 * 0.05
> > seconds vs. 1.4 seconds)
> > 2. Time to complete a batch REST call plateaus at about 10,000
> application
> > reports per call.
> >
> > Full story:
> > I experimented and measure how long it takes to fetch Application Reports
> > from YARN with the REST API. My objective was to compare doing a batch
> REST
> > call to get all ApplicationReports vs doing individual REST calls for
> each
> > Application Report.
> >
> > I did the tests on 4 different cluster: 1) a test cluster, 2) a
> moderately
> > used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> > used production cluster. For each cluster I made 7 REST call to get 1,
> 10,
> > 100, 1000, 1, 10, 100 application reports respectively. I
> > repeated each call 200 times to count for variations and I reported the
> > median time.
> > To measure the time, I used the following curl command:
> >
> > $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> > $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> > applicationTypes=$limit"
> >
> > The attached charts show the results. In all the charts, the x axis show
> > the number of results that were request in the call.
> > The bar chart show the time it takes to complete a REST call on each
> > cluster.
> > The first line plot also shows the same results as the bar chart on a log
> > scale (it is easier to see that the time to complete the REST call
> plateaus
> > at 10,000
> > The last chart shows the size of data that is being downloaded on each
> > REST call, which explains why the time plateaus  at 10,000.
> >
> >
> > [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][
> image:
> > size_downloaded_line_plot.png]
> >
> >>
> >>
> > Thanks,
> > Meisam
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu]

based on this result, I think we should follow the bulk operation pattern

Shall we move forward with the PR from Paypal?

Best,

Nan

On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi 
wrote:

> Bottom line up front:
> 1. The cost of calling 1 individual REST calls is about two order of
> magnitude higher than calling a single batch REST call (1 * 0.05
> seconds vs. 1.4 seconds)
> 2. Time to complete a batch REST call plateaus at about 10,000 application
> reports per call.
>
> Full story:
> I experimented and measure how long it takes to fetch Application Reports
> from YARN with the REST API. My objective was to compare doing a batch REST
> call to get all ApplicationReports vs doing individual REST calls for each
> Application Report.
>
> I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
> used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> used production cluster. For each cluster I made 7 REST call to get 1, 10,
> 100, 1000, 1, 10, 100 application reports respectively. I
> repeated each call 200 times to count for variations and I reported the
> median time.
> To measure the time, I used the following curl command:
>
> $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> applicationTypes=$limit"
>
> The attached charts show the results. In all the charts, the x axis show
> the number of results that were request in the call.
> The bar chart show the time it takes to complete a REST call on each
> cluster.
> The first line plot also shows the same results as the bar chart on a log
> scale (it is easier to see that the time to complete the REST call plateaus
> at 10,000
> The last chart shows the size of data that is being downloaded on each
> REST call, which explains why the time plateaus  at 10,000.
>
>
> [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
> size_downloaded_line_plot.png]
>
>>
>>
> Thanks,
> Meisam
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi]

Hi Marcelo,


> I'm not really familiar with how multi-node HA was implemented (I
> stopped at session recovery), but why isn't a single server doing the
> update and storing the results in ZK? Unless it's actually doing
> load-balancing, it seems like that would avoid multiple servers having
> to hit YARN.
>

We considered having one server update ZooKeeper, but the extra benefits
that we would get from polling yarn fewer times is not worth the extra
complexity needed to implement it. For example, we would have to make
servers aware of each other, and aware of each others failures. We would've
needed a voting mechanism to select a new leader to update ZooKeeper each
time the current leader had a failure. Also rolling out updates would be
tricker with servers that are aware of each other.

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi]

> Just an FYI, apache mailing lists cant share attachments. If you could
> please upload the files to another file sharing site and include links
> instead.
>
Thanks for the information. I added the files to the JIRA ticket and put
the contents of the previous email as a comment. Here are the links to the
ticket and to the files:

JIRA ticket: https://issues.apache.org/jira/browse/LIVY-336
Time to complete REST calls to YARN:
https://issues.apache.org/jira/secure/attachment/12882985/transfer_time_bar_plot.png
Trends in time to complete REST calls to YARN:
https://issues.apache.org/jira/secure/attachment/12882984/transfer_time_line_plot.png
Size of response from REST calls to YARN:
https://issues.apache.org/jira/secure/attachment/12882983/size_downloaded_line_plot.png

Also, should we move the discussion to JIRA now that it is up and running?

Thanks,
Meisam

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi]

I forgot to attach the first chart. Sorry about that.

[image: transfer_time_bar_plot.png]

Thanks,
Meisam

On Mon, Aug 21, 2017 at 12:21 PM Meisam Fathi 
wrote:

> Bottom line up front:
> 1. The cost of calling 1 individual REST calls is about two order of
> magnitude higher than calling a single batch REST call (1 * 0.05
> seconds vs. 1.4 seconds)
> 2. Time to complete a batch REST call plateaus at about 10,000 application
> reports per call.
>
> Full story:
> I experimented and measure how long it takes to fetch Application Reports
> from YARN with the REST API. My objective was to compare doing a batch REST
> call to get all ApplicationReports vs doing individual REST calls for each
> Application Report.
>
> I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
> used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> used production cluster. For each cluster I made 7 REST call to get 1, 10,
> 100, 1000, 1, 10, 100 application reports respectively. I
> repeated each call 200 times to count for variations and I reported the
> median time.
> To measure the time, I used the following curl command:
>
> $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$applicationTypes=$limit"
>
> The attached charts show the results. In all the charts, the x axis show
> the number of results that were request in the call.
> The bar chart show the time it takes to complete a REST call on each
> cluster.
> The first line plot also shows the same results as the bar chart on a log
> scale (it is easier to see that the time to complete the REST call plateaus
> at 10,000
> The last chart shows the size of data that is being downloaded on each
> REST call, which explains why the time plateaus  at 10,000.
>
>
> [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
> size_downloaded_line_plot.png]
>
>>
>>
> Thanks,
> Meisam
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi]

Bottom line up front:
1. The cost of calling 1 individual REST calls is about two order of
magnitude higher than calling a single batch REST call (1 * 0.05
seconds vs. 1.4 seconds)
2. Time to complete a batch REST call plateaus at about 10,000 application
reports per call.

Full story:
I experimented and measure how long it takes to fetch Application Reports
from YARN with the REST API. My objective was to compare doing a batch REST
call to get all ApplicationReports vs doing individual REST calls for each
Application Report.

I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
used dev cluster, 3) a lightly used production cluster, and 4) a heavily
used production cluster. For each cluster I made 7 REST call to get 1, 10,
100, 1000, 1, 10, 100 application reports respectively. I
repeated each call 200 times to count for variations and I reported the
median time.
To measure the time, I used the following curl command:

$ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
$rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$applicationTypes=$limit"

The attached charts show the results. In all the charts, the x axis show
the number of results that were request in the call.
The bar chart show the time it takes to complete a REST call on each
cluster.
The first line plot also shows the same results as the bar chart on a log
scale (it is easier to see that the time to complete the REST call plateaus
at 10,000
The last chart shows the size of data that is being downloaded on each REST
call, which explains why the time plateaus  at 10,000.


[image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
size_downloaded_line_plot.png]

>
>
Thanks,
Meisam

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

I like that approach on paper, although I currently don't have much
time to actually be able to review the PR and provide decent feedback.

I think that regardless of the approach, one goal should be to
probably separate what is being monitored from how it's being
monitored; that way you can later change the monitoring code to be
smarter without having to change the rest of the code that calls into
it. I remember reviewing this code when it first was submitted and it
could definitely use some refactoring.

> we have Livy Multi-Node HA i.e livy running on 6 servers for each cluster,

I'm not really familiar with how multi-node HA was implemented (I
stopped at session recovery), but why isn't a single server doing the
update and storing the results in ZK? Unless it's actually doing
load-balancing, it seems like that would avoid multiple servers having
to hit YARN.



On Wed, Aug 16, 2017 at 4:18 PM, Prabhu Kasinathan
 wrote:
> As Meisam highlighted, in our case, we have Livy Multi-Node HA i.e livy
> running on 6 servers for each cluster, load-balanced, sharing livy metadata
> on zookeeper and running thousands of applications. With below changes, we
> are seeing good improvements due to batching the requests (one per livy
> node) instead of each livy node making multiple requests. Please review the
> changes and let us know if improvements needed or we are open to explore
> other alternative option if works.
>
>> We are making one big request to get ApplicationReports, Then we make an
>> individual + thread pool request to get the tracking URL, Spark UI URL,
>> YARN diagnostics, etc for each application separately. For our cluster
>> settings and our workloads, one big request turned out to be a better
>> solution. But we were limited to the API provided in YarnClient. With the
>> home-made REST client a separate request is not needed and that can change
>> the whole equation.
>
>
>
> On Wed, Aug 16, 2017 at 3:33 PM, Meisam Fathi 
> wrote:
>
>>
>> On Wed, Aug 16, 2017 at 2:09 PM Nan Zhu  wrote:
>>
>>> With time goes, the reply from YARN can only be larger and larger. Given
>>> the consistent workload pattern, the cost of a large query can be
>>> eventually larger than individual request
>>>
>>
>> I am under the impression that there is a limit to the number of reports
>> that YARN retains, which is set by 
>> yarn.resourcemanager.max-completed-applications
>> in yarn.xml and defaults to 10,000. But I could be wrong about the
>> semantics of yarn.resourcemanager.max-completed-applications.
>>
>> I would say go with individual request + thread pool  or large batch for
>>> all first, if any performance issue is observed, add the optimization on
>>> top of it
>>>
>>
>> We are making one big request to get ApplicationReports, Then we make an
>> individual + thread pool request to get the tracking URL, Spark UI URL,
>> YARN diagnostics, etc for each application separately. For our cluster
>> settings and our workloads, one big request turned out to be a better
>> solution. But we were limited to the API provided in YarnClient. With the
>> home-made REST client a separate request is not needed and that can change
>> the whole equation.
>>
>> @Prabhu, can you chime in?
>>
>>
>>> However, even with rest API, there are some corner cases, e.g. a
>>> long running app lasting for days (training some models), and some short
>>> ones which last only for minutes
>>>
>>
>> We are running Spark streaming jobs on Livy that virtually run for ever.
>>
>> Thanks,
>> Meisam
>>



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Prabhu Kasinathan]

As Meisam highlighted, in our case, we have Livy Multi-Node HA i.e livy
running on 6 servers for each cluster, load-balanced, sharing livy metadata
on zookeeper and running thousands of applications. With below changes, we
are seeing good improvements due to batching the requests (one per livy
node) instead of each livy node making multiple requests. Please review the
changes and let us know if improvements needed or we are open to explore
other alternative option if works.

> We are making one big request to get ApplicationReports, Then we make an
> individual + thread pool request to get the tracking URL, Spark UI URL,
> YARN diagnostics, etc for each application separately. For our cluster
> settings and our workloads, one big request turned out to be a better
> solution. But we were limited to the API provided in YarnClient. With the
> home-made REST client a separate request is not needed and that can change
> the whole equation.



On Wed, Aug 16, 2017 at 3:33 PM, Meisam Fathi 
wrote:

>
> On Wed, Aug 16, 2017 at 2:09 PM Nan Zhu  wrote:
>
>> With time goes, the reply from YARN can only be larger and larger. Given
>> the consistent workload pattern, the cost of a large query can be
>> eventually larger than individual request
>>
>
> I am under the impression that there is a limit to the number of reports
> that YARN retains, which is set by 
> yarn.resourcemanager.max-completed-applications
> in yarn.xml and defaults to 10,000. But I could be wrong about the
> semantics of yarn.resourcemanager.max-completed-applications.
>
> I would say go with individual request + thread pool  or large batch for
>> all first, if any performance issue is observed, add the optimization on
>> top of it
>>
>
> We are making one big request to get ApplicationReports, Then we make an
> individual + thread pool request to get the tracking URL, Spark UI URL,
> YARN diagnostics, etc for each application separately. For our cluster
> settings and our workloads, one big request turned out to be a better
> solution. But we were limited to the API provided in YarnClient. With the
> home-made REST client a separate request is not needed and that can change
> the whole equation.
>
> @Prabhu, can you chime in?
>
>
>> However, even with rest API, there are some corner cases, e.g. a
>> long running app lasting for days (training some models), and some short
>> ones which last only for minutes
>>
>
> We are running Spark streaming jobs on Livy that virtually run for ever.
>
> Thanks,
> Meisam
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi]

Hi Nan,

In the highlighted line
>
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75
>
> I assume that it will get the reports of all applications in YARN, even
> they are finished?


That's right. That line will return reports for all Spark Applications,
even applications that completed a long time ago. For us YARN retains
reports for a few thousand completed applications (not a big concern).

Livy needs to get the reports for applications that finished recently, but
I didn't find an API in YARN 2.7 to get those only reports.

Thanks,
Meisam

>
>

> On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi 
> wrote:
>
> > Hi Nan,
> >
> >
> > >
> > > my question related to the undergoing discussion is simply "have you
> seen
> > > any performance issue in
> > >
> > > https://github.com/apache/incubator-livy/pull/36/files#diff-
> > a3f879755cfe10a678cc08ddbe60a4d3R75
> > > ?
> > >  > a3f879755cfe10a678cc08ddbe60a4d3R75?>
> > > "
> > >
> > > The short answer is yes. This PR fixes one part of the scalability
> > problem, which is, it prevents Livy from creating many
> > yarnAppMinotorThreads. But the two other parts are still there
> >
> > 1. one call to spark-submit for each application
> > 2. once thread that waits for the exit code of spark-submit.
> >
> > Out of these two problems, calling one spark-submit per application is
> the
> > biggest problem, but it can be solved by adding more Livy servers. We
> > modified Livy so if an application status changes on one Livy instance,
> all
> > other Livy instances get the updated information about the application.
> > From users' perspective, this is transparent because users just see the
> > load balancer.
> >
> > So, refactoring the yarn poll mechanism + a load balancer and a grid of
> > Livy servers fixed the scalability issue.
> >
> > On the performance of the code itself, we have not had an issue. The time
> > consuming parts in the code are calls to YARN and not filtering and
> > updating the data structures. On memory usage, this all needs less than
> 1GB
> > at peak time.
> >
> > I hope this answers your question.
> >
> > Thanks,
> > Meisam
> >
> >
> > > We have several scenarios that a large volume of applications are
> > submitted
> > > to YARN every day and it easily accumulates a lot to be fetched with
> this
> > > call
> > >
> > > Best,
> > >
> > > Nan
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi  >
> > > wrote:
> > >
> > > > Here are my two pennies on both designs (actor-based design vs.
> > > > single-thread polling design)
> > > >
> > > > *Single-thread polling design*
> > > > We implemented a single-thread polling mechanism for Yarn here at
> > PayPal.
> > > > Our solution is more involved because we added many new features to
> > Livy
> > > > that we had to consider when we refactored Livy's YARN interface. But
> > we
> > > > are willing to hammer our changes so it suits the need of the Livy
> > > > community best :-)
> > > >
> > > > *Actor-based design*
> > > > It seems to me that the proposed actor based design (
> > > > https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> > > > nbTQTdDFXl2XQhXDiwA/edit)
> > > > needs a few more messages and actors. Here is why.
> > > > Livy makes three (blocking) calls to YARN
> > > > 1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
> > > > 2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which
> > gives
> > > > Livy `getAMContainerId`
> > > > 3. `yarnClient.getContainerReport`, which gives Livy tracking URLs
> > > >
> > > > The result of the previous call is needed to make the next call. The
> > > > proposed actor system needs to be designed to handles all these
> > blocking
> > > > calls.
> > > >
> > > > I do agree that actor based design is cleaner and more maintainable.
> > But
> > > we
> > > > had to discard it because it adds more dependencies to Livy. We faced
> > too
> > > > many dependency-version-mismatch problems with Livy interactive
> > sessions
> > > > (when applications depend on a different version of a library that is
> > > used
> > > > internally by Livy). If the livy community prefers an actor based
> > design,
> > > > we are willing to reimplement our changes with an actor system.
> > > >
> > > > Finally, either design is only the first step in fixing this
> particular
> > > > scalability problem. The reason is that the *"yarnAppMinotorThread"
> is
> > > not
> > > > the only thread that Livy spawns per Spark application.* For batch
> > jobs,
> > > > Livy
> > > > 1. calls spark-submit, which lunches a new JVM (an operations that is
> > far
> > > > more heavy than creating a thread and can easily drain the system)
> > > > 2. It create a thread that waits for the exist code of spark-submit.
> > Even
> > > > though this thread is "short-lived", at peak time thousands of 

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 12:57 PM, Nan Zhu  wrote:
> yes, we finally converge on the idea
>
> how large the reply can be? if I have only one running applications and I
> still need to fetch 1000
>
> on the other side
>
> I have 1000 running apps, what's the cost of sending 1000 requests even the
> thread pool and yarn client are shared?

I don't know the answers, but I'm asking you, since you are proposing
the design, to consider that as an option, since it does not seem like
you considered that tradeoff when suggesting your current approach.

My comments about filtering are targeted at making things better in
your first case; if there's really only one app being monitored, and
you can figure out a filter that returns let's say 50 apps instead of
1000 that may be monitored by YARN, then you can do that.

Or maybe you can go with a hybrid approach, where you use individual
requests but past a certain threshold you fall back to bulk requests
to avoid overloading YARN.

Again, I'm asking you to consider alternatives that are not mentioned
in your design document, because I identified potential performance
issues in the current approach.


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu]

yes, we finally converge on the idea

how large the reply can be? if I have only one running applications and I
still need to fetch 1000

on the other side

I have 1000 running apps, what's the cost of sending 1000 requests even the
thread pool and yarn client are shared?



On Wed, Aug 16, 2017 at 12:36 PM, Marcelo Vanzin 
wrote:

> On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu  wrote:
> > I am using your words *current*. What's the definition of "current" in
> > livy? I think that's all application which still keep some records in the
> > livy's process's memory space
>
> There are two views of what is current: Livy's and YARN's. They may
> not be the same.
>
> From your reply below, you seem to want to query YARN for the state of
> applications that are current to Livy. There's no API for that, as you
> said. But that is not what I'm talking about.
>
> I'm saying that Livy should query YARN for YARN's current view of what
> applications exist, and then match those against its own view.
>
> Again, it's all a question about what is cheaper: a single request to
> YARN that results in a large reply, parts of which Livy will ignore
> because it's not interested in the data, or hundreds of small requests
> to YARN polling specific applications?
>
> > 1. How you express this "current" in a query to YARN? I think you have to
> > use ApplicationID (maybe there are some other ways) in a query
> >
> > 2. The problem is that I didn't see such an API to make such a "big call"
> > by passing in all applications's IDs
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu  wrote:
> I am using your words *current*. What's the definition of "current" in
> livy? I think that's all application which still keep some records in the
> livy's process's memory space

There are two views of what is current: Livy's and YARN's. They may
not be the same.

>From your reply below, you seem to want to query YARN for the state of
applications that are current to Livy. There's no API for that, as you
said. But that is not what I'm talking about.

I'm saying that Livy should query YARN for YARN's current view of what
applications exist, and then match those against its own view.

Again, it's all a question about what is cheaper: a single request to
YARN that results in a large reply, parts of which Livy will ignore
because it's not interested in the data, or hundreds of small requests
to YARN polling specific applications?

> 1. How you express this "current" in a query to YARN? I think you have to
> use ApplicationID (maybe there are some other ways) in a query
>
> 2. The problem is that I didn't see such an API to make such a "big call"
> by passing in all applications's IDs


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu  wrote:
> Yes, I know there is such an API, what I don't understand is what I should
> pass in the filtering API you mentioned, say we query YARN for every 5
> tickets
>
> 0: Query and get App A is running
>
> 4: App A is done
>
> 5: Query...so what I should fill as filtering parameters at 5 get capture
> the changes of App A's state?

You don't query for app state *changes*. You query for the current app
state, and compare against what you have, and then you can detect
changes that way. The trick is how to filter to get the information
you want, so you limit how much data you request from YARN.

I'm not aware of any YARN API to query for state changes like that. So
even in the individual request case, you'd have to get app A's state,
and update the Livy handle if the state has changed from what was
previously know.

That's most probably why Meisam's PR only filters by app type. If
there are further filters than can be applied, then great, but you
still need logic in Livy to detect the state changes you want.

> If you look at Meisam's PR, they can only filter based on appType
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu]

Yes, I know there is such an API, what I don't understand is what I should
pass in the filtering API you mentioned, say we query YARN for every 5
tickets

0: Query and get App A is running

4: App A is done

5: Query...so what I should fill as filtering parameters at 5 get capture
the changes of App A's state?

If you look at Meisam's PR, they can only filter based on appType
https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75

I guess it's due to the same reason

Best,

Nan


On Wed, Aug 16, 2017 at 11:28 AM, Marcelo Vanzin 
wrote:

> On Wed, Aug 16, 2017 at 11:17 AM, Nan Zhu  wrote:
> > Looks like non-REST API also contains this https://hadoop.apache.
> > org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/
> > api/YarnClient.html#line.225
> >
> > my concern which was skipped in your last email (again) is that, how many
> > app states we want to fetch through this API. What I can see is we cannot
> > filter applications since application state can change between two polls,
> > any thoughts?
>
> I didn't skip it. I'm intentionally keeping the discussion high level
> because there's no code here to compare. It's purely a "multiple
> requests for single app state" vs. "single request for multiple
> applications' statuses" discussion.
>
> The bulk API I suggested you to investigate should be able to support
> enough filtering so that Livy only gets the information it needs
> (maybe with a little extra noise). It should't get every single YARN
> application ever run, for example.
>
> This method is more what I was thinking of:
>
> 287  public abstract List getApplications(
> 288  Set applicationTypes,
> 289  EnumSet applicationStates) throws
> YarnException,
> 290  IOException;
>
> Lets you query apps with a given type and multiple states that you're
> interested in. It's not optimal (doesn't let you filter by tags, for
> example), but it's better than getting all apps. Maybe that's now
> enough either, but you're proposing the changes, so please explain why
> that is not enough instead of just throwing the question back at me.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu]

yes, it is going to be Akka if moving forward (at least not going to
introduce an actor framework to livy)

On Wed, Aug 16, 2017 at 11:24 AM, Meisam Fathi 
wrote:

> That is true, but I was under the impression that this will be implemented
> with Akka (maybe because it is mentioned in the design doc).
>
> On Wed, Aug 16, 2017 at 11:21 AM Marcelo Vanzin 
> wrote:
>
> > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi 
> > wrote:
> > > I do agree that actor based design is cleaner and more maintainable.
> But
> > we
> > > had to discard it because it adds more dependencies to Livy.
> >
> > I've been reading "actor system" as a design pattern, not as
> > introducing a new dependency to Livy.
> >
> > If the document is actually proposing using Akka (instead of just
> > using Akka as an example of an actor system implementation), then I'm
> > a -1 on that.
> >
> > --
> > Marcelo
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi]

That is true, but I was under the impression that this will be implemented
with Akka (maybe because it is mentioned in the design doc).

On Wed, Aug 16, 2017 at 11:21 AM Marcelo Vanzin  wrote:

> On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi 
> wrote:
> > I do agree that actor based design is cleaner and more maintainable. But
> we
> > had to discard it because it adds more dependencies to Livy.
>
> I've been reading "actor system" as a design pattern, not as
> introducing a new dependency to Livy.
>
> If the document is actually proposing using Akka (instead of just
> using Akka as an example of an actor system implementation), then I'm
> a -1 on that.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi  wrote:
> I do agree that actor based design is cleaner and more maintainable. But we
> had to discard it because it adds more dependencies to Livy.

I've been reading "actor system" as a design pattern, not as
introducing a new dependency to Livy.

If the document is actually proposing using Akka (instead of just
using Akka as an example of an actor system implementation), then I'm
a -1 on that.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu]

> What I proposed is having a single request to YARN to get all applications'
statuses, if that's possible. You'd still have multiple application handles
that are independent of each other. They'd all be updated separately from
that one thread talking to YARN. This has nothing to do with a "shared data
structure". There's no shared data structure here to track application
status.

You are still avoiding the questions how you make all "application handles"
accessible to this thread

Please go with direct discussion

> No, but I suggested that you look whether that exists since I think that's
a better solution both from YARN and Livy's perspectives, since it requires
less resources. It should at least be mentioned as an alternative in your
mini-spec and, if it doesn't work for whatever reason, deserves an
explanation.

"I would investigate whether there's any API in YARN to do a bulk get of
running applications with a particular filter;" - from your email

If you suggest something, please find evidence to support you

> Irrelevant.

Please go with direct discussion

> What if YARN goes down? What if your datacenter has a massive power
failure? You have to handle errors in any scenario.

Again, I am describing one concrete scenario which is always involved in
any bulk operation and even we go to bulk direction, you have to handle
this. Since you proposed this bulk operation, I am asking you what's your
expectation about this. But you are throwing some imaginations without any
values

Please go with direct discussion




On Wed, Aug 16, 2017 at 9:11 AM, Marcelo Vanzin  wrote:

> On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu  wrote:
> >> I'm not really sure what you're talking about here, since I did not
> > suggest a "shared data structure", and I'm not really sure what that
> > means in this context.
> >
> > What you claimed is just monitoring/updating the state with a single
> thread
> > *given* all applications have been there.
>
> What I proposed is having a single request to YARN to get all
> applications' statuses, if that's possible. You'd still have multiple
> application handles that are independent of each other. They'd all be
> updated separately from that one thread talking to YARN.
>
> This has nothing to do with a "shared data structure". There's no
> shared data structure here to track application status.
>
> >> Yes. While there are applications that need monitoring, you poll YARN
> > at a constant frequency. Basically what would be done by multiple
> > threads, but there's a single one.
> >
> > Did you find the bulk API?
>
> No, but I suggested that you look whether that exists since I think
> that's a better solution both from YARN and Livy's perspectives, since
> it requires less resources. It should at least be mentioned as an
> alternative in your mini-spec and, if it doesn't work for whatever
> reason, deserves an explanation.
>
> >> Why not. The expensive part is not parsing results, I'll bet, but
> > having a whole bunch of different tasks opening and closing YARN
> > connections.
> >
> > First, YARNClient is thread safe and can be shared by multiple
> threads
>
> Irrelevant.
>
> > Second, If I have 1000 applications, what's your expectation to the
> > following cases
> >
> > 1. YARN processed request for 999 and failed on the last one for some
> reason
> >
> > 2. Livy received 999 well-formatted response but get 1 malformed response
>
> What if YARN goes down? What if your datacenter has a massive power
> failure?
>
> You have to handle errors in any scenario.
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin]

On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu  wrote:
>> I'm not really sure what you're talking about here, since I did not
> suggest a "shared data structure", and I'm not really sure what that
> means in this context.
>
> What you claimed is just monitoring/updating the state with a single thread
> *given* all applications have been there.

What I proposed is having a single request to YARN to get all
applications' statuses, if that's possible. You'd still have multiple
application handles that are independent of each other. They'd all be
updated separately from that one thread talking to YARN.

This has nothing to do with a "shared data structure". There's no
shared data structure here to track application status.

>> Yes. While there are applications that need monitoring, you poll YARN
> at a constant frequency. Basically what would be done by multiple
> threads, but there's a single one.
>
> Did you find the bulk API?

No, but I suggested that you look whether that exists since I think
that's a better solution both from YARN and Livy's perspectives, since
it requires less resources. It should at least be mentioned as an
alternative in your mini-spec and, if it doesn't work for whatever
reason, deserves an explanation.

>> Why not. The expensive part is not parsing results, I'll bet, but
> having a whole bunch of different tasks opening and closing YARN
> connections.
>
> First, YARNClient is thread safe and can be shared by multiple threads

Irrelevant.

> Second, If I have 1000 applications, what's your expectation to the
> following cases
>
> 1. YARN processed request for 999 and failed on the last one for some reason
>
> 2. Livy received 999 well-formatted response but get 1 malformed response

What if YARN goes down? What if your datacenter has a massive power failure?

You have to handle errors in any scenario.


-- 
Marcelo

resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu]

Hi, all

In HDInsight, we (Microsoft) use Livy as the Spark job submission service.
We keep seeing the customers fall into the problem when they submit many
concurrent applications to the system, or recover livy from a state with
many concurrent applications

By looking at the code and the customers' exception stack, we lock down the
problem to the application monitoring module where a new thread is created
for each application.

To resolve the issue, we propose a actor-based design of application
monitoring module and share it here (as new JIRA seems not working
yet) 
*https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
*

We are glad to hear feedbacks from the community and improve the design
before we start implementing it!

Best,

Nan