Re: [Proposal] Use jemalloc as default memory allocator for Mesos

2017-08-30 Thread Till Toenshoff 
Ow, I forgot my actual vote :)

+1 for using jemalloc
+1 for making it non bundled
+1 for using it by default if the configuration phase locates it

> On Aug 25, 2017, at 3:28 PM, Benno Evers  wrote:
> 
> Hi Jeff,
> 
> from looking around on the internet, it seems like Firefox builds with
> jemalloc on all platforms, and I've also seen reports of people
> successfully using tcmalloc heap profiling on windows. I'm afraid I don't
> currently have a Windows machine with development environment set up, so I
> can't provide direct user experience.
> 
> In the worst case, we would have to disable jemalloc-support on windows,
> but I hope it won't be necessary.
> 
> Since you probably have more experience with memory management on windows,
> is there a reason to suspect that it should or shouldn't work?
> 
> Best regards,
> Benno
> 
> On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler <
> jeff.coff...@microsoft.com.invalid> wrote:
> 
>> Hi Benno,
>> 
>> What's the availability of both jemalloc and tcmalloc on the Windows
>> platform? Do the products work there properly?
>> 
>> There are solutions that I know work on Windows (from past work I've
>> done). I'm unsure about either jemalloc and tcmalloc, however.
>> 
>> Thanks,
>> 
>> /Jeff
>> 
>> -Original Message-
>> From: Benno Evers [mailto:bev...@mesosphere.com]
>> Sent: Tuesday, August 22, 2017 3:16 AM
>> To: dev@mesos.apache.org
>> Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos
>> 
>> Hi Alexander,
>> 
>> in general, jemalloc and tcmalloc are very similar, and seem to be taking
>> ideas from each other (in fact the jeprof executable started as a copy of
>> pprof and there are still references the pprof documentation in some
>> comments)
>> 
>> From what I've seen, the main difference is that the profiling seems
>> better-suited to multi-threaded programs, in particular the profile file
>> format includes per-thread memory statistics and the profiling features can
>> be turned on and off individually per thread. From an API perspective, all
>> settings can be accessed by the mallctl() function, while it seems that
>> tcmalloc requires some options to be set by environment variable (
>> https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fgperftools.github.io%2Fgperftools%
>> 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%
>> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
>> db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%
>> 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also
>> found the documentation to be more thorough.
>> 
>> But again, the two are very similar, so I think the main decision here
>> isn't whether to choose jemalloc or tcmalloc but whether to switch to a
>> custom memory allocator that has support for profiling heap memory usage.
>> 
>> 
>> On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas 
>> wrote:
>> 
>>> Hi Benno,
>>> 
>>> This does sound like a great addition to Mesos. Can you however
>>> explain how jemalloc is better than tcmalloc? I think that for such
>>> important change, we probably need some more information.
>>> 
>>> Your comment in MESOS-7876 mentions that we already have tcmalloc
>>> since it is part of gperftools, so I would like to have a whole
>>> picture of the advantages and disadvantages of both options.
>>> 
>>> Alexander Rojas
>>> alexan...@mesosphere.io
>>> 
>>> 
>>> 
>>> 
 On 18. Aug 2017, at 12:49, Benno Evers  wrote:
 
 Hi all,
 
 I would like to propose bundling jemalloc as a new dependency under
 `3rdparty/`, and to link Mesos against this new memory allocator by
 default.
 
 
 # Motivation
 
 The Mesos master and agent binaries are, ideally, very long-running
 processes. This makes them susceptible to memory issues, because
 even small leaks have a chance to build up over time to the point
 where they become problematic.
 
 We have seen several such issues on our internal Mesos
 installations, for example
 https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
 ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Coff
 ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=L016YGyEkK5
 0WtvhgSNS%2FT5ntkkd9qINorRI2Utp5lk%3D&reserved=0
 or https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMESOS-
>> 7800&data=02%7C01%7CJeff.Coffler%40microsoft.com%
>> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
>> db47%7C1%7C0%7C636389937852266742&sdata=IrzDO6o1VL9a8eGJIW3jKbWXk6U4fH
>> Fn3Xbn4po1r3c%3D&reserved=0.
 
 I imagine any organization running Mesos for an extended period of
 time has had its share of similar issues, so I expect this proposal
 to be useful for the whole community.
 
 
 # Why jemalloc?
 
 Given that memory issu

Re: [Proposal] Use jemalloc as default memory allocator for Mesos

2017-08-30 Thread Till Toenshoff 
It appears that jemalloc does support Windows (64bit) 
See: https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt 


tcmalloc on the other hand appears to only offer a minimal variant on Windows.
See: https://github.com/gperftools/gperftools 
 - grep for "COMPILING ON NON-LINUX 
SYSTEMS”
See: https://github.com/gperftools/gperftools/blob/master/INSTALL 
 - grep for 
“Windows  (MSVC, Cygwin, and MinGW)"

So both options rely on Cygwin or MinGW for building - a requirement that the 
Mesos build itself does not have and proves your point of stuff not really 
“just working” at least when it comes to the build step of those packages.

Seems that without trying it, we won’t find out if jemalloc works as hoped on 
Windows for us - the Firefox project results however are encouraging. On the 
other hand, if it doesn’t work, we could simply decide to disable it on Windows 
just like some other Mesos features will remain disabled on that platform 
unless someone decides to port them -  e.g. SASL based authn.

> On Aug 25, 2017, at 3:28 PM, Benno Evers  wrote:
> 
> Hi Jeff,
> 
> from looking around on the internet, it seems like Firefox builds with
> jemalloc on all platforms, and I've also seen reports of people
> successfully using tcmalloc heap profiling on windows. I'm afraid I don't
> currently have a Windows machine with development environment set up, so I
> can't provide direct user experience.
> 
> In the worst case, we would have to disable jemalloc-support on windows,
> but I hope it won't be necessary.
> 
> Since you probably have more experience with memory management on windows,
> is there a reason to suspect that it should or shouldn't work?
> 
> Best regards,
> Benno
> 
> On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler <
> jeff.coff...@microsoft.com.invalid> wrote:
> 
>> Hi Benno,
>> 
>> What's the availability of both jemalloc and tcmalloc on the Windows
>> platform? Do the products work there properly?
>> 
>> There are solutions that I know work on Windows (from past work I've
>> done). I'm unsure about either jemalloc and tcmalloc, however.
>> 
>> Thanks,
>> 
>> /Jeff
>> 
>> -Original Message-
>> From: Benno Evers [mailto:bev...@mesosphere.com]
>> Sent: Tuesday, August 22, 2017 3:16 AM
>> To: dev@mesos.apache.org
>> Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos
>> 
>> Hi Alexander,
>> 
>> in general, jemalloc and tcmalloc are very similar, and seem to be taking
>> ideas from each other (in fact the jeprof executable started as a copy of
>> pprof and there are still references the pprof documentation in some
>> comments)
>> 
>> From what I've seen, the main difference is that the profiling seems
>> better-suited to multi-threaded programs, in particular the profile file
>> format includes per-thread memory statistics and the profiling features can
>> be turned on and off individually per thread. From an API perspective, all
>> settings can be accessed by the mallctl() function, while it seems that
>> tcmalloc requires some options to be set by environment variable (
>> https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fgperftools.github.io%2Fgperftools%
>> 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%
>> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
>> db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%
>> 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also
>> found the documentation to be more thorough.
>> 
>> But again, the two are very similar, so I think the main decision here
>> isn't whether to choose jemalloc or tcmalloc but whether to switch to a
>> custom memory allocator that has support for profiling heap memory usage.
>> 
>> 
>> On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas 
>> wrote:
>> 
>>> Hi Benno,
>>> 
>>> This does sound like a great addition to Mesos. Can you however
>>> explain how jemalloc is better than tcmalloc? I think that for such
>>> important change, we probably need some more information.
>>> 
>>> Your comment in MESOS-7876 mentions that we already have tcmalloc
>>> since it is part of gperftools, so I would like to have a whole
>>> picture of the advantages and disadvantages of both options.
>>> 
>>> Alexander Rojas
>>> alexan...@mesosphere.io
>>> 
>>> 
>>> 
>>> 
 On 18. Aug 2017, at 12:49, Benno Evers  wrote:
 
 Hi all,
 
 I would like to propose bundling jemalloc as a new dependency under
 `3rdparty/`, and to link Mesos against this new memory allocator by
 default.
 
 
 # Motivation
 
 The Mesos master and agent binaries are, ideally, very long-running
 processes. This makes them susceptible to memory issues, because
 even small leaks have a chance to build up over time to the point
 where they become problematic.
 
 We ha

Re: [VOTE] Release Apache Mesos 1.1.3 (rc2)

2017-08-30 Thread Till Toenshoff 
+1

Tested on internal CI as well as on macOS 10.12 and macOS 10.13 DP 8 using 
Apple’s clang (Xcode 8.3.3 and Xcode 9.0.0 beta 6).

> On Aug 27, 2017, at 8:33 PM, Vinod Kone  wrote:
> 
> +1 (binding)
> 
> Tested on ASF CI. The only red build was the known perf core dump issue.
> 
> Revision: ce77d91bd3a59227d5684ce0783b460c54ea311f
> refs/tags/1.1.3-rc2
> Configuration Matrix  gcc clang
> centos:7  --verbose --enable-libevent --enable-sslautotools   
>  
> 
>   
> 
> cmake 
>  
> 
>   
> 
> --verbose autotools   
>  
> 
>  
> 
> cmake 
>  
> 
>  
> 
> ubuntu:14.04  --verbose --enable-libevent --enable-sslautotools   
>  
> 
>   
>  
> 
> cmake 
>  
> 
>   
>  
> 
> --verbose autotools   
>  
> 
>  
>  
> 
> cmake 
>  
> 
>  
>  
> 
> 
> On Fri, Aug 25, 2017 at 7:48 AM, Alex Rukletsov  > wrote:
> Folks,
> 
> Please vote on releasing the following candidate as Apache Mesos 1.1.3. Note 
> that this will be the last 1.1.x release.
> 
> 1.1.3 includes the following:
> 
> ** Bug
>  * [MESOS-5187] - The filesystem/linux isolator does not set the permissions 
> of the host_path.
>   * [MESOS-6743] - Docker executor hangs forever if `docker stop` fails.
>   * [MESOS-6950] - Launching two tasks with the same Docker image 
> simultaneously may cause a staging dir never cleaned up.
>   * [MESOS-7540] - Add an agent flag for executor re-registration timeout.
>   * [MESOS-7569] - Allow "old" executors with half-open connections to be 
> preserved during agent upgrade / restart.
>   * [MESOS-7689] - Libprocess can crash on malformed request paths for 
> libprocess messages.
>   * [MESOS-7690] - The agent can crash when an unknown executor tries to 
> register.
>   * [MESOS-7581] - Fix interference of exte

Re: Offer operation reconciliation discussion notes

2017-08-30 Thread Jie Yu 
> What are the next steps for moving this forward, Jie

James, the next step is to create a design doc for this.

Looks like we're aligned on the high level approach.

On Wed, Aug 30, 2017 at 11:40 AM, Jie Yu  wrote:

> + Gaston and Greg
>
> Who might be working on this.
>
> On Tue, Aug 29, 2017 at 6:59 AM, James DeFelice 
> wrote:
>
>> What are the next steps for moving this forward, Jie? I'm very interested
>> in seeing status updates for operations land sooner than later.
>>
>> On Wed, Aug 23, 2017 at 5:55 PM, Gabriel Hartmann 
>> wrote:
>>
>>> Please can the "reason" be the reason for the failure and NOT the reason
>>> the message was sent, e.g. "RECONCILIATION"
>>>
>>> On Wed, Aug 23, 2017 at 1:58 PM Yan Xu  wrote:
>>>
 Yeah a reason for failed operations is probably useful for all resource
 operations. It looks like the task-style status update is still the best
 approach.

 ---
 @xujyan 

 On Wed, Aug 23, 2017 at 11:40 AM, Jie Yu  wrote:

> We should continue the discussion here:
>
> I think I forgot to mention one important reason that I went for the
> operation based reconciliation API proposal. For new operations like
> CREATE_VOLUME/CREATE_BLOCK, not only we need to know the end result (the
> resources) if it's successful, we also need to know the failure reason if
> it fails. For instance, imagine you're creating an EBS volume by talking 
> to
> a CSI EBS plugin. Surfacing the creation error (e.g., retryable or not 
> from
> the CSI plugin) will be useful for scheduler to determine the next step.
>
> I don't think a resources based reconciliation API can address this.
> Maybe we can add both if we feel both are useful?
>
> Thoughts?
> - Jie
>
> On Wed, Aug 23, 2017 at 11:26 AM, Jie Yu  wrote:
>
>> Hi,
>>
>> We had a discussion on some very early proposal (see the attached
>> slides) on providing feedback for offer operations (e.g., CREATE/DESTORY,
>> RESERVE/UNRESERVE, etc.) with a bunch of folks from the community. Here 
>> are
>> the notes I captured in the meeting:
>>
>>
>>- One alternative approach discussed was to have best effort
>>feedback, and a resources based reconciliation API allowing framework 
>> to
>>query the resources on a given resource provider or agent. That way, 
>> we
>>don't necessarily need the status update mechanism for offer 
>> operations,
>>which causes complexity in the frameworks.
>>- In the current proposal, do we need agent_id (or resource
>>provider id) when performing reconciliation for that operation? The 
>> reason
>>we require that in the task reconciliation case is because agent 
>> might not
>>re-register yet during master failover.
>>- We need to mock up the operator API for this work.
>>- What's the order guarantee for the operations specified in one
>>API call?
>>- Wish list
>>   - Reservation tie to framework instead of role.
>>   - When a framework teardown, auto release resources reserved
>>   for that framework
>>
>> If I miss anything, please reply to this thread! Thanks!
>>
>> https://docs.google.com/presentation/d/1Mef8K3aLIuzcFVc3MnAo
>> 64TkjpyTWarYVShtvCN4e48/edit?usp=sharing
>>
>> - Jie
>>
>
>

>>
>

Re: Offer operation reconciliation discussion notes

2017-08-30 Thread Jie Yu 
+ Gaston and Greg

Who might be working on this.

On Tue, Aug 29, 2017 at 6:59 AM, James DeFelice  wrote:

> What are the next steps for moving this forward, Jie? I'm very interested
> in seeing status updates for operations land sooner than later.
>
> On Wed, Aug 23, 2017 at 5:55 PM, Gabriel Hartmann 
> wrote:
>
>> Please can the "reason" be the reason for the failure and NOT the reason
>> the message was sent, e.g. "RECONCILIATION"
>>
>> On Wed, Aug 23, 2017 at 1:58 PM Yan Xu  wrote:
>>
>>> Yeah a reason for failed operations is probably useful for all resource
>>> operations. It looks like the task-style status update is still the best
>>> approach.
>>>
>>> ---
>>> @xujyan 
>>>
>>> On Wed, Aug 23, 2017 at 11:40 AM, Jie Yu  wrote:
>>>
 We should continue the discussion here:

 I think I forgot to mention one important reason that I went for the
 operation based reconciliation API proposal. For new operations like
 CREATE_VOLUME/CREATE_BLOCK, not only we need to know the end result (the
 resources) if it's successful, we also need to know the failure reason if
 it fails. For instance, imagine you're creating an EBS volume by talking to
 a CSI EBS plugin. Surfacing the creation error (e.g., retryable or not from
 the CSI plugin) will be useful for scheduler to determine the next step.

 I don't think a resources based reconciliation API can address this.
 Maybe we can add both if we feel both are useful?

 Thoughts?
 - Jie

 On Wed, Aug 23, 2017 at 11:26 AM, Jie Yu  wrote:

> Hi,
>
> We had a discussion on some very early proposal (see the attached
> slides) on providing feedback for offer operations (e.g., CREATE/DESTORY,
> RESERVE/UNRESERVE, etc.) with a bunch of folks from the community. Here 
> are
> the notes I captured in the meeting:
>
>
>- One alternative approach discussed was to have best effort
>feedback, and a resources based reconciliation API allowing framework 
> to
>query the resources on a given resource provider or agent. That way, we
>don't necessarily need the status update mechanism for offer 
> operations,
>which causes complexity in the frameworks.
>- In the current proposal, do we need agent_id (or resource
>provider id) when performing reconciliation for that operation? The 
> reason
>we require that in the task reconciliation case is because agent might 
> not
>re-register yet during master failover.
>- We need to mock up the operator API for this work.
>- What's the order guarantee for the operations specified in one
>API call?
>- Wish list
>   - Reservation tie to framework instead of role.
>   - When a framework teardown, auto release resources reserved
>   for that framework
>
> If I miss anything, please reply to this thread! Thanks!
>
> https://docs.google.com/presentation/d/1Mef8K3aLIuzcFVc3MnAo
> 64TkjpyTWarYVShtvCN4e48/edit?usp=sharing
>
> - Jie
>


>>>
>

08/31 community meeting agenda

2017-08-30 Thread Benjamin Hindman 
Please see the doc

for
an up to date agenda and Hangouts links.

tl;dr; In addition to going over the community kanban board we'll also
discuss some upcoming changes and proposals around libprocess `Future`
discard and abandonment semantics. If you're hacking in Mesos or interested
to learn more about libprocess please join the meeting tomorrow at 3 PM PT!

RE: [Proposal] Use jemalloc as default memory allocator for Mesos

2017-08-30 Thread Jeff Coffler 
Hi Benno,

These days, I never assume that a Linux package "just works" on Windows. I'm 
well aware (all too much) of just how different Windows is from Linux, and thus 
how difficult it can be to write software that works on both platforms. So 
unless the author of the memory management software (deployed on Linux) 
happened to consider Windows important enough to support, I'd assume that 
Windows is not supported (and would almost certainly not "just work" by chance).

I really asked my question because Mesos is intended to be a portable memory 
package. Thus, if we pick up a new third party dependency that would affect the 
Mesos agent, we should pick third party dependencies that work on Windows, Mac, 
and other platforms. This is something we should be thinking about.

/Jeff

-Original Message-
From: Benno Evers [mailto:bev...@mesosphere.com] 
Sent: Friday, August 25, 2017 6:28 AM
To: dev@mesos.apache.org
Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos

Hi Jeff,

from looking around on the internet, it seems like Firefox builds with jemalloc 
on all platforms, and I've also seen reports of people successfully using 
tcmalloc heap profiling on windows. I'm afraid I don't currently have a Windows 
machine with development environment set up, so I can't provide direct user 
experience.

In the worst case, we would have to disable jemalloc-support on windows, but I 
hope it won't be necessary.

Since you probably have more experience with memory management on windows, is 
there a reason to suspect that it should or shouldn't work?

Best regards,
Benno

On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler < 
jeff.coff...@microsoft.com.invalid> wrote:

> Hi Benno,
>
> What's the availability of both jemalloc and tcmalloc on the Windows 
> platform? Do the products work there properly?
>
> There are solutions that I know work on Windows (from past work I've 
> done). I'm unsure about either jemalloc and tcmalloc, however.
>
> Thanks,
>
> /Jeff
>
> -Original Message-
> From: Benno Evers [mailto:bev...@mesosphere.com]
> Sent: Tuesday, August 22, 2017 3:16 AM
> To: dev@mesos.apache.org
> Subject: Re: [Proposal] Use jemalloc as default memory allocator for 
> Mesos
>
> Hi Alexander,
>
> in general, jemalloc and tcmalloc are very similar, and seem to be 
> taking ideas from each other (in fact the jeprof executable started as 
> a copy of pprof and there are still references the pprof documentation 
> in some
> comments)
>
> From what I've seen, the main difference is that the profiling seems 
> better-suited to multi-threaded programs, in particular the profile 
> file format includes per-thread memory statistics and the profiling 
> features can be turned on and off individually per thread. From an API 
> perspective, all settings can be accessed by the mallctl() function, 
> while it seems that tcmalloc requires some options to be set by 
> environment variable ( 
> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgperftools.github.io%2Fgperftools%
> 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%
> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
> db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%
> 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I 
> also found the documentation to be more thorough.
>
> But again, the two are very similar, so I think the main decision here 
> isn't whether to choose jemalloc or tcmalloc but whether to switch to 
> a custom memory allocator that has support for profiling heap memory usage.
>
>
> On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas 
> 
> wrote:
>
> > Hi Benno,
> >
> > This does sound like a great addition to Mesos. Can you however 
> > explain how jemalloc is better than tcmalloc? I think that for such 
> > important change, we probably need some more information.
> >
> > Your comment in MESOS-7876 mentions that we already have tcmalloc 
> > since it is part of gperftools, so I would like to have a whole 
> > picture of the advantages and disadvantages of both options.
> >
> > Alexander Rojas
> > alexan...@mesosphere.io
> >
> >
> >
> >
> > > On 18. Aug 2017, at 12:49, Benno Evers  wrote:
> > >
> > > Hi all,
> > >
> > > I would like to propose bundling jemalloc as a new dependency 
> > > under `3rdparty/`, and to link Mesos against this new memory 
> > > allocator by default.
> > >
> > >
> > > # Motivation
> > >
> > > The Mesos master and agent binaries are, ideally, very 
> > > long-running processes. This makes them susceptible to memory 
> > > issues, because even small leaks have a chance to build up over 
> > > time to the point where they become problematic.
> > >
> > > We have seen several such issues on our internal Mesos 
> > > installations, for example 
> > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fi
> > > ss 
> > > ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Co
> > > ff 
> > > ler%40microsoft.com%7Ccb0bfb1eb3

[Proposal] Replicated log replica catch-up

2017-08-30 Thread Ilya Pronin 
Hi everyone,

I've created a document with a proposal to add eager catch-up functionality
to non-leading replicated log replicas. It would allow frameworks that use
the replicated log (e.g. Aurora) to reduce failover time (by keeping
in-memory storage "hot") and potentially offload some eventually consistent
reading from a leader to standbys.

I have a working prototype of the logic described in the doc.

https://docs.google.com/document/d/1dERXJeAsi3Lnq9Akt82JGWK4pKNeJ6k7PTVCpM9ic_8/edit?usp=sharing

Please feel free to comment on the doc and let me know if you have any
concerns.

Thanks!

-- 
Ilya Pronin