Mesos masters and zookeeper running together?

[Ron Lipke]

Hello, I've been working on setting up a mesos cluster for eventual
production use and I have a question on configuring zookeeper alongside
the mesos masters.  
Is it best practice to run zookeeper/exhibitor as a separate cluster (in
our case, three nodes) or on the same machines as the mesos masters?  I
understand the drawbacks of increased cost for compute resources that
will just be running a single service and most of the reference docs
have them running together, but just wondering if it's beneficial to
have them uncoupled.

Thanks in advance for any input.

Ron Lipke
@neverminding

Re: Mesos fetcher in dockerized slave

[Tim Chen]

Hi Marica/Shuai,

Sorry haven't been able to spend the time to repro, but looks like Shuai
confirmed it.

Can one of you file a JIRA?

Thanks!

Tim

On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin  wrote:

> Hi Marica,
>
> I can reproduce the problem exactly as you described in the first email of
> this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set,
> the fetcher works just fine; With it, the fetcher steps seems skipped. This
> looks like a bug to me.
>
> Regards,
> Shuai
>
> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
> marica.antona...@ba.infn.it> wrote:
>
>> Dear all,
>>
>> I have not solved this issue yet. Please, can anyone run the same test
>> and let me know if the fetcher is correctly invoked?
>> The test is really simple, just try to start a dockerized app (see json
>> definition file below) through marathon on a mesos slave running in a
>> docker container started with the option —docker_mesos_image=> image>.
>> I would appreciate very much any feedback.
>>
>> Sample Marathon app:
>> {
>>  "id": "test-app",
>>  "container": {
>>"type": "DOCKER",
>>"docker": {
>>  "image": "libmesos/ubuntu"
>>}
>>  },
>>  "cpus": 1,
>>  "mem": 512,
>>  "uris": [ "
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv";
>> ],
>>  "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>> done"
>> }
>>
>> Docker run command to start dockerized mesos slave:
>>
>> # docker run -d MESOS_HOSTNAME= -e MESOS_IP= -e
>> MESOS_MASTER=zk://:2181,:2181,:2181/mesos -e
>> MESOS_CONTAINERIZERS=docker,mesos
>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>> --name slave --net host --privileged --pid host mesos-slave
>>
>> Thank you very much in advance!
>> Best regards,
>> Marica
>>
>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>> marica.antona...@ba.infn.it> ha scritto:
>>
>> Dear Tim,
>>
>> I have collected some information from my test environment, starting the
>> slave container with and without the —docker_mesos_image startup flag.
>> Please let me know if you need further input. Thank you very much for your
>> support!
>>
>> Using the flag —docker_mesos_image:
>>
>> root@mesos-slave:~# docker ps
>> CONTAINER IDIMAGE   COMMAND  CREATED
>> STATUS  PORTS   NAMES
>> b30cea22a07clibmesos/ubuntu "/bin/sh -c 'cd $MESO"   2
>> minutes ago   Up 2 minutes
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> da9c78ec5727mesos-slave "/bin/sh -c '/usr/lib"   2
>> minutes ago   Up 2 minutes
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> 150f78fbf327mesos-slave "/entrypoint.sh /usr/"   3
>> minutes ago   Up 3 minutesslave
>>
>> *root@mesos-slave:~# docker logs slave*
>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-
>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-
>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> to user 'root'
>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a- with resources cpus(*):0.1;
>> mem(*):32 in work directory
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-
>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>> '246b272b-d649-47c0-88ca-6b1ff35f437a-'
>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to
>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/p

Re: Mesos masters and zookeeper running together?

[craig w]

We've run the mesos-master and marathon processes in the same servers as
zookeeper.
On Dec 24, 2015 11:01 AM, "Ron Lipke"  wrote:

> Hello, I've been working on setting up a mesos cluster for eventual
> production use and I have a question on configuring zookeeper alongside
> the mesos masters.
> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
> our case, three nodes) or on the same machines as the mesos masters?  I
> understand the drawbacks of increased cost for compute resources that
> will just be running a single service and most of the reference docs
> have them running together, but just wondering if it's beneficial to
> have them uncoupled.
>
> Thanks in advance for any input.
>
> Ron Lipke
> @neverminding
>

Re: Mesos masters and zookeeper running together?

[Rodrick Brown]

With our design we end up building out a stand alone zookeeper cluster 3 nodes. 
 Zookeeper seems to be the default dumping ground for many Apache based 
products these days. You will eventually see many services and frameworks 
require a zk instance for leader election, coordination, Kv store etc.. I've 
seen situations where the masters can become extremely busy and cause 
performance problem with Zk which can be huge issue for mesos. 
Sent from Outlook Mobile




On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"  wrote:










Hello, I've been working on setting up a mesos cluster for eventual
production use and I have a question on configuring zookeeper alongside
the mesos masters.  
Is it best practice to run zookeeper/exhibitor as a separate cluster (in
our case, three nodes) or on the same machines as the mesos masters?  I
understand the drawbacks of increased cost for compute resources that
will just be running a single service and most of the reference docs
have them running together, but just wondering if it's beneficial to
have them uncoupled.

Thanks in advance for any input.

Ron Lipke
@neverminding






-- 
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for 
the use of the addressee only. If you are not an intended recipient of this 
communication, please delete it immediately and notify the sender by return 
email. Unauthorized reading, dissemination, distribution or copying of this 
communication is prohibited. This communication does not constitute an 
offer to sell or a solicitation of an indication of interest to purchase 
any loan, security or any other financial product or instrument, nor is it 
an offer to sell or a solicitation of an indication of interest to purchase 
any products or services to any persons who are prohibited from receiving 
such information under applicable law. The contents of this communication 
may not be accurate or complete and are subject to change without notice. 
As such, Orchard App, Inc. (including its subsidiaries and affiliates, 
"Orchard") makes no representation regarding the accuracy or completeness 
of the information contained herein. The intended recipient is advised to 
consult its own professional advisors, including those specializing in 
legal, tax and accounting matters. Orchard does not provide legal, tax or 
accounting advice.

Re: Mesos masters and zookeeper running together?

[Dick Davies]

zookeeper really wants a dedicated cluster IMO; preferably with SSD
under it - if zookeeper
starts to run slow then everything else will start to bog down. I've
co-hosted it with mesos masters
before now for demo purposes etc. but for production it's probably
worth choosing dedicated hosts.

On 24 December 2015 at 20:36, Rodrick Brown  wrote:
> With our design we end up building out a stand alone zookeeper cluster 3
> nodes.  Zookeeper seems to be the default dumping ground for many Apache
> based products these days. You will eventually see many services and
> frameworks require a zk instance for leader election, coordination, Kv store
> etc.. I've seen situations where the masters can become extremely busy and
> cause performance problem with Zk which can be huge issue for mesos.
>
> Sent from Outlook Mobile
>
>
>
>
> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"  wrote:
>
>> Hello, I've been working on setting up a mesos cluster for eventual
>> production use and I have a question on configuring zookeeper alongside
>> the mesos masters.
>> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
>> our case, three nodes) or on the same machines as the mesos masters?  I
>> understand the drawbacks of increased cost for compute resources that
>> will just be running a single service and most of the reference docs
>> have them running together, but just wondering if it's beneficial to
>> have them uncoupled.
>>
>> Thanks in advance for any input.
>>
>> Ron Lipke
>> @neverminding
>
>
> NOTICE TO RECIPIENTS: This communication is confidential and intended for
> the use of the addressee only. If you are not an intended recipient of this
> communication, please delete it immediately and notify the sender by return
> email. Unauthorized reading, dissemination, distribution or copying of this
> communication is prohibited. This communication does not constitute an offer
> to sell or a solicitation of an indication of interest to purchase any loan,
> security or any other financial product or instrument, nor is it an offer to
> sell or a solicitation of an indication of interest to purchase any products
> or services to any persons who are prohibited from receiving such
> information under applicable law. The contents of this communication may not
> be accurate or complete and are subject to change without notice. As such,
> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
> makes no representation regarding the accuracy or completeness of the
> information contained herein. The intended recipient is advised to consult
> its own professional advisors, including those specializing in legal, tax
> and accounting matters. Orchard does not provide legal, tax or accounting
> advice.

Re: Mesos masters and zookeeper running together?

[craig w]

What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.
On Dec 24, 2015 5:13 PM, "Dick Davies"  wrote:

> zookeeper really wants a dedicated cluster IMO; preferably with SSD
> under it - if zookeeper
> starts to run slow then everything else will start to bog down. I've
> co-hosted it with mesos masters
> before now for demo purposes etc. but for production it's probably
> worth choosing dedicated hosts.
>
> On 24 December 2015 at 20:36, Rodrick Brown 
> wrote:
> > With our design we end up building out a stand alone zookeeper cluster 3
> > nodes.  Zookeeper seems to be the default dumping ground for many Apache
> > based products these days. You will eventually see many services and
> > frameworks require a zk instance for leader election, coordination, Kv
> store
> > etc.. I've seen situations where the masters can become extremely busy
> and
> > cause performance problem with Zk which can be huge issue for mesos.
> >
> > Sent from Outlook Mobile
> >
> >
> >
> >
> > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" 
> wrote:
> >
> >> Hello, I've been working on setting up a mesos cluster for eventual
> >> production use and I have a question on configuring zookeeper alongside
> >> the mesos masters.
> >> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
> >> our case, three nodes) or on the same machines as the mesos masters?  I
> >> understand the drawbacks of increased cost for compute resources that
> >> will just be running a single service and most of the reference docs
> >> have them running together, but just wondering if it's beneficial to
> >> have them uncoupled.
> >>
> >> Thanks in advance for any input.
> >>
> >> Ron Lipke
> >> @neverminding
> >
> >
> > NOTICE TO RECIPIENTS: This communication is confidential and intended for
> > the use of the addressee only. If you are not an intended recipient of
> this
> > communication, please delete it immediately and notify the sender by
> return
> > email. Unauthorized reading, dissemination, distribution or copying of
> this
> > communication is prohibited. This communication does not constitute an
> offer
> > to sell or a solicitation of an indication of interest to purchase any
> loan,
> > security or any other financial product or instrument, nor is it an
> offer to
> > sell or a solicitation of an indication of interest to purchase any
> products
> > or services to any persons who are prohibited from receiving such
> > information under applicable law. The contents of this communication may
> not
> > be accurate or complete and are subject to change without notice. As
> such,
> > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
> > makes no representation regarding the accuracy or completeness of the
> > information contained herein. The intended recipient is advised to
> consult
> > its own professional advisors, including those specializing in legal, tax
> > and accounting matters. Orchard does not provide legal, tax or accounting
> > advice.
>

Re: Mesos masters and zookeeper running together?

[Ron Lipke]

Thanks, this makes a lot of sense.  

It also allows us to do rolling upgrades to the zookeeper EC2 instances as we 
build out the cluster.  

Thanks everyone for the help!  Happy Holidays.

Ron Lipke
@neverminding
> On Dec 24, 2015, at 5:13 PM, Dick Davies  wrote:
> 
> zookeeper really wants a dedicated cluster IMO; preferably with SSD
> under it - if zookeeper
> starts to run slow then everything else will start to bog down. I've
> co-hosted it with mesos masters
> before now for demo purposes etc. but for production it's probably
> worth choosing dedicated hosts.
> 
> On 24 December 2015 at 20:36, Rodrick Brown  
> wrote:
>> With our design we end up building out a stand alone zookeeper cluster 3
>> nodes.  Zookeeper seems to be the default dumping ground for many Apache
>> based products these days. You will eventually see many services and
>> frameworks require a zk instance for leader election, coordination, Kv store
>> etc.. I've seen situations where the masters can become extremely busy and
>> cause performance problem with Zk which can be huge issue for mesos.
>> 
>> Sent from Outlook Mobile
>> 
>> 
>> 
>> 
>> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"  wrote:
>> 
>>> Hello, I've been working on setting up a mesos cluster for eventual
>>> production use and I have a question on configuring zookeeper alongside
>>> the mesos masters.
>>> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
>>> our case, three nodes) or on the same machines as the mesos masters?  I
>>> understand the drawbacks of increased cost for compute resources that
>>> will just be running a single service and most of the reference docs
>>> have them running together, but just wondering if it's beneficial to
>>> have them uncoupled.
>>> 
>>> Thanks in advance for any input.
>>> 
>>> Ron Lipke
>>> @neverminding
>> 
>> 
>> NOTICE TO RECIPIENTS: This communication is confidential and intended for
>> the use of the addressee only. If you are not an intended recipient of this
>> communication, please delete it immediately and notify the sender by return
>> email. Unauthorized reading, dissemination, distribution or copying of this
>> communication is prohibited. This communication does not constitute an offer
>> to sell or a solicitation of an indication of interest to purchase any loan,
>> security or any other financial product or instrument, nor is it an offer to
>> sell or a solicitation of an indication of interest to purchase any products
>> or services to any persons who are prohibited from receiving such
>> information under applicable law. The contents of this communication may not
>> be accurate or complete and are subject to change without notice. As such,
>> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
>> makes no representation regarding the accuracy or completeness of the
>> information contained herein. The intended recipient is advised to consult
>> its own professional advisors, including those specializing in legal, tax
>> and accounting matters. Orchard does not provide legal, tax or accounting
>> advice.

Re: Mesos masters and zookeeper running together?

[Rodrick Brown]

3x - r3.xlarge  @ 4x vCPU 30GB Mem 1x80GB SSD

> On Dec 24 2015, at 6:01 pm, craig w  wrote:  

>

> What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.

>

> On Dec 24, 2015 5:13 PM, "Dick Davies"
<[d...@hellooperator.net](mailto:d...@hellooperator.net)> wrote:  

>

>> zookeeper really wants a dedicated cluster IMO; preferably with SSD  
under it - if zookeeper  
starts to run slow then everything else will start to bog down. I've  
co-hosted it with mesos masters  
before now for demo purposes etc. but for production it's probably  
worth choosing dedicated hosts.  
  
On 24 December 2015 at 20:36, Rodrick Brown
<[rodr...@orchardplatform.com](mailto:rodr...@orchardplatform.com)>
wrote:  
> With our design we end up building out a stand alone zookeeper cluster 3  
> nodes.  Zookeeper seems to be the default dumping ground for many Apache  
> based products these days. You will eventually see many services and  
> frameworks require a zk instance for leader election, coordination, Kv
store  
> etc.. I've seen situations where the masters can become extremely busy
and  
> cause performance problem with Zk which can be huge issue for mesos.  
>  
> Sent from Outlook Mobile  
>  
>  
>  
>  
> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"
<[m...@ronlipke.com](mailto:m...@ronlipke.com)> wrote:  
>  
>> Hello, I've been working on setting up a mesos cluster for eventual  
>> production use and I have a question on configuring zookeeper
alongside  
>> the mesos masters.  
>> Is it best practice to run zookeeper/exhibitor as a separate cluster
(in  
>> our case, three nodes) or on the same machines as the mesos masters?
I  
>> understand the drawbacks of increased cost for compute resources that  
>> will just be running a single service and most of the reference docs  
>> have them running together, but just wondering if it's beneficial to  
>> have them uncoupled.  
>>  
>> Thanks in advance for any input.  
>>  
>> Ron Lipke  
>> @neverminding  
>  
>  
> NOTICE TO RECIPIENTS: This communication is confidential and intended for  
> the use of the addressee only. If you are not an intended recipient of
this  
> communication, please delete it immediately and notify the sender by
return  
> email. Unauthorized reading, dissemination, distribution or copying of
this  
> communication is prohibited. This communication does not constitute an
offer  
> to sell or a solicitation of an indication of interest to purchase any
loan,  
> security or any other financial product or instrument, nor is it an offer
to  
> sell or a solicitation of an indication of interest to purchase any
products  
> or services to any persons who are prohibited from receiving such  
> information under applicable law. The contents of this communication may
not  
> be accurate or complete and are subject to change without notice. As
such,  
> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")  
> makes no representation regarding the accuracy or completeness of the  
> information contained herein. The intended recipient is advised to
consult  
> its own professional advisors, including those specializing in legal, tax  
> and accounting matters. Orchard does not provide legal, tax or accounting  
> advice.  


-- 
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for 
the use of the addressee only. If you are not an intended recipient of this 
communication, please delete it immediately and notify the sender by return 
email. Unauthorized reading, dissemination, distribution or copying of this 
communication is prohibited. This communication does not constitute an 
offer to sell or a solicitation of an indication of interest to purchase 
any loan, security or any other financial product or instrument, nor is it 
an offer to sell or a solicitation of an indication of interest to purchase 
any products or services to any persons who are prohibited from receiving 
such information under applicable law. The contents of this communication 
may not be accurate or complete and are subject to change without notice. 
As such, Orchard App, Inc. (including its subsidiaries and affiliates, 
"Orchard") makes no representation regarding the accuracy or completeness 
of the information contained herein. The intended recipient is advised to 
consult its own professional advisors, including those specializing in 
legal, tax and accounting matters. Orchard does not provide legal, tax or 
accounting advice.

Re: Mesos masters and zookeeper running together?

[Adam Bordelon]

If you're only using ZK for Mesos, then it should be fine to have it
on the same nodes as the Mesos masters, since Mesos only uses ZK for
leader election, and not any other metadata storage. Marathon and
other services, however, use ZK more heavily, and if you're seeing
significant ZK load, it would be better to put ZK on a separate
cluster. Alternatively, you could have 1 ZK cluster just for Mesos,
co-located with the masters, and a separate ZK cluster (or multiple)
for your other services.

On Thu, Dec 24, 2015 at 9:39 PM, Rodrick Brown  wrote:
> 3x - r3.xlarge  @ 4x vCPU 30GB Mem 1x80GB SSD
>>
>> On Dec 24 2015, at 6:01 pm, craig w  wrote:
>>
>> What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.
>>
>> On Dec 24, 2015 5:13 PM, "Dick Davies"  wrote:
>>
>> zookeeper really wants a dedicated cluster IMO; preferably with SSD
>> under it - if zookeeper
>> starts to run slow then everything else will start to bog down. I've
>> co-hosted it with mesos masters
>> before now for demo purposes etc. but for production it's probably
>> worth choosing dedicated hosts.
>>
>> On 24 December 2015 at 20:36, Rodrick Brown 
>> wrote:
>> > With our design we end up building out a stand alone zookeeper cluster 3
>> > nodes.  Zookeeper seems to be the default dumping ground for many Apache
>> > based products these days. You will eventually see many services and
>> > frameworks require a zk instance for leader election, coordination, Kv
>> > store
>> > etc.. I've seen situations where the masters can become extremely busy
>> > and
>> > cause performance problem with Zk which can be huge issue for mesos.
>> >
>> > Sent from Outlook Mobile
>> >
>> >
>> >
>> >
>> > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" 
>> > wrote:
>> >
>> >> Hello, I've been working on setting up a mesos cluster for eventual
>> >> production use and I have a question on configuring zookeeper alongside
>> >> the mesos masters.
>> >> Is it best practice to run zookeeper/exhibitor as a separate cluster
>> >> (in
>> >> our case, three nodes) or on the same machines as the mesos masters?  I
>> >> understand the drawbacks of increased cost for compute resources that
>> >> will just be running a single service and most of the reference docs
>> >> have them running together, but just wondering if it's beneficial to
>> >> have them uncoupled.
>> >>
>> >> Thanks in advance for any input.
>> >>
>> >> Ron Lipke
>> >> @neverminding
>> >
>> >
>> > NOTICE TO RECIPIENTS: This communication is confidential and intended
>> > for
>> > the use of the addressee only. If you are not an intended recipient of
>> > this
>> > communication, please delete it immediately and notify the sender by
>> > return
>> > email. Unauthorized reading, dissemination, distribution or copying of
>> > this
>> > communication is prohibited. This communication does not constitute an
>> > offer
>> > to sell or a solicitation of an indication of interest to purchase any
>> > loan,
>> > security or any other financial product or instrument, nor is it an
>> > offer to
>> > sell or a solicitation of an indication of interest to purchase any
>> > products
>> > or services to any persons who are prohibited from receiving such
>> > information under applicable law. The contents of this communication may
>> > not
>> > be accurate or complete and are subject to change without notice. As
>> > such,
>> > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
>> > makes no representation regarding the accuracy or completeness of the
>> > information contained herein. The intended recipient is advised to
>> > consult
>> > its own professional advisors, including those specializing in legal,
>> > tax
>> > and accounting matters. Orchard does not provide legal, tax or
>> > accounting
>> > advice.
>
>
> NOTICE TO RECIPIENTS: This communication is confidential and intended for
> the use of the addressee only. If you are not an intended recipient of this
> communication, please delete it immediately and notify the sender by return
> email. Unauthorized reading, dissemination, distribution or copying of this
> communication is prohibited. This communication does not constitute an offer
> to sell or a solicitation of an indication of interest to purchase any loan,
> security or any other financial product or instrument, nor is it an offer to
> sell or a solicitation of an indication of interest to purchase any products
> or services to any persons who are prohibited from receiving such
> information under applicable law. The contents of this communication may not
> be accurate or complete and are subject to change without notice. As such,
> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
> makes no representation regarding the accuracy or completeness of the
> information contained herein. The intended recipient is advised to consult
> its own professional advisors, including those specializing in legal, tax
> and accounting matters. Orchard does not p