Hi Dragos,With this contribution, the cold start is around 25 seconds. That
number could go much lower, and I hope that this contribution will open the
door for a deeper integration between OpenWhisk and Hadoop. I am still looking
for numbers on the most recent work, but Microsoft reported the following
metrics at Hadoop Summit 2017:
https://www.slideshare.net/Hadoop_Summit/lessons-learned-from-scaling-yarn-to-40k-machines-in-a-multi-tenancy-environment
Cluster size: 4,000 nodesSustained load: less than 3000 QPS
Allocation/Promotion latency @ 95th percentile: less than 3sRack locality:
99.84%
Note: these containers are most likely JVMs not docker containers, but they are
cold starts. Without a data locality requirement, it should be even faster, but
locality is import for data heavy use cases. It would be great to add an HDFS
feed and support for data-local actions.
-Sam
On Monday, February 25, 2019, 4:16:06 PM PST, Dascalita Dragos
<[email protected]> wrote:
Hi Samuel,
This is an interesting contribution. Do you happen to have any performance
numbers with YARN ? I'd be particularly interested in the cold start
latencies.
Thanks,
dragos
On Fri, Feb 22, 2019 at 5:21 PM Samuel Hjelmfelt
<[email protected]> wrote:
>
> Hi Rodric and Carlos,
>
>
> ApacheHadoop has three major components: HDFS (distributed filesystem),
> MapReduce(distributed batch processing engine), YARN (Yet Another Resource
> Negotiator) (containerengine). While MapReduce has been largely replaced by
> Apache Tez, Apache Spark,and Apache Flink, HDFS and YARN are still widely
> used for data analytics use cases.
>
>
>
> YARN is unique as a container engine because, unlike Mesos and Kubernetes,
> it was designed for ephemeral, short-livedcontainers rather than for long
> running micro-services. The jobs and queries that run on YARN are split
> intosmall tasks that run to completion and generally only last for seconds
> or maybe minutes. Overthe last couple years, YARN has been expanding its
> support for long running usecases, but is still focused on data-driven use
> cases over more generic micro-serviceuse cases (like web apps). The primary
> long running technologies on YARN are currently Spark Streamingand
> TensorFlow. Here is an articlefrom LinkedIn about why they created a
> project for TensorFlow on YARN. Asimilar case could be made for OpenWhisk:
> https://engineering.linkedin.com/blog/2018/09/open-sourcing-tony--native-support-of-tensorflow-on-hadoop.
>
>
>
>
> Bringing OpenWhisk onto YARN makes FaaS more accessible to thethousands of
> organizations with existing Hadoop clusters. Between Cloudera’s 2,000+
> customers; Azure, AWS,and GCP cloud customers; and the organizations
> self-supporting like Netflix, theinstall base of YARN is very high and
> still growing.
>
>
>
> ThisPR is a first level of integration, but YARN’s focus on ephemeral
> containerscould be more fully leveraged by OpenWhisk to improve scalability
> andperformance. Here is an interesting article on the scalability of YARN
> fromMicrosoft:
> https://azure.microsoft.com/en-us/blog/how-microsoft-drives-exabyte-analytics-on-the-world-s-largest-yarn-cluster/
>
> Thanks,
> Sam Hjelmfelt
>
