Repository: samza Updated Branches: refs/heads/master 7eac1b98a -> 76dc4cdd4
SAMZA-1924: Documentation for case-studies and release-notes This PR covers the following - Documentation work for case-studies (Slack and Linkedin use-case) - Documentation work for past release notes - Assigns priority to case studies - Removes unnecessary files from case-studies and release-notes - Adds assets for case studies.    Author: bharathkk <codin.mart...@gmail.com> Reviewers: Jagadish<jagad...@apache.org> Closes #674 from bharathkk/documentation-1.0 Project: http://git-wip-us.apache.org/repos/asf/samza/repo Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/76dc4cdd Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/76dc4cdd Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/76dc4cdd Branch: refs/heads/master Commit: 76dc4cdd43df7694a89f4c2856b8b1d9583c87e3 Parents: 7eac1b9 Author: bharathkk <codin.mart...@gmail.com> Authored: Fri Sep 28 14:19:31 2018 -0700 Committer: Jagadish <jvenkatra...@linkedin.com> Committed: Fri Sep 28 14:19:31 2018 -0700 ---------------------------------------------------------------------- docs/_case-studies/digitalsmiths.md | 1 + docs/_case-studies/fortscale.md | 1 + docs/_case-studies/intuit.md | 1 + docs/_case-studies/linkedin.md | 27 ++++++++- docs/_case-studies/netflix.md | 30 ---------- docs/_case-studies/optimizely.md | 1 + docs/_case-studies/redfin.md | 1 + docs/_case-studies/slack.md | 55 +++++++++++++++++++ docs/_case-studies/state.md | 31 ----------- docs/_case-studies/uber.md | 30 ---------- docs/_config.yml | 2 +- docs/_layouts/case-study.html | 9 ++- docs/_layouts/default.html | 5 +- docs/_layouts/page.html | 2 +- docs/_menu/index.html | 10 ++-- docs/_releases/0.10.md | 25 ++++++++- docs/_releases/0.11.md | 21 ++++++- docs/_releases/0.12.md | 28 +++++++++- docs/_releases/0.13.md | 30 +++++++++- docs/_releases/0.14.md | 32 ++++++++++- docs/_releases/0.7.0.md | 21 ------- docs/_releases/0.8.md | 21 ------- docs/_releases/0.9.md | 21 ------- .../linkedin-atc-samza-pipeline.png | Bin 0 -> 77325 bytes .../case-studies/slack-samza-pipeline.png | Bin 0 -> 108380 bytes 25 files changed, 228 insertions(+), 177 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/digitalsmiths.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/digitalsmiths.md b/docs/_case-studies/digitalsmiths.md index 339a4f0..fcab0fc 100644 --- a/docs/_case-studies/digitalsmiths.md +++ b/docs/_case-studies/digitalsmiths.md @@ -3,6 +3,7 @@ layout: case-study # the layout to use hide_title: true # so we have control in case-study layout, but can still use page title: Totally awesome use-case of samza by DigitalSmiths # title of case study page study_domain: digitalsmiths.com # just the domain, not the protocol +priority: 7 menu_title: DigitalSmiths # what shows up in the menu excerpt_separator: <!--more--> --- http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/fortscale.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/fortscale.md b/docs/_case-studies/fortscale.md index 38ae4d9..fce8115 100644 --- a/docs/_case-studies/fortscale.md +++ b/docs/_case-studies/fortscale.md @@ -3,6 +3,7 @@ layout: case-study # the layout to use hide_title: true # so we have control in case-study layout, but can still use page title: Totally awesome use-case of samza by FortScale # title of case study page study_domain: fortscale.com # just the domain, not the protocol +priority: 6 menu_title: FortScale # what shows up in the menu excerpt_separator: <!--more--> --- http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/intuit.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/intuit.md b/docs/_case-studies/intuit.md index 6aff6cf..39f133e 100644 --- a/docs/_case-studies/intuit.md +++ b/docs/_case-studies/intuit.md @@ -3,6 +3,7 @@ layout: case-study # the layout to use hide_title: true # so we have control in case-study layout, but can still use page title: Totally awesome use-case of samza by Intuit # title of case study page study_domain: intuit.com # just the domain, not the protocol +priority: 5 menu_title: Intuit # what shows up in the menu excerpt_separator: <!--more--> --- http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/linkedin.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/linkedin.md b/docs/_case-studies/linkedin.md index 3c35b6b..a300172 100644 --- a/docs/_case-studies/linkedin.md +++ b/docs/_case-studies/linkedin.md @@ -1,8 +1,9 @@ --- layout: case-study # the layout to use hide_title: true # so we have control in case-study layout, but can still use page -title: Totally awesome use-case of samza by LinkedIn # title of case study page +title: Air Traffic Controller with Samza at LinkedIn study_domain: linkedin.com # just the domain, not the protocol +priority: 4 menu_title: LinkedIn # what shows up in the menu excerpt_separator: <!--more--> --- @@ -27,4 +28,26 @@ Testing the excerpt <!--more--> -Markdown content goes here \ No newline at end of file +LinkedIn is a professional networking company that offers various services and platform for job seekers, employers and sales professionals. With a growing user base and multiple product offerings, it becomes imperative to streamline and standardize our communications to the users. In order to ensure member experience comes first before individual product metrics, LinkedIn developed a new email and notifications platform called *Air Traffic Controller*. + +ATC is an intelligent platform, that is capable of tracking all the outgoing communications to the user and delivering the communication through the right channel to the right member at the right time. + +<img src="/img/{{site.version}}/case-studies/linkedin-atc-samza-pipeline.png" alt="architecture" style="max-width: 80%; height: auto;" onclick="window.open(this.src)"/> + +It has a three main components, + +- **Partitioner**: Partition communication requests, metrics based on user +- **Pipeline**: Handle partitioned communication requests which performs aggregation and consults with the relevance model to determine delivery time +- **Relevance processor**: Provide insights on how relevant is the content to the user, the right delivery time, etc. + +ATC, leverages Samza extensively and uses a lot of features including but not limited to: + +- **Stateful processing**: The ML models in the relevance module are stored locally in RocksDb which are updated realtime time based on user feedback. +- **Async APIs and Multi-threading**: Samzaâs multi-threading and Async APIs allows ATC to perform remote calls with high-throughput. This helps bring down the 90th percentile (P90) end-to-end latency for end to end latency for push notifications from about 12 seconds to about 1.5 seconds. +- **Host affinity**: Co-location of local state stores along with host awareness helps ATC to achieve zero downtime and instant recovery. + +Key Samza Features: *Stateful processing*, *Async API*, *Host affinity* + +More information + +- [Sending less email is just the beginning](https://blog.linkedin.com/2015/11/10/sending-less-email-is-just-the-beginning) http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/netflix.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/netflix.md b/docs/_case-studies/netflix.md deleted file mode 100644 index 7f3cb3b..0000000 --- a/docs/_case-studies/netflix.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -layout: case-study # the layout to use -hide_title: true # so we have control in case-study layout, but can still use page -title: Totally awesome use-case of samza by Netflix # title of case study page -study_domain: netflix.com # just the domain, not the protocol -menu_title: Netflix # what shows up in the menu -excerpt_separator: <!--more--> ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - -Testing the excerpt - -<!--more--> - -Markdown content goes here \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/optimizely.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/optimizely.md b/docs/_case-studies/optimizely.md index 6c3e240..a681f64 100644 --- a/docs/_case-studies/optimizely.md +++ b/docs/_case-studies/optimizely.md @@ -3,6 +3,7 @@ layout: case-study hide_title: true # so we have control in case-study layout, but can still use page title: Real Time Session Aggregation study_domain: optimizely.com +priority: 2 menu_title: Optimizely excerpt_separator: <!--more--> --- http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/redfin.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/redfin.md b/docs/_case-studies/redfin.md index ba03582..fa7b4df 100644 --- a/docs/_case-studies/redfin.md +++ b/docs/_case-studies/redfin.md @@ -3,6 +3,7 @@ layout: case-study # the layout to use hide_title: true # so we have control in case-study layout, but can still use page title: Realtime Notifications at Redfin study_domain: redfin.com # just the domain, not the protocol +priority: 3 menu_title: Redfin # what shows up in the menu excerpt_separator: <!--more--> --- http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/slack.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/slack.md b/docs/_case-studies/slack.md new file mode 100644 index 0000000..d9f59f6 --- /dev/null +++ b/docs/_case-studies/slack.md @@ -0,0 +1,55 @@ +--- +layout: case-study # the layout to use +hide_title: true # so we have control in case-study layout, but can still use page +title: Building streaming data pipelines for monitoring and analytics at Slack # title of case study page +study_domain: slack.com # just the domain, not the protocol +priority: 1 +menu_title: Slack # what shows up in the menu +excerpt_separator: <!--more--> +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Testing the excerpt + +<!--more--> + +Slack is a cloud based company that offers collaboration tools and services to increase productivity. With a rapidly growing user base, and a daily active users north of 8 million, there is an imminent need to react quickly to issues and proactively monitor the health of the application. With a lack of existing monitoring solution, the team went on to build a new data pipeline with the following requirements + +- Near realtime alerting +- Fault tolerant and high throughput data pipeline +- Process billions of metric data, logs and derive timely insights on the health of application +- Extend the pipeline to other use cases such as experimentation, performance etc. + +<img src="/img/{{site.version}}/case-studies/slack-samza-pipeline.png" alt="architecture" style="max-width: 80%; height: auto;" onclick="window.open(this.src)"/> + +The engineering team built a data platform using Apache Samza. It has three main components, + +- **Router**: Deserialize Kafka events and add instrumentation +- **Processor**: Registers with the routers to process subset of message types and performs aggregation +- **Converter**: Enrich the processed data before piping the data to analytics store. + +The clients and backend servers channels the logs and exceptions through Kafka to content routers a.k.a samza partitioners. The partitioned data then flows through processors where it is stored in RocksDb before being joined with other metrics data. The enriched data is stored in druid which powers analytics queries and also acts as a trigger to alert slackbot notifications. + +Other notable use case includes experimentation framework that leverage the data pipeline to track the results of A/B testing in near realtime. The metrics data is joined with the exposure table (members part of the experiment) to derive insights on the experiment. The periodic snapshots of RocksDb is also used to perform data quality check with the batch pipeline. + +Key Samza Features: *Stateful processing*, *Join*, *Windowing* + +More information + +- [Talk: Streaming data pipelines at Slack](https://www.youtube.com/watch?v=wbS1P9ehgd0) +- [Slides: Streaming data pipelines at Slack](https://speakerdeck.com/vananth22/streaming-data-pipelines-at-slack) http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/state.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/state.md b/docs/_case-studies/state.md deleted file mode 100644 index 419620e..0000000 --- a/docs/_case-studies/state.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -layout: case-study # the layout to use -hide_title: true # so we have control in case-study layout, but can still use page -title: Totally awesome use-case of samza by State # title of case study page -study_domain: state.com # just the domain, not the protocol -menu_title: State # what shows up in the menu -excerpt_separator: <!--more--> ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - -Testing the excerpt - -<!--more--> - - -Markdown content goes here \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_case-studies/uber.md ---------------------------------------------------------------------- diff --git a/docs/_case-studies/uber.md b/docs/_case-studies/uber.md deleted file mode 100644 index 64e7a4f..0000000 --- a/docs/_case-studies/uber.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -layout: case-study # the layout to use -hide_title: true # so we have control in case-study layout, but can still use page -title: Totally awesome use-case of samza by Uber # title of case study page -study_domain: uber.com # just the domain, not the protocol -menu_title: Uber # what shows up in the menu -excerpt_separator: <!--more--> ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - -Testing the excerpt - -<!--more--> - -Markdown content goes here \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_config.yml ---------------------------------------------------------------------- diff --git a/docs/_config.yml b/docs/_config.yml index ad9c3cf..15e0c19 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -40,6 +40,6 @@ collections: committers: output: false releases: - output: false + output: true powered-by: output: false \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_layouts/case-study.html ---------------------------------------------------------------------- diff --git a/docs/_layouts/case-study.html b/docs/_layouts/case-study.html index a8b85a3..732448a 100644 --- a/docs/_layouts/case-study.html +++ b/docs/_layouts/case-study.html @@ -116,8 +116,8 @@ <div class="case-studies"> <div class="case-studies-list"> - - {% for study in site.case-studies %} + {% assign case-studies_sorted = site.case-studies | sort: 'priority' %} + {% for study in case-studies_sorted %} {% if study.exclude_from_loop or study.url == page.url %} {% continue %} @@ -128,9 +128,8 @@ <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/{{ study.study_domain}}?size=512');"></div> </a> </div> - - {% endfor %} - + {% endfor %} + </div> </div> http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_layouts/default.html ---------------------------------------------------------------------- diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html index 339e8a6..a176c4d 100644 --- a/docs/_layouts/default.html +++ b/docs/_layouts/default.html @@ -159,7 +159,8 @@ <div class="section__items"> - {% for study in site.case-studies %} + {% assign case-studies_sorted = site.case-studies | sort: 'priority' %} + {% for study in case-studies_sorted %} {% if study.exclude_from_loop %} {% continue %} @@ -170,7 +171,6 @@ <div class="section__item-logo-bg" style="background-image: url('https://logo.clearbit.com/{{ study.study_domain}}?size=256');"></div> </a> </div> - {% endfor %} </div> @@ -180,7 +180,6 @@ <!-- footer starts here --> {% include footer.html %} - </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_layouts/page.html ---------------------------------------------------------------------- diff --git a/docs/_layouts/page.html b/docs/_layouts/page.html index 0b1bfa6..edb2087 100644 --- a/docs/_layouts/page.html +++ b/docs/_layouts/page.html @@ -68,7 +68,7 @@ <h2>{{ page.title }}</h2> {% endif %} - {% if page.url contains "/learn/documentation" or page.url contains "/releases/" %} + {% if page.url contains "/learn/documentation" %} <div class="releases-list-container"> http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_menu/index.html ---------------------------------------------------------------------- diff --git a/docs/_menu/index.html b/docs/_menu/index.html index 681dffd..03444ce 100644 --- a/docs/_menu/index.html +++ b/docs/_menu/index.html @@ -12,8 +12,6 @@ items: items_attributes: 'data-documentation="/learn/documentation/version/"' - menu_title: Releases get_items: releases - map_title: version - map_url: link sort: 'order' reverse: true - menu_title: Blog @@ -22,19 +20,20 @@ items: items: - menu_title: Contact Us url: '/community/mailing-lists.html' - - menu_title: Contributor's Corner + - menu_title: Contributor's Corner url: '/contribute/contributors-corner.html' - menu_title: PMC members and committers url: '/community/committers.html' - - menu_title: Talks and Meetups + - menu_title: Talks and Meetups url: '/meetups/' - menu_title: Case Studies get_items: case-studies + sort: 'priority' items: - menu_title: View All url: /case-studies/ - - divider: true + - divider: true --- <!-- Licensed to the Apache Software Foundation (ASF) under one or more @@ -69,7 +68,6 @@ items: {% else %} {% assign has_sub = false %} {% endif %} - <!-- Start Group --> {% if has_sub %} <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.10.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.10.md b/docs/_releases/0.10.md index d4e8850..7a86eb2 100644 --- a/docs/_releases/0.10.md +++ b/docs/_releases/0.10.md @@ -1,7 +1,9 @@ --- version: '0.10' order: 10 -link: /learn/documentation/0.10/introduction/background.html +layout: page +menu_title: 0.10 +title: Apache Samza 0.10 --- <!-- Licensed to the Apache Software Foundation (ASF) under one or more @@ -19,3 +21,24 @@ link: /learn/documentation/0.10/introduction/background.html See the License for the specific language governing permissions and limitations under the License. --> + +We are excited to announce that the much awaited **Apache Samza 0.10.0** has been released. It is a major release with new features; notably HDFS and ElasticSearch support, broadcast streams. + +####Enhancements and Bug Fixes +Overall, [130 JIRAs](https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20updated%20DESC) were resolved in this release. Here are few highlights + +- [SAMZA-348](https://issues.apache.org/jira/browse/SAMZA-348) Add Coordinator Stream to support large and dynamic configuration in a Samza job +- [SAMZA-704](https://issues.apache.org/jira/browse/SAMZA-704) Command-line tool to write to the Coordinator Stream +- [SAMZA-676](https://issues.apache.org/jira/browse/SAMZA-676) Added support for Broadcast Stream +- [SAMZA-617](https://issues.apache.org/jira/browse/SAMZA-617) Add host affinity support for stateful jobs in YARN +- [SAMZA-537](https://issues.apache.org/jira/browse/SAMZA-537) Support for TTL for RocksDB stores +- [SAMZA-693](https://issues.apache.org/jira/browse/SAMZA-693) Add HDFS system producer +- [SAMZA-654](https://issues.apache.org/jira/browse/SAMZA-654) Add ElasticSearch system producer +- [SAMZA-598](https://issues.apache.org/jira/browse/SAMZA-598) Create CLI tool to query RocksDB stores +- [SAMZA-798](https://issues.apache.org/jira/browse/SAMZA-798) Fix performance issues related to merging checkpoint and coordinator stream +- [SAMZA-754](https://issues.apache.org/jira/browse/SAMZA-754) Fix race condition in job runner between coordinator consumer and producer + +####Sources and Artifacts +[Samza-sources-0.10.0.tgz](http://www.apache.org/dyn/closer.cgi/samza/0.10.0) + +For more details about this release, please check out the [release blog post](https://blogs.apache.org/samza/). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.11.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.11.md b/docs/_releases/0.11.md index 6f21b05..f985271 100644 --- a/docs/_releases/0.11.md +++ b/docs/_releases/0.11.md @@ -1,7 +1,9 @@ --- version: '0.11' order: 11 -link: /learn/documentation/0.11/introduction/background.html +layout: page +menu_title: 0.11 +title: Apache Samza 0.11 --- <!-- Licensed to the Apache Software Foundation (ASF) under one or more @@ -19,3 +21,20 @@ link: /learn/documentation/0.11/introduction/background.html See the License for the specific language governing permissions and limitations under the License. --> + +We are excited to announce that the **Apache Samza 0.11.0** has been released. It is a major release with several large improvements in runtime performance, operational stability and ease of use. + +####Enhancements and Bug Fixes +Overall, [37 JIRAs](https://issues.apache.org/jira/browse/SAMZA-1010?jql=project%20%3D%20SAMZA%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20in%20(0.11%2C%200.11.0)) were resolved in this release. Here are few highlights + +- [SAMZA-863](https://issues.apache.org/jira/browse/SAMZA-863) Add multi-threading support +- [SAMZA-849](https://issues.apache.org/jira/browse/SAMZA-849) Add split deployment to Samza framework +- [SAMZA-865](https://issues.apache.org/jira/browse/SAMZA-865) Introduce rest API to start and stop jobs +- [SAMZA-924](https://issues.apache.org/jira/browse/SAMZA-924) Add disk space monitoring +- [SAMZA-972](https://issues.apache.org/jira/browse/SAMZA-972) Add holistic monitoring of memory in Samza applications +- [SAMZA-963](https://issues.apache.org/jira/browse/SAMZA-963) Add performance metrics for accessing state + +####Sources and Artifacts +[Samza-sources-0.11.0.tgz](http://www.apache.org/dyn/closer.cgi/samza/0.11.0) + +For more details about this release, please check out the [release blog post](https://blogs.apache.org/samza/). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.12.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.12.md b/docs/_releases/0.12.md index bd9eb3b..e6ea2bb 100644 --- a/docs/_releases/0.12.md +++ b/docs/_releases/0.12.md @@ -1,7 +1,9 @@ --- version: '0.12' order: 12 -link: /learn/documentation/0.12/introduction/background.html +layout: page +menu_title: 0.12 +title: Apache Samza 0.12 --- <!-- Licensed to the Apache Software Foundation (ASF) under one or more @@ -19,3 +21,27 @@ link: /learn/documentation/0.12/introduction/background.html See the License for the specific language governing permissions and limitations under the License. --> + +We are excited to announce that the **Apache Samza 0.12.0** has been released. It is a major release with several features; notably batch processing and enhancements to improve stability, performance and ease of use. + +####Enhancements and Bug Fixes +Overall, [30 JIRAs](https://issues.apache.org/jira/browse/SAMZA-469?jql=project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.12.0%20%20AND%20status%20%3D%20Resolved%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC) were resolve in this release. Here are few highlights + +- [SAMZA-967](https://issues.apache.org/jira/browse/SAMZA-967) Add HDFS system consumer +- [SAMZA-1042](https://issues.apache.org/jira/browse/SAMZA-1042) Allow offset notifications for input systems +- [SAMZA-1013](https://issues.apache.org/jira/browse/SAMZA-1013) Support YARN node labels +- [SAMZA-1079](https://issues.apache.org/jira/browse/SAMZA-1079) Add timeout for blocking reads when localizing containers +- [SAMZA-1077](https://issues.apache.org/jira/browse/SAMZA-1077) Handle all exceptions and errors in container +- [SAMZA-1069](https://issues.apache.org/jira/browse/SAMZA-1069) Deadlock between kafka system producer and kafka producer from kafka-clients lib +- [SAMZA-1065](https://issues.apache.org/jira/browse/SAMZA-1065) Handle commit order correctly to guarantee at least once processing when using local store for dedup + +####Upgrade Notes +- Upgraded Kafka version to 0.10. This enables us to take advantage of the critical fixes and improvements in Kafka. +- Upgraded to Jetty 9 from Jetty 8. +- Full support for Scala 2.11. All Samza jars will now have the scala version as 2.11 as a part of their file name. For example, samza-yarn_2.11-0.12.jar. +- Samza is now source compatible with JDK 8 and above. Older JDKs are no longer supported. + +####Sources and Artifacts +[Samza-sources-0.12.0.tgz](http://www.apache.org/dyn/closer.cgi/samza/0.12.0) + +For more details about this release, please check out the [release blog post](https://blogs.apache.org/samza/). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.13.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.13.md b/docs/_releases/0.13.md index 65e64f0..a7993f4 100644 --- a/docs/_releases/0.13.md +++ b/docs/_releases/0.13.md @@ -1,7 +1,9 @@ --- version: '0.13' order: 13 -link: /learn/documentation/0.13/introduction/background.html +layout: page +menu_title: 0.13 +title: Apache Samza 0.13 --- <!-- Licensed to the Apache Software Foundation (ASF) under one or more @@ -19,3 +21,29 @@ link: /learn/documentation/0.13/introduction/background.html See the License for the specific language governing permissions and limitations under the License. --> + +We are very excited to announce the release of **Apache Samza 0.13.0**. It contains previews for the following highly anticipated features; notably new [programming model](http://samza.apache.org/startup/preview/#high-level-api) and [deployment model](http://samza.apache.org/startup/preview/#flexible-deployment-model). + +####Enhancements and Bug Fixes +Overall, [150 JIRAs](https://issues.apache.org/jira/browse/SAMZA-1110?jql=project%20%3D%2012314526%20AND%20fixVersion%20%3D%200.13.0%20%20AND%20status%20%3D%20Resolved%20%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC) were resolve in this release. Here are few highlights + +- [SAMZA-871](https://issues.apache.org/jira/browse/SAMZA-871) Adds container heartbeat +- [SAMZA-1140](https://issues.apache.org/jira/browse/SAMZA-1140) Enable async commit +- [SAMZA-1143](https://issues.apache.org/jira/browse/SAMZA-1143) Add configurations for localizing arbitrary resources in YARN. +- [SAMZA-1154](https://issues.apache.org/jira/browse/SAMZA-1154) Add new endpoint to samza-rest to get information about all tasks in a job +- [SAMZA-1158](https://issues.apache.org/jira/browse/SAMZA-1158) Add samza-rest monitor to clean up stale local stores +- [SAMZA-1083](https://issues.apache.org/jira/browse/SAMZA-1083) Handle task stores that are older than delete tombstones correctly +- [SAMZA-1100](https://issues.apache.org/jira/browse/SAMZA-1100) Handle empty stream for both bootstrap and broadcast scenarios +- [SAMZA-1112](https://issues.apache.org/jira/browse/SAMZA-1112) Fix broker proxy to log fatal errors +- [SAMZA-1121](https://issues.apache.org/jira/browse/SAMZA-1121) Fix stream appender to prevent propagating exceptions to the caller +- [SAMZA-1157](https://issues.apache.org/jira/browse/SAMZA-1157) Fix logging for serialization and deserialization errors + +####Upgrade Notes +- Samza now supports Scala 2.12. +- Kafka version to 0.10.1.1. +- Elasticsearch version to 2.2.0 + +####Sources and Artifacts +[Samza-sources-0.13.0.tgz](http://www.apache.org/dyn/closer.cgi/samza/0.13.0) + +For more details about this release, please check out the [release blog post](https://blogs.apache.org/samza/). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.14.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.14.md b/docs/_releases/0.14.md index 8a0ce10..89fa020 100644 --- a/docs/_releases/0.14.md +++ b/docs/_releases/0.14.md @@ -1,7 +1,9 @@ --- version: '0.14' order: 14 -link: /learn/documentation/0.14/introduction/background.html +layout: page +menu_title: 0.14 +title: Apache Samza 0.14 --- <!-- Licensed to the Apache Software Foundation (ASF) under one or more @@ -19,3 +21,31 @@ link: /learn/documentation/0.14/introduction/background.html See the License for the specific language governing permissions and limitations under the License. --> + +We are very excited to announce the release of **Apache Samza 0.14.0**. It is a major release with highly anticipated features viz Samza SQL, Azure EventHubs support and AWS Kinesis consumer. + +####Enhancements and Bug Fixes +Overall, [65 JIRAs](https://issues.apache.org/jira/browse/SAMZA-1109?jql=project%20%3D%20SAMZA%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.14.0%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC) were resolved in this release. Here are few highlights + +- [SAMZA-1510](https://issues.apache.org/jira/browse/SAMZA-1510) Introduce SQL semantics to Samza +- [SAMZA-1438](https://issues.apache.org/jira/browse/SAMZA-1438) Implement Producer and consumer for Azure EventHubs +- [SAMZA-1515](https://issues.apache.org/jira/browse/SAMZA-1515) Implement Kinesis consumer +- [SAMZA-1486](https://issues.apache.org/jira/browse/SAMZA-1486) Checkpoint provider for Azure tables +- [SAMZA-1421](https://issues.apache.org/jira/browse/SAMZA-1421) Support for durable state in high-level API +- [SAMZA-1392](https://issues.apache.org/jira/browse/SAMZA-1392) Fix performance and correctness issues with concurrent sends and flushes in kafka system producer +- [SAMZA-1406](https://issues.apache.org/jira/browse/SAMZA-1406) Enhancements to the Zookeeper based deployment model +- [SAMZA-1321](https://issues.apache.org/jira/browse/SAMZA-1321) Support for multi-stage batch processing + +####Upgrade Notes +- Introduced a new mandatory configuration - `job.coordination.utils.factory`. It impacts applications using non-YARN deployment models. Read more about it [here](/learn/documentation/{{site.version}}/jobs/configuration-table.html). +- The following APIs in SystemAdmin have been deprecated in the previous versions and hence, replaced with newer APIs. If you have a custom System implementation, then you have to update to the newer APIs. + - `void createChangelogStream(String streamName, int numOfPartitions); -> boolean createStream(StreamSpec streamSpec);` + - `void createCoordinatorStream(String streamName); -> boolean createStream(StreamSpec streamSpec);` + - `void validateChangelogStream(String streamName, int numOfPartitions); -> void validateStream(StreamSpec streamSpec) throws StreamValidationException;` +- New API has been added to SystemAdmin that clear a stream. + - `boolean clearStream(StreamSpec streamSpec);` Read more about it in the [API docs](/learn/documentation/{{site.version}}/api/javadocs/org/apache/samza/system/SystemAdmin.html). + +####Sources and Artifacts +[Samza-sources-0.14.tgz](http://www.apache.org/dyn/closer.cgi/samza/0.14.0) + +For more details about this release, please check out the [release blog post](https://blogs.apache.org/samza/). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.7.0.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.7.0.md b/docs/_releases/0.7.0.md deleted file mode 100644 index a46019e..0000000 --- a/docs/_releases/0.7.0.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -version: '0.7.0' -order: 7 -link: /learn/documentation/0.7.0/introduction/background.html ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.8.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.8.md b/docs/_releases/0.8.md deleted file mode 100644 index 3438356..0000000 --- a/docs/_releases/0.8.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -version: '0.8' -order: 8 -link: /learn/documentation/0.8/introduction/background.html ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/_releases/0.9.md ---------------------------------------------------------------------- diff --git a/docs/_releases/0.9.md b/docs/_releases/0.9.md deleted file mode 100644 index 71f3659..0000000 --- a/docs/_releases/0.9.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -version: '0.9' -order: 9 -link: /learn/documentation/0.9/introduction/background.html ---- -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/img/versioned/case-studies/linkedin-atc-samza-pipeline.png ---------------------------------------------------------------------- diff --git a/docs/img/versioned/case-studies/linkedin-atc-samza-pipeline.png b/docs/img/versioned/case-studies/linkedin-atc-samza-pipeline.png new file mode 100644 index 0000000..fc9a54e Binary files /dev/null and b/docs/img/versioned/case-studies/linkedin-atc-samza-pipeline.png differ http://git-wip-us.apache.org/repos/asf/samza/blob/76dc4cdd/docs/img/versioned/case-studies/slack-samza-pipeline.png ---------------------------------------------------------------------- diff --git a/docs/img/versioned/case-studies/slack-samza-pipeline.png b/docs/img/versioned/case-studies/slack-samza-pipeline.png new file mode 100644 index 0000000..aaae73a Binary files /dev/null and b/docs/img/versioned/case-studies/slack-samza-pipeline.png differ
