Repository: kafka-site Updated Branches: refs/heads/asf-site 4ed4dd1bd -> 46fccbe1e
http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/kafka/javaapi/consumer/package-summary.html ---------------------------------------------------------------------- diff --git a/0100/javadoc/kafka/javaapi/consumer/package-summary.html b/0100/javadoc/kafka/javaapi/consumer/package-summary.html new file mode 100644 index 0000000..4b5539a --- /dev/null +++ b/0100/javadoc/kafka/javaapi/consumer/package-summary.html @@ -0,0 +1,136 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";> +<!-- NewPage --> +<html lang="en"> +<head> +<!-- Generated by javadoc (version 1.7.0_79) on Mon Mar 28 13:06:16 PDT 2016 --> +<title>kafka.javaapi.consumer (core 0.10.0.0 API)</title> +<meta name="date" content="2016-03-28"> +<link rel="stylesheet" type="text/css" href="../../../stylesheet.css" title="Style"> +</head> +<body> +<script type="text/javascript"><!-- + if (location.href.indexOf('is-external=true') == -1) { + parent.document.title="kafka.javaapi.consumer (core 0.10.0.0 API)"; + } +//--> +</script> +<noscript> +<div>JavaScript is disabled on your browser.</div> +</noscript> +<!-- ========= START OF TOP NAVBAR ======= --> +<div class="topNav"><a name="navbar_top"> +<!-- --> +</a><a href="#skip-navbar_top" title="Skip navigation links"></a><a name="navbar_top_firstrow"> +<!-- --> +</a> +<ul class="navList" title="Navigation"> +<li><a href="../../../kafka/javaapi/consumer/package-summary.html">Package</a></li> +<li>Class</li> +<li><a href="package-tree.html">Tree</a></li> +<li><a href="../../../deprecated-list.html">Deprecated</a></li> +<li><a href="../../../index-all.html">Index</a></li> +<li><a href="../../../help-doc.html">Help</a></li> +</ul> +</div> +<div class="subNav"> +<ul class="navList"> +<li>Prev Package</li> +<li>Next Package</li> +</ul> +<ul class="navList"> +<li><a href="../../../index.html?kafka/javaapi/consumer/package-summary.html" target="_top">Frames</a></li> +<li><a href="package-summary.html" target="_top">No Frames</a></li> +</ul> +<ul class="navList" id="allclasses_navbar_top"> +<li><a href="../../../allclasses-noframe.html">All Classes</a></li> +</ul> +<div> +<script type="text/javascript"><!-- + allClassesLink = document.getElementById("allclasses_navbar_top"); + if(window==top) { + allClassesLink.style.display = "block"; + } + else { + allClassesLink.style.display = "none"; + } + //--> +</script> +</div> +<a name="skip-navbar_top"> +<!-- --> +</a></div> +<!-- ========= END OF TOP NAVBAR ========= --> +<div class="header"> +<h1 title="Package" class="title">Package kafka.javaapi.consumer</h1> +</div> +<div class="contentContainer"> +<ul class="blockList"> +<li class="blockList"> +<table class="packageSummary" border="0" cellpadding="3" cellspacing="0" summary="Interface Summary table, listing interfaces, and an explanation"> +<caption><span>Interface Summary</span><span class="tabEnd"> </span></caption> +<tr> +<th class="colFirst" scope="col">Interface</th> +<th class="colLast" scope="col">Description</th> +</tr> +<tbody> +<tr class="altColor"> +<td class="colFirst"><a href="../../../kafka/javaapi/consumer/ConsumerConnector.html" title="interface in kafka.javaapi.consumer">ConsumerConnector</a></td> +<td class="colLast"> </td> +</tr> +<tr class="rowColor"> +<td class="colFirst"><a href="../../../kafka/javaapi/consumer/ConsumerRebalanceListener.html" title="interface in kafka.javaapi.consumer">ConsumerRebalanceListener</a></td> +<td class="colLast"> +<div class="block">This listener is used for execution of tasks defined by user when a consumer rebalance + occurs in <code>ZookeeperConsumerConnector</code></div> +</td> +</tr> +</tbody> +</table> +</li> +</ul> +</div> +<!-- ======= START OF BOTTOM NAVBAR ====== --> +<div class="bottomNav"><a name="navbar_bottom"> +<!-- --> +</a><a href="#skip-navbar_bottom" title="Skip navigation links"></a><a name="navbar_bottom_firstrow"> +<!-- --> +</a> +<ul class="navList" title="Navigation"> +<li><a href="../../../kafka/javaapi/consumer/package-summary.html">Package</a></li> +<li>Class</li> +<li><a href="package-tree.html">Tree</a></li> +<li><a href="../../../deprecated-list.html">Deprecated</a></li> +<li><a href="../../../index-all.html">Index</a></li> +<li><a href="../../../help-doc.html">Help</a></li> +</ul> +</div> +<div class="subNav"> +<ul class="navList"> +<li>Prev Package</li> +<li>Next Package</li> +</ul> +<ul class="navList"> +<li><a href="../../../index.html?kafka/javaapi/consumer/package-summary.html" target="_top">Frames</a></li> +<li><a href="package-summary.html" target="_top">No Frames</a></li> +</ul> +<ul class="navList" id="allclasses_navbar_bottom"> +<li><a href="../../../allclasses-noframe.html">All Classes</a></li> +</ul> +<div> +<script type="text/javascript"><!-- + allClassesLink = document.getElementById("allclasses_navbar_bottom"); + if(window==top) { + allClassesLink.style.display = "block"; + } + else { + allClassesLink.style.display = "none"; + } + //--> +</script> +</div> +<a name="skip-navbar_bottom"> +<!-- --> +</a></div> +<!-- ======== END OF BOTTOM NAVBAR ======= --> +</body> +</html> http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/kafka/javaapi/consumer/package-tree.html ---------------------------------------------------------------------- diff --git a/0100/javadoc/kafka/javaapi/consumer/package-tree.html b/0100/javadoc/kafka/javaapi/consumer/package-tree.html new file mode 100644 index 0000000..c1642a0 --- /dev/null +++ b/0100/javadoc/kafka/javaapi/consumer/package-tree.html @@ -0,0 +1,117 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";> +<!-- NewPage --> +<html lang="en"> +<head> +<!-- Generated by javadoc (version 1.7.0_79) on Mon Mar 28 13:06:16 PDT 2016 --> +<title>kafka.javaapi.consumer Class Hierarchy (core 0.10.0.0 API)</title> +<meta name="date" content="2016-03-28"> +<link rel="stylesheet" type="text/css" href="../../../stylesheet.css" title="Style"> +</head> +<body> +<script type="text/javascript"><!-- + if (location.href.indexOf('is-external=true') == -1) { + parent.document.title="kafka.javaapi.consumer Class Hierarchy (core 0.10.0.0 API)"; + } +//--> +</script> +<noscript> +<div>JavaScript is disabled on your browser.</div> +</noscript> +<!-- ========= START OF TOP NAVBAR ======= --> +<div class="topNav"><a name="navbar_top"> +<!-- --> +</a><a href="#skip-navbar_top" title="Skip navigation links"></a><a name="navbar_top_firstrow"> +<!-- --> +</a> +<ul class="navList" title="Navigation"> +<li><a href="../../../kafka/javaapi/consumer/package-summary.html">Package</a></li> +<li>Class</li> +<li class="navBarCell1Rev">Tree</li> +<li><a href="../../../deprecated-list.html">Deprecated</a></li> +<li><a href="../../../index-all.html">Index</a></li> +<li><a href="../../../help-doc.html">Help</a></li> +</ul> +</div> +<div class="subNav"> +<ul class="navList"> +<li>Prev</li> +<li>Next</li> +</ul> +<ul class="navList"> +<li><a href="../../../index.html?kafka/javaapi/consumer/package-tree.html" target="_top">Frames</a></li> +<li><a href="package-tree.html" target="_top">No Frames</a></li> +</ul> +<ul class="navList" id="allclasses_navbar_top"> +<li><a href="../../../allclasses-noframe.html">All Classes</a></li> +</ul> +<div> +<script type="text/javascript"><!-- + allClassesLink = document.getElementById("allclasses_navbar_top"); + if(window==top) { + allClassesLink.style.display = "block"; + } + else { + allClassesLink.style.display = "none"; + } + //--> +</script> +</div> +<a name="skip-navbar_top"> +<!-- --> +</a></div> +<!-- ========= END OF TOP NAVBAR ========= --> +<div class="header"> +<h1 class="title">Hierarchy For Package kafka.javaapi.consumer</h1> +</div> +<div class="contentContainer"> +<h2 title="Interface Hierarchy">Interface Hierarchy</h2> +<ul> +<li type="circle">kafka.javaapi.consumer.<a href="../../../kafka/javaapi/consumer/ConsumerConnector.html" title="interface in kafka.javaapi.consumer"><span class="strong">ConsumerConnector</span></a></li> +<li type="circle">kafka.javaapi.consumer.<a href="../../../kafka/javaapi/consumer/ConsumerRebalanceListener.html" title="interface in kafka.javaapi.consumer"><span class="strong">ConsumerRebalanceListener</span></a></li> +</ul> +</div> +<!-- ======= START OF BOTTOM NAVBAR ====== --> +<div class="bottomNav"><a name="navbar_bottom"> +<!-- --> +</a><a href="#skip-navbar_bottom" title="Skip navigation links"></a><a name="navbar_bottom_firstrow"> +<!-- --> +</a> +<ul class="navList" title="Navigation"> +<li><a href="../../../kafka/javaapi/consumer/package-summary.html">Package</a></li> +<li>Class</li> +<li class="navBarCell1Rev">Tree</li> +<li><a href="../../../deprecated-list.html">Deprecated</a></li> +<li><a href="../../../index-all.html">Index</a></li> +<li><a href="../../../help-doc.html">Help</a></li> +</ul> +</div> +<div class="subNav"> +<ul class="navList"> +<li>Prev</li> +<li>Next</li> +</ul> +<ul class="navList"> +<li><a href="../../../index.html?kafka/javaapi/consumer/package-tree.html" target="_top">Frames</a></li> +<li><a href="package-tree.html" target="_top">No Frames</a></li> +</ul> +<ul class="navList" id="allclasses_navbar_bottom"> +<li><a href="../../../allclasses-noframe.html">All Classes</a></li> +</ul> +<div> +<script type="text/javascript"><!-- + allClassesLink = document.getElementById("allclasses_navbar_bottom"); + if(window==top) { + allClassesLink.style.display = "block"; + } + else { + allClassesLink.style.display = "none"; + } + //--> +</script> +</div> +<a name="skip-navbar_bottom"> +<!-- --> +</a></div> +<!-- ======== END OF BOTTOM NAVBAR ======= --> +</body> +</html> http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/overview-tree.html ---------------------------------------------------------------------- diff --git a/0100/javadoc/overview-tree.html b/0100/javadoc/overview-tree.html new file mode 100644 index 0000000..762a6e6 --- /dev/null +++ b/0100/javadoc/overview-tree.html @@ -0,0 +1,121 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";> +<!-- NewPage --> +<html lang="en"> +<head> +<!-- Generated by javadoc (version 1.7.0_79) on Mon Mar 28 13:06:16 PDT 2016 --> +<title>Class Hierarchy (core 0.10.0.0 API)</title> +<meta name="date" content="2016-03-28"> +<link rel="stylesheet" type="text/css" href="stylesheet.css" title="Style"> +</head> +<body> +<script type="text/javascript"><!-- + if (location.href.indexOf('is-external=true') == -1) { + parent.document.title="Class Hierarchy (core 0.10.0.0 API)"; + } +//--> +</script> +<noscript> +<div>JavaScript is disabled on your browser.</div> +</noscript> +<!-- ========= START OF TOP NAVBAR ======= --> +<div class="topNav"><a name="navbar_top"> +<!-- --> +</a><a href="#skip-navbar_top" title="Skip navigation links"></a><a name="navbar_top_firstrow"> +<!-- --> +</a> +<ul class="navList" title="Navigation"> +<li><a href="kafka/javaapi/consumer/package-summary.html">Package</a></li> +<li>Class</li> +<li class="navBarCell1Rev">Tree</li> +<li><a href="deprecated-list.html">Deprecated</a></li> +<li><a href="index-all.html">Index</a></li> +<li><a href="help-doc.html">Help</a></li> +</ul> +</div> +<div class="subNav"> +<ul class="navList"> +<li>Prev</li> +<li>Next</li> +</ul> +<ul class="navList"> +<li><a href="index.html?overview-tree.html" target="_top">Frames</a></li> +<li><a href="overview-tree.html" target="_top">No Frames</a></li> +</ul> +<ul class="navList" id="allclasses_navbar_top"> +<li><a href="allclasses-noframe.html">All Classes</a></li> +</ul> +<div> +<script type="text/javascript"><!-- + allClassesLink = document.getElementById("allclasses_navbar_top"); + if(window==top) { + allClassesLink.style.display = "block"; + } + else { + allClassesLink.style.display = "none"; + } + //--> +</script> +</div> +<a name="skip-navbar_top"> +<!-- --> +</a></div> +<!-- ========= END OF TOP NAVBAR ========= --> +<div class="header"> +<h1 class="title">Hierarchy For All Packages</h1> +<span class="strong">Package Hierarchies:</span> +<ul class="horizontal"> +<li><a href="kafka/javaapi/consumer/package-tree.html">kafka.javaapi.consumer</a></li> +</ul> +</div> +<div class="contentContainer"> +<h2 title="Interface Hierarchy">Interface Hierarchy</h2> +<ul> +<li type="circle">kafka.javaapi.consumer.<a href="kafka/javaapi/consumer/ConsumerConnector.html" title="interface in kafka.javaapi.consumer"><span class="strong">ConsumerConnector</span></a></li> +<li type="circle">kafka.javaapi.consumer.<a href="kafka/javaapi/consumer/ConsumerRebalanceListener.html" title="interface in kafka.javaapi.consumer"><span class="strong">ConsumerRebalanceListener</span></a></li> +</ul> +</div> +<!-- ======= START OF BOTTOM NAVBAR ====== --> +<div class="bottomNav"><a name="navbar_bottom"> +<!-- --> +</a><a href="#skip-navbar_bottom" title="Skip navigation links"></a><a name="navbar_bottom_firstrow"> +<!-- --> +</a> +<ul class="navList" title="Navigation"> +<li><a href="kafka/javaapi/consumer/package-summary.html">Package</a></li> +<li>Class</li> +<li class="navBarCell1Rev">Tree</li> +<li><a href="deprecated-list.html">Deprecated</a></li> +<li><a href="index-all.html">Index</a></li> +<li><a href="help-doc.html">Help</a></li> +</ul> +</div> +<div class="subNav"> +<ul class="navList"> +<li>Prev</li> +<li>Next</li> +</ul> +<ul class="navList"> +<li><a href="index.html?overview-tree.html" target="_top">Frames</a></li> +<li><a href="overview-tree.html" target="_top">No Frames</a></li> +</ul> +<ul class="navList" id="allclasses_navbar_bottom"> +<li><a href="allclasses-noframe.html">All Classes</a></li> +</ul> +<div> +<script type="text/javascript"><!-- + allClassesLink = document.getElementById("allclasses_navbar_bottom"); + if(window==top) { + allClassesLink.style.display = "block"; + } + else { + allClassesLink.style.display = "none"; + } + //--> +</script> +</div> +<a name="skip-navbar_bottom"> +<!-- --> +</a></div> +<!-- ======== END OF BOTTOM NAVBAR ======= --> +</body> +</html> http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/package-list ---------------------------------------------------------------------- diff --git a/0100/javadoc/package-list b/0100/javadoc/package-list new file mode 100644 index 0000000..ec6060f --- /dev/null +++ b/0100/javadoc/package-list @@ -0,0 +1 @@ +kafka.javaapi.consumer http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/resources/background.gif ---------------------------------------------------------------------- diff --git a/0100/javadoc/resources/background.gif b/0100/javadoc/resources/background.gif new file mode 100644 index 0000000..f471940 Binary files /dev/null and b/0100/javadoc/resources/background.gif differ http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/resources/tab.gif ---------------------------------------------------------------------- diff --git a/0100/javadoc/resources/tab.gif b/0100/javadoc/resources/tab.gif new file mode 100644 index 0000000..1a73a83 Binary files /dev/null and b/0100/javadoc/resources/tab.gif differ http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/resources/titlebar.gif ---------------------------------------------------------------------- diff --git a/0100/javadoc/resources/titlebar.gif b/0100/javadoc/resources/titlebar.gif new file mode 100644 index 0000000..17443b3 Binary files /dev/null and b/0100/javadoc/resources/titlebar.gif differ http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/resources/titlebar_end.gif ---------------------------------------------------------------------- diff --git a/0100/javadoc/resources/titlebar_end.gif b/0100/javadoc/resources/titlebar_end.gif new file mode 100644 index 0000000..3ad78d4 Binary files /dev/null and b/0100/javadoc/resources/titlebar_end.gif differ http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/javadoc/stylesheet.css ---------------------------------------------------------------------- diff --git a/0100/javadoc/stylesheet.css b/0100/javadoc/stylesheet.css new file mode 100644 index 0000000..0aeaa97 --- /dev/null +++ b/0100/javadoc/stylesheet.css @@ -0,0 +1,474 @@ +/* Javadoc style sheet */ +/* +Overall document style +*/ +body { + background-color:#ffffff; + color:#353833; + font-family:Arial, Helvetica, sans-serif; + font-size:76%; + margin:0; +} +a:link, a:visited { + text-decoration:none; + color:#4c6b87; +} +a:hover, a:focus { + text-decoration:none; + color:#bb7a2a; +} +a:active { + text-decoration:none; + color:#4c6b87; +} +a[name] { + color:#353833; +} +a[name]:hover { + text-decoration:none; + color:#353833; +} +pre { + font-size:1.3em; +} +h1 { + font-size:1.8em; +} +h2 { + font-size:1.5em; +} +h3 { + font-size:1.4em; +} +h4 { + font-size:1.3em; +} +h5 { + font-size:1.2em; +} +h6 { + font-size:1.1em; +} +ul { + list-style-type:disc; +} +code, tt { + font-size:1.2em; +} +dt code { + font-size:1.2em; +} +table tr td dt code { + font-size:1.2em; + vertical-align:top; +} +sup { + font-size:.6em; +} +/* +Document title and Copyright styles +*/ +.clear { + clear:both; + height:0px; + overflow:hidden; +} +.aboutLanguage { + float:right; + padding:0px 21px; + font-size:.8em; + z-index:200; + margin-top:-7px; +} +.legalCopy { + margin-left:.5em; +} +.bar a, .bar a:link, .bar a:visited, .bar a:active { + color:#FFFFFF; + text-decoration:none; +} +.bar a:hover, .bar a:focus { + color:#bb7a2a; +} +.tab { + background-color:#0066FF; + background-image:url(resources/titlebar.gif); + background-position:left top; + background-repeat:no-repeat; + color:#ffffff; + padding:8px; + width:5em; + font-weight:bold; +} +/* +Navigation bar styles +*/ +.bar { + background-image:url(resources/background.gif); + background-repeat:repeat-x; + color:#FFFFFF; + padding:.8em .5em .4em .8em; + height:auto;/*height:1.8em;*/ + font-size:1em; + margin:0; +} +.topNav { + background-image:url(resources/background.gif); + background-repeat:repeat-x; + color:#FFFFFF; + float:left; + padding:0; + width:100%; + clear:right; + height:2.8em; + padding-top:10px; + overflow:hidden; +} +.bottomNav { + margin-top:10px; + background-image:url(resources/background.gif); + background-repeat:repeat-x; + color:#FFFFFF; + float:left; + padding:0; + width:100%; + clear:right; + height:2.8em; + padding-top:10px; + overflow:hidden; +} +.subNav { + background-color:#dee3e9; + border-bottom:1px solid #9eadc0; + float:left; + width:100%; + overflow:hidden; +} +.subNav div { + clear:left; + float:left; + padding:0 0 5px 6px; +} +ul.navList, ul.subNavList { + float:left; + margin:0 25px 0 0; + padding:0; +} +ul.navList li{ + list-style:none; + float:left; + padding:3px 6px; +} +ul.subNavList li{ + list-style:none; + float:left; + font-size:90%; +} +.topNav a:link, .topNav a:active, .topNav a:visited, .bottomNav a:link, .bottomNav a:active, .bottomNav a:visited { + color:#FFFFFF; + text-decoration:none; +} +.topNav a:hover, .bottomNav a:hover { + text-decoration:none; + color:#bb7a2a; +} +.navBarCell1Rev { + background-image:url(resources/tab.gif); + background-color:#a88834; + color:#FFFFFF; + margin: auto 5px; + border:1px solid #c9aa44; +} +/* +Page header and footer styles +*/ +.header, .footer { + clear:both; + margin:0 20px; + padding:5px 0 0 0; +} +.indexHeader { + margin:10px; + position:relative; +} +.indexHeader h1 { + font-size:1.3em; +} +.title { + color:#2c4557; + margin:10px 0; +} +.subTitle { + margin:5px 0 0 0; +} +.header ul { + margin:0 0 25px 0; + padding:0; +} +.footer ul { + margin:20px 0 5px 0; +} +.header ul li, .footer ul li { + list-style:none; + font-size:1.2em; +} +/* +Heading styles +*/ +div.details ul.blockList ul.blockList ul.blockList li.blockList h4, div.details ul.blockList ul.blockList ul.blockListLast li.blockList h4 { + background-color:#dee3e9; + border-top:1px solid #9eadc0; + border-bottom:1px solid #9eadc0; + margin:0 0 6px -8px; + padding:2px 5px; +} +ul.blockList ul.blockList ul.blockList li.blockList h3 { + background-color:#dee3e9; + border-top:1px solid #9eadc0; + border-bottom:1px solid #9eadc0; + margin:0 0 6px -8px; + padding:2px 5px; +} +ul.blockList ul.blockList li.blockList h3 { + padding:0; + margin:15px 0; +} +ul.blockList li.blockList h2 { + padding:0px 0 20px 0; +} +/* +Page layout container styles +*/ +.contentContainer, .sourceContainer, .classUseContainer, .serializedFormContainer, .constantValuesContainer { + clear:both; + padding:10px 20px; + position:relative; +} +.indexContainer { + margin:10px; + position:relative; + font-size:1.0em; +} +.indexContainer h2 { + font-size:1.1em; + padding:0 0 3px 0; +} +.indexContainer ul { + margin:0; + padding:0; +} +.indexContainer ul li { + list-style:none; +} +.contentContainer .description dl dt, .contentContainer .details dl dt, .serializedFormContainer dl dt { + font-size:1.1em; + font-weight:bold; + margin:10px 0 0 0; + color:#4E4E4E; +} +.contentContainer .description dl dd, .contentContainer .details dl dd, .serializedFormContainer dl dd { + margin:10px 0 10px 20px; +} +.serializedFormContainer dl.nameValue dt { + margin-left:1px; + font-size:1.1em; + display:inline; + font-weight:bold; +} +.serializedFormContainer dl.nameValue dd { + margin:0 0 0 1px; + font-size:1.1em; + display:inline; +} +/* +List styles +*/ +ul.horizontal li { + display:inline; + font-size:0.9em; +} +ul.inheritance { + margin:0; + padding:0; +} +ul.inheritance li { + display:inline; + list-style:none; +} +ul.inheritance li ul.inheritance { + margin-left:15px; + padding-left:15px; + padding-top:1px; +} +ul.blockList, ul.blockListLast { + margin:10px 0 10px 0; + padding:0; +} +ul.blockList li.blockList, ul.blockListLast li.blockList { + list-style:none; + margin-bottom:25px; +} +ul.blockList ul.blockList li.blockList, ul.blockList ul.blockListLast li.blockList { + padding:0px 20px 5px 10px; + border:1px solid #9eadc0; + background-color:#f9f9f9; +} +ul.blockList ul.blockList ul.blockList li.blockList, ul.blockList ul.blockList ul.blockListLast li.blockList { + padding:0 0 5px 8px; + background-color:#ffffff; + border:1px solid #9eadc0; + border-top:none; +} +ul.blockList ul.blockList ul.blockList ul.blockList li.blockList { + margin-left:0; + padding-left:0; + padding-bottom:15px; + border:none; + border-bottom:1px solid #9eadc0; +} +ul.blockList ul.blockList ul.blockList ul.blockList li.blockListLast { + list-style:none; + border-bottom:none; + padding-bottom:0; +} +table tr td dl, table tr td dl dt, table tr td dl dd { + margin-top:0; + margin-bottom:1px; +} +/* +Table styles +*/ +.contentContainer table, .classUseContainer table, .constantValuesContainer table { + border-bottom:1px solid #9eadc0; + width:100%; +} +.contentContainer ul li table, .classUseContainer ul li table, .constantValuesContainer ul li table { + width:100%; +} +.contentContainer .description table, .contentContainer .details table { + border-bottom:none; +} +.contentContainer ul li table th.colOne, .contentContainer ul li table th.colFirst, .contentContainer ul li table th.colLast, .classUseContainer ul li table th, .constantValuesContainer ul li table th, .contentContainer ul li table td.colOne, .contentContainer ul li table td.colFirst, .contentContainer ul li table td.colLast, .classUseContainer ul li table td, .constantValuesContainer ul li table td{ + vertical-align:top; + padding-right:20px; +} +.contentContainer ul li table th.colLast, .classUseContainer ul li table th.colLast,.constantValuesContainer ul li table th.colLast, +.contentContainer ul li table td.colLast, .classUseContainer ul li table td.colLast,.constantValuesContainer ul li table td.colLast, +.contentContainer ul li table th.colOne, .classUseContainer ul li table th.colOne, +.contentContainer ul li table td.colOne, .classUseContainer ul li table td.colOne { + padding-right:3px; +} +.overviewSummary caption, .packageSummary caption, .contentContainer ul.blockList li.blockList caption, .summary caption, .classUseContainer caption, .constantValuesContainer caption { + position:relative; + text-align:left; + background-repeat:no-repeat; + color:#FFFFFF; + font-weight:bold; + clear:none; + overflow:hidden; + padding:0px; + margin:0px; +} +caption a:link, caption a:hover, caption a:active, caption a:visited { + color:#FFFFFF; +} +.overviewSummary caption span, .packageSummary caption span, .contentContainer ul.blockList li.blockList caption span, .summary caption span, .classUseContainer caption span, .constantValuesContainer caption span { + white-space:nowrap; + padding-top:8px; + padding-left:8px; + display:block; + float:left; + background-image:url(resources/titlebar.gif); + height:18px; +} +.overviewSummary .tabEnd, .packageSummary .tabEnd, .contentContainer ul.blockList li.blockList .tabEnd, .summary .tabEnd, .classUseContainer .tabEnd, .constantValuesContainer .tabEnd { + width:10px; + background-image:url(resources/titlebar_end.gif); + background-repeat:no-repeat; + background-position:top right; + position:relative; + float:left; +} +ul.blockList ul.blockList li.blockList table { + margin:0 0 12px 0px; + width:100%; +} +.tableSubHeadingColor { + background-color: #EEEEFF; +} +.altColor { + background-color:#eeeeef; +} +.rowColor { + background-color:#ffffff; +} +.overviewSummary td, .packageSummary td, .contentContainer ul.blockList li.blockList td, .summary td, .classUseContainer td, .constantValuesContainer td { + text-align:left; + padding:3px 3px 3px 7px; +} +th.colFirst, th.colLast, th.colOne, .constantValuesContainer th { + background:#dee3e9; + border-top:1px solid #9eadc0; + border-bottom:1px solid #9eadc0; + text-align:left; + padding:3px 3px 3px 7px; +} +td.colOne a:link, td.colOne a:active, td.colOne a:visited, td.colOne a:hover, td.colFirst a:link, td.colFirst a:active, td.colFirst a:visited, td.colFirst a:hover, td.colLast a:link, td.colLast a:active, td.colLast a:visited, td.colLast a:hover, .constantValuesContainer td a:link, .constantValuesContainer td a:active, .constantValuesContainer td a:visited, .constantValuesContainer td a:hover { + font-weight:bold; +} +td.colFirst, th.colFirst { + border-left:1px solid #9eadc0; + white-space:nowrap; +} +td.colLast, th.colLast { + border-right:1px solid #9eadc0; +} +td.colOne, th.colOne { + border-right:1px solid #9eadc0; + border-left:1px solid #9eadc0; +} +table.overviewSummary { + padding:0px; + margin-left:0px; +} +table.overviewSummary td.colFirst, table.overviewSummary th.colFirst, +table.overviewSummary td.colOne, table.overviewSummary th.colOne { + width:25%; + vertical-align:middle; +} +table.packageSummary td.colFirst, table.overviewSummary th.colFirst { + width:25%; + vertical-align:middle; +} +/* +Content styles +*/ +.description pre { + margin-top:0; +} +.deprecatedContent { + margin:0; + padding:10px 0; +} +.docSummary { + padding:0; +} +/* +Formatting effect styles +*/ +.sourceLineNo { + color:green; + padding:0 30px 0 0; +} +h1.hidden { + visibility:hidden; + overflow:hidden; + font-size:.9em; +} +.block { + display:block; + margin:3px 0 0 0; +} +.strong { + font-weight:bold; +} http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/ops.html ---------------------------------------------------------------------- diff --git a/0100/ops.html b/0100/ops.html index 4cfe17b..541a01d 100644 --- a/0100/ops.html +++ b/0100/ops.html @@ -34,6 +34,8 @@ The replication factor controls how many servers will replicate each message tha <p> The partition count controls how many logs the topic will be sharded into. There are several impacts of the partition count. First each partition must fit entirely on a single server. So if you have 20 partitions the full data set (and read and write load) will be handled by no more than 20 servers (no counting replicas). Finally the partition count impacts the maximum parallelism of your consumers. This is discussed in greater detail in the <a href="#intro_consumers">concepts section</a>. <p> +Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a dash (-) and the partition id. Since a typical folder name can not be over 255 characters long, there will be a limitation on the length of topic names. We assume the number of partitions will not ever be above 100,000. Therefore, topic names cannot be longer than 249 characters. This leaves just enough room in the folder name for a dash and a potentially 5 digit long partition id. +<p> The configurations added on the command line override the default settings the server has for things like the length of time data should be retained. The complete set of per-topic configurations is documented <a href="#topic-config">here</a>. <h4><a id="basic_ops_modify_topic" href="#basic_ops_modify_topic">Modifying topics</a></h4> http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/quickstart.html ---------------------------------------------------------------------- diff --git a/0100/quickstart.html b/0100/quickstart.html index 1238316..7a923c6 100644 --- a/0100/quickstart.html +++ b/0100/quickstart.html @@ -21,11 +21,11 @@ This tutorial assumes you are starting fresh and have no existing Kafka or ZooKe <h4><a id="quickstart_download" href="#quickstart_download">Step 1: Download the code</a></h4> -<a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz"; title="Kafka downloads">Download</a> the 0.9.0.0 release and un-tar it. +<a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz"; title="Kafka downloads">Download</a> the 0.10.0.0 release and un-tar it. <pre> -> <b>tar -xzf kafka_2.11-0.9.0.0.tgz</b> -> <b>cd kafka_2.11-0.9.0.0</b> +> <b>tar -xzf kafka_2.11-0.10.0.0.tgz</b> +> <b>cd kafka_2.11-0.10.0.0</b> </pre> <h4><a id="quickstart_startserver" href="#quickstart_startserver">Step 2: Start the server</a></h4> @@ -249,3 +249,112 @@ The connectors continue to process data, so we can add data to the file and see </pre> You should see the line appear in the console consumer output and in the sink file. + +<h4><a id="quickstart_kafkastreams" href="#quickstart_kafkastreams">Step 8: Use Kafka Streams to process data</a></h4> + +<p> +Kafka Streams is a client library of Kafka for real-time stream processing and analyzing data stored in Kafka brokers. +This quickstart example will demonstrate how to run a streaming application coded in this library. Here is the gist +of the <code>WordCountDemo</code> example code (converted to use Java 8 lambda expressions for easy reading). +</p> +<pre> +KStream<String, Long> wordCounts = textLines +// Split each text line, by whitespace, into words. +.flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+"))) +// Ensure the words are available as message keys for the next aggregate operation. +.map((key, value) -> new KeyValue<>(value, value)) +// Count the occurrences of each word (message key). +.countByKey(stringSerializer, longSerializer, stringDeserializer, longDeserializer, "Counts") +// Convert the resulted aggregate table into another stream. +.toStream(); +</pre> + +<p> +It implements the WordCount +algorithm, which computes a word occurrence histogram from the input text. However, unlike other WordCount examples +you might have seen before that operate on bounded data, the WordCount demo application behaves slightly differently because it is +designed to operate on an <b>infinite, unbounded stream</b> of data. Similar to the bounded variant, it is a stateful algorithm that +tracks and updates the counts of words. However, since it must assume potentially +unbounded input data, it will periodically output its current state and results while continuing to process more data +because it cannot know when it has processed "all" the input data. +</p> +<p> +We will now prepare input data to a Kafka topic, which will subsequently processed by a Kafka Streams application. +</p> + +<!-- +<pre> +> <b>./bin/kafka-topics --create \</b> + <b>--zookeeper localhost:2181 \</b> + <b>--replication-factor 1 \</b> + <b>--partitions 1 \</b> + <b>--topic streams-file-input</b> + +</pre> + +--> + +<pre> +> <b>echo -e "all streams lead to kafka\nhello kafka streams\njoin kafka summit" > file-input.txt</b> +</pre> + +<p> +Next, we send this input data to the input topic named <b>streams-file-input</b> using the console producer (in practice, +stream data will likely be flowing continuously into Kafka where the application will be up and running): +</p> + +<pre> +> <b>cat /tmp/file-input.txt | ./bin/kafka-console-producer --broker-list localhost:9092 --topic streams-file-input</b> +</pre> + +<p> +We can now run the WordCount demo application to process the input data: +</p> + +<pre> +> <b>./bin/kafka-run-class org.apache.kafka.streams.examples.wordcount.WordCountDemo</b> +</pre> + +<p> +There won't be any STDOUT output except log entries as the results are continuously written back into another topic named <b>streams-wordcount-output</b> in Kafka. +The demo will run for a few seconds and then, unlike typical stream processing applications, terminate automatically. +</p> +<p> +We can now inspect the output of the WordCount demo application by reading from its output topic: +</p> + +<pre> +> <b>./bin/kafka-console-consumer --zookeeper localhost:2181 \</b> + <b>--topic streams-wordcount-output \</b> + <b>--from-beginning \</b> + <b>--formatter kafka.tools.DefaultMessageFormatter \</b> + <b>--property print.key=true \</b> + <b>--property print.key=true \</b> + <b>--property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \</b> + <b>--property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer</b> +</pre> + +<p> +with the following output data being printed to the console (You can stop the console consumer via <b>Ctrl-C</b>): +</p> + +<pre> +all 1 +streams 1 +lead 1 +to 1 +kafka 1 +hello 1 +kafka 2 +streams 2 +join 1 +kafka 3 +summit 1 +<b>^C</b> +</pre> + +<p> +Here, the first column is the Kafka message key, and the second column is the message value, both in in <code>java.lang.String</code> format. +Note that the output is actually a continuous stream of updates, where each data record (i.e. each line in the original output above) is +an updated count of a single word, aka record key such as "kafka". For multiple records with the same key, each later record is an update of the previous one. +</p> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/streams.html ---------------------------------------------------------------------- diff --git a/0100/streams.html b/0100/streams.html new file mode 100644 index 0000000..9b94bb3 --- /dev/null +++ b/0100/streams.html @@ -0,0 +1,341 @@ +<!--~ + ~ Licensed to the Apache Software Foundation (ASF) under one or more + ~ contributor license agreements. See the NOTICE file distributed with + ~ this work for additional information regarding copyright ownership. + ~ The ASF licenses this file to You under the Apache License, Version 2.0 + ~ (the "License"); you may not use this file except in compliance with + ~ the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + ~--> + +<h3><a id="streams_overview" href="#streams_overview">9.1 Overview</a></h3> + +<p> +Kafka Streams is a client library for processing and analyzing data stored in Kafka and either write the resulting data back to Kafka or send the final output to an external system. It builds upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple yet efficient management of application state. +Kafka Streams has a <b>low barrier to entry</b>: You can quickly write and run a small-scale proof-of-concept on a single machine; and you only need to run additional instances of your application on multiple machines to scale up to high-volume production workloads. Kafka Streams transparently handles the load balancing of multiple instances of the same application by leveraging Kafka's parallelism model. +</p> +<p> +Some highlights of Kafka Streams: +</p> + +<ul> + <li>Designed as a <b>simple and lightweight client library</b>, which can be easily embedded in any Java application and integrated with any existing packaging, deployment and operational tools that users have for their streaming applications.</li> + <li>Has <b>no external dependencies on systems other than Apache Kafka itself</b> as the internal messaging layer; notably, it uses Kafka's partitioning model to horizontally scale processing while maintaining strong ordering guarantees.</li> + <li>Supports <b>fault-tolerant local state</b>, which enables very fast and efficient stateful operations like joins and windowed aggregations.</li> + <li>Employs <b>one-record-at-a-time processing</b> to achieve low processing latency, and supports <b>event-time based windowing operations</b>.</li> + <li>Offers necessary stream processing primitives, along with a <b>high-level Streams DSL</b> and a <b>low-level Processor API</b>.</li> + +</ul> + +<h3><a id="streams_developer" href="#streams_developer">9.2 Developer Guide</a></h3> + +<p> +There is a <a href="#quickstart_kafkastreams">quickstart</a> example that provides how to run a stream processing program coded in the Kafka Streams library. +This section focuses on how to write, configure, and execute a Kafka Streams application. +</p> + +<h4><a id="streams_concepts" href="#streams_concepts">Core Concepts</a></h4> + +<p> +We first summarize the key concepts of Kafka Streams. +</p> + +<h5><a id="streams_topology" href="#streams_topology">Stream Processing Topology</a></h5> + +<ul> + <li>A <b>stream</b>is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a <b>data record</b> is defined as a key-value pair.</li> + <li>A stream processing application written in Kafka Streams defines its computational logic through one or more <b>processor topologies</b>, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).</li> + <li>A <b>stream processor</b> is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently producing one or more output records to its downstream processors.</li> +</ul> + +<p> +Kafka Streams offers two ways to define the stream processing topology: the <a href="#streams_dsl"><b>Kafka Streams DSL</b></a> provides +the most common data transformation operations such as <code>map</code> and <code>filter</code>; the lower-level <a href="#streams_processor"><b>Processor API</b></a> allows +developers define and connect custom processors as well as to interact with <a href="#streams_state">state stores</a>. +</p> + +<h5><a id="streams_time" href="#streams_time">Time</a></h5> + +<p> +A critical aspect in stream processing is the the notion of <b>time</b>, and how it is modeled and integrated. +For example, some operations such as <b>windowing</b> are defined based on time boundaries. +</p> +<p> +Common notions of time in streams are: +</p> + +<ul> + <li><b>Event time</b> - The point in time when an event or data record occurred, i.e. was originally created "at the source".</li> + <li><b>Processing time</b> - The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is being consumed. The processing time may be milliseconds, hours, or days etc. later than the original event time.</li> +</ul> + +<p> +Kafka Streams assigns a <b>timestamp</b> to every data record +via the <code>TimestampExtractor</code> interface. +Concrete implementations of this interface may retrieve or compute timestamps based on the actual contents of data records such as an embedded timestamp field +to provide event-time semantics, or use any other approach such as returning the current wall-clock time at the time of processing, +thereby yielding processing-time semantics to stream processing applications. +Developers can thus enforce different notions of time depending on their business needs. For example, +per-record timestamps describe the progress of a stream with regards to time (although records may be out-of-order within the stream) and +are leveraged by time-dependent operations such as joins. +</p> + +<h5><a id="streams_state" href="#streams_state">States</a></h5> + +<p> +Some stream processing applications don't require state, which means the processing of a message is independent from +the processing of all other messages. +However, being able to maintain state opens up many possibilities for sophisticated stream processing applications: you +can join input streams, or group and aggregate data records. Many such stateful operators are provided by the <a href="#streams_dsl"><b>Kafka Streams DSL</b></a>. +</p> +<p> +Kafka Streams provides so-called <b>state stores</b>, which can be used by stream processing applications to store and query data. +This is an important capability when implementing stateful operations. +Every task in Kafka Streams embeds one or more state stores that can be accessed via APIs to store and query data required for processing. +These state stores can either be a persistent key-value store, an in-memory hashmap, or another convenient data structure. +Kafka Streams offers fault-tolerance and automatic recovery for local state stores. +</p> +<br> +<p> +As we have mentioned above, the computational logic of a Kafka Streams application is defined as a <a href="#streams_topology">processor topology</a>. +Currently Kafka Streams provides two sets of APIs to define the processor topology, which will be described in the subsequent sections. +</p> + +<h4><a id="streams_processor" href="#streams_processor">Low-Level Processor API</a></h4> + +<h5><a id="streams_processor_process" href="#streams_processor_process">Processor</a></h5> + +<p> +Developers can define their customized processing logic by implementing the <code>Processor</code> interface, which +provides <code>process</code> and <code>punctuate</code> methods. The <code>process</code> method is performed on each +of the received record; and the <code>punctuate</code> method is performed periodically based on elapsed time. +In addition, the processor can maintain the current <code>ProcessorContext</code> instance variable initialized in the +<code>init</code> method, and use the context to schedule the punctuation period (<code>context().schedule</code>), to +forward the modified / new key-value pair to downstream processors (<code>context().forward</code>), to commit the current +processing progress (<code>context().commit</code>), etc. +</p> + +<pre> + public class MyProcessor extends Processor<String, String> { + private ProcessorContext context; + private KeyValueStore<String, Integer> kvStore; + + @Override + @SuppressWarnings("unchecked") + public void init(ProcessorContext context) { + this.context = context; + this.context.schedule(1000); + this.kvStore = (KeyValueStore<String, Integer>) context.getStateStore("Counts"); + } + + @Override + public void process(String dummy, String line) { + String[] words = line.toLowerCase().split(" "); + + for (String word : words) { + Integer oldValue = this.kvStore.get(word); + + if (oldValue == null) { + this.kvStore.put(word, 1); + } else { + this.kvStore.put(word, oldValue + 1); + } + } + } + + @Override + public void punctuate(long timestamp) { + KeyValueIterator<String, Integer> iter = this.kvStore.all(); + + while (iter.hasNext()) { + KeyValue<String, Integer> entry = iter.next(); + context.forward(entry.key, entry.value.toString()); + } + + iter.close(); + context.commit(); + } + + @Override + public void close() { + this.kvStore.close(); + } + }; +</pre> + +<p> +In the above implementation, the following actions are performed: + +<ul> + <li>In the <code>init</code> method, schedule the punctuation every 1 second and retrieve the local state store by its name "Counts".</li> + <li>In the <code>process</code> method, upon each received record, split the value string into words, and update their counts into the state store (we will talk about this feature later in the section).</li> + <li>In the <code>punctuate</code> method, iterate the local state store and send the aggregated counts to the downstream processor, and commit the current stream state.</li> +</ul> +</p> + +<h5><a id="streams_processor_topology" href="#streams_processor_topology">Processor Topology</a></h5> + +<p> +With the customized processors defined in the Processor API, developers can use the <code>TopologyBuilder</code> to build a processor topology +by connecting these processors together: + +<pre> + TopologyBuilder builder = new TopologyBuilder(); + + builder.addSource("SOURCE", "src-topic") + + .addProcessor("PROCESS1", MyProcessor1::new /* the ProcessorSupplier that can generate MyProcessor1 */, "SOURCE") + .addProcessor("PROCESS2", MyProcessor2::new /* the ProcessorSupplier that can generate MyProcessor2 */, "PROCESS1") + .addProcessor("PROCESS3", MyProcessor3::new /* the ProcessorSupplier that can generate MyProcessor3 */, "PROCESS1") + + .addSink("SINK1", "sink-topic1", "PROCESS1") + .addSink("SINK2", "sink-topic2", "PROCESS2") + .addSink("SINK3", "sink-topic3", "PROCESS3"); +</pre> + +There are several steps in the above code to build the topology, and here is a quick walk through: + +<ul> + <li>First of all a source node named "SOURCE" is added to the topology using the <code>addSource</code> method, with one Kafka topic "src-topic" fed to it.</li> + <li>Three processor nodes are then added using the <code>addProcessor</code> method; here the first processor is a child of the "SOURCE" node, but is the parent of the other two processors.</li> + <li>Finally three sink nodes are added to complete the topology using the <code>addSink</code> method, each piping from a different parent processor node and writing to a separate topic.</li> +</ul> +</p> + +<h5><a id="streams_processor_statestore" href="#streams_processor_statestore">Local State Store</a></h5> + +<p> +Note that the Processor API is not limited to only accessing the current records as they arrive, but can also maintain local state stores +that keep recently arrived records to use in stateful processing operations such as aggregation or windowed joins. +To take advantage of this local states, developers can use the <code>TopologyBuilder.addStateStore</code> method when building the +processor topology to create the local state and associate it with the processor nodes that needs to access it; or they can connect a created +local state store with the existing processor nodes through <code>TopologyBuilder.connectProcessorAndStateStores</code>. + +<pre> + TopologyBuilder builder = new TopologyBuilder(); + + builder.addSource("SOURCE", "src-topic") + + .addProcessor("PROCESS1", MyProcessor1::new, "SOURCE") + // create the in-memory state store "COUNTS" associated with processor "PROCESS1" + .addStateStore(Stores.create("COUNTS").withStringKeys().withStringValues().inMemory().build(), "PROCESS1") + .addProcessor("PROCESS2", MyProcessor3::new /* the ProcessorSupplier that can generate MyProcessor3 */, "PROCESS1") + .addProcessor("PROCESS3", MyProcessor3::new /* the ProcessorSupplier that can generate MyProcessor3 */, "PROCESS1") + + // connect the state store "COUNTS" with processor "PROCESS2" + .connectProcessorAndStateStores("PROCESS2", "COUNTS"); + + .addSink("SINK1", "sink-topic1", "PROCESS1") + .addSink("SINK2", "sink-topic2", "PROCESS2") + .addSink("SINK3", "sink-topic3", "PROCESS3"); +</pre> + +</p> + +In the next section we present another way to build the processor topology: the Kafka Streams DSL. + +<h4><a id="streams_dsl" href="#streams_dsl">High-Level Streams DSL</a></h4> + +To build a processor topology using the Streams DSL, developers can apply the <code>KStreamBuilder</code> class, which is extended from the <code>TopologyBuilder</code>. +A simple example is included with the source code for Kafka in the <code>streams/examples</code> package. The rest of this section will walk +through some code to demonstrate the key steps in creating a topology using the Streams DSL, but we recommend developers to read the full example source +codes for details. + +<h5><a id="streams_dsl_source" href="#streams_dsl_source">Create Source Streams from Kafka</a></h5> + +<p> +Either a <b>record stream</b> (defined as <code>KStream</code>) or a <b>changelog stream</b> (defined as <code>KTable</code>) +can be created as a source stream from one or more Kafka topics (for <code>KTable</code> you can only create the source stream +from a single topic). +</p> + +<pre> + KStreamBuilder builder = new KStreamBuilder(); + + KStream<String, GenericRecord> source1 = builder.stream("topic1", "topic2"); + KTable<String, GenericRecord> source2 = builder.table("topic3"); +</pre> + +<h5><a id="streams_dsl_transform" href="#streams_dsl_transform">Transform a stream</a></h5> + +<p> +There is a list of transformation operations provided for <code>KStream</code> and <code>KTable</code> respectively. +Each of these operations may generate either one or more <code>KStream</code> and <code>KTable</code> objects and +can be translated into one or more connected processors into the underlying processor topology. +All these transformation methods can be chained together to compose a complex processor topology. +Since <code>KStream</code> and <code>KTable</code> are strongly typed, all these transformation operations are defined as +generics functions where users could specify the input and output data types. +</p> + +<p> +Among these transformations, <code>filter</code>, <code>map</code>, <code>mapValues</code>, etc, are stateless +transformation operations and can be applied to both <code>KStream</code> and <code>KTable</code>, +where users can usually pass a customized function to these functions as a parameter, such as <code>Predicate</code> for <code>filter</code>, +<code>KeyValueMapper</code> for <code>map</code>, etc: + +</p> + +<pre> + // written in Java 8+, using lambda expressions + KStream<String, GenericRecord> mapped = source1.mapValue(record -> record.get("category")); +</pre> + +<p> +Stateless transformations, by definition, do not depend on any state for processing, and hence implementation-wise +they do not require a state store associated with the stream processor; Stateful transformations, on the other hand, +require accessing an associated state for processing and producing outputs. +For example, in <code>join</code> and <code>aggregate</code> operations, a windowing state is usually used to store all the received records +within the defined window boundary so far. The operators can then access these accumulated records in the store and compute +based on them. +</p> + +<pre> + // written in Java 8+, using lambda expressions + KTable<Windowed<String>, Long> counts = source1.aggregateByKey( + () -> 0L, // initial value + (aggKey, value, aggregate) -> aggregate + 1L, // aggregating value + HoppingWindows.of("counts").with(5000L).every(1000L), // intervals in milliseconds + ); + + KStream<String, String> joined = source1.leftJoin(source2, + (record1, record2) -> record1.get("user") + "-" + record2.get("region"); + ); +</pre> + +<h5><a id="streams_dsl_sink" href="#streams_dsl_sink">Write streams back to Kafka</a></h5> + +<p> +At the end of the processing, users can choose to (continuously) write the final resulted streams back to a Kafka topic through +<code>KStream.to</code> and <code>KTable.to</code>. +</p> + +<pre> + joined.to("topic4"); +</pre> + +If your application needs to continue reading and processing the records after they have been materialized +to a topic via <code>to</code> above, one option is to construct a new stream that reads from the output topic; +Kafka Streams provides a convenience method called <code>through</code>: + +<pre> + // equivalent to + // + // joined.to("topic4"); + // materialized = builder.stream("topic4"); + KStream<String, String> materialized = joined.through("topic4"); +</pre> + + +<br> +<p> +Besides defining the topology, developers will also need to configure their applications +in <code>StreamsConfig</code> before running it. A complete list of +Kafka Streams configs can be found <a href="#streamsconfigs"><b>here</b></a>. +</p> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kafka-site/blob/46fccbe1/0100/upgrade.html ---------------------------------------------------------------------- diff --git a/0100/upgrade.html b/0100/upgrade.html index ba3d024..060c3de 100644 --- a/0100/upgrade.html +++ b/0100/upgrade.html @@ -64,7 +64,7 @@ work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9 is set to 0.10.0, one should not change it back to an earlier format as it may break consumers on versions before 0.10.0.0. </p> -<h5><a id="upgrade_10_breaking" href="#upgrade_10_breaking">potential breaking changes in 0.10.0.0</a></h5> +<h5><a id="upgrade_10_breaking" href="#upgrade_10_breaking">Potential breaking changes in 0.10.0.0</a></h5> <ul> <li> Starting from Kafka 0.10.0.0, the message format version in Kafka is represented as the Kafka version. For example, message format 0.9.0 refers to the highest message version supported by Kafka 0.9.0. </li> <li> Message format 0.10.0 has been introduced and it is used by default. It includes a timestamp field in the messages and relative offsets are used for compressed messages. </li> @@ -78,6 +78,13 @@ work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9 <li> MessageFormatter's package was changed from <code>kafka.tools</code> to <code>kafka.common</code> </li> <li> MessageReader's package was changed from <code>kafka.tools</code> to <code>kafka.common</code> </li> <li> MirrorMakerMessageHandler no longer exposes the <code>handle(record: MessageAndMetadata[Array[Byte], Array[Byte]])</code> method as it was never called. </li> + <li> The 0.7 KafkaMigrationTool is no longer packaged with Kafka. If you need to migrate from 0.7 to 0.10.0, please migrate to 0.8 first and then follow the documented upgrade process to upgrade from 0.8 to 0.10.0. </li> +</ul> + +<h5><a id="upgrade_10_notable" href="#upgrade_10_notable">Notable changes in 0.10.0.0</a></h5> + +<ul> + <li> The default value of the configuration parameter <code>receive.buffer.bytes</code> is now 64K for the new consumer </li> </ul> <h4><a id="upgrade_9" href="#upgrade_9">Upgrading from 0.8.0, 0.8.1.X or 0.8.2.X to 0.9.0.0</a></h4>
