Modified: nifi/site/trunk/docs/nifi-docs/html/user-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/html/user-guide.html?rev=1758318&r1=1758317&r2=1758318&view=diff ============================================================================== --- nifi/site/trunk/docs/nifi-docs/html/user-guide.html (original) +++ nifi/site/trunk/docs/nifi-docs/html/user-guide.html Tue Aug 30 03:18:37 2016 @@ -443,17 +443,21 @@ body.book #toc,body.book #preamble,body. <li><a href="#browser-support">Browser Support</a> <ul class="sectlevel2"> <li><a href="#unsupported-browsers">Unsupported Browsers</a></li> +<li><a href="#viewing-the-ui-in-variably-sized-browsers">Viewing the UI in Variably Sized Browsers</a></li> </ul> </li> <li><a href="#terminology">Terminology</a></li> <li><a href="#User_Interface">NiFi User Interface</a></li> +<li><a href="#UI-with-multi-tenant-authorization">Accessing the UI with Multi-Tenant Authorization</a></li> <li><a href="#logging-in">Logging In</a></li> <li><a href="#building-dataflow">Building a DataFlow</a> <ul class="sectlevel2"> <li><a href="#adding-components-to-the-canvas">Adding Components to the Canvas</a></li> <li><a href="#Configuring_a_Processor">Configuring a Processor</a></li> <li><a href="#additional-help">Additional Help</a></li> -<li><a href="#Controller_Services_and_Reporting_Tasks">Controller Services and Reporting Tasks</a></li> +<li><a href="#Using_Custom_Properties">Using Custom Properties with Expression Language</a></li> +<li><a href="#Controller_Services">Controller Services</a></li> +<li><a href="#Reporting_Tasks">Reporting Tasks</a></li> <li><a href="#Connecting_Components">Connecting Components</a></li> <li><a href="#processor-validation">Processor Validation</a></li> <li><a href="#site-to-site">Site-to-Site</a></li> @@ -482,6 +486,7 @@ body.book #toc,body.book #preamble,body. <li><a href="#templates">Templates</a> <ul class="sectlevel2"> <li><a href="#Create_Template">Creating a Template</a></li> +<li><a href="#Import_Template">Importing a Template</a></li> <li><a href="#instantiating-a-template">Instantiating a Template</a></li> <li><a href="#Manage_Templates">Managing Templates</a></li> </ul> @@ -511,8 +516,8 @@ high throughput, and priority-based queu cloned, modified, sent, and ultimately dropped upon reaching its configured end-state.</p> </div> <div class="paragraph"> -<p>See the Admin Guide for information about system requirements, installation, and configuration. Once NiFi is installed, -use a supported web browser to view the User Interface (UI).</p> +<p>See the <a href="administration-guide.html">System Administratorâs Guide</a> for information about system requirements, installation, and configuration. Once NiFi is installed, +use a supported web browser to view the UI.</p> </div> </div> </div> @@ -567,6 +572,17 @@ against the supported browsers. Any prob experience and is not currently supported in mobile browsers.</p> </div> </div> +<div class="sect2"> +<h3 id="viewing-the-ui-in-variably-sized-browsers"><a class="anchor" href="#viewing-the-ui-in-variably-sized-browsers"></a>Viewing the UI in Variably Sized Browsers</h3> +<div class="paragraph"> +<p>In most environments, all of the UI is visible in your browser. However, the UI has a responsive design that allows you +to scroll through screens as needed, in smaller sized browsers or tablet environments.</p> +</div> +<div class="paragraph"> +<p>In environments where your browser width is less than 800 pixels and the height less than 600 pixels, portions of the +UI may become unavailable.</p> +</div> +</div> </div> </div> <div class="sect1"> @@ -642,7 +658,7 @@ experience and is not currently supporte Whenever a component reports a Bulletin, a bulletin icon is displayed on that component. System-level bulletins are displayed on the Status bar near the top of the page. Using the mouse to hover over that icon will provide a tool-tip that shows the time and severity (Debug, Info, Warning, Error) of the Bulletin, as well as the message of the Bulletin. - Bulletins from all components can also be viewed and filtered in the Bulletin Board Page, available in the Management Toolbar.</p> + Bulletins from all components can also be viewed and filtered in the Bulletin Board Page, available in the Global Menu.</p> </div> <div class="paragraph"> <p><strong>Template</strong>: Often times, a dataflow is comprised of many sub-flows that could be reused. NiFi allows DFMs to select a part of the dataflow @@ -651,10 +667,13 @@ experience and is not currently supporte These templates can also be exported as XML and imported into another NiFi instance, allowing these building blocks to be shared.</p> </div> <div class="paragraph"> -<p><strong>flow.xml.gz</strong>: Everything the DFM puts onto the NiFi User Interface canvas is written, in real time, to one file called the flow.xml.gz. This file is located in the nifi/conf directory. - Any change made on the canvas is automatically saved to this file, without the user needing to click a "save" button. In addition, the user may create a back-up copy of this file at any time - by selecting the Controller Settings button in the far-right section of the tool bar and clicking "Back-up flow" on the General tab. By default, this action saves a copy of the current flow in the nifi/conf/archive directory. - See <a href="#Controller_Settings">Controller Settings</a> for a description of where the "Back-up flow" button may be found. (Note that in a NiFi Cluster, the NiFi Cluster Manager’s copy of this file is named flow.tar, whereas this file is still named flow.xml.gz on the nodes.)</p> +<p><strong>flow.xml.gz</strong>: Everything the DFM puts onto the NiFi User Interface canvas is written, in real time, to one file called the flow.xml.gz. This file is located in the nifi/conf directory by default. + Any change made on the canvas is automatically saved to this file, without the user needing to click a "save" button. + In addition, NiFi automatically creates a backup copy of this file in the archive directory when it is updated. + You can use these archived files to rollback flow configuration. To do so, stop NiFi, replace flow.xml.gz with a desired backup copy, then restart NiFi. + In a clustered environment, stop the entire NiFi cluster, replace the flow.xml.gz of one of nodes, and restart the node. Remove flow.xml.gz from other nodes. + Once you confirmed the node starts up as a one-node cluster, start the other nodes. The replaced flow configuration will be synchronized across the cluster. + The name and location of flow.xml.gz, and auto archive behavior are configurable. See the <a href="administration-guide.html#core-properties-br">System Administratorâs Guide</a> for further details.</p> </div> </div> </div> @@ -662,70 +681,181 @@ experience and is not currently supporte <h2 id="User_Interface"><a class="anchor" href="#User_Interface"></a>NiFi User Interface</h2> <div class="sectionbody"> <div class="paragraph"> -<p>The NiFi User Interface (UI) provides mechanisms for creating automated dataflows, as well as visualizing, +<p>The NiFi UI provides mechanisms for creating automated dataflows, as well as visualizing, editing, monitoring, and administering those dataflows. The UI can be broken down into several segments, each responsible for different functionality of the application. This section provides screenshots of the application and highlights the different segments of the UI. Each segment is discussed in further detail later in the document.</p> </div> <div class="paragraph"> -<p>When the application is started, the user is able to navigate to the User Interface by going to the default address of +<p>When the application is started, the user is able to navigate to the UI by going to the default address of <code>http://<hostname>:8080/nifi</code> in a web browser. There are no permissions configured by default, so anyone is -able to view and modify the dataflow. For information on securing the system, see the Systems Administrator guide.</p> +able to view and modify the dataflow. For information on securing the system, see the <a href="administration-guide.html">System Administratorâs Guide</a>.</p> </div> <div class="paragraph"> <p>When a DFM navigates to the UI for the first time, a blank canvas is provided on which a dataflow can be built:</p> </div> <div class="imageblock"> <div class="content"> -<img src="./images/new-flow.png" alt="Empty Flow"> +<img src="./images/nifi-toolbar-components.png" alt="NiFi Components Toolbar"> </div> </div> <div class="paragraph"> -<p>Along the top of the of the screen is a toolbar that contains several of these segments. -To the left is the Components Toolbar. This toolbar consists of the different components that can be dragged onto the canvas.</p> +<p>The Components Toolbar runs across the top left portion of your screen. It consists of the components you can drag onto the +canvas to build your dataflow. Each component is described in more detail in <a href="building-dataflow.html">Building a Dataflow</a>.</p> </div> <div class="paragraph"> -<p>Next to the Components Toolbar is the Actions Toolbar. This toolbar consists of buttons to manipulate the existing -components on the canvas. To the right of the Actions Toolbar is the Search Toolbar. This toolbar consists of a single -Search field that allows users to easily find components on the canvas. Users are able to search by component name, -type, identifier, configuration properties, and their values.</p> +<p>The Status Bar is under the Components Toolbar. The Status bar provides information about how many Processors exist on the canvas in +each state (Stopped, Running, Invalid, Disabled), how many Remote Process Groups exist on the canvas in each state +(Transmitting, Not Transmitting), the number of threads that are currently active in the flow, the amount of data that currently +exists in the flow, and the timestamp at which all of this information was last refreshed. Additionally, if the instance of NiFi is clustered, the Status bar shows how many nodes +are in the cluster and how many are currently connected.</p> </div> <div class="paragraph"> -<p>The Management Toolbar sits to the right-hand side of the screen. This toolbar consists of buttons that are -used by DFMs to manage the flow as well as by administrators who manage user access +<p>The Operate Palette sits to the left-hand side of the screen. It consists of buttons that are +used by DFMs to manage the flow, as well as by administrators who manage user access and configure system properties, such as how many system resources should be provided to the application.</p> </div> +<div class="paragraph"> +<p>On the right side of the canvas is Search, and the Global Menu. You can use Search to easily find components on the +canvas and can to search by component name, type, identifier, configuration properties, and their values. The Global Menu +contain options that allow you to manipulate existing components on the canvas:</p> +</div> <div class="imageblock"> <div class="content"> -<img src="./images/nifi-toolbar-components.png" alt="NiFi Components Toolbar"> +<img src="./images/global-menu.png" alt="NiFi Global Menu"> </div> </div> <div class="paragraph"> -<p>Next, we have segments that provide capabilities to easily navigate around the canvas. On the left-hand side is a toolbar that -provides the ability to pan around the canvas and zoom in and out. On the right-hand side is a âBirds-Eye Viewâ of the dataflow. -This provides a high-level view of the dataflow and allows the user to quickly and easily pan across large portions of the dataflow. -Along the top of the screen is a trail of breadcrumbs. As users navigate into and out of Process Groups, the breadcrumbs show -the depth in the flow and each Process Group that was entered to reach this depth. Each of the Process Groups listed in the breadcrumbs -is a link that will take you back up to that level in the flow.</p> +<p>Additionally, the UI has allows has some features that allow you to easily navigate around the canvas. You can use the +Navigate Palette to pan around the canvas, and to zoom in and out. The âBirds Eye Viewâ of the dataflow provides a high-level +view of the dataflow and allows you to pan across large portions of the dataflow. You can also find breadcrumbs along the +bottom of the screen. As you navigate into and out of Process Groups, the breadcrumbs show +the depth in the flow, and each Process Group that you entered to reach this depth. Each of the Process Groups listed in the +breadcrumbs is a link that will take you back up to that level in the flow.</p> </div> <div class="imageblock"> <div class="content"> <img src="./images/nifi-navigation.png" alt="NiFi Navigation"> </div> </div> -<div id="status_bar" class="paragraph"> -<p>Below the breadcrumbs lives the Status bar. The Status bar provides information about how many Processors exist on the canvas in -each state (Stopped, Running, Invalid, Disabled), how many Remote Process Groups exist on the canvas in each state -(Transmitting, Not Transmitting), the number of threads that are currently active in the flow, the amount of data that currently -exists in the flow, and the timestamp at which all of this information was last refreshed. If there are any System-Level bulletins, -these are shown in the Status bar as well. Additionally, if the instance of NiFi is clustered, the Status bar shows how many nodes -are in the cluster and how many are currently connected.</p> </div> -<div class="imageblock"> -<div class="content"> -<img src="./images/status-bar.png" alt="NiFi Status Bar"> </div> +<div class="sect1"> +<h2 id="UI-with-multi-tenant-authorization"><a class="anchor" href="#UI-with-multi-tenant-authorization"></a>Accessing the UI with Multi-Tenant Authorization</h2> +<div class="sectionbody"> +<div class="paragraph"> +<p>Multi-tenant authorization enables multiple groups of users (tenants) to command, control, and observe different parts of the dataflow, +with varying levels of authorization. When an authenticated user attempts to view or modify a NiFi resource, the system checks whether the +user has privileges to perform that action. These privileges are defined by policies that you can apply system wide or to individual +components. What this means from a Dataflow Manager perspective is that once you have access to the NiFi canvas, a range of functionality +is visible and available to you, depending on the privileges assigned to you.</p> +</div> +<div class="paragraph"> +<p>The available global access policies are:</p> +</div> +<table class="tableblock frame-all grid-all spread"> +<colgroup> +<col style="width: 50%;"> +<col style="width: 50%;"> +</colgroup> +<thead> +<tr> +<th class="tableblock halign-left valign-top">Policy</th> +<th class="tableblock halign-left valign-top">Privilege</th> +</tr> +</thead> +<tbody> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">view the UI</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view the UI</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">access the controller</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify the controller including reporting tasks, Controller Services, and nodes in the cluster</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">query provenance</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to submit a provenance search and request even lineage</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">access all policies</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify the policies for all components</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">access users/groups</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users view and modify the users and user groups</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">retrieve site-to-site details</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows other NiFi instances to retrieve Site-To-Site details</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">view system diagnostics</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view System Diagnostics</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">proxy user requests</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows proxy machines to send requests on the behalf of others</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">access counters</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view and modify counters</p></td> +</tr> +</tbody> +</table> +<div class="paragraph"> +<p>The available component-level access policies are:</p> +</div> +<table class="tableblock frame-all grid-all spread"> +<colgroup> +<col style="width: 50%;"> +<col style="width: 50%;"> +</colgroup> +<thead> +<tr> +<th class="tableblock halign-left valign-top">Policy</th> +<th class="tableblock halign-left valign-top">Privilege</th> +</tr> +</thead> +<tbody> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">view the component</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view component configuration details</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">modify the component</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to modify component configuration details</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">view the data</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view metadata and content for this component through provenance data and flowfile queues in outbound connection</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">modify the data</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to empty flowfile queues in outbound connections and to submit replays</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">view the policies</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to view the list of users who can view and modify a component</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">modify the policies</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows users to modify the list of users who can view and modify a component</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">retrieve data via site-to-site</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows a port to receive data from NiFi instances</p></td> +</tr> +<tr> +<td class="tableblock halign-left valign-top"><p class="tableblock">send data via site-to-site</p></td> +<td class="tableblock halign-left valign-top"><p class="tableblock">Allows a port to send data from NiFi instances</p></td> +</tr> +</tbody> +</table> +<div class="paragraph"> +<p>If you are unable to view or modify a NiFi resource, contact your System Administrator or see Configuring Users and Access Policies in the +<a href="administration-guide.html">System Administratorâs Guide</a> for more information.</p> </div> </div> </div> @@ -734,14 +864,9 @@ are in the cluster and how many are curr <div class="sectionbody"> <div class="paragraph"> <p>If NiFi is configured to run securely, users will be able to request access to the DataFlow. For information on configuring NiFi to run -securely, see the <a href="administration-guide.html">Admin Guide</a>. If NiFi supports anonymous access, users will be given access +securely, see the <a href="administration-guide.html">System Administratorâs Guide</a>. If NiFi supports anonymous access, users will be given access accordingly and given an option to log in.</p> </div> -<div class="imageblock"> -<div class="content"> -<img src="./images/anonymous-access.png" alt="Anonymous Access"> -</div> -</div> <div class="paragraph"> <p>Clicking the <em>login</em> link will open the log in page. If the user is logging in with their username/password they will be presented with a form to do so. If NiFi is not configured to support anonymous access and the user is logging in with their username/password, they will @@ -752,35 +877,20 @@ be immediately sent to the login form by <img src="./images/login.png" alt="Log In"> </div> </div> -<div class="paragraph"> -<p>Once the user has logged in or if they are accessing NiFi using a client certificate loaded in their browser, they will be prompted -to request access by submitting a justification if this is the first time they have accessed this NiFi. Fill in an optional justification -that the administrator will review while granting the account access. If NiFi is not configured to support anonymous access and the -user is using a client certificate, they will be immediately sent to the form to request access bypassing the canvas and login form.</p> -</div> -<div class="imageblock"> -<div class="content"> -<img src="./images/request-access.png" alt="Request Access"> -</div> -</div> -<div class="paragraph"> -<p>Press Submit to send the account request. If NiFi supports anonymous access, the user can continue accessing the DataFlow by closing the -login page. Returning to the login page will check the status of the account request. If access has been granted, press the home link or -reload the page to assume the new roles.</p> -</div> </div> </div> <div class="sect1"> <h2 id="building-dataflow"><a class="anchor" href="#building-dataflow"></a>Building a DataFlow</h2> <div class="sectionbody"> <div class="paragraph"> -<p>A DFM is able to build an automated dataflow using the NiFi User Interface (UI). Simply drag components from the toolbar to the canvas, configure the components to meet specific needs, and connect +<p>A DFM is able to build an automated dataflow using the NiFi UI. Simply drag components from the toolbar to the canvas, +configure the components to meet specific needs, and connect the components together.</p> </div> <div class="sect2"> <h3 id="adding-components-to-the-canvas"><a class="anchor" href="#adding-components-to-the-canvas"></a>Adding Components to the Canvas</h3> <div class="paragraph"> -<p>In the User Interface section above outlined the different segments of the UI and pointed out a Components Toolbar. +<p>The User Interface section above outlined the different segments of the UI and pointed out a Components Toolbar. This section looks at each of the Components in that toolbar:</p> </div> <div class="imageblock"> @@ -818,18 +928,21 @@ Processors that allow us to ingest data location that it was dropped.</p> </div> <div class="paragraph"> -<p><strong>Note</strong>: For any component added to the canvas, it is possible to select it with the mouse and move it anywhere on the canvas. Also, it is possible to select multiple items at once by either holding down the Shift key and selecting each item or by holding down the Shift key and dragging a selection box around the desired components.</p> +<p><strong>Note</strong>: For any component added to the canvas, it is possible to select it with the mouse and move it anywhere on the canvas. +Also, it is possible to select multiple items at once by either holding down the Shift key and selecting each item or by holding +down the Shift key and dragging a selection box around the desired components.</p> </div> <div class="paragraph"> -<p>Once a Processor has been dragged onto the canvas, the user may interact with it by right-clicking on the Processor and selecting an option from the context menu.</p> +<p>Once you have dragged a Processor onto the canvas, you can interact with it by right-clicking on the Processor and +selecting an option from the context menu. The options available to you from the context menu vary, depending on the privileges assigned to you.</p> </div> <div class="imageblock"> <div class="content"> -<img src="./images/nifi-processor-menu.png" alt="Processor Menu" width="300"> +<img src="./images/nifi-processor-menu.png" alt="Processor Menu"> </div> </div> <div class="paragraph"> -<p>The following options are available:</p> +<p>While the options available from the context menu vary, the following options are typically available when you have full privileges to work with a Processor:</p> </div> <div class="ulist"> <ul> @@ -843,10 +956,7 @@ location that it was dropped.</p> <p><strong>Stats</strong>: This option opens a graphical representation of the Processor’s statistical information over time.</p> </li> <li> -<p><strong>Upstream connections</strong>: This option allows the user to see and "jump to" upstream connections that are coming into the Processor. This is particularly useful when processors connect into and out of other Process Groups.</p> -</li> -<li> -<p><strong>Downstream connections</strong>: This option allows the user to see and "jump to" downstream connections that are going out of the Processor. This is particularly useful when processors connect into and out of other Process Groups.</p> +<p><strong>Data provenance</strong>: This option displays the NiFi Data Provenance table, with information about data provenance events for the FlowFiles routed through that Processor</p> </li> <li> <p><strong>Usage</strong>: This option takes the user to the Processor’s usage documentation.</p> @@ -876,7 +986,7 @@ in the Root Process Group. If the Input to receive data from remote instances of NiFi via <a href="#site-to-site">Site-to-Site</a>. In this case, the Input Port can be configured to restrict access to appropriate users, if NiFi is configured to run securely. For information on configuring NiFi to run securely, see the -<a href="administration-guide.html">Admin Guide</a>.</p> +<a href="administration-guide.html">System Administratorâs Guide</a>.</p> </div> <div id="output_port" class="paragraph"> <p><span class="image"><img src="./images/iconOutputPort.png" alt="Output Port" width="32"></span> @@ -890,7 +1000,7 @@ remote instances of NiFi via <a href="#s of NiFi pull data from the port, that data is removed from the queues of the incoming Connections. If NiFi is configured to run securely, the Output Port can be configured to restrict access to appropriate users. For information on configuring NiFi to run securely, see the -<a href="administration-guide.html">Admin Guide</a>.</p> +<a href="administration-guide.html">System Administratorâs Guide</a>.</p> </div> <div id="process_group" class="paragraph"> <p><span class="image"><img src="./images/iconProcessGroup.png" alt="Process Group" width="32"></span> @@ -899,16 +1009,16 @@ and maintain. When a Process Group is dr Groups within the same parent group must have unique names. The Process Group will then be nested within that parent group.</p> </div> <div class="paragraph"> -<p>Once a Process Group has been dragged onto the canvas, the user may interact with it by right-clicking on the Process Group and selecting an option from -context menu.</p> +<p>Once you have dragged a Process Group onto the canvas, you can interact with it by right-clicking on the Process Group and selecting an option from +context menu.The options available to you from the context menu vary, depending on the privileges assigned to you.</p> </div> <div class="imageblock"> <div class="content"> -<img src="./images/nifi-process-group-menu.png" alt="Process Group Menu" width="300"> +<img src="./images/nifi-process-group-menu.png" alt="Process Group Menu"> </div> </div> <div class="paragraph"> -<p>The following options are available:</p> +<p>While the options available from the context menu vary, the following options are typically available when you have full privileges to work with the Process Group:</p> </div> <div class="ulist"> <ul> @@ -928,12 +1038,6 @@ context menu.</p> <p><strong>Stats</strong>: This option opens a graphical representation of the Process Group’s statistical information over time.</p> </li> <li> -<p><strong>Upstream connections</strong>: This option allows the user to see and "jump to" upstream connections that are coming into the Process Group.</p> -</li> -<li> -<p><strong>Downstream connections</strong>: This option allows the user to see and "jump to" downstream connections that are going out of the Process Group.</p> -</li> -<li> <p><strong>Center in view</strong>: This option centers the view of the canvas on the given Process Group.</p> </li> <li> @@ -957,15 +1061,15 @@ recalculate the load balancing based on </div> <div class="paragraph"> <p>Once a Remote Process Group has been dragged onto the canvas, the user may interact with it by right-clicking on the Remote Process Group and selecting an option from -context menu.</p> +context menu. The options available to you from the context menu vary, depending on the privileges assigned to you.</p> </div> <div class="imageblock"> <div class="content"> -<img src="./images/nifi-rpg-menu.png" alt="Remote Process Group Menu" width="300"> +<img src="./images/nifi-rpg-menu.png" alt="Remote Process Group Menu"> </div> </div> <div class="paragraph"> -<p>The following options are available:</p> +<p>While the options available from the context menu vary, the following options are typically available when you have full privileges to work with the Remote Process Group:</p> </div> <div class="ulist"> <ul> @@ -1008,7 +1112,7 @@ context menu.</p> </ul> </div> <div id="funnel" class="paragraph"> -<p><span class="image"><img src="./images/iconFunnel.png" alt="Funnel" width="32"></span> +<p><span class="image"><img src="./images/iconFunnel.png" alt="Funnel"></span> <strong>Funnel</strong>: Funnels are used to combine the data from many Connections into a single Connection. This has two advantages. First, if many Connections are created with the same destination, the canvas can become cluttered if those Connections have to span a large space. By funneling these Connections into a single Connection, that single Connection can then be @@ -1017,7 +1121,7 @@ several Connections can be funneled into one Connection, rather than prioritizing the data on each Connection independently.</p> </div> <div id="template" class="paragraph"> -<p><span class="image"><img src="./images/iconTemplate.png" alt="Template" width="32"></span> +<p><span class="image"><img src="./images/iconTemplate.png" alt="Template"></span> <strong>Template</strong>: Templates can be created by DFMs from sections of the flow, or they can be imported from other dataflows. These Templates provide larger building blocks for creating a complex flow quickly. When the Template is dragged onto the canvas, the DFM is provided a dialog to choose which Template to add to the canvas:</p> @@ -1142,8 +1246,8 @@ the administrator has configured.</p> Timer driven scheduling mode. However, the CRON driven mode provides significantly more flexibility at the expense of increasing the complexity of the configuration. This value is made up of six fields, each separated by a space. These fields include:</p> -<div class="olist arabic"> -<ol class="arabic"> +<div class="ulist"> +<ul> <li> <p>Seconds</p> </li> @@ -1165,7 +1269,10 @@ fields include:</p> <li> <p>Year</p> </li> -</ol> +</ul> +</div> +</li> +</ul> </div> <div class="paragraph"> <p>The value for each of these fields should be a number, range, or increment. @@ -1183,9 +1290,6 @@ values are 0, 10, 20, 30, 40, and 50. Ho <p>For the Day of Week field, valid values are 1 (Sunday) through 7 (Saturday). Additionally, a value of <code>L</code> may be appended to one of these values to indicate the last occurrence of this day in the month. For example, <code>1L</code> can be used to indicate the last Monday of the month.</p> </div> -</li> -</ul> -</div> <div class="paragraph"> <p>Next, the Scheduling Tab provides a configuration option named ‘Concurrent tasks.’ This controls how many threads the Processor will use. Said a different way, this controls how many FlowFiles should be processed by this Processor at the same time. Increasing @@ -1259,7 +1363,7 @@ In fact, this Processor will not be vali <p>Some processors also have an Advanced User Interface (UI) built into them. For example, the UpdateAttribute processor has an Advanced UI. To access the Advanced UI, click the <code>Advanced</code> button that appears at the bottom of the Configure Processor window. Only processors that have an Advanced UI will have this button.</p> </div> <div class="paragraph"> -<p>Some processors have properties that refer to other components, such as Controller Services, which also need to be configured. For example, the GetHTTP processor has an SSLContextService property, which refers to the StandardSSLContextService controller service. When DFMs want to configure this property but have not yet created and configured the controller service, they have the option to create the service on the spot, as depicted in the image below. For more information about configuring Controller Services, see the <a href="#Controller_Services_and_Reporting_Tasks">Controller Services and Reporting Tasks</a> section.</p> +<p>Some processors have properties that refer to other components, such as Controller Services, which also need to be configured. For example, the GetHTTP processor has an SSLContextService property, which refers to the StandardSSLContextService controller service. When DFMs want to configure this property but have not yet created and configured the controller service, they have the option to create the service on the spot, as depicted in the image below. For more information about configuring Controller Services, see the <a href="#Controller_Services_and_Reporting_Tasks">[Controller_Services_and_Reporting_Tasks]</a> section.</p> </div> <div class="paragraph"> <p><span class="image"><img src="./images/create-service-ssl-context.png" alt="Create Service" width="700"></span></p> @@ -1281,72 +1385,207 @@ whatever comments are appropriate for th <div class="sect2"> <h3 id="additional-help"><a class="anchor" href="#additional-help"></a>Additional Help</h3> <div class="paragraph"> -<p>The user may access additional documentation about each Processor’s usage by right-clicking -on the Processor and then selecting ‘Usage’ from the context menu. Alternatively, clicking the ‘Help’ link in the top-right -corner of the User Interface will provide a Help page with all of the documentation, including usage documentation -for all the Processors that are available. Clicking on the desired Processor in the list will display its usage documentation.</p> +<p>You can access additional documentation about each Processor’s usage by right-clicking +on the Processor and selecting ‘Usage’ from the context menu. Alternatively, select Help from the Global Menu in the top-right +corner of the UI to display a Help page with all of the documentation, including usage documentation +for all the Processors that are available. Click on the desired Processor to view usage documentation.</p> </div> </div> <div class="sect2"> -<h3 id="Controller_Services_and_Reporting_Tasks"><a class="anchor" href="#Controller_Services_and_Reporting_Tasks"></a>Controller Services and Reporting Tasks</h3> +<h3 id="Using_Custom_Properties"><a class="anchor" href="#Using_Custom_Properties"></a>Using Custom Properties with Expression Language</h3> <div class="paragraph"> -<p>While DFMs have the ability to create Controller Services from the Configure Processor window, there is also a central place within the User Interface for adding and configuring both Controller Services and Reporting Tasks. To get there, click on the Controller Settings button in the Management section of the toolbar.</p> +<p>You can use NiFi Expression Language to reference FlowFile attributes, compare them to other values, +and manipulate their values when you are creating and configuring dataflows.</p> +</div> +<div class="paragraph"> +<p>In addition to using FlowFile attributes, system properties, and environment properties within Express +Language, you can also define custom properties for Expression Language use. Defining custom properties +gives you more flexibility in handling and processing dataflows. You can also create custom properties +for connection, server, and service properties, for easier dataflow configuration.</p> +</div> +<div class="paragraph"> +<p>To create custom properties for use with Expression Language, identify one or more sets of key/value +pairs, and give them to your system administrator.</p> +</div> +<div class="paragraph"> +<p>NiFi properties have resolution precedence of which you should be aware when creating custom properties:â¨</p> +</div> +<div class="ulist"> +<ul> +<li> +<p>Processor-specific attributes</p> +</li> +<li> +<p>FlowFile propertiesâ¨</p> +</li> +<li> +<p>FlowFile attributesâ¨</p> +</li> +<li> +<p>From variable registry:â¨</p> +<div class="ulist"> +<ul> +<li> +<p>User defined properties (custom properties)</p> +</li> +<li> +<p>System propertiesâ¨</p> +</li> +<li> +<p>Operating System environment variables</p> +</li> +</ul> +</div> +</li> +</ul> +</div> +<div class="paragraph"> +<p>When you are creating custom properties, ensure that each custom property contains a distinct property value, +so that it is not overridden by existing environment properties, system properties, or FlowFile attributes. +Once you have added the new custom properties, ensure that you have updated the nifi.variable.registry.properties +field in the nifi.properties file, with the custom properties location.</p> +</div> +<div class="paragraph"> +<p>For more information on Expression Language, see the <a href="expression-language-guide.html">Expression Lanuaguage Guide</a>. +For information on how to define custom properties, see the <a href="administration-guide.html">System Administratorâs Guide</a>.</p> +</div> +</div> +<div class="sect2"> +<h3 id="Controller_Services"><a class="anchor" href="#Controller_Services"></a>Controller Services</h3> +<div class="paragraph"> +<p>Controller Services are available for reporting tasks, processors, and other services to utilize for +configuration or task execution. You can use the NiFi UI to add Controller Services for either reporting +tasks or dataflows.</p> +</div> +<div class="paragraph"> +<p>Your ability to view and add Controller Services is dependent on the roles and privileges assigned to you. If you +do not have access to one or more Controller Services, you are not able to see or access it in the UI. Roles and +privileges can be assigned on a global or Controller Service-specific basis.</p> +</div> +<div class="paragraph"> +<p>Controller Services are not reporting task or dataflow specific. You have access to the full set of available Controller +Services whether you are adding it for a reporting task or a dataflow.</p> </div> <div class="sect3"> -<h4 id="Controller_Settings"><a class="anchor" href="#Controller_Settings"></a>Controller Settings</h4> +<h4 id="Controller_Services_for_Reporting_Tasks"><a class="anchor" href="#Controller_Services_for_Reporting_Tasks"></a>Adding Controller Settings for Reporting Tasks</h4> <div class="paragraph"> -<p><span class="image"><img src="./images/controller-settings-button.png" alt="Controller Settings Button" width="200"></span></p> +<p>To add a Controller Service for a reporting task, select Controller Settings from the Global Menu. +This displays the NiFi Settings window.</p> </div> <div class="paragraph"> -<p>The Controller Settings window has three tabs across the top: General, Controller Services, and Reporting Tasks. The General tab is for settings that pertain to general information about the NiFi instance. For example, here, the DFM can provide a unique name for the overall dataflow, as well as comments that describe the flow. Be aware that this information is visible to any other NiFi instance that connects remotely to this instance (using Remote Process Groups, a.k.a., Site-to-Site).</p> +<p>The NiFi Settings window has three tabs across the top: General, Controller Services, and Reporting Tasks. The General tab is +for settings that pertain to general information about the NiFi instance. For example, here, the DFM can provide a unique name for +the overall dataflow, as well as comments that describe the flow. Be aware that this information is visible to any other NiFi instance +that connects remotely to this instance (using Remote Process Groups, a.k.a., Site-to-Site).</p> </div> <div class="paragraph"> -<p>The General tab also provides settings for the overall maximum thread counts of the instance, as well as the ability to click "Back-up flow" to create a backup copy of the current flow, which is saved by default in the /conf/archive directory.</p> +<p>The General tab also provides settings for the overall maximum thread counts of the instance.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/settings-general-tab.png" alt="Controller Settings General Tab" width="700"></span></p> +<p><span class="image"><img src="./images/settings-general-tab.png" alt="Controller Settings General Tab"></span></p> </div> <div class="paragraph"> -<p>To the right of the General tab is the Controller Services tab. From this tab, the DFM may click the "+" button in the upper-right corner to create a new Controller Service.</p> +<p>To the right of the General tab is the Controller Services tab. From this tab, the DFM may click the "+" button in the upper-right +corner to create a new Controller Service.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/controller-services-tab.png" alt="Controller Services Tab" width="900"></span></p> +<p><span class="image"><img src="./images/controller-services-tab.png" alt="Controller Services Tab"></span></p> </div> <div class="paragraph"> -<p>The Add Controller Service window opens. This window is similar to the Add Processor window. It provides a list of the available Controller Services on the right and a tag cloud, showing the most common category tags used for Controller Services, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Controller Services to those that fit the categories desired. The DFM may also use the Filter field at the top of the window to search for the desired Controller Service. Upon selecting a Controller Service from the list, the DFM can see a description of the the service below. Select the desired controller service and click Add, or simply double-click the name of the service to add it.</p> +<p>The Add Controller Service window opens. This window is similar to the Add Processor window. It provides a list of the +available Controller Services on the right and a tag cloud, showing the most common category tags used for Controller +Services, on the left. The DFM may click any tag in the tag cloud in order to narrow down the list of Controller Services +to those that fit the categories desired. The DFM may also use the Filter field at the top of the window to search +for the desired Controller Service. Upon selecting a Controller Service from the list, the DFM can see a description of +the the service below. Select the desired controller service and click Add, or simply double-click the name of the service +to add it.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/add-controller-service-window.png" alt="Add Controller Service Window" width="700"></span></p> +<p><span class="image"><img src="./images/add-controller-service-window.png" alt="Add Controller Service Window"></span></p> </div> <div class="paragraph"> -<p>Once a Controller Service has been added, the DFM may configure it by clicking the Edit button (pencil icon) in the far-right column. Other buttons in this column include the Enable button (to enable a configured service), the Remove button, and the Usage button, which links to the documentation for the particular Controller Service.</p> +<p>Once you have added a Controller Service, you can configure it by clicking the Edit button in the +far-right column. Other buttons in this column include Remove and Access Policies.</p> </div> <div class="paragraph"> <p><span class="image"><img src="./images/controller-services-edit-buttons.png" alt="Controller Services Buttons"></span></p> </div> <div class="paragraph"> -<p>When the DFM clicks the Edit button, a Configure Controller Service window opens. It has three tabs: Settings, Properties, and Comments. This window is similar to the Configure Processor window. The Settings tab provides a place for the DFM to give the Controller Service a unique name (if desired). It also lists the UUID for the service and provides a list of other components (processors or other controller services) that reference the service.</p> +<p>You can obtain information about Controller Services by clicking the Details, Usage, and Alerts buttons in the left-hand column.</p> +</div> +<div class="paragraph"> +<p><span class="image"><img src="./images/controller-services-information-buttons.png" alt="Controller Services Information Buttons"></span></p> +</div> +<div class="paragraph"> +<p>When the DFM clicks the Edit button, a Configure Controller Service window opens. It has three tabs: Settings, Properties, +and Comments. This window is similar to the Configure Processor window. The Settings tab provides a place for the DFM +to give the Controller Service a unique name (if desired). It also lists the UUID for the service and provides a list +of other components (processors or other controller services) that reference the service.</p> +</div> +<div class="paragraph"> +<p><span class="image"><img src="./images/configure-controller-service-settings.png" alt="Configure Controller Service Settings"></span></p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/configure-controller-service-settings.png" alt="Configure Controller Service Settings" width="700"></span></p> +<p>The Properties tab lists the various properties that apply to the particular controller service. As with configuring +processors, the DFM may hover the over the question mark icons to see more information about each property.</p> </div> <div class="paragraph"> -<p>The Properties tab lists the various properties that apply to the particular controller service. As with configuring processors, the DFM may hover the mouse over the question mark icons to see more information about each property.</p> +<p><span class="image"><img src="./images/configure-controller-service-properties.png" alt="Configure Controller Service Properties"></span></p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/configure-controller-service-properties.png" alt="Configure Controller Service Properties" width="700"></span></p> +<p>The Comments tab is just an open-text field, where the DFM may include comments about the service. After configuring +a Controller Service, click the Apply button to apply the configuration and close the window, or click the Cancel +button to cancel the changes and close the window.</p> </div> <div class="paragraph"> -<p>The Comments tab is just an open-text field, where the DFM may include comments about the service. After configuring a Controller Service, click the Apply button to apply the configuration and close the window, or click the Cancel button to cancel the changes and close the window.</p> +<p>Note that after a Controller Service has been configured, it must be enabled in order to run. Do this using the +Enable button in the far-right column of the Controller Services tab of the Controller Settings window. Then, +in order to modify an existing/running controller service, the DFM needs to stop/disable it (as well as all referencing processors, reporting tasks, and controller services). Rather than having to hunt down each component that is referenced by that controller service, the DFM has the ability to stop/disable them when disabling the controller service in question. Likewise, when enabling a controller service, the DFM has the option to start/enable all referencing processors, reporting tasks, and controller services.</p> </div> +</div> +<div class="sect3"> +<h4 id="Controller_Services_for_Dataflows"><a class="anchor" href="#Controller_Services_for_Dataflows"></a>Adding Controller Services for Dataflows</h4> <div class="paragraph"> -<p>Note that after a Controller Service has been configured, it must be enabled in order to run. Do this using the Enable button in the far-right column of the Controller Services tab of the Controller Settings window. Then, in order to modify an existing/running controller service, the DFM needs to stop/disable it (as well as all referencing processors, reporting tasks, and controller services). Rather than having to hunt down each component that is referenced by that controller service, the DFM has the ability to stop/disable them when disabling the controller service in question. Likewise, when enabling a controller service, the DFM has the option to start/enable all referencing processors, reporting tasks, and controller services.</p> +<p>To add a Controller Service for a dataflow, you can either right click a Process Group and select Configure, or click +Configure from the Operate Palette. When you click Configure from the Operate Palette with nothing selected on your +canvas, you add a Controller Service for your root Process Group. That Controller Service is then available to all +nested Process Groups in your dataflow. When you select a Process Group on the canvas and then click Configure from +either the Operate Palette or the Process Group context menu, you add a Controller Service only for use with the selected +Process Group.</p> </div> <div class="paragraph"> +<p>In either case, use the following steps to add a Controller Service:</p> +</div> +<div class="olist arabic"> +<ol class="arabic"> +<li> +<p>Click Configure, either from the Operate Palette, or from the Process Group context menu.</p> +</li> +<li> +<p>From the Process Group Configuration page, select the Controller Services tab.</p> +</li> +<li> +<p>Click the Add button to display the Add Controller Service dialog.</p> +</li> +<li> +<p>Select the Controller Service you want to add, and click Add.</p> +</li> +<li> +<p>Perform any necessary Controller Service configuration tasks by clicking the View Details icon ( +<span class="image"><img src="./images/iconViewDetails.png" alt="View Details"></span> +) in the left-hand column.</p> +</li> +</ol> +</div> +</div> +</div> +<div class="sect2"> +<h3 id="Reporting_Tasks"><a class="anchor" href="#Reporting_Tasks"></a>Reporting Tasks</h3> +<div class="paragraph"> <p>The Reporting Tasks tab behaves similarly to the Controller Services tab. The DFM has the option to add Reporting Tasks and configure them in the same way as Controller Services.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/reporting-tasks-tab.png" alt="Reporting Tasks Tab" width="900"></span></p> +<p><span class="image"><img src="./images/reporting-tasks-tab.png" alt="Reporting Tasks Tab"></span></p> </div> <div class="paragraph"> <p>Once a Reporting Task has been added, the DFM may configure it by clicking the Edit (pencil icon) in the far-right column. Other buttons in this column include the Start button, Remove button, and Usage button, which links to the documentation for the particular Reporting Task.</p> @@ -1358,13 +1597,13 @@ for all the Processors that are availabl <p>When the DFM clicks the Edit button, a Configure Reporting Task window opens. It has three tabs: Settings, Properties, and Comments. This window is also similar to the Configure Processor window. The Settings tab provides a place for the DFM to give the Reporting Task a unique name (if desired). It also lists a UUID for the Reporting Task and provides settings for the task’s Scheduling Strategy and Run Schedule (similar to the same settings in a processor). The DFM may hover the mouse over the question mark icons to see more information about each setting.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/configure-reporting-task-settings.png" alt="Configure Reporting Task Settings" width="700"></span></p> +<p><span class="image"><img src="./images/configure-reporting-task-settings.png" alt="Configure Reporting Task Settings"></span></p> </div> <div class="paragraph"> <p>The Properties tab for a Reporting Task lists the properties that may be configured for the task. The DFM may hover the mouse over the question mark icons to see more information about each property.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/configure-reporting-task-properties.png" alt="Configure Reporting Task Properties" width="700"></span></p> +<p><span class="image"><img src="./images/configure-reporting-task-properties.png" alt="Configure Reporting Task Properties"></span></p> </div> <div class="paragraph"> <p>The Comments tab is just an open-text field, where the DFM may include comments about the task. After configuring the Reporting Task, click the Apply button to apply the configuration and close the window, or click Cancel to cancel the changes and close the window.</p> @@ -1373,7 +1612,6 @@ for all the Processors that are availabl <p>When you want to run the Reporting Task, click the Start button in the far-right column of the Reporting Tasks tab.</p> </div> </div> -</div> <div class="sect2"> <h3 id="Connecting_Components"><a class="anchor" href="#Connecting_Components"></a>Connecting Components</h3> <div class="paragraph"> @@ -1391,7 +1629,8 @@ Connection between each component. When releases the mouse, a ‘Create Connection’ dialog appears. This dialog consists of two tabs: ‘Details’ and ‘Settings’. They are discussed in detail below. Note that it is possible to draw a connection so that it loops back on the same processor. This can be useful if the DFM wants the processor to try to re-process FlowFiles if they go down a failure Relationship. To create this type of looping -connection, simply drag the connection bubble away and then back to the same processor until it is highlighted. Then release the mouse and the same <em>Create Connection</em> dialog appears.</p> +connection, simply drag the connection bubble away and then back to the same processor until it is highlighted. Then release the mouse +and the same <em>Create Connection</em> dialog appears.</p> </div> <div class="sect3"> <h4 id="details-tab"><a class="anchor" href="#details-tab"></a>Details Tab</h4> @@ -1474,13 +1713,13 @@ prioritizers’ list to the ‘A <p><strong>Note</strong>: After a connection has been drawn between two components, the connection’s configuration may be changed, and the connection may be moved to a new destination; however, the processors on either side of the connection must be stopped before a configuration or destination change may be made.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/nifi-connection.png" alt="Connection" width="300"></span></p> +<p><span class="image"><img src="./images/nifi-connection.png" alt="Connection"></span></p> </div> <div class="paragraph"> <p>To change a connection’s configuration or interact with the connection in other ways, right-click on the connection to open the connection context menu.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/nifi-connection-menu.png" alt="Connection Menu" width="200"></span></p> +<p><span class="image"><img src="./images/nifi-connection-menu.png" alt="Connection Menu"></span></p> </div> <div class="paragraph"> <p>The following options are available:</p> @@ -1519,7 +1758,7 @@ the bend point and drag it so that the c bend point. To remove a bend point, simply double-click it again.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/nifi-connection-bend-points.png" alt="Connection Bend Points" width="500"></span></p> +<p><span class="image"><img src="./images/nifi-connection-bend-points.png" alt="Connection Bend Points"></span></p> </div> </div> </div> @@ -1528,7 +1767,7 @@ bend point. To remove a bend point, simp <div class="paragraph"> <p>Before trying to start a Processor, it’s important to make sure that the Processor’s configuration is valid. A status indicator is shown in the top-left of the Processor. If the Processor is invalid, the indicator -will show a yellow Warning indicator with an exclamation mark indicating that there is a problem:</p> +will show a red Warning indicator with an exclamation mark indicating that there is a problem:</p> </div> <div class="imageblock"> <div class="content"> @@ -1577,7 +1816,7 @@ one NiFi instance or data producing appl to allow only specific users, and only those users will be able to see that the port even exists. For information on configuring the Certificates, see the <a href="administration-guide.html#security-configuration">Security Configuration</a> section of the -<a href="administration-guide.html">Admin Guide</a>.</p> +<a href="administration-guide.html">System Administratorâs Guide</a>.</p> </li> </ul> </div> @@ -1654,7 +1893,28 @@ or deficiency is ever discovered in a pr </ul> </div> <div class="paragraph"> -<p>In order to communicate with a remote NiFi instance via Site-to-Site, simply drag a <a href="#remote_process_group">Remote Process Group</a> onto the canvas +<p>Site-to-Site is a protocol transferring data between two NiFi instances. Both end can be a standalone NiFi or a NiFi cluster. In this section, the NiFi instance initiates the communications is called <em>Site-to-Site client NiFi instance</em> and the other end as <em>Site-to-Site server NiFi instance</em> to clarify what configuration needed on each NiFi instances.</p> +</div> +<div class="paragraph"> +<p>A NiFi instance can be both client and server for Site-to-Site protocol, however, it can only be a client or server within a specific Site-to-Site communication. For example, if there are three NiFi instances A, B and C. A pushes data to B, and B pulls data from C. <em>A — push → B ← pull — C</em>. Then B is not only a <em>server</em> in the communication between A and B, but also a <em>client</em> in B and C.</p> +</div> +<div class="paragraph"> +<p>It is important to understand which NiFi instance will be the client or server in order to design your data flow, and configure each instance accordingly. Here is a summary of what components run on which side based on data flow direction:</p> +</div> +<div class="ulist"> +<ul> +<li> +<p>Push: a client <em>sends</em> data to a Remote Process Group, the server <em>receives</em> it with an Input Port</p> +</li> +<li> +<p>Pull: a client <em>receives</em> data from a Remote Process Group, the server <em>sends</em> data through an Output Port</p> +</li> +</ul> +</div> +<div class="sect3"> +<h4 id="configure-site-to-site-client-nifi-instance"><a class="anchor" href="#configure-site-to-site-client-nifi-instance"></a>Configure Site-to-Site client NiFi instance</h4> +<div id="Site-to-Site_Remote_Process_Group" class="paragraph"> +<p><strong>Remote Process Group</strong>: In order to communicate with a remote NiFi instance via Site-to-Site, simply drag a <a href="#remote_process_group">Remote Process Group</a> onto the canvas and enter the URL of the remote NiFi instance (for more information on the components of a Remote Process Group, see <a href="#Remote_Group_Transmission">Remote Process Group Transmission</a> section of this guide.) The URL is the same URL you would use to go to that instance’s User Interface. At that point, you can drag a connection to or from the Remote Process Group @@ -1670,10 +1930,22 @@ the ports shown will be the Input Ports <div class="paragraph"> <p><strong>Note</strong>: if the remote instance is configured to use secure data transmission, you will see only ports that you are authorized to communicate with. For information on configuring NiFi to run securely, see the -<a href="administration-guide.html">Admin Guide</a>.</p> +<a href="administration-guide.html">System Administratorâs Guide</a>.</p> +</div> +<div id="Site-to-Site_Transport_Protocol" class="paragraph"> +<p><strong>Transport Protocol</strong>: On a Remote Process Group creation or configuration dialog, you can choose Transport Protocol to use for Site-to-Site communication as shown in the following image:</p> +</div> +<div class="paragraph"> +<p><span class="image"><img src="./images/configure-remote-process-group.png" alt="Configure Remote Process Group"></span></p> </div> <div class="paragraph"> -<p>In order to allow another NiFi instance to push data to your local instance, you can simply drag an <a href="#input_port">Input Port</a> onto the Root Process Group +<p>By default, it is set to <em>RAW</em> which uses raw socket communication using a dedicated port. <em>HTTP</em> transport protocol is especially useful if the remote NiFi instance is in a restricted network that only allow access through HTTP(S) protocol or only accessible from a specific HTTP Proxy server. For accessing through a HTTP Proxy Server, BASIC and DIGEST authentication are supported.</p> +</div> +</div> +<div class="sect3"> +<h4 id="configure-site-to-site-server-nifi-instance"><a class="anchor" href="#configure-site-to-site-server-nifi-instance"></a>Configure Site-to-Site server NiFi instance</h4> +<div id="Site-to-Site_Input_Port" class="paragraph"> +<p><strong>Input Port</strong>: In order to allow another NiFi instance to push data to your local instance, you can simply drag an <a href="#input_port">Input Port</a> onto the Root Process Group of your canvas. After entering a name for the port, it will be added to your flow. You can now right-click on the Input Port and choose Configure in order to adjust the name and the number of concurrent tasks that are used for the port. If Site-to-Site is configured to run securely, you will also be given the ability to adjust who has access to the port. If secure, only those who have been granted access to communicate with the port will be able to see @@ -1683,8 +1955,8 @@ that the port exists.</p> <p>After being given access to a particular port, in order to see that port, the operator of a remote NiFi instance may need to right-click on their Remote Process Group and choose to "Refresh" the flow.</p> </div> -<div class="paragraph"> -<p>Similar to an Input Port, a DataFlow Manager may choose to add an <a href="#output_port">Output Port</a> to the Root Process Group. The Output Port allows an +<div id="Site-to-Site_Output_Port" class="paragraph"> +<p><strong>Output Port</strong>: Similar to an Input Port, a DataFlow Manager may choose to add an <a href="#output_port">Output Port</a> to the Root Process Group. The Output Port allows an authorized NiFi instance to remotely connect to your instance and pull data from the Output Port. Configuring the Output Port will again allow the DFM to control how many concurrent tasks are allowed, as well as which NiFi instances are authorized to pull data from the instance being configured.</p> </div> @@ -1692,8 +1964,8 @@ DFM to control how many concurrent tasks <p>In addition to other instances of NiFi, some other applications may use a Site-to-Site client in order to push data to or receive data from a NiFi instance. For example, NiFi provides an Apache Storm spout and an Apache Spark Receiver that are able to pull data from NiFi’s Root Group Output Ports.</p> </div> -<div class="paragraph"> -<p>If your instance of NiFi is running securely, the first time that a client establishes a connection to your instance, the client will be forbidden and +<div id="Site-to-Site_Access_Control" class="paragraph"> +<p><strong>Access Control</strong>: If your instance of NiFi is running securely, the first time that a client establishes a connection to your instance, the client will be forbidden and a request for an account for that client will automatically be generated. The client will need to be granted the <em>NiFi</em> role in order to communicate via Site-to-Site. For more information on managing user accounts, see the <a href="administration-guide.html#controlling-levels-of-access">Controlling Levels of Access</a> @@ -1702,7 +1974,8 @@ section of the <a href="administration-g <div class="paragraph"> <p>For information on how to enable and configure Site-to-Site on a NiFi instance, see the <a href="administration-guide.html#site_to_site_properties">Site-to-Site Properties</a> section of the -<a href="administration-guide.html">Admin Guide</a>.</p> +<a href="administration-guide.html">System Administratorâs Guide</a>.</p> +</div> </div> </div> <div class="sect2"> @@ -1750,7 +2023,7 @@ to build a quick flow for demonstration </div> <div class="imageblock"> <div class="content"> -<img src="./images/simple-flow.png" alt="Simple Flow" width="900"> +<img src="./images/simple-flow.png" alt="Simple Flow"> </div> </div> <div class="paragraph"> @@ -2148,16 +2421,6 @@ within the Process Group. The following <div class="ulist"> <ul> <li> -<p><span class="image"><img src="./images/iconInputPortSmall.png" alt="Input Port" width="16"></span> -<strong>Input Ports</strong>: The number of Input Ports that exist directly within this Process Group. This does not include any - Input Ports that exist within child Process Groups, as child groups' ports cannot be accessed directly.</p> -</li> -<li> -<p><span class="image"><img src="./images/iconOutputPortSmall.png" alt="Output Port"></span> -<strong>Output Ports</strong>: The number of Output Ports that exist directly within this Process Group. This does not include any - Output Ports that exist within child Process Groups as child groups' ports cannot be accessed directly.</p> -</li> -<li> <p><span class="image"><img src="./images/iconTransmissionActive.png" alt="Transmission Active"></span> <strong>Transmitting Ports</strong>: The number of Remote Process Group Ports that currently are configured to transmit data to remote instances of NiFi or pull data from remote instances of NiFi.</p> @@ -2251,56 +2514,6 @@ This URL is entered when the Remote Proc and the data is not encrypted while it is transferred between instances of NiFi.</p> </li> <li> -<p><strong>Input Ports</strong>: This section shows three pieces of information:</p> -<div class="ulist"> -<ul> -<li> -<p><span class="image"><img src="./images/iconInputPortSmall.png" alt="Input Ports"></span> -<strong>Input Ports</strong>: The number of Input Ports that are available to send data to on the remote instance of NiFi. - If the remote instance is secure, only the ports to which this instance of NiFi has been granted access - will be counted.</p> -</li> -<li> -<p><span class="image"><img src="./images/iconTransmissionActive.png" alt="Transmitting"></span> -<strong>Transmitting Ports</strong>: The number of Input Ports to which this NiFi is connected and currently configured to - send data to. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <a href="#Remote_Group_Transmission">Remote Process Group Transmission</a>) or via the <a href="#Remote_Port_Configuration">Individual Port Transmission</a> dialog.</p> -</li> -<li> -<p><span class="image"><img src="./images/iconTransmissionInactive.png" alt="Not Transmitting"></span> -<strong>Non-Transmitting Ports</strong>: The number of Input Ports to which this NiFi is connected but is not currently configured - to send data to. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <a href="#Remote_Group_Transmission">Remote Process Group Transmission</a>) or via the <a href="#Remote_Port_Configuration">Individual Port Transmission</a> dialog.</p> -</li> -</ul> -</div> -</li> -<li> -<p><strong>Output Ports</strong>: Similar to the “Input Ports” section above, this element shows three pieces of information:</p> -<div class="ulist"> -<ul> -<li> -<p><span class="image"><img src="./images/iconOutputPortSmall.png" alt="Output Ports"></span> -<strong>Output Ports</strong>: The number of Output Ports that are available to pull data from the remote instance of NiFi. - If the remote instance is secure, only the ports to which this instance of NiFi has been granted access - will be counted.</p> -</li> -<li> -<p><span class="image"><img src="./images/iconTransmissionActive.png" alt="Transmitting"></span> -<strong>Transmitting Ports</strong>: The number of Output Ports from which this NiFi is connected and currently configured - to pull data from. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <a href="#Remote_Group_Transmission">Remote Process Group Transmission</a>) or via the <a href="#Remote_Port_Configuration">Individual Port Transmission</a> dialog.</p> -</li> -<li> -<p><span class="image"><img src="./images/iconTransmissionInactive.png" alt="Not Transmitting"></span> -<strong>Non-Transmitting Ports</strong>: The number of Output Ports to which this NiFi is connected but is not currently configured - to pull data from. Ports can be turned on and off by enabling and disabling transmission on the Remote Process - Group (see <a href="#Remote_Group_Transmission">Remote Process Group Transmission</a>) or via the <a href="#Remote_Port_Configuration">Individual Port Transmission</a> dialog.</p> -</li> -</ul> -</div> -</li> -<li> <p><strong>5-Minute Statistics</strong>: Two statistics are shown for Remote Process Groups: <strong>Sent</strong> and <strong>Received</strong>. Both of these are in the format <count> / <size> where <count> is the number of FlowFiles that have been sent or received in the previous five minutes and <size> is the total size of those FlowFiles' content.</p> @@ -2346,13 +2559,11 @@ are actively running.</p> <div class="paragraph"> <p>While the NiFi canvas is useful for understanding how the configured DataFlow is laid out, this view is not always optimal when trying to discern the status of the system. In order to help the user understand how the DataFlow is functioning -at a higher level, NiFi provides a Summary page. This page is available in the Management Toolbar in the top-right corner +at a higher level, NiFi provides a Summary page. This page is available in the Global Menu in the top-right corner of the User Interface. See the <a href="#User_Interface">NiFi User Interface</a> section for more information about the location of this toolbar.</p> </div> <div class="paragraph"> -<p>The Summary Page is opened by clicking the Summary icon ( -<span class="image"><img src="./images/iconSummary.png" alt="Summary"></span> -) from the Management Toolbar. This opens the Summary table dialog:</p> +<p>The Summary Page is opened by selecting Summary from the Global Menu. This opens the Summary table dialog:</p> </div> <div class="imageblock"> <div class="content"> @@ -2438,7 +2649,7 @@ for more information).</p> </div> <div class="paragraph"> <p>The amount of historical information that is stored is configurable in the NiFi properties but defaults to 24 hours. For specific -configuration information reference the Component Status Repository of the Administration guide. +configuration information reference the Component Status Repository of the <a href="administration-guide.html">System Administratorâs Guide</a>. When the Stats dialog is opened, it provides a graph of historical statistics:</p> </div> <div class="imageblock"> @@ -2520,7 +2731,7 @@ received from others can then be importe <p>To create a Template, select the components that are to be a part of the template, and then click the “Create Template” ( <span class="image"><img src="./images/iconNewTemplate.png" alt="Create Template"></span> -) button in the Actions Toolbar (See <a href="#User_Interface">NiFi User Interface</a> for more information on the Actions Toolbar).</p> +) button in the Operate Palette (See <a href="#User_Interface">NiFi User Interface</a> for more information on the Operate Palette).</p> </div> <div class="paragraph"> <p>Clicking this button without selecting anything will create a Template that contains all of the contents of the @@ -2547,6 +2758,23 @@ destination of the Connection is not als </div> </div> <div class="sect2"> +<h3 id="Import_Template"><a class="anchor" href="#Import_Template"></a>Importing a Template</h3> +<div class="paragraph"> +<p>After receiving a Template that has been exported from another NiFi, the first step needed to use the template is to import +the template into this instance of NiFi. You may import templates into any Process Group where you have the appropriate +authorization.</p> +</div> +<div class="paragraph"> +<p>From the Operate Palette, click the “Upload Template” ( +<span class="image"><img src="./images/iconUploadTemplate.png" alt="Upload Template"></span> +) button (See <a href="#User_Interface">NiFi User Interface</a> for more information on the Operate Palette). This will display the Upload Template +dialog. Click the find icon and use the File Selection dialog to choose which template file to upload. +Select the file and click Open. +Clicking the “Upload” button will attempt to import the Template into this instance of NiFi. +The Upload Template dialog will update to show “Success” or an error message if there was a problem importing the template.</p> +</div> +</div> +<div class="sect2"> <h3 id="instantiating-a-template"><a class="anchor" href="#instantiating-a-template"></a>Instantiating a Template</h3> <div class="paragraph"> <p>Once a Template has been created (see <a href="#Create_Template">Creating a Template</a>) or imported (see <a href="#Import_Template">Importing a Template</a>), it is ready to be @@ -2569,22 +2797,9 @@ longer wanted, it may be deleted.</p> <div class="paragraph"> <p>One of the most powerful features of NiFi Templates is the ability to easily export a Template to an XML file and to import a Template that has already been exported. This provides a very simple mechanism for sharing parts -of a DataFlow with others. The Management Toolbar (see <a href="#User_Interface">NiFi User Interface</a>) has a button for managing Templates ( -<span class="image"><img src="./images/iconTemplate.png" alt="Manage Templates"></span> -). Clicking this button opens a dialog that allows the user to see all of the Templates that are currently available, -filter the templates to see only those of interest, and import, export, and delete Templates.</p> -</div> -<div class="sect3"> -<h4 id="Import_Template"><a class="anchor" href="#Import_Template"></a>Importing a Template</h4> -<div class="paragraph"> -<p>After receiving a Template that has been exported from another NiFi, the first step needed to use the template is to import -the template into this instance of NiFi. From the Template Management dialog (see <a href="#Manage_Templates">Managing Templates</a>), click the -“Browse” button. This provides a File Selection dialog that allows the user to choose which template file to upload. -Select the file and click Open. The “Browse” button is replaced by an “Import” button and a “Cancel” button. -Clicking the “Import” button will attempt to import the Template into this instance of NiFi. If unable to import -the Template, an error message will appear in red next to the “Browse” button. Otherwise, the Template will be -added to the table and the “Browse” button will reappear.</p> -</div> +of a DataFlow with others. You can select Templates from the Global Menu (see <a href="#User_Interface">NiFi User Interface</a>) to open a dialog +that displays all of the Templates that are currently available, +filter the templates to see only those of interest, export, and delete Templates.</p> </div> <div class="sect3"> <h4 id="Export_Template"><a class="anchor" href="#Export_Template"></a>Exporting a Template</h4> @@ -2622,9 +2837,8 @@ like dataflow compliance and optimizatio is configurable.</p> </div> <div class="paragraph"> -<p>To access the Data Provenance page, click the Data Provenance button in the Management Toolbar (see <a href="#User_Interface">NiFi User Interface</a>) -( <span class="image"><img src="./images/iconProvenance.png" alt="Data Provenance" width="28"></span> -). Clicking this button opens a dialog window that allows the user to see the most recent Data Provenance information available, +<p>To access the Data Provenance page, select Data Provenance from the Global Menu. Clicking this button opens a dialog window t +hat allows the user to see the most recent Data Provenance information available, search the information for specific items, and filter the search results. It is also possible to open additional dialog windows to see event details, replay data at any point within the dataflow, and see a graphical representation of the data’s lineage, or path through the flow. (These features are described in depth below.)</p> @@ -2646,7 +2860,7 @@ click the <code>Search</code> button in define for the search. The parameters include the processing event of interest, distinguishing characteristics about the FlowFile or the component that produced the event, the timeframe within which to search, and the size of the FlowFile.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/search-events.png" alt="Search Events" width="400"></span></p> +<p><span class="image"><img src="./images/search-events.png" alt="Search Events"></span></p> </div> <div class="paragraph"> <p>For example, to determine if a particular FlowFile was received, search for an Event Type of "RECEIVE" and include an @@ -2655,7 +2869,7 @@ So, to determine whether a FlowFile with image could be performed:</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/search-receive-event-abc.png" alt="Search for RECEIVE Event" width="400"></span></p> +<p><span class="image"><img src="./images/search-receive-event-abc.png" alt="Search for RECEIVE Event"></span></p> </div> </div> <div class="sect2"> @@ -2697,8 +2911,11 @@ the FlowFile is sent to the connection f <div class="sect2"> <h3 id="viewing-flowfile-lineage"><a class="anchor" href="#viewing-flowfile-lineage"></a>Viewing FlowFile Lineage</h3> <div class="paragraph"> -<p>It is often useful to see a graphical representation of the lineage or path a FlowFile took within the dataflow. To see a FlowFile’s lineage, click on the "Show Lineage" icon ( <span class="image"><img src="./images/iconLineage.png" alt="Show Lineage" width="28"></span> ) in the far-right column -of the Data Provenance table. This opens a graph displaying the FlowFile ( <span class="image"><img src="./images/lineage-flowfile.png" alt="FlowFile" width="32"></span> ) and the various processing events that have occurred. The selected event will be highlighted in yellow. It is possible to right-click on any event to see that event’s details (See <a href="#event_details">Details of an Event</a>). +<p>It is often useful to see a graphical representation of the lineage or path a FlowFile took within the dataflow. To see a FlowFile’s lineage, +click on the "Show Lineage" icon ( <span class="image"><img src="./images/iconLineage.png" alt="Show Lineage" width="28"></span> ) in the far-right column +of the Data Provenance table. This opens a graph displaying the FlowFile ( <span class="image"><img src="./images/lineage-flowfile.png" alt="FlowFile" width="32"></span> ) and the +various processing events that have occurred. The selected event will be highlighted in red. It is possible to right-click on any +event to see that event’s details (See <a href="#event_details">Details of an Event</a>). To see how the lineage evolved over time, click the slider at the bottom-left of the window and move it to the left to see the state of the lineage at earlier stages in the dataflow.</p> </div> <div class="paragraph"> @@ -2712,13 +2929,13 @@ track of the parent FlowFile that produc lineage graph and select "Find parents" from the context menu.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/find-parents.png" alt="Find Parents" width="250"></span></p> +<p><span class="image"><img src="./images/find-parents.png" alt="Find Parents"></span></p> </div> <div class="paragraph"> <p>Once "Find parents" is selected, the graph is re-drawn to show the parent FlowFile and its lineage as well as the child and its lineage.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/parent-found.png" alt="Parent Found" width="250"></span></p> +<p><span class="image"><img src="./images/parent-found.png" alt="Parent Found"></span></p> </div> </div> <div class="sect3"> @@ -2727,13 +2944,13 @@ lineage graph and select "Find parents" <p>In the same way that it is useful to find a parent FlowFile, the user may also want to determine what children were spawned from a given FlowFile. To do this, right-click on the event in the lineage graph and select "Expand" from the context menu.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/expand-event.png" alt="Expand Event" width="250"></span></p> +<p><span class="image"><img src="./images/expand-event.png" alt="Expand Event"></span></p> </div> <div class="paragraph"> <p>Once "Expand" is selected, the graph is re-drawn to show the children and their lineage.</p> </div> <div class="paragraph"> -<p><span class="image"><img src="./images/expanded-events.png" alt="Expanded Events" width="300"></span></p> +<p><span class="image"><img src="./images/expanded-events.png" alt="Expanded Events"></span></p> </div> </div> </div> @@ -2743,17 +2960,26 @@ lineage graph and select "Find parents" <h2 id="other_management_features"><a class="anchor" href="#other_management_features"></a>Other Management Features</h2> <div class="sectionbody"> <div class="paragraph"> -<p>In addition to the Summary Page, Data Provenance Page, Template Management Page, and Bulletin Board Page, there are other tools in the Management Toolbar (See <a href="#User_Interface">NiFi User Interface</a>) that are useful to the DFM. The Flow Configuration History, which is available by clicking on the clock icon ( <span class="image"><img src="./images/iconFlowHistory.png" alt="Flow History" width="28"></span> ) in the Management Toolbar, shows all the changes that have been made to the dataflow. The history can aid in troubleshooting, such as if a recent change to the dataflow has caused a problem and needs to be fixed. The DFM can see what changes have been made and adjust the flow as needed to fix the problem. While NiFi does not have an "undo" feature, the DFM can make new changes to the dataflow that will fix the problem.</p> +<p>In addition to the Summary Page, Data Provenance Page, Template Management Page, and Bulletin Board Page, there are +other tools in the Global Menu (See <a href="#User_Interface">NiFi User Interface</a>) that are useful to the DFM. Select Flow Configuration History to view +all the changes that have been made to the dataflow. The history can aid in troubleshooting, such as if a +recent change to the dataflow has caused a problem and needs to be fixed. The DFM can see what changes have been made and +adjust the flow as needed to fix the problem. While NiFi does not have an "undo" feature, the DFM can make new changes to the +dataflow that will fix the problem.</p> </div> <div class="paragraph"> -<p>Two other tools in the Management Toolbar are the Controller Settings page ( <span class="image"><img src="./images/iconSettings.png" alt="Settings" width="28"></span> ) and the Users page ( <span class="image"><img src="./images/iconUsers.png" alt="Users" width="28"></span> ). The Controller Settings page provides the ability to change the name of the NiFi instance, add comments describing the NiFi instance, set the maximum number of threads that are available to the application, and create a back-up copy of the dataflow(s) currently on the canvas. It also provides tabs where DFMs may add and configure Controller Services and Reporting Tasks (see <a href="#Controller_Services_and_Reporting_Tasks">Controller Services and Reporting Tasks</a>). The Users page is used to manage user access, which is described in the <a href="administration-guide.html">Admin Guide</a>.</p> +<p>Two other tools in the Global Menu are Controller Settings and Users. The Controller Settings page provides the ability to change +the name of the NiFi instance, add comments describing the NiFi instance, set the maximum number of threads that are available +to the application. It also provides tabs where DFMs may add and configure Controller Services and Reporting Tasks +(see <a href="#Controller_Services_and_Reporting_Tasks">[Controller_Services_and_Reporting_Tasks]</a>). The Users page is used to manage user access, which is described in +the <a href="administration-guide.html">System Administrator’s Guide</a>.</p> </div> </div> </div> </div> <div id="footer"> <div id="footer-text"> -Last updated 2016-07-09 13:42:05 -04:00 +Last updated 2016-08-26 11:39:00 -04:00 </div> </div> </body>
