Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ExtractText/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ExtractText/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ExtractText/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ExtractText/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1,2 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ExtractText</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ExtractText</h1><h2>Description: </h2><p>Evaluates one or more Regular Expressions against the content of a FlowFile. The results of those Regular Expressions are assigned to FlowFile Attributes. Regular Expressions are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed. The first capture group, if any found, will be placed into that attribute name.But all capture groups, including the matching string sequence itself will also be provided at that attribute name with an index value provided, with the exception of a ca pturing group that is optional and does not match - for example, given the attribute name "regex" and expression "abc(def)?(g)" we would add an attribute "regex.1" with a value of "def" if the "def" matched. If the "def" did not match, no attribute named "regex.1" would be added but an attribute named "regex.2" with a value of "g" will be added regardless.The value of the property must be a valid Regular Expressions with one or more capturing groups. If the Regular Expression matches more than once, only the first match will be used unless the property enabling repeating capture group is set to true. If any provided Regular Expression matches, the FlowFile(s) will be routed to 'matched'. If no provided Regular Expression matches, the FlowFile will be routed to 'unmatched' and no attributes will be applied to the FlowFile.</p><h3>Tags: </h3><p>evaluate, extract, Text, Regular Expression, regex</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <s trong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Character Set</strong></td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">The Character Set in which the file is encoded</td></tr><tr><td id="name"><strong>Maximum Buffer Size</strong></td><td id="default-value">1 MB</td><td id="allowable-values"></td><td id="description">Specifies the maximum amount of data to buffer (per file) in order to apply the regular expressions. Files larger than the specified maximum will not be fully evaluated.</td></tr><tr><td id="name">Maximum Capture Group Length</td><td id="default-value">1024</td><td id="allowable-values"></td><td id="description">Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated.</td></tr><tr><td id="name"><strong>Enable Canonical Equivalence</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates that two characters match only when their full canonical decompositions match.</td></tr><tr><td id="name"><strong>Enable Case-insensitive Matching</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates that two characters match even if they are in a different case. Can also be specified via the embedded flag (?i).</td></tr><tr><td id="name"><strong>Permit Whitespace and Comments in Pattern</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">In this mode, whitespace is ignored, and embedded comments starting with # are ignored until the end of a line. Can also be specified via the em bedded flag (?x).</td></tr><tr><td id="name"><strong>Enable DOTALL Mode</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates that the expression '.' should match any character, including a line terminator. Can also be specified via the embedded flag (?s).</td></tr><tr><td id="name"><strong>Enable Literal Parsing of the Pattern</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates that Metacharacters and escape characters should be given no special meaning.</td></tr><tr><td id="name"><strong>Enable Multiline Mode</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates that '^' and '$' should match just after and just before a line terminator or end of sequence, instead of only the beginning or end of the entire input. Can also be specified via the embeded flag (?m).</td></tr><tr><td id="name"><strong>Enable Unicode-aware Case Folding</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">When used with 'Enable Case-insensitive Matching', matches in a manner consistent with the Unicode Standard. Can also be specified via the embedded flag (?u).</td></tr><tr><td id="name"><strong>Enable Unicode Predefined Character Classes</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Specifies conformance with the Unicode Technical Standard #18: Unicode Regular Expression Annex C: Compatibility Properties. Can also be specified via the embedded flag (?U).</td></tr><tr><td id="name"><strong>Enable Unix Lines Mode</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Ind icates that only the ' +' line terminator is recognized in the behavior of '.', '^', and '$'. Can also be specified via the embedded flag (?d).</td></tr><tr><td id="name"><strong>Include Capture Group 0</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates that Capture Group 0 should be included as an attribute. Capture Group 0 represents the entirety of the regular expression match, is typically not used, and could have considerable length.</td></tr><tr><td id="name"><strong>Enable repeating capture group</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If set to true, every string matching the capture groups will be extracted. Otherwise, if the Regular Expression matches more than once, only the first match will be extracted.</td></tr></table><h3>Dynamic Properties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Value</th><th>Description</th></tr><tr><td id="name">A FlowFile attribute</td><td id="value">A Regular Expression with one or more capturing group</td><td>The first capture group, if any found, will be placed into that attribute name.But all capture groups, including the matching string sequence itself will also be provided at that attribute name with an index value provided.<br/><strong>Supports Expression Language: false</strong></td></tr></table></p><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>unmatched</td><td>FlowFiles are routed to this relationship when no provided Regular Expression matches the content of the FlowFile</td></tr><tr><td>matched</td><td>FlowFiles are routed to this relationship when the Regular Expression is successfully evaluated and the FlowFile is modified as a result</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3 >Writes Attributes: </h3>None specified.<h3>State management: </h3>This >component does not store state.<h3>Restricted: </h3>This component is not >restricted.<h3>Input requirement: </h3>This component requires an incoming >relationship.<h3>System Resource Considerations:</h3>None >specified.</body></html> \ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchDistributedMapCache/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchDistributedMapCache/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchDistributedMapCache/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchDistributedMapCache/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchDistributedMapCache</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchDistributedMapCache</h1><h2>Description: </h2><p>Computes cache key(s) from FlowFile attributes, for each incoming FlowFile, and fetches the value(s) from the Distributed Map Cache associated with each key. If configured without a destination attribute, the incoming FlowFile's content is replaced with the binary data received by the Distributed Map Cache. If there is no value stored under that key then the flow file will be routed to 'not-found'. Note that the processor will always attempt to read the entire cached value into memory before placing it in it's destination. This could be potentially problematic if the cached value is very large.</p><h3>Tags: </h3><p>map, cache, fetch, distributed</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Cache Entry Identifier</strong></td><td id="default-value">${hash.value}</td><td id="allowable-values"></td><td id="description">A comma-delimited list of FlowFile attributes, or the results of Attribute Expression Language statements, which will be evaluated against a FlowFile in order to determine the value(s) used to identify duplicates; it is these values that are cached. NOTE: Only a singl e Cache Entry Identifier is allowed unless Put Cache Value In Attribute is specified. Multiple cache lookups are only supported when the destination is a set of attributes (see the documentation for 'Put Cache Value In Attribute' for more details including naming convention.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Distributed Cache Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>DistributedMapCacheClient<br/><strong>Implementations: </strong><a href="../../../nifi-distributed-cache-services-nar/1.7.0/org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService/index.html">DistributedMapCacheClientService</a><br/><a href="../../../nifi-redis-nar/1.7.0/org.apache.nifi.redis.service.RedisDistributedMapCacheClientService/index.html">RedisDistributedMapCacheClientService</a><br/><a hre f="../../../nifi-hbase_1_1_2-client-service-nar/1.7.0/org.apache.nifi.hbase.HBase_1_1_2_ClientMapCacheService/index.html">HBase_1_1_2_ClientMapCacheService</a><br/><a href="../../../nifi-couchbase-nar/1.7.0/org.apache.nifi.couchbase.CouchbaseMapCacheClient/index.html">CouchbaseMapCacheClient</a></td><td id="description">The Controller Service that is used to get the cached values.</td></tr><tr><td id="name">Put Cache Value In Attribute</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">If set, the cache value received will be put into an attribute of the FlowFile instead of a the content of theFlowFile. The attribute key to put to is determined by evaluating value of this property. If multiple Cache Entry Identifiers are selected, multiple attributes will be written, using the evaluated value of this property, appended by a period (.) and the name of the cache entry identifier.<br/><strong>Supports Expression Language: true (will be evaluated using f low file attributes and variable registry)</strong></td></tr><tr><td id="name">Max Length To Put In Attribute</td><td id="default-value">256</td><td id="allowable-values"></td><td id="description">If routing the cache value to an attribute of the FlowFile (by setting the "Put Cache Value in attribute" property), the number of characters put to the attribute value will be at most this amount. This is important because attributes are held in memory and large attributes will quickly cause out of memory issues. If the output goes longer than this value, it will be truncated to fit. Consider making this smaller if able.</td></tr><tr><td id="name">Character Set</td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">The Character Set in which the cached value is encoded. This will only be used when routing to an attribute.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>If the cache was successfully communicated with it will be routed to this relationship</td></tr><tr><td>failure</td><td>If unable to communicate with the cache or if the cache entry is evaluated to be blank, the FlowFile will be penalized and routed to this relationship</td></tr><tr><td>not-found</td><td>If a FlowFile's Cache Entry Identifier was not found in the cache, it will be routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>user-defined</td><td>If the 'Put Cache Value In Attribute' property is set then whatever it is set to will become the attribute key and the value would be whatever the response was from the Distributed Map Cache. If multiple cache entry identifiers are selected, multiple attributes will be written, using the evaluated value of this property, appended by a period (.) and the name of the cache entry identifier. For exa mple, if the Cache Entry Identifier property is set to 'id,name', and the user-defined property is named 'fetched', then two attributes will be written, fetched.id and fetched.name, containing their respective values.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../../../nifi-distributed-cache-services-nar/1.7.0/org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService/index.html">DistributedMapCacheClientService</a>, <a href="../../../nifi-distributed-cache-services-nar/1.7.0/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html">DistributedMapCacheServer</a>, <a href="../org.apache.nifi.processors.standard.PutDistributedMapCache/index.html">PutDistributedMapCache</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFTP/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFTP/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchFTP</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchFTP</h1><h2>Description: </h2><p>Fetches the content of a file from a remote SFTP server and overwrites the contents of an incoming FlowFile with the content of the remote file.</p><h3>Tags: </h3><p>ftp, get, retrieve, files, fetch, remote, ingest, source, input</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, a nd whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Hostname</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully-qualified hostname or IP address of the host to fetch the data from<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Port</strong></td><td id="default-value">21</td><td id="allowable-values"></td><td id="description">The port to connect to on the remote host to fetch the data from<br/><strong>Supports Expression Language: true (will be evaluated using flow file at tributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Username</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Username<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Password</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Password for the user account<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Remote File</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully qualified filename on the remote system<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Completion Str ategy</strong></td><td id="default-value">None</td><td id="allowable-values"><ul><li>None <img src="../../../../../html/images/iconInfo.png" alt="Leave the file as-is" title="Leave the file as-is"></img></li><li>Move File <img src="../../../../../html/images/iconInfo.png" alt="Move the file to the directory specified by the <Move Destination Directory> property" title="Move the file to the directory specified by the <Move Destination Directory> property"></img></li><li>Delete File <img src="../../../../../html/images/iconInfo.png" alt="Deletes the original file from the remote system" title="Deletes the original file from the remote system"></img></li></ul></td><td id="description">Specifies what to do with the original file on the server once it has been pulled into NiFi. If the Completion Strategy fails, a warning will be logged but the data will still be transferred.</td></tr><tr><td id="name">Move Destination Directory</td><td id="default-value"></td><td id="allowabl e-values"></td><td id="description">The directory on the remote server to move the original file to once it has been ingested into NiFi. This property is ignored unless the Completion Strategy is set to 'Move File'. The specified directory must already exist on the remote system if 'Create Directory' is disabled, or the rename will fail.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Create Directory</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Used when 'Completion Strategy' is 'Move File'. Specifies whether or not the remote directory should be created if it does not exist.</td></tr><tr><td id="name"><strong>Connection Timeout</strong></td><td id="default-value">30 sec</td><td id="allowable-values"></td><td id="description">Amount of time to wait before timing out while creating a connection</td></t r><tr><td id="name"><strong>Data Timeout</strong></td><td id="default-value">30 sec</td><td id="allowable-values"></td><td id="description">When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems</td></tr><tr><td id="name"><strong>Use Compression</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates whether or not ZLIB compression should be used when transferring files</td></tr><tr><td id="name">Connection Mode</td><td id="default-value">Passive</td><td id="allowable-values"><ul><li>Active</li><li>Passive</li></ul></td><td id="description">The FTP Connection Mode</td></tr><tr><td id="name">Transfer Mode</td><td id="default-value">Binary</td><td id="allowable-values"><ul><li>Binary</li><li>ASCII</li></ul></td><td id="description">The FTP Transfer Mode</td></tr><tr><td id="name">Proxy Confi guration Service</td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>ProxyConfigurationService<br/><strong>Implementation: </strong><a href="../../../nifi-proxy-configuration-nar/1.7.0/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html">StandardProxyConfigurationService</a></td><td id="description">Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS</td></tr><tr><td id="name">Proxy Type</td><td id="default-value">DIRECT</td><td id="allowable-values"><ul><li>DIRECT</li><li>HTTP</li><li>SOCKS</li></ul></td><td id="description">Proxy type used for file transfers</td></tr><tr><td id="name">Proxy Host</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully qualified hostname or IP address of the proxy server<br/><strong>Supports Expression Language: true ( will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Proxy Port</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The port of the proxy server<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Http Proxy Username</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Http Proxy Username<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Http Proxy Password</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Http Proxy Password<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Internal Buffer Size</td><td id="default-value">16KB</td><td id="allowable-values"></td><td id="des cription">Set the internal buffer size for buffered data streams</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All FlowFiles that are received are routed to success</td></tr><tr><td>comms.failure</td><td>Any FlowFile that could not be fetched from the remote server due to a communications failure will be transferred to this Relationship.</td></tr><tr><td>not.found</td><td>Any FlowFile for which we receive a 'Not Found' message from the remote server will be transferred to this Relationship.</td></tr><tr><td>permission.denied</td><td>Any FlowFile that could not be fetched from the remote server due to insufficient permissions will be transferred to this Relationship.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>ftp.remote.host</td><td>The hostname or IP address from which the fi le was pulled</td></tr><tr><td>ftp.remote.port</td><td>The port that was used to communicate with the remote FTP server</td></tr><tr><td>ftp.remote.filename</td><td>The name of the remote file that was pulled</td></tr><tr><td>filename</td><td>The filename is updated to point to the filename fo the remote file</td></tr><tr><td>path</td><td>If the Remote File contains a directory name, that directory name will be added to the FlowFile using the 'path' attribute</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.GetSFTP/index.html">GetSFTP</a>, <a href="../org.apache.nifi.processors.standard.PutSFTP/index.html">PutSFTP</a>, <a href="../org.apache.nifi.processors.standard.GetFTP/index.html">GetFTP</a>, <a hr ef="../org.apache.nifi.processors.standard.PutFTP/index.html">PutFTP</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFile/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchFile/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchFile</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchFile</h1><h2>Description: </h2><p>Reads the contents of a file from disk and streams it into the contents of an incoming FlowFile. Once this is done, the file is optionally moved elsewhere or deleted to help keep the file system organized.</p><h3>Tags: </h3><p>local, files, filesystem, ingest, ingress, get, source, input, fetch</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../.. /../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>File to Fetch</strong></td><td id="default-value">${absolute.path}/${filename}</td><td id="allowable-values"></td><td id="description">The fully-qualified filename of the file to fetch from the file system<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Completion Strategy</strong></td><td id="default-value">None</td><td id="allowable-values"><ul><li>None <img src="../../../../../html/images/iconInfo.png" alt="Leave the file as-is" title="Leave the file as-is"></img></li><li>Move File <img src="../../../../../html/images/iconInfo.png" alt="Moves the file to the directory specified by the <Move Destination Directory> property" title="Moves the file to the dire ctory specified by the <Move Destination Directory> property"></img></li><li>Delete File <img src="../../../../../html/images/iconInfo.png" alt="Deletes the original file from the file system" title="Deletes the original file from the file system"></img></li></ul></td><td id="description">Specifies what to do with the original file on the file system once it has been pulled into NiFi</td></tr><tr><td id="name">Move Destination Directory</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The directory to the move the original file to once it has been fetched from the file system. This property is ignored unless the Completion Strategy is set to "Move File". If the directory does not exist, it will be created.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Move Conflict Strategy</strong></td><td id="default-value">Rename</td><td id="al lowable-values"><ul><li>Rename <img src="../../../../../html/images/iconInfo.png" alt="The existing destination file should remain intact. The newly ingested file should be moved to the destination directory but be renamed to a random filename" title="The existing destination file should remain intact. The newly ingested file should be moved to the destination directory but be renamed to a random filename"></img></li><li>Replace File <img src="../../../../../html/images/iconInfo.png" alt="The newly ingested file should replace the existing file in the Destination Directory" title="The newly ingested file should replace the existing file in the Destination Directory"></img></li><li>Keep Existing <img src="../../../../../html/images/iconInfo.png" alt="The existing file should in the Destination Directory should stay intact and the newly ingested file should be deleted" title="The existing file should in the Destination Directory should stay intact and the newly ingested file should be deleted"></img></li><li>Fail <img src="../../../../../html/images/iconInfo.png" alt="The existing destination file should remain intact and the incoming FlowFile should be routed to failure" title="The existing destination file should remain intact and the incoming FlowFile should be routed to failure"></img></li></ul></td><td id="description">If Completion Strategy is set to Move File and a file already exists in the destination directory with the same name, this property specifies how that naming conflict should be resolved</td></tr><tr><td id="name"><strong>Log level when file not found</strong></td><td id="default-value">ERROR</td><td id="allowable-values"><ul><li>TRACE</li><li>DEBUG</li><li>INFO</li><li>WARN</li><li>ERROR</li><li>FATAL</li><li>NONE</li></ul></td><td id="description">Log level to use in case the file does not exist when the processor is triggered</td></tr><tr><td id="name"><strong>Log level when permission denied</strong></td><td id="default-value">ERROR</td><t d id="allowable-values"><ul><li>TRACE</li><li>DEBUG</li><li>INFO</li><li>WARN</li><li>ERROR</li><li>FATAL</li><li>NONE</li></ul></td><td id="description">Log level to use in case user alopresto does not have sufficient permissions to read the file</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Any FlowFile that is successfully fetched from the file system will be transferred to this Relationship.</td></tr><tr><td>not.found</td><td>Any FlowFile that could not be fetched from the file system because the file could not be found will be transferred to this Relationship.</td></tr><tr><td>permission.denied</td><td>Any FlowFile that could not be fetched from the file system due to the user running NiFi not having sufficient permissions will be transferred to this Relationship.</td></tr><tr><td>failure</td><td>Any FlowFile that could not be fetched from the file system for any reason other than insufficien t permissions or the file not existing will be transferred to this Relationship.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3><table id="restrictions"><tr><th>Required Permission</th><th>Explanation</th></tr><tr><td>read filesystem</td><td>Provides operator the ability to read from any file that NiFi has access to.</td></tr><tr><td>write filesystem</td><td>Provides operator the ability to delete any file that NiFi has access to.</td></tr></table><h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.GetFile/index.html">GetFile</a>, <a href="../org.apache.nifi.processors.standard.PutFile/index.html">PutFile</a>, <a href="../org.apache.nifi.processors.standard.ListFile/index.html">ListFile</a></p></body ></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchSFTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchSFTP/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchSFTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FetchSFTP/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchSFTP</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchSFTP</h1><h2>Description: </h2><p>Fetches the content of a file from a remote SFTP server and overwrites the contents of an incoming FlowFile with the content of the remote file.</p><h3>Tags: </h3><p>sftp, get, retrieve, files, fetch, remote, ingest, source, input</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a> , and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Hostname</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully-qualified hostname or IP address of the host to fetch the data from<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Port</strong></td><td id="default-value">22</td><td id="allowable-values"></td><td id="description">The port to connect to on the remote host to fetch the data from<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Username</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Username<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Password</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Password for the user account<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Private Key Path</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully qualified path to the Private Key file<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Private Key Passphrase</td><td i d="default-value"></td><td id="allowable-values"></td><td id="description">Password for the private key<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Remote File</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully qualified filename on the remote system<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Completion Strategy</strong></td><td id="default-value">None</td><td id="allowable-values"><ul><li>None <img src="../../../../../html/images/iconInfo.png" alt="Leave the file as-is" title="Leave the file as-is"></img></li><li>Move File <img src="../../../../../html/images/iconInfo.png" alt="Move the file to the directory specified by the <Move Destination Directory&g t; property" title="Move the file to the directory specified by the <Move Destination Directory> property"></img></li><li>Delete File <img src="../../../../../html/images/iconInfo.png" alt="Deletes the original file from the remote system" title="Deletes the original file from the remote system"></img></li></ul></td><td id="description">Specifies what to do with the original file on the server once it has been pulled into NiFi. If the Completion Strategy fails, a warning will be logged but the data will still be transferred.</td></tr><tr><td id="name">Move Destination Directory</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The directory on the remote server to move the original file to once it has been ingested into NiFi. This property is ignored unless the Completion Strategy is set to 'Move File'. The specified directory must already exist on the remote system if 'Create Directory' is disabled, or the rename will fail.<br/><strong>Suppor ts Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name">Create Directory</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Used when 'Completion Strategy' is 'Move File'. Specifies whether or not the remote directory should be created if it does not exist.</td></tr><tr><td id="name">Disable Directory Listing</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Control how 'Move Destination Directory' is created when 'Completion Strategy' is 'Move File' and 'Create Directory' is enabled. If set to 'true', directory listing is not performed prior to create missing directories. By default, this processor executes a directory listing command to see target directory existence before creating missing directories. However, there are situations that you might need to di sable the directory listing such as the following. Directory listing might fail with some permission setups (e.g. chmod 100) on a directory. Also, if any other SFTP client created the directory after this processor performed a listing and before a directory creation request by this processor is finished, then an error is returned because the directory already exists.</td></tr><tr><td id="name"><strong>Connection Timeout</strong></td><td id="default-value">30 sec</td><td id="allowable-values"></td><td id="description">Amount of time to wait before timing out while creating a connection</td></tr><tr><td id="name"><strong>Data Timeout</strong></td><td id="default-value">30 sec</td><td id="allowable-values"></td><td id="description">When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems</td></tr><tr><td id="name"><strong>Send Keep Alive On Timeout</strong></td><td id="default-val ue">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates whether or not to send a single Keep Alive message when SSH socket times out</td></tr><tr><td id="name">Host Key File</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">If supplied, the given file will be used as the Host Key; otherwise, no use host key file will be used</td></tr><tr><td id="name"><strong>Strict Host Key Checking</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates whether or not strict enforcement of hosts keys should be applied</td></tr><tr><td id="name"><strong>Use Compression</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates whether or not ZLIB compression should be used when transferring files</td></tr><tr><td id="name">Proxy Configuratio n Service</td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>ProxyConfigurationService<br/><strong>Implementation: </strong><a href="../../../nifi-proxy-configuration-nar/1.7.0/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html">StandardProxyConfigurationService</a></td><td id="description">Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS + AuthN</td></tr><tr><td id="name">Proxy Type</td><td id="default-value">DIRECT</td><td id="allowable-values"><ul><li>DIRECT</li><li>HTTP</li><li>SOCKS</li></ul></td><td id="description">Proxy type used for file transfers</td></tr><tr><td id="name">Proxy Host</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The fully qualified hostname or IP address of the proxy server<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Proxy Port</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The port of the proxy server<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Http Proxy Username</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Http Proxy Username<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name">Http Proxy Password</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Http Proxy Password<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success< /td><td>All FlowFiles that are received are routed to success</td></tr><tr><td>comms.failure</td><td>Any FlowFile that could not be fetched from the remote server due to a communications failure will be transferred to this Relationship.</td></tr><tr><td>not.found</td><td>Any FlowFile for which we receive a 'Not Found' message from the remote server will be transferred to this Relationship.</td></tr><tr><td>permission.denied</td><td>Any FlowFile that could not be fetched from the remote server due to insufficient permissions will be transferred to this Relationship.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>sftp.remote.host</td><td>The hostname or IP address from which the file was pulled</td></tr><tr><td>sftp.remote.port</td><td>The port that was used to communicate with the remote SFTP server</td></tr><tr><td>sftp.remote.filename</td><td>The name of the remot e file that was pulled</td></tr><tr><td>filename</td><td>The filename is updated to point to the filename fo the remote file</td></tr><tr><td>path</td><td>If the Remote File contains a directory name, that directory name will be added to the FlowFile using the 'path' attribute</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.GetSFTP/index.html">GetSFTP</a>, <a href="../org.apache.nifi.processors.standard.PutSFTP/index.html">PutSFTP</a>, <a href="../org.apache.nifi.processors.standard.GetFTP/index.html">GetFTP</a>, <a href="../org.apache.nifi.processors.standard.PutFTP/index.html">PutFTP</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FlattenJson/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FlattenJson/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FlattenJson/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.FlattenJson/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FlattenJson</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FlattenJson</h1><h2>Description: </h2><p>Provides the user with the ability to take a nested JSON document and flatten it into a simple key/value pair document. The keys are combined at each level with a user-defined separator that defaults to '.'. Support three kinds of flatten mode, normal, keep-arrays and dot notation for MongoDB query. Default flatten mode is 'keep-arrays'.</p><h3>Tags: </h3><p>json, flatten</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also ind icates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Separator</td><td id="default-value">.</td><td id="allowable-values"></td><td id="description">The separator character used for joining keys. Must be a JSON-legal character.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong></td></tr><tr><td id="name"><strong>Flatten Mode</strong></td><td id="default-value">keep arrays</td><td id="allowable-values"><ul><li>normal <img src="../../../../../html/images/iconInfo.png" alt="Flattens every objects into a single level json" title="Flattens every objects into a single level json"></img></li><li>keep arrays <img src="../../../../../html/images/iconInfo.png" alt="Flattens every objects and keep arrays format" title="Flattens every objects and keep arrays format"></img></li><li>dot notation <img src="../../../../../html/images/iconInfo.png" alt="Conforms to MongoDB dot notation to update also nested documents" title="Conforms to MongoDB dot notation to update also nested documents"></img></li></ul></td><td id="description">Specifies how json is flattened</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Successfully flattened files go to this relationship.</td></tr><tr><td>failure</td><td>Files that cannot be flattened go to this relationship.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/additionalDetails.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/additionalDetails.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1,481 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>ForkRecord</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <p> + ForkRecord allows the user to fork a record into multiple records. To do that, the user must specify + one or multiple <a href="../../../../../html/record-path-guide.html">RecordPath</a> (as dynamic + properties of the processor) pointing to a field of type ARRAY containing RECORD elements. + </p> + <p> + The processor accepts two modes: + <ul> + <li>Split mode - in this mode, the generated records will have the same schema as the input. For + every element in the array, one record will be generated and the array will only contain this + element.</li> + <li>Extract mode - in this mode, the generated records will be the elements contained in the array. + Besides, it is also possible to add in each record all the fields of the parent records from the root + level to the record element being forked. However it supposes the fields to add are defined in the + schema of the Record Writer controller service. </li> + </ul> + </p> + + <h2>Examples</h2> + + <h3>EXTRACT mode</h3> + + <p> + To better understand how this Processor works, we will lay out a few examples. For the sake of these examples, let's assume that our input + data is JSON formatted and looks like this: + </p> + +<code> +<pre> +[{ + "id": 1, + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "accounts": [{ + "id": 42, + "balance": 4750.89 + }, { + "id": 43, + "balance": 48212.38 + }] +}, +{ + "id": 2, + "name": "Jane Doe", + "address": "345 My Street", + "city": "Her City", + "state": "NY", + "zipCode": "22222", + "country": "USA", + "accounts": [{ + "id": 45, + "balance": 6578.45 + }, { + "id": 46, + "balance": 34567.21 + }] +}] +</pre> +</code> + + + <h4>Example 1 - Extracting without parent fields</h4> + + <p> + For this case, we want to create one record per <code>account</code> and we don't care about + the other fields. We'll add a dynamic property "path" set to <code>/accounts</code>. The resulting + flow file will contain 4 records and will look like (assuming the Record Writer schema is correctly set): + </p> + +<code> +<pre> +[{ + "id": 42, + "balance": 4750.89 +}, { + "id": 43, + "balance": 48212.38 +}, { + "id": 45, + "balance": 6578.45 +}, { + "id": 46, + "balance": 34567.21 +}] +</pre> +</code> + + + <h4>Example 2 - Extracting with parent fields</h4> + + <p> + Now, if we set the property "Include parent fields" to true, this will recursively include + the parent fields into the output records assuming the Record Writer schema allows it. In + case multiple fields have the same name (like we have in this example for <code>id</code>), + the child field will have the priority over all the parent fields sharing the same name. In + this case, the <code>id</code> of the array <code>accounts</code> will be saved in the + forked records. The resulting flow file will contain 4 records and will look like: + </p> + +<code> +<pre> +[{ + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "id": 42, + "balance": 4750.89 +}, { + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "id": 43, + "balance": 48212.38 +}, { + "name": "Jane Doe", + "address": "345 My Street", + "city": "Her City", + "state": "NY", + "zipCode": "22222", + "country": "USA", + "id": 45, + "balance": 6578.45 +}, { + "name": "Jane Doe", + "address": "345 My Street", + "city": "Her City", + "state": "NY", + "zipCode": "22222", + "country": "USA", + "id": 46, + "balance": 34567.21 +}] +</pre> +</code> + + + <h4>Example 3 - Multi-nested arrays</h4> + + <p> + Now let's say that the input record contains multi-nested arrays like the below example: + </p> + +<code> +<pre> +[{ + "id": 1, + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "accounts": [{ + "id": 42, + "balance": 4750.89, + "transactions": [{ + "id": 5, + "amount": 150.31 + }, + { + "id": 6, + "amount": -15.31 + }] + }, { + "id": 43, + "balance": 48212.38, + "transactions": [{ + "id": 7, + "amount": 36.78 + }, + { + "id": 8, + "amount": -21.34 + }] + }] +}] +</pre> +</code> + + <p> + If we want to have one record per <code>transaction</code> for each <code>account</code>, then + the Record Path should be set to <code>/accounts[*]/transactions</code>. If we have the following + schema for our Record Reader: + </p> + +<code> +<pre> +{ + "type" : "record", + "name" : "bank", + "fields" : [ { + "name" : "id", + "type" : "int" + }, { + "name" : "name", + "type" : "string" + }, { + "name" : "address", + "type" : "string" + }, { + "name" : "city", + "type" : "string" + }, { + "name" : "state", + "type" : "string" + }, { + "name" : "zipCode", + "type" : "string" + }, { + "name" : "country", + "type" : "string" + }, { + "name" : "accounts", + "type" : { + "type" : "array", + "items" : { + "type" : "record", + "name" : "accounts", + "fields" : [ { + "name" : "id", + "type" : "int" + }, { + "name" : "balance", + "type" : "double" + }, { + "name" : "transactions", + "type" : { + "type" : "array", + "items" : { + "type" : "record", + "name" : "transactions", + "fields" : [ { + "name" : "id", + "type" : "int" + }, { + "name" : "amount", + "type" : "double" + } ] + } + } + } ] + } + } + } ] +} +</pre> +</code> + + <p> + And if we have the following schema for our Record Writer: + </p> + +<code> +<pre> +{ + "type" : "record", + "name" : "bank", + "fields" : [ { + "name" : "id", + "type" : "int" + }, { + "name" : "name", + "type" : "string" + }, { + "name" : "address", + "type" : "string" + }, { + "name" : "city", + "type" : "string" + }, { + "name" : "state", + "type" : "string" + }, { + "name" : "zipCode", + "type" : "string" + }, { + "name" : "country", + "type" : "string" + }, { + "name" : "amount", + "type" : "double" + }, { + "name" : "balance", + "type" : "double" + } ] +} +</pre> +</code> + + <p> + Then, if we include the parent fields, we'll have 4 records as below: + </p> + +<code> +<pre> +[ { + "id" : 5, + "name" : "John Doe", + "address" : "123 My Street", + "city" : "My City", + "state" : "MS", + "zipCode" : "11111", + "country" : "USA", + "amount" : 150.31, + "balance" : 4750.89 +}, { + "id" : 6, + "name" : "John Doe", + "address" : "123 My Street", + "city" : "My City", + "state" : "MS", + "zipCode" : "11111", + "country" : "USA", + "amount" : -15.31, + "balance" : 4750.89 +}, { + "id" : 7, + "name" : "John Doe", + "address" : "123 My Street", + "city" : "My City", + "state" : "MS", + "zipCode" : "11111", + "country" : "USA", + "amount" : 36.78, + "balance" : 48212.38 +}, { + "id" : 8, + "name" : "John Doe", + "address" : "123 My Street", + "city" : "My City", + "state" : "MS", + "zipCode" : "11111", + "country" : "USA", + "amount" : -21.34, + "balance" : 48212.38 +} ] +</pre> +</code> + + <h3>SPLIT mode</h3> + + <h4>Example</h4> + + <p> + Assuming we have the below data and we added a property "path" set to <code>/accounts</code>: + </p> + +<code> +<pre> +[{ + "id": 1, + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "accounts": [{ + "id": 42, + "balance": 4750.89 + }, { + "id": 43, + "balance": 48212.38 + }] +}, +{ + "id": 2, + "name": "Jane Doe", + "address": "345 My Street", + "city": "Her City", + "state": "NY", + "zipCode": "22222", + "country": "USA", + "accounts": [{ + "id": 45, + "balance": 6578.45 + }, { + "id": 46, + "balance": 34567.21 + }] +}] +</pre> +</code> + + <p> + Then we'll get 4 records as below: + </p> + +<code> +<pre> +[{ + "id": 1, + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "accounts": [{ + "id": 42, + "balance": 4750.89 + }] +}, +{ + "id": 1, + "name": "John Doe", + "address": "123 My Street", + "city": "My City", + "state": "MS", + "zipCode": "11111", + "country": "USA", + "accounts": [{ + "id": 43, + "balance": 48212.38 + }] +}, +{ + "id": 2, + "name": "Jane Doe", + "address": "345 My Street", + "city": "Her City", + "state": "NY", + "zipCode": "22222", + "country": "USA", + "accounts": [{ + "id": 45, + "balance": 6578.45 + }] +}, +{ + "id": 2, + "name": "Jane Doe", + "address": "345 My Street", + "city": "Her City", + "state": "NY", + "zipCode": "22222", + "country": "USA", + "accounts": [{ + "id": 46, + "balance": 34567.21 + }] +}] +</pre> +</code> + + </body> +</html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.ForkRecord/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ForkRecord</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ForkRecord</h1><h2>Description: </h2><p>This processor allows the user to fork a record into multiple records. The user must specify at least one Record Path, as a dynamic property, pointing to a field of type ARRAY containing RECORD objects. The processor accepts two modes: 'split' and 'extract'. In both modes, there is one record generated per element contained in the designated array. In the 'split' mode, each generated record will preserve the same schema as given in the input but the array will contain only one element. In the 'extract' mode, the element of the array must be of record type and w ill be the generated record. Additionally, in the 'extract' mode, it is possible to specify if each generated record should contain all the fields of the parent records from the root level to the extracted record. This assumes that the fields to add in the record are defined in the schema of the Record Writer controller service. See examples in the additional details documentation of this processor.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>fork, record, content, array, stream, event</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Record Reader</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Servic e API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-scripting-nar/1.7.0/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.json.JsonTreeReader/index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.xml.XMLReader/index.html">XMLReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a></td><td id="description">Specifie s the Controller Service to use for reading incoming data</td></tr><tr><td id="name"><strong>Record Writer</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.7.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetW riter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.7.0/org.apache.nifi.xml.XMLRecordSetWriter/index.html">XMLRecordSetWriter</a></td><td id="description">Specifies the Controller Service to use for writing out the records</td></tr><tr><td id="name"><strong>Mode</strong></td><td id="default-value">split</td><td id="allowable-values"><ul><li>Extract <img src="../../../../../html/images/iconInfo.png" alt="Generated records will be the elements of the array" title="Generated records will be the elements of the array"></img></li><li>Split <img src="../../../../../html/images/iconInfo.png" alt="Generated records will preserve the input schema and will contain a one-element array" title="Generated records will preserve the input schema and will contain a one-element array"></img></li></ul></td><td id="description">Specifies the forking mode of the processor</td></tr><tr><td id="name"><strong>Include Parent Fields</strong></td><td id="default-value">false</td><td id="a llowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">This parameter is only valid with the 'extract' mode. If set to true, all the fields from the root level to the given array will be added as fields of each element of the array to fork.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>failure</td><td>In case a FlowFile generates an error during the fork operation, it will be routed to this relationship</td></tr><tr><td>original</td><td>The original FlowFiles will be routed to this relationship</td></tr><tr><td>fork</td><td>The FlowFiles containing the forked records will be routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>record.count</td><td>The generated FlowFile will have a 'record.count' attribute indicating the number of records that were w ritten to the FlowFile.</td></tr><tr><td>mime.type</td><td>The MIME Type indicated by the Record Writer</td></tr><tr><td><Attributes from Record Writer></td><td>Any Attribute that the configured Record Writer returns will be added to the FlowFile.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateFlowFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateFlowFile/index.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateFlowFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateFlowFile/index.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>GenerateFlowFile</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">GenerateFlowFile</h1><h2>Description: </h2><p>This processor creates FlowFiles with random data or custom content. GenerateFlowFile is usefulfor load testing, configuration, and simulation.</p><h3>Tags: </h3><p>test, random, generate</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="propertie s"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>File Size</strong></td><td id="default-value">0B</td><td id="allowable-values"></td><td id="description">The size of the file that will be used</td></tr><tr><td id="name"><strong>Batch Size</strong></td><td id="default-value">1</td><td id="allowable-values"></td><td id="description">The number of FlowFiles to be transferred in each invocation</td></tr><tr><td id="name"><strong>Data Format</strong></td><td id="default-value">Text</td><td id="allowable-values"><ul><li>Binary</li><li>Text</li></ul></td><td id="description">Specifies whether the data should be Text or Binary</td></tr><tr><td id="name"><strong>Unique FlowFiles</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, each FlowFile that is generated will be unique. If false, a random value will be generated and all FlowFil es will get the same content but this offers much higher throughput</td></tr><tr><td id="name">Custom Text</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">If Data Format is text and if Unique FlowFiles is false, then this custom text will be used as content of the generated FlowFiles and the File Size will be ignored. Finally, if Expression Language is used, evaluation will be performed only once per batch of generated FlowFiles<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name"><strong>Character Set</strong></td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">Specifies the character set to use when writing the bytes of Custom Text to a flow file.</td></tr></table><h3>Dynamic Properties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Val ue</th><th>Description</th></tr><tr><td id="name">Generated FlowFile attribute name</td><td id="value">Generated FlowFile attribute value</td><td>Specifies an attribute on generated FlowFiles defined by the Dynamic Property's key and value. If Expression Language is used, evaluation will be performed only once per batch of generated FlowFiles.<br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr></table></p><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td></td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component does not allow an incoming relationship.<h3>System Resource Considerations:</h3>None specified.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateTableFetch/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateTableFetch/additionalDetails.html?rev=1834461&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateTableFetch/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.GenerateTableFetch/additionalDetails.html Wed Jun 27 00:37:46 2018 @@ -0,0 +1,69 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>GenerateTableFetch</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <p> + GenerateTableFetch uses its properties and the specified database connection to generate flow files + containing SQL statements that can be used to fetch "pages" (aka "partitions") of data from a table. + GenerateTableFetch executes a query to the database to determine the current row count and maximum value, + and if Maximum Value Columns are specified, will collect the count of rows whose values for the Maximum + Value Columns are larger than those last observed by GenerateTableFetch. This allows for incremental + fetching of "new" rows, rather than generating SQL to fetch the entire table each time. If no Maximum + Value Columns are set, then the processor will generate SQL to fetch the entire table each time. + </p> + + <p> + In order to generate SQL that will fetch pages/partitions of data, by default GenerateTableFetch will + generate SQL that orders the data based on the Maximum Value Columns (if present) and utilize the row + numbers of the result set to determine each page. For example if the Maximum Value Column is an integer "id" + and the partition size is 10, then the SQL for the first page might be "SELECT * FROM myTable LIMIT 10" and + the second page might be "SELECT * FROM myTable OFFSET 10 LIMIT 10", and so on. + </p> + + <p> + Ordering the data can be an expensive operation depending on the database, the number of rows, etc. + Alternatively, it is possible to specify a column whose values will be used to determine the pages, using + the Column for Value Partitioning property. If set, GenerateTableFetch will determine the minimum and + maximum values for the column, and uses the minimum value as the initial offset. The SQL to fetch a page is + then based on this initial offset and the total difference in values (i.e. maximum - minimum) divided by + the page size. For example, if the column "id" is used for value partitioning, and the column contains + values 100 to 200, then with a page size of 10 the SQL to fetch the first page might be "SELECT * FROM + myTable WHERE id >= 100 AND id < 110" and the second page might be "SELECT * FROM myTable WHERE id >= 110 + AND id < 120", and so on. + </p> + + <p> + It is important that the Column for Value Partitioning be set to a column whose type can be coerced to a + long integer (i.e. not date or timestamp), and that the column values are evenly distributed and not + sparse, for best performance. As a counterexample to the above, consider a column "id" whose values are 100, + 2000, and 30000. If the Partition Size is 100, then the column values are relatively sparse, so the SQL + for the "second page" (see above example) will return zero rows, and so will every page until the value in + the query becomes "id >= 2000". Another counterexample is when the values are not uniformly distributed. + Consider a column "id" with values 100, 200, 201, 202, ... 299. Then the SQL for the first page (see above + example) will return one row with value id = 100, and the second page will return 100 rows with values 200 + ... 299. This can cause inconsistent processing times downstream, as the pages may contain a very different + number of rows. For these reasons it is recommended to use a Column for Value Partitioning that is + sufficiently dense (not sparse) and fairly evenly distributed. + </p> + </body> +</html> \ No newline at end of file
