Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html Tue May 9 15:27:39 2017 @@ -0,0 +1,155 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8" /> + <title>ExecuteFlumeSink</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> +</head> + +<body> +<h2>Data Model</h2> +<p> + This processor executes an Apache Flume sink. FlowFiles are wrapped in Flume's + Event interface. The content of the FlowFile becomes the body of the Event and + the attributes of the FlowFile become Event headers. The following special + headers are also set: +</p> +<table id="headers"> + <tr> + <th>Flume Event Header</th> + <th>FlowFile Attribute</th> + </tr> + <tr> + <td>nifi.entry.date</td> + <td>FlowFile#getEntryDate()</td> + </tr> + <tr> + <td>nifi.id</td> + <td>FlowFile#getId()</td> + </tr> + <tr> + <td>nifi.last.queue.date</td> + <td>FlowFile#getLastQueueDate()</td> + </tr> + <tr> + <td>nifi.lineage.start.date</td> + <td>FlowFile#getLineageStartDate()</td> + </tr> + <tr> + <td>nifi.size</td> + <td>FlowFile#getSize()</td> + </tr> +</table> +<h2>Warning</h2> +<p> + In NiFi, the contents of a FlowFile are accessed via a stream, but in Flume it is + stored in a byte array. This means the full content will be loaded into memory when + a FlowFile is processed by the ExecuteFlumeSink processor. You should consider the + typical size of the FlowFiles you'll process and the batch size, if any, your sink + is configured with when setting NiFi's heap size. +</p> +<h2>Configuration Details</h2> +<p> + This processor is designed to execute arbitrary Flume sinks. Most of the details + of configuring the sink is deferred to Flume's built-in configuration system. + For details on the available settings for each sink type, refer to the Flume + <a href="http://flume.apache.org/FlumeUserGuide.html#flume-sinks";>User Guide</a>. + Configuring the Flume sink is a four step process: +</p> +<ol> + <li>Set the Sink Type property to a valid Flume sink type.</li> + <li> + Set the Agent Name property to the name of the agent in your + Flume configuration. This is the prefix of the properties in the Flume + configuration file. Example: <code>tier1</code> + </li> + <li> + Set the Sink Name property to the name of the sink in your Flume + configuration. If Agent Name is <code>tier1</code>, then the Sink Name + is the value of the <code>tier1.sinks</code> property. Example: <code>sink-1</code> + </li> + <li> + Copy and paste the configuration for the sink from your Flume configuration + file into the Flume Configuration property. Assuming you're using + the same Agent Name and Sink Name as in the examples above, this will be all + of the properties that start with <code>tier1.sinks.sink-1</code>. + Do not copy the <code>tier1.sinks.sink-1.type</code> or + <code>tier1.sinks.sink-1.channel</code> properties. + </li> +</ol> +<h2>Usage Example</h2> +<p> + Assuming you had the following existing Flume configuration file: +</p> + <pre> +a1.sources = r1 +a1.sinks = k1 +a1.channels = c1 + +a1.sources.r1.type = netcat +a1.sources.r1.bind = localhost +a1.sources.r1.port = 44444 + +a1.sinks.k1.type = hdfs +a1.sinks.k1.channel = c1 +a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S +a1.sinks.k1.hdfs.filePrefix = events- +a1.sinks.k1.hdfs.round = true +a1.sinks.k1.hdfs.roundValue = 10 +a1.sinks.k1.hdfs.roundUnit = minute + +a1.channels.c1.type = memory +a1.channels.c1.capacity = 1000 +a1.channels.c1.transactionCapacity = 100 + +a1.sources.r1.channels = c1 +a1.sinks.k1.channel = c1</pre> +<p> + Then you'd configure the ExecuteFlumeSink as follows: +</p> +<table id="example"> + <tr> + <th>Property</th> + <th>Value</th> + </tr> + <tr> + <td>Sink Type</td> + <td>hdfs</td> + </tr> + <tr> + <td>Agent Name</td> + <td>a1</td> + </tr> + <tr> + <td>Sink Name</td> + <td>k1</td> + </tr> + <tr> + <td>Flume Configuration</td> + <td> + <code> + a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S<br> + a1.sinks.k1.hdfs.filePrefix = events-<br> + a1.sinks.k1.hdfs.round = true<br> + a1.sinks.k1.hdfs.roundValue = 10<br> + a1.sinks.k1.hdfs.roundUnit = minute + </code> + </td> + </tr> +</table> +</body> +</html> \ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ExecuteFlumeSink</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ExecuteFlumeSink</h1><h2>Description: </h2><p>Execute a Flume sink. Each input FlowFile is converted into a Flume Event for processing by the sink.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>flume, hadoop, put, sink, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></t r><tr><td id="name"><strong>Sink Type</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The component type name for the sink. For some sinks, this is a short, symbolic name (e.g. hdfs). For others, it's the fully-qualified name of the Sink class. See the Flume User Guide for details.</td></tr><tr><td id="name"><strong>Agent Name</strong></td><td id="default-value">tier1</td><td id="allowable-values"></td><td id="description">The name of the agent used in the Flume sink configuration</td></tr><tr><td id="name"><strong>Sink Name</strong></td><td id="default-value">sink-1</td><td id="allowable-values"></td><td id="description">The name of the sink used in the Flume sink configuration</td></tr><tr><td id="name"><strong>Flume Configuration</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The Flume configuration for the sink copied from the flume.properties file</td></tr></table><h3>Relationships: </h3> <table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td></td></tr><tr><td>failure</td><td></td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to execute arbitrary Flume configurations assuming all permissions that NiFi has.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html Tue May 9 15:27:39 2017 @@ -0,0 +1,114 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8" /> + <title>ExecuteFlumeSource</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> +</head> + +<body> +<h2>Data Model</h2> +<p> + This processor executes an Apache Flume source. Each Flume Event is turned into + a FlowFile. The content of the FlowFile is set to the body of the Event and + the Event headers become FlowFile attributes. +</p> +<h2>Configuration Details</h2> +<p> + This processor is designed to execute arbitrary Flume sources. Most of the details + of configuring the source is deferred to Flume's built-in configuration system. + For details on the available settings for each source type, refer to the Flume + <a href="http://flume.apache.org/FlumeUserGuide.html#flume-sources";>User Guide</a>. + Configuring the Flume source is a four step process: +</p> +<ol> + <li>Set the Source Type property to a valid Flume source type.</li> + <li> + Set the Agent Name property to the name of the agent in your + Flume configuration. This is the prefix of the properties in the Flume + configuration file. Example: <code>tier1</code> + </li> + <li> + Set the Source Name property to the name of the source in your Flume + configuration. If Agent Name is <code>tier1</code>, then the Source Name + is the value of the <code>tier1.sources</code> property. Example: <code>src-1</code> + </li> + <li> + Copy and paste the configuration for the source from your Flume configuration + file into the Flume Configuration property. Assuming you're using + the same Agent Name and Source Name as in the examples above, this will be all + of the properties that start with <code>tier1.sources.src-1</code>. + Do not copy the <code>tier1.sources.src-1.type</code> or + <code>tier1.sources.src-1.channel</code> properties. + </li> +</ol> +<h2>Usage Example</h2> +<p> + Assuming you had the following existing Flume configuration file: +</p> + <pre> +a1.sources = r1 +a1.sinks = k1 +a1.channels = c1 + +a1.sources.r1.type = multiport_syslogtcp +a1.sources.r1.channels = c1 +a1.sources.r1.host = 0.0.0.0 +a1.sources.r1.ports = 10001 10002 10003 +a1.sources.r1.portHeader = port + +a1.sinks.k1.type = logger + +a1.channels.c1.type = memory +a1.channels.c1.capacity = 1000 +a1.channels.c1.transactionCapacity = 100 + +a1.sources.r1.channels = c1 +a1.sinks.k1.channel = c1</pre> +<p> + Then you'd configure the ExecuteFlumeSource as follows: +</p> +<table id="example"> + <tr> + <th>Property</th> + <th>Value</th> + </tr> + <tr> + <td>Source Type</td> + <td>multiport_syslogtcp</td> + </tr> + <tr> + <td>Agent Name</td> + <td>a1</td> + </tr> + <tr> + <td>Source Name</td> + <td>r1</td> + </tr> + <tr> + <td>Flume Configuration</td> + <td> + <code> + a1.sources.r1.host = 0.0.0.0<br> + a1.sources.r1.ports = 10001 10002 10003<br> + a1.sources.r1.portHeader = port + </code> + </td> + </tr> +</table> +</body> +</html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.2.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ExecuteFlumeSource</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ExecuteFlumeSource</h1><h2>Description: </h2><p>Execute a Flume source. Each Flume Event is sent to the success relationship as a FlowFile</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>flume, hadoop, get, source, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr ><td id="name"><strong>Source Type</strong></td><td >id="default-value"></td><td id="allowable-values"></td><td >id="description">The component type name for the source. For some sources, >this is a short, symbolic name (e.g. spooldir). For others, it's the >fully-qualified name of the Source class. See the Flume User Guide for >details.</td></tr><tr><td id="name"><strong>Agent Name</strong></td><td >id="default-value">tier1</td><td id="allowable-values"></td><td >id="description">The name of the agent used in the Flume source >configuration</td></tr><tr><td id="name"><strong>Source Name</strong></td><td >id="default-value">src-1</td><td id="allowable-values"></td><td >id="description">The name of the source used in the Flume source >configuration</td></tr><tr><td id="name"><strong>Flume >Configuration</strong></td><td id="default-value"></td><td >id="allowable-values"></td><td id="description">The Flume configuration for >the source copied from the flume.properties file</td></tr></table><h3>Rela tionships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td></td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to execute arbitrary Flume configurations assuming all permissions that NiFi has.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>GCPCredentialsControllerService</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">GCPCredentialsControllerService</h1><h2>Description: </h2><p>Defines credentials for Google Cloud Platform processors. Uses Application Default credentials without configuration. Application Default credentials support environmental variable (GOOGLE_APPLICATION_CREDENTIALS) pointing to a credential file, the config generated by `gcloud auth application-default login`, AppEngine/Compute Engine service accounts, etc.</p><h3>Tags: </h3><p>gcp, credentials, provider</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any oth er properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Use Application Default Credentials</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, uses Google Application Default Credentials, which checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filepath to a service account JSON key, the config generated by the gcloud sdk, the A pp Engine service account, and the Compute Engine service account.</td></tr><tr><td id="name">Use Compute Engine Credentials</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, uses Google Compute Engine Credentials of the Compute Engine VM Instance which NiFi is running on.</td></tr><tr><td id="name">Service Account JSON File</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Path to a file containing a Service Account key file in JSON format.</td></tr><tr><td id="name">Service Account JSON</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The raw JSON containing a Service Account keyfile.<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.</body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>DeleteGCSObject</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">DeleteGCSObject</h1><h2>Description: </h2><p>Deletes objects from a Google Cloud Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.</p><h3>Tags: </h3><p>google cloud, gcs, google, storage, delete</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properti es"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><br/><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td id="default-value">6</td><td id="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong> Bucket</strong></td><td id="default-value">${gcs.bucket}</td><td id="allowable-values"></td><td id="description">Bucket of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Key</strong></td><td id="default-value">${filename}</td><td id="allowable-values"></td><td id="description">Name of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Generation</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The generation of the object to be deleted. If null, will use latest version of the object.<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google Cl oud Storage operation fails.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html">PutGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html">FetchGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html">ListGCSBucket</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchGCSObject</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchGCSObject</h1><h2>Description: </h2><p>Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.</p><h3>Tags: </h3><p>google cloud, google, storage, gcs, fetch</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><br/><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Num ber of retries</strong></td><td id="default-value">6</td><td id="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucket</strong></td><td id="default-value">${gcs.bucket}</td><td id="allowable-values"></td><td id="description">Bucket of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Key</strong></td><td id="default-value">${filename}</td><td id="allowable-values"></td><td id="description">Name of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Object Generation</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The generation of the Object to download. If null, will download latest generation.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Server Side Encryption Key</td><td id="default-value"></td><td i d="allowable-values"></td><td id="description">An AES256 Key (encoded in base64) which the object has been encrypted in.<br/><strong>Sensitive Property: true</strong><br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google Cloud Storage operation fails.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The name of the file, parsed if possible from the Content-Disposition response header</td></tr><tr><td>gcs.bucket</td><td>Bucket of the object.</td></tr><tr><td>gcs.key</td><td>Name of the object.</td></tr><tr><td>gcs.size</td><td>Size of the o bject.</td></tr><tr><td>gcs.cache.control</td><td>Data cache control of the object.</td></tr><tr><td>gcs.component.count</td><td>The number of components which make up the object.</td></tr><tr><td>gcs.content.disposition</td><td>The data content disposition of the object.</td></tr><tr><td>gcs.content.encoding</td><td>The content encoding of the object.</td></tr><tr><td>gcs.content.language</td><td>The content language of the object.</td></tr><tr><td>mime.type</td><td>The MIME/Content-Type of the object</td></tr><tr><td>gcs.crc32c</td><td>The CRC32C checksum of object's data, encoded in base64 in big-endian order.</td></tr><tr><td>gcs.create.time</td><td>The creation time of the object (milliseconds)</td></tr><tr><td>gcs.update.time</td><td>The last modification time of the object (milliseconds)</td></tr><tr><td>gcs.encryption.algorithm</td><td>The algorithm used to encrypt the object.</td></tr><tr><td>gcs.encryption.sha256</td><td>The SHA256 hash of the key used to encrypt the objec t</td></tr><tr><td>gcs.etag</td><td>The HTTP 1.1 Entity tag for the object.</td></tr><tr><td>gcs.generated.id</td><td>The service-generated for the object</td></tr><tr><td>gcs.generation</td><td>The data generation of the object.</td></tr><tr><td>gcs.md5</td><td>The MD5 hash of the object's data encoded in base64.</td></tr><tr><td>gcs.media.link</td><td>The media download link to the object.</td></tr><tr><td>gcs.metageneration</td><td>The metageneration of the object.</td></tr><tr><td>gcs.owner</td><td>The owner (uploader) of the object.</td></tr><tr><td>gcs.owner.type</td><td>The ACL entity type of the uploader of the object.</td></tr><tr><td>gcs.uri</td><td>The URI of the object as a string.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html">ListGCSBucket</a>, <a href="../org.apache.nifi.processors.gcp. storage.PutGCSObject/index.html">PutGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html">DeleteGCSObject</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ListGCSBucket</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ListGCSBucket</h1><h2>Description: </h2><p>Retrieves a listing of objects from an GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.</p><h3>Tags: </h3><p>google cloud, google, storage, gcs, list</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</str ong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><br/><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td id="default-value">6</td><td id="allowable-values"></td><td id ="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucket</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Bucket of the object.</td></tr><tr><td id="name">Prefix</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The prefix used to filter the object list. In most cases, it should end with a forward slash ('/').</td></tr><tr><td id="name"><strong>Use Generations</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Specifies whether to use GCS Generations, if applicable. If false, only the latest version of each object will be returned.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The name of the file</td></tr><tr><td>gcs.bucket</td><td>Bucket of the object.</td></tr><tr><td>gcs.key</td><td>Name of the object.</td></tr><tr><td>gcs.size</td><td>Size of the object.</td></tr><tr><td>gcs.cache.control</td><td>Data cache control of the object.</td></tr><tr><td>gcs.component.count</td><td>The number of components which make up the object.</td></tr><tr><td>gcs.content.disposition</td><td>The data content disposition of the object.</td></tr><tr><td>gcs.content.encoding</td><td>The content encoding of the object.</td></tr><tr><td>gcs.content.language</td><td>The content language of the object.</td></tr><tr><td>mime.type</td><td>The MIME/Content-Type of the object</td></tr><tr><td>gcs.crc32c</td><td>The CRC32C checksum of object's data, encoded in base64 in big-endian order.</ td></tr><tr><td>gcs.create.time</td><td>The creation time of the object (milliseconds)</td></tr><tr><td>gcs.update.time</td><td>The last modification time of the object (milliseconds)</td></tr><tr><td>gcs.encryption.algorithm</td><td>The algorithm used to encrypt the object.</td></tr><tr><td>gcs.encryption.sha256</td><td>The SHA256 hash of the key used to encrypt the object</td></tr><tr><td>gcs.etag</td><td>The HTTP 1.1 Entity tag for the object.</td></tr><tr><td>gcs.generated.id</td><td>The service-generated for the object</td></tr><tr><td>gcs.generation</td><td>The data generation of the object.</td></tr><tr><td>gcs.md5</td><td>The MD5 hash of the object's data encoded in base64.</td></tr><tr><td>gcs.media.link</td><td>The media download link to the object.</td></tr><tr><td>gcs.metageneration</td><td>The metageneration of the object.</td></tr><tr><td>gcs.owner</td><td>The owner (uploader) of the object.</td></tr><tr><td>gcs.owner.type</td><td>The ACL entity type of the uploader of the object.</td></tr><tr><td>gcs.uri</td><td>The URI of the object as a string.</td></tr></table><h3>State management: </h3><table id="stateful"><tr><th>Scope</th><th>Description</th></tr><tr><td>CLUSTER</td><td>After performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.</td></tr></table><h3>Restricted: </h3>This component is not restricted.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html">PutGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html">DeleteGCSObject</a>, <a href="../org .apache.nifi.processors.gcp.storage.FetchGCSObject/index.html">FetchGCSObject</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.2.0/org.apache.nifi.processors.gcp.storage.PutGCSObject/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>PutGCSObject</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">PutGCSObject</h1><h2>Description: </h2><p>Puts flow files to a Google Cloud Bucket.</p><h3>Tags: </h3><p>google, google cloud, gcs, archive, put</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the <strong>nifi.properties</strong> file has an entry for the property <strong>nifi.sensitive.props.key</strong>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>GCP Credentials Provider Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>GCPCredentialsService<br/><strong>Implementation:</strong><br/><a href="../org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html">GCPCredentialsControllerService</a></td><td id="description">The Controller Service used to obtain Google Cloud Platform credentials.</td></tr><tr><td id="name"><strong>Project ID</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Google Cloud Project ID</td></tr><tr><td id="name"><strong>Number of retries</strong></td><td id="default-value">6</td> <td id="allowable-values"></td><td id="description">How many retry attempts should be made before routing to the failure relationship.</td></tr><tr><td id="name"><strong>Bucket</strong></td><td id="default-value">${gcs.bucket}</td><td id="allowable-values"></td><td id="description">Bucket of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Key</strong></td><td id="default-value">${filename}</td><td id="allowable-values"></td><td id="description">Name of the object.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Content Type</td><td id="default-value">${mime.type}</td><td id="allowable-values"></td><td id="description">Content Type for the file, i.e. text/plain<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">MD5 Hash</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">MD5 Hash (encoded in Base64) of the file for server-side valid ation.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">CRC32C Checksum</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">CRC32C Checksum (encoded in Base64, big-Endian order) of the file for server-side validation.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Object ACL</td><td id="default-value"></td><td id="allowable-values"><ul><li>All Authenticated Users <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER and WRITER permissions. All other permissions are removed." title="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER and WRITER permissions. All other permissions are removed."></img></li><li>Authenticated Read <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER permission. All other permissions are removed." title="Gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER permission. All other permissions are removed."></img></li><li>Bucket Owner Full Control <img src="../../../../../html/images/iconInfo.png" alt="Gives the object and bucket owners OWNER permission. All other permissions are removed." title="Gives the object and bucket owners OWNER permission. All other permissions are removed."></img></li><li>Bucket Owner Read Only <img src="../../../../../html/images/iconInfo.png" alt="Gives the object owner OWNER permission, and gives the bucket owner READER permission. All other permissions are removed." title="Gives the object owner OWNER permission, and gives the bucket owner READER permission. All other permissions are removed."></img></li><li>Private <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER permission for a bucket or object, and removes all other access permissions." title="Gives the bucket or object owner OWNER permission for a bucket or object, and removes all other access permissions."></img></li><li>Project Private <img src="../../../../../html/images/iconInfo.png" alt="Gives permission to the project team based on their roles. Anyone who is part of the team has READER permission. Project owners and project editors have OWNER permission. This is the default ACL for newly created buckets. This is also the default ACL for newly created objects unless the default object ACL for that bucket has been changed." title="Gives permission to the project team based on their roles. Anyone who is part of the team has READER permission. Project owners and project editors have OWNER permission. This is the default ACL for newly created buckets. This is also the default ACL for newly created objects unless the default object ACL for that bucket has been changed."></img></li><li>Public Read Only <img src="../../../../../html/images/iconInfo.png" alt="Gives the bucket or object owner OWNER permission, and gives all users, both authenticated and anonymous, READER permission. When you apply this to an object, anyone on the Internet can read the object without authenticating." title="Gives the bucket or object owner OWNER permission, and gives all users, both authenticated and anonymous, READER permission. When you apply this to an object, anyone on the Internet can read the object without authenticating."></img></li></ul></td><td id="description">Access Control to be attached to the object uploaded. Not providing this will revert to bucket defaults.</td></tr><tr><td id="name">Server Side Encryption Key</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">An AES256 Encryption Key (encoded in base64) for server-side encryption of the object.<br/><strong>Sensitive Property: true</strong><br/><strong>Su pports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Overwrite Object</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If false, the upload to GCS will succeed only if the object does not exist.</td></tr><tr><td id="name">Content Disposition Type</td><td id="default-value"></td><td id="allowable-values"><ul><li>Inline <img src="../../../../../html/images/iconInfo.png" alt="Indicates that the object should be loaded and rendered within the browser." title="Indicates that the object should be loaded and rendered within the browser."></img></li><li>Attachment <img src="../../../../../html/images/iconInfo.png" alt="Indicates that the object should be saved (using a Save As... dialog) rather than opened directly within the browser" title="Indicates that the object should be saved (using a Save As... dialog) rather than opened directly within the browser"></img></li></ul></td><td id ="description">Type of RFC-6266 Content Disposition to be attached to the object</td></tr></table><h3>Dynamic Properties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Value</th><th>Description</th></tr><tr><td id="name">The name of a User-Defined Metadata field to add to the GCS Object</td><td id="value">The value of a User-Defined Metadata field to add to the GCS Object</td><td>Allows user-defined metadata to be added to the GCS object as key/value pairs<br/><strong>Supports Expression Language: true</strong></td></tr></table></p><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.</td></tr><tr><td>failure</td><td>FlowFiles are routed to this relationship if the Google Cloud Storage operation fails.</td></tr></table><h3>Reads Attrib utes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>Uses the FlowFile's filename as the filename for the GCS object</td></tr><tr><td>mime.type</td><td>Uses the FlowFile's MIME type as the content-type for the GCS object</td></tr></table><h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>gcs.bucket</td><td>Bucket of the object.</td></tr><tr><td>gcs.key</td><td>Name of the object.</td></tr><tr><td>gcs.size</td><td>Size of the object.</td></tr><tr><td>gcs.cache.control</td><td>Data cache control of the object.</td></tr><tr><td>gcs.component.count</td><td>The number of components which make up the object.</td></tr><tr><td>gcs.content.disposition</td><td>The data content disposition of the object.</td></tr><tr><td>gcs.content.encoding</td><td>The content encoding of the object.</td></tr><tr><td>gcs.content.language</td><td>The content language of the object.</td></tr><tr><td>mi me.type</td><td>The MIME/Content-Type of the object</td></tr><tr><td>gcs.crc32c</td><td>The CRC32C checksum of object's data, encoded in base64 in big-endian order.</td></tr><tr><td>gcs.create.time</td><td>The creation time of the object (milliseconds)</td></tr><tr><td>gcs.update.time</td><td>The last modification time of the object (milliseconds)</td></tr><tr><td>gcs.encryption.algorithm</td><td>The algorithm used to encrypt the object.</td></tr><tr><td>gcs.encryption.sha256</td><td>The SHA256 hash of the key used to encrypt the object</td></tr><tr><td>gcs.etag</td><td>The HTTP 1.1 Entity tag for the object.</td></tr><tr><td>gcs.generated.id</td><td>The service-generated for the object</td></tr><tr><td>gcs.generation</td><td>The data generation of the object.</td></tr><tr><td>gcs.md5</td><td>The MD5 hash of the object's data encoded in base64.</td></tr><tr><td>gcs.media.link</td><td>The media download link to the object.</td></tr><tr><td>gcs.metageneration</td><td>The metageneratio n of the object.</td></tr><tr><td>gcs.owner</td><td>The owner (uploader) of the object.</td></tr><tr><td>gcs.owner.type</td><td>The ACL entity type of the uploader of the object.</td></tr><tr><td>gcs.uri</td><td>The URI of the object as a string.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html">FetchGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html">DeleteGCSObject</a>, <a href="../org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html">ListGCSBucket</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/additionalDetails.html Tue May 9 15:27:39 2017 @@ -0,0 +1,46 @@ +<!DOCTYPE html> +<html lang="en"> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + <head> + <meta charset="utf-8" /> + <title>CreateHadoopSequenceFile</title> + + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" /> + </head> + + <body> + <!-- Processor Documentation ================================================== --> + <h2>Description:</h2> + <p>This processor is used to create a Hadoop Sequence File, which essentially is a file of key/value pairs. The key + will be a file name and the value will be the flow file content. The processor will take either a merged (a.k.a. packaged) flow + file or a singular flow file. Historically, this processor handled the merging by type and size or time prior to creating a + SequenceFile output; it no longer does this. If creating a SequenceFile that contains multiple files of the same type is desired, + precede this processor with a <code>RouteOnAttribute</code> processor to segregate files of the same type and follow that with a + <code>MergeContent</code> processor to bundle up files. If the type of files is not important, just use the + <code>MergeContent</code> processor. When using the <code>MergeContent</code> processor, the following Merge Formats are + supported by this processor: + <ul> + <li>TAR</li> + <li>ZIP</li> + <li>FlowFileStream v3</li> + </ul> + The created SequenceFile is named the same as the incoming FlowFile with the suffix '.sf'. For incoming FlowFiles that are + bundled, the keys in the SequenceFile are the individual file names, the values are the contents of each file. + </p> + NOTE: The value portion of a key/value pair is loaded into memory. While there is a max size limit of 2GB, this could cause memory + issues if there are too many concurrent tasks and the flow file sizes are large. +</body> +</html> Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>CreateHadoopSequenceFile</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">CreateHadoopSequenceFile</h1><h2>Description: </h2><p>Creates Hadoop Sequence Files from incoming flow files</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>hadoop, sequence file, create, sequencefile</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Hado op Configuration Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.</td></tr><tr><td id="name">Kerberos Principal</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Keytab</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Relogin Period</td><td id="default-value">4 hours</td><td id="allowable-values"></td><td id="description">Period of time which should pass befo re attempting a kerberos relogin</td></tr><tr><td id="name">Additional Classpath Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of paths to files and/or directories that will be added to the classpath. When specifying a directory, all files with in the directory will be added to the classpath, but further sub-directories will not be included.</td></tr><tr><td id="name">Compression type</td><td id="default-value"></td><td id="allowable-values"><ul><li>NONE</li><li>RECORD</li><li>BLOCK</li></ul></td><td id="description">Type of compression to use when creating Sequence File</td></tr><tr><td id="name"><strong>Compression codec</strong></td><td id="default-value">NONE</td><td id="allowable-values"><ul><li>NONE</li><li>DEFAULT</li><li>BZIP</li><li>GZIP</li><li>LZ4</li><li>SNAPPY</li><li>AUTOMATIC</li></ul></td><td id="description">No Description Provided.</td></tr></table><h3>Relationships: </h3><table id="relationship s"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Generated Sequence Files are sent to this relationship</td></tr><tr><td>failure</td><td>Incoming files that failed to generate a Sequence File are sent to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.hadoop.PutHDFS/index.html">PutHDFS</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.DeleteHDFS/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.DeleteHDFS/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.DeleteHDFS/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.DeleteHDFS/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>DeleteHDFS</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">DeleteHDFS</h1><h2>Description: </h2><p>Deletes one or more files or directories from HDFS. The path can be provided as an attribute from an incoming FlowFile, or a statically set path that is periodically removed. If this processor has an incoming connection, itwill ignore running on a periodic basis and instead rely on incoming FlowFiles to trigger a delete. Note that you may use a wildcard character to match multiple files or directories. If there are no incoming connections no flowfiles will be transfered to any output relationships. If there is an incoming flowfile then provided there are no de tected failures it will be transferred to success otherwise it will be sent to false. If knowledge of globbed files deleted is necessary use ListHDFS first to produce a specific list of files to delete. </p><h3>Tags: </h3><p>hadoop, HDFS, delete, remove, filesystem, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Hadoop Configuration Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search t he classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.</td></tr><tr><td id="name">Kerberos Principal</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Keytab</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Relogin Period</td><td id="default-value">4 hours</td><td id="allowable-values"></td><td id="description">Period of time which should pass before attempting a kerberos relogin</td></tr><tr><td id="name">Additional Classpath Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of paths to files and/or directories tha t will be added to the classpath. When specifying a directory, all files with in the directory will be added to the classpath, but further sub-directories will not be included.</td></tr><tr><td id="name"><strong>Path</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The HDFS file or directory to delete. A wildcard expression may be used to only delete certain files<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Recursive</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Remove contents of a non-empty directory recursively</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>When an incoming flowfile is used then if there are no errors invoking delete the flowfile will route here.</td></tr><tr><td>failure</td><td>When an incoming flowfile is used and there is a failure while deleting then the flowfile will route here.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>hdfs.filename</td><td>HDFS file to be deleted</td></tr><tr><td>hdfs.path</td><td>HDFS Path specified in the delete request</td></tr><tr><td>hdfs.error.message</td><td>HDFS error message related to the hdfs.error.code</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to delete any file that NiFi has access to in HDFS or the local filesystem.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.hadoop.ListHDFS/index.html">ListHDFS</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.FetchHDFS/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.FetchHDFS/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.FetchHDFS/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.FetchHDFS/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>FetchHDFS</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">FetchHDFS</h1><h2>Description: </h2><p>Retrieves a file from HDFS. The content of the incoming FlowFile is replaced by the content of the file in HDFS. The file in HDFS is left intact without any changes being made to it.</p><h3>Tags: </h3><p>hadoop, hdfs, get, ingest, fetch, source, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html ">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Hadoop Configuration Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.</td></tr><tr><td id="name">Kerberos Principal</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Keytab</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr>< tr><td id="name">Kerberos Relogin Period</td><td id="default-value">4 hours</td><td id="allowable-values"></td><td id="description">Period of time which should pass before attempting a kerberos relogin</td></tr><tr><td id="name">Additional Classpath Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of paths to files and/or directories that will be added to the classpath. When specifying a directory, all files with in the directory will be added to the classpath, but further sub-directories will not be included.</td></tr><tr><td id="name"><strong>HDFS Filename</strong></td><td id="default-value">${path}/${filename}</td><td id="allowable-values"></td><td id="description">The name of the HDFS file to retrieve<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Compression codec</strong></td><td id="default-value">NONE</td><td id="allowable-values"><ul><li>NONE</li><li>DEFAULT</li>< li>BZIP</li><li>GZIP</li><li>LZ4</li><li>SNAPPY</li><li>AUTOMATIC</li></ul></td><td id="description">No Description Provided.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>FlowFiles will be routed to this relationship once they have been updated with the content of the HDFS file</td></tr><tr><td>comms.failure</td><td>FlowFiles will be routed to this relationship if the content of the HDFS file cannot be retrieve due to a communications failure. This generally indicates that the Fetch should be tried again.</td></tr><tr><td>failure</td><td>FlowFiles will be routed to this relationship if the content of the HDFS file cannot be retrieved and trying again will likely not be helpful. This would occur, for instance, if the file is not found or if there is a permissions issue</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name< /th><th>Description</th></tr><tr><td>hdfs.failure.reason</td><td>When a FlowFile is routed to 'failure', this attribute is added indicating why the file could not be fetched from HDFS</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to retrieve any file that NiFi has access to in HDFS or the local filesystem.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.hadoop.ListHDFS/index.html">ListHDFS</a>, <a href="../org.apache.nifi.processors.hadoop.GetHDFS/index.html">GetHDFS</a>, <a href="../org.apache.nifi.processors.hadoop.PutHDFS/index.html">PutHDFS</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFS/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFS/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFS/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFS/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>GetHDFS</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">GetHDFS</h1><h2>Description: </h2><p>Fetch files from Hadoop Distributed File System (HDFS) into FlowFiles. This Processor will delete the file from HDFS after fetching it.</p><h3>Tags: </h3><p>hadoop, HDFS, get, fetch, ingest, source, filesystem, restricted</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><tab le id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Hadoop Configuration Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.</td></tr><tr><td id="name">Kerberos Principal</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Keytab</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Relogin Perio d</td><td id="default-value">4 hours</td><td id="allowable-values"></td><td id="description">Period of time which should pass before attempting a kerberos relogin</td></tr><tr><td id="name">Additional Classpath Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of paths to files and/or directories that will be added to the classpath. When specifying a directory, all files with in the directory will be added to the classpath, but further sub-directories will not be included.</td></tr><tr><td id="name"><strong>Directory</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The HDFS directory from which files should be read<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Recurse Subdirectories</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates whethe r to pull files from subdirectories of the HDFS directory</td></tr><tr><td id="name"><strong>Keep Source File</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Determines whether to delete the file from HDFS after it has been successfully transferred. If true, the file will be fetched repeatedly. This is intended for testing only.</td></tr><tr><td id="name">File Filter Regex</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A Java Regular Expression for filtering Filenames; if a filter is supplied then only files whose names match that Regular Expression will be fetched, otherwise all files will be fetched</td></tr><tr><td id="name"><strong>Filter Match Name Only</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true then File Filter Regex will match on just the filename, otherwise subdirectory names will be included with filename in the regex comparison</td></tr><tr><td id="name"><strong>Ignore Dotted Files</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, files whose names begin with a dot (".") will be ignored</td></tr><tr><td id="name"><strong>Minimum File Age</strong></td><td id="default-value">0 sec</td><td id="allowable-values"></td><td id="description">The minimum age that a file must be in order to be pulled; any file younger than this amount of time (based on last modification date) will be ignored</td></tr><tr><td id="name">Maximum File Age</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The maximum age that a file must be in order to be pulled; any file older than this amount of time (based on last modification date) will be ignored</td></tr><tr><td id="name"><strong>Polling Interval</strong></td><td id="default-value">0 s ec</td><td id="allowable-values"></td><td id="description">Indicates how long to wait between performing directory listings</td></tr><tr><td id="name"><strong>Batch Size</strong></td><td id="default-value">100</td><td id="allowable-values"></td><td id="description">The maximum number of files to pull in each iteration, based on run schedule.</td></tr><tr><td id="name">IO Buffer Size</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Amount of memory to use to buffer file contents during IO. This overrides the Hadoop Configuration</td></tr><tr><td id="name"><strong>Compression codec</strong></td><td id="default-value">NONE</td><td id="allowable-values"><ul><li>NONE</li><li>DEFAULT</li><li>BZIP</li><li>GZIP</li><li>LZ4</li><li>SNAPPY</li><li>AUTOMATIC</li></ul></td><td id="description">No Description Provided.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All files r etrieved from HDFS are transferred to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The name of the file that was read from HDFS.</td></tr><tr><td>path</td><td>The path is set to the relative path of the file's directory on HDFS. For example, if the Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to "./". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "abc/1/2/3".</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>Provides operator the ability to retrieve and delete any file that NiFi has access to in HDFS or the local filesystem.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.hadoop.PutHDFS/index.html">PutHDFS</a>, <a href="../o rg.apache.nifi.processors.hadoop.ListHDFS/index.html">ListHDFS</a></p></body></html> \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFSSequenceFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFSSequenceFile/index.html?rev=1794596&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFSSequenceFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.2.0/org.apache.nifi.processors.hadoop.GetHDFSSequenceFile/index.html Tue May 9 15:27:39 2017 @@ -0,0 +1 @@ +<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>GetHDFSSequenceFile</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">GetHDFSSequenceFile</h1><h2>Description: </h2><p>Fetch sequence files from Hadoop Distributed File System (HDFS) into FlowFiles</p><h3>Tags: </h3><p>hadoop, HDFS, get, fetch, ingest, source, sequence file</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>De fault Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Hadoop Configuration Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.</td></tr><tr><td id="name">Kerberos Principal</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Keytab</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your nifi.properties</td></tr><tr><td id="name">Kerberos Relogin Period</td><td id="default-value">4 hours</td>< td id="allowable-values"></td><td id="description">Period of time which should pass before attempting a kerberos relogin</td></tr><tr><td id="name">Additional Classpath Resources</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of paths to files and/or directories that will be added to the classpath. When specifying a directory, all files with in the directory will be added to the classpath, but further sub-directories will not be included.</td></tr><tr><td id="name"><strong>Directory</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The HDFS directory from which files should be read<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Recurse Subdirectories</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Indicates whether to pull files from subdirectories of the HDFS directory</td></tr><tr><td id="name"><strong>Keep Source File</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Determines whether to delete the file from HDFS after it has been successfully transferred. If true, the file will be fetched repeatedly. This is intended for testing only.</td></tr><tr><td id="name">File Filter Regex</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A Java Regular Expression for filtering Filenames; if a filter is supplied then only files whose names match that Regular Expression will be fetched, otherwise all files will be fetched</td></tr><tr><td id="name"><strong>Filter Match Name Only</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true then File Filter Regex will match on just the filename, otherwise subdirectory names will be included with f ilename in the regex comparison</td></tr><tr><td id="name"><strong>Ignore Dotted Files</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, files whose names begin with a dot (".") will be ignored</td></tr><tr><td id="name"><strong>Minimum File Age</strong></td><td id="default-value">0 sec</td><td id="allowable-values"></td><td id="description">The minimum age that a file must be in order to be pulled; any file younger than this amount of time (based on last modification date) will be ignored</td></tr><tr><td id="name">Maximum File Age</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The maximum age that a file must be in order to be pulled; any file older than this amount of time (based on last modification date) will be ignored</td></tr><tr><td id="name"><strong>Polling Interval</strong></td><td id="default-value">0 sec</td><td id="allowable-values"></td><td id="description">Indicates how long to wait between performing directory listings</td></tr><tr><td id="name"><strong>Batch Size</strong></td><td id="default-value">100</td><td id="allowable-values"></td><td id="description">The maximum number of files to pull in each iteration, based on run schedule.</td></tr><tr><td id="name">IO Buffer Size</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Amount of memory to use to buffer file contents during IO. This overrides the Hadoop Configuration</td></tr><tr><td id="name"><strong>Compression codec</strong></td><td id="default-value">NONE</td><td id="allowable-values"><ul><li>NONE</li><li>DEFAULT</li><li>BZIP</li><li>GZIP</li><li>LZ4</li><li>SNAPPY</li><li>AUTOMATIC</li></ul></td><td id="description">No Description Provided.</td></tr><tr><td id="name"><strong>FlowFile Content</strong></td><td id="default-value">VALUE ONLY</td><td id="allowable-values"><ul><li>VALUE ONLY</li><li>KEY VALUE PAIR</li></ul></td>< td id="description">Indicate if the content is to be both the key and value of the Sequence File, or just the value.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All files retrieved from HDFS are transferred to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>The name of the file that was read from HDFS.</td></tr><tr><td>path</td><td>The path is set to the relative path of the file's directory on HDFS. For example, if the Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to "./". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "abc/1/2/3".</td></tr></table><h3>State management: </h3>This component does not sto re state.<h3>Restricted: </h3>Provides operator the ability to retrieve and delete any file that NiFi has access to in HDFS or the local filesystem.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.hadoop.PutHDFS/index.html">PutHDFS</a></p></body></html> \ No newline at end of file
