Re: Regarding Nifi packaging and deployment

[Matthew Clarke]

The NiFi Expression language (EL) can also use system user environment
variables as well as NiFI JVM properties as input.  So you could create
templates that have EL statements that reference these environment
variables which would allow each system running the same dataflow to pull
in different values.  NiFI JVM properties would be added to the
bootstrap.conf file in your NiFi installation.  While system environment
variables are set at the OS level.  There is an order of preference is NiFi
will first search for FlowFIle Attributes matching the defined subject/key
name, then system environment variables, and last JVM environment variables.

On Thu, Dec 10, 2015 at 10:25 PM, Joe Witt  wrote:

> Shweta,
>
> The primary mechanism is flow templates [1].
>
> They do have some important limitations today though that you'll want
> to understand.  First, some properties are sensitive, like passwords,
> and thus are not included in the templates so you'll have to reenter
> them when you apply the template in the new environment.  Second,
> there are at times properties you'd want to have different values for
> in different environments.  We need to provide a property/env variable
> mapping mechanism.  Both of these we intend to address but neither are
> presently actively being worked as far as I'm aware of.
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates
>
> Thanks
> Joe
>
> On Thu, Dec 10, 2015 at 9:25 PM, shweta  wrote:
> > Hi all,
> >
> > I'm new to Nifi. I have created some sample flows. I want to know how
> can I
> > package and deploy the same
> > from development environment to testing environment or do I need to
> recreate
> > the entire data flow again in different environment.
> >
> > Thanks,
> > Shweta
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Regarding-Nifi-packaging-and-deployment-tp5716.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Facing Issue while connecting with HDFS

[Bryan Bende]

Digvijay,

Are you talking about Kerberos authentication to the HDFS cluster?

If so, in nifi.properties you specify your krb5.conf file:

nifi.kerberos.krb5.file=/etc/krb5.conf  (or wherever your conf file is)

 This is the file that would have your realms defined. Then on the HDFS
processors your
properties would be something like:

Kerberos Principal = myprinicpal@MYREALM
Kerberos Keytab = /etc/security/keytabs/myprinciapl.keytab

-Bryan


On Fri, Dec 11, 2015 at 6:00 AM, digvijayp 
wrote:

> Hi Bryan,
>
> Thanks for nice info...It would really help me.
> Facing one issue while designing putHDFS process.After putting the putHDFS
> configuration details I am getting the error mentioning security
> authentication .After the analysis I found that these is due to permission
> of userid.
> So where can we put the userid and password so that when it connect to hdfs
> first login by that id and then pulls back data.As I know it should be done
> in the nifi.properties file.But can you please share where I can add these
> details?
>
> Thanks
> Digvijay P.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5720.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Questions about the ordering of the FlowFile.

[Brandon DeVries]

Paresh,

 You might want to look at the PriorityAttributePrioritizer[1]:

*PriorityAttributePrioritizer*: Given two FlowFiles that both have a
"priority" attribute, the one that has the highest priority value will be
prprocessed first. Note that an UpdateAttribute processor should be used to
add the "priority" attribute to the FlowFiles before they reach a
connection that has this prioritizer set. Values for the "priority"
attribute may be alphanumeric, where "a" is a higher priority than "z", and
"1" is a higher priority than "9", for example.

 You can set a "priority" attribute in your custom processor.  However,
I would caution against absolutely relying on in-order delivery. Just
because a FlowFile begins processing first doesn't mean it will complete
first (assuming the processor has multiple concurrent tasks).  If it is
only critical that they be in order for the last processor, you might also
consider the MergeContent processor in "Defragment" mode.  Similar to the
"priority" attribute, you would set a "fragment.identifier" common to all
of the FlowFiles comprising a record, and then a "fragment.index" for each
FlowFile in the record.  At the end of the flow, you could then create a
single FlowFile comprised of all of the pieces of the record, in order.
Alternatively, you could extend the same class as MergeContent
(BinFiles[3]) in your last processor to ensure that all files are received
in order before beginning the final step.  Hope this helps

[1]
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#connecting-components
[2]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html
[3]
https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/BinFiles.java


Brandon


On Thu, Dec 10, 2015 at 11:37 PM, Paresh Shah 
wrote:

> Here’s my use case.
> We have a application protocol between the start and end processors in a
> data flow, that expect the flow files to arrive in the order they are
> generated. For e.g
>
> Start Record Flowfile
> 
> End Record Flowfile.
>
> The first processor does the following.
>
>   1.  Generates and transfers the StartRecord flow file.
>   2.  Generates data records and transfers them.
>   3.  Generates and transfers the EndRecord flow file
>
> The last processor in the data flow does the following.
>
>   1. Looks for the StartRecord flow file and does its thing.
>   2. Looks for the DataRecord flow file and does its thing.
>   3.  Looks for the EndRecord flow file and updates and cleanups up
> the target state.
>
> The first processor is doing multiple transfers on the session object
> before calling commit.
>
> We see that they are being received in random order. As a result we are
> not able to execute the app protocol. We have tried the
> FirstInFirstOutPrioritizer and OldestFlowFilePrioritizer.
>
> We would appreciate any insights into this we can get as it seems to be a
> blocking issue for us.
>
> Thanks
> Paresh
> 
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> 
>

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Matt Gilman]

+1 (binding)

On Wed, Dec 9, 2015 at 7:45 PM, Chris Lim 
wrote:

> +1 (non binding)
>
> On Thursday, 10 December 2015, Ryan Blue  wrote:
>
> > +1 (non-binding)
> >
> > Checked source artifact, spot-checked some license docs on new modules
> >
> > rb
> >
> > On 12/08/2015 02:58 PM, Ricky Saltzer wrote:
> >
> >> +1
> >>
> >> build works
> >> test works
> >> hashes/keys check out
> >> tested a couple simple workflows
> >>
> >> On Tue, Dec 8, 2015 at 4:57 PM, Joe Percivall <
> >> joeperciv...@yahoo.com.invalid> wrote:
> >>
> >> Ran through the helper: validated keys, built binaries on Windows and
> Mac,
> >>> ran some templates.
> >>> Everything worked as expected.
> >>> +1 Release this package as Apache NiFi 0.4.0. - - - - - - Joseph
> >>> Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com
> >>>
> >>>
> >>>
> >>>  On Tuesday, December 8, 2015 3:29 PM, Joe Witt <
> joew...@apache.org>
> >>> wrote:
> >>>
> >>>
> >>>   Hello NiFi Community,
> >>>
> >>> I am pleased to be calling this vote for the source release of Apache
> >>> NiFi 0.4.0.
> >>>
> >>> The source zip, including signatures, digests, and associated
> >>> convenience binaries can be found at
> >>>https://dist.apache.org/repos/dist/dev/nifi/nifi-0.4.0/
> >>>
> >>> The staged maven artifacts of the build can be found at
> >>>
> https://repository.apache.org/content/repositories/orgapachenifi-1065
> >>>
> >>> The Git tag is nifi-0.4.0-RC2
> >>> The Git commit ID is b66c029090f395c0cbc001fd483e86895b133e46
> >>>
> >>>
> >>>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b66c029090f395c0cbc001fd483e86895b133e46
> >>>
> >>> Checksums of NiFi 0.4.0 Source Release
> >>> MD5: da733f8fdb520a0346dcda59940b2c12
> >>> SHA1: 82fffbc5f8d7e4724bbe2f794bdde39396dae745
> >>>
> >>> Release artifacts are signed with the following key
> >>>https://people.apache.org/keys/committer/joewitt.asc
> >>>
> >>> KEYS file available here
> >>>https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>>
> >>> 161 issues were closed/resolved for this release
> >>>
> >>>
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12333070
> >>>
> >>> Release note highlights
> >>>
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.4.0
> >>>
> >>> Migration/Upgrade guidance
> >>>https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
> >>>https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
> >>>
> >>> The vote will be open for 72 hours.
> >>> Please download the release candidate and evaluate the necessary items
> >>> including checking hashes, signatures, build from source, and test.
> >>>
> >>> Then please vote:
> >>>
> >>> [ ] +1 Release this package as Apache NiFi 0.4.0
> >>> [ ] +0 no opinion
> >>> [ ] -1 Do not release this package because...
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Cloudera, Inc.
> >
>

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Sean Busbey]

On Fri, Dec 11, 2015 at 8:06 AM, Sean Busbey  wrote:

> +1 (binding)
>
> * verified signatures / checksums
> * source and binary artifacts passed check from RAT.
> * verified source artifact against tag
> * verified bin artifacts match each other and one generated from source
> artifact
> * spot checked licenses
> * build passes
> * bin artifact runs simple flow
>
> Ran into NIFI-1278, but I don't think it's sufficient to block the release.
>
>
>
Sorry, was in a hurry when I wrote this up. Should have been

* build passes (after accounting for failures/errors that I think are from
NIFI-1278



-- 
Sean

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Sean Busbey]

+1 (binding)

* verified signatures / checksums
* source and binary artifacts passed check from RAT.
* verified source artifact against tag
* verified bin artifacts match each other and one generated from source
artifact
* spot checked licenses
* build passes
* bin artifact runs simple flow

Ran into NIFI-1278, but I don't think it's sufficient to block the release.



On Tue, Dec 8, 2015 at 12:29 PM, Joe Witt  wrote:

> Hello NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 0.4.0.
>
> The source zip, including signatures, digests, and associated
> convenience binaries can be found at
>   https://dist.apache.org/repos/dist/dev/nifi/nifi-0.4.0/
>
> The staged maven artifacts of the build can be found at
>   https://repository.apache.org/content/repositories/orgapachenifi-1065
>
> The Git tag is nifi-0.4.0-RC2
> The Git commit ID is b66c029090f395c0cbc001fd483e86895b133e46
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b66c029090f395c0cbc001fd483e86895b133e46
>
> Checksums of NiFi 0.4.0 Source Release
> MD5: da733f8fdb520a0346dcda59940b2c12
> SHA1: 82fffbc5f8d7e4724bbe2f794bdde39396dae745
>
> Release artifacts are signed with the following key
>   https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here
>   https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 161 issues were closed/resolved for this release
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12333070
>
> Release note highlights
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.4.0
>
> Migration/Upgrade guidance
>   https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>   https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
>
> Then please vote:
>
> [ ] +1 Release this package as Apache NiFi 0.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>



-- 
Sean

[RESULT][VOTE] Release Apache NiFi 0.4.0

[Joe Witt]

Hello Apache NiFi Community,

The release of Apache NiFi 0.4.0 passes with

6 +1 (binding) votes
6 +1 (non-binding) votes
0 -1 votes.

This release is the result of a tremendous level of contribution from
many people.  Thanks to all who made this release possible.

Here is the PMC vote thread:
http://mail-archives.apache.org/mod_mbox/nifi-dev/201512.mbox/%3CCALJK9a56_OQBm64ePYVr6fM7oEe5zJ5Db0jUTvyTZ%3D%3DN7u89bw%40mail.gmail.com%3E

Thanks
Joe

XML Questions

[DAVID SMITH]

Hi

I have two questions which I think may require bespoke processors.

1)  I have extracted a set of  values from xml tags and made them into flowfile 
attributes, how a can I add these flowfile attributes to one or more flowfiles?

2)  I have multiple xml files which I need to merge into one xml file, can I do 
this with the current processor set?

Thanks
Dave

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Joe Witt]

+1 (binding)

Will send the vote result momentarily.

Thanks
Joe

On Fri, Dec 11, 2015 at 12:02 PM, Joe Skora  wrote:
> +1 (non-binding)
>
> Signatures and checksums good.
> Builds and tests without problems.
> Contrib check passes.
>
> On Tue, Dec 8, 2015 at 3:29 PM, Joe Witt  wrote:
>
>> Hello NiFi Community,
>>
>> I am pleased to be calling this vote for the source release of Apache
>> NiFi 0.4.0.
>>
>> The source zip, including signatures, digests, and associated
>> convenience binaries can be found at
>>   https://dist.apache.org/repos/dist/dev/nifi/nifi-0.4.0/
>>
>> The staged maven artifacts of the build can be found at
>>   https://repository.apache.org/content/repositories/orgapachenifi-1065
>>
>> The Git tag is nifi-0.4.0-RC2
>> The Git commit ID is b66c029090f395c0cbc001fd483e86895b133e46
>>
>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b66c029090f395c0cbc001fd483e86895b133e46
>>
>> Checksums of NiFi 0.4.0 Source Release
>> MD5: da733f8fdb520a0346dcda59940b2c12
>> SHA1: 82fffbc5f8d7e4724bbe2f794bdde39396dae745
>>
>> Release artifacts are signed with the following key
>>   https://people.apache.org/keys/committer/joewitt.asc
>>
>> KEYS file available here
>>   https://dist.apache.org/repos/dist/release/nifi/KEYS
>>
>> 161 issues were closed/resolved for this release
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12333070
>>
>> Release note highlights
>>
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.4.0
>>
>> Migration/Upgrade guidance
>>   https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>>   https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>>
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test.
>>
>> Then please vote:
>>
>> [ ] +1 Release this package as Apache NiFi 0.4.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
>>

Moving BinFiles

[Joe Gresock]

In NIFI-305, we refactored a parent class for MergeContent so the binning
functionality could be reused.  In practice, this isn't quite useful yet,
because BinFiles is still in the nifi-standard-processors, which contains
an org.apache.nifi.processor.Processor file.  Therefore, any processor that
tries to extend BinFiles must automatically pull in all the processors in
nifi-standard-processors, which is probably undesirable if it's an
extension.

Is there anywhere in the code we could put parent classes like this for
extensibility purposes?

Thanks,
Joe

-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: Force Processor to stop

[Joe Witt]

Ian,

There is no forced stop/interrupt mechanism.  This is being addressed
in NIFI-1183 and NIFI-78.

Ultimately there are some cases where we simply cannot force java to
give us back our thread.  However, we should certainly be able to
quarantine the thread itself and move on to doing other work.

Thanks
Joe

On Fri, Dec 11, 2015 at 6:39 PM, ianwork  wrote:
> Is there a function to call to force a processor shutdown?
>
> Ian
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Force-Processor-to-stop-tp5732.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.