Change data capture processor

[Buntu Dev]

Is the new CaptureChangeMySQL processor available in 1.2 and is there any
documentation available for this processor?

 https://issues.apache.org/jira/browse/NIFI-3413


Thanks!

Re: MaxFileSize in timeBasedFileNamingAndTriggeringPolicy not work?

[Jeff]

Hello Prabhu,

I think you're running into a logback bug [1] that is fixed with 1.1.7.
Unfortunately, it looks like NiFi 0.6.1 is using logback 1.1.3.

[1] https://jira.qos.ch/browse/LOGBACK-747

On Tue, Apr 11, 2017 at 8:37 AM prabhu Mahendran 
wrote:

> In NiFi-0.6.1 ,i have try to reduce size of nifi-app.log to be stored in
> local directory.
>
> In that conf\logback.xml i have configured "MaxFileSize" to be 1MB.I think
> this only stores nifi-app.log should be under 1 MB Size only.But it doesn't
> do like that.It always store every logs.
>
>  class="ch.qos.logback.core.rolling.RollingFileAppender">
> logs/nifi-app.log
>  class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> 
> 
> ./logs/nifi-app_%d{-MM-dd_HH}.%i.log
>  class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
> 1MB
> 
> 
> 1
> 
> 
> %date %level [%thread] %logger{40} %msg%n
> true
> 
> 
>
> Now i need to set 1MB for size of an nifi-app.log.
>
> *How to set size for nifi-app.log?*
>

Re: Logging changes to workflow

[Joe Witt]

John

The code i referenced in the previous note on this thread points to
the API support for connecting a reporting task to the audit data.
This will give you more stable/reliable information than having a
Logger send the data via syslog/etc..  However, there is no
ReportingTask that I am aware of available in the Apache NiFi codebase
that leverages that function so for now you'd want to write a custom
one with Java or with the scripted reporting task support.

Thanks
Joe

On Tue, Apr 11, 2017 at 2:31 AM, HARRIOTT, John
 wrote:
> Hi Joe,
>
> What is meant by the following statement: "From that one can access the list 
> of changes from a given offset onward."
> ReportingTask.java is an interface class. That does mean there exists a class 
> specifically for reporting configuration changes?
>
> Regards John
>
> -Original Message-
> From: Joe Witt [mailto:joe.w...@gmail.com]
> Sent: Tuesday, 11 April 2017 4:02 AM
> To: users@nifi.apache.org
> Subject: Re: Logging changes to workflow
>
> Juan,
>
> They were introduced in https://issues.apache.org/jira/browse/NIFI-986
>
> I don't know if there is a reporting task which leverages this yet but the 
> key classes to notice are here:
>  ReportingTask [1]
>  EventAccess [2]
>
> From that one can access the list of changes from a given offset onward.
>
> [1] 
> https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/reporting/ReportingTask.java
> [2] 
> https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/reporting/EventAccess.java
>
> Thanks
> Joe
>
> On Mon, Apr 10, 2017 at 2:25 PM, Juan Sequeiros  wrote:
>> Hi Joe / all,
>>
>> " we make it available via reporting tasks "
>>
>> What reporting task is that? I've looked at NIFI 1 available reporting
>> tasks and I don't see anything that would let me export "flow
>> configuration history".
>> Coincidentally we are having to meet similar requirement.
>>
>> Our solution was to get that through restAPI but if there is a cleaner
>> way though reporting tasks, would like to experiment with that too.
>>
>> On Mon, Apr 10, 2017 at 9:09 AM Joe Witt  wrote:
>>>
>>> John
>>>
>>> To build on Andrew's comments there is already a very significant
>>> amount of audit data made available for flow changes.  We store this
>>> in an audit database so we can drive some user experiences off it and
>>> we make it available via reporting tasks so one could report this
>>> information to an external system.  I believe what you're asking for
>>> is well supported.
>>>
>>> In addition to that sort of configuraiton auditing remember too we
>>> have actual data provenance which shows not just what the
>>> configuration was but the lineage/path that data actually takes.
>>> These too together form an extremely powerful audit basis.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Mon, Apr 10, 2017 at 9:05 AM, Andrew Grande  wrote:
>>> > John, the Flow Configuration History menu item already captures it.
>>> > Does it look like a fit?
>>> >
>>> > Andrew
>>> >
>>> >
>>> > On Sun, Apr 9, 2017, 10:50 PM HARRIOTT, John
>>> > 
>>> > wrote:
>>> >>
>>> >> Is it possible to log changes to a workflow so they can be
>>> >> captured by a central logging/auditing capability e.g. syslog.
>>> >>
>>> >> This will allow security auditors to detect changes to a systems
>>> >> configuration e.g. addition/removal/modification of a processor(s).
>>> >>
>>> >>
>>> >>
>>> >> Thanks
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> This email has been sent on behalf of one of the following
>>> >> companies within the BAE Systems Australia group of companies:
>>> >>
>>> >> BAE Systems Australia Limited - Australian Company Number 008
>>> >> 423
>>> >> 005
>>> >> BAE Systems Australia Defence Pty Limited - Australian Company
>>> >> Number
>>> >> 006 870 846
>>> >> BAE Systems Australia Logistics Pty Limited - Australian
>>> >> Company Number 086 228 864
>>> >>
>>> >> Our registered office is Evans Building, Taranaki Road, Edinburgh
>>> >> Parks, Edinburgh, South Australia, 5111. If the identity of the
>>> >> sending company is not clear from the content of this email please
>>> >> contact the sender.
>>> >>
>>> >> This email and any attachments may contain confidential and
>>> >> legally privileged information.  If you are not the intended
>>> >> recipient, do not copy or disclose its content, but please reply
>>> >> to this email immediately and highlight the error to the sender
>>> >> and then immediately delete the message.
> This email has been sent on behalf of one of the following companies within 
> the BAE Systems Australia group of companies:
>
> BAE Systems Australia Limited - Australian Company Number 008 423 005
> BAE Systems Australia Defence Pty Limited - Australian Company Number 006 
> 870 846
> BAE Systems Australia Logistics Pty Limited - Australian 

Re: parsing html

[Jeremy Dyer]

Sven,

There is also the GetHTML processor I added awhile back. If the input is valid 
HTML you should always be able to use a CSS selector to extract that HTML 
value. If you can provide a sample of the HTML I would be glad to make a flow 
for you doing so as an example

Jeremy

Sent from my iPhone

> On Apr 11, 2017, at 1:01 PM, Andy LoPresto  wrote:
> 
> Sven,
> 
> Currently I would recommend using ExecuteScript and simply streaming & 
> slicing the content bytes at line 10 (a one-line operation in Groovy, I 
> believe the same in Ruby and Python). 
> 
> This isn’t the first time I’ve heard of a similar request though, so I think 
> if you were to open a Jira requesting a “GetLine(s)” or “SliceText” 
> processor, it could be valuable to the community. The current component 
> solution would probably involve SplitText/SplitContent and as you said, 
> decent overhead, especially if the desired content is early in the flowfile. 
> 
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Apr 11, 2017, at 9:38 AM, Sven Davison  wrote:
>> 
>> I'm looking to parse some HTML. It's not the cleanest but i know that my 
>> content is always on line 10 of the file. I could use splittext then compare 
>> it to ensure it starts with XYZBeginningString, i supose.. but i'm looking 
>> for something w/ less overhead. Especially knowing the content is always on 
>> line 10.
>> 
>> Anyone have other/cleaner ideas on how to get the content of line 10?
> 

Re: parsing html

[Andy LoPresto]

Sven,

Currently I would recommend using ExecuteScript and simply streaming & slicing 
the content bytes at line 10 (a one-line operation in Groovy, I believe the 
same in Ruby and Python).

This isn’t the first time I’ve heard of a similar request though, so I think if 
you were to open a Jira requesting a “GetLine(s)” or “SliceText” processor, it 
could be valuable to the community. The current component solution would 
probably involve SplitText/SplitContent and as you said, decent overhead, 
especially if the desired content is early in the flowfile.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 11, 2017, at 9:38 AM, Sven Davison  wrote:
> 
> I'm looking to parse some HTML. It's not the cleanest but i know that my 
> content is always on line 10 of the file. I could use splittext then compare 
> it to ensure it starts with XYZBeginningString, i supose.. but i'm looking 
> for something w/ less overhead. Especially knowing the content is always on 
> line 10.
> 
> Anyone have other/cleaner ideas on how to get the content of line 10?



signature.asc
Description: Message signed with OpenPGP using GPGMail

parsing html

[Sven Davison]

I'm looking to parse some HTML. It's not the cleanest but i know that my
content is always on line 10 of the file. I could use splittext then
compare it to ensure it starts with XYZBeginningString, i supose.. but i'm
looking for something w/ less overhead. Especially knowing the content is
always on line 10.

Anyone have other/cleaner ideas on how to get the content of line 10?

Re: Logging changes to workflow

[Toivo Adams]

Hi,



It seems such a reporting task is not yet implemented.



I wonder if you need only output to syslog, can’t this be configured adding
*SyslogAppender* to logback.xml configuration?



And maybe new Logger is also needed in logback.xml?

Which might be Logger name?



Thanks

Toivo



2017-04-11 9:31 GMT+03:00 HARRIOTT, John :

> Hi Joe,
>
> What is meant by the following statement: "From that one can access the
> list of changes from a given offset onward."
> ReportingTask.java is an interface class. That does mean there exists a
> class specifically for reporting configuration changes?
>
> Regards John
>
> -Original Message-
> From: Joe Witt [mailto:joe.w...@gmail.com]
> Sent: Tuesday, 11 April 2017 4:02 AM
> To: users@nifi.apache.org
> Subject: Re: Logging changes to workflow
>
> Juan,
>
> They were introduced in https://issues.apache.org/jira/browse/NIFI-986
>
> I don't know if there is a reporting task which leverages this yet but the
> key classes to notice are here:
>  ReportingTask [1]
>  EventAccess [2]
>
> From that one can access the list of changes from a given offset onward.
>
> [1] https://github.com/apache/nifi/blob/master/nifi-api/src/
> main/java/org/apache/nifi/reporting/ReportingTask.java
> [2] https://github.com/apache/nifi/blob/master/nifi-api/src/
> main/java/org/apache/nifi/reporting/EventAccess.java
>
> Thanks
> Joe
>
> On Mon, Apr 10, 2017 at 2:25 PM, Juan Sequeiros 
> wrote:
> > Hi Joe / all,
> >
> > " we make it available via reporting tasks "
> >
> > What reporting task is that? I've looked at NIFI 1 available reporting
> > tasks and I don't see anything that would let me export "flow
> > configuration history".
> > Coincidentally we are having to meet similar requirement.
> >
> > Our solution was to get that through restAPI but if there is a cleaner
> > way though reporting tasks, would like to experiment with that too.
> >
> > On Mon, Apr 10, 2017 at 9:09 AM Joe Witt  wrote:
> >>
> >> John
> >>
> >> To build on Andrew's comments there is already a very significant
> >> amount of audit data made available for flow changes.  We store this
> >> in an audit database so we can drive some user experiences off it and
> >> we make it available via reporting tasks so one could report this
> >> information to an external system.  I believe what you're asking for
> >> is well supported.
> >>
> >> In addition to that sort of configuraiton auditing remember too we
> >> have actual data provenance which shows not just what the
> >> configuration was but the lineage/path that data actually takes.
> >> These too together form an extremely powerful audit basis.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Mon, Apr 10, 2017 at 9:05 AM, Andrew Grande 
> wrote:
> >> > John, the Flow Configuration History menu item already captures it.
> >> > Does it look like a fit?
> >> >
> >> > Andrew
> >> >
> >> >
> >> > On Sun, Apr 9, 2017, 10:50 PM HARRIOTT, John
> >> > 
> >> > wrote:
> >> >>
> >> >> Is it possible to log changes to a workflow so they can be
> >> >> captured by a central logging/auditing capability e.g. syslog.
> >> >>
> >> >> This will allow security auditors to detect changes to a systems
> >> >> configuration e.g. addition/removal/modification of a processor(s).
> >> >>
> >> >>
> >> >>
> >> >> Thanks
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> This email has been sent on behalf of one of the following
> >> >> companies within the BAE Systems Australia group of companies:
> >> >>
> >> >> BAE Systems Australia Limited - Australian Company Number 008
> >> >> 423
> >> >> 005
> >> >> BAE Systems Australia Defence Pty Limited - Australian Company
> >> >> Number
> >> >> 006 870 846
> >> >> BAE Systems Australia Logistics Pty Limited - Australian
> >> >> Company Number 086 228 864
> >> >>
> >> >> Our registered office is Evans Building, Taranaki Road, Edinburgh
> >> >> Parks, Edinburgh, South Australia, 5111. If the identity of the
> >> >> sending company is not clear from the content of this email please
> >> >> contact the sender.
> >> >>
> >> >> This email and any attachments may contain confidential and
> >> >> legally privileged information.  If you are not the intended
> >> >> recipient, do not copy or disclose its content, but please reply
> >> >> to this email immediately and highlight the error to the sender
> >> >> and then immediately delete the message.
> This email has been sent on behalf of one of the following companies
> within the BAE Systems Australia group of companies:
>
> BAE Systems Australia Limited - Australian Company Number 008 423 005
> BAE Systems Australia Defence Pty Limited - Australian Company Number
> 006 870 846
> BAE Systems Australia Logistics Pty Limited - Australian Company
> Number 086 228 864
>
> Our registered office is Evans Building, Taranaki Road, Edinburgh Parks,
> Edinburgh, South 

Re: How to configure logging via SLF4J ?

[Russell Bateman]

Take a look at ${NIFI_ROOT}//conf/logback.xml/. If you've developed a 
processor and are asking about it, likely this question would fit better 
in d...@nifi.apache.org, no?


Hope this helps.


On 04/11/2017 08:24 AM, Vince Cole wrote:

Hi

I have developed a processor which uses qpid-jms-client to send
messages to ActiveMQ. The jms client uses SLF4J and I need to
configure that via NIFI... how?


Thanks

How to configure logging via SLF4J ?

[Vince Cole]

Hi

I have developed a processor which uses qpid-jms-client to send messages to
ActiveMQ. The jms client uses SLF4J and I need to configure that via
NIFI... how?


Thanks

Re: Location for script using ExecuteStreamCommand

[James McMahon]

We also keep all our files including scripts that are specific to our NiFi
installation in a Git project.

On Tue, Apr 11, 2017 at 9:49 AM, James McMahon  wrote:

> Good morning Jennifer. We have been working with NiFi for only a year and
> so can only offer you that level of "hands on" experience. With that in
> mind, we established a scripts subdirectory at the same level as the NiFi
> config, logs, etc. This does mean that we must make migrating that folder
> part of our upgrade process when we do upgrade, but that does not impose
> any impediment. And since we link to the latest installation instance
> through /home/nifi/nifi/latest/[config, logs, scripts, etc etc], we never
> need to change the script directory location in our ExecuteScript or
> InvokeScriptedProcessor processors following upgrades.
>
> One other comment: some of the more experienced users had suggested that
> we not use ExecuteStreamCommand. And in fact I have encountered situations
> where ExecuteStreamCommand would deep dive into a large file and never
> return a result or error code. I have no empirical evidence to offer, but
> we have since heeded that advice and stuck to ExecuteScript and
> InvokeScriptedProcessor. -Jim
>
> On Tue, Apr 11, 2017 at 9:37 AM, Tress, Jennifer L. 
> wrote:
>
>> Is there a best practice for where to store scripts used by
>> ExecuteStreamCommand processors?
>>
>>
>>
>>
>>
>> *Jennifer Tress*
>>
>>
>>
>
>

Re: Location for script using ExecuteStreamCommand

[James McMahon]

Good morning Jennifer. We have been working with NiFi for only a year and
so can only offer you that level of "hands on" experience. With that in
mind, we established a scripts subdirectory at the same level as the NiFi
config, logs, etc. This does mean that we must make migrating that folder
part of our upgrade process when we do upgrade, but that does not impose
any impediment. And since we link to the latest installation instance
through /home/nifi/nifi/latest/[config, logs, scripts, etc etc], we never
need to change the script directory location in our ExecuteScript or
InvokeScriptedProcessor processors following upgrades.

One other comment: some of the more experienced users had suggested that we
not use ExecuteStreamCommand. And in fact I have encountered situations
where ExecuteStreamCommand would deep dive into a large file and never
return a result or error code. I have no empirical evidence to offer, but
we have since heeded that advice and stuck to ExecuteScript and
InvokeScriptedProcessor. -Jim

On Tue, Apr 11, 2017 at 9:37 AM, Tress, Jennifer L. 
wrote:

> Is there a best practice for where to store scripts used by
> ExecuteStreamCommand processors?
>
>
>
>
>
> *Jennifer Tress*
>
>
>

Re: Location for script using ExecuteStreamCommand

[Bryan Rosander]

Hi Jennifer,

I would think that for a script of moderate or more complexity, having it
versioned (git, svn, etc) would be useful.  Also, it should definitely not
be writable by users other than (and possibly including) NiFi as it could
be modified to do something malicious.

Thanks,
Bryan

On Tue, Apr 11, 2017 at 9:37 AM, Tress, Jennifer L. 
wrote:

> Is there a best practice for where to store scripts used by
> ExecuteStreamCommand processors?
>
>
>
>
>
> *Jennifer Tress*
>
>
>

Location for script using ExecuteStreamCommand

[Tress, Jennifer L.]

Is there a best practice for where to store scripts used by 
ExecuteStreamCommand processors?


Jennifer Tress

MaxFileSize in timeBasedFileNamingAndTriggeringPolicy not work?

[prabhu Mahendran]

In NiFi-0.6.1 ,i have try to reduce size of nifi-app.log to be stored in
local directory.

In that conf\logback.xml i have configured "MaxFileSize" to be 1MB.I think
this only stores nifi-app.log should be under 1 MB Size only.But it doesn't
do like that.It always store every logs.


logs/nifi-app.log



./logs/nifi-app_%d{-MM-dd_HH}.%i.log

1MB


1


%date %level [%thread] %logger{40} %msg%n
true



Now i need to set 1MB for size of an nifi-app.log.

*How to set size for nifi-app.log?*

RE: GetHDFS from Azure Blob

[Giovanni Lanzani]

Hi Bryan,

That indeed did the trick (2.8.0 has been out for a while).

All the tests passed. For future reference, and for anybody trying this on a 
HDInsight managed node, this is what I had to do

Create a wasb-site.xml with inside the following



fs.wasb.impl
org.apache.hadoop.fs.azure.NativeAzureFileSystem



In the Hadoop configuration resources, I’ve put

/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml,/path/to/wasb-site.xml

And in the Additional Classpath resource

/usr/hdp/current/hadoop-client,/usr/hdp/current/hadoop-client/lib

That way, if the edge node is managed by HDInsight/Ambari, there’s *no* need to 
pass an unencrypted storage key, as it will automatically use the (encrypted) 
one provided by HDInsight.

Thanks for the pointers everyone!

Cheers,

Giovanni Lanzani
Chief Science Officer GoDataDriven
T: @gglanzani
M: +31 6 5120 6163

From: Bryan Bende
Sent: Tuesday, April 4, 2017 5:41 PM
To: users@nifi.apache.org
Subject: Re: GetHDFS from Azure Blob

Giovanni,

In the pom.xml at the root of the NiFi source tree:

2.7.3

You can change that to 2.8.0 (if 2.8.0 is released) and then run a
full build, assuming 2.8.0 doesn't break any code that NiFi is using.

I don't really view this as the same issue as NIFI-1922... even if the
solution to NIFI-1922 was to directly bundle the Azure/Wasb JARs in
NiFi, we would still have to bundle the JARs that are compatible with
the Hadoop client we are using, which is currently 2.7.3.

In the future when we have an extension registry, we could presumably
publish variations of the nifi-hadoop-nar + nifi-hadoop-libraries-nar
built against different versions of the Hadoop client (2.7.x, 2.8.x,
HDP, CDH, MapR, etc), and with the component versioning work currently
going on in master, it would be easy for people to run as many of
these in parallel as they want.

For now I think the easiest thing to do is maintain your own build of
nifi-hadoop-libraries-nar by changing the the version mentioned above.

At some point the NiFi community will likely move to a newer Hadoop
client as they come out (we fairly recently moved from 2.6.x to
2.7.x), but this a bigger decision that depends on how stable the
client is and what (if any) ramifications it has for compatibility.

Thanks,

Bryan



On Tue, Apr 4, 2017 at 11:20 AM, Giovanni Lanzani
 wrote:
> Hi Brian,
>
> Thanks for the reply.
>
> Is there a way to compile NiFi using the Hadoop 2.8.0 libraries?
>
> It's of course unfortunate, but the libraries you mentioned before works in 
> their very specific version. Once you use a newer version (like 
> azure-storage-2.2.0) then things seem to break.
>
> Maybe this jira [^1] could be reopened then? 
>
> Cheers,
>
> Giovanni
>
> [^1]: https://issues.apache.org/jira/browse/NIFI-1922
>
>> -Original Message-
>> From: Bryan Bende [mailto:bbe...@gmail.com]
>> Sent: Tuesday, April 4, 2017 3:59 PM
>> To: users@nifi.apache.org
>> Subject: Re: GetHDFS from Azure Blob
>>
>> Giovanni,
>>
>> I'm not that familiar with using a key provider, but NiFi currently bundles 
>> the
>> Hadoop 2.7.3 client, and looking at ProviderUtils from 2.7.3, there doesn't
>> appear to be a method
>> "excludeIncompatibleCredentialProviders":
>>
>> https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-common-
>> project/hadoop-
>> common/src/main/java/org/apache/hadoop/security/ProviderUtils.java
>>
>> It looks like it is introduced in 2.8.0:
>>
>> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-
>> project/hadoop-
>> common/src/main/java/org/apache/hadoop/security/ProviderUtils.java
>>
>> Most likely some code that is present in one of the JARs specified through
>> Additional Resources is dependent on Hadoop 2.8.0, and since NiFi is bundling
>> 2.7.3, there are some things not lining up.
>>
>> -Bryan
>>
>>
>> On Tue, Apr 4, 2017 at 9:50 AM, Giovanni Lanzani
>>  wrote:
>> > Bryan,
>> >
>> > Allow me to chime in (to ask for help).
>> >
>> > What about when I'm using an encrypted key?
>> >
>> > In my case I have (in core-site.xml)
>> >
>> >
>> >
>> fs.azure.account.keyprovider.nsanalyticsstorage.blob.core.windows.ne
>> t
>> >   org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider
>> > 
>> >
>> > Everything works from the command line (hdfs dfs).
>> >
>> > But NiFi complains with:
>> >
>> > java.lang.NoSuchMethodError:
>> > org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredential
>> > Providers
>> >
>> > Any ideas? I've already linked hadoop-commons.jar as well (besides what you
>> suggested below).
>> >
>> > Cheers,
>> >
>> > Giovanni
>> >
>> >
>> >> -Original Message-
>> >> From: Bryan Bende [mailto:bbe...@gmail.com]
>> >> Sent: Tuesday, March 28, 2017 7:41 PM
>> >> To: users@nifi.apache.org
>> >> Subject: Re: GetHDFS from Azure Blob
>> >>
>> >> Austin,
>> >>
>> >> Can you provide the full 

RE: Logging changes to workflow

[HARRIOTT, John]

Hi Joe,

What is meant by the following statement: "From that one can access the list of 
changes from a given offset onward."
ReportingTask.java is an interface class. That does mean there exists a class 
specifically for reporting configuration changes?

Regards John

-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com] 
Sent: Tuesday, 11 April 2017 4:02 AM
To: users@nifi.apache.org
Subject: Re: Logging changes to workflow

Juan,

They were introduced in https://issues.apache.org/jira/browse/NIFI-986

I don't know if there is a reporting task which leverages this yet but the key 
classes to notice are here:
 ReportingTask [1]
 EventAccess [2]

From that one can access the list of changes from a given offset onward.

[1] 
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/reporting/ReportingTask.java
[2] 
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/reporting/EventAccess.java

Thanks
Joe

On Mon, Apr 10, 2017 at 2:25 PM, Juan Sequeiros  wrote:
> Hi Joe / all,
>
> " we make it available via reporting tasks "
>
> What reporting task is that? I've looked at NIFI 1 available reporting 
> tasks and I don't see anything that would let me export "flow 
> configuration history".
> Coincidentally we are having to meet similar requirement.
>
> Our solution was to get that through restAPI but if there is a cleaner 
> way though reporting tasks, would like to experiment with that too.
>
> On Mon, Apr 10, 2017 at 9:09 AM Joe Witt  wrote:
>>
>> John
>>
>> To build on Andrew's comments there is already a very significant 
>> amount of audit data made available for flow changes.  We store this 
>> in an audit database so we can drive some user experiences off it and 
>> we make it available via reporting tasks so one could report this 
>> information to an external system.  I believe what you're asking for 
>> is well supported.
>>
>> In addition to that sort of configuraiton auditing remember too we 
>> have actual data provenance which shows not just what the 
>> configuration was but the lineage/path that data actually takes.
>> These too together form an extremely powerful audit basis.
>>
>> Thanks
>> Joe
>>
>> On Mon, Apr 10, 2017 at 9:05 AM, Andrew Grande  wrote:
>> > John, the Flow Configuration History menu item already captures it. 
>> > Does it look like a fit?
>> >
>> > Andrew
>> >
>> >
>> > On Sun, Apr 9, 2017, 10:50 PM HARRIOTT, John 
>> > 
>> > wrote:
>> >>
>> >> Is it possible to log changes to a workflow so they can be 
>> >> captured by a central logging/auditing capability e.g. syslog.
>> >>
>> >> This will allow security auditors to detect changes to a systems 
>> >> configuration e.g. addition/removal/modification of a processor(s).
>> >>
>> >>
>> >>
>> >> Thanks
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> This email has been sent on behalf of one of the following 
>> >> companies within the BAE Systems Australia group of companies:
>> >>
>> >> BAE Systems Australia Limited - Australian Company Number 008 
>> >> 423
>> >> 005
>> >> BAE Systems Australia Defence Pty Limited - Australian Company 
>> >> Number
>> >> 006 870 846
>> >> BAE Systems Australia Logistics Pty Limited - Australian 
>> >> Company Number 086 228 864
>> >>
>> >> Our registered office is Evans Building, Taranaki Road, Edinburgh 
>> >> Parks, Edinburgh, South Australia, 5111. If the identity of the 
>> >> sending company is not clear from the content of this email please 
>> >> contact the sender.
>> >>
>> >> This email and any attachments may contain confidential and 
>> >> legally privileged information.  If you are not the intended 
>> >> recipient, do not copy or disclose its content, but please reply 
>> >> to this email immediately and highlight the error to the sender 
>> >> and then immediately delete the message.
This email has been sent on behalf of one of the following companies within the 
BAE Systems Australia group of companies:

BAE Systems Australia Limited - Australian Company Number 008 423 005
BAE Systems Australia Defence Pty Limited - Australian Company Number 006 
870 846
BAE Systems Australia Logistics Pty Limited - Australian Company Number 086 
228 864

Our registered office is Evans Building, Taranaki Road, Edinburgh Parks,
Edinburgh, South Australia, 5111. If the identity of the sending company is
not clear from the content of this email please contact the sender.

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy or
disclose its content, but please reply to this email immediately and highlight
the error to the sender and then immediately delete the message.