Re: Deadlock bug in PersistentProvenanceRepository

[Brian Davis]

I will get you all of the stuff you asked for tomorrow morning.

Thank you for making the code easy to follow and understand.  Plus, taking
your time on this bug that is affecting me.

You have a good one.

Brian



On Tue, Jul 26, 2016 at 9:50 PM, Joe Witt  wrote:

> Thanks Brian.  Will discuss on the JIRA.  If you could possibly share
> the stack dumps for anything involving the prov repo that would be
> really helpful.
>
> Thanks for reporting and also providing really helpful context/analysis.
>
> Joe
>
> On Tue, Jul 26, 2016 at 9:23 PM, Brian Davis
>  wrote:
> > I have been running nifi for about a week on a machine that has 32 cpus.
> > During this week I have seen the nifi instance lockup about 5 times (it
> > could of been more if it was not a weekend).  Lockup being no flowfiles
> > moving.  All the threads are in the PersistentProvenanceRepository.  I
> have
> > submitted a bug here:
> >
> > https://issues.apache.org/jira/browse/NIFI-2395
> >
> > I think the fix is as easy as changing a || to a &&.  But there are a
> lot of
> > parts to this class and I could easily be missing something.  I tried to
> > document where all my threads were.  I believe this bug would hit any
> system
> > that runs nifi that has low to unused cpus.  I know it effects 0.6.0 and
> it
> > looks like it is in 0.7.0.  It is a very crippling bug.  I do not think I
> > have lost any data, but cannot be sure.
> >
> > If you have seen flows that just stop processing data, this could be the
> > cause.
> >
> > You have a good one.
> >
> > Brian
>

Re: Deadlock bug in PersistentProvenanceRepository

[Joe Witt]

Thanks Brian.  Will discuss on the JIRA.  If you could possibly share
the stack dumps for anything involving the prov repo that would be
really helpful.

Thanks for reporting and also providing really helpful context/analysis.

Joe

On Tue, Jul 26, 2016 at 9:23 PM, Brian Davis
 wrote:
> I have been running nifi for about a week on a machine that has 32 cpus.
> During this week I have seen the nifi instance lockup about 5 times (it
> could of been more if it was not a weekend).  Lockup being no flowfiles
> moving.  All the threads are in the PersistentProvenanceRepository.  I have
> submitted a bug here:
>
> https://issues.apache.org/jira/browse/NIFI-2395
>
> I think the fix is as easy as changing a || to a &&.  But there are a lot of
> parts to this class and I could easily be missing something.  I tried to
> document where all my threads were.  I believe this bug would hit any system
> that runs nifi that has low to unused cpus.  I know it effects 0.6.0 and it
> looks like it is in 0.7.0.  It is a very crippling bug.  I do not think I
> have lost any data, but cannot be sure.
>
> If you have seen flows that just stop processing data, this could be the
> cause.
>
> You have a good one.
>
> Brian

Deadlock bug in PersistentProvenanceRepository

[Brian Davis]

I have been running nifi for about a week on a machine that has 32 cpus.
During this week I have seen the nifi instance lockup about 5 times (it
could of been more if it was not a weekend).  Lockup being no flowfiles
moving.  All the threads are in the PersistentProvenanceRepository.  I have
submitted a bug here:

https://issues.apache.org/jira/browse/NIFI-2395

I think the fix is as easy as changing a || to a &&.  But there are a lot
of parts to this class and I could easily be missing something.  I tried to
document where all my threads were.  I believe this bug would hit any
system that runs nifi that has low to unused cpus.  I know it effects 0.6.0
and it looks like it is in 0.7.0.  It is a very crippling bug.  I do not
think I have lost any data, but cannot be sure.

If you have seen flows that just stop processing data, this could be the
cause.

You have a good one.

Brian

RE: JsonSplit Question/Help

[Sven Davison]

WOOT! It took a minute but.. I got it! Thanks for the help!

http://prntscr.com/bxx5jc



Sent from Mail for Windows 10

From: Matt Burgess

RE: JsonSplit Question/Help

[Sven Davison]

Does it just become $twitter.hashtags without the index locators then? I’m not 
sure how to reference the content of the flowfile after it’s been split off. It 
LOOKS like it’s still the original full list vice element of the array.



Sent from Mail for Windows 10

From: Matt Burgess

JsonSplit Question/Help

[Sven Davison]

I’m trying to store each of the hash tags from a twitter post. They can be 0 or 
more and I’ve found the array via JSON but I’m not sure how to reference them 
other than via

$twitter.hashtags[0].text that’s not very dynamic… as I never know how many 
there will be.

Here’s a screenshot showing that they are extracted via EvaluateJsonPath… but 
I’m not sure how to split the array for use now.

http://prntscr.com/bxvse3


the plan is to do something with each tag, so they probably need to become 
their own flow file after being split up.. still learning!



Sent from Mail for Windows 10

Re: JsonSplit Question/Help

[Matt Burgess]

Sven,

You can use the SplitJson processor with a JSONPath value of
$.twitter.hashtags, it will create a new flowfile for each hashtag.
Then you can use EvaluateJsonPath to get the text value from each of
the flow files.

Regards,
Matt

On Tue, Jul 26, 2016 at 5:09 PM, Sven Davison  wrote:
> I’m trying to store each of the hash tags from a twitter post. They can be 0
> or more and I’ve found the array via JSON but I’m not sure how to reference
> them other than via
>
>
>
> $twitter.hashtags[0].text that’s not very dynamic… as I never know how many
> there will be.
>
>
>
> Here’s a screenshot showing that they are extracted via EvaluateJsonPath…
> but I’m not sure how to split the array for use now.
>
>
>
> http://prntscr.com/bxvse3
>
>
>
>
>
> the plan is to do something with each tag, so they probably need to become
> their own flow file after being split up.. still learning!
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>

Re: authentication problem

[Matt Gilman]

Ralph,

I'm guessing that every user is not disabled since you said that you are
manually adding them back in through the UI. Is there anything in common
with the user's that are being revoked?

To follow up on Clarke's comment... As your updating the users through the
UI, the authorized-users.xml file should be updated to reflect those
changes. Are you seeing any errors logged there? Can you confirm that the
authorized-users.xml is accurate at that point?

Matt

On Tue, Jul 26, 2016 at 1:31 PM, Perko, Ralph J 
wrote:

> Thanks for the responses.
>
> Matt Clarke: The permissions look fine.  Nifi runs as user “nifi” and all
> files are owned by “nifi” (with write permissions of course)
>
> Matt Gilman:
> As far as additional logging here is what I found – No errors of any sort:
>
> 016-07-25 16:10:59,420 INFO [main]
> o.a.nifi.admin.UserDataSourceFactoryBean Existing database found and
> connected to at:
> jdbc:h2:./database_repository/nifi-users;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
> 2016-07-25 16:10:59,473 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User account already created:
> . Updating authorities...
> 2016-07-25 16:10:59,485 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User account already
> created: . Updating authorities...
> 2016-07-25 16:10:59,487 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User account already
> created: . Updating authorities...
> 2016-07-25 16:10:59,492 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User account already
> created: . Updating authorities...
> 2016-07-25 16:10:59,494 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User account already
> created:. Updating authorities…
> ..
> 2016-07-25 16:10:59,508 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User not authorized with configured
> provider: . Disabling account...
> 2016-07-25 16:10:59,509 INFO [main]
> o.a.n.a.s.action.SeedUserAccountsAction User not authorized with configured
> provider: . Disabling account...
> …
>
> From: Matthew Clarke 
> Reply-To: "users@nifi.apache.org" 
> Date: Tuesday, July 26, 2016 at 10:03 AM
> To: "users@nifi.apache.org" 
> Subject: Re: authentication problem
>
> Verify the user running your nifi has the correct permissions to read and
> edit all the database files in the NiFi database repository directory. Also
> make sure the user running NiFi had permissions to edit the
> authorized-users.xml file. This file is only read on start-up. After nifi
> is running it operates solely from the memory and DBs. My guess is here
> that nifi can not update the authorized-users.xml file with the changes you
> are making in the User management UI.
>
> Thanks,
> Matt
>
> On Jul 26, 2016 12:47 PM, "Perko, Ralph J"  wrote:
>
> Hi – whenever we restart Nifi half the accounts are disabled with this
> message:
>
> INFO [main] o.a.n.a.s.action.SeedUserAccountsAction User not authorized
> with configured provider: . Disabling account...
>
> The users are in the authorized-users.xml file.  Is this a configuration
> issue on our part or a bug?  It is curious that not all the accounts get
> disabled only some and it is always the same accounts.
>
> To re-enable the accounts I go into the users page, select the disabled
> user (click the little pencil) and click ‘apply’ with no changes - the
> account is re-enabled.
>
> Details:
> Nifi 0.6.1
> authority-providers.xml: default file provider
> login-identy-management.xml: kerberos-provider (corporate system –
> everyone is in it)
> authorized-users.xml:setup for each user
>
> Thanks,
> Ralph
>
>
>

Re: authentication problem

[Perko, Ralph J]

Thanks for the responses.

Matt Clarke: The permissions look fine.  Nifi runs as user “nifi” and all files 
are owned by “nifi” (with write permissions of course)

Matt Gilman:
As far as additional logging here is what I found – No errors of any sort:

016-07-25 16:10:59,420 INFO [main] o.a.nifi.admin.UserDataSourceFactoryBean 
Existing database found and connected to at: 
jdbc:h2:./database_repository/nifi-users;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
2016-07-25 16:10:59,473 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User account already created: . Updating authorities...
2016-07-25 16:10:59,485 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User account already created: . Updating authorities...
2016-07-25 16:10:59,487 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User account already created: . Updating authorities...
2016-07-25 16:10:59,492 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User account already created: . Updating authorities...
2016-07-25 16:10:59,494 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User account already created:. Updating authorities…
..
2016-07-25 16:10:59,508 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User not authorized with configured provider: . Disabling 
account...
2016-07-25 16:10:59,509 INFO [main] o.a.n.a.s.action.SeedUserAccountsAction 
User not authorized with configured provider: . Disabling 
account...
…

From: Matthew Clarke 
mailto:matt.clarke@gmail.com>>
Reply-To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Tuesday, July 26, 2016 at 10:03 AM
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: Re: authentication problem


Verify the user running your nifi has the correct permissions to read and edit 
all the database files in the NiFi database repository directory. Also make 
sure the user running NiFi had permissions to edit the authorized-users.xml 
file. This file is only read on start-up. After nifi is running it operates 
solely from the memory and DBs. My guess is here that nifi can not update the 
authorized-users.xml file with the changes you are making in the User 
management UI.

Thanks,
Matt

On Jul 26, 2016 12:47 PM, "Perko, Ralph J" 
mailto:ralph.pe...@pnnl.gov>> wrote:
Hi – whenever we restart Nifi half the accounts are disabled with this message:

INFO [main] o.a.n.a.s.action.SeedUserAccountsAction User not authorized with 
configured provider: . Disabling account...

The users are in the authorized-users.xml file.  Is this a configuration issue 
on our part or a bug?  It is curious that not all the accounts get disabled 
only some and it is always the same accounts.

To re-enable the accounts I go into the users page, select the disabled user 
(click the little pencil) and click ‘apply’ with no changes - the account is 
re-enabled.

Details:
Nifi 0.6.1
authority-providers.xml: default file provider
login-identy-management.xml: kerberos-provider (corporate system – everyone is 
in it)
authorized-users.xml:setup for each user

Thanks,
Ralph

Re: authentication problem

[Matthew Clarke]

Verify the user running your nifi has the correct permissions to read and
edit all the database files in the NiFi database repository directory. Also
make sure the user running NiFi had permissions to edit the
authorized-users.xml file. This file is only read on start-up. After nifi
is running it operates solely from the memory and DBs. My guess is here
that nifi can not update the authorized-users.xml file with the changes you
are making in the User management UI.

Thanks,
Matt

On Jul 26, 2016 12:47 PM, "Perko, Ralph J"  wrote:

Hi – whenever we restart Nifi half the accounts are disabled with this
message:

INFO [main] o.a.n.a.s.action.SeedUserAccountsAction User not authorized
with configured provider: . Disabling account...

The users are in the authorized-users.xml file.  Is this a configuration
issue on our part or a bug?  It is curious that not all the accounts get
disabled only some and it is always the same accounts.

To re-enable the accounts I go into the users page, select the disabled
user (click the little pencil) and click ‘apply’ with no changes - the
account is re-enabled.

Details:
Nifi 0.6.1
authority-providers.xml: default file provider
login-identy-management.xml: kerberos-provider (corporate system – everyone
is in it)
authorized-users.xml:setup for each user

Thanks,
Ralph

Re: authentication problem

[Matt Gilman]

Ralph,

Based on the details in your email, it sounds like it should be working. At
start up, it will go through the users database to re-verify each user with
the configured provider. The authorizations for a user are cache in this
database for a configurable time. The re-verification is done to ensure the
cached authorizations were correct in the event that an admin changed the
configured authority provider.

Looking at the code there should be some additional logging about the issue
that occurred while attempting to verify the user in question. Are you
seeing anything else in the logs?

Matt

On Tue, Jul 26, 2016 at 12:47 PM, Perko, Ralph J 
wrote:

> Hi – whenever we restart Nifi half the accounts are disabled with this
> message:
>
> INFO [main] o.a.n.a.s.action.SeedUserAccountsAction User not authorized
> with configured provider: . Disabling account...
>
> The users are in the authorized-users.xml file.  Is this a configuration
> issue on our part or a bug?  It is curious that not all the accounts get
> disabled only some and it is always the same accounts.
>
> To re-enable the accounts I go into the users page, select the disabled
> user (click the little pencil) and click ‘apply’ with no changes - the
> account is re-enabled.
>
> Details:
> Nifi 0.6.1
> authority-providers.xml: default file provider
> login-identy-management.xml: kerberos-provider (corporate system –
> everyone is in it)
> authorized-users.xml:setup for each user
>
> Thanks,
> Ralph
>
>

authentication problem

[Perko, Ralph J]

Hi – whenever we restart Nifi half the accounts are disabled with this message:

INFO [main] o.a.n.a.s.action.SeedUserAccountsAction User not authorized with 
configured provider: . Disabling account...

The users are in the authorized-users.xml file.  Is this a configuration issue 
on our part or a bug?  It is curious that not all the accounts get disabled 
only some and it is always the same accounts.

To re-enable the accounts I go into the users page, select the disabled user 
(click the little pencil) and click ‘apply’ with no changes - the account is 
re-enabled.

Details:
Nifi 0.6.1
authority-providers.xml: default file provider
login-identy-management.xml: kerberos-provider (corporate system – everyone is 
in it)
authorized-users.xml:setup for each user

Thanks,
Ralph

Re: Json routing

[Anuj Handa]

never mind.. i got that working .. i just wanted to understand the flow in
the invokescripted processor

first i am reading the input stream to JSON slurper
identify whether the element name is ORDER or REFUND
based on the element name transfer the flow file to appropriate
relationship.. i am sketchy about this step since in the examples you have
you only have one relationship.. do you have an example where you have
multiple realtionship defined.. i might have missed it on the blog.

Anuj

On Tue, Jul 26, 2016 at 11:28 AM, Anuj Handa  wrote:

> Hi Matt,
>
> To use JSon slurper into the invokeScripted processor do i have to import
> any classes, i didnt see any clases in your examples and when i try to use
> new JsonSlurper it give me an error unable to resolve class.
>
> Anuj
>
> On Thu, Jul 7, 2016 at 11:45 AM, Matt Burgess  wrote:
>
>> It will be fixed in 0.7.0 [1].  Also you could use
>> InvokeScriptedProcessor to replace both the ExecuteScript and
>> RouteOnAttribute, since the scripted processor can define the
>> relationships and provide the logic to extract the arbitrary JSON
>> keys.
>>
>> Regards,
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1822
>>
>> On Thu, Jul 7, 2016 at 11:21 AM, Bryan Bende  wrote:
>> > I think you actually need to use both ExecuteScript and RouteOnAttribute
>> > together.
>> >
>> > The reason is that ExecuteScript only has success and failure
>> relationships,
>> > so all you can do there is extract the value you are interested in
>> (order or
>> > refund) and put it in an attribute and route to success,
>> > then send it to RouteOnAttribute which can have the dynamic
>> relationships.
>> >
>> > There was a limitation where ExecuteScript could not run concurrent
>> tasks,
>> > and I know it was fixed, but not sure if it was fixed in 0.6.1 or for
>> the
>> > next upcoming release (0.7.0) which is happening soon.
>> > Maybe someone else can chime about when this was fixed.
>> >
>> > On Thu, Jul 7, 2016 at 11:13 AM, Anuj Handa 
>> wrote:
>> >>
>> >> Hi Bryan,
>> >>
>> >> Thats correct i want to route on element name so "ORDER" through one
>> and
>> >> "REFUND" to another. The reason is based on element  name the
>> structure of
>> >> the JSON could be different so the subsequent processing depends on the
>> >> element name.
>> >>
>> >> Processing thousands of messages per minute which would be better using
>> >> RouteonAttribute or ExecuteScript ? Since the entire JSON doc could
>> come in
>> >> the attribute if i use  $.POSTransaction. i would think it would slow
>> it
>> >> down compared with if the attribute value only has "ORDER"
>> >>
>> >> Anuj
>> >>
>> >>
>> >>
>> >> On Thu, Jul 7, 2016 at 10:59 AM, Bryan Bende  wrote:
>> >>>
>> >>> Anuj,
>> >>>
>> >>> Just to clarify, you want to route on the name of the element under
>> >>> POSTransaction? Meaning, route "Order" to one place and "Refund" to
>> another?
>> >>>
>> >>> I'm not a JSON Path expert, but I can't come up with a way to get
>> just
>> >>> an element name from JSON path, it is usually used to get the value
>> of a
>> >>> known path.
>> >>>
>> >>> If you used $.POSTransaction. as the expression I think you would get
>> >>> back everything under POSTransaction  including the "Order" or
>> "Refund"
>> >>> part, and
>> >>> then in RouteOnAttribute you could use expression language to see
>> what it
>> >>> starts ${yourAttribute:startsWith('Order')} or maybe use contains()
>> instead
>> >>> of startsWith.
>> >>>
>> >>> Another completely different option is to use the ExecuteScript
>> processor
>> >>> to write a Groovy/Jython/etc script that gets the name of the first
>> element
>> >>> under POSTransaction and adds it as an attribute.
>> >>>
>> >>> -Bryan
>> >>>
>> >>>
>> >>> On Thu, Jul 7, 2016 at 8:34 AM, Anuj Handa 
>> wrote:
>> 
>>  Hi Folks,
>> 
>>  I have following two JSON documents and i would like to route them
>> based
>>  on what the value is. In the below examples its order and refund. i
>> want
>>  this to be dynamic as i can expect range of values.
>> 
>>  i was thinking of using EvaluateJsonPath and reading the Value of
>> this
>>  field in the attribute. i was unable to get what Path expression
>> should be .
>> 
>>  $.POSTransaction.* returns me the entire JSON and not just the Order
>>  value
>> 
>>  is it possible to make it dynamic ?  or is there a better/different
>> way
>>  to do this
>> 
>>  {
>>  "POSTransaction": {
>>  "Order": {
>> 
>>  {
>>  "POSTransaction": {
>>  "Refund": {
>> 
>> 
>>  Regards,
>>  Anuj
>> >>>
>> >>>
>> >>
>> >
>>
>
>

Re: Mantain Connection Pool Live

[Toivo Adams]

Hi Carlos,

Yes, we can add only one property – validationQuery.
And when validationQuery is provided this means connections should be
validated.
This in turn should give us dropping invalid connections and create new
ones.

Thanks
Toivo

2016-07-25 16:37 GMT+03:00 Carlos Manuel Fernandes (DSI) <
carlos.antonio.fernan...@cgd.pt>:

> HI Toivos,
>
>
>
> if I request a connection and there isn’t a free one in the pool , a new
> one is created  if the maximum  wasn’t reached, it’s the normal function,
> right?
>
> If all the connections (max)  are created on startup of the service we
> need to create new connections if some become invalid, but this isn’t the
> case, right?
>
>
>
> If the former is correct, Probably its simpler  to have just the property
> for validation query and drop the invalid connections.
>
>
>
> Thanks
>
> Carlos
>
>
>
>
>
>
>
>
>
>
>
> *From:* Toivo Adams [mailto:toivo.ad...@gmail.com]
> *Sent:* domingo, 24 de Julho de 2016 17:34
>
> *To:* users@nifi.apache.org
> *Subject:* Re: Mantain Connection Pool Live
>
>
>
> Hi Carlos,
>
> You are right, as far I know there isn't single validation query for all
> databases.
>
> It seems we must add 2 Property's to DBCPConnectionPool service which user
> must fill.
>
> 1. Do you want drop invalid connections and create new ones automatically
> (which might affect performance little bit).
>
> 2. Validation query.
>
>
> Thanks
> Toivo
>
>
>
> 2016-07-24 19:01 GMT+03:00 Carlos Manuel Fernandes (DSI) <
> carlos.antonio.fernan...@cgd.pt>:
>
> Hi Toivo, Thanks for you reply.
>
>
>
> I opened  a Jira  ticket NIFI-2381 (
> https://issues.apache.org/jira/browse/NIFI-2381)
>
>
>
> Its good to Know Apache commons DBC P already has this capability,
> unfortunately as I know there isn’t a universal simple validation query,
> its database specific. I obtain this list in stackoverflow (
> http://stackoverflow.com/questions/10684244/dbcp-validationquery-for-different-databases
> ):
>
>
>
> Oracle - select 1 from dual
>
> DB2 - select 1 from sysibm.sysdummy1
>
> mysql - select 1
>
> microsoft SQL Server - select 1 (tested on SQL-Server 9.0, 10.5 [2008])
>
> postgresql - select 1
>
> Netezza – select 1
>
> Informix - select 1 from systables
>
> ingres - select 1
>
> derby - values 1
>
> H2 - select 1
>
> Firebird - select 1 from rdb$database
>
> hsqldb - select 1 from INFORMATION_SCHEMA.SYSTEM_USERS
>
>
>
> I had test myself Oracle, Db2, microsoft SQL Server , Netezza, the others
> probably are correct too.
>
>
>
> Thanks
>
>
>
> Carlos Fernandes
>
>
>
>
>
>
>
> *From:* Toivo Adams [mailto:toivo.ad...@gmail.com]
> *Sent:* sábado, 23 de Julho de 2016 13:20
> *To:* users@nifi.apache.org
> *Subject:* Re: Mantain Connection Pool Live
>
>
>
> Hi Carlos,
>
> I think we should add capability dropping invalid connections and create
> new ones to DBCPConnectionPool.
>
> DBCPConnectionPool uses internally Apache commons DBCP and DBCP itself
> supports dropping invalid connections.
>
> See testOnBorrow
>
> http://commons.apache.org/proper/commons-dbcp/api-1.4/src-html/org/apache/commons/dbcp/BasicDataSource.html#line.506
>
> Also validationQuery is need to be set up.
>
> Please can you create Jira ticket.
>
> Thanks
> Toivo
>
>
>
> 2016-07-21 21:22 GMT+03:00 Carlos Manuel Fernandes (DSI) <
> carlos.antonio.fernan...@cgd.pt>:
>
> Hi all,
>
>
>
> I have a  DBCPConnectionPool to  Netezza with 8 Connections, but if  some
> connections go idle too much time,  Netezza will throw :
>
> “org.netezza.error.NzSQLException: FATAL 1:  Connection Terminated -
> session timeout exceeded” when I attempt to retrieve one of them. After
> this that   number of connections available decrease. I must restart the
> service to maintain all the connections.
>
>
>
> The Connection Pooling Service don’t handle to create a new connection?
>
>
>
> I  can resolve the problem creating a ExecuteScript processor to maintain
> the connections alive  but I think a new property (keepAlive) on Pooling
> service was a better solution.
>
>
>
> Carlos Fernandes
>
>
>
>
>
> Full  Exception:
>
>
>
> 2016-07-21 15:00:10,258 ERROR [Timer-Driven Process Thread-7]
> o.a.nifi.processors.script.ExecuteScript
>
> org.netezza.error.NzSQLException: FATAL 1:  Connection Terminated -
> session timeout exceeded
>
>
>
> at
> org.netezza.internal.QueryExecutor.getNextResult(QueryExecutor.java:280)
> ~[na:na]
>
> at
> org.netezza.internal.QueryExecutor.execute(QueryExecutor.java:76) ~[na:na]
>
> at org.netezza.sql.NzConnection.execute(NzConnection.java:2750)
> ~[na:na]
>
> at org.netezza.sql.NzStatement._execute(NzStatement.java:849)
> ~[na:na]
>
> at org.netezza.sql.NzStatement.execute(NzStatement.java:306)
> ~[na:na]
>
>at
> org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
> ~[na:na]
>
> at
> org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
> ~[na:na]
>
> at groovy.sql.Sql.execute(Sql.java:2272)
> ~[gr

Re: 0.6.1 Master running out of heap space

[Corey Flowers]

This is probably because of your stats. All the stats across your cluster are 
kept within the NCMs heap space. Within Apache nifi it stores 1440 stats per 
processor, per minute. If you look in the conf file you will see the settings 
toward the bottom of the file. Sorry, I don't have a running instance in front 
of me to get the exact lines. You still may have issues with heap at that size 
but if you change the settings to 280 every 5 minutes it should help. 

Good luck!

Sent from my iPhone

> On Jul 26, 2016, at 4:06 AM, "michael.griffit...@baesystems.com" 
>  wrote:
> 
> Hi all,
>  
> I have a cluster with a relatively large amount of processors (150ish) in a 
> flow. I’m finding that the master frequently runs out of heap space (set to a 
> maximum of 1GB) after a few hours of being online.
>  
> Is this a known issue, or is the 1GB of ‘allocated’ RAM not enough to hold 
> that many processors? As the master doesn’t do any processing, I’m unsure why 
> I’m seeing the master run out of heap space and not the slaves themselves.
>  
> “2016-07-26 09:05:08,115 WARN [Process NCM Request-17] 
> org.apache.nifi.io.socket.SocketListener Dispatching socket request 
> encountered exception due to: java.lang.OutOfMemoryError: Java heap space”
>  
> From my bootstrap file:
>  
> # JVM memory settings
> java.arg.2=-Xms1024m
> java.arg.3=-Xmx1024m
>  
> Many thanks,
>  
> Michael
>  
> Michael Griffiths
> System Developer
> BAE Systems Applied Intelligence
> ___
> 
>  E: michael.griffit...@baesystems.com
> 
> BAE Systems Applied Intelligence, Surrey Research Park, Guildford, Surrey, 
> GU2 7RQ. 
> www.baesystems.com/ai
> 
>  
> Please consider the environment before printing this email. This message 
> should be regarded as confidential. If you have received this email in error 
> please notify the sender and destroy it immediately. Statements of intent 
> shall only become binding when confirmed in hard copy by an authorised 
> signatory. The contents of this email may relate to dealings with other 
> companies under the control of BAE Systems Applied Intelligence Limited, 
> details of which can be found at 
> http://www.baesystems.com/Businesses/index.htm.

Re: Syslog to avro format

[Madhukar Thota]

HI Conrad

Thanks for your response. I am not opposed to convert into JSON. Just
checking to see if there is a way to get direct Avro conversion from
attributes.

-Madhu

On Tue, Jul 26, 2016 at 4:55 AM, Conrad Crampton <
conrad.cramp...@secdata.com> wrote:

> Why not converting to JSON?
>
> I do exactly this, parse the syslog (into attributes), convert attributes
> to JSON, JSON->Avro.
>
> I had to have an intermediate Avro schema that was only strings due to a
> problem converting JSON integers into equivalent Avro, then convert Avro
> schema to final one (that included ints).
>
> HTH,
>
> Conrad
>
>
>
> *From: *Madhukar Thota 
> *Reply-To: *"users@nifi.apache.org" 
> *Date: *Tuesday, 26 July 2016 at 09:52
> *To: *"users@nifi.apache.org" 
> *Subject: *Syslog to avro format
>
>
>
> Friends,
>
>
>
> What is the best way to get Syslog data into avro format without
> converting to JSON?
>
>
>
> Any suggestions?
>
>
>
>
>
> ***This email originated outside SecureData***
>
> Click here  to
> report this email as spam.
>
>
> SecureData, combating cyber threats
>
> --
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email
> are those of the individual and not necessarily of SecureData Europe Ltd.
> Any prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
>

Re: Syslog to avro format

[Conrad Crampton]

Why not converting to JSON?
I do exactly this, parse the syslog (into attributes), convert attributes to 
JSON, JSON->Avro.
I had to have an intermediate Avro schema that was only strings due to a 
problem converting JSON integers into equivalent Avro, then convert Avro schema 
to final one (that included ints).
HTH,
Conrad

From: Madhukar Thota 
Reply-To: "users@nifi.apache.org" 
Date: Tuesday, 26 July 2016 at 09:52
To: "users@nifi.apache.org" 
Subject: Syslog to avro format

Friends,

What is the best way to get Syslog data into avro format without converting to 
JSON?

Any suggestions?



***This email originated outside SecureData***

Click here to report 
this email as spam.


SecureData, combating cyber threats
__ 
The information contained in this message or any of its attachments may be 
privileged and confidential and intended for the exclusive use of the intended 
recipient. If you are not the intended recipient any disclosure, reproduction, 
distribution or other dissemination or use of this communications is strictly 
prohibited. The views expressed in this email are those of the individual and 
not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if 
followed up by a formal written quote.

SecureData Europe Limited. Registered in England & Wales 04365896. Registered 
Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, 
ME16 9NT

Syslog to avro format

[Madhukar Thota]

Friends,

What is the best way to get Syslog data into avro format without converting
to JSON?

Any suggestions?

0.6.1 Master running out of heap space

[michael.griffit...@baesystems.com]

Hi all,

I have a cluster with a relatively large amount of processors (150ish) in a 
flow. I'm finding that the master frequently runs out of heap space (set to a 
maximum of 1GB) after a few hours of being online.

Is this a known issue, or is the 1GB of 'allocated' RAM not enough to hold that 
many processors? As the master doesn't do any processing, I'm unsure why I'm 
seeing the master run out of heap space and not the slaves themselves.

"2016-07-26 09:05:08,115 WARN [Process NCM Request-17] 
org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered 
exception due to: java.lang.OutOfMemoryError: Java heap space"

>From my bootstrap file:

# JVM memory settings
java.arg.2=-Xms1024m
java.arg.3=-Xmx1024m

Many thanks,

Michael

Michael Griffiths
System Developer
BAE Systems Applied Intelligence
___

 E: michael.griffit...@baesystems.com

BAE Systems Applied Intelligence, Surrey Research Park, Guildford, Surrey, GU2 
7RQ.
www.baesystems.com/ai

Please consider the environment before printing this email. This message should 
be regarded as confidential. If you have received this email in error please 
notify the sender and destroy it immediately. Statements of intent shall only 
become binding when confirmed in hard copy by an authorised signatory. The 
contents of this email may relate to dealings with other companies under the 
control of BAE Systems Applied Intelligence Limited, details of which can be 
found at http://www.baesystems.com/Businesses/index.htm.