Re: nifi process running at 114% of node CPU !!?

[Mike Harding]

Hi Mike,

Thanks for creating the JIRA - I'm unable to create a JIRA as I'm away on
holiday at the minute.

Mike
On Fri, 15 Jul 2016 at 14:30, Michael Moser  wrote:

> I went ahead and created NIFI-2268 for this, since it was fresh in my
> mind.  ListenHTTP calls ProcessContext.yield() whenever it doesn't have
> work to do, so HandleHttpRequest could do the same.
>
> -- Mike
>
>
> On Thu, Jul 14, 2016 at 7:00 PM, Joe Witt  wrote:
>
>> Mike,
>>
>> If you don't mind could you file a JIRA for this.  Frankly it sounds
>> like a bug to me.  We should consider making a default scheduling
>> period of something a bit slower.  Frankly just dialing back to 100 ms
>> would be sufficient most likely.  If you agree this is a bug please
>> file one here: https://issues.apache.org/jira/browse/NIFI
>>
>> If you could attach a template of the flow that behaves badly and the
>> one that behaves better that would be ideal but if not just a good
>> description should do.
>>
>> Thanks
>> Joe
>>
>> On Thu, Jul 14, 2016 at 6:57 PM, Mike Harding 
>> wrote:
>> > Thanks all - I checked the logs and there is nothing I can see thats
>> seems
>> > erroneous. I increased the number of threads for the processor and
>> added the
>> > 10 second scheduling and it has dropped dramatically from 2.5M tasks to
>> 300
>> > over 5 minute period. CPU for the nifi java process is now running at
>> 8-10%
>> > CPU.
>> >
>> > I don't think I saw this issue when using HTTPListen processor which I
>> > recently from to HttpRequestHandle.
>> >
>> > Cheers,
>> > Mike
>> >
>> > On 14 July 2016 at 16:41, Aldrin Piri  wrote:
>> >>
>> >> Mike,
>> >>
>> >> To add some context, while NiFi will intelligently schedule processors
>> to
>> >> execute, given HandleHTTPRequest's function as a listener, it is
>> constantly
>> >> scheduled to run, checking for a request to handle.  I assume by
>> number of
>> >> tasks, you mean the rolling count over the last 5 minutes.  As
>> mentioned by
>> >> Andy, you can tamper this rate by increasing the run scheduld if the
>> >> handling of the HTTP requests with a slight latency is acceptable to
>> you and
>> >> your needs.
>> >>
>> >>
>> >> On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto 
>> >> wrote:
>> >>>
>> >>> Mike,
>> >>>
>> >>> You can adjust the processor properties for the HandleHTTPRequest
>> >>> processor in the scheduling tab.
>> >>>
>> >>> “Concurrent tasks” limits the number of threads this processor will
>> use
>> >>> (default is 1)
>> >>> “Run schedule” determines the frequency that this processor will be
>> run
>> >>> (default is ‘0 sec’ which means continuously)
>> >>>
>> >>> If you are only getting requests on a much slower schedule, you could
>> >>> reduce the run schedule to ~10 seconds and see if this is better for
>> you. I
>> >>> have not encountered NiFi running at such high CPU percentage with
>> that
>> >>> little data.
>> >>>
>> >>> As for the high number of tasks, that is definitely an anomaly.
>> >>> Configuration best practices [1] currently recommend increasing the
>> limit to
>> >>> the 10k range, but 2.5M for a single processor is unusual. Can you
>> inspect
>> >>> the logs (located in $NIFI_HOME/logs) to see if there are errors or
>> more
>> >>> insight there?
>> >>>
>> >>> [1]
>> >>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
>> >>>
>> >>>
>> >>>
>> >>> Andy LoPresto
>> >>> alopre...@apache.org
>> >>> alopresto.apa...@gmail.com
>> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> >>>
>> >>> On Jul 14, 2016, at 10:36 AM, Mike Harding 
>> >>> wrote:
>> >>>
>> >>> ps - also noticed it seems to generate a lot of tasks, currently 2.5M
>> >>> compared to other processes in the pipeline which reports 10s of
>> tasks.
>> >>>
>> >>> Mike
>> >>>
>> >>> On 14 July 2016 at 15:34, Mike Harding 
>> wrote:
>> 
>> 
>>  Hi All,
>> 
>>  The node in my cluster running nifi crashed due to a CPU overload
>> event.
>>  After restarting I analysed the CPU consumption and found that nifi
>> was the
>>  issue. As you can see below it was running at 133% CPU:
>> 
>>  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
>>  COMMAND
>> 
>>  2031 nifi  20   0 3392960 990.7m  34124 S 113.4 12.4   1179:47
>> java]
>> 
>>  I have only one data pipe line setup that is receiving data through a
>>  HandleHTTPRequest processor and after playing around and turning
>> other
>>  processors off in the pipe it was only when I stopped this process
>> that the
>>  CPU dropped significantly to around 10% CPU.
>> 
>>  Its receiving around 67KB of data every 5 minutes from multiple
>> requests
>>  from a up stream web app.
>> 
>>  Has any one else seen this behaviour and or know whether there are

Re: NiFi 0.6.1 Documentation

[David Wynne]

Jennifer,

If you download NiFi version 0.6.1, the documentation is embedded in NiFi.
Just click the help word in the upper right hand corner of NiFi

[image: Inline image 1]

The documentation will open in a new window.  I am not sure if it is still
online somewhere.



On Wed, Jul 20, 2016 at 12:12 PM, Jennifer Coston <
jennifer.cos...@raytheon.com> wrote:

> Hello,
>
> Can anyone tell me where to find the NiFi 0.6.1 documentation? Is it still
> online?
>
> Thank you!
>
>*Jennifer *
>
>

Stuck Thread Monitor?

[Sven Davison]

I have a custom processor that has a stuck thread from time to time. Once
the thread's stuck, it fails to process data. Restarting the NiFi instance
is enough to fix it... for a while.

Without changing the processor, is there a way to monitor stuck threads
w/in NiFi?

Re: export from Teradata

[Oleg Zhurakousky]

Just to add to what Mark said, perhaps you can raise a JIRA (as new Feature) to 
add an optional property to the processor config where such property would 
point to an additional class path (to support your case and many others).
We are doing it with few processors already.

Cheers
Oleg

On Jul 20, 2016, at 8:37 AM, Mark Payne 
> wrote:

Hi Dima,

That will work, however it is very dangerous to put jar files into NiFi's lib/ 
directory. We generally recommend
only putting NAR's into the lib/ directory, as any jar that is there will be 
inherited by all NARs and all classloaders.
So you may end up getting some really funny results from other NARs if any of 
the classes in those jars conflicts
with the classes in one of the NARs.

The preferred approach is to create the uber jar, as described by Anuj, but NOT 
put it into the NiFi lib/ directory.
Instead, it should be placed outside of the lib/ directory and then referenced 
by the JAR URL property.

Thanks
-Mark


On Jul 20, 2016, at 4:10 AM, Dima Fadeyev 
> wrote:

Just in case someone has the same problem. Copying both teradata jars to nifi's 
lib directory and leaving the "jar url" blank solved the problem for us. I was 
able to start exporting data from Teradata.

Best regards,
Dima

On Thu, Jul 14, 2016 at 4:34 PM, Anuj Handa 
> wrote:
Hi Dima,

You will have to create an über jar from the JDBC drivers provided by Teradata 
and copy the uberjar into the lib folder of nifi.

As Matt pointed out the instructions are in the email thread. You can refer 
them.

Anuj

> On Jul 14, 2016, at 9:35 AM, Matt Burgess 
> > wrote:
>
> Dima,
>
> There was a discussion on how to get the SQL processors working with
> Teradata a little while ago:
>
> http://mail-archives.apache.org/mod_mbox/nifi-users/201605.mbox/%3CCAEXY4srXZkb2pMGiOFGs%3DrSc_mHCFx%2BvjW32RjPhz_K1pMr%2B%2Bg%40mail.gmail.com%3E
>
> Looks like it involves making a fat JAR to include the Teradata driver
> and all its dependencies, since the DBCPControllerService asks for a
> single JAR containing the driver class
>
> Regards,
> Matt
>
>> On Thu, Jul 14, 2016 at 9:26 AM, Dima Fadeyev 
>> > wrote:
>> Hello, everyone,
>>
>> I'm new to nifi and this mailing list, I'm evaluating if we could extract
>> data (tables) from Teradata to local fs or HDFS with nifi. Is that possible?
>>
>> What I've done so far was creating an "Execute SQL" processor with the query
>> and the a database connection pooling service with the following
>> configuration:
>>
>> connection url:  jdbc:teradata://teradata.host/database=mydb
>> class name: com.teradata.jdbc.TeraDriver
>> jar url: file:///root/nifi-0.7.0/lib/terajdbc4.jar
>> ...
>>
>> I'm seeing this error in the log file:
>>
>> org.apache.nifi.processor.exception.ProcessException:
>> org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of
>> class 'com.teradata.jdbc.TeraDriver' for connect URL '
>> jdbc:teradata://teradata.host/database=mydb'
>>at
>> org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:225)
>> ~[na:na]
>> ...
>> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC
>> driver of class 'com.teradata.jdbc.TeraDriver' for connect URL '
>> jdbc:teradata://teradata.host/database=mydb'
>> ...
>> Caused by: java.sql.SQLException: No suitable driver
>>
>> It looks like nifi can't find or load the driver, even though the jar is
>> located in /root/nifi-0.7.0/lib/terajdbc4.jar
>>
>> Please, help me resolve this.
>> Thanks in advance.

Re: export from Teradata

[Mark Payne]

Hi Dima,

That will work, however it is very dangerous to put jar files into NiFi's lib/ 
directory. We generally recommend
only putting NAR's into the lib/ directory, as any jar that is there will be 
inherited by all NARs and all classloaders.
So you may end up getting some really funny results from other NARs if any of 
the classes in those jars conflicts
with the classes in one of the NARs.

The preferred approach is to create the uber jar, as described by Anuj, but NOT 
put it into the NiFi lib/ directory.
Instead, it should be placed outside of the lib/ directory and then referenced 
by the JAR URL property.

Thanks
-Mark


> On Jul 20, 2016, at 4:10 AM, Dima Fadeyev  wrote:
> 
> Just in case someone has the same problem. Copying both teradata jars to 
> nifi's lib directory and leaving the "jar url" blank solved the problem for 
> us. I was able to start exporting data from Teradata.
> 
> Best regards,
> Dima
> 
> On Thu, Jul 14, 2016 at 4:34 PM, Anuj Handa  > wrote:
> Hi Dima,
> 
> You will have to create an über jar from the JDBC drivers provided by 
> Teradata and copy the uberjar into the lib folder of nifi.
> 
> As Matt pointed out the instructions are in the email thread. You can refer 
> them.
> 
> Anuj
> 
> > On Jul 14, 2016, at 9:35 AM, Matt Burgess  > > wrote:
> >
> > Dima,
> >
> > There was a discussion on how to get the SQL processors working with
> > Teradata a little while ago:
> >
> > http://mail-archives.apache.org/mod_mbox/nifi-users/201605.mbox/%3CCAEXY4srXZkb2pMGiOFGs%3DrSc_mHCFx%2BvjW32RjPhz_K1pMr%2B%2Bg%40mail.gmail.com%3E
> >  
> > 
> >
> > Looks like it involves making a fat JAR to include the Teradata driver
> > and all its dependencies, since the DBCPControllerService asks for a
> > single JAR containing the driver class
> >
> > Regards,
> > Matt
> >
> >> On Thu, Jul 14, 2016 at 9:26 AM, Dima Fadeyev  >> > wrote:
> >> Hello, everyone,
> >>
> >> I'm new to nifi and this mailing list, I'm evaluating if we could extract
> >> data (tables) from Teradata to local fs or HDFS with nifi. Is that 
> >> possible?
> >>
> >> What I've done so far was creating an "Execute SQL" processor with the 
> >> query
> >> and the a database connection pooling service with the following
> >> configuration:
> >>
> >> connection url:  jdbc:teradata://teradata.host/database=mydb
> >> class name: com.teradata.jdbc.TeraDriver
> >> jar url: file:///root/nifi-0.7.0/lib/terajdbc4.jar
> >> ...
> >>
> >> I'm seeing this error in the log file:
> >>
> >> org.apache.nifi.processor.exception.ProcessException:
> >> org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of
> >> class 'com.teradata.jdbc.TeraDriver' for connect URL '
> >> jdbc:teradata://teradata.host/database=mydb'
> >>at
> >> org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:225)
> >> ~[na:na]
> >> ...
> >> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC
> >> driver of class 'com.teradata.jdbc.TeraDriver' for connect URL '
> >> jdbc:teradata://teradata.host/database=mydb'
> >> ...
> >> Caused by: java.sql.SQLException: No suitable driver
> >>
> >> It looks like nifi can't find or load the driver, even though the jar is
> >> located in /root/nifi-0.7.0/lib/terajdbc4.jar
> >>
> >> Please, help me resolve this.
> >> Thanks in advance.
> 

Re: export from Teradata

[Dima Fadeyev]

Just in case someone has the same problem. Copying both teradata jars to
nifi's lib directory and leaving the "jar url" blank solved the problem for
us. I was able to start exporting data from Teradata.

Best regards,
Dima

On Thu, Jul 14, 2016 at 4:34 PM, Anuj Handa  wrote:

> Hi Dima,
>
> You will have to create an über jar from the JDBC drivers provided by
> Teradata and copy the uberjar into the lib folder of nifi.
>
> As Matt pointed out the instructions are in the email thread. You can
> refer them.
>
> Anuj
>
> > On Jul 14, 2016, at 9:35 AM, Matt Burgess  wrote:
> >
> > Dima,
> >
> > There was a discussion on how to get the SQL processors working with
> > Teradata a little while ago:
> >
> >
> http://mail-archives.apache.org/mod_mbox/nifi-users/201605.mbox/%3CCAEXY4srXZkb2pMGiOFGs%3DrSc_mHCFx%2BvjW32RjPhz_K1pMr%2B%2Bg%40mail.gmail.com%3E
> >
> > Looks like it involves making a fat JAR to include the Teradata driver
> > and all its dependencies, since the DBCPControllerService asks for a
> > single JAR containing the driver class
> >
> > Regards,
> > Matt
> >
> >> On Thu, Jul 14, 2016 at 9:26 AM, Dima Fadeyev 
> wrote:
> >> Hello, everyone,
> >>
> >> I'm new to nifi and this mailing list, I'm evaluating if we could
> extract
> >> data (tables) from Teradata to local fs or HDFS with nifi. Is that
> possible?
> >>
> >> What I've done so far was creating an "Execute SQL" processor with the
> query
> >> and the a database connection pooling service with the following
> >> configuration:
> >>
> >> connection url:  jdbc:teradata://teradata.host/database=mydb
> >> class name: com.teradata.jdbc.TeraDriver
> >> jar url: file:///root/nifi-0.7.0/lib/terajdbc4.jar
> >> ...
> >>
> >> I'm seeing this error in the log file:
> >>
> >> org.apache.nifi.processor.exception.ProcessException:
> >> org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of
> >> class 'com.teradata.jdbc.TeraDriver' for connect URL '
> >> jdbc:teradata://teradata.host/database=mydb'
> >>at
> >>
> org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:225)
> >> ~[na:na]
> >> ...
> >> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
> JDBC
> >> driver of class 'com.teradata.jdbc.TeraDriver' for connect URL '
> >> jdbc:teradata://teradata.host/database=mydb'
> >> ...
> >> Caused by: java.sql.SQLException: No suitable driver
> >>
> >> It looks like nifi can't find or load the driver, even though the jar is
> >> located in /root/nifi-0.7.0/lib/terajdbc4.jar
> >>
> >> Please, help me resolve this.
> >> Thanks in advance.
>

RE: Passing the values to the processors Dynamically

[Gadiputi, Sravani]

Thanks for the reply Bryan.

I can use UpdateAttribute processor to set the attribute value like 
username=myuser but how username value will set to myuser.
If I am trying in wrong way Could you please explain with some example.
Looking forward for your valuable inputs/suggestions.

Thanks in advance!

Regards,
Sravani

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Tuesday, July 19, 2016 10:02 PM
To: users@nifi.apache.org
Subject: Re: Passing the values to the processors Dynamically

Hello,

The properties for hostname, username, and remote path, all support expression 
language.

Somewhere before FetchSFTP if you set an attribute on a flow file like username 
= myuser then in the username property of FetchSFTP you could set the value to 
${username} to reference it.

Hope that helps.

-Bryan


On Tue, Jul 19, 2016 at 3:19 AM, Gadiputi, Sravani 
> wrote:
Hi Team,

I have a requirement where I have  to move the files from Teradata to S3.
Here I am using FetchSFTP processor to fetch the file and putS3 processor to 
store the file into S3 , and I have configured FetchSFTP processor with 
properties like hostname, username,
Password.
Now I wanted to pass the Host name,username and  Remote path values dynamically.
How can we achieve this.

Please suggest me, It would be great help.

Thanks,
Sravani



This message contains information that may be privileged or confidential and is 
the property of the Capgemini Group. It is intended only for the person to whom 
it is addressed. If you are not the intended recipient, you are not authorized 
to read, print, retain, copy, disseminate, distribute, or use this message or 
any part thereof. If you receive this message in error, please notify the 
sender immediately and delete all copies of this message.