NiFi WebGUI Timestamp issue

[Josef.Zahner1]

Hi Guys,

We just faced an issue with the time shown in the NiFi webgui (Screenshot 
below).

We are in Switzerland, so at the moment we have “CET” (UTC + 1h). After a 
restart of our 8-node NiFi 1.9.2 cluster, NiFi suddenly showed “CET” - but the 
timestamp was in fact UTC, so minus 1h. Then we decided to restart all NiFi 
nodes again and now it’s even more confusing, it shows “UTC” but the time is 
from CET?

[cid:image001.png@01D58F10.B9FC5C30]


Can anybody explain where the timestamp and the localization letters are coming 
from? After the reboot for a very short time period it showed another 
localization and timestamp, but then it changed to what I mentioned in the 
screenshot above.

We have more clusters and single nifi’s and all the other machines show the 
correct timestamp and location… all have the same config.

Cheers Josef


smime.p7s
Description: S/MIME Cryptographic Signature

Re: NiFi WebGUI Timestamp issue

[Wesley C. Dias de Oliveira]

Hi, Josef.

It seems to be related to machine time settings.

Have you checked this?

Em qua, 30 de out de 2019 às 06:57,  escreveu:

> Hi Guys,
>
>
>
> We just faced an issue with the time shown in the NiFi webgui (Screenshot
> below).
>
>
>
> We are in Switzerland, so at the moment we have “CET” (UTC + 1h). After a
> restart of our 8-node NiFi 1.9.2 cluster, NiFi suddenly showed “CET” - but
> the timestamp was in fact UTC, so minus 1h. Then we decided to restart all
> NiFi nodes again and now it’s even more confusing, it shows “UTC” but the
> time is from CET?
>
>
>
>
>
>
>
> Can anybody explain where the timestamp and the localization letters are
> coming from? After the reboot for a very short time period it showed
> another localization and timestamp, but then it changed to what I mentioned
> in the screenshot above.
>
>
>
> We have more clusters and single nifi’s and all the other machines show
> the correct timestamp and location… all have the same config.
>
>
>
> Cheers Josef
>


-- 
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.

Re: NiFi WebGUI Timestamp issue

[Josef.Zahner1]

Hi Wesley

I didn’t change anything on the machine during the time when I restarted the 
cluster (multiple times), however NiFi showed after each restart different 
timestamps/localizations.

Cheers Josef

From: "Wesley C. Dias de Oliveira" 
Reply to: "users@nifi.apache.org" 
Date: Wednesday, 30 October 2019 at 14:41
To: "users@nifi.apache.org" 
Subject: Re: NiFi WebGUI Timestamp issue

Hi, Josef.

It seems to be related to machine time settings.

Have you checked this?

Em qua, 30 de out de 2019 às 06:57, 
mailto:josef.zahn...@swisscom.com>> escreveu:
Hi Guys,

We just faced an issue with the time shown in the NiFi webgui (Screenshot 
below).

We are in Switzerland, so at the moment we have “CET” (UTC + 1h). After a 
restart of our 8-node NiFi 1.9.2 cluster, NiFi suddenly showed “CET” - but the 
timestamp was in fact UTC, so minus 1h. Then we decided to restart all NiFi 
nodes again and now it’s even more confusing, it shows “UTC” but the time is 
from CET?

[cid:16e1ce4a8a64cff311]


Can anybody explain where the timestamp and the localization letters are coming 
from? After the reboot for a very short time period it showed another 
localization and timestamp, but then it changed to what I mentioned in the 
screenshot above.

We have more clusters and single nifi’s and all the other machines show the 
correct timestamp and location… all have the same config.

Cheers Josef


--
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.


smime.p7s
Description: S/MIME Cryptographic Signature

RE: How to deploy NiFi processors change to multiple NiFi instances?

[Woodcock, Michael W., M.B.A.]

The approach we use for disaster recovery might be a start.

  *   nifi (user) has a cron job that runs BASH script on PROD server node  
which copies the flow.xml.gz file to be used on another cluster to a temp 
folder.
 *   The script decrypts the current passwords/properties in flow.xml.gz 
and encrypts with new BACKUP's password which re-encrypts sensitive properties.
*   
/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/encrypt-config.sh 
-b /usr/hdf/current/nifi/conf/bootstrap.conf -f 
/apps/var/lib/nifi/conf/flow.xml.gz -g ~/flow.xml.gz -n 
/usr/hdf/current/nifi/conf/nifi.properties -o 
/usr/hdf/current/nifi/conf/backup/nifi.properties -s $#$#$#$#$#$#$#$ -x
 *   Script also unzips flow.xml.gz, makes modifications by calling python 
script that changes a few labels and parameters using xml.etree.
  *   The flow.xml.gz is zipped and copied to /tmp with chmod changes
  *   Another cron copies the flow.xml.gz to the BACKUP servers offsite.
  *   Cron on BACKUP servers runs script that stops NiFi, copies new 
flow.xml.gz overwriting existing one.  Restarts NiFi with all processors 
stopped. Otherwise system starts with flows in last known state in PROD.
Caveats are:
NiFi versions and configurations the same
Users should be the same
Should point to same NiFi Registry, if used
If using Ranger, run a script exporting configuration, renaming it and sending 
it to BACKUP's Ranger
User authorization and authentication mechanisms the same.
Note that the password for encrypting flow for the BACKUP server is exposed in 
script ($#$#$#$#$#$#$#$).
I don't  remember what else...
Mike

From: Andy LoPresto [mailto:alopre...@apache.org]
Sent: Monday, October 28, 2019 12:09 PM
To: users@nifi.apache.org
Subject: Re: How to deploy NiFi processors change to multiple NiFi instances?

I would generally want a human intervention step in this process, but if you 
want this to be fully-automated, you can script that using the Event Hooks [1] 
feature. As Edward noted, the NiFi Toolkit or NiPyAPI could be invoked from 
here to execute whatever steps you want taken.

[1] 
https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#event-hooks


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Oct 28, 2019, at 1:38 AM, Edward Armes 
mailto:edward.ar...@gmail.com>> wrote:

Hi Lei,

As far as I'm aware there currently isn't an out of the box method to do this
 However I would look at using a combination of the nifi registry and tools in 
nifi toolkit to roll your own.

Also, there have been multiple questions to both this and the dev mailing list, 
asking what could be done. I would also suggest searching the list archives and 
see what others did and didn't do as well.

Edward

On Mon, 28 Oct 2019, 08:14 
wangl...@geekplus.com.cn, 
mailto:wangl...@geekplus.com.cn>> wrote:

We have many standalone NiFi instances and all running the same  NiFi Flow.
If the flow changes,  how to deploy the change to all NiFi instances 
automatically?

Thanks,
Lei



wangl...@geekplus.com.cn

Re: NiFi WebGUI Timestamp issue

[Wesley C. Dias de Oliveira]

Are you using NTP servers?

Em qua, 30 de out de 2019 às 11:36,  escreveu:

> Hi Wesley
>
>
>
> I didn’t change anything on the machine during the time when I restarted
> the cluster (multiple times), however NiFi showed after each restart
> different timestamps/localizations.
>
>
>
> Cheers Josef
>
>
>
> *From: *"Wesley C. Dias de Oliveira" 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Wednesday, 30 October 2019 at 14:41
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi WebGUI Timestamp issue
>
>
>
> Hi, Josef.
>
>
>
> It seems to be related to machine time settings.
>
> Have you checked this?
>
>
>
> Em qua, 30 de out de 2019 às 06:57,  escreveu:
>
> Hi Guys,
>
>
>
> We just faced an issue with the time shown in the NiFi webgui (Screenshot
> below).
>
>
>
> We are in Switzerland, so at the moment we have “CET” (UTC + 1h). After a
> restart of our 8-node NiFi 1.9.2 cluster, NiFi suddenly showed “CET” - but
> the timestamp was in fact UTC, so minus 1h. Then we decided to restart all
> NiFi nodes again and now it’s even more confusing, it shows “UTC” but the
> time is from CET?
>
>
>
> [image: cid:16e1ce4a8a64cff311]
>
>
>
>
>
> Can anybody explain where the timestamp and the localization letters are
> coming from? After the reboot for a very short time period it showed
> another localization and timestamp, but then it changed to what I mentioned
> in the screenshot above.
>
>
>
> We have more clusters and single nifi’s and all the other machines show
> the correct timestamp and location… all have the same config.
>
>
>
> Cheers Josef
>
>
>
>
> --
>
> Grato,
>
> Wesley C. Dias de Oliveira.
>
>
>
> Linux User nº *576838.*
>


-- 
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.

Re: HandleHTTPRequest 503 error issue

[Peter Turcsanyi]

Hi Wyllys, sorry for my late reply. 1.10 release in progress, expected soon.

Regards,
Peter Turcsanyi

On Fri, Oct 4, 2019 at 10:22 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

>
> Yes, thanks.  Is there an ETA for 1.10?
>
> On Fri, Oct 4, 2019 at 3:05 PM Peter Turcsanyi 
> wrote:
>
>> Hi Wyllys,
>>
>> It has been fixed in NIFI-6317
>>  which will be shipped
>> in the upcoming 1.10 release.
>> Could you please retest your workload on that version when it is
>> available?
>>
>> Regards,
>> Peter Turcsanyi
>>
>> On Fri, Oct 4, 2019 at 6:28 PM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>>
>>>
>>> I believe this issue: https://issues.apache.org/jira/browse/NIFI-5522
>>> is a regression in Nifi 1.9.2, it was marked as fixed in 1.8.0, but Im
>>> seeing it a lot in 1.9.2
>>>
>>> I can recreate it fairly regularly when I am hitting the endpoint pretty
>>> heavily and an error is encountered.  The process return 503 error (or
>>> nothing at all) and cannot be stopped or restarted without rebooting the
>>> entire node.
>>>
>>> -Wyllys Ingersoll
>>>
>>>

Re: NiFi WebGUI Timestamp issue

[Josef.Zahner1]

Yes of course… 4 servers.

From: "Wesley C. Dias de Oliveira" 
Reply to: "users@nifi.apache.org" 
Date: Wednesday, 30 October 2019 at 16:56
To: "users@nifi.apache.org" 
Subject: Re: NiFi WebGUI Timestamp issue

Are you using NTP servers?

Em qua, 30 de out de 2019 às 11:36, 
mailto:josef.zahn...@swisscom.com>> escreveu:
Hi Wesley

I didn’t change anything on the machine during the time when I restarted the 
cluster (multiple times), however NiFi showed after each restart different 
timestamps/localizations.

Cheers Josef

From: "Wesley C. Dias de Oliveira" 
mailto:wcdolive...@gmail.com>>
Reply to: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Wednesday, 30 October 2019 at 14:41
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi WebGUI Timestamp issue

Hi, Josef.

It seems to be related to machine time settings.

Have you checked this?

Em qua, 30 de out de 2019 às 06:57, 
mailto:josef.zahn...@swisscom.com>> escreveu:
Hi Guys,

We just faced an issue with the time shown in the NiFi webgui (Screenshot 
below).

We are in Switzerland, so at the moment we have “CET” (UTC + 1h). After a 
restart of our 8-node NiFi 1.9.2 cluster, NiFi suddenly showed “CET” - but the 
timestamp was in fact UTC, so minus 1h. Then we decided to restart all NiFi 
nodes again and now it’s even more confusing, it shows “UTC” but the time is 
from CET?

[cid:16e1ce4a8a64cff311]


Can anybody explain where the timestamp and the localization letters are coming 
from? After the reboot for a very short time period it showed another 
localization and timestamp, but then it changed to what I mentioned in the 
screenshot above.

We have more clusters and single nifi’s and all the other machines show the 
correct timestamp and location… all have the same config.

Cheers Josef


--
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.


--
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.


smime.p7s
Description: S/MIME Cryptographic Signature

invokehttp timeout

[Michael Di Domenico]

i'm trying to use invokehttp to post a file to a webserver.  for
smaller files this seems to work just fine, but i try to send a larger
file, i get snagged with this

2019-10-30 14:47:59,188 ERROR [Timer-Driven Process Thread-10]
o.a.nifi.processors.standard.InvokeHTTP
InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610] Routing to Failure
due to exception:
org.apache.nifi.processor.exception.ProcessException: IOException
thrown from InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610]:
java.net.SocketTimeoutException: timeout:
org.apache.nifi.processor.exception.ProcessException: IOException
thrown from InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610]:
java.net.SocketTimeoutException: timeout
org.apache.nifi.processor.exception.ProcessException: IOException
thrown from InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610]:
java.net.SocketTimeoutException: timeout

i've lengthened the connect and read timeout in the properties to 1min
and 1hrs respectively, but it doesn't seem to help.

is there a setting i'm missing?  I also semi-related, i need to ensure
i get response from the webserver.  however, the web server might take
up 30 mins to respond.  will invokehttp hold the connection open, if
not is there a timeout setting somewhere?

Re: Implementing a custom provenance reporting task

[Tim Dean]

Thanks Matt - and sorry for the delayed response

I think this approach would work well for me, so I look forward to trying it 
out once it has been released.

-Tim


> On Oct 18, 2019, at 1:49 PM, Matt Burgess  wrote:
> 
> Tim,
> 
> The "secondary flow" issue is something I wanted to address as well,
> so I decoupled the formatting of data from the transmission of that
> data into a new paradigm/service I'm calling a RecordServiceSink. A
> ReportingTask can use RecordServiceSink to allow the user to choose
> where the reporting "records" are to go. I'm doing this work under
> NIFI-6780 [1] and the PR is currently under review [2], so hopefully
> it'll be coming soon to a NiFi near you :)
> 
> With this capability you could have a ProvenanceRecordReportingTask
> for example, one that uses a RecordSinkService rather than strictly
> site-to-site. My PR does include a SiteToSiteReportingRecordSink in
> order to add feature parity in case such a
> "ProvenanceRecordReportingTask" was implemented. If you use a
> SiteToSiteReportingRecordSink with it, I'd expect that to be pretty
> much the same as using the SiteToSiteProvenanceReportingTask. In your
> case you'd want to add a RecordSinkService implementation for your own
> in-house system, but in either case the reporting task itself would be
> the same. I expect it would probably look similar to the current
> SiteToSiteReportingTask in terms of code; the changes would be to get
> the results into a RecordSet and call sendData() on the
> RecordSinkService. With this we'd have a more generic reporting task
> that anyone could use to get provenance data to any supported "record
> sink"/target, without the need for a sub-flow, input port, and the
> associated extraneous provenance events generated for that sub-flow.
> 
> I should mention there's also a DatabaseRecordSink (that uses a
> DBCPConnectionPool controller service) in the PR as well, so there are
> two initial possible sinks/destinations available. I'll follow those
> up with a PrometheusRecordSink and perhaps some KafkaRecordSink
> implementations as well.
> 
> Happy to discuss this more, and please feel free to look at the PR and
> comment on it as you like, I'm always looking for feedback!
> 
> Regards,
> Matt
> 
> [1] https://issues.apache.org/jira/browse/NIFI-6780
> [2] https://github.com/apache/nifi/pull/3826
> 
> On Fri, Oct 18, 2019 at 1:50 PM Tim Dean  wrote:
>> 
>> Thanks Mike,
>> 
>> I understand what you are saying but I am really trying to avoid having a 
>> secondary flow in NiFi if I can avoid it.
>> 
>> It seems like NiFi was designed to allow this kind of custom reporting task 
>> and that I should ideally be able to do this without using the NiFi-provided 
>> S2S implementation if that doesn’t suit my needs.
>> 
>> - Tim
>> 
>> Sent from my iPhone
>> 
>>> On Oct 18, 2019, at 12:36 PM, Mike Thomsen  wrote:
>>>