[ https://issues.apache.org/jira/browse/NIFI-8423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325735#comment-17325735 ]
Josef Zahner edited comment on NIFI-8423 at 4/20/21, 12:06 PM: --------------------------------------------------------------- I don't know whether p-li-nifi-05 is the only node which shows the discrepancy, under normal circumstances no ERRORs will be generated so I don't know when and if it occurs on other nodes. I've already restarted NiFi on all nodes multiple times. It's really difficult to tell if it's always the same node (or multiple nodes) What I can tell is that NiFi runs only once: {code:java} [user@p-li-nifi-05 ~]$ ps aux | grep nifi user 19597 0.0 0.0 112808 968 pts/0 S+ 13:13 0:00 grep --color=auto nifi nifi 51798 0.0 0.0 113412 760 ? S 10:46 0:00 /bin/sh /opt/nifi/bin/nifi.sh start nifi 51800 0.0 0.0 7093000 197364 ? Sl 10:46 0:06 /usr/lib/jvm/java/bin/java -cp /opt/nifi-1.13.2/conf:/opt/nifi-1.13.2/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi -Dorg.apache.nifi.bootstrap.config.pid.dir=/var/run/nifi -Dorg.apache.nifi.bootstrap.config.file=/opt/nifi-1.13.2/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start nifi 51872 332 14.0 74496404 37058284 ? Sl 10:47 487:17 /usr/lib/jvm/java/bin/java -classpath /opt/nifi-1.13.2/./conf:/opt/nifi-1.13.2/./lib/javax.servlet-api-3.1.0.jar:/opt/nifi-1.13.2/./lib/jetty-schemas-3.1.jar:/opt/nifi-1.13.2/./lib/logback-classic-1.2.3.jar:/opt/nifi-1.13.2/./lib/logback-core-1.2.3.jar:/opt/nifi-1.13.2/./lib/jcl-over-slf4j-1.7.30.jar:/opt/nifi-1.13.2/./lib/jul-to-slf4j-1.7.30.jar:/opt/nifi-1.13.2/./lib/log4j-over-slf4j-1.7.30.jar:/opt/nifi-1.13.2/./lib/slf4j-api-1.7.30.jar:/opt/nifi-1.13.2/./lib/nifi-api-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-framework-api-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-server-api-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-runtime-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-nar-utils-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-properties-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-stateless-bootstrap-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-stateless-api-1.13.2.jar -XX:+PrintGCDetails -Dcom.sun.management.jmxremote.rmi.port=30008 -XX:+UseGCLogFileRotation -Xloggc:/var/log/nifi/gc.log -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=30008 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=true -Duser.timezone=Europe/Zurich -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx31g -Xms8g -Dcurator-log-only-first-connection-issue-as-error-level=true -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:ReservedCodeCacheSize=256m -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -XX:NumberOfGCLogFiles=10 -XX:+UseCodeCacheFlushing -XX:CodeCacheMinimumFreeSpace=10m -XX:GCLogFileSize=10m -Dnifi.properties.file.path=/opt/nifi-1.13.2/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39853 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi nifi 53618 0.3 0.0 31964 23484 ? Ssl 09:11 0:44 /opt/pushgateway/pushgateway --web.listen-address 0.0.0.0:9091 {code} We have a systemd services for NiFi, so in theory it should never start more than one NiFi: {code:java} [user@p-li-nifi-05 ~]$ cat /etc/systemd/system/nifi.service [Unit] Description=Apache NiFi After=network.target[Service] Type=forking User=nifi Group=nifi RuntimeDirectory=nifi RuntimeDirectoryMode=0775 ExecStart=/opt/nifi/bin/nifi.sh start ExecStop=/opt/nifi/bin/nifi.sh stop ExecReload=/opt/nifi/bin/nifi.sh restart LimitNOFILE=1000000[Install] WantedBy=multi-user.target {code} Is there an easy way to generate an ERROR on NiFi? was (Author: jzahner): I don't know whether p-li-nifi-05 is the only node which shows the discrepancy, under normal circumstances no ERRORs will be generated so I don't know when and if it occurs on other nodes. I've already restarted NiFi on all nodes multiple times. It's really difficult to tell if it's always the same node (or multiple nodes) What I can tell is that NiFi runs only once: {code:java} [user@p-li-nifi-05 ~]$ ps aux | grep nifi user 19597 0.0 0.0 112808 968 pts/0 S+ 13:13 0:00 grep --color=auto nifi nifi 51798 0.0 0.0 113412 760 ? S 10:46 0:00 /bin/sh /opt/nifi/bin/nifi.sh start nifi 51800 0.0 0.0 7093000 197364 ? Sl 10:46 0:06 /usr/lib/jvm/java/bin/java -cp /opt/nifi-1.13.2/conf:/opt/nifi-1.13.2/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi -Dorg.apache.nifi.bootstrap.config.pid.dir=/var/run/nifi -Dorg.apache.nifi.bootstrap.config.file=/opt/nifi-1.13.2/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start nifi 51872 332 14.0 74496404 37058284 ? Sl 10:47 487:17 /usr/lib/jvm/java/bin/java -classpath /opt/nifi-1.13.2/./conf:/opt/nifi-1.13.2/./lib/javax.servlet-api-3.1.0.jar:/opt/nifi-1.13.2/./lib/jetty-schemas-3.1.jar:/opt/nifi-1.13.2/./lib/logback-classic-1.2.3.jar:/opt/nifi-1.13.2/./lib/logback-core-1.2.3.jar:/opt/nifi-1.13.2/./lib/jcl-over-slf4j-1.7.30.jar:/opt/nifi-1.13.2/./lib/jul-to-slf4j-1.7.30.jar:/opt/nifi-1.13.2/./lib/log4j-over-slf4j-1.7.30.jar:/opt/nifi-1.13.2/./lib/slf4j-api-1.7.30.jar:/opt/nifi-1.13.2/./lib/nifi-api-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-framework-api-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-server-api-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-runtime-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-nar-utils-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-properties-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-stateless-bootstrap-1.13.2.jar:/opt/nifi-1.13.2/./lib/nifi-stateless-api-1.13.2.jar -XX:+PrintGCDetails -Dcom.sun.management.jmxremote.rmi.port=30008 -XX:+UseGCLogFileRotation -Xloggc:/var/log/nifi/gc.log -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=30008 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=true -Duser.timezone=Europe/Zurich -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx31g -Xms8g -Dcurator-log-only-first-connection-issue-as-error-level=true -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:ReservedCodeCacheSize=256m -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -XX:NumberOfGCLogFiles=10 -XX:+UseCodeCacheFlushing -XX:CodeCacheMinimumFreeSpace=10m -XX:GCLogFileSize=10m -Dnifi.properties.file.path=/opt/nifi-1.13.2/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39853 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi nifi 53618 0.3 0.0 31964 23484 ? Ssl 09:11 0:44 /opt/pushgateway/pushgateway --web.listen-address 0.0.0.0:9091 {code} We have a systemd services for NiFi, so in theory it should never start more than one NiFi: {code:java} [user@p-li-nifi-05 ~]$ cat /etc/systemd/system/nifi.service [Unit] Description=Apache NiFi After=network.target[Service] Type=forking User=nifi Group=nifi RuntimeDirectory=nifi RuntimeDirectoryMode=0775 ExecStart=/opt/nifi/bin/nifi.sh start ExecStop=/opt/nifi/bin/nifi.sh stop ExecReload=/opt/nifi/bin/nifi.sh restart LimitNOFILE=1000000[Install] WantedBy=multi-user.target {code} > Timezone wrong in UI for an 8 node cluster > ------------------------------------------ > > Key: NIFI-8423 > URL: https://issues.apache.org/jira/browse/NIFI-8423 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI > Affects Versions: 1.13.2 > Environment: 8 Node NiFi Cluster on CentOS 7 > OpenJDK 1.8.0_282 > Local timezone: Europe/Zurich (CEST or UTC+2h) > Reporter: Josef Zahner > Priority: Critical > Labels: centos, cluster, openjdk, timezone > Attachments: image-2021-04-13-15-14-02-162.png, > image-2021-04-13-15-14-06-930.png, image-2021-04-13-15-14-56-690.png, > manual_configured_timezone_gui_output.png, nifi-app_log.png > > > We just upgraded to NiFi 1.13.2 and Java 1.8.0_282 > On our 8 node NiFi 1.13.2 cluster with timezone Europe/Zurich (CEST/CET), we > have the issue that the UI does display the correct timezone (CEST, so UTC > +2h), but in fact the time is displayed as UTC. NTP is enabled and working. > The OS configuration/location is everywhere the same (doesn't matter if > single or cluster NiFi). My tests below are all done at around 15:xx:xx local > time (CEST). > As you can see below, the timezone seems to be correct, but the time itself > within NiFi is 2h behind (so in fact UTC) compared to Windows. In earlier > NiFi/java versions it was enough to multiple times restart the cluster, but > on the newest versions this doesn't help anymore. It shows most of the time > CEST with the wrong time or directly UTC. > !image-2021-04-13-15-14-06-930.png! > > The single NiFi instances or the 2 node clusters are always fine. The issue > exists only on our 8 node cluster. > NiFi Single Node Screenshot, which is fine (CEST, so UTC + 2h): > !image-2021-04-13-15-14-02-162.png! > > If we set the -Duser.timezone to "Europe/Zurich" in bootstrap.conf, the UI > shows no summer time, so only GMT+1 instead of GMT+2. As well not what we > want. > {code:java} > java.arg.20=-Duser.timezone="Europe/Zurich"{code} > !image-2021-04-13-15-14-56-690.png! > > What Matt below suggested has been verified, all servers (single nodes as > well as clusters) are reporting the same time/timezone. > [https://community.cloudera.com/t5/Support-Questions/NiFi-clock-is-off-by-one-hour-daylight-savings-problem/td-p/192942] > > So the question remains, where on a NiFi cluster comes the time from the UI > and what could cause it that it is wrong? Sometimes I get UTC, sometimes I'm > getting CEST but the time is anyhow UTC instead of CEST... I really need to > have the correct time in the UI as I don't know what the impact could be on > our dataflows. > > Any help would be really appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)