Hi David,
Thanks for reponse. Yes, I've already read this article from exceptionFactory.
I will check with my kubernetes team.
Thanks
Envoyé: lundi 30 septembre 2024 à 20:39
De: "David Handermann"
À: users@nifi.apache.org
Objet: Re: cluster nifi hosted in kubernetes
Hi Minh,
There
Hello all,
Until now we hosted our clusters nifi from VM installed.
We want now to create clusters nifi to kubernetes to have the possibility to scale up and down more easier than just with VM.
Could you tell me it is possible with kubernetes and have we got some documentations ?
Hello nifi team.
Somebody has already deployed nifi cluster with Kubernetes ?
Do you need to develop some customs operators to manage the cluster ?
Have you got some tutos to do this thing ?
Thanks all
Regards
Minh
Thanks for reply Mark,
I checked in the nifi-app.log, we got 0 error :(
Only "INFO" message and the last messages are the same.
Below the message from another node of cluster
2024-07-09 06:42:52,175 INFO [pool-2-thread-1] org.apache.nifi.BootstrapListener Received SHUTDOWN request from
The only except INFO, there are only this line in bootstrap.log
[root@ope-nifi-dfp-vm-19 nifi]# grep -v INFO nifi-bootstrap.log
2024-07-08 07:13:39,883 WARN [main] org.apache.nifi.bootstrap.Command NiFi PID [71123] shutdown not completed after 20 seconds: Killing process
2024-07-08 07:14:08
Hello all,
How it is possible to have more details about this error ?
Here the last logs we got before NIFI shutdown without another Error in the nifi-app.log:
2024-07-08 10:01:57,988 INFO [pool-2-thread-1] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap
2024-
Hello Deepak,
We have migrated all our nifi instances to Rocky Linux 8 and until now no got any issue.
Regards
Minh
Envoyé: vendredi 14 juin 2024 à 19:59
De: "Joe Witt"
À: users@nifi.apache.org
Objet: Re: Nifi on RockyLinux
Deepak
In Apache we don't have mechanisms to veri
Hello all,
Somebody could tell me what is impacted when we increase the number of "Concurrent Tasks" in the inputPort ?
Regards
Minh
Hello all,
I'm looking for the connection and fetch data from FIRESTORE GCP.
Have we something in NIFI to do this thing ?
Thanks
Minh
Hello Joe,
Thanks for the response.
But I found the solution.
Like the table size is around 7,6GB (avro) for 161000 rows
And I setup Fetch size : 20 so the processor ExecuteSQL try to fetch all datas.
Like the heap size is setup to 8GB
I have reduced the "Fetch Size" to 1 a
Hello
I got the executeSQL processor does the sql command "select * from public.table1"
It is a postgresql database.
Here the end of properties of processor.
Max Wait Time 0 seconds
Normalize Table/Column Names false
Use Avro Logical Types false
Compression Format NONE
D
Hello Lehel,
Yes, I have already try to use UpdateAttribute with "Store State Locally".
The issue I saw it is when the same dataflow run multiple time the value "token" is increase all the times
I need to have the "counter" reset to Zero when the dataflow is consider to done.
Also, I have
Hello all,
Somebody has already use the Counter or UpdateCounter ?
My use case is sending a message (success) to topic pubsub if my nifi dataflow has consider to finish.
The input of dataflow is for example 3 flowfiles (do so treatements) and the ouput will be push to the GCS Bucket.
Hi all,
I don't how explain this error.
We use the MQV9.2 as producer and consume by NIFI 1.19.1 with processor ConsumeJMS and drivers :
ls -rtl mqm_ibm
total 11384
-rw-r--r-- 1 nifi nifi 2111220 Jun 26 2018 ojdbc-11.2.0.3-jdk16.jar
-rw-r--r-- 1 nifi nifi 88374 Jun 26 2018 com.ibm
Hello all,
Somebody has already seen this error and how to correct it ?
Unable to write flowfile content to content repository container repo0 due to archive file size constraints; waiting for archive cleanup. Total number of files currently archived = 1
my filesystem is up to
Hello Mark,
I shared the screenshot about ExecuteSQL properties.
This 3 properties together work very well for PostGRESQL in the NIFI with 2CPU/8Gb.
Without the properties "Set Auto Commit" to FALSE, the NIFI stuck and crash.
Thanks a lot for help
Minh
Envoyé: vendredi
So after some tests, here the result perhaps could help someone.
With nifi (2CPU / 8Go Ram)
I have tested with these couples properties :
> 1 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 10
> 2 executeSQL with "select * from table"
Output Batch S
And the important thing for us it has only one avro file by table.
So it is possible to merge avro files to one avro file ?
Regards
Envoyé: jeudi 4 janvier 2024 à 19:01
De: e-soci...@gmx.fr
À: users@nifi.apache.org
Cc: users@nifi.apache.org
Objet: Re: Hardware requirement for NIFI i
Hello all,
Thanks a lot for the reply.
So for more details.
All the properties for the ExecuteSQL are set by default, except "Set Auto Commit: false".
The sql command could not be more simple than "select * from ${db.table.fullname}"
The nifi version is 1.16.3 and 1.23.2
Hello all,
Who could help me to determine the cpu/memory need for nifi instance to fetch the data from Postgresql hosted in google ?
We got this error :
==> Error : executesql.error.message
Ran out of memory retrieving query results.
The procesor ExecuteSQL has this config : Set Au
Hello all,
It was working with NIFI 1.18.0 and JDK 1.8.0_311
Have you got some issues with ListenHTTP with the length of file receive by NIFI ?
Error : "Form is larger than max length 20"
Do you know how correct it ?
Regards
Minh
java.lang.IllegalStateException: Form i
Hi Steve,
I found the error :)
It seems block by the security team...
"This category is not allowed by the company IS security policies."
Regards
Minh
Envoyé: lundi 27 novembre 2023 à 16:47
De: e-soci...@gmx.fr
À: users@nifi.apache.org, stephen.hindma...@bt.com
Cc: users@nifi.apache.
Thanks, for the informations, I will recheck from my side
Minh
Envoyé: lundi 27 novembre 2023 à 16:38
De: "stephen.hindmarch.bt.com via users"
À: users@nifi.apache.org
Objet: RE: invokeHTTP SSL error NIFI : 1.23.2
Minh,
I would be reluctant to simply trust an external web site’s
Hello,
It is not working
I have used :
# true | openssl s_client -showcerts -connect api-eu.reputation.com:443
I saw it is manage by Let Encrypt
And the let Encrypt CA is already setup in the file 11.0.17/lib/security/cacerts => alias name: letsencryptisrgx1 [jdk]
so I co
Hello
Thank for reply, the weird thing it is until now, I don't use SSL context and it is working.
Good anyway, I will get the server certificate and add it in the truststore and configure invokeHTTP to user SSL context also
Thanks
Minh
Envoyé: lundi 27 novembre 2023 à 10:48
D
Hello all,
Since I've upgraded the nifi version from 1.18 to 1.23.2 - Java Version 11.0.17
I got the error concerning the invokeHTTP (GET https://api-eu.reputation.com/v3/ ..) even if I setup SSL Context or not
Do you have informations about what has changed between the 2 nifi version ?
Hello Quentin,
In the normal way, if the processor is started correctly, you have this logs
My schedule ==> 0 0/2 * * * ?
==> 2023-11-16 10:28:00,002 DEBUG [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=7edea667-2e42-39c5-7252-bda580ba683a] Returning C
Hello Quentin,
It seems strange to have log at 06h00 PM, it is not related to your CRON strategy.
What is you have you do this CRON "«0 15 1/1 * * ? *»
Anyway, logs seems saying it is running once time.
Have you got "clear state" for ListSFTP to be sure ?
Regards
Minh
E
Hello
Thanks Mark
What could be a good way to debug "google processor"
(not working)
Thanks
Envoyé: mardi 14 novembre 2023 à 16:11
De: "Mark Payne"
À: "users@nifi.apache.org"
Objet: Re: ListSFTP Processor CRON doesn't start
Hi Minh,
No - you can configure l
Hello Mark,
Have we got the documentation about exhaustive list about logger we got have in NIFI ?
Regards
Minh
Envoyé: mardi 14 novembre 2023 à 15:25
De: "Mark Payne"
À: "users"
Objet: Re: ListSFTP Processor CRON doesn't start
Hi Quentin,
What is the CRON schedule that you
Hello,
Seems not working, I don't have any logs concerning google/putgcsObjects in the nifi-app.log
Missing something ?
Minh
Envoyé: vendredi 10 novembre 2023 à 15:36
De: "Pierre Villard"
À: users@nifi.apache.org
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
You'd just ne
Hello Pierre,
I'm not developper, so it is a bit confuse for me.
I'm trying to temporarily push this config in the logback.xml but it is not working.
.
.
.
.
.
It is missing something ?
Thanks a lot for your help
Regards
Minh
Envoyé: mercred
Hi all,
It is strange for the putGCSObject with properties "GZIP Compression Enabled"
The description said the most time is better to setup at "false" but by default is "true"
"Signals to the GCS Blob Writer whether GZIP compression during transfer is desired. False means do not gzi
Hi Pierre,
Thanks for this information.
Minh
Envoyé: mercredi 8 novembre 2023 à 19:57
De: "Pierre Villard"
À: users@nifi.apache.org
Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2
So I believe the client library, when using the private endpoint, is making multiple calls, the fi
Hi Pierre,
I confirm, it works with Proxy.
But how we are sure that the data are transfert from the "private connection" and not through the proxy...
Regards
Envoyé: mercredi 8 novembre 2023 à 16:36
De: e-soci...@gmx.fr
À: users@nifi.apache.org
Objet: Re: Bug : PutGCSObject : Ver
Hi Pierre,
But it seems weird because when I put the wrong "storage api url : https://storage-euwest1.p.googleapis.com/storage/v1/"
We got the wrong url but has the correct hostname
Not Found POST https://storage-euwest1.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/m
Hi
I have checked the DNS entry for "storage-euwest1.p.googleapis.com" it seems OK
From the VM nifi, in the terminal, I could reach the port 443 it works fine.
I check with the devops google team also.
Thanks
Envoyé: mercredi 8 novembre 2023 à 15:40
De: "Joe Witt"
À: users@nifi.
Hello Joe
Until now we leave the "storage api url" empty by default, but we use "proxy configuration service" because we are behind the corporate network.
Since some month ago we have configured "private connection" with google.
So I have setup "storage api url" with the private url and le
Hello,
I'm trying to user the overwriting storage url parameter in the nifi version 1.23.2.
Do you know what is the correct parameter to set for the "Storage API URL" in the processor PutGCSObject ?
If I setup "https://storage-euwest1.p.googleapis.com/storage/v1/"
It got this error :
Hello Joe,
It is a VM with Rocky 8 as OS. We use the systemd to start/stop nifi process
The nifi process is die. I could see with the journalctl -u nifi
Oct 27 14:53:11 server01 systemd[1]: Started Apache NiFi.
Oct 28 03:31:04 server01 systemd[1]: nifi.service: Supervising proces
Hello all,
I have activated the "diagnosticts on stop" in the nifi.properties.
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#nifi_diagnostics
My node has stop twice time now but I don't find anything.
Have we got some documentations about the log of diagnosticts t
Hello
Thanks for your reply, I think I'm doing something like you to find the roort cause but it is often I need to check a lot of thing.
For exemple, when I saw the "Java heap space" in the nifi-app.log
It is not easy to see what processor has cause this issue.
Anyway thanks
Regar
Hello all,
Since I'm working with NIFI, I got a lot of difficulty to find the root cause why the node is disconnected to the cluster.
I check nifi-app.log nifi-bootstrap.log but it is a bit complicated to find the correct information.
Somebody has some informations for me ?
Regards
Hello all,
Somebody has already done some configuration for schedule the processor with the particular (CET/France) timezone even if the NIFI is running in the UTC ?
Here the definition of summer/winter time
Metropolitan France uses Central European Time (heure d'Europe centrale, UT
Hello Pierre,
Thanks for new option.
So it means, the dataflow_1 register in the registry_A, will be register in the same way in registry_B after migration ?
Or needs we also use the URL aliasing ?
Regards
Envoyé: vendredi 29 septembre 2023 à 12:41
De: "Pierre Villard"
À: users@
Hello all,
Someone has already migrate their nifi-registry from hostA to hostB (nifi-registry) and keep track the versionning for the nifi client points to the hostB ?
Have we got some documentations about migration nifi-registry ?
regards
Minh
Hello,
Sorry, I have not correctly checked in the release note.
Great new, thanks a lot
regards
Minh
Envoyé: mardi 19 septembre 2023 à 11:38
De: "Pierre Villard"
À: users@nifi.apache.org
Objet: Re: GCS processors : endpoint Override URL: " storage-xyz.p.googleapis.com"
Hi
Hello all,
Is it possible to add an option in the processor GCS so we can write the custom url for storage.googleapis.com something like "storage-xyz.p.googleapis.com"?
I know it is possible for the "S3 Amazon processor" :
Endpoint Override URL
Endpoint Override URL
Hello Mike,
Could you please give me the details about the resolution ?
Have you change something in the processor or just changing the sql command ?
Regards
Envoyé: samedi 2 septembre 2023 à 00:00
De: "Mike Thomsen"
À: users@nifi.apache.org
Objet: NiFi hanging during large sql
Hello Jens,
Yes it is a bit too long to read but very interesting.
Thanks a lot for this information. I'm not sure can do this thing in my work.
I do simple for testing and use only one Haproxy.
Thanks
Minh
Envoyé: jeudi 7 septembre 2023 à 18:58
De: "Jens M. Kofoed"
À: use
Hello Quentin,
I also used the python with ExecuteScript but it is not native and not really supported.
The better thing it is use executeGroovy processor.
Minh
Envoyé: vendredi 8 septembre 2023 à 22:55
De: "Rafael Fracasso"
À: users@nifi.apache.org
Objet: Re: [NIFI] ExecuteScrip
Hello Jens
Thanks a lot for haproxy conf.
Could you give more details about this point :
- I have 2 proxy nodes, which is running in a HA setup with keepalived and with a vip.-
- I have a dns record nifi-cluster01.foo.bar pointing to the vip address to keepalived
Thanks
Hello YolandaWhere is the document about accepted parameter to set after “includedRegistries”Thanks for helpEnvoyé à partir de l'app mail mobile sur 14/08/2023 le 17:38, Yolanda Davis écrivit:
Hello all,
We're facing a lot of issue concerning the timeout for "PutGCSObject"
PutGCSObject[id=d5a6d876-486f-3f6d-acba-8c661b3527af] Failed to put FlowFile[filename=xxx] to Google Cloud Storage due to Read timed out: com.google.cloud.storage.StorageException: Read timed out - Caused by:
54 matches
Mail list logo