RE: node died unexpectedly

2019-09-25 Thread DEHAY Aurelien
Classification\_- INTERNAL & PARTNERS Hello. By any chance, did you take thread dumps? If it happens again, take thread dumps of the java nifi process (with jstack: jstack -l >> threaddumps.log). Take 3 or 4 every 10/15 seconds . Regards. From: Jean-Sebastien Vachon Sent: lundi 23

FreeFormTextWriter and flowfiles attributes

2019-09-03 Thread DEHAY Aurelien
Classification\_- INTERNAL & PARTNERS Hello. FreeFormTextWriter is not reading the flowfile attributes when using ${}. I've try ${attributename} and ${anyAttribute('attributename')} but both fails silently and always return null (isNull is always true). Can FreeFormTextWriter access

RE: Process group name in S2SStatusReporterTask

2019-09-02 Thread DEHAY Aurelien
/e2ca50e66a3b1a7d810ea8eac256d21bc > a3fd07f/nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site- > reporting- > task/src/main/java/org/apache/nifi/reporting/SiteToSiteStatusReportingTas > k.java#L236 > > On Thu, Aug 29, 2019 at 5:59 AM DEHAY Aurelien >

RE: Process group name in S2SStatusReporterTask

2019-08-29 Thread DEHAY Aurelien
https://github.com/apache/nifi/blob/e2ca50e66a3b1a7d810ea8eac256d21bca3fd07f/nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteStatusReportingTask.java#L236> > > On Thu, Aug 29, 2019 at 5:59 AM DEHA

Process group name in S2SStatusReporterTask

2019-08-29 Thread DEHAY Aurelien
Classification\_- INTERNAL & PARTNERS Hello. I'm using the site2site status reporter task to monitor my Nifi Cluster. I'm collecting Input ports metrics and put them in InfluxDB. The applicationName is always the root process group name, wherever is my input put, even inside a process group.

Kerberized PutKudu

2019-08-01 Thread DEHAY Aurelien
Classification\_- INTERNAL & PARTNERS Hello. I still have some issues with the kerberized PutKudu on my 1.9.2 nifi cluster: 2019-08-01 16:15:28,207 DEBUG [Timer-Driven Process Thread-64] o.a.n.security.krb.AbstractKerberosUser TGT was found, but has not reached expiration window 2019-08-01

PutKudu and Kerberos issue at token renewal

2019-06-25 Thread DEHAY Aurelien
Hello. I got this error in app log: I'm using a keytab for a nifi service my cluster. When I restart the PutKudu processor, it works again, until next renewal. Anything I can do? Thanks; 2019-06-24 10:03:46,220 WARN [New I/O worker #796] org.apache.kudu.client.SecurityContext Caller-provided

Split Json and PathnotFoundException

2019-06-03 Thread DEHAY Aurelien
Hello. I have a question on splitjson regarding handling on path not found. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitJson.java#L201 Should not the log be in ERROR level? As

RE: HiveQL to load data in hive after generating file on HDFS

2019-04-10 Thread DEHAY Aurelien
r-template-faster.html On Tue, Apr 9, 2019 at 9:49 AM DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> wrote: Hello. We would like, in a flow to: 1) Read data from kafka 2) Merge records 3) PutParquet on HDFS 4) Load data in Hive using LOAD DATA hiveql. The fi

HiveQL to load data in hive after generating file on HDFS

2019-04-09 Thread DEHAY Aurelien
Hello. We would like, in a flow to: 1) Read data from kafka 2) Merge records 3) PutParquet on HDFS 4) Load data in Hive using LOAD DATA hiveql. The first 3 steps are no problem, it's working fine. But we wonder what is the best way to launch the hiveql command. We

RE: Empty "nifi users" page.

2019-03-29 Thread DEHAY Aurelien
ork around this by ensuring that your users and groups have unique names. Matt [1] https://issues.apache.org/jira/browse/NIFI-6161 On Thu, Mar 28, 2019 at 12:38 PM DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> wrote: Well, I can compare a lot of things I don’t really know where

RE: Empty "nifi users" page.

2019-03-28 Thread DEHAY Aurelien
ory server? To see what's being loaded from your directory server, if your running 1.9.x you should be able to enable debug level logging on org.apache.nifi.ldap.tenants.LdapUserGroupProvider Thanks Matt On Wed, Mar 27, 2019 at 5:46 AM DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> wrote

RE: Empty "nifi users" page.

2019-03-27 Thread DEHAY Aurelien
le its already addressed. Thanks Matt On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende mailto:bbe...@gmail.com>> wrote: What browser and browser version are you using? Have you tried clearing your browser cache just to make sure the page is loading properly? On Tue, Mar 26, 2019 at 10:

RE: Empty "nifi users" page.

2019-03-26 Thread DEHAY Aurelien
mmand line tool such as ldapsearch? I'm not sure if a search filter is required (although it would probably fail with an error if it was), but maybe try setting a "match all" filter such as (uid=*). Hope this helps, Kevin On Fri, Mar 22, 2019 at 11:08 AM DEHAY Aurelien wrote: &

Empty "nifi users" page.

2019-03-22 Thread DEHAY Aurelien
Hello. My "nifi users" page is empty, with only the spinner spinning at the bottom of the page. I've tried a lot of conf, and for now everything work fine, except that. There is nothing in the log, even when I tried to put "debug" for authorization parts in logback. I had the same behavior

RE: Re:LDAP, Groups and nifi users

2019-03-11 Thread DEHAY Aurelien
Hello. Thanks for your answer. I did not put the entire file, but the user and group entries are from the ldapusergroupprovider. Pierre already points me out to his docs and help me here to configure this, the ldap auth is working fine. I just can’t use anything related to groups From: Vishal

RE: LDAP, Groups and nifi users

2019-03-11 Thread DEHAY Aurelien
and nifi users Did you delete users.xml and authorizations.xml files? On Mon, 11 Mar 2019 at 09:16 DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> wrote: Hello. I’m struggling to configure a correct authorizes.xml to achieve the following. I’m using nifi 1.8 and 1.9 (freshy install) in

LDAP, Groups and nifi users

2019-03-11 Thread DEHAY Aurelien
Hello. I'm struggling to configure a correct authorizes.xml to achieve the following. I'm using nifi 1.8 and 1.9 (freshy install) in secure mode + ldap auth. - I have a LDAP serveur (RH identity manager) where users/groups are stored. - I'd like to be able to grant rights

Re: Json to SQL empty field

2019-02-03 Thread DEHAY Aurelien
>> a écrit : Hey, By no mean an expert on the subject, but, just to be sure, would expect the null value to be added in the database or do you really want the empty string? Pierre Le ven. 1 févr. 2019 à 16:37, DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> a écrit : Hello. We’re tryi

Json to SQL empty field

2019-02-01 Thread DEHAY Aurelien
Hello. We're trying to inject json data into impala (via jdbc connection and a DBCPConnectionPool) in Nifi 1.8.0. Given the following json (note the empty fields event): { "_msgid": "5c40a721b02db11867a3db95", "event": "" } PutDataBaseRecord or ConvertJsonToSQL+PutSQL is generating

RE: PutKudu & Kerberos

2019-02-01 Thread DEHAY Aurelien
type who is inclined to build the branch and test it out, please do so. Otherwise, I suspect it will make it into the next release. Thanks -Mark [1] https://github.com/apache/nifi/pull/3279 [2] https://issues.apache.org/jira/browse/NIFI-5984 On Jan 31, 2019, at 4:27 AM, DEHAY Aure

PutKudu & Kerberos

2019-01-31 Thread DEHAY Aurelien
Hello. Is there a way / plan to add support for Kerberos authentication in PutKudu processor? Thanks. This electronic transmission (and any attachments thereto) is intended solely for the use of the addressee(s). It may contain confidential or legally privileged information. If you are not

Minifi S2Sreporting

2018-12-11 Thread DEHAY Aurelien
Hello. I was looking a way to send bulletin, metrics & status from minify instances to Nifi, and s2sreporting tasks are not bundled with minify. I'd rather to using the query & export feature from the command line as I don't necessarily have access to the servers where minify are

RE: Nifi Cluster & LDAP

2018-11-09 Thread DEHAY Aurelien
sers/identities, it's not updated on next restarts. Pierre Le ven. 9 nov. 2018 à 11:42, DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> a écrit : Hello Pierre. https://gist.github.com/zorel/3655188026b0355c8860030932884fa9 I’ve tried to comment edge2 and 3, but I have now the error

RE: Nifi Cluster & LDAP

2018-11-09 Thread DEHAY Aurelien
tial users/identities, it's not updated on next restarts. Pierre Le ven. 9 nov. 2018 à 11:42, DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> a écrit : Hello Pierre. https://gist.github.com/zorel/3655188026b0355c8860030932884fa9 I’ve tried to comment edge2 and 3, but I have now the error

RE: Nifi Cluster & LDAP

2018-11-09 Thread DEHAY Aurelien
ated in the authorizers.xml file. Do you mind sharing it (after removing anything sensitive)? No need to add the nodes in the LDAP and it's definitely not related to keystores. Pierre Le ven. 9 nov. 2018 à 11:30, DEHAY Aurelien mailto:aurelien.de...@faurecia.com>> a écrit : Hello. I'm

Nifi Cluster & LDAP

2018-11-09 Thread DEHAY Aurelien
Hello. I'm struggling to configure the very first node of my 3 nodes nifi 1.8.0 cluster. I've used the toolkit to create the jks: bin/tls-toolkit.sh standalone -n 'par01prdedge[1-3].fqdn' -C "CN=admin,OU=nifi" -c "par01prdedge1" -d 3650 -o ~/nifi-data/toolkit -O I have then 3 directories,

RE: How to only get the newest flowfile from a queue

2018-09-13 Thread DEHAY Aurelien
Hello. Simpliest solution in my opinion, rather than trying to do that entirely in nifi, is to use an shell script and run it via ExecuteStreamCommand as an input for FetchFile. -- Aurélien From: scott [mailto:tcots8...@gmail.com] Sent: mercredi 12 septembre 2018 16:56 To:

RE: saving record into different tables?

2018-09-03 Thread DEHAY Aurelien
Hello. I would flatten the records with jolt, with a kind of flag to know if it’s “root” data or not. Then I would extract the flag in an attribute with evaluatejson path, and at the end route with RouteOnAttribute. Aurélien From: l vic [mailto:lvic4...@gmail.com] Sent: samedi 1 septembre

RE: Wrapping a JSON string

2018-09-03 Thread DEHAY Aurelien
Hello. You will have to remove the “ in the wrapped json, and process it like a standard string, so I would use the text processors instead of a json one. We did something different from our side: we use encodeJSON do encapsulate the json payload, encoded (“ => \” mainly) in a payload string:

RE: QueryDatabaseTable date/timestamp and Oracle.

2018-05-02 Thread DEHAY Aurelien
and persisting timestamps in MySql table on a side. You will have more control that way and you can add your own audit and logging. It was quite easy with NiFi and I was in full control of Oracle queries, executed by NiFi. Boris On Tue, Apr 24, 2018 at 6:04 AM, DEHAY Aurelien <aurelien

RE: QueryDatabaseTable date/timestamp and Oracle.

2018-04-27 Thread DEHAY Aurelien
. It was quite easy with NiFi and I was in full control of Oracle queries, executed by NiFi. Boris On Tue, Apr 24, 2018 at 6:04 AM, DEHAY Aurelien <aurelien.de...@faurecia.com<mailto:aurelien.de...@faurecia.com>> wrote: Hello. I'm using Oracle 11g with nifi 1.6.0. I'm trying to use QueryD

QueryDatabaseTable date/timestamp and Oracle.

2018-04-24 Thread DEHAY Aurelien
Hello. I'm using Oracle 11g with nifi 1.6.0. I'm trying to use QueryDatabaseTable to fetch newly modified/created rows from a table with a few million records. The field I use as maximum value column is "sysmodtime", a DATE field. When I put something like 2018-04-23 00:00:00.0 in

MergeRecord, queue & backpressure

2018-04-13 Thread DEHAY Aurelien
Hello. It's me again regarding my mergerecord question. I still don't manage to have what I want, I may have understand how bin based processor works, it's for clarification and a question regarding performance. I want to merge a huge number of 300 octets flowfiles in 128 MB parquet file. My

RE: MergeRecord

2018-04-13 Thread DEHAY Aurelien
i-stand > ard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/proc > essors/standard/MergeRecord.java#L348 > > So, even if schema is the same, if actual values vary, merging group > id will be different. > If you can use SchemaRegistry, it should work as expected. >

Re: MergeRecord

2018-04-12 Thread DEHAY Aurelien
ctual MergeRecord behavior so I may be wrong, but > worth to change the configuration. > > Hope this helps, > Koji > > > On Fri, Apr 13, 2018 at 12:26 AM, DEHAY Aurelien > <aurelien.de...@faurecia.com> wrote: >> Hello. >> >> Please see the template att