Re: Run Nifi in IntelliJ to debug?
Did you follow Kevin’s guide below? If you put a breakpoint inside any of the code other than the bootstrap, the IDE should pause there. Many of the NiFi developers perform this process daily. If you are still having trouble, please share your bootstrap.conf file and the locations of the breakpoints within your code. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 27, 2020, at 9:08 AM, Darren Govoni wrote: > > Hello, >So i was able to get intelliJ to debug nifi but only inside the bootstrap > process. It looks like nifi spawns a new process and that process does not > run the debug options. > > Is there a way to instruct nifi to enable debug port on its main process? > That will have the actual app code i need to trace. > > Thanks for any tips. Much appreciated! > Darren > > Sent from my Verizon, Samsung Galaxy smartphone > Get Outlook for Android <https://aka.ms/ghei36> > From: Mike Thomsen > Sent: Monday, October 26, 2020 10:15:33 PM > To: users@nifi.apache.org > Subject: Re: Run Nifi in IntelliJ to debug? > > Are you using a binary derived from the source code in your IDE? Like > a 1.12.1 binary and the source code from the release? > > On Mon, Oct 26, 2020 at 7:47 PM Russell Bateman wrote: > > > > Hmmm... It's rare that I debug NiFi code. And it's also rare that I debug > > my own in that context since the NiFi test runner allows me to fend off > > most surprises via my JUnit tests. > > > > I think back in 2016, I was debugging a start-up problem involving NiFi > > start-up and incompatibility with the Java Flight Recorder. As I recall, I > > downloaded the relevant NiFi code sources matching the version of NiFi I > > was debugging remotely. I remember ultimately making a slight (and only > > temporary) change to NiFi start-up that fixed the problem. At that point I > > must have been building my own copy to have seen it fixed.. It had to do > > with the order in which NiFi was getting command-line arguments making it > > so the JFR wasn't running. I'd have to dig back to figure out what I was > > doing, but it's probably not too relevant to what you need to do. > > > > What do you need to see in this? > > > > Russ > > > > On 10/26/20 5:38 PM, Darren Govoni wrote: > > > > Correct. Primarily the nifi-web-api module and AccessResource class. For > > starters. > > > > Sent from my Verizon, Samsung Galaxy smartphone > > Get Outlook for Android > > > > > > From: Russell Bateman > > Sent: Monday, October 26, 2020 7:37:13 PM > > To: Darren Govoni ; users@nifi.apache.org > > > > Subject: Re: Run Nifi in IntelliJ to debug? > > > > Darren, > > > > This is just Apache NiFi code out of NARs you want to step through or is it > > yours? You haven't stripped debug information or anything, right? > > > > Russ > > > > On 10/26/20 5:30 PM, Darren Govoni wrote: > > > > Kevin/Russel > > > > Thanks for the info. I did set things up this way. > > > > IntelliJ does connect to the nifi jvm and nifi runs and works but intellij > > isnt breaking on code it should. > > > > I did set the module where the code/classes are located (in the remote > > connection dialog) and i see the exception im tracking print on the console > > output but intellij never breaks. > > > > Is there an extra step needed? Generate sources? > > > > For future it would be nice if there was a maven goal for debug. > > > > Much appreciated! > > Darren > > > > Sent from my Verizon, Samsung Galaxy smartphone > > Get Outlook for Android > > > > From: Russell Bateman > > Sent: Monday, October 26, 2020 4:09:50 PM > > To: users@nifi.apache.org ; Darren Govoni > > > > Subject: Re: Run Nifi in IntelliJ to debug? > > > > Darren, > > > > I was out this morning and didn't see your plea until I got in just now. > > Here's a step by step I wrote up for both IntelliJ IDEA and Eclipse (I'm > > more an IntelliJ guy). It also covers using an IP tunnel. > > > > https://www.javahotchocolate.com/notes/nifi.html#20160323 > > <https://www.javahotchocolate.com/notes/nifi.html#20160323> > > > > On 10/26/20 9:52 AM, Darren Govoni wrote: > > > > Hi > >Is it possible to run Nifi from inside IntelliJ with debugging such that > > I can hit the app from my browser and trigger breakpoints? > > > > If anyone has done this can you please share any info? > > > > Thanks in advance! > > Darren > > > > Sent from my Verizon, Samsung Galaxy smartphone > > Get Outlook for Android > > > > > > > >
Re: Securing MiNiFi
We’ll try to clean up the NiFi docs, but vendor documentation is out of our control. Glad it’s working for you now. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 21, 2020, at 17:46, scotty wrote: > > That was it! I didn't have nifi.remote.input.http.enabled= true. > > I was going off of the link below where it says "By default, it is set to > true." I guess I misread it to mean that if nothing was specifically entered > there then it would default to true. > > https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.2/bk_administration/content/site_to_site_properties.html > > > > Anyway, thank you, greatly, for the help. > > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Securing MiNiFi
I think you need to at least populate one of, if not both, nifi.remote.input.socket.port= and nifi.remote.input.http.enabled=. If there is no socket port specified, and HTTP transmission is not enabled, I don’t think S2S will work. I also don’t see any keystore or truststore configured; did you just not share that portion? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 21, 2020, at 11:18 AM, scotty wrote: > > Hi, > > Here's the info: > > # Site to Site properties > > nifi.remote.input.host=mydomain.com > nifi.remote.input.secure=true > nifi.remote.input.socket.port= > nifi.remote.input.http.enabled= > nifi.remote.input.http.transaction.ttl=30 sec > nifi.remote.contents.cache.expiration=30 secs > > # web properties # > > nifi.web.war.directory=./lib > #nifi.web.http.host=mydomain.com > #nifi.web.http.port=8080 > nifi.web.http.network.interface.default= > nifi.web.https.host=mydomain.com > nifi.web.https.port=9443 > nifi.web.https.network.interface.default= > nifi.web.jetty.working.directory=./work/jetty > nifi.web.jetty.threads=200 > nifi.web.max.header.size=16 KB > > > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: [EXT] sslcontext certs
The flow.xml.gz file which defines the flow does contain these paths. You could automate the flow change via Puppet (keeping in mind that other values will change as you modify NiFi flows via the canvas), or you could define these values as parameters [1] within NiFi and have each of the components reference those parameters in the path properties. [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 15, 2020, at 7:45 AM, Michael Di Domenico > wrote: > > On Wed, Oct 14, 2020 at 3:59 PM Nathan Gough wrote: >> >> Is there a reason each ListenHTTP has a unique SSLContextService if they're >> all using the same certificates? >> >> If it were me, I'd use a single shared SSLContextService, and when I needed >> to update the certificate in the keystore/truststore, I would change it on >> disk by renaming the old file and putting the new file in place with the >> original name. Now NiFi and the context service refers to the updated >> certificates and no NiFi configuration changed. Does this work for you? > > Possibly. Its likely the sslcontext documentation wasn't clear when i > read it and didn't realize i could do this. > > and yes i came to same conclusion about symlinking the keystores on > the local filesystem, which should also work. > > ideally, these parameters would be managed via some xml file that i > could have puppet control. so when the certs change, puppet can push > out the changes and update everything. yes i can do this with a > symlink, but it's not my preferred method to declare the resources and > changes.
Re: Securing MiNiFi
MiNiFi’s flow definition is persisted in the config.yml file. You should examine that file for the plaintext URL below and update it with the HTTPS URL (protocol & port). That will indicate that it should connect over TLS. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 17, 2020, at 4:06 AM, scotty wrote: > > Ok, I found some references online and got the proper permissions setup for > MiNiFi through the NiFi UI. > > Now nifi-user.log shows MiNiFi authenticating with no more authorization > errors, but minifi-app.log shows that it's still trying to communicate > unsecured: > > nifi.remote.client.http.HttpClient Penalizing a peer > Peer[url=http://example.com:8080/nifi-api] due to java.net.ConnectException: > Connection refused. > > nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate > with. > > nifi.remote.client.PeerSelector > org.apache.nifi.remote.client.PeerSelector@7cd5363e Unable to refresh Remote > Group's peers due to Remote instance of NiFi is not configured to allow > HTTP site-to-site communications. > > SiteToSiteRestApiClient Failed to create transaction for > http://example.com:8080/nifi-api/data-transfer/input-ports/0190d471-0175-1000-f6dd-2c8caa09aa24/transactions > > java.net.ConnectException: Connection refused. > > > I see a reference online to "the HTTPClient [relying] on an SSL Context > Service," but I don't see that processor available (NiFi 1.9.1). > > I also see a reference online to configuring the MiNiFi bootstrap.config > instead of using an SSL Context Service; however, my bootstrap.config has no > 'security properties' section as described: > https://docs.cloudera.com/cem/1.2.0/installation/topics/cem-configure-agent-tls.html > > Any further guidance appreciated. > > > > > > > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Securing MiNiFi
You can use the toolkit to generate another “server/node” certificate just as you did for the NiFi instance (do this from the same toolkit instance & directory to ensure both are signed by the same CA) and this will generate both the keystore and truststore JKS to use for MiNiFi. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 11, 2020, at 12:01 AM, scotty wrote: > > I have a working, unsecured, standalone NiFi setup with a remote MiNiFi > instance. Now I want to secure MiNiFi, and I'm trying to figure out what to > put in the security section of the config.yml. > I'm using the NiFi toolkit. > > From what I've been able to find, my current understanding is that I would > use the the same truststore information that is in the nifi.properties file, > but that i would have a different keystore. > > Do I generate a standalone client certificate for my MiNiFi instance and > convert that PCKS12 keystore to a jks keystore - using .password file for > the keystore password? > > > Thanks. > > > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
[ANNOUNCE] Apache NiFi CVE-2020-9486, CVE-2020-9487, CVE-2020-9491, CVE-2020-13940
Apache NiFi PMC would like to announce the discovery and resolution of CVE-2020-9486, CVE-2020-9487, CVE-2020-9491, and CVE-2020-13940. These issues have been resolved and a new version of the Apache NiFi project was released in accordance with the Apache Release Process. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Fixed in Apache NiFi 1.12.0 (Released: August 18, 2020) CVE-2020-9486: Apache NiFi information disclosure in logs Severity: Important Versions Affected: Apache NiFi 1.10.0 - 1.11.4 Description: The NiFi stateless execution engine produced log output which included sensitive property values. When a flow was triggered, the flow definition configuration JSON was printed, potentially containing sensitive values in plaintext. Mitigation: Implemented Argon2 secure hashing to provide a deterministic loggable value which does not reveal the sensitive value. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Andy LoPresto and Pierre Villard. CVE-2020-9487: Apache NiFi denial of service Severity: Important Versions Affected: Apache NiFi 1.0.0 - 1.11.4 Description: The NiFi download token (one-time password) mechanism used a fixed cache size and did not authenticate a request to create a download token, only when attempting to use the token to access the content. An unauthenticated user could repeatedly request download tokens, preventing legitimate users from requesting download tokens. Mitigation: Disabled anonymous authentication, implemented a multi-indexed cache, and limited token creation requests to one concurrent request per user. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Dennis Detering (IT Security Consultant at Spike Reply). CVE-2020-9491: Apache NiFi use of weak TLS protocols Severity: Critical Versions Affected: Apache NiFi 1.2.0 - 1.11.4 Description: The NiFi UI and API were protected by mandating TLS v1.2, as well as listening connections established by processors like ListenHTTP, HandleHttpRequest, etc. However intracluster communication such as cluster request replication, Site-to-Site, and load balanced queues continued to support TLS v1.0 or v1.1. Mitigation: Refactored disparate internal SSL and TLS code, reducing exposure for extension and framework developers to low-level primitives. Added support for TLS v1.3 on supporting JVMs. Restricted all incoming TLS communications to TLS v1.2+. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Juan Carlos Sequeiros and Andy LoPresto. CVE-2020-13940: Apache NiFi information disclosure by XXE Severity: Low Versions Affected: Apache NiFi 1.0.0 - 1.11.4 Description: The notification service manager and various policy authorizer and user group provider objects allowed trusted administrators to inadvertently configure a potentially malicious XML file. The XML file has the ability to make external calls to services (via XXE). Mitigation: An XML validator was introduced to prevent malicious code from being parsed and executed. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Matt Burgess and Andy LoPresto. For more information: https://nifi.apache.org/security.html <https://nifi.apache.org/security.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
Re: WebSocket Service - Using Trusted Certificates
I think the word “trusted” is doing a lot of work here. As it stands, only certificates that are either explicitly present or signed by a certificate present in the corresponding truststore will be accepted. If the certificate is self-signed, all that means is that an external entity (a certificate authority or CA) did not evaluate the identity & ownership of the certificate and sign it. So any certificate (self-signed or not) is still required to be “trusted” by the truststore for the connection to work. If you mean you want it to accept “any certificate signed by a generally accepted CA, you can rely on a generic truststore. Your OS, browser(s), and even Java come with these truststores pre-populated with the public certificates of the commercial and government CAs (what allows your computer to connect to and verify a generic internet site out of the box). The Java Virtual Machine (JVM) from the JRE or JDK will contain a JKS truststore called “cacerts” with the default password “changeit”. The location will vary slightly depending on the version of Java you’re using, but look inside your Java home directory for "jre/lib/security/cacerts”. Also, is there a reason you’re using web sockets between two NiFi instances? The NiFi Site-to-site protocol [1] offers a number of advantages. [1] https://medium.com/@abdelkrim.hadjidj/hub-and-spoke-architectures-with-nifi-site-to-site-communications-at-any-level-a-nifi-1-10-a8702f77c66e Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 10, 2020, at 12:32 AM, Madhan Vishwas > wrote: > > Hi All, > I am using WebSocket for communication between two independently running > instances of NiFi. > SSLContextService is being used for Secure Communication(WSS). > Everything works fine and is tested with Self signed certificates. > However, I would like to make sure that the communication works only with > trusted Certificates. Is there some way this can be ensured ? > Please advise. > Thanks in advance. > Madhan.
Re: Jks password migration issue
Hi Sanjeet,
If the root encryption key used in bootstrap.conf is identical, the encrypted
representation of the password should be reusable. Ensure you copied the entire
string (it consists of an IV encoded in Base64, || as a delimiter, and then the
actual cipher text (the encrypted password) also encoded in Base64.
You can also use the Encrypt-Config Toolkit [1] to perform a migration
operation if you prefer.
The first error you are encountering is likely because the complete encrypted
password was not copied successfully. The cipher text cannot be less than 17
characters long due to the cipher algorithm and minimum input length.
The second error is likely because of an incorrect encryption key being used.
The use of the correct key will result in proper padding detection and
successful decryption.
[1]
https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#encrypt_config_tool
<https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#encrypt_config_tool>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 27, 2020, at 6:52 AM, sanjeet rath wrote:
>
> Hi All,
>
> I am facing one ussue during my migration from 1.8 to 1.11.4
>
> My 1.8 env has jks key password is "xyz"
> The newly created 1.11.4 has jks password "abc".
>
> The encripyion key used in the bootstrap file is same for both the env.
>
>
> I have modified the pasaword of the 1.11.4 env's jks file using keytool to
> "xyz".
> However when i am changing its values("xyz") in nifi.properties &
> authoriser.xml in 1.11.4 env.I am getting below error.
>
> Error in creating authoriser bean ,
> IlligalArgumentException can't decrypt a ciphertext less than 17 characters .
>
> When i am copying the encripted values for jks password from 1.8 env's
> nifi.properties and replacing directly in nifi.properties& authoriser.xml of
> 1.11.4 env , as the encription key is same in both, then getting.
>
> java.security.UnrecoverableKeyException: Get Key failed: Given final block
> not properly padded. Such issues can arise if a bad key is used during
> decryption.
>
> Could you please help me how can i use my old jks password here.
> Thanks in advance.
> Sanjeet
Re: Connect NiFi to MongoDB(Atlas)
This was discussed in the Slack channel [1] and NIFI-7768 [2] was filed to address it. [1] https://apachenifi.slack.com/archives/C0L9VCD47/p1598402012117700 [2] https://issues.apache.org/jira/browse/NIFI-7768 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 25, 2020, at 5:35 PM, Ravi Teja Kondisetty > wrote: > > hello all, > > I am trying to connect Apache Nifi with Mongo DB. We have Mongo DB Atlas > instance and the connection string starts with mongodb+srv://. Now when I > give the same URL in NiFi, it throws a message saying that the Mongo DB URL > should start with mongodb://. Can someone throw some light on this topic pls? >
Re: Site to Site with multi-entry keystore?
All S2S authentication is performed using mutual authentication TLS, so there would not be a WWW-Authenticate request. You’re saying each endpoint has the appropriate keystore and truststore in place, and each trusts the other? You’ve also set the appropriate user policies (different from certificate trust; the user identity is proxied in the request itself and used for authorization)? Have you checked the logs/nifi-app.log and logs/nifi-user.log files to see what identity the incoming authentication request is presenting? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 25, 2020, at 8:09 AM, Pat White wrote: > > Hi Folks, > > Does S2S require use of a single entry keystore, or will multiple entries > work ok? > > I thought i saw documentation which stated S2S will only work with single > entry keystores, but i'm not able to find the reference. Trying to track down > a 401 Unauthorized error trying to do S2S with a peer cluster, without > receiving a followup credential request. > > Everything seems ok, policies allow both sides access, credentials are valid > and set both client and server Auth. Just appears as if the response to the > nifi-api/site-to-site GET doesn't trust the peer node and drops the > connection, without a follow up WWW-Authenticate request. However, i can't > find a reason for the reject. > > > patw
Re: SSL/LDAP Configuration
Ok to diagnose, look at the users.xml to see if there is a user matching that DN, and if so, it should have a UUID. Then in the authorizations.xml there should be policies defined in a hierarchical manner associating those users with a right on a specific resource (component/processor). If so, you can copy/paste as many as you want to define them. Again, this is not the ideal situation; most of this should be possible through the UI but I’m not sitting there to diagnose the issue. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 22, 2020, at 16:56, White, Daniel wrote: > > > Hi Andy, > > I tried removing users.xml and authorizations.xml but I’m still getting the > same error. > > Suspect it’s something to do with authorizers.xml, but I can’t see any issues > with it. > > I see this in the nifi-user.log : > > > Thanks > Dan > > From: Andy LoPresto > Sent: 23 August 2020 00:12 > To: users@nifi.apache.org > Subject: Re: SSL/LDAP Configuration > > CAUTION: This email originated from outside of the organisation. Do not click > links or open attachments unless you recognise the sender and know the > content is safe. > > Daniel, > > A couple options: > > The “easy way” is to shut down NiFi, delete “users.xml” and > “authorizations.xml” in the “conf/“ directory, and then restart NiFi. > Whatever user was specified as the IAI should have enough permissions to get > started now. > > Once you can access the main canvas, you’ll want to go into the global > policies dialog (global menu top right > policies) and give yourself the > specific view & modify permissions on the root process group. I understand > this manual effort is less than ideal, but the stages in which things are > defined has mandated this for now. > > I think the User Guide does a good job of explaining the theory here as well > as specific component steps (but doesn’t go soup to nuts on the process), so > I’d recommend that as well as the “end” (the last 3-4 steps) of the > Walkthrough guide section on securing NiFi. > > I’m on my phone so I don’t have all my usual resources available, but > hopefully this guides you in the right direction. If not, please let me know > and tomorrow I can provide more specific instructions. > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Aug 22, 2020, at 16:05, White, Daniel wrote: > > > Hi Andy, > > I’ve now managed to login to Nifi using my AD account but am getting the > following error : > > Insufficient Permissions – No applicable policies could be found. > > > > Any pointers would be gratefully received. > > Thanks > Dan > > From: Andy LoPresto > Sent: 03 August 2020 03:07 > To: users@nifi.apache.org > Subject: Re: SSL/LDAP Configuration > > CAUTION: This email originated from outside of the organisation. Do not click > links or open attachments unless you recognise the sender and know the > content is safe. > > Also, your authorizers.xml is not correct — you haven’t configured (or even > uncommented) the LDAP user group provider, so the specified user group > provider is the file users.xml, and you haven’t configured any initial > admins, so no users will be allowed to log in. Did you follow the steps in > the NiFi Admin Guide [3][4] for configuring this? Authentication and > authorization are decoupled in NiFi, and while you can use LDAP for both, > you’ll have to configure it for each. > > Also, your login-identity-providers.xml uses START_TLS as the authentication > strategy but does not specify any properties for the keystore or truststore, > which will be required. > > [3] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider > [4] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider > > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Aug 2, 2020, at 7:02 PM, Andy LoPresto wrote: > > Hi Daniel, > > Did you verify that the provided credentials are correct? There will be two > sets — the “manager” DN and password which are provided as configuration > values in the authorizers.xml file, and the individual user credentials > provided on each login attempt. The manager credentials allow NiFi to make an > au
Re: SSL/LDAP Configuration
Daniel, A couple options: The “easy way” is to shut down NiFi, delete “users.xml” and “authorizations.xml” in the “conf/“ directory, and then restart NiFi. Whatever user was specified as the IAI should have enough permissions to get started now. Once you can access the main canvas, you’ll want to go into the global policies dialog (global menu top right > policies) and give yourself the specific view & modify permissions on the root process group. I understand this manual effort is less than ideal, but the stages in which things are defined has mandated this for now. I think the User Guide does a good job of explaining the theory here as well as specific component steps (but doesn’t go soup to nuts on the process), so I’d recommend that as well as the “end” (the last 3-4 steps) of the Walkthrough guide section on securing NiFi. I’m on my phone so I don’t have all my usual resources available, but hopefully this guides you in the right direction. If not, please let me know and tomorrow I can provide more specific instructions. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 22, 2020, at 16:05, White, Daniel wrote: > > > Hi Andy, > > I’ve now managed to login to Nifi using my AD account but am getting the > following error : > > Insufficient Permissions – No applicable policies could be found. > > > > Any pointers would be gratefully received. > > Thanks > Dan > > From: Andy LoPresto > Sent: 03 August 2020 03:07 > To: users@nifi.apache.org > Subject: Re: SSL/LDAP Configuration > > CAUTION: This email originated from outside of the organisation. Do not click > links or open attachments unless you recognise the sender and know the > content is safe. > > Also, your authorizers.xml is not correct — you haven’t configured (or even > uncommented) the LDAP user group provider, so the specified user group > provider is the file users.xml, and you haven’t configured any initial > admins, so no users will be allowed to log in. Did you follow the steps in > the NiFi Admin Guide [3][4] for configuring this? Authentication and > authorization are decoupled in NiFi, and while you can use LDAP for both, > you’ll have to configure it for each. > > Also, your login-identity-providers.xml uses START_TLS as the authentication > strategy but does not specify any properties for the keystore or truststore, > which will be required. > > [3] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider > [4] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider > > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Aug 2, 2020, at 7:02 PM, Andy LoPresto wrote: > > Hi Daniel, > > Did you verify that the provided credentials are correct? There will be two > sets — the “manager” DN and password which are provided as configuration > values in the authorizers.xml file, and the individual user credentials > provided on each login attempt. The manager credentials allow NiFi to make an > authenticated request to the LDAP service, and the request itself contains > the user’s credentials. > > You can verify these credentials by using the ldapsearch [1][2] tool from one > of the machines where NiFi is installed. This allows you to verify TLS, > ports, network reachability, and the correctness of the credentials > themselves. > > Something like: > > $ ldapsearch -x -b “dc=,dc=com" -H ldap:// -D > "cn=admin,dc=,dc=com" -W > > That will conduct a general search using the account provided by -D, and > prompt for the password with -W. You can also switch out the account in -D > for the specific user you’re trying to log in as to verify those credentials. > > [1] > https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811 > [2] https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/ > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Aug 2, 2020, at 1:11 PM, White, Daniel wrote: > > Confidential > > Hi All, > > Looking for some assistance with setting up SSL/LDAP to enable user admin > within Nifi. > > I’ve setup and configured my non-prod environment but am having issue login > in : > > Unable to validate the supplied credentials. Please cont
Re: Flow.xml.gz gets overwritten with Empty Flow.xml
Hi John, I’m glad you found a documented workaround which solved your problem. While we try to build NiFi to operate at high scale, I don’t think it was ever anticipated that a flow definition would be 70 GB. Gzip compression can often achieve > 10:1 ratio for structured text data like XML, so that could actually be ~ 1 TB of uncompressed data. The Encrypt-Config Toolkit usually runs with 8 GB of heap space, far short of what is necessary to hold the deserialized data in memory to perform the encryption/decryption process for a flow of that size. There is an open Jira [1] which captures some of these issues and will direct resolution & improvements, but I am curious to gather more information about your scenario to hopefully provide the best solution. Usually when we encounter flow definitions of more than 100 MB, there are a few causes: * A high number of components on the canvas (10k+ processors) * A high number of saved templates Some NiFi instances are deployed for multitenant access and “legitimately” need hundreds or thousands of processors in many flows. However, we have often seen duplication of identical or near-identical flows due to “copy/paste” rather than parameterizing a single property value, which leads to significant opportunities for flow refactoring and deduplication. An analogy in writing code would be writing a new method to print “Hello Andy”, “Hello John”, “Hello Yolanda”, etc. rather than a single method which accepts a name and prints “Hello ”. Reducing these duplicate flows within the definition will reduce the size. Templates aren’t constantly visible and thus can grow over time, adding sizable impact to the flow definition even if no longer necessary. With the introduction of the NiFi Registry, templates have been deprecated, and cleaning up unused templates via the Global Menu > Templates option will drastically reduce the size of the flow definition. As NiFi has become more suited for elastic scaling and containerization, we’ve also seen better success with orchestrated containerized deployments on infrastructure abstractions like Kubernetes, which allows “one (or a logical group of) flow per cluster” and provides resource isolation & contention management, easier authorization management, independent scaling & monitoring, etc. If your flow definition is 70 GB, you may be very interested in pursuing this approach. [1] https://issues.apache.org/jira/browse/NIFI-6999 <https://issues.apache.org/jira/browse/NIFI-6999> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 21, 2020, at 8:09 PM, jgunvaldson wrote: > > Just FYI, I found the answer > > https://community.cloudera.com/t5/Support-Questions/NiFi-Toolkit-Can-t-start-NiFi-with-a-75MB-gzipped-flow-xml/td-p/288712 > > Exact same problem with proper solution > > Fixed… > > >> On Aug 21, 2020, at 3:48 PM, jgunvaldson wrote: >> >> We have a production instance of NIFI (1.9.0.3.4.1.1-4 built 05/01/2019 >> 02:15:30 UTC Tagged nifi-1.9.0-RC2) with an unusual symptom. >> >> We know, that on a new instance of NIFI, the canvas comes up empty (result >> of a new flow.xml). Developers then start building process >> groups and more from this start. >> >> What we are experiencing (never seen this before) - is that on restart (via >> startup log file entries verified) of NIFI using AMBARI (or other means), >> NIFI is overwriting flow.xml which is about 70 GB, with new empty flow.xml? >> Each time we copy >> back the original Flow.xml and restart (single node cluster) - NIFI >> overwrites with an empty Flow.xml >> >> Possibly related to toolkit and encryption of flow.xml.gz >> >> Obviously we are down until we can solve this >> >> Any ideas? >> >> John >
Re: NiFi 1.12.0 - KeyStores with multiple certificates are not supported
Hi Josef and Kotaro,
Thanks for identifying this scenario. I am away from the office for a bit but
will try to review Kotaro’s changes in the linked PR. The regression is within
Jetty’s code, and requires a new API to be invoked. NiFi does not have an
existing method to configure a specific key to use within the keystore, and
thus has always encouraged the use of a keystore with a single certificate and
key (PrivateKeyEntry).
However, I will note that the initial scenario described by Josef seems to use
a wildcard certificate, and this is explicitly mentioned in the documentation
as not supported and discouraged [1].
> Wildcard certificates (i.e. two nodes node1.nifi.apache.org and
> node2.nifi.apache.org being assigned the same certificate with a CN or SAN
> entry of *.nifi.apache.org) are not officially supported and not recommended.
> There are numerous disadvantages to using wildcard certificates, and a
> cluster working with wildcard certificates has occurred in previous versions
> out of lucky accidents, not intentional support. Wildcard SAN entries are
> acceptable if each cert maintains an additional unique SAN entry and CN entry.
I understand the challenges around automating key and certificate management
and regenerating/expiring certificates appropriately. The TLS Toolkit exists to
assist with this process, and there are ongoing improvements being made.
However, fully supporting wildcard certificates would require substantial
refactoring in the core framework and is not planned for any immediate
attention.
[1]
https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#wildcard_certificates
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 19, 2020, at 11:13 AM, Kotaro Terada wrote:
>
> Hi Josef and teams,
>
> I encountered the same problem, and I have created a patch to fix it [1].
>
> I guess the only way to fix the problem is to apply the patch and rebuild
> NiFi, since the current implementation unfortunately doesn't seem to support
> keystores with multiple certificates. Could someone please give support to
> review the PR and proceed to fix it?
>
> [1] https://issues.apache.org/jira/browse/NIFI-7730
> <https://issues.apache.org/jira/browse/NIFI-7730>
>
> Thanks,
> Kotaro
>
>
> On Thu, Aug 20, 2020 at 12:51 AM <mailto:josef.zahn...@swisscom.com>> wrote:
> Hi guys
>
>
>
> As we are waiting for some fixed bugs in NiFI 1.12.0, we upgraded today from
> 1.11.4 to the newest version on one of our secured test single VM instances.
> However, NiFi crashed during startup, error message below. It tells us that
> KeyStores with multiple certificates are not supported. As you know we have
> to use two keystores (keystore & truststore):
>
> Keystore with PrivateKey and Signed Cert -> only one Cert, the one belongs to
> the PrivateKey (picture far below)
> Truststore Keystore with CA Certs -> Multiple CA certs as we have imported
> the cacerts from linux
>
>
> I see two potential issues now, but I didn’t found the time to execute
> further tests.
>
>
>
> We don’t have multiple certs in the keystore with the privateKey as you can
> see in the picture far below, but of course we have SAN (Subject Alternative
> Names) as we have ton’s of NiFi instances running and it’s more than annoying
> to configure/generate a keypair for each instance. So the workaround was to
> insert all our NiFi instances as SAN and that way we were able to use one
> single keystore for all our NiFi instances (some of them are even clustered,
> some not). However my assumption is that the mentioned workaround potentially
> breaks now NiFi, this was working until NiFi 1.11.4. We know from security
> perspective the workaround is/was not ideal, but we don’t have the manpower
> to generate manually that many certs every 1-2 years when the certs are
> expiring and it’s anyway completely separated from public networks.
>
>
>
> In the truststore we have multiple certs, but that’s very common that you use
> eg. Linux cacerts as template for that.
>
>
>
> So to sum up, we can’t start NiFi anymore after the upgrade – any thoughts
> how to fix the issue with the keystores? Or shall I open a bugticket on Jira?
>
>
>
> Cheers Josef
>
>
>
> 2020-08-19 16:23:43,334 INFO [main] o.e.jetty.server.handler.ContextHandler
> Started
> o.e.j.w.WebAppContext@2a4f5433{nifi-error,/,file:///opt/nifi-1.12.0/work/jetty/nifi-web-error-1.12.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.12.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.12.0.war}
>
> 2020-08-19 16:23:43,346 INFO
Re: Nifi takes too long to start(~ 30 minutes)
Mohit, I’m a little confused by some of the details you report. Initially you said there were ~100 flows with 25 processors each (2500); now there are 25,000. If you examine the logs, you note that “reaching the election process” takes the majority of the time — what messages are being printed in the log before the election process starts? Have you taken thread dumps at regular intervals during this time to see where the time is being spent? This could be caused by the cost of fingerprinting such a large flow during cluster startup. Are there any custom processors in your NiFi instance? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 16, 2020, at 10:28 PM, Mohit Jain wrote: > > Hi Pierre, > > No, the election process takes only a minute or two max, reaching the > election process takes a lot of time. > There are around 25k stopped/invalid components in the UI, I marked all of > them as disabled hoping it would resolve the issue but no improvement in the > startup time. > > On a similar note, NiFi Rest api response is also quite slow when there are a > lot of components. Would it improve by disabling the components which are not > in use? Or is there something else that could have been causing the issue? > > Thanks, > Mohit > > On Thu, Aug 13, 2020 at 9:49 PM Pierre Villard <mailto:pierre.villard...@gmail.com>> wrote: > Hi, > > I'm surprised this is something you observe in the election process part. > I've constantly seen quick startup times even with thousands of components in > the flow. > I'd look into the logs and maybe turn on some debug logs to find out what's > going on. > > Pierre > > Le jeu. 13 août 2020 à 16:33, Joe Witt <mailto:joe.w...@gmail.com>> a écrit : > Mohit, > > You almost certainly want to take that same flow and setup a cluster on a > more recent version of NiFi to compare startup times. For flows with > thousands of components there are important improvements which have occurred > in the past year and a half. > > Startup time, user perceived behavior in the UI on continuous operations, > etc.. have been improved. Further you can now hot load new versions of nars > which should reduce the need to restart. > > We also will have 1.12 out hopefully within days so that could be interesting > for you as well. > > Thanks > > On Thu, Aug 13, 2020 at 7:18 AM Mohit Jain <mailto:mo...@open-insights.com>> wrote: > Hi Team, > > I am using a single node NiFi 1.9.0 cluster. It takes more than 30 minutes to > start each time it is restarted. There are more than 100 flows on the NiFi UI > with an average of 25 processors per flow. It takes around 25-30 minutes to > reach the cluster election process after which it gets started in a minute. > > Is this an expected behaviour that startup time is directly proportional to > the number of processors in the Canvas? Or is there a way to reduce the NiFi > startup time? > > Any leads would be appreciated. > > Thanks, > Mohit
Re: FetchSFTP: Rename file on move
You can also use an UpdateAttribute processor to change the “filename” attribute, which is what any “file persistence” processor (PutSFTP, PutFile, etc.) will use when writing the file out. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 11, 2020, at 3:47 PM, Joe Witt wrote: > > Jairo > > You can use a PutSFTP after Fetch to place it where you want. > > Thanks > > On Tue, Aug 11, 2020 at 3:16 PM Jairo Henao <mailto:jairohenaoro...@gmail.com>> wrote: > Hi community, > > Is there a way to rename a file before moving it with FetchSFTP? > > After processing a file, I need to move it to a folder and add a timestamp > suffix to it. The file in the source always has the same name, but I need > that when moving it they are not overwritten. > > Any ideas or is it necessary to request a modification to the processor? > > Thanks > > -- > Jairo Henao > @jairohenaorojas >
Re: Data Provenance Stops Working
Shawn, I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line. 1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes <https://cwiki.apache.org/confluence/display/NIFI/Release+Notes> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 10, 2020, at 10:00 AM, Shawn Weeks wrote: > > I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper > and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For > some reason after a period of time Data Provenance goes blank, old records > are no longer queryable and new data provenance doesn’t appear to get > written. No Lucene or other exceptions are logged and restarting the node > causes data provenance to go back to being written however old data > provenance does not re-appear. No exceptions appear when querying data > provenance. All tests have been run as the initial admin user and roles and > permissions appear to be correct. I’ve also checked selinux audit logs to see > if something is being blocked but nothing appears. > > WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 > seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, > compress on rollover, and don’t sync always. > > Thanks > Shawn Weeks
Re: Get all available variables in the InvokeScriptedProcessor
Those variables are available to be referenced via Expression Language in the
flowfile attributes. They are not intended for direct programmatic access via
code, so you don’t need to address them directly in your Groovy code.
If you need to populate specific values at configuration time, you can define
dynamic properties on the processor config and reference those directly in code
(see any existing processor source for examples).
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 9, 2020, at 10:40 PM, Saloni Udani wrote:
>
> Thanks Andy.
>
> By variables I meant NiFi process group variables.
>
>
> On Sat, Aug 8, 2020 at 12:39 AM Andy LoPresto <mailto:alopre...@apache.org>> wrote:
> I think we need additional clarification on what you mean by “variables”. If
> you are referring to actual Groovy variables, you can enumerate them using
> the binding available in the context (see below). If you mean the attributes
> available on a flowfile, you can access them similarly.
>
> Find all variables starting with prefix:
>
> def varsStartingWithABC = this.binding.variables.findAll { k,v ->
> k.startsWith(“a.b.c”) }
>
> Find all attributes starting with prefix:
>
> def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v ->
> k.startsWith(“a.b.c”) }
>
>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Aug 7, 2020, at 2:11 AM, Saloni Udani > <mailto:saloniudani.t...@gmail.com>> wrote:
>>
>> Hi,
>> We use NiFi 1.5.0.
>> Our use case is to get particular key pattern variables (key and value) in
>> the groovy InvokeScriptedProcessor. E.g I want all variables whose key
>> starts with "a.b.c". By this I can write a generic logic on certain
>> categories of variables for further use.
>>
>> Is there a way programmatically to get all variables with a certain key
>> pattern? Or for that matter is there a way programmatically to get all
>> available variables Map ? In NiFi 1.5.0 or further versions.
>>
>>
>> Thanks
>> Saloni Udani
>
Re: External Access using InvokeHTTP_Test processor and StandardSSLContextService
Hi Valentina, I am not sure why it would have worked on a different NiFi 1.10 instance previously unless the remote host introduced TLS/HSTS during that time and that’s what you’re encountering. If you are connecting to an HTTPS endpoint, you will need to provide an SSLContextService with a truststore containing either the exact certificate or one of the signing certificates for the remote endpoint. If it’s a commonly-accessible commercially-signed endpoint, use the cacerts truststore as referenced in this thread. If it’s an internal endpoint or you want to be more restrictive, obtain the public certificate of that endpoint and manually create a truststore containing that cert. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 7, 2020, at 1:14 AM, Valentina Ivanova wrote: > > Hello everyone! > > I am facing the same problem using InvokeHTTP 1.11.0 configured with the > default values (except for the Remote URL field): > sun.security.validator.ValidatorException: PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: unable to find > valid certification path to requested target > > The SSL Context Service property in the processor is not configured at all. > Previously the same workflow worked on another machine with NiFi 1.10. > > Many thanks in advance for your help! > > Valentina > > > > From: Andy LoPresto > Sent: Thursday, 6 August 2020 18:01 > To: users@nifi.apache.org > Subject: Re: External Access using InvokeHTTP_Test processor and > StandardSSLContextService > > This is not a JVM issue. Josef is correct that the external site you are > trying to communicate with is presenting a certificate which the configured > NiFi truststore has no way to verify (it can’t find the “path” between cert X > and any of its signing certs to one already known by NiFi). > > The solution is to acquire the external server public certificate or a > signing certificate and import it directly into the truststore you have > configured for the InvokeHTTP processor via the SSLStandardContextService, or > reference a different truststore which already has the certificate present. > > If you are pointing at the same truststore you use for NiFi as an > application, it’s not suggested to import the external cert directly, as this > will have an impact on authentication mechanisms. Rather, use a new > truststore explicitly for this use case, or reference the JRE provided > “cacerts” truststore [1] directly from the SSLContextService (default > password is “changeit” and it comes with many commercial/public CAs imported > automatically, just like your browser or OS). > > [1] https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html > <https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html> > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Aug 6, 2020, at 6:32 AM, Jorge Machado > <mailto:jom...@me.com>> wrote: >> >> Hi Dan, >> >> Seems like this is a jvm issue. >> >> Try this: >> https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html >> >> <https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html> >> Diagnosis >> Use SSL Poke to verify connectivity >> Try the Java class SSLPoke to see if your truststore contains the right >> certificates. This will let you connect to a SSL service, send a byte of >> input, and watch the output. >> Download SSLPoke.class >> <https://confluence.atlassian.com/kb/files/779355358/779355357/1/1441897666313/SSLPoke.class> >> Execute the class as per the below, changing the URL and port appropriately. >> Take care that you are running the same Java as what Confluence is running >> with. If you used the installer you will need to use >> /jre/java >> $JAVA_HOME/bin/java SSLPoke jira.example.com <http://jira.example.com/> 443 >> A mail server may be mail.example.com <http://mail.example.com/> 465 . >> >> The jira.example.com <http://jira.example.com/> is your custom site. Add the >> CA Certs >> >> >> >>> On 6. Aug 2020, at 14:08, >> <mailto:josef.zahn...@swisscom.com>> >> <mailto:josef.zahn...@swisscom.com>> wrote: >>> >>> It tells you most probably that the CA cert from the remote HTTP
Re: Get all available variables in the InvokeScriptedProcessor
I think we need additional clarification on what you mean by “variables”. If
you are referring to actual Groovy variables, you can enumerate them using the
binding available in the context (see below). If you mean the attributes
available on a flowfile, you can access them similarly.
Find all variables starting with prefix:
def varsStartingWithABC = this.binding.variables.findAll { k,v ->
k.startsWith(“a.b.c”) }
Find all attributes starting with prefix:
def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v ->
k.startsWith(“a.b.c”) }
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 7, 2020, at 2:11 AM, Saloni Udani wrote:
>
> Hi,
> We use NiFi 1.5.0.
> Our use case is to get particular key pattern variables (key and value) in
> the groovy InvokeScriptedProcessor. E.g I want all variables whose key starts
> with "a.b.c". By this I can write a generic logic on certain categories of
> variables for further use.
>
> Is there a way programmatically to get all variables with a certain key
> pattern? Or for that matter is there a way programmatically to get all
> available variables Map ? In NiFi 1.5.0 or further versions.
>
>
> Thanks
> Saloni Udani
Re: External Access using InvokeHTTP_Test processor and StandardSSLContextService
This is not a JVM issue. Josef is correct that the external site you are trying to communicate with is presenting a certificate which the configured NiFi truststore has no way to verify (it can’t find the “path” between cert X and any of its signing certs to one already known by NiFi). The solution is to acquire the external server public certificate or a signing certificate and import it directly into the truststore you have configured for the InvokeHTTP processor via the SSLStandardContextService, or reference a different truststore which already has the certificate present. If you are pointing at the same truststore you use for NiFi as an application, it’s not suggested to import the external cert directly, as this will have an impact on authentication mechanisms. Rather, use a new truststore explicitly for this use case, or reference the JRE provided “cacerts” truststore [1] directly from the SSLContextService (default password is “changeit” and it comes with many commercial/public CAs imported automatically, just like your browser or OS). [1] https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html <https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 6, 2020, at 6:32 AM, Jorge Machado wrote: > > Hi Dan, > > Seems like this is a jvm issue. > > Try this: > https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html > > <https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html> > Diagnosis > Use SSL Poke to verify connectivity > Try the Java class SSLPoke to see if your truststore contains the right > certificates. This will let you connect to a SSL service, send a byte of > input, and watch the output. > Download SSLPoke.class > <https://confluence.atlassian.com/kb/files/779355358/779355357/1/1441897666313/SSLPoke.class> > Execute the class as per the below, changing the URL and port appropriately. > Take care that you are running the same Java as what Confluence is running > with. If you used the installer you will need to use > /jre/java > $JAVA_HOME/bin/java SSLPoke jira.example.com <http://jira.example.com/> 443 > A mail server may be mail.example.com <http://mail.example.com/> 465 . > > The jira.example.com <http://jira.example.com/> is your custom site. Add the > CA Certs > > > >> On 6. Aug 2020, at 14:08, > <mailto:josef.zahn...@swisscom.com>> > <mailto:josef.zahn...@swisscom.com>> wrote: >> >> It tells you most probably that the CA cert from the remote HTTPS server >> hasn’t been found in the truststore you’ve defined to access the site. So >> please check again the CA cert and the truststore… >> >> Cheers Josef >> >> >> From: "White, Daniel" mailto:daniel.wh...@lgim.com>> >> Reply to: "users@nifi.apache.org <mailto:users@nifi.apache.org>" >> mailto:users@nifi.apache.org>> >> Date: Thursday, 6 August 2020 at 13:07 >> To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" >> mailto:users@nifi.apache.org>> >> Subject: External Access using InvokeHTTP_Test processor and >> StandardSSLContextService >> >> Confidential >> >> Hi All, >> >> We’ve setup the truststore from the NiFi processor. However we get the >> following error when trying to connect to an external HTTPS location >> >> The error I get is: PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: unable to find >> valid certification path to requested target >> >> Any ideas? Assume this is a cert issue on the Nifi server. >> >> Thanks >> >> Dan White >> Lead Technical Architect >> Legal & General Investment Management >> One Coleman Street, London, EC2R 5AA >> Tel: +44 203 124 4048 >> Mob: +44 7980 027 656 >> www.lgim.com <http://www.lgim.com/> >> >> This e-mail (and any attachments) may contain privileged and/or confidential >> information. If you are not the intended recipient please do not disclose, >> copy, distribute, disseminate or take any action in reliance on it. If you >> have received this message in error please reply and tell us and then delete >> it. Should you wish to communicate with us by e-mail we cannot guarantee the >> security of any data outside our own computer systems. >> >> Any information contained in t
Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"
Dan, Thanks for reporting this. Case-sensitivity in these kinds of things is important but it also seems like low-hanging fruit for us to at least detect & alert on when errors occur. “Failed to connect to external service X with principal n...@x.net <mailto:n...@x.net> …. Did you mean n...@x.net <mailto:n...@x.net>?” Or even potentially trying to do case-conversion internally as a fallback. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 5, 2020, at 2:50 PM, dan young wrote: > > On a related note, I noticed that the ACL are getting set, but also for each > znode under the /nifi, the Read ACL for world is being set. Is there a way > to have nifi only set with the sasl? > > zk: nifi1-5.X.net:2181(CONNECTED) 12] getAcl /nifi > 'sasl,'n...@x.net <mailto:n...@x.net> > : cdrwa > 'world,'anyone > : r > > On Wed, Aug 5, 2020 at 1:56 PM Mark Payne <mailto:marka...@hotmail.com>> wrote: > No worries, thanks for following up and letting us know! > > Thanks > -Mark > > >> On Aug 5, 2020, at 3:42 PM, dan young > <mailto:danoyo...@gmail.com>> wrote: >> >> Hello, >> >> Sorry for all the noise...doohwas due to the realm in the jaas.conf >> being lowercase...i'm a knucklehead... >> >> Dano >> >> On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende > <mailto:bbe...@gmail.com>> wrote: >> I don't see how this would relate to the problem, but shouldn't the ACL be >> set to "Creator" when Sasl/Kerberos is setup correctly? >> >> In addition to the nifi configs you showed, you would also need a jaas conf >> file specified in bootstrap.conf and in that file you would need the jaas >> entry for the ZK client. >> >> On Wed, Aug 5, 2020 at 3:02 PM dan young > <mailto:danoyo...@gmail.com>> wrote: >> Hello Mark, >> >> Attached is a dump from one of the nodesI replaced the domain related >> entries with X/x. I'm not sure if it's relevant or not, but I did notice >> that in the log there's entries "Looking for keys for n...@x.net >> <mailto:n...@x.net>" the x (domain) is lowercase whereas in the keytab >> file it's uppercase X. Also not sure if the Found unsupported keytype (1) >> is meaningful. Not that when I delete the znode in zookeeper=, at least the >> initial znode is created /nifi, but we never see the other typical suspect, >> i.e Coordinator, Primary, etc... >> >> Seems to be something stuck in Curator??? >> >> Regards. >> >> Dano >> >> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne > <mailto:marka...@hotmail.com>> wrote: >> Dan, >> >> Can you grab a thread dump and provide that? Specifically, the “main” thread >> is the important one with startup. The note that the role is already >> registered is normal. It probably could be changed to a DEBUG level, really. >> It should not be concerning. A thread dump, though, would show us exactly >> where it’s at. >> >> Thanks >> -Mark >> >> >>> On Aug 5, 2020, at 2:02 PM, dan young >> <mailto:danoyo...@gmail.com>> wrote: >>> >>> Hello, >>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled >>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck >>> in : >>> >>> 2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService >>> There is currently no Cluster Coordinator. This often happens upon restart >>> of NiFi >>> when running an embedded ZooKeeper. Will register this node to become the >>> active Cluster Coordinator and will attempt to connect to cluster again >>> 2020-08-05 17:10:18,907 INFO [main] >>> o.a.n.c.l.e.CuratorLeaderElectionManager >>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader >>> Election >>> for role 'Cluster Coordinator' but this role is already registered >>> >>> >>> >>> I've checked zookeeper and I can see that the /nifi znode has been created, >>> although empty, and the ACL seem to look correct >>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181/>(CONNECTED) 3] getAcl >>> /nifi >>> 'sasl,'n...@x.net <mailto:n...@x.net> >>> : cdrwa >>> 'world,'anyone >>> : r >>> >>> >>> relevant Nifi config settings >>> >>> nifi.properties: >>> >>> nifi.zo
Re: SSL/LDAP Configuration
Also, your authorizers.xml is not correct — you haven’t configured (or even uncommented) the LDAP user group provider, so the specified user group provider is the file users.xml, and you haven’t configured any initial admins, so no users will be allowed to log in. Did you follow the steps in the NiFi Admin Guide [3][4] for configuring this? Authentication and authorization are decoupled in NiFi, and while you can use LDAP for both, you’ll have to configure it for each. Also, your login-identity-providers.xml uses START_TLS as the authentication strategy but does not specify any properties for the keystore or truststore, which will be required. [3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider> [4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 2, 2020, at 7:02 PM, Andy LoPresto wrote: > > Hi Daniel, > > Did you verify that the provided credentials are correct? There will be two > sets — the “manager” DN and password which are provided as configuration > values in the authorizers.xml file, and the individual user credentials > provided on each login attempt. The manager credentials allow NiFi to make an > authenticated request to the LDAP service, and the request itself contains > the user’s credentials. > > You can verify these credentials by using the ldapsearch [1][2] tool from one > of the machines where NiFi is installed. This allows you to verify TLS, > ports, network reachability, and the correctness of the credentials > themselves. > > Something like: > > $ ldapsearch -x -b “dc=,dc=com" -H ldap:// -D > "cn=admin,dc=,dc=com" -W > > That will conduct a general search using the account provided by -D, and > prompt for the password with -W. You can also switch out the account in -D > for the specific user you’re trying to log in as to verify those credentials. > > [1] > https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811 > > <https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811> > [2] https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/ > <https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/> > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Aug 2, 2020, at 1:11 PM, White, Daniel > <mailto:daniel.wh...@lgim.com>> wrote: >> >> Confidential >> >> Hi All, >> >> Looking for some assistance with setting up SSL/LDAP to enable user admin >> within Nifi. >> >> I’ve setup and configured my non-prod environment but am having issue login >> in : >> >> Unable to validate the supplied credentials. Please contact the system >> administrator >> >> I’ve followed the config guide and am stuck as to what the issue could be. >> >> The steps I followed : >> >> Generate keys etc using tls-toolkit.sh >> Updated nifi.properties to set >> nifi.security.user.login.identity.provider=ldap-provider >> Modified login-identity-providers.xml (copy attached) >> Modified authorizers.xml (copy attached) >> >> Nifi starts and I can get to the login page, just unable to login (with >> error shown above). >> >> Any help will be very grateful. >> >> Thanks >> >> Dan White >> Lead Technical Architect >> Legal & General Investment Management >> One Coleman Street, London, EC2R 5AA >> Tel: +44 203 124 4048 >> Mob: +44 7980 027 656 >> www.lgim.com <http://www.lgim.com/> >> >> This e-mail (and any attachments) may contain privileged and/or confidential >> information. If you are not the intended recipient please do not disclose, >> copy, distribute, disseminate or take any action in reliance on it. If you >> have received this message in error please reply and tell us and then delete >> it. Should you wish to communicate with us by e-mail we cannot guarantee the >> security of any data outside our own computer systems. >> >> Any information contained in this m
Re: SSL/LDAP Configuration
Hi Daniel, Did you verify that the provided credentials are correct? There will be two sets — the “manager” DN and password which are provided as configuration values in the authorizers.xml file, and the individual user credentials provided on each login attempt. The manager credentials allow NiFi to make an authenticated request to the LDAP service, and the request itself contains the user’s credentials. You can verify these credentials by using the ldapsearch [1][2] tool from one of the machines where NiFi is installed. This allows you to verify TLS, ports, network reachability, and the correctness of the credentials themselves. Something like: $ ldapsearch -x -b “dc=,dc=com" -H ldap:// -D "cn=admin,dc=,dc=com" -W That will conduct a general search using the account provided by -D, and prompt for the password with -W. You can also switch out the account in -D for the specific user you’re trying to log in as to verify those credentials. [1] https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811 <https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811> [2] https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/ <https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 2, 2020, at 1:11 PM, White, Daniel wrote: > > Confidential > > Hi All, > > Looking for some assistance with setting up SSL/LDAP to enable user admin > within Nifi. > > I’ve setup and configured my non-prod environment but am having issue login > in : > > Unable to validate the supplied credentials. Please contact the system > administrator > > I’ve followed the config guide and am stuck as to what the issue could be. > > The steps I followed : > > Generate keys etc using tls-toolkit.sh > Updated nifi.properties to set > nifi.security.user.login.identity.provider=ldap-provider > Modified login-identity-providers.xml (copy attached) > Modified authorizers.xml (copy attached) > > Nifi starts and I can get to the login page, just unable to login (with error > shown above). > > Any help will be very grateful. > > Thanks > > Dan White > Lead Technical Architect > Legal & General Investment Management > One Coleman Street, London, EC2R 5AA > Tel: +44 203 124 4048 > Mob: +44 7980 027 656 > www.lgim.com <http://www.lgim.com/> > > This e-mail (and any attachments) may contain privileged and/or confidential > information. If you are not the intended recipient please do not disclose, > copy, distribute, disseminate or take any action in reliance on it. If you > have received this message in error please reply and tell us and then delete > it. Should you wish to communicate with us by e-mail we cannot guarantee the > security of any data outside our own computer systems. > > Any information contained in this message may be subject to applicable terms > and conditions and must not be construed as giving investment advice within > or outside the United Kingdom or Republic of Ireland. > > Telephone Conversations may be recorded for your protection and to ensure > quality of service > > Legal & General Investment Management Limited (no 2091894), LGIM Real Assets > (Operator) Limited (no 05522016), LGIM (International) Limited (no 7716001) > Legal & General Unit Trust Managers (no 1009418), GO ETF Solutions LLP > (OC329482) and LGIM Corporate Director Limited (no 7105051) are authorised > and regulated by the Financial Conduct Authority. All are registered in > England & Wales with a registered office at One Coleman Street, London, EC2R > 5AA > > Legal & General Assurance (Pensions Management) Limited (no 1006112) is > authorised by the Prudential Regulation Authority and regulated by the > Financial Conduct Authority and the Prudential Regulation Authority. It is > registered in England & Wales with a registered office at One Coleman Street, > London, EC2R 5AA. > > Legal & General Property Limited (no 2091897) is authorised and regulated by > the Financial Conduct Authority for insurance mediation activities. It is > registered in England & Wales with a registered office at One Coleman Street, > London, EC2R 5AA. > > LGIM Managers (Europe) Limited is authorised and regulated by the Central > Bank of Ireland (C173733). It is registered in the Republic of Ireland (no > 609677) with a registered office at 33/34 Sir John Rogerson's Quay, Dublin 2, > D02 XK09. > >
Re: Template not saving parameter context assignment
Parameter contexts were added after the use of templates was no longer recommended and substantially deprecated and NiFi Registry was selected as the managed flow definition version control and environment promotion system. There are future efforts to facilitate the export/import of flow segments directly to/from a NiFi instance without NiFi Registry but they are not released yet. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 28, 2020, at 2:00 PM, bsavard wrote: > > Hi, > > I have a flow that uses parameter contexts. I understand that the parameter > context definitions themselves are persisted to flow.xml.gz. > > However, when I save my flow as a template, delete the flow from the canvas, > then add the template back onto a blank canvas, all of the processor context > assignments I made are gone. When I download and search the template file > itself, there's no reference to the parameter context assignments. > > So, where are the assignments stored? And how would I go about deploying > this template to many servers without someone having to manually go back > into the flow on each server and adding the assignments all over the place? > We don't use NiFi Registry; we use a traditional Maven based deployment > cycle. > > TIA! > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Issue with Secured NiFi on kubernetes using Helm charts
Chris has a lot of good suggestions there. NiFi can accept certificates from any provider as long as they meet certain requirements (EKU, SAN, no wildcard, etc.). The toolkit was designed to make the process easier for people who could not obtain their certificates elsewhere. Maybe I am misunderstanding your statement, but I am curious why the toolkit can’t run on the node — if you don’t have Java available, how does NiFi itself run? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 23, 2020, at 12:35 AM, Chris Sampson wrote: > > My suggestion would be to run the apache/nifi-toolkit image as another Pod > within your k8s namespace and have it running as a TLS Server[1]. You'll > probably need to do that separately from your Helm chart (I'm not familiar > with Helm or this chart). > > Then connect to that from your NiFi instances as they start up, e.g. with an > init-container based on the same apache/nifi-toolkit image using the TLS > client function [1] to obtain the required TLS certificate files from the TLS > Server. You can use an emptyDir [2] volume to pass the files from the > init-container to the NiFi container within the Pod. > > If you run the TLS Server as a StatefulSet (or a Deployment) with a > Persistent Volume Claim that backed by an external volume within your cloud > provider (whatever the GKE equivalent is of AWS's EBS volumes), then the TLS > Server can be setup with its own Certificate Authority that persists between > Pod restarts and thus your NiFi certificates shouldn't become invalid over > time (if the TLS Server is restarted and generates a new CA, then subsequent > NiFi restarts would mean your NiFi cluster instances would no longer be able > to communicate with one another as they wouldn't trust one another's > certificates). > > > An alternative, if it's available in your k8s cluster, is to use something > like cert-manager [3] to provision certificates for your instances, then use > an init-container within the NiFi Pods to convert the PEM files to Java > Keystore or PKCS12 format as required by NiFi. > > > [1]: > https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#client-server > <https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#client-server> > [2]: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir > <https://kubernetes.io/docs/concepts/storage/volumes/#emptydir> > [3]: https://github.com/jetstack/cert-manager > <https://github.com/jetstack/cert-manager> > > > Chris Sampson > IT Consultant > chris.samp...@naimuri.com <mailto:chris.samp...@naimuri.com> > > > > On Thu, 23 Jul 2020 at 07:09, Atul Wankhade <mailto:atul.wankhad...@gmail.com>> wrote: > Thanks a lot Andy for your reply, it definitely helped pinpointing what is > going wrong. I tried simulating the same with the docker image from Apache > and generating the keystore/truststore files on the Docker host. For one node > NiFi it worked fine. The problem comes when I am trying the same on > Kubernetes. Nodes in GKE have Container optimized OS (no pkg installer) , so > it does not support using NiFi tls-toolkit as Java cannot be installed. Can > you please give some pointers/workaround on how to solve this issue with k8s? > Once the files are generated we can mount it using Host mount in the pod. > > Thanks again for your help :) > Atul > > On Tue, Jul 21, 2020 at 10:37 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Atul, > > I am not a Kubernetes/ingress expert, but that error is indicating that you > specified NiFi should be secure (i.e. use TLS/HTTPS) and yet there is no > keystore or truststore provided to the application, so it fails to start. > NiFi differs from some other applications in that you cannot configure > authentication and authorization without explicitly enabling and configuring > TLS for NiFi itself, not just delegating that data in transit encryption to > an external system (like a load balancer, proxy, or service mesh). > > I suggest you read the NiFi walkthrough for “Securing NiFi with TLS” [1] > which will provide some context around what the various requirements are, and > the Admin Guide [2] sections on authentication and authorization for more > background. > > [1] > https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-tls > > <https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-tls> > [2] > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_configuration > > <https://nifi.apache.org/docs/nifi-docs/html/
Re: Issue with Secured NiFi on kubernetes using Helm charts
Atul,
I am not a Kubernetes/ingress expert, but that error is indicating that you
specified NiFi should be secure (i.e. use TLS/HTTPS) and yet there is no
keystore or truststore provided to the application, so it fails to start. NiFi
differs from some other applications in that you cannot configure
authentication and authorization without explicitly enabling and configuring
TLS for NiFi itself, not just delegating that data in transit encryption to an
external system (like a load balancer, proxy, or service mesh).
I suggest you read the NiFi walkthrough for “Securing NiFi with TLS” [1] which
will provide some context around what the various requirements are, and the
Admin Guide [2] sections on authentication and authorization for more
background.
[1]
https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-tls
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_configuration
<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_configuration>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 20, 2020, at 11:58 PM, Atul Wankhade wrote:
>
> Hi All,
> I am trying to install NiFi with SSL on Kubernetes using Helm(cetic/nifi),
> Below is my values.yaml. I keep getting an error on NiFi containers as - Am I
> missing something?
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'clusterCoordinationProtocolSender' defined in class
> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
> to bean 'protocolSocketConfiguration' while setting constructor argument;
> nested exception is org.springframework.beans.factory.BeanCreationException:
> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
> threw exception on object creation; nested exception is
> java.io.FileNotFoundException: (No such file or directory)
>
> VALUES.YAML:
> ---
> # Number of nifi nodes
> replicaCount: 1
>
> ## Set default image, imageTag, and imagePullPolicy.
> ## ref: https://hub.docker.com/r/apache/nifi/
> <https://hub.docker.com/r/apache/nifi/>
> ##
> image:
> repository: apache/nifi
> tag: "1.11.4"
> pullPolicy: IfNotPresent
>
> ## Optionally specify an imagePullSecret.
> ## Secret must be manually created in the namespace.
> ## ref:
> https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
>
> <https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/>
> ##
> # pullSecret: myRegistrKeySecretName
>
> securityContext:
> runAsUser: 1000
> fsGroup: 1000
>
> sts:
> # Parallel podManagementPolicy for faster bootstrap and teardown. Default
> is OrderedReady.
> podManagementPolicy: Parallel
> AntiAffinity: soft
> hostPort: null
>
> ## Useful if using any custom secrets
> ## Pass in some secrets to use (if required)
> # secrets:
> # - name: myNifiSecret
> # keys:
> # - key1
> # - key2
> # mountPath: /opt/nifi/secret
>
> ## Useful if using any custom configmaps
> ## Pass in some configmaps to use (if required)
> # configmaps:
> # - name: myNifiConf
> # keys:
> # - myconf.conf
> # mountPath: /opt/nifi/custom-config
>
>
> properties:
> # use externalSecure for when inbound SSL is provided by nginx-ingress or
> other external mechanism
> externalSecure: true
> isNode: true
> httpPort: null
> httpsPort: 8443
> clusterPort: 6007
> clusterSecure: true
> needClientAuth: true
> provenanceStorage: "8 GB"
> siteToSite:
> secure: true
> port: 1
> authorizer: managed-authorizer
> # use properties.safetyValve to pass explicit 'key: value' pairs that
> overwrite other configuration
> safetyValve:
> #nifi.variable.registry.properties: "${NIFI_HOME}/example1.properties,
> ${NIFI_HOME}/example2.properties"
> nifi.web.http.network.interface.default: eth0
> # listen to loopback interface so "kubectl port-forward ..." works
> nifi.web.http.network.interface.lo: lo
>
> ## Include additional libraries in the Nifi containers by using the postStart
> handler
> ## ref:
> https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
>
> <https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/>
> # postStart: /opt/nifi/psql; wget -P /opt/nifi/psql
> https://jdbc.postgresql.org/download/postgresql-42.2.6.jar
> <https://jdbc.postgresql.org/download/postgresql-42.2.6.j
Re: Processor setup - Outbound Proxy Config
Hi Dan, There are a number of ways a NiFi instance can interact with a proxy service. Rather than deploying behind a reverse proxy or load balancer, it sounds like your use case is for an outgoing request to a specific external service to make use of a proxy. Without knowing what specific processor you’re using, it’s difficult to answer with any specificity. For example, using InvokeHTTP [1], you can configure the proxy settings directly in the processor properties, or configure a StandardProxyConfigurationService [2] controller service once and reference those same values in multiple processor instances. If you can provide more concrete information, we may have additional suggestions. Hopefully this helps. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.InvokeHTTP/index.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.11.4/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 10, 2020, at 8:07 AM, White, Daniel wrote: > > Confidential > > Hi All, > > The project I’m working on needs to configure a processor to connect to an > external data source via our corporate proxy. > > How do we configure the outbound proxy within Nifi? > > thanks > > Dan White > Lead Technical Architect > Legal & General Investment Management > One Coleman Street, London, EC2R 5AA > Tel: +44 203 124 4048 > Mob: +44 7980 027 656 > www.lgim.com <http://www.lgim.com/> > > This e-mail (and any attachments) may contain privileged and/or confidential > information. If you are not the intended recipient please do not disclose, > copy, distribute, disseminate or take any action in reliance on it. If you > have received this message in error please reply and tell us and then delete > it. Should you wish to communicate with us by e-mail we cannot guarantee the > security of any data outside our own computer systems. > > Any information contained in this message may be subject to applicable terms > and conditions and must not be construed as giving investment advice within > or outside the United Kingdom or Republic of Ireland. > > Telephone Conversations may be recorded for your protection and to ensure > quality of service > > Legal & General Investment Management Limited (no 2091894), LGIM Real Assets > (Operator) Limited (no 05522016), LGIM (International) Limited (no 7716001) > Legal & General Unit Trust Managers (no 1009418), GO ETF Solutions LLP > (OC329482) and LGIM Corporate Director Limited (no 7105051) are authorised > and regulated by the Financial Conduct Authority. All are registered in > England & Wales with a registered office at One Coleman Street, London, EC2R > 5AA > > Legal & General Assurance (Pensions Management) Limited (no 1006112) is > authorised by the Prudential Regulation Authority and regulated by the > Financial Conduct Authority and the Prudential Regulation Authority. It is > registered in England & Wales with a registered office at One Coleman Street, > London, EC2R 5AA. > > Legal & General Property Limited (no 2091897) is authorised and regulated by > the Financial Conduct Authority for insurance mediation activities. It is > registered in England & Wales with a registered office at One Coleman Street, > London, EC2R 5AA. > > LGIM Managers (Europe) Limited is authorised and regulated by the Central > Bank of Ireland (C173733). It is registered in the Republic of Ireland (no > 609677) with a registered office at 33/34 Sir John Rogerson's Quay, Dublin 2, > D02 XK09. > > Legal & General Group PLC, Registered Office One Coleman Street, London, EC2R > 5AA. > > Registered in England no: 1417162 > > This email has come from the internet and has been scanned for all > viruses and potentially offensive content by Messagelabs on behalf of Legal & > General
Re: PutFile set Last Modified Time without file.creationTime
To fix the date formatting specific error, you are correct that you need to use
the Expression Language functions toDate() [1] and format() [2] to convert
to/from plain strings to date objects. You are currently concatenating the two
date values (the year-month-day segment and the hour:minute:second segment),
then changing the delimiter from a space to a ’T’ (you can just do this
explicitly in the first step), then concatenating the timezone offset and
trying to convert this to a timestamp via a prescribed format, but the format
doesn’t match the input you have.
Please use the values below (I tested these against the current main branch
build, but nothing should have changed since prior releases):
To concatenate the string attributes into a parseable format and convert it to
a date object (internally represented as the number of milliseconds since the
epoch began at Jan 1, 1970 00:00:00 UTC):
${fileMetadata.6:append('T'):append(${fileMetadata.7:substringBefore('.')}):append('
'):append(${fileMetadata.8}):toDate("-MM-dd'T'HH:mm:ss Z”)}
To parse the result of the above into various timezones:
Local timezone: ${parsedTimestamp:format("-MM-dd'T'HH:mm:ss Z”)}
UTC timezone: ${parsedTimestamp:format("-MM-dd'T'HH:mm:ss Z", "UTC”)}
If you set the PutFile Last Modified Time to ${timestampUTCString} (or whatever
you name the attribute mentioned in Step 2 above), it will successfully set the
file’s timestamp when writing it out (06:14 in February in my timezone is equal
to 14:14 UTC):
/tmp ll timestamptest 15:28:55
total 0
drwxrwxrwx 14 alopresto wheel 448B Jul 2 15:29 ./
drwxrwxrwt 7 root wheel 224B Jul 2 15:28 ../
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
0eec229c-5658-4a86-b6ba-3fe507672bd4
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
113fb95e-5a10-48e4-ba9b-616909b68684
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
13fd2b13-fc8e-455d-8ca9-4afa2886a8e8
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
3228111c-476d-4cf6-a141-587270d821e2
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
397e7a21-944b-4a0c-a0d7-6150e10b385e
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
400313d8-9511-451a-ba40-6a37e7649906
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
46c587f6-06ee-463e-8e91-b432073aa98d
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
4a783b61-2304-44c6-9820-045e0cfaac52
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
a30a3e6c-e3ed-4180-9486-3de274116652
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
d4cdafc4-b5f3-4a18-9548-c7a5a2a3ea68
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
e6b94e07-9bd1-4fbc-aee2-27b687681849
-rw-r--r-- 1 alopresto wheel 0B Feb 14 06:14
f6802781-d820-4e18-b803-ceeaf5abee11
I’m not sure I understand your other concerns — ListFile and GetFile do not
accept incoming connections because they are designed to retrieve the list of
or explicit files from a particular file system location (e.g. you want to list
all the files that appear in
/some/location/where/another/process/puts/them/over/time as they appear). If
you have some other initial process to determine an absolute file path, you can
pass it to FetchFile as you’re doing.
You can also file a feature request Jira to also read the file metadata and
make it available as named attributes in the flowfile after reading the file,
as this seems like a useful behavior for you and others moving forward.
[1]
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#todate
<https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#todate>
[2]
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#format
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 2, 2020, at 6:32 AM, Valentina Ivanova wrote:
>
> Hello!
>
> I need to set Last Modified Time in PutFile however I cannot use
> file.creationTime as it is retrieved from either ListFile or GetFile.
>
> I am retrieving files from a folder in the middle of my flow using FetchFile
> and passing the absolute path to the files (as ListFile and GetFile have no
> input connections).
> After FetchFile I retrieve the file metadata with ls - l
> --time-style=full-iso which outputs something like this:
>
> -rw-r--r-- 1 nifi nifi 60 2020-02-14 14:14:07.0 + file.txt
>
> From this I retrieve all components of the date and time that are needed and
> merge them together with the following:
>
> fileMetadata.6 value:2020-02-14
> fileMetadata.7 value:14:14:07.0
> fileMetadata.8 value:+
>
> dateMetadata value:${fileMetadata.6:append('
> '):append(${fileMetadata.7:substringBefore('.')})}
> Last Modified Time value:${dateMetadata:replace(' ', 'T'):append('
>
Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
You should not need to explicitly set the additional module directory to cover
that location. Is there a reason you can’t use the native Groovy JSON [1]
parsing? That way you don’t have to download any additional libraries.
[1] http://groovy-lang.org/json.html# <http://groovy-lang.org/json.html#>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 29, 2020, at 7:41 AM, Myklebust, Bjørn Magnar
> wrote:
>
> Andy, just a quick followup on this.
>
> I wanted to test a groovy-script with this code (not finished by far yet),
> and the script is placed in the Script Body part of an
> ExecuteGroovyScript-process in NiFi:
>
>
> import org.json.JSONObject
> import org.json.XML
> import org.apache.commons.io.IOUtils
> import java.nio.charset.*
>
> def flowFile = session.get()
> if (!flowFile) return
>
> flowFile = session.write(flowFile,
> {inputStream, outputStream ->
> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
> def xmlJSONObj = XML.toJSONObject(text);
> def json = xmlJSONObj.toString();
> outputStream.write(json.getBytes(StandardCharsets.UTF_8))
> } as StreamCallback)
>
> session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
>
> But when trying to run this I get the message «unable to resolve class
> org.json.JSONObject @ line 1»
> I have downloaded the jar file from this site:
> https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar
> <https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar>
> And placed it in my nifi/lib-directory.
> And the content of this jar you can see in the enclosed png-picture.
>
> Do I need to set a value for the property Additional Classpath when the
> jar-file is stored in the lib-directory?
>
> Thanks,
> Bjørn
>
>
> Fra: Andy LoPresto
> Sendt: torsdag 25. juni 2020 19:20
> Til: users@nifi.apache.org
> Emne: Re: Replacing a base64-encoded field in a JSON-document with its
> decoded/converted value
>
> Hi Bjørn,
>
> No, XML to JSON conversion is not an Expression Language feature. You’ll need
> to either get this data into a flowfile as the complete content to perform
> the conversion with existing built-in tools, or add that step to your Groovy
> script.
>
> With that additional requirement, I think using the Groovy script to perform
> those steps in tandem is probably the most performant and logical approach
> here.
>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
> On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar
> mailto:bjorn.mykleb...@skatteetaten.no>>
> wrote:
>
> Thanks Andy.
> The XML-content is around 5 kB-ish. But I also need to convert the XML to
> JSON before replacing it back into the original JSON-file. Can this be done
> with e.g a ConvertAttribute before the ReplaceText?
>
> Thanks,
> Bjørn
>
>
>
> Fra: Andy LoPresto mailto:alopre...@apache.org>>
> Sendt: onsdag 24. juni 2020 17:24
> Til: users@nifi.apache.org <mailto:users@nifi.apache.org>
> Emne: Re: Replacing a base64-encoded field in a JSON-document with its
> decoded/converted value
>
> Hello Bjørn,
>
> If the size of the encoded XML document is small (under ~1 KB), you can
> extract the Base64-encoded value to a flowfile attribute using
> EvaluateJSONPath, perform the decoding using the base64Decode Expression
> Language function [1], and then replace it into the flowfile JSON content
> using ReplaceText (using some regex like "content": ".*" -> “content":
> ”${decodedXML}” where decodedXML is the name of the attribute you are using).
>
> If the XML content could be very large, this will negatively affect your
> performance, as attributes are stored directly in memory and handling large
> amounts of data will impact the heap. In this case, I would recommend writing
> a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly
> JSON handling and extract the value, Base64 decode it, and replace it in a
> couple lines.
>
> Hope this helps.
>
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode
>
> <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@
Re: unsubscribe
Please send a message to users-unsubscr...@nifi.apache.org <mailto:users-unsubscr...@nifi.apache.org> to unsubscribe. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 29, 2020, at 8:36 AM, obaidul karim wrote: > >
Re: Duplicate Attribute Values in Extract Text Processor Output
The resulting flowfile will always have at least two attributes because the whole match is extracted as an attribute and every capture group is extracted as an attribute, and the expression must contain at least one capture group. What is the objective you are trying to accomplish? If you want to route flowfiles based on their text contents, you can use RouteText. If you want to extract text content to attributes, use ExtractText. The use case you described above basically retrieves a log file from blob storage, splits each file to individual lines, extracts the content of each line (minus the final character) into an attribute, and then sends the values to Syslog. You may want to look at the record processors to improve the performance and simplicity of the flow substantially. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 25, 2020, at 11:53 AM, muhyid72 wrote: > > Hi Andy, > > Thank you for your quick answer and interest. > > Actually I tried that but there were still 2 attributes on the flow file. As > far as I understand it is by design, I can't set just one attribute, it has > at least 2. Am i right? > > Can I use Route Text Processor instead of Extract Text (I have given my > Extract Text configuration at the above) Dou you have comment? > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
Hi Bjørn,
No, XML to JSON conversion is not an Expression Language feature. You’ll need
to either get this data into a flowfile as the complete content to perform the
conversion with existing built-in tools, or add that step to your Groovy
script.
With that additional requirement, I think using the Groovy script to perform
those steps in tandem is probably the most performant and logical approach
here.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar
> wrote:
>
> Thanks Andy.
> The XML-content is around 5 kB-ish. But I also need to convert the XML to
> JSON before replacing it back into the original JSON-file. Can this be done
> with e.g a ConvertAttribute before the ReplaceText?
>
> Thanks,
> Bjørn
>
>
>
> Fra: Andy LoPresto
> Sendt: onsdag 24. juni 2020 17:24
> Til: users@nifi.apache.org
> Emne: Re: Replacing a base64-encoded field in a JSON-document with its
> decoded/converted value
>
> Hello Bjørn,
>
> If the size of the encoded XML document is small (under ~1 KB), you can
> extract the Base64-encoded value to a flowfile attribute using
> EvaluateJSONPath, perform the decoding using the base64Decode Expression
> Language function [1], and then replace it into the flowfile JSON content
> using ReplaceText (using some regex like "content": ".*" -> “content":
> ”${decodedXML}” where decodedXML is the name of the attribute you are using).
>
> If the XML content could be very large, this will negatively affect your
> performance, as attributes are stored directly in memory and handling large
> amounts of data will impact the heap. In this case, I would recommend writing
> a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly
> JSON handling and extract the value, Base64 decode it, and replace it in a
> couple lines.
>
> Hope this helps.
>
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode
>
> <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
> On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar
> mailto:bjorn.mykleb...@skatteetaten.no>>
> wrote:
>
>
> Hi.
> I have a set of Json-files which contain a base64-coded field (Jsonpath to
> this field is $.data.content), and this field contains a XML-document.
> Decoding the field works as expected, so does the conversion from xml to
> json, and I'm able to write the content from this field to a file in a
> bucket in S3. But what I would like to do is to be able to replace the coded
> value for this field in the original file with the decoded/converted value in
> stead of writing the decoded/converted value to file. And after replacing the
> json-value then I can write the updated Json-file to a new S3 bucket.
> My process look like this at the moment, and works fine for getting the data
> to file, but it's missing the last part of replacing $.data.content with the
> decoded/converted data.
>
> So how can I do the last part?
>
>
>
> The EvaluedJsonPath looks like this:
>
>
>
>
> The ReplaceText looks like this:
>
>
>
> The Base64EncodeContent looks like this:
>
>
>
> and finally, the CovertRecord looks like this:
>
>
>
>
>
> This is a testfile for that I'm working with:
>
> {
> "header": {
> "dokumentidentifikator": null,
> "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",
> "revisjonsnummer": 1,
> "dokumentnavn":
> "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",
> "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",
> "dokumenttilstand": "OPPRETTET",
> "gyldig": true,
> "gjelderInntektsaar": 2017,
> "gjelderPeriode": "2017_10",
> "gjelderPart": {
> "partsnummer": 5544332211,
> "identifiseringstype": "MASKINELL",
> "identifikator": null
> },
> "opphavspart": {
> "partsnummer": 5544332211,
> "identifikator": null
> },
> &
Re: Duplicate Attribute Values in Extract Text Processor Output
The regex you’re using contains a capture group, and so the entire string is
captured as one attribute, and then the contained capture groups are also
extracted as attributes. You can set the property “Include Capture Group 0” to
false to remove one of them. The others are provided as expected.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 25, 2020, at 8:27 AM, muhyid72 wrote:
>
> Dear All
> I need an information about Flow Files Attribute of Extract Text Processor.
> My flow is that;
>
> 1. Getting IIS Log files from Azure Blob Storage
> 2. Splitting each IIS Log File to line by line with Split Text Processor.
> 2.1. Line Split Count:1
> 2.2. Maximum Fragment Size: No value set
> 2.3. Header Line Count: 0
> 2.4. Header Line Marker Characters: No value set
> 2.5. Remove Trailing Newlines: True
> 3. Transferring new flow files which is produced by Split Text Processor to
> Extract Text Processor.
> 3.1. All Properties are Default
> 3.2. I added one RegEx in the Properties. I would like to carry on Flow
> Files attributes to Syslog
> 3.2.1. Property Name: msg
> 3.2.2. Value: (.*).
> 4. Transferring all flow files where is coming from Extract Text to Put
> Syslog Processor.
> 4.1. All Properties are Default or configured properly for requirements
> (such as IP address of the Syslog, port etc.)
> 4.2. Message Body: IISHttp${msg}
>
> When I check Flow Files Attribute from Data Provenance in the Extract Text
> Processor, I see 3 attributes same each other.
> Msg: 2020-06-24 13:33:49 GET /Test/Service/test.css
> 200 0 0 852 7005 921
> Msg.1: 2020-06-24 13:33:49 GET /Test/Service/test.css
> 200 0 0 852 7005 921
> Msg.2: 2020-06-24 13:33:49 GET /Test/Service/test.css
> 200 0 0 852 7005 921
>
> How can I remove duplicate attributes from extract text output? Or I need to
> use another way?
> Do you have any comment or suggestion?
>
> My environment details are below:
> Apache NiFi 1.11.3
> Windows Server 2016
> Java JRE 1.8.0_241 (64 Bit)
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
Hello Bjørn,
If the size of the encoded XML document is small (under ~1 KB), you can extract
the Base64-encoded value to a flowfile attribute using EvaluateJSONPath,
perform the decoding using the base64Decode Expression Language function [1],
and then replace it into the flowfile JSON content using ReplaceText (using
some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML
is the name of the attribute you are using).
If the XML content could be very large, this will negatively affect your
performance, as attributes are stored directly in memory and handling large
amounts of data will impact the heap. In this case, I would recommend writing a
Groovy script in ExecuteScript processor to leverage Groovy’s very friendly
JSON handling and extract the value, Base64 decode it, and replace it in a
couple lines.
Hope this helps.
[1]
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode
<https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar
> wrote:
>
>
> Hi.
> I have a set of Json-files which contain a base64-coded field (Jsonpath to
> this field is $.data.content), and this field contains a XML-document.
> Decoding the field works as expected, so does the conversion from xml to
> json, and I'm able to write the content from this field to a file in a
> bucket in S3. But what I would like to do is to be able to replace the coded
> value for this field in the original file with the decoded/converted value in
> stead of writing the decoded/converted value to file. And after replacing the
> json-value then I can write the updated Json-file to a new S3 bucket.
> My process look like this at the moment, and works fine for getting the data
> to file, but it's missing the last part of replacing $.data.content with the
> decoded/converted data.
>
> So how can I do the last part?
>
>
>
> The EvaluedJsonPath looks like this:
>
>
>
>
> The ReplaceText looks like this:
>
>
>
> The Base64EncodeContent looks like this:
>
>
>
> and finally, the CovertRecord looks like this:
>
>
>
>
>
> This is a testfile for that I'm working with:
>
> {
> "header": {
> "dokumentidentifikator": null,
> "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",
> "revisjonsnummer": 1,
> "dokumentnavn":
> "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",
> "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",
> "dokumenttilstand": "OPPRETTET",
> "gyldig": true,
> "gjelderInntektsaar": 2017,
> "gjelderPeriode": "2017_10",
> "gjelderPart": {
> "partsnummer": 5544332211,
> "identifiseringstype": "MASKINELL",
> "identifikator": null
> },
> "opphavspart": {
> "partsnummer": 5544332211,
> "identifikator": null
> },
> "kildereferanse": {
> "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",
> "gruppe": "",
> "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",
> "tidspunkt": "2018-01-16T13:28:02.49+01:00"
> }
> },
> "data": {
> "metadata": {
> "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",
> "bytes": 4420,
> "mimeType": "application/xml",
> "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="
> },
> "content":
> "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiA
Re: 3 node of nifi generating 3 different flow files
Sanjeet, Did you stop every node in the cluster before deleting these files? Can you share the actual output of the log files? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 18, 2020, at 5:15 PM, sanjeet rath wrote: > > Hi Andy, > Already tried by deleting flow.xml.gz ,users.xml,authorization.xml file of 3 > nodes. > But still getting the same error. > in nifi-app.log is showing exception that" local flow is differet than > cluster flow" > Also new flow.xml.gz file is having different 'rootGroupid' on each node > Thanks. > Sanjeet > > On Fri, 19 Jun 2020, 5:16 am Andy LoPresto, <mailto:alopre...@apache.org>> wrote: > Sanjeet, > > If this is for a new cluster, you can delete the flow.xml.gz file from all > nodes and restart NiFi. When the nodes start up again, they will create the > new flow definition file on each node respectively with the synced root > process group ID. > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 18, 2020, at 4:44 PM, sanjeet rath > <mailto:rath.sanj...@gmail.com>> wrote: >> >> Thanks, Andy for your quick response. >> I did it as you suggested, now nifi is using its default encryption. >> >> But still gettting the error, in nifi-app.log is showing exception that" >> local flow is differet than cluster flow" >> As per my analysis, The issue is it got 3 different flow.xml.gz files in >> each node.The 'rootGroupid' of each flow.xml.gz file is different. >> As per my understanding ,the flow.xml.gz fie should be same accross all >> nodes.so i copied flow.xml.gz file from one node to other two nodes to make >> it sync.but still getting same error. >> The zookeeper is connected properly to all nodes. >> >> Thanks, >> Sanjeet >> >> >> On Thu, 18 Jun 2020, 10:55 pm Andy LoPresto, > <mailto:alopre...@apache.org>> wrote: >> You do not need to manually run this command, as it migrates the encryption >> key used for sensitive processor properties (e.g. database passwords in the >> flow) that are stored in your flow.xml.gz file. You only need this command >> when you have a cluster which has been using one encryption key to secure >> these values and you want to migrate to a new key. >> >> When starting a new cluster, set the nifi.sensitive.props.key value to the >> desired value on all cluster nodes, and NiFi will automatically encrypt and >> decrypt the sensitive processor properties with it. >> >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> He/Him >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Jun 18, 2020, at 3:55 AM, sanjeet rath >> <mailto:rath.sanj...@gmail.com>> wrote: >>> >>> Hi , >>> >>> I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing >>> exception that" local flow file is differet than cluster flow" >>> The issue is i got 3 different flow files in each node.'rootGroupid' of >>> each flow file is different. >>> >>> Previous i was facing no issue in creating 3 node clister.but this time ,i >>> made a change here during creation the clusters i have encripted the flow >>> file with using nifi toolkit.(encript-config.sh) >>> >>> Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties >>> --bootstrapconf /nifi/bootstrap.conf -s password -x >>> >>> Please note that there is are no flows are availavle in canvas.its a new >>> nifi cluster. >>> >>> Could you please help me to understand what i am doing wrong? >>> >>> Thanks a lot in advance for helping me. >>> >>> Regards, >>> Sanjeet >> >
Re: 3 node of nifi generating 3 different flow files
Sanjeet, If this is for a new cluster, you can delete the flow.xml.gz file from all nodes and restart NiFi. When the nodes start up again, they will create the new flow definition file on each node respectively with the synced root process group ID. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 18, 2020, at 4:44 PM, sanjeet rath wrote: > > Thanks, Andy for your quick response. > I did it as you suggested, now nifi is using its default encryption. > > But still gettting the error, in nifi-app.log is showing exception that" > local flow is differet than cluster flow" > As per my analysis, The issue is it got 3 different flow.xml.gz files in each > node.The 'rootGroupid' of each flow.xml.gz file is different. > As per my understanding ,the flow.xml.gz fie should be same accross all > nodes.so i copied flow.xml.gz file from one node to other two nodes to make > it sync.but still getting same error. > The zookeeper is connected properly to all nodes. > > Thanks, > Sanjeet > > > On Thu, 18 Jun 2020, 10:55 pm Andy LoPresto, <mailto:alopre...@apache.org>> wrote: > You do not need to manually run this command, as it migrates the encryption > key used for sensitive processor properties (e.g. database passwords in the > flow) that are stored in your flow.xml.gz file. You only need this command > when you have a cluster which has been using one encryption key to secure > these values and you want to migrate to a new key. > > When starting a new cluster, set the nifi.sensitive.props.key value to the > desired value on all cluster nodes, and NiFi will automatically encrypt and > decrypt the sensitive processor properties with it. > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 18, 2020, at 3:55 AM, sanjeet rath > <mailto:rath.sanj...@gmail.com>> wrote: >> >> Hi , >> >> I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing >> exception that" local flow file is differet than cluster flow" >> The issue is i got 3 different flow files in each node.'rootGroupid' of each >> flow file is different. >> >> Previous i was facing no issue in creating 3 node clister.but this time ,i >> made a change here during creation the clusters i have encripted the flow >> file with using nifi toolkit.(encript-config.sh) >> >> Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties >> --bootstrapconf /nifi/bootstrap.conf -s password -x >> >> Please note that there is are no flows are availavle in canvas.its a new >> nifi cluster. >> >> Could you please help me to understand what i am doing wrong? >> >> Thanks a lot in advance for helping me. >> >> Regards, >> Sanjeet >
Re: 3 node of nifi generating 3 different flow files
You do not need to manually run this command, as it migrates the encryption key used for sensitive processor properties (e.g. database passwords in the flow) that are stored in your flow.xml.gz file. You only need this command when you have a cluster which has been using one encryption key to secure these values and you want to migrate to a new key. When starting a new cluster, set the nifi.sensitive.props.key value to the desired value on all cluster nodes, and NiFi will automatically encrypt and decrypt the sensitive processor properties with it. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 18, 2020, at 3:55 AM, sanjeet rath wrote: > > Hi , > > I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing > exception that" local flow file is differet than cluster flow" > The issue is i got 3 different flow files in each node.'rootGroupid' of each > flow file is different. > > Previous i was facing no issue in creating 3 node clister.but this time ,i > made a change here during creation the clusters i have encripted the flow > file with using nifi toolkit.(encript-config.sh) > > Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties > --bootstrapconf /nifi/bootstrap.conf -s password -x > > Please note that there is are no flows are availavle in canvas.its a new nifi > cluster. > > Could you please help me to understand what i am doing wrong? > > Thanks a lot in advance for helping me. > > Regards, > Sanjeet
Re: AmazonDocumentDB
If that was the issue, you can also import those PEM files into a Java Keystore (.jks) file and configure the SSL Context Service your Mongo processors use to reference it as a truststore. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 15, 2020, at 6:36 PM, Trevor Dunn wrote: > > FYI for those that might be looking for the same thing. > > Yes the mongodb processors work with Amazon DocumentDB. However in order to > get the SSL to work you need to break the .pem file from Amazon into its > individual keys and then import the certificates individually into the > cacerts file for the java runtime that is running Nifi. Restart NiFi and > hopefully it will work for you. > > > > From: Trevor Dunn <mailto:trevor.d...@ihsmarkit.com>> > Sent: Monday, June 15, 2020 2:21 PM > To: users@nifi.apache.org <mailto:users@nifi.apache.org> > Subject: AmazonDocumentDB > > [CAUTION] EXTERNAL EMAIL .. > > Hi All. > > Has anyone used the Mongodb processors to connect to Amazon DocumentDB. > > I am getting an issue connecting but the mongo shell connects fine. > > Thanks > Trevor > > > This e-mail, including accompanying communications and attachments, is > strictly confidential and only for the intended recipient. Any retention, use > or disclosure not expressly authorised by IHSMarkit is prohibited. This email > is subject to all waivers and other terms at the following link: > https://ihsmarkit.com/Legal/EmailDisclaimer.html > <https://ihsmarkit.com/Legal/EmailDisclaimer.html> > > Please visit www.ihsmarkit.com/about/contact-us.html > <http://www.ihsmarkit.com/about/contact-us.html> for contact information on > our offices worldwide. > > > This e-mail, including accompanying communications and attachments, is > strictly confidential and only for the intended recipient. Any retention, use > or disclosure not expressly authorised by IHSMarkit is prohibited. This email > is subject to all waivers and other terms at the following link: > https://ihsmarkit.com/Legal/EmailDisclaimer.html > <https://ihsmarkit.com/Legal/EmailDisclaimer.html> > > Please visit www.ihsmarkit.com/about/contact-us.html > <http://www.ihsmarkit.com/about/contact-us.html> for contact information on > our offices worldwide.
Re: In memoriam of Jeff Storck
Jeff loved mocks, both friendly impressions and in his tests. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 15, 2020, at 12:12 PM, Jeremy Dyer wrote: > > This is shocking and heartbreaking news. Jeff was a great guy and will be > deeply missed. > > The last time I saw Jeff in person was with Aldrin. We were eating at Bonchon > chicken and he was mocking me for how little spice I could handle XD. I could > always count on him for a good Dumb and Dumber reference and laugh. We also > shared a common hatred for conference food. > > RIP Jeff > > On Mon, Jun 15, 2020 at 2:33 PM Joe Witt <mailto:joe.w...@gmail.com>> wrote: > You will be greatly missed. Your impact to this community has been > tremendous. The items Andy summarizes were huge efforts that you drove over > periods of many many months if not a year or more and they make NiFi so much > more accessible than before. > > RIP Jeff. > > > > On Mon, Jun 15, 2020 at 11:24 AM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > It is with a heavy heart that I write to the NiFi community today. Jeff > Storck, a PMC member, committer, and genuine and helpful presence in the > community, has passed away. > > I was lucky enough to know Jeff personally for many years, and his absence is > a huge loss to all of us who did. Jeff was incredibly intelligent, but also > kind and willing to share his experience with everyone. Whether playing > volleyball (I am nowhere near as good but he humored me), discussing the best > ramen and sushi spots, or evaluating a new exercise regime, Jeff brought > passion to everything. A number of us are sharing stories of our favorite > times with Jeff, and I am touched by how many people have a memory of Jeff > reaching out and patiently helping them when they were new or struggling with > a task. > > While other colleagues would happily transition to any topic _but_ work when > we went to a nearby brewery at the end of a long day, Jeff would sit down > next to me and say with a smile, "Ok Andy, work's done, now we can _really_ > talk about Groovy unit testing." He never shied away from expressing his > perspective and stood on conviction, but he was also open and genuinely > wanted to hear other views to expand his mind. > > If you come across a Spock test in the NiFi codebase, that was most likely > Jeff's work. He was intimately involved in much of the most challenging code > - especially Kerberos integration, making the difficult but critical > processes easier for our users. Anyone running NiFi on Java 11 should thank > Jeff, as that was a labor of love, pushing against the headwinds of so many > compatibility issues and language changes. The ease with which NiFi runs on > multiple versions and platforms belies the immense amount of effort and > dedication that he put into making this happen. > > There are so many aspects to Jeff that a note like this could never capture, > but one that stands above the rest to me is Jeff's passion for learning and > growth. He devoted himself to doing the best he could and constantly > improving that. That is a noble philosophy that I know I will remember and > admire moving forward. I’ve already started learning Kotlin because of Jeff’s > enthusiasm and encouragement. > > Jeff’s family has created a GoFundMe page [1] and there they describe their > intent to celebrate his life. I think that message is very positive and > uplifting. To anyone wondering how they can honor Jeff's legacy, I suggest > offering a helping hand to someone who needs it. Something as simple as > responding to an extra "newbie" mailing list question at the end of a long > day, or taking on a challenging task because your neighbor has their plate > full. That's how Jeff lived, and he made the world a better place. > > > > Andy > > [1] https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck > <https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck> > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >
Re: MergeContent resulting in corrupted JSON
Sorry, TWR = try-with-resources. Definitely a lot of old code that “still works” but is brittle. We should do better about highlighting modern implementations and paying down tech debt, but the project just moves so quickly. Not a perfect rule, but if I see code from one of the core contributors with a date in the last couple years, I trust it much more than even code by those same people from 5 years ago (which was likely written even longer ago than that; time “starts” from the initial import in 2014). Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 11, 2020, at 18:16, Jason Iannone wrote: > > > We currently have it encapsulated in code that allows proper isolation and > testing, as this is the same methodology applied for standard development. > What I wasn't sure is whether Nifi is opinionated and actually preferred > and/or performed better with callbacks. There's a lot of older Nifi examples > out there, including Nifi core processors and its hard to discern what's > recommended. > > What's TWR? > > Appreciate the replies, you both have been immensely helpful! > > Thanks, > Jason > >> On Thu, Jun 11, 2020 at 8:40 PM Andy LoPresto wrote: >> To give another perspective on the “callback vs. non”, I’d say “heavy” or >> “messy” operations (like encryption, for example) should be contained in >> encapsulated code (other classes which provide a service) and then invoked >> from the callback or TWR. This allows for much more testable business logic, >> separation of concerns (a service which implements the behavior and then a >> component effectively calling the API), and composability/flexibility. If I >> want to build a processor which exposes a property allowing the user to >> select different encryption algorithms, I can either detect which one and >> delegate that to an implementation, or I could have a giant switch statement >> and the raw crypto primitive code all in a giant spaghetti method/callback >> definition. I know I would prefer the former. >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> He/Him >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Jun 11, 2020, at 8:14 AM, Mark Payne wrote: >>> >>> Jason, >>> >>> Modify vs. clone vs. create new: >>> >>> You would clone a FlowFile if you want an exact copy of the FlowFile (with >>> the exception that the clone will have a unique UUID, Entry Date, etc.). >>> Very rare that a Processor will actually do this. >>> >>> Modify vs. create a “Child” FlowFiles (i.e., `session.create(original);` ) >>> - This is a judgment call really. Do you think it will be necessary to have >>> a copy of the original FlowFile and a modified version of it? If so, you >>> may want to create a child FlowFile and send the original FlowFile to >>> original. In reality, you shouldn’t need this often. In most cases, if the >>> user wants both the original and the modified version, they can just add >>> two connections, one going to this processor and one going to wherever else >>> they want the FlowFile. This will cause NiFi to implicitly clone the >>> FlowFile. Where the “create a child and send out the original” matters is >>> just when there’s a feasible use case in which the user would want to have >>> a modified version of the FlowFile and the original version of the FlowFile >>> and also not want to process the original version until after the modified >>> version has been created. This is not common. However, over the years, it >>> has become a common practice to create “original” relationships when they >>> are not needed, likely because a few developers saw a pattern of creating >>> an original relationship and duplicated this to many other processors >>> without really understanding the difference. >>> >>> “Net New” - there are two ways to create a FlowFile: `session.create()` and >>> `session.create(original);` - the first creates a FlowFile with no parent >>> FlowFile. This should be done only if there is no inbound FlowFile to >>> create it from. I.e., when this is a “source” processor. In 100% of all >>> other cases, it should be done as `session.create(original);` Providing the >>> original FlowFile does 2 important things. Firstly, it creates a linkage in >>> provenance between them. Secondly, it causes the newly created FlowFile to >>> inherit all attribut
Re: NiFi Expression Language in UpdateAttribute
Russell,
I think it would be fine to include an example like this in the Expression
Language Guide. You can submit a PR to add that if you like.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 11, 2020, at 7:31 AM, Russell Bateman wrote:
>
> That solves confusion I had after and despite reading the NiFi Expression
> Language documentation. Having written documentation before, I get why it's
> never ending and danger-fraught to write "bigger" examples in such
> documentation, and we resist doing it, but that's what I needed to clear up
> in my head the combination of how the two constructs, endsWith and ifElse,
> work together.
>
> Many thanks!
>
> On 6/11/20 8:24 AM, Shawn Weeks wrote:
>> Your syntax is incorrect. Try this instead.
>>
>> ${filename:endsWith('xml'):ifElse('XML','JSON')}
>>
>> From: Russell Bateman <mailto:r...@windofkeltia.com>
>> Reply-To: "users@nifi.apache.org" <mailto:users@nifi.apache.org>
>> <mailto:users@nifi.apache.org>
>> Date: Thursday, June 11, 2020 at 9:15 AM
>> To: NiFi Users <mailto:users@nifi.apache.org>
>> Subject: NiFi Expression Language in UpdateAttribute
>>
>> I wnat to create an attribute, format-type, which I hoped would be set based
>> on the in-coming flowfile name:
>>
>> ${ ${filename:endsWith( 'xml' )} : ifElse( 'XML', 'JSON' ) }
>>
>> If the in-coming flowfile name is sample.xml, the result is nevertheless and
>> always
>>
>> format-type = JSON
>>
>> I want it to be XML, obviously. What have I screwed up in my NEL syntax?
>>
>> (Yes, I have tried taking out all the white space.)
>>
>> Thanks,
>> Russ
Re: Expression language null value
Yes, can open a Jira for a feature request here [1] and submit a pull request implementing this change here [2]. Please read the Contributor Guide [3] and Developer Guide [4] for more information about this process. [1] https://issues.apache.org/jira/secure/CreateIssue!default.jspa <https://issues.apache.org/jira/secure/CreateIssue!default.jspa> [2] https://github.com/apache/nifi/pulls <https://github.com/apache/nifi/pulls> [3] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> [4] https://nifi.apache.org/developer-guide.html <https://nifi.apache.org/developer-guide.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 11, 2020, at 7:40 AM, Fabrizio Spataro wrote: > > ... ping ... > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: MergeContent resulting in corrupted JSON
To give another perspective on the “callback vs. non”, I’d say “heavy” or “messy” operations (like encryption, for example) should be contained in encapsulated code (other classes which provide a service) and then invoked from the callback or TWR. This allows for much more testable business logic, separation of concerns (a service which implements the behavior and then a component effectively calling the API), and composability/flexibility. If I want to build a processor which exposes a property allowing the user to select different encryption algorithms, I can either detect which one and delegate that to an implementation, or I could have a giant switch statement and the raw crypto primitive code all in a giant spaghetti method/callback definition. I know I would prefer the former. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 11, 2020, at 8:14 AM, Mark Payne wrote: > > Jason, > > Modify vs. clone vs. create new: > > You would clone a FlowFile if you want an exact copy of the FlowFile (with > the exception that the clone will have a unique UUID, Entry Date, etc.). Very > rare that a Processor will actually do this. > > Modify vs. create a “Child” FlowFiles (i.e., `session.create(original);` ) - > This is a judgment call really. Do you think it will be necessary to have a > copy of the original FlowFile and a modified version of it? If so, you may > want to create a child FlowFile and send the original FlowFile to original. > In reality, you shouldn’t need this often. In most cases, if the user wants > both the original and the modified version, they can just add two > connections, one going to this processor and one going to wherever else they > want the FlowFile. This will cause NiFi to implicitly clone the FlowFile. > Where the “create a child and send out the original” matters is just when > there’s a feasible use case in which the user would want to have a modified > version of the FlowFile and the original version of the FlowFile and also not > want to process the original version until after the modified version has > been created. This is not common. However, over the years, it has become a > common practice to create “original” relationships when they are not needed, > likely because a few developers saw a pattern of creating an original > relationship and duplicated this to many other processors without really > understanding the difference. > > “Net New” - there are two ways to create a FlowFile: `session.create()` and > `session.create(original);` - the first creates a FlowFile with no parent > FlowFile. This should be done only if there is no inbound FlowFile to create > it from. I.e., when this is a “source” processor. In 100% of all other cases, > it should be done as `session.create(original);` Providing the original > FlowFile does 2 important things. Firstly, it creates a linkage in provenance > between them. Secondly, it causes the newly created FlowFile to inherit all > attributes from the child. > > Call vs. non-callback: It doesn’t matter. The callback was originally the > only way to read or write content of FlowFiles. It was done this way because > it was a straight-forward way to ensure that the framework was able to > properly manage InputStream, OutputStream, etc. But there were use cases that > didn’t fit the callback mechanism well so we eventually added ability to get > the InputStreams and OutputStreams directly and callers can just use > try-with-resources. This is probably preferred now for most cases just > because it results in cleaner code. > > Thanks > -Mark > >> On Jun 11, 2020, at 10:43 AM, Jason Iannone > <mailto:bread...@gmail.com>> wrote: >> >> I confirmed what you mentioned as well. >> >> I also looked over many custom processor examples and looking for >> clarification on a few things which I didn't see explicitly called out in >> the developers guide. >> Are their guidelines on when one should modify the original flowfile vs when >> you should clone vs when you should create net new? >> Should heavier lifting such as decryption, formatting, etc. be done in a >> callback? >> >> Thanks, >> Jason >> >> On Wed, Jun 10, 2020 at 4:32 PM Mark Payne > <mailto:marka...@hotmail.com>> wrote: >> I don’t think flushing should matter, if you’re writing directly to the >> provided OutputStream. If you wrap it in a BufferedOutputStream or something >> like that, then of course you’ll want to flush that. Assuming that you are >> extending AbstractProcessor, it will call session.commit() for you >> automatically when onTr
Re: MergeContent resulting in corrupted JSON
It may just be a copy/paste or retyping issue, but in the example you provided,
I see unpaired double quotes (the hexBytes values have trailing quotes but not
leading ones), which could be causing issues in parsing…
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 9, 2020, at 5:02 PM, Mark Payne wrote:
>
> Hey Jason,
>
> Thanks for reaching out. That is definitely odd and not something that I’ve
> seen or heard about before.
>
> Are you certain that the data is not being corrupted upstream of the
> processor? I ask because the code for the processor that handles writing out
> the content is pretty straight forward and hasn’t been modified in over 3
> years, so I would expect to see it happen often if it were a bug in the
> MergeContent processor itself. Any chance that you can create a flow
> template/sample data that recreates the issue? Anything particularly unique
> about your flow?
>
> Thanks
> -Mark
>
>
>> On Jun 9, 2020, at 6:47 PM, Jason Iannone wrote:
>>
>> Hi all,
>>
>> Within Nifi 1.10.0 we're seeing unexpected behavior with mergecontent. The
>> processor is being fed in many flowfiles with individual JSON records. The
>> records have various field types including a hex-encoded byte[]. We are not
>> trying to merge JSON records themselves but rather consolidate many
>> flowfiles into fewer flowfiles.
>>
>> What we're seeing is that a random flowfile is split causing the merge file
>> to be invalid JSON. When running multiple bins we saw the flowfile split
>> across bins.
>>
>> Example
>> Flowfile 1: {"name": "1", "hexbytes": A10F15B11D14", timestamp: "123456789" }
>> Flowfile 2: {"name": "2", "hexbytes": A10F15D14B11", timestamp: "123456790"
>> }
>> Flowfile 3: {"name": "3", "hexbytes": A10F15D14B11", timestamp: "123456790"
>> }
>>
>> Merged Result:
>> {"name": "1", "hexbyters": A10F15B11D14", timestamp: "123456789" }
>> xbytes": A10F15D14B11", timestamp: "123456790" }
>> {"name": "3", "hexbytes": A10F15D14B11", timestamp: "123456790" }
>> {"name": "3", "h
>>
>> Mergecontent Configuration:
>> Concurrent Tasks: 4
>> Merge Strategy: Bin-Packing Algorithm
>> Merge Format: Binary Concatenation
>> Attribute Strategy: Keep Only Common Attributes
>> Min. number of entries 1000
>> Max number of entries: 2
>> Minimum group size: 10 KB
>> Maximum number of bins: 5
>> Header, Footer, and Demaractor are not set.
>>
>> We then backed off the below to reduce min and max entries, bin to 1, and
>> thread to 1 and still see the same issue.
>>
>> Any insights?
>>
>> Thanks,
>> Jason
>
Re: Execute Script - Groovy get attribute
It also seems like you would rather set the flowfile batch count as a
_property_ on the ExecuteScript processor and obtain it from the session,
rather than check a specific flowfile for an attribute which may or may not
exist.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 3, 2020, at 8:07 AM, Matt Burgess wrote:
>
> Asmath,
>
> When you pass an integer argument into session.get(), you get an ArrayList
> back, not a single FlowFile. You can just use session.get() to get (up to) 1
> FlowFile, or if you call session.get(1) you can just use the first element
> (once checking to see if it's null), so trigger[0].getAttribute('cnt')
>
> Regards,
> Matt
>
>
> On Wed, Jun 3, 2020 at 8:58 AM KhajaAsmath Mohammed <mailto:mdkhajaasm...@gmail.com>> wrote:
> Hi,
>
> I have followed the below link to wait for 100 files and then process the
> next step.
>
> https://community.cloudera.com/t5/Support-Questions/NiFi-Count-Fileflows-via-attribute/td-p/178860
>
> <https://community.cloudera.com/t5/Support-Questions/NiFi-Count-Fileflows-via-attribute/td-p/178860>
>
>
> This works well but my use case is that count number is dynamic. To test the
> functionality, I have passed an attribute count to get the value. Below
> script is not working. any help?
>
> import org.apache.commons.io.IOUtils
> import java.nio.charset.StandardCharsets
> def trigger=session.get(1)
> def flowFiles=session.get(99)
> if (!trigger) return
>
> String attr=trigger.getAttribute("cnt");
>
>
> if(!flowFiles || flowFiles.size() < 7)
> {
> session.rollback()
> }
> else
> {
> session.transfer(trigger,REL_SUCCESS)
> session.transfer(flowFiles,REL_FAILURE)
>
> }
>
>
>
> Here is the error I got.
>
> Thanks,
> Asmath
Re: Prometheus reporting task
At a quick glance it does not look like there is much logging. Perhaps Matt Burgess can offer more context around this? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 1, 2020, at 2:45 PM, Pavel S wrote: > > Yes I did enable it on "All Components". I will check it again, I wish there > was more documentation on this. Is there a log anywhere that I can see about > what the reporting task is doing? > > On Mon, Jun 1, 2020 at 5:19 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Yolanda replied to a similar question a while ago with some good notes on > setting this up [1]. Specifically, you should be able to configure the > reporting task to be scoped to the root process group, a specific process > group, or the entire canvas. > > [1] > http://apache-nifi-users-list.2361937.n4.nabble.com/Metrics-via-Prometheus-tp9102p9107.html > > <http://apache-nifi-users-list.2361937.n4.nabble.com/Metrics-via-Prometheus-tp9102p9107.html> > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 1, 2020, at 2:10 PM, Pavel S > <mailto:pavel.sapozhniko...@gmail.com>> wrote: >> >> Hello >> >> I have upgraded Nifi to version 1.10.0 just to be able to enable Prometheus >> Reporting Task. Enabling the task was easy and installing Prometheus itself >> was easy and pointing it to Nifi to scrape. I have number of processors >> running in my Nifi. However, the metrics that get exposed only are bound to >> one of those processors and no matter what I try to do it is always that >> same processor that gets exposed. What am I doing wrong? How do I get all of >> my processors to expose? Is there a configuration that I am missing >> somewhere? >
Re: Prometheus reporting task
Yolanda replied to a similar question a while ago with some good notes on setting this up [1]. Specifically, you should be able to configure the reporting task to be scoped to the root process group, a specific process group, or the entire canvas. [1] http://apache-nifi-users-list.2361937.n4.nabble.com/Metrics-via-Prometheus-tp9102p9107.html <http://apache-nifi-users-list.2361937.n4.nabble.com/Metrics-via-Prometheus-tp9102p9107.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 1, 2020, at 2:10 PM, Pavel S wrote: > > Hello > > I have upgraded Nifi to version 1.10.0 just to be able to enable Prometheus > Reporting Task. Enabling the task was easy and installing Prometheus itself > was easy and pointing it to Nifi to scrape. I have number of processors > running in my Nifi. However, the metrics that get exposed only are bound to > one of those processors and no matter what I try to do it is always that same > processor that gets exposed. What am I doing wrong? How do I get all of my > processors to expose? Is there a configuration that I am missing somewhere?
Re: Accessing flow attributes from ExecuteStreamCommand
I think Mike is referring to the “command arguments” & “command arguments strategy” processor properties when he says “parameters”, as this is different from NiFi parameters. If the Python script can run in Jython (2.x only, no native libs), you could put the script into an ExecuteScript or InvokeScriptedProcessor, in which case the NiFi session API is available to your code and you can interact directly with attributes. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 28, 2020, at 7:57 AM, Mike Thomsen wrote: > > There's not way at the moment to interact with the NiFi API from that > processor. The closest work around would be to pass in flowfile attributes as > parameters using the parameter configuration field and expression language. > > On Thu, May 28, 2020 at 10:28 AM Jean-Sebastien Vachon > mailto:jsvac...@brizodata.com>> wrote: > Hi all, > > > > I am using the ExecuteStreamCommand processor to run a python script to > crunch different data and I was curious > > to know if such a processor could both read and/or write from/to the flow > attributes. > > > > Can someone point me to the documentation if this is possible? I could not > find it by myself. > > > > Thanks > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 > > >
Re: Error to connect using ConsumeAMQP processor
Is the .NET application which can connect also running in a Docker container? Does the connection use TLS at all? Are the ports mapped to the host machine running both Docker containers? Running from a shell on the container hosting NIFi, can you connect to the AMQP server by address & port? I suspect the issue is with Docker networking making those ports available to resolve within the other container. Have you followed the steps in [1]? [1] https://www.rabbitmq.com/troubleshooting-networking.html <https://www.rabbitmq.com/troubleshooting-networking.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 25, 2020, at 8:18 AM, Marcus Vinnicius Brandão > wrote: > > Dears, > > I have a problem to connect my RabbitMQ with Apache Nifi. > I run a docker container RabbitMQ with shared ports 15672 and 5672. And i run > other docker container Apache Nifi, with shared ports 9090. > > So, when I try to connect my ConsumeAMQP processor, I receive the message: > "Failed to establish connection with AMQP broker" > > My Configuration in processor is: > Queue : TestQueueNifi (this is the same name in Rabbit) > Auto-Acknowledge messages : false > Batch Size : 10 > Host Name : localhost > Port : 5672 > Virtual Host : > User Name : admin > Password : admin (I set this password and login in rabbit) > AMQP Version : 0.9.1 > SSL Context Service : > Use Certificate Authentication : false > Client Auth : NONE > > I tryed to connect in RabbitMQ with my .net application with the same > connection and worked! > > Please, > someone can help me? > > Thank you!
Re: ExecuteProcess processor with TLS1.2 error: "failed setting cipher list"
Thanks Eric. Glad to know what the issue was and this should help people in the
future. Always appreciate when people follow up and document a solved problem
for the benefit of the community.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On May 24, 2020, at 16:45, Eric Chaves wrote:
>
>
> Hi Andy, sorry for not answering before. I Just figured this one out (after a
> lot of trial and error). This one was tricky. ;)
>
> The curl being used was the same one that I ran on bash. The error was
> related to how I was passing the arguments to curl. In bash I was passing the
> argument --ciphers 'DEFAULT:!DH' with a single quote to prevent bash
> expansion and when I declared the arguments on the processor I did the same
> however it seems that the processor does some quoting on it's own and curl
> was getting confused with the name of the cipher.
>
> Once I removed the quotes the command worked just fine.
>
> Thanks for the help anyway.
>
>
>> Em sex., 22 de mai. de 2020 às 15:11, Andy LoPresto
>> escreveu:
>> Hi Eric,
>>
>> Can you verify a couple things?
>>
>> 1. The specific curl instance you’re using in the terminal and in NiFi are
>> the same? (i.e. run this command on the terminal and in an ExecuteProcess
>> processor: $ which curl)
>> 2. Run curl -V to see which version of openssl curl is using in both
>> scenarios.
>> 3. Run curl -vvv to see increased verbosity output.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>> On May 22, 2020, at 8:08 AM, Eric Chaves wrote:
>>>
>>> Hi folks,
>>>
>>> I have a flow that downloads files from an FTP server over SSL using
>>> TLS1.2. To achieve this I use curl command line in an ExecuteProcess
>>> processor. This routine has been working ok until recently when we tried it
>>> on an upgraded NiFi server.
>>>
>>> After tracking down the error we noticed that it was due to the updated
>>> version of open-ssl recommendation of not allowing the use of old ciphers.
>>> The FTP server in question is using TLS1.2 with a weak certificate but
>>> since it is not managed by me updating the server is not an option.
>>>
>>> After some troubleshooting I managed to adjust my curl command and it is
>>> working when I execute it manually in a bash session on my nifi server (to
>>> be precise I ran it inside the docker container that is running the nifi)
>>> but when I execute the same command line with the ExecuteProcess processor
>>> I got the following error: "failed setting cipher list"
>>>
>>> The curl command and argument line I'm executing is:
>>>
>>> curl -v -slk --tlsv1.2 --ciphers 'DEFAULT:!DH' --user
>>> ${FTP_USER}:${FTP_PASS} --ftp-ssl ftp://${FTP_HOST}:${FTP_PORT}/${FTP_DIR}/
>>>
>>> The actual verbose error from inside the ExecuteProcess processor is:
>>>
>>> * Trying 200.230.161.229...
>>> * TCP_NODELAY set
>>> * Expire in 200 ms for 4 (transfer 0x55f98e691f50)
>>> * Connected to () port
>>> (#0)
>>> < 220 ProFTPD 1.3.4d Server (...) []
>>> > AUTH SSL
>>> < 234 AUTH SSL successful
>>> * failed setting cipher list: 'DEFAULT:!DH'
>>> * Closing connection 0
>>>
>>> So it seems that some configuration either on the nifi or the
>>> ExecuteProcess is not allowing me to force my curl command to use insecure
>>> ciphers with openssl.
>>>
>>> How can I circumvent this?
>>>
>>> Best regards,
>>>
>>> Eric
>>
Re: Connecting Controller Services Automatically
Yes, I should have clarified this. Thanks Bryan. This is the solution for the generic use case. The original question was about reducing the controller to only a single instance of a specific controller service implementation, which is how the tangent got started. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 23, 2020, at 3:27 PM, Bryan Bende wrote: > > If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto select > external controller services with the same name as long as there is only one > of the same type with same name (name is not unique). > > On Sat, May 23, 2020 at 3:34 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > My position is that we don’t claim completely automated deployment as a > feature, so manually setting the controller service IDs is not exposed. > Technically, they are defined in the flow.xml.gz and could be modified by an > administrator to be static after generation. This would require frequent > manual manipulation of the flow.xml.gz in various environments and frequent > restarts of the NiFi service. I do not recommend this. > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On May 23, 2020, at 11:05 AM, Andrew Grande > <mailto:apere...@gmail.com>> wrote: >> >> Aren't those IDs generated? How can one enforce it? >> >> Andrew >> >> On Sat, May 23, 2020, 10:53 AM Andy LoPresto > <mailto:alopre...@apache.org>> wrote: >> If you want the process to be completely automated, you would have to >> enforce the controller service IDs to be identical across environments. >> Otherwise deployment would need a manual intervention to reference the >> specific controller service in the proper component. >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> He/Him >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On May 22, 2020, at 3:57 PM, Eric Secules >> <mailto:esecu...@gmail.com>> wrote: >>> >>> Hi Andy, >>> >>> Given that you have a flow which operates on two different S3 accounts for >>> example, how would you do deployment automation? Do you mandate that the >>> controller service with the same ID must exist in both a development and >>> production environment rather than try to connect a processor to a matching >>> controller service? >>> >>> -Eric >>> >>> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto >> <mailto:alopre...@apache.org>> wrote: >>> Eric, >>> >>> I can’t answer all these questions but I would definitely have hesitations >>> around building an expectation that there is only one instance of any given >>> controller service type in an entire canvas. I can think of numerous flows >>> (this may not affect your particular flows, but the concepts still apply) >>> which require multiple instances of the same controller service type to be >>> available: >>> >>> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes >>> data, transforms it, and posts it to another mTLS API >>> * A flow which retrieves objects from one S3 bucket and puts them into an >>> S3 bucket in a different AWS account >>> * A flow which connects to one database and retrieves data, transforms it, >>> and persists it to another database >>> >>> If there is only _one_ StandardSSLContextService, >>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in >>> the entire controller, these flows cannot exist. >>> >>> I am not saying the retrieval of new flow versions and the matching of >>> referenced controller services cannot be improved, but I would definitely >>> advise caution before going too far down this path without considering all >>> possible side effects and potential constraints on future flow development. >>> >>> >>> >>> Andy LoPresto >>> alopre...@apache.org <mailto:alopre...@apache.org> >>> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >>> He/Him >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
Re: Connecting Controller Services Automatically
My position is that we don’t claim completely automated deployment as a feature, so manually setting the controller service IDs is not exposed. Technically, they are defined in the flow.xml.gz and could be modified by an administrator to be static after generation. This would require frequent manual manipulation of the flow.xml.gz in various environments and frequent restarts of the NiFi service. I do not recommend this. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 23, 2020, at 11:05 AM, Andrew Grande wrote: > > Aren't those IDs generated? How can one enforce it? > > Andrew > > On Sat, May 23, 2020, 10:53 AM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > If you want the process to be completely automated, you would have to enforce > the controller service IDs to be identical across environments. Otherwise > deployment would need a manual intervention to reference the specific > controller service in the proper component. > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On May 22, 2020, at 3:57 PM, Eric Secules > <mailto:esecu...@gmail.com>> wrote: >> >> Hi Andy, >> >> Given that you have a flow which operates on two different S3 accounts for >> example, how would you do deployment automation? Do you mandate that the >> controller service with the same ID must exist in both a development and >> production environment rather than try to connect a processor to a matching >> controller service? >> >> -Eric >> >> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto > <mailto:alopre...@apache.org>> wrote: >> Eric, >> >> I can’t answer all these questions but I would definitely have hesitations >> around building an expectation that there is only one instance of any given >> controller service type in an entire canvas. I can think of numerous flows >> (this may not affect your particular flows, but the concepts still apply) >> which require multiple instances of the same controller service type to be >> available: >> >> * A flow which invokes a mutually-authenticated TLS HTTP API, consumes data, >> transforms it, and posts it to another mTLS API >> * A flow which retrieves objects from one S3 bucket and puts them into an S3 >> bucket in a different AWS account >> * A flow which connects to one database and retrieves data, transforms it, >> and persists it to another database >> >> If there is only _one_ StandardSSLContextService, >> AWSCredentialsProviderControllerService, or DBCPConnectionPool available in >> the entire controller, these flows cannot exist. >> >> I am not saying the retrieval of new flow versions and the matching of >> referenced controller services cannot be improved, but I would definitely >> advise caution before going too far down this path without considering all >> possible side effects and potential constraints on future flow development. >> >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> He/Him >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On May 22, 2020, at 3:01 PM, Eric Secules >> <mailto:esecu...@gmail.com>> wrote: >>> >>> Hello everyone, >>> >>> I am running into an issue with automated deployment using nipyapi >>> <https://nipyapi.readthedocs.io/en/latest/>. We would like to be able to >>> pull down flows from a registry and have them ready to go once all their >>> controller services have been turned on. But there are a few issues. >>> Sometimes the flows that we download from the registry reference controller >>> service IDs that don't exist on this machine because the flow was developed >>> in a different environment. That's easy enough to fix if there is just one >>> applicable controller service, but not when there are two or more. >>> >>> We have taken the decision to put all our controller services at the top >>> level and have one of each kind we need, rather than have multiple of the >>> same controller service attached to individual process groups. >>> >>> We are running into a problem where some processors can either connect to a >>> JSONTreeReader or a CSVReader and there's no indication in the ProcessorDTO >>> object which type it was originally connected to, just a GUID of a >>> controller service that doesn't exist in this deployment. >>> >>> Would it be possible to include the type or name of the controller service >>> in the component.config.descriptors section? Are we going about it the >>> wrong way trying to simplify down to the least number of controller >>> services? >>> >>> Thanks, >>> Eric >> >
Re: Connecting Controller Services Automatically
If you want the process to be completely automated, you would have to enforce the controller service IDs to be identical across environments. Otherwise deployment would need a manual intervention to reference the specific controller service in the proper component. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 22, 2020, at 3:57 PM, Eric Secules wrote: > > Hi Andy, > > Given that you have a flow which operates on two different S3 accounts for > example, how would you do deployment automation? Do you mandate that the > controller service with the same ID must exist in both a development and > production environment rather than try to connect a processor to a matching > controller service? > > -Eric > > On Fri, May 22, 2020 at 3:44 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Eric, > > I can’t answer all these questions but I would definitely have hesitations > around building an expectation that there is only one instance of any given > controller service type in an entire canvas. I can think of numerous flows > (this may not affect your particular flows, but the concepts still apply) > which require multiple instances of the same controller service type to be > available: > > * A flow which invokes a mutually-authenticated TLS HTTP API, consumes data, > transforms it, and posts it to another mTLS API > * A flow which retrieves objects from one S3 bucket and puts them into an S3 > bucket in a different AWS account > * A flow which connects to one database and retrieves data, transforms it, > and persists it to another database > > If there is only _one_ StandardSSLContextService, > AWSCredentialsProviderControllerService, or DBCPConnectionPool available in > the entire controller, these flows cannot exist. > > I am not saying the retrieval of new flow versions and the matching of > referenced controller services cannot be improved, but I would definitely > advise caution before going too far down this path without considering all > possible side effects and potential constraints on future flow development. > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On May 22, 2020, at 3:01 PM, Eric Secules > <mailto:esecu...@gmail.com>> wrote: >> >> Hello everyone, >> >> I am running into an issue with automated deployment using nipyapi >> <https://nipyapi.readthedocs.io/en/latest/>. We would like to be able to >> pull down flows from a registry and have them ready to go once all their >> controller services have been turned on. But there are a few issues. >> Sometimes the flows that we download from the registry reference controller >> service IDs that don't exist on this machine because the flow was developed >> in a different environment. That's easy enough to fix if there is just one >> applicable controller service, but not when there are two or more. >> >> We have taken the decision to put all our controller services at the top >> level and have one of each kind we need, rather than have multiple of the >> same controller service attached to individual process groups. >> >> We are running into a problem where some processors can either connect to a >> JSONTreeReader or a CSVReader and there's no indication in the ProcessorDTO >> object which type it was originally connected to, just a GUID of a >> controller service that doesn't exist in this deployment. >> >> Would it be possible to include the type or name of the controller service >> in the component.config.descriptors section? Are we going about it the wrong >> way trying to simplify down to the least number of controller services? >> >> Thanks, >> Eric >
Re: Connecting Controller Services Automatically
Eric, I can’t answer all these questions but I would definitely have hesitations around building an expectation that there is only one instance of any given controller service type in an entire canvas. I can think of numerous flows (this may not affect your particular flows, but the concepts still apply) which require multiple instances of the same controller service type to be available: * A flow which invokes a mutually-authenticated TLS HTTP API, consumes data, transforms it, and posts it to another mTLS API * A flow which retrieves objects from one S3 bucket and puts them into an S3 bucket in a different AWS account * A flow which connects to one database and retrieves data, transforms it, and persists it to another database If there is only _one_ StandardSSLContextService, AWSCredentialsProviderControllerService, or DBCPConnectionPool available in the entire controller, these flows cannot exist. I am not saying the retrieval of new flow versions and the matching of referenced controller services cannot be improved, but I would definitely advise caution before going too far down this path without considering all possible side effects and potential constraints on future flow development. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 22, 2020, at 3:01 PM, Eric Secules wrote: > > Hello everyone, > > I am running into an issue with automated deployment using nipyapi > <https://nipyapi.readthedocs.io/en/latest/>. We would like to be able to pull > down flows from a registry and have them ready to go once all their > controller services have been turned on. But there are a few issues. > Sometimes the flows that we download from the registry reference controller > service IDs that don't exist on this machine because the flow was developed > in a different environment. That's easy enough to fix if there is just one > applicable controller service, but not when there are two or more. > > We have taken the decision to put all our controller services at the top > level and have one of each kind we need, rather than have multiple of the > same controller service attached to individual process groups. > > We are running into a problem where some processors can either connect to a > JSONTreeReader or a CSVReader and there's no indication in the ProcessorDTO > object which type it was originally connected to, just a GUID of a controller > service that doesn't exist in this deployment. > > Would it be possible to include the type or name of the controller service in > the component.config.descriptors section? Are we going about it the wrong way > trying to simplify down to the least number of controller services? > > Thanks, > Eric
Re: Use of SNI routing in Nifi ?
Thanks Pat. The S2S protocol uses TLS as a component, and attempts to use the highest protocol version supported by both endpoints. For Java 8, this should be TLSv1.2, and for Java 11, TLSv1.3 (introduced in upcoming NiFi 1.12.0). NiFi itself doesn’t support hosting multiple instances on the same port, so the only way I see this being applicable is if a load balancer/reverse proxy in front of NiFi + other services attempted to identify and route incoming traffic based on SNI. I tried to craft a realistic scenario for this email but I couldn’t get to a point where it made sense. If you have a specific desired scenario, I can try to analyze it, but the entire concept of having multiple NiFi services or NiFi + other services be exposed on the same port and use SNI to differentiate seems unnecessary to me. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 22, 2020, at 12:05 PM, Pat White wrote: > > Hi Andy, > Thanks very much for the feedback, and my apologies for being vague. I have > not used SNI so i have some learning to do. > > Specific use case we were asked about relates with Nifi to Nifi transfers, so > not the webservice itself but rather S2S. > I was wondering if S2S protocol supports SNI, and if so some pointers on how > to configure that. > > patw > > On Fri, May 22, 2020 at 1:14 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Hi Pat, > > Are you asking if NiFi’s internal web server supports SNI or if NiFi > processors/framework connecting to external services can resolve SNI? Maybe > some more context around your question would help us answer. > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On May 22, 2020, at 9:19 AM, Pat White > <mailto:patwh...@verizonmedia.com>> wrote: >> >> Hi Folks, >> >> Has anyone tried using SNI routing with Nifi? >> >> I believe Jetty supports the TLS extension for SNI but have not tried using >> it, would appreciate any feedback if someone has tried this. Thank you. >> >> >
Re: Use of SNI routing in Nifi ?
Hi Pat, Are you asking if NiFi’s internal web server supports SNI or if NiFi processors/framework connecting to external services can resolve SNI? Maybe some more context around your question would help us answer. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 22, 2020, at 9:19 AM, Pat White wrote: > > Hi Folks, > > Has anyone tried using SNI routing with Nifi? > > I believe Jetty supports the TLS extension for SNI but have not tried using > it, would appreciate any feedback if someone has tried this. Thank you. > >
Re: Secure Nifi with DigiCert
If this is the same question posted to the Slack channel earlier, I’ll reply here as well. Importing the .p12 file into your browser provides the client certificate identifying you as a user to the site. When you visit google.com, only one end of the connection (Google, the server) presents a certificate, which you the user (your browser) verify and decide to trust. When you visit a NiFi instance which is secured and has no other authentication mechanism configured, the only way to authenticate is to present a client certificate. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 19, 2020, at 7:24 PM, Ren Yang wrote: > > > >> Hi Nifi Team, >> Thanks for reading my email. I have encountered an issue of securing Nifi >> with Digicert issue. Could you please read the following details. >> >> I have got the Digicert related files and generated the keystore.jks and >> truststore.jks. And all other setup steps have finished. However, when I >> come to my nifi site with HTTPS URL, it denied. >> >> >> Next, I double clicked the nifi.p12 which generated by openssl command, >> imported it into Keychain access. >> >> >> >> Then I access my Nifi https url again, the cert confirmation window comes. >> After pressed “OK”, I arrived the Nifi home page. My question is why I need >> to manfully import the .p12 file into browser. Hasn’t it been working like >> any other public websites (such as https://www.google.com >> <https://www.google.com/>) without doing anything on client side? >> >> >> >> >> Please let me know if you have any questions. Awaiting for your reply. Thank >> you! >> >> > > > > > Ren Yang > ryang...@gmail.com <mailto:ryang...@gmail.com> > > >
Re: NiFi and real-time data lake
Thanks Boris, this is really interesting to read and I appreciate that you’re sharing it with the community. We’re glad NiFi can help with these important use cases. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 12, 2020, at 12:29 PM, Boris Tyukin wrote: > > Hi guys, > > wanted to share with you how we used Nifi to build near real-time data lake > for a large healthcare system > https://boristyukin.com/building-near-real-time-big-data-lake-part-2/ > <https://boristyukin.com/building-near-real-time-big-data-lake-part-2/> > > Our real-time infra was pretty crucial during Covid19 to support all kind of > analytics. > > I truly appreciate everyone who helped us answering our questions and > suggesting ideas. > > Matt Burgess (we love groovy and custom processors and your posts inspired > us!), Mike, Mark, Bryan, Koji, Andrew - I am sure I forgot someone. you guys > rock and amazing. You care about your baby project and your care about users > and devs who use it - thanks for that!! > > Boris
Re: Is provenance data preserved when processors are deleted?
Eric, The provenance exported via the reporting task does not contain the flowfile content. NiFi wasn’t designed as a long term store for the content or provenance data, but given appropriate resources, you can certainly increase the retention policies significantly. This is not an endorsement, but there are other metadata storage systems like Apache Atlas [1] which you may want to look at for longer retention and some of the features you’re looking for, like a UI for lineage graphs. [1] https://atlas.apache.org/#/ <https://atlas.apache.org/#/> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 5, 2020, at 11:39 PM, Eric Secules wrote: > > Thanks Mike, > > So the content gets sent over the wire too or just a content URI? I see that > the content gets aged out according to nifi.content.repository properties. > Given that the defaults for retention are so short would nifi crumble on a > long running system if the retention period is years and the available disk > space is huge? Shipping the provenance info off to MongoDB or something isn't > as attractive because we loose the provenance web UI and the ability to view > the configuration of a processor that a flowfile went through. > > Thanks, > Eric > > On Mon, May 4, 2020 at 4:13 PM Mike Thomsen <mailto:mikerthom...@gmail.com>> wrote: > It copies all of the provenance data, and no, there's no way yet to back the > provenance repository with one of those nosql databases yet unfortunately. > > On Mon, May 4, 2020 at 6:40 PM Eric Secules <mailto:esecu...@gmail.com>> wrote: > What information is transmitted by SiteToSiteProvenanceReporting? Is it the > content, the attributes, and the path the flowfile takes through the system? > Is there any way to connect the provenance view from NiFi to the nosql > database instead of the internal provenance storage? > > On Mon, May 4, 2020 at 3:07 PM Mike Thomsen <mailto:mikerthom...@gmail.com>> wrote: > One way to do it would be to set up a SiteToSiteProvenanceReporting task and > have it send the data to another NiFi instance. That instance can post all of > the provenance data into a NoSQL database like Mongo or Elasticsearch very > quickly. > > On Mon, May 4, 2020 at 5:47 PM Eric Secules <mailto:esecu...@gmail.com>> wrote: > Hello everyone, > > If I am upgrading a process group to the latest version, do you know whether > provenance is preserved for processors that may get deleted in the upgrade? > I have noticed that if I delete my process group and redownload it from the > registry, I am no longer able to see the provenance data from flowfiles that > went through the first process group. > > What is the best way to view and archive provenance data for older versions > of flows? For background I am running NiFi in a docker container. > I think I might have to archive the currently running container and bring the > new version up on a new container. > > Thanks, > Eric
Re: ConsumeIMAP certificates issue
Since the ConsumeIMAP processor does not expose an SSLContextService controller service to allow you to configure a custom truststore, it looks like the certificate verification is done internally in the underlying Spring library. I would try adding the public certificate of the IMAP server to the following truststores, one at a time, in this order: 1. JRE cacerts (copy the actual cacerts and ensure you have a backup before you start modifying it) 2. The NiFi truststore configured in nifi.properties Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 5, 2020, at 8:42 PM, Luis Carmona wrote: > > Hi guys, > > I have a project that needs to receive the mails flow from an Imap > server. > > If I try to read from port 993, get the error: > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > find valid certification path to requested target > > If I try to read from port 143, get the error: > > Unrecognized SSL message, plaintext connection > > > > As my mail server accepts only secure login, I presume it is claiming > about the corresponding certificate. > > The question is how to configure from where it has to read the > certificate ? > > > Thanks in advance. > > Regards, > > LC > > > >
Re: POST multipart/form-data with Invokehttp
ExecuteProcess and ExecuteStreamCommand both allow shell commands to be run; ExecuteProcess does not allow incoming flowfiles but ExecuteStreamCommand does. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 27, 2020, at 2:31 PM, Luis Carmona wrote: > > Hi Wesley, > > couldn't use Execute Processor as it doesn't receive an input FlowFile > ( or I didn't find out how to connect it ) and I need to give to the > procesor the file that should be sent. > > Thanks anyway. > > Will try with a script now. > > LC > > > > > On Mon, 2020-04-27 at 14:26 -0300, Wesley C. Dias de Oliveira wrote: >> Hello, Luis. >> >> Have you tried to send with ExecuteProcessor? >> >> >> >> Using that way you can invoke curl explicit to run your command. >> >> Em seg., 27 de abr. de 2020 às 14:21, Luis Carmona < >> lcarm...@openpartner.cl> escreveu: >>> Hi everyone, >>> >>> Hoping everybody is doing ok, wherever you are, need some help >>> please. >>> >>> Does anyone has sent a file and parameters to a REST point using >>> Invokehhtp with multipart/form-data as mime-type ? >>> >>> I can't figure out how to include the -F , speaking in >>> terms >>> of curl syntax. >>> >>> I really need this done throught NIFIso any help will be highly >>> apreciated. >>> >>> Thanks in advance. >>> >>> LC >>> >> >> >
Re: OIDC Redirect loop
Can you verify the initial redirect to OIDC and the callback are going to the
same node in NiFi? I see your LB configs are set to sticky sessions, but it may
be that if the callback is originating from the OIDC IDP server rather than the
actual client IP, the session affinity is not being applied. Regardless, the
error appears to indicate that the JWT provided in the request to NiFi isn’t
able to be validated, which indicates that the key used to sign it isn’t
present on that node, which is likely due to the request being sent to a node
other than the one that signed it.
Quick and easy way to validate this would be to change the stateful set # to 1
node and attempt the same sequence of operations.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 27, 2020, at 8:12 AM, Ami Goldenberg wrote:
>
> Hi Nathan,
> Indeed, that's the case
>
> On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough <mailto:thena...@gmail.com>> wrote:
> Hi Ami,
>
> Just to confirm, the OAuth Client ID redirect URL in OIDC is set to
> "https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback; and the
> NiFi property is set
> "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configu
> <https://accounts.google.com/.well-known/openid-configu>ration".
>
> Nathan
>
> On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <mailto:ami.g...@gmail.com>> wrote:
> Hi,
>
> We are trying to deploy NiFi on kubernetes after successfully using it for a
> while.
> The issue we are having is that every time we enter our nifi URL it will
> redirect us to Google and once we sign in we just get redirected again.
>
> The error I see on users.log is:
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET
> https://XXX.XXX./nifi-api/flow/current-user
> <https://xxx.xxx./nifi-api/flow/current-user> (source ip: 172.32.34.99)
> 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
> Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating
> the JWT
> 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
> Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access
> token.
> 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match
> locally computed signature. JWT validity cannot be asserted and should not be
> trusted.
> 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web
> Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web
> api:Unable to validate the access token.
>
> We're trying to follow practices from blogs and pvillard's repo:
> https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
>
> <https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc>
> https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
> <https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect>
> https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c
> <https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c>Our
> set up is as such:
> OIDC provider is Google
> TLS-toolkit running in server mode inside k8s
> StatefulSet of 3 replicas
> Zookeeper in K8s
> Ingress that is set up to create a load balancer in AWS - with sticky
> sessions (based on cookie)
> Service that is set up with sessionAffinity: ClientIP
>
> Any idea which direction I should be checking next?anks!
Re: UpdateAttribute processor question...
You might be able to wire something together using the Advanced mode and
chained conditionals, but I still don’t think order is enforced there, so even
with a rule of “b exists & is not empty” then “evaluate c”, it still might
occur before b is defined, so it wouldn’t throw an exception but would still
not provide the result you’re expecting.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 22, 2020, at 12:38 PM, Andy LoPresto wrote:
>
> Dan,
>
> Unfortunately I don’t believe there is a way to consolidate this in a single
> UA processor because currently the application of the attributes is not
> deterministically ordered, so b may not be available when c is evaluated and
> applied. The current work around is to use linear dependent processors as you
> are doing. I do think this is a valid feature request that could be
> introduced in the UA processor without breaking backwards compatibility if
> you’re interested in filing the ticket. Changing the internal data structure
> of the dynamic properties to be ordered _should_ be possible, but I think
> currently the API doesn’t request that order so it would require a code
> change there, with a default practice being “order as received".
>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Apr 22, 2020, at 12:04 PM, dan young > <mailto:danoyo...@gmail.com>> wrote:
>>
>> Hello,
>>
>> I haven't been able to figure out if this is doable with a single
>> UpdateAttrbute processor, but is it possible to create an attribute that
>> references a different attribute you're setting in the same UpdateAttribute
>> processor? ie.
>>
>> UpdateAttribute
>> a = "foo"
>> b = "bar"
>> c = ${allAttributes("a", "b"):join(" _")}
>>
>> What I've been doing in the past is just have c in a downstream
>> UpdateAttributewonder if there's a better way of doing this, or maybe in
>> the Advance section of the UpdateAttribute.
>>
>>
>> Regards,
>>
>> Dano
>>
>
Re: UpdateAttribute processor question...
Dan,
Unfortunately I don’t believe there is a way to consolidate this in a single UA
processor because currently the application of the attributes is not
deterministically ordered, so b may not be available when c is evaluated and
applied. The current work around is to use linear dependent processors as you
are doing. I do think this is a valid feature request that could be introduced
in the UA processor without breaking backwards compatibility if you’re
interested in filing the ticket. Changing the internal data structure of the
dynamic properties to be ordered _should_ be possible, but I think currently
the API doesn’t request that order so it would require a code change there,
with a default practice being “order as received".
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 22, 2020, at 12:04 PM, dan young wrote:
>
> Hello,
>
> I haven't been able to figure out if this is doable with a single
> UpdateAttrbute processor, but is it possible to create an attribute that
> references a different attribute you're setting in the same UpdateAttribute
> processor? ie.
>
> UpdateAttribute
> a = "foo"
> b = "bar"
> c = ${allAttributes("a", "b"):join(" _")}
>
> What I've been doing in the past is just have c in a downstream
> UpdateAttributewonder if there's a better way of doing this, or maybe in
> the Advance section of the UpdateAttribute.
>
>
> Regards,
>
> Dano
>
Re: Avro Single Object Binary Encoding
Hi Nathan, I don’t believe this has been addressed yet. Please file a Jira ticket [1] requesting this as a new feature/improvement and document as many of the specific requirements you’re encountering as possible to provide context around the development effort. It’s interesting a new feature was added in a x.y.z release, which indicates they may not be using semantic versioning and we would have to perform serious evaluation before upgrading the library to ensure no other behavior was changed. [1] https://issues.apache.org/jira/projects/NIFI/issues <https://issues.apache.org/jira/projects/NIFI/issues> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 20, 2020, at 2:59 AM, nathan.engl...@bt.com wrote: > > Hi There, > > Apologies if this has been asked before! > > One of our Inputs into our NIFI cluster is looking to start using Avro Single > Object Encoding on their messages. This was added V1.8.2 of the Avro Schema, > but from what I can see NIFI is using V1.8.1 (looking at the pom.xml on > GitHub, > https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml > > <https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml>) > of the Avro Library. Further information on the Single Object Encoding can > be found here > https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec > <https://avro.apache.org/docs/1.8.2/spec.html#single_object_encoding_spec>. > > Firstly, are there any plans to upgrade the Avro library in NIFI? Secondly > with an upgrade how and will NIFI support the Avro Single Object Encoding? > Has anybody implemented support for this in their NIFI clusters? > > Once again apologies if this has been asked before. > > Kind Regards, > > Nathan
Re: Include parent fields into the output record fields in XML data
Can you share your XPath and XQuery properties? I think this should be possible with queries that return an array of results. If the results are in multiple attributes, you may be able to recombine them in the way you want using ExecuteScript or a ScriptedRecordSetWriter to translate them to CSV. You can also use the QueryRecord processor to perform SQL-like queries over large datasets in a flowfile which might be helpful in forming the output you’re looking for. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 17, 2020, at 6:26 PM, ALAM Mahabub > wrote: > > Hello All, > I am new in NiFI. I have below nested xml file and I need to keep the parents > node with its multiple record in a same table. I > already able to separate them but not able to align them in a same flow file > attributes record. > > So, it will be highly appreciated if anyone please help me how can I Include > parent fields into the output record and what will be my > flow file? > > track_id > track_name > swithc_id > track_continue_course > pos > 1 > TR_3B_ASW_ITW > 2 > Straight > 554.05 > 1 > TR_3B_ASW_ITW > 3 > Straight > 2654.64 > 1 > TR_3B_ASW_ITW > 4 > Straight > 2767.56 > … > … > … > … > … > > XML file: > > > > NiFi flow: > In the left flow file, I am able to split track id=1, name=”TR_3B_ASW_ITW” > In the right flow file, I am able to split records with switch id=2, > trackContinueCourse=”straight”, pos=”544.05” etc. > > Output needed: > > track_id > track_name > swithc_id > track_continue_course > pos > 1 > TR_3B_ASW_ITW > 2 > Straight > 554.05 > 1 > TR_3B_ASW_ITW > 3 > Straight > 2654.64 > 1 > TR_3B_ASW_ITW > 4 > Straight > 2767.56 > … > … > … > … > … > > > > > Regards, > Mahabub ALAM > > CONFIDENTIALITY : This e-mail and any attachments are confidential and may be > privileged. If you are not a named recipient, please notify the sender > immediately and do not disclose the contents to another person, use it for > any purpose or store or copy the information in any medium.
Re: Storing output of shellscript in s3 bucket
If that script cannot write directly to STDOUT in this scenario (which the docs do not make clear), there are multiple ways to achieve what you need. 1. Instead of invoking the command directly in the ExecuteStreamCommand processor, invoke a custom script file “example_zk_script.sh zk_source_data.json” which contains: ``` zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/“$@“ cat /path/to/export/"$@" ``` This script will execute the command you were originally calling (which sends data _to_ ZooKeeper that is read _from_ the file location you provided) and then output the file contents to STDOUT, which will be captured as the flowfile content. 2. Read directly from the file using a FetchFile processor. The path to the file is known in the context of NiFi, either as a static value in the ExecuteStreamCommand property descriptor, or as an attribute or variable, so that file path can be read from directly using another processor. The FetchFile processor replaces a flowfile’s content with the content of a file on disk, so pass the result of the ExecuteStreamCommand to the FetchFile processor with the path in the filename attribute (update using an UpdateAttribute processor if necessary). I would recommend the first option as a cleaner and more robust solution. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 10, 2020, at 7:19 PM, Joe Witt wrote: > > If the zk-migrator can be configured to write to stdout instead of a file > then yes. > > On Fri, Apr 10, 2020 at 3:52 PM sanjeet rath <mailto:rath.sanj...@gmail.com>> wrote: > Hi, > > Thanks for your quick reply.Yeah i am using executestreamcommand to execute > bellow script > > zk-migrator.sh -s -z > destinationHostname:destinationClientPort/destinationRootPath/components -f > /path/to/export/zk-source-data.json > > Can the zk-source-data.json file written as a output flow file of the above > processor.if yes please let me know how. > > Many thanks > sanjeet > > On Fri, 10 Apr 2020, 9:25 pm Bryan Bende, <mailto:bbe...@gmail.com>> wrote: > Hello, > > Assuming you are using the ExecuteStreamCommand processors, then the output > of the command is written to the flow file content. So if your command writes > the JSON to stdout, then it should end up in the flow file. > > Thanks, > > Bryan > > > On Fri, Apr 10, 2020 at 11:23 AM sanjeet rath <mailto:rath.sanj...@gmail.com>> wrote: > Hi, > > I have a scenario , where i have to Execute a shell script amd the output of > the script is a json file and i want to put the file in s3 bucket. > > I am able to do it by building 2 flows. > > One flow for executestremproces and store the json file in folder in System > > Then another flow to get the file from system and using puts3object to put in > s3 bucket. But building write and read in separate flow is bit risky and > noway i am make it dependent as getfile has no preprocessor. > > > Is it possible not to store the json file (output of shell script) in file > system and as a flow file move it to s3 bucket ? > Means in one flow to execute shellscript and store output in s3 bucket. > > Regards, > Sanjeet >
Re: How to iterate over all dynamic properties in python invokeScriptedProcessor?
My Python is not great but I would look at NiPyAPI as an example and the NiFi
REST API [1] to see how objects are nested. The ProcessorConfigDTO [2] contains
a dict of str: PropertyDescriptorDTO at descriptors, so you could iterate over
that as you’re asking. For example, the list_sensitive_processors function [3]
operates very similarly.
def list_sensitive_processors(pg_id='root', summary=False):
"""
Returns a flattened list of all Processors on the canvas which have
sensitive properties that would need to be managed during deployment
Args:
pg_id (str): The UUID of the Process Group to start from, defaults to
the Canvas root
summary (bool): True to return just the list of relevant
properties per Processor, False for the full listing
Returns:
list[ProcessorEntity] or list(dict)
"""
assert isinstance(pg_id, six.string_types), "pg_id should be a string"
assert isinstance(summary, bool)
cache = nipyapi.config.cache.get('list_sensitive_processors')
if not cache:
cache = []
matches = []
proc_list = list_all_processors(pg_id)
for proc in proc_list:
if proc.component.type in cache:
matches.append(proc)
else:
sensitive_test = False
for _, detail in proc.component.config.descriptors.items():
if detail.sensitive is True:
sensitive_test = True
break
if sensitive_test:
matches.append(proc)
cache.append(str(proc.component.type))
if cache:
nipyapi.config.cache['list_sensitive_processors'] = cache
if summary:
return [
{x.id: [
p for p, q in x.component.config.descriptors.items()
if q.sensitive is True]}
for x in matches
]
return matches
[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
<https://nifi.apache.org/docs/nifi-docs/rest-api/index.html>
[2]
https://github.com/Chaffelson/nipyapi/blob/master/nipyapi/nifi/models/processor_config_dto.py
<https://github.com/Chaffelson/nipyapi/blob/master/nipyapi/nifi/models/processor_config_dto.py>
[3] https://github.com/Chaffelson/nipyapi/blob/master/nipyapi/canvas.py#L225
<https://github.com/Chaffelson/nipyapi/blob/master/nipyapi/canvas.py#L225>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 18, 2020, at 12:12 PM, Eric Chaves wrote:
>
> Hi folks,
>
> I'm trying to write a quick python InvokeProcessorScript where I need to
> iterate over all Dynamic Properties from the processor to select just a few
> and I'm having some difficulties with the class types between Jython and Java.
>
> Can someone show me how to iterate over "context.properties" to get each
> PropertyDescriptor?
>
> I'd like do something like this:
> for prop in context.properties:
> name = prop.name <http://prop.name/>
> value =
> context.getProperty(prop).evaluateAttributeExpressions(flowFile).getValue()
> self.log.info <http://self.log.info/>("attr {name}:
> {value}".format(name=name, value=value))
> if prop.dynamic:
> if name in lista and re.search(value, filename):
> attrMap['TipoArquivo'] = name
> else:
> attrMap[name] = value
>
> Cheers
Re: alternate web root directory?
Hi Wyllys, As I started reading, I was going to suggest the proxy approach. Unfortunately, at this time I am unaware of any way to change the paths within NiFi itself - there would be substantial refactoring required to make that an option. You can open a feature request Jira for that, or perhaps the ability to inject a path prefix, but I expect it to be a high level of effort to implement. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll > wrote: > > > Is there a way to configure nifi to use a different root directory for web > requests? > > We would like everything to be under a common root such as: > /XXX/nifi/... > /XXX/nifi-api/... > > Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to > setup a reverse proxy that also can incorporate OpenID authentication tokens > to a secure backend cluster of nodes. > >
Re: Nifi - Python SDKs
You can use a number of processors to interact with Python. The ExecuteScript and InvokeScriptedProcessor components allow you to write Python (actually Jython) code and have it be executed by a JSR-223 script execution engine within the same NiFi JVM. This allows the script to be persisted in the NiFi flow directly and be reusable without external dependencies. You can also use the ExecuteProcess or ExecuteStreamCommand processors to invoke shell commands, including calling an external Python script. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 10, 2020, at 4:53 AM, Karthick Subramanian > wrote: > > Hi, > > Can anyone point me to any sources or samples on how to use python SDK > (Google Analytics, Facebook etc offers client SDK to access their API) inside > NiFi. > > Regards, > K'sub
Re: zookeeper connection string question/clarification
Thanks Dan. If it works, we can update the MG with that example as well. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 20, 2020, at 3:28 PM, dan young wrote: > > ok, great thank you. Yes, we're using external zookeeper; 3.5.6. I'm going > to test this change out on a dev cluster real quick. > > On Thu, Feb 20, 2020 at 4:22 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Sorry, I should have elaborated that I was referencing the link from the MG. > I realize you’re using external ZK and this is for embedded. Yes, I believe > you will need to change the format of your connection string. > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Feb 20, 2020, at 3:20 PM, Andy LoPresto > <mailto:alopre...@apache.org>> wrote: >> >> Hi Dan, >> >> I believe the changes you’re looking for are here [1], copied below: >> >> The Zookeeper dependency that NiFi uses for state management and cluster >> elections was upgraded to v3.5.5. From v3.5.x onwards, Zookeeper changed the >> zookeeper.properties file format and as a result NiFi users using an >> existing embedded zookeeper will need to adjust their existing >> zookeeper.properties file accordingly. More details here: >> https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_clientport >> >> <https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_clientport>. >> For new deployments of the 1.10.0 release onwards, NiFi will be packaged >> with an updated template zookeeper.properties file. >> To update an existing zookeeper.properties file however, edit the >> conf/zookeeper.properties file: >> Remove the clientPort=2181 line (or whatever your port number may be) >> Add the client port to the end of the server string eg: >> server.1=localhost:2888:3888;2181 >> >> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance >> <https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance> >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Feb 20, 2020, at 3:18 PM, dan young >> <mailto:danoyo...@gmail.com>> wrote: >>> >>> Hello, >>> >>> Using Nifi 1.11.1 in cluster mode with external zookeeper. Does the >>> nifi.zookeeper.connect string in the nifi.properties need to change from >>> say: >>> >>> >>> nifi.zookeeper.connect.string=10.xxx.x.xxx:2181,10.xxx.x.xxx:2181,10.xxx.x.xxx:2181 >>> >>> >>> nifi.zookeeper.connect.string=10.xxx.x.xxx;2181,10.xxx.x.xxx;2181,10.xxx.x.xxx;2181 >>> >>> Changing the : to ; between the host and client port? >> >
Re: zookeeper connection string question/clarification
Sorry, I should have elaborated that I was referencing the link from the MG. I realize you’re using external ZK and this is for embedded. Yes, I believe you will need to change the format of your connection string. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 20, 2020, at 3:20 PM, Andy LoPresto wrote: > > Hi Dan, > > I believe the changes you’re looking for are here [1], copied below: > > The Zookeeper dependency that NiFi uses for state management and cluster > elections was upgraded to v3.5.5. From v3.5.x onwards, Zookeeper changed the > zookeeper.properties file format and as a result NiFi users using an existing > embedded zookeeper will need to adjust their existing zookeeper.properties > file accordingly. More details here: > https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_clientport > > <https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_clientport>. > For new deployments of the 1.10.0 release onwards, NiFi will be packaged with > an updated template zookeeper.properties file. > To update an existing zookeeper.properties file however, edit the > conf/zookeeper.properties file: > Remove the clientPort=2181 line (or whatever your port number may be) > Add the client port to the end of the server string eg: > server.1=localhost:2888:3888;2181 > > [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance > <https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance> > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Feb 20, 2020, at 3:18 PM, dan young > <mailto:danoyo...@gmail.com>> wrote: >> >> Hello, >> >> Using Nifi 1.11.1 in cluster mode with external zookeeper. Does the >> nifi.zookeeper.connect string in the nifi.properties need to change from say: >> >> >> nifi.zookeeper.connect.string=10.xxx.x.xxx:2181,10.xxx.x.xxx:2181,10.xxx.x.xxx:2181 >> >> >> nifi.zookeeper.connect.string=10.xxx.x.xxx;2181,10.xxx.x.xxx;2181,10.xxx.x.xxx;2181 >> >> Changing the : to ; between the host and client port? >
Re: zookeeper connection string question/clarification
Hi Dan, I believe the changes you’re looking for are here [1], copied below: The Zookeeper dependency that NiFi uses for state management and cluster elections was upgraded to v3.5.5. From v3.5.x onwards, Zookeeper changed the zookeeper.properties file format and as a result NiFi users using an existing embedded zookeeper will need to adjust their existing zookeeper.properties file accordingly. More details here: https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_clientport <https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_clientport>. For new deployments of the 1.10.0 release onwards, NiFi will be packaged with an updated template zookeeper.properties file. To update an existing zookeeper.properties file however, edit the conf/zookeeper.properties file: Remove the clientPort=2181 line (or whatever your port number may be) Add the client port to the end of the server string eg: server.1=localhost:2888:3888;2181 [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance <https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 20, 2020, at 3:18 PM, dan young wrote: > > Hello, > > Using Nifi 1.11.1 in cluster mode with external zookeeper. Does the > nifi.zookeeper.connect string in the nifi.properties need to change from say: > > > nifi.zookeeper.connect.string=10.xxx.x.xxx:2181,10.xxx.x.xxx:2181,10.xxx.x.xxx:2181 > > > nifi.zookeeper.connect.string=10.xxx.x.xxx;2181,10.xxx.x.xxx;2181,10.xxx.x.xxx;2181 > > Changing the : to ; between the host and client port?
Re: NiFi user and access rights
You could use MiNiFi agents on each external resource to consume data in a siloed manner and transmit it to a central NiFi instance over Site-to-site protocol. This would allow each producer of data to remain isolated (either physically disconnected or each using a distinct OS user for ACL with the respective MiNiFi agents running as that user) and communicate the necessary data back to a central processing instance. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 12, 2020, at 6:54 AM, Tomislav Novosel wrote: > > Hi guys, > > I'm having this situation inside my company projects. We are using NiFi as > DataFlow platform and there are multiple projects. > Every project has files on shared disk/folder from which one Nifi > instance(standalone instance) is reading data. > NiFi instance service is running under one generic user which has read rights > for every shared folder/project and that is fine. > > As there will be more and more projects and only one generic user will need > to have read rights on all shared disks/folders of all projects. So which is > better solution: > > To have one NiFi instance running with one generic user which has read rights > on all shared disks/folders. From security standpoint it is not ok. Shared > folders are from various customers. Data volume and load is not too big for > only one standalone NiFi instance. > To have Multiple NiFi instances on one server each running under different > generic user and every generic user belongs to one customer shared folder > regarding read rights, 1:1 relationship. > In the future there will be need to scure NiFi instances with SSL, maybe to > add more nodes and to establish multi-tenancy. > > Is there maybe some other third solution for this situation? How to setup > that kind of data flow where are multiple data sources and security is > important? > > Thanks in advance and best regards. > > Tom
Re: Can jetty reload keystore credentials dynamically?
This is available in Jetty versions 9.3+ [1], but in NiFi this is not currently supported. I have filed a number of enhancement Jiras [2] to improve the TLS handling throughout the application, and now that encrypted repositories are available, hope to address some of these in the near future. Please file a Jira for this specifically and include it in the linked epic. [1] https://github.com/eclipse/jetty.project/issues/918 <https://github.com/eclipse/jetty.project/issues/918> [2] https://issues.apache.org/jira/browse/NIFI-5458 <https://issues.apache.org/jira/browse/NIFI-5458> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 11, 2020, at 12:23 PM, Pat White wrote: > > Hi Folks, > > Can Nifi's jetty automatically detect and reload its keystore when the > keystore is changed, such as during credentials update or rotation? > > Thank you > patw
Re: NiFi Cluster setup
Anurag, There is extensive documentation on the NiFi site under the User Guide [1] and Admin Guide [2]. Pierre has also written a number of articles around setting up a cluster [3] and monitoring status [4]. [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#monitoring <https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#monitoring> [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering> [3] https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/ <https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/> [4] https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/ <https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 25, 2020, at 6:29 AM, Anurag Sharma wrote: > > Hi All, > > Just getting started with NiFi. Could you please pass on resources about best > practices around cluster setup and monitoring. > > Regards > Anurag > > This message may contain confidential and/or privileged information. If you > are not the addressee or authorized to receive this for the addressee, you > must not use, copy, disclose, or take any action based on this message or any > information herein. If you have received this message in error, please > advise the sender immediately by reply e-mail and delete this message. The > opinion expressed in this mail is that of the sender and do not necessarily > reflect that of Quicko Technosoft Labs Private Limited. Thank you for your > co-operation.
Re: Need help for TLS implementation with CA signed certificates
You will likely want to read the TLS Toolkit Guide [1] closely, especially the section for using an external CA [2] or externally-signed certificates [3]. Whether generated using the TLS Toolkit or provided from an external source (i.e. your organization generates signed certificates per NiFi node), the truststore for each node needs to be aware of the presented public certificates of every other node. Therefore, the easiest solution is to use a common intermediate CA to sign all node certificates and import the public certificate of the CA into a single truststore which is present on every node. However, you can use external certificates, provided that either each public certificate is populated in every truststore, or a common ancestor of all node certificates is. When you generate the certificates using TLS Toolkit, the resulting files in the output directories each contain the keystore, truststore, and generated nifi.properties file for a single node. The generated nifi.properties file has been populated with the keystore and truststore locations and passwords, and that file is based either on a generic template or the existing nifi.properties file being used by the node, depending on the command-line flags provided. If it uses the actual nifi.properties file, you can copy this newly-generated file directly into the conf/ directory. However, if you use the generic template or there are additional changes you need to make (for example, you generate the certificates one at a time and don’t configure the cluster settings), you’ll have to merge these changes manually. Importing nifi-cert.pem (the public certificate of the NiFi CA) into the truststore for each node is already part of the toolkit process. However, if you run the toolkit command independently on each node, it will generate a unique CA certificate on each node, and you will have to cross-import these CA certs into every truststore. Again, the recommended process is to generate all of the certs at once in the same location, thus using the same CA cert to sign all the certificates, or use the client/server mode to generate a single CA cert in one node and use it to sign all other certificates. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit <https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit> [2] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_intermediate_ca [3] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_external-signed_ca Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 24, 2019, at 8:02 AM, Krishnakishore Ganta > wrote: > > Hi All, > > I am implementing TLS for NIFI and we are able to connect to primary node > with https and nifi page is displaying cluster with 1/3 status. We observed > following errors in nifi-app.log file - > > 2019-12-23 14:01:47,286 WARN [main] o.a.nifi.controller.StandardFlowService > Failed to connect to cluster due to: > org.apache.nifi.cluster.protocol.ProtocolException: Failed to create socket > to node03:9081 due to: java.net.ConnectException: Connection refused > (Connection refused) > 2019-12-23 14:01:52,288 INFO [main] > o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster > Coordinator is located at node03:9081; will use this address for sending > heartbeat messages > 2019-12-23 14:01:52,367 WARN [main] o.a.nifi.controller.StandardFlowService > Failed to connect to cluster due to: > org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling > 'CONNECTION_REQUEST' protocol message due to: > javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown > 2019-12-23 14:01:57,371 INFO [main] > o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster > Coordinator is located at node03:9081; will use this address for sending > heartbeat messages > 2019-12-23 14:01:57,392 WARN [main] o.a.nifi.controller.StandardFlowService > Failed to connect to cluster due to: > org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling > 'CONNECTION_REQUEST' protocol message due to: > javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown > 2019-12-23 14:02:02,395 INFO [main] > o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster > Coordinator is located at node03:9081; will use this address for sending > heartbeat messages > 2019-12-23 14:02:02,409 WARN [main] o.a.nifi.controller.StandardFlowService > Failed to connect to cluster due to: > org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling > 'CONNECTION_REQUEST' protocol message due to: > javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_u
Re: SSLContextService configuration
I am not sure I follow all of the issues you are describing, but I will try to address them as I understand them. In the 1.10.0 release, parameters [1] were introduced, which allow any component property to use a reference to externally-defined values (this does not rely on Expression Language, which must be explicitly enabled for each property, and allows for sensitive parameter values so it can be used for passwords). You should be able to define a single controller service with the proper parameters specified for the passwords, and then each environment will have a parameter context defined once which contains the appropriate passwords. Any time a new flow is loaded or updated that references the controller service, no changes will be required. We do not recommend using templates for flow deployment as NiFi Registry [2] is available and is a more robust solution. There is currently no plan for a 1.9.3 release, and even if it came to fruition, it would not include this behavior as it would be a bug fix only, not a feature-bearing release. We use semantic versioning [3], so the next release which would contain new features is 1.11.0. I do not anticipate adding a feature for a “proxy” controller service which wraps another controller service in this manner because I don’t see this addressing the problem you have. I believe there was a fix [4] in the most recent version of NiFi Registry which addresses a similar issue. [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters <https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters> [2] https://nifi.apache.org/docs/nifi-registry-docs/index.html <https://nifi.apache.org/docs/nifi-registry-docs/index.html> [3] https://semver.org <https://semver.org/> [4] https://issues.apache.org/jira/browse/NIFIREG-288 <https://issues.apache.org/jira/browse/NIFIREG-288> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 3, 2020, at 10:50 AM, Etienne Jouvin wrote: > > Hello all. > > Those last days, I spent some times to deploy process to my client, using a > Template. > In the template, I have many InvokeHTTP processors, and some services related > to the SSLContextService. > > My client have 3 environments, and for two of them, I can not configure the > SSLContextService, because I do not have to know password for keystore and > trustore. > > So we decide to setup a SSLContextService at "root" level in NiFi once for > all. > Each time I deploy the Template, a new service is deployed (this was a > previous question by someone here). > I just have to delete the serice created during the template import. And then > modify all processors. > > I was thinking of something than my help me and ask here if you think I could > be nice to have it in future release (ideally in 1.9.3, if planned) > We could have a sort of "proxy" for SSLContextService. The only property > would be an instance of SSLContextService. > And each call on the proxy will be a delegation to to "wrapped" instance. > > Like this, during deploy, I will just have to update the instance set on the > "proxy". > For other usage, we will be able to switch easly between SSL Context. > > The problem could be the implementation that may produce circular reference. > But it is not our fault if user make things stupid. > > > What do you think about that ? > > > Regards > > Etienne Jouvin >
Re: Clarification regarding NiFi registry
The NiFi Registry documentation [1] explains the interaction between the two services in more detail, but the basic process is that a Registry Client is configured in NiFi which contains the information necessary for NiFi to connect to the registry. Assets (currently flow definitions or extensions) are persisted or retrieved to/from the registry instance over HTTP(S). In NiFi Registry, those assets are persisted in either Git or a relational database configured locally. When NiFi restarts, the flow controller (global scheduling system) uses the configured Registry Client to connect to the registry instance as necessary. [1] https://nifi.apache.org/docs/nifi-registry-docs/index.html <https://nifi.apache.org/docs/nifi-registry-docs/index.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 10, 2019, at 1:51 PM, sanjeet rath wrote: > > Hi , > > I just have confusion but no where i am finding a proper documentation > regarding how nifi nodes intigrates with nifi registry. means internally how > it works.as <http://works.as/> per the code , the node calls API to get the > registry details and storing the entiry in RegistryDTO object. > > So my question is , Is it storing in persistant storage like db or in memory. > If it stores in memory, then when we restarting the nifi application how it > again connect to registry , is it again call the API . > > I am sorry if my understanding is wrong. > please let me know. > > Thanks in advance > Regards, > Sanjeet
Re: Encrypting passwords - Nifi 1.10.0
Thanks Juan. A couple notes: Using the same plaintext value for multiple keys will not cause a technical problem, but it is bad security practice and is strongly discouraged. It would not be the source of the issue here (however, you need to use a fully-formed AES key for the provenance encryption key, and it’s unlikely that would be the same value or format as a password for the sensitive properties. That can cause other problems later on). As you are using the plain WriteAheadProvenanceRepository and not the EncryptedWriteAheadProvenanceRepository, you do not need to provide (and in fact, they are currently ignored) any properties for nifi.provenance.encryption.*. So you can remove those lines entirely (and probably should just for clarity and not to confuse anyone else who looks at these properties). If you want to use the encrypted repository, you’ll need to change the repository implementation (see step-by-step details in the link I provided earlier). The nested exception was that one of the encrypted properties did not contain the “||” delimiter. From visual inspection, it appears that all properties you have listed here do contain the delimiter. That exception is only thrown in one condition, and that is a simple string contains check for the delimiter. Are you sure these are the only encrypted values in your nifi.properties file, and that you are referencing the correct file? Can you look for any other entries of the form “nifi.xyz.protected=“? You mentioned that it generates two unique entries for “nifi.provenance.repository.encryption.key” and you remove the plaintext one. Are you sure that is being removed? If the system believes that property is encrypted (as indicated by the nifi.provenance.repository.encryption.key.protected=aes/gcm/256” line following it) and tries to decrypt the plaintext value, that would cause the exception to be thrown. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 9, 2019, at 2:22 PM, Juan Pablo Gardella > wrote: > > Thanks for answering my questions Andy, > > Below are the sensitive properties: > > # Provenance Repository Properties > nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository > nifi.provenance.repository.debug.frequency=1_000_000 > nifi.provenance.repository.encryption.key=fbRg/ZgK7U8qJcrU||4nI1n1aRD0Tooq7TLSTyVDhkmX8 > nifi.provenance.repository.encryption.key.protected=aes/gcm/256 > nifi.provenance.repository.encryption.key.provider.location= > nifi.provenance.repository.encryption.key.id > <http://nifi.provenance.repository.encryption.key.id/>= > # security properties # > nifi.sensitive.props.key=jtZiGY+mZyHPQIc1||/IJnMQBBXKN7VNkwMf6Oo7vZmAs > nifi.sensitive.props.key.protected=aes/gcm/256 > nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL > nifi.sensitive.props.provider=BC > nifi.sensitive.props.additional.keys= > > nifi.security.keystore=/opt/certs/keystore.jks > nifi.security.keystoreType=JKS > nifi.security.keystorePasswd=GuuOm4fyK6yvo76H||av/NQmH7Hw8qK9k0NOMRSjp08tw+walt4D5JLpYPiCHG/Z7DDq5QZ+ui/dKOXxtapH76Gjpt3hMwmP0 > nifi.security.keystorePasswd.protected=aes/gcm/256 > nifi.security.keyPasswd=y4spsJvsy5Fzc3Uq||Q1vMntNgfLLMMSJuyPNn8+9aHlH+banQy82Ly0qrLWf6hNUTNgA+akyh86rlf2J5XZCONL3JCLX6mY0 > nifi.security.keyPasswd.protected=aes/gcm/256 > nifi.security.truststore=/opt/certs/truststore.jks > nifi.security.truststoreType=JKS > nifi.security.truststorePasswd=9r+fyOSjRUXQLcZG||YwAtPYorADqHSKFUmU4H3SbyqvYqqYNZiGidgCOUCibPdP2jiEAMGtLt5xyFsMcNPm5Pye2qXEioLR8 > nifi.security.truststorePasswd.protected=aes/gcm/256 > > These properties are generated by the toolkit. I using the same value for > nifi.sensitive.props.key value and the > nifi.provenance.repository.encryption.key, I was not aware they should be > different. Could be that the problem? > > Juan > > On Mon, 9 Dec 2019 at 08:20, Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Hi Juan, > > The error you are getting is saying that one of the protected properties is > not of the expected format. While the Sensitive Property Provider mechanism > is extensible (see NIFI-5481 [1] for additional options being added), the > only natively supported one in 1.10.0 is AES/GCM encryption. This requires > the sensitive properties to be in the format > > Wl9bXjSWX5DXs4Gm||EDnf18wwAAMJFckgNNfkRWiA4daSDWJCuRvSsbe99AaefQrkpmSqehJtyJGgEbhn402zSyztXi1EGPU > > Where the segment preceding the “||” delimiter is the Base64-encoded 16 byte > initialization vector (IV), which is random and unique for each property, and > the segment following the delimiter is the Base64-encoded cipher text. > > The error states that wh
Re: Encrypting passwords - Nifi 1.10.0
Hi Juan,
The error you are getting is saying that one of the protected properties is not
of the expected format. While the Sensitive Property Provider mechanism is
extensible (see NIFI-5481 [1] for additional options being added), the only
natively supported one in 1.10.0 is AES/GCM encryption. This requires the
sensitive properties to be in the format
Wl9bXjSWX5DXs4Gm||EDnf18wwAAMJFckgNNfkRWiA4daSDWJCuRvSsbe99AaefQrkpmSqehJtyJGgEbhn402zSyztXi1EGPU
Where the segment preceding the “||” delimiter is the Base64-encoded 16 byte
initialization vector (IV), which is random and unique for each property, and
the segment following the delimiter is the Base64-encoded cipher text.
The error states that when NiFi tries to decrypt one of the five encrypted
properties (it does not specify which in this case), it is not encoded in the
proper form. Assuming you are using a strong key for
nifi.bootstrap.sensitive.key in conf/bootstrap.conf, you can share the
nifi.properties file with the encoded and encrypted values with this list to be
verified for format, as no one will be able to decrypt them. However, if you do
not wish to share them, please validate that they are all of the format
specified above and encrypted with the same key that is present in
bootstrap.conf.
Another thing I noted is that you are replacing the nifi.sensitive.props.key
value and the nifi.provenance.repository.encryption.key value with the same
environment variable. These keys should not have the same value. The provenance
repository key is designed to protect the provenance repository on disk and be
rotated/migrated automatically. The formatting and provision of these keys is
documented in the User Guide [2]. The key can be present in plaintext (raw
hexadecimal encoding) or encrypted as any other sensitive configuration value
in the nifi.properties file.
The nifi.sensitive.props.key value is a password or other key derivation
material used by NiFi to derive a strong key to encrypt the sensitive
_property_ values - this means things like database passwords, FTP server
passwords, keystore passwords, etc. that the NiFi flow uses and persists in an
encrypted format in the flow.xml.gz file.
If you believe the sensitive properties key you are injecting into the file is
in the correct format (encoded as described above), check the value of your
master key to ensure it is the same key that encrypted that value. If you are
injecting a plaintext value like “my_bad_sensitive_props_password”, you must
remove the master key from the bootstrap.conf file and ensure there is no
sibling property present called NiFi.sensitive.props.key.protected which
indicates that the value must be decrypted.
I.e. the existing section like:
nifi.sensitive.props.key=xPqEWK8a34r19J4z||UOFzOfZE/NQK4Xua8WWblf1/Ld+Pf7eQ1zg0U/qYW2sPwxyhhOXWwQmrUft6qA
nifi.sensitive.props.key.protected=aes/gcm/128
Should change to look like:
nifi.sensitive.props.key=my_bad_sensitive_props_password
NiFi.sensitive.props.key.protected= # or remove this line entirely
[1] https://github.com/apache/nifi/pull/3672
<https://github.com/apache/nifi/pull/3672>
[2]
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#encrypted-provenance
<https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#encrypted-provenance>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Dec 8, 2019, at 8:01 PM, Juan Pablo Gardella
> wrote:
>
> Hello all,
>
> I am trying to protect plain text passwords. I am using the latest docker
> image (1.10.0), and edited manually nifi.sensitive.props.key as below
>
> sed -i -e
> "s|^nifi.sensitive.props.key=.*$|nifi.sensitive.props.key=${NIFI_SENSITIVE_PROPS_KEY}|"
> /opt/nifi/nifi-current/conf/nifi.properties
> sed -i -e
> "s|^nifi.provenance.repository.encryption.key=.*$|nifi.provenance.repository.encryption.key=${NIFI_SENSITIVE_PROPS_KEY}|"
> /opt/nifi/nifi-current/conf/nifi.properties
>
> (this command for some reason does not update the file inside the Dockerfile,
> I have to do inside the container).
>
> After updated that property, I run following command inside the container:
>
> bash /opt/nifi/nifi-toolkit-current/bin/encrypt-config.sh -n
> /opt/nifi/nifi-current/conf/nifi.properties -b
> /opt/nifi/nifi-current/conf/bootstrap.conf -a
> /opt/nifi/nifi-current/conf/authorizers.xml -l
> /opt/nifi/nifi-current/conf/login-identity-providers.xml
>
> It prompts to put a master password and after that, I restart[1] the
> container but it failed to start with below error:
>
> nifi | 2019-12-08 18:57:31,777 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 162 properties from
> /opt/nifi/nifi-current/./conf/nifi.properties
> nifi | 2019-12-08 18:57:31,933
Re: NiFi Upgrade 1.9.2 to 1.10.0 - LDAP Failure
Hi Josef, My inclination is that somehow the password NiFi is trying to send to the LDAP service is no longer sufficiently protected? The only other change I am aware of that could influence this is the Spring Security upgrade from 4.2.8 to 4.2.13 (NiFi-6412) [1]; the new version of Spring Security might enforce a new restriction on how the password is sent that LDAP doesn’t like. The LDAP error code 13 refers to the password being sent in plaintext [2]. As you are using StartTLS, I am assuming the LDAP port you’re connecting to is still 389? Did anything change on the LDAP server? Can you verify a simple lookup using ldapsearch still works? If you get the same error code, you may need to add -Z to the command to initialize a secure TLS channel. [1] https://issues.apache.org/jira/browse/NIFI-6412 <https://issues.apache.org/jira/browse/NIFI-6412> [2] https://ldap.com/ldap-result-code-reference-core-ldapv3-result-codes/#rc-confidentialityRequired <https://ldap.com/ldap-result-code-reference-core-ldapv3-result-codes/#rc-confidentialityRequired> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 11, 2019, at 4:59 PM, josef.zahn...@swisscom.com wrote: > > Hi guys > > We would like to upgrade from NiFi 1.9.2 to 1.10.0 and we have HTTPS with > LDAP (START_TLS) authentication successfully enabled on 1.9.2. Now after > upgrading, we have an issue which prevents nifi from startup: > > > 2019-11-11 08:29:30,447 ERROR [main] o.s.web.context.ContextLoader Context > initialization failed > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': > Unsatisfied dependency expressed through method > 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is > org.springframework.beans.factory.BeanExpressionException: Expression parsing > failed; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency > expressed through method 'setJwtAuthenticationProvider' parameter 0; nested > exception is org.springframework.beans.factory.BeanCreationException: Error > creating bean with name 'jwtAuthenticationProvider' defined in class path > resource [nifi-web-security-context.xml]: Cannot resolve reference to bean > 'authorizer' while setting constructor argument; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name 'authorizer': FactoryBean threw exception on object creation; > nested exception is > org.springframework.ldap.AuthenticationNotSupportedException: [LDAP: error > code 13 - confidentiality required]; nested exception is > javax.naming.AuthenticationNotSupportedException: [LDAP: error code 13 - > confidentiality required] > at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:666) > at > org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87) > at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1269) > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551) > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481) > at > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) > at > org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) > at > org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) > at > org.springframework.context.support.AbstractAp
Re: How to replace multi character delimiter with ASCII 001
I haven’t tried this, but you might be able to use ${"AQ==“:base64Decode()} as
AQ== is the Base64 encoded \u0001 ?
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 6, 2019, at 12:25 PM, Shawn Weeks wrote:
>
> I'm trying to process a delimited file with a multi character delimiter which
> is not supported by the CSV Record Reader. To get around that I'm trying to
> replace the delimiter with ASCII 001 the same delimiter used by Hive and one
> I know isn't in the data. Here is my current configuration but NiFi isn't
> interpreting \u0001. I've also tried \001 and ${literal('\u0001')}. None of
> which worked. What is the correct way to do this?
>
> Thanks
> Shawn Weeks
>
>
Re: Converting long string to JSON format.
I think you could accomplish this using ConvertRecord. For the Record Reader,
use a CSVReader with the delimiter character set to |, and for the Record
Writer, use a JsonRecordSetWriter. You may have to use a
ScriptedRecordSetWriter to parse the key/value pair tokens out individually.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Nov 6, 2019, at 9:24 AM, Chandrashekhar Kotekar
> wrote:
>
> I would prefer to write custom processor which will first convert those key
> value pairs into Properties object and then map that object to POJO and use
> json4s to convert pojo to JSON.
>
> Sent from my iPhone
>
>> On 6 Nov 2019, at 9:13 pm, Rivasa wrote:
>>
>> Hi, so I'm having a bit of trouble where I can't really figure out how to
>> accomplish what I need, since i'm fairly new to NIFI in general.
>>
>> So I have a string in the following format:
>> item1=value1|item2=value2|item3=value3|item4=value4... so on and so forth.
>>
>> What I would like to do is convert this into a valid JSON object of the
>> format of:
>> {
>> "item1":"value1",
>> "item2":"value2",
>> "item3":"value3",
>> "item4":"value4"
>> ...
>> etc
>> }
>>
>> I think I need to use extract text or similar maybe? But I'm not exactly
>> sure what to do. Could someone point me in the right direction?
>>
>>
>>
>> --
>> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Nifi Single Instance mode to Cluster Mode
(Removing dev@; only use one list please) Hi Bimal, That error generally means the encryption key used to decrypt the sensitive value was not correct — i.e. not the key used to encrypt the value originally. It sounds like you will need to contact the vendor for this specific issue. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 28, 2019, at 12:12 PM, Bimal Mehta wrote: > > Hi, > > We are doing a migration from a single instance NiFi to a NiFi cluster mode > (with 3 nodes) using Cloudera Manager. > After staring all the 3 nodes from Cloudera Manager, only 1 shows connected > and the others appear disconnected. > In the log file for the 2 nodes that are showing disconnected, we get the > following error: > 455 ERROR org.apache.nifi.controller.serialization.FlowFromDOMFactory: There > was a problem decrypting a sensitive flow configuration value. Check that the > nifi.sensitive.props.key value in nifi.properties matches the value used to > encrypt the flow.xml.gz file > org.apache.nifi.encrypt.EncryptionException: > org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive > value > > I am unable to update the nifi.sensitive.props.key as it is generated by > Cloudera Manager when we start the nodes. > Any suggestions? > > Thanks > Bimal Mehta
Re: How to deploy NiFi processors change to multiple NiFi instances?
I would generally want a human intervention step in this process, but if you want this to be fully-automated, you can script that using the Event Hooks [1] feature. As Edward noted, the NiFi Toolkit or NiPyAPI could be invoked from here to execute whatever steps you want taken. [1] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#event-hooks Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 28, 2019, at 1:38 AM, Edward Armes wrote: > > Hi Lei, > > As far as I'm aware there currently isn't an out of the box method to do this > However I would look at using a combination of the nifi registry and tools > in nifi toolkit to roll your own. > > Also, there have been multiple questions to both this and the dev mailing > list, asking what could be done. I would also suggest searching the list > archives and see what others did and didn't do as well. > > Edward > > On Mon, 28 Oct 2019, 08:14 wangl...@geekplus.com.cn > <mailto:wangl...@geekplus.com.cn>, <mailto:wangl...@geekplus.com.cn>> wrote: > > We have many standalone NiFi instances and all running the same NiFi Flow. > If the flow changes, how to deploy the change to all NiFi instances > automatically? > > Thanks, > Lei > > > wangl...@geekplus.com.cn <mailto:wangl...@geekplus.com.cn>
Re: Curious about Best Practices for Deployment of Workflows
Ken, There are a lot of disjoint resources that discuss some of these concepts, but they are not well-organized at the moment. I would recommend Kevin Doran’s presentation at Dataworks Summit 2018 [1] as a good starting point, specifically including some activity hooks which can automate much of the process you describe. Drew Lim has also recorded some videos showing features and workflows (I separate workflow [“the process of performing some manual activities against the NiFi ecosystem applications”] from dataflow [“the graph of components which runs within NiFi and ingests, routes, transforms, and persists data”]) [2] (scroll down to “Videos”). If others can chime in with their resources as well, we can collect these into a more useful format and hopefully enable more people to use these tools productively. [1] https://dataworkssummit.com/san-jose-2018/session/sdlc-with-apache-nifi/ <https://dataworkssummit.com/san-jose-2018/session/sdlc-with-apache-nifi/> [2] https://nifi.apache.org/registry.html <https://nifi.apache.org/registry.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 18, 2019, at 12:53 PM, Ken Swanson wrote: > > Hi all, > > My group is getting started with NiFi and I'm hoping to draw on people's > experience. I was curious if there were any best practices for NiFi when > deploying workflows. When I deploy new code to production, there's often pull > requests, code review, and then CI/CD to actually put it in production. Is > there anything similar for workflows, or do people have solutions that they > use? > > I'd love it if there was a way for someone to make a local change to a > workflow/process group, push that up to the registry, and then a reviewer see > a diff of the change, approve it, and then it get automatically put onto > master. Is this something people tend to do with NiFi? > > -Ken
Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch
Hi Peter, If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following: * the server is behind an interface which responds differently to GET and POST/PUT requests * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t) * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further. Hopefully this helps. [1] https://tools.ietf.org/html/rfc6125#section-6.4.4 <https://tools.ietf.org/html/rfc6125#section-6.4.4> s_client example: $ openssl s_client -connect -debug -state -cert -key -CAfile Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 16, 2019, at 8:37 PM, Peter Moberg wrote: > > I have an Elastic Search cluster that is setup with SSL. It uses a > self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow > that has the PutElasticSearchHttp component. I have setup a SSLContextService > for that component where I have specified a trust store that has the > self-signed cert from ES. I specify an https endpoint to access Elastic > Search and Im having no issues populating my Elastic Search instance using > this flow. > > I have another flow where I want to do some lookups. So I have been using the > LookupRecord processor. That one I have associated with an > ElasticSearchClientServiceImpl which I have setup to point to the same > SSLContextService as used above. I specified the same HTTPS Url (triple > checked this). However, when I run this second Flow I am not able to verify > the ES server's self-signed certificate. > > I check the nifi-app.log and it says: > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable > to find valid certification path to requested target > > I am a bit surprised that I am not able to verify the same server certificate > in the two different flows. > > Completely stuck on this so if anyone have any pointers please let me know. > > Thanks, > > Peter
Re: High CPU consumption
Evan, Thanks for sharing that diagnosing technique. While ideally we would have other controls to prevent excess CPU usage, this seems like a useful tool which could be automated using NiPyAPI [1] to perform a “bisect” command. I’ve used this for git commit searching as well as side-effect unit test identification. [1] https://nipyapi.readthedocs.io/en/latest/readme.html <https://nipyapi.readthedocs.io/en/latest/readme.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 15, 2019, at 1:40 PM, Evan Reynolds wrote: > > I have found two issues that can cause high CPU when idle (high being about > 200% CPU when idle.) I haven’t verified these with 1.9.2, but it doesn’t hurt > to tell you. > > If you are using ports, make sure each input port is connected. If you have a > one that isn’t connected, that can bring your CPU to 200% and stay there. > > If you have any processors that are set to run on the primary node of a > cluster, that can also take your CPU to 200%. I know RouteOnAttribute will do > that (again, haven’t tested 1.9.2, but it was a problem for me for a bit!) > The fix – either don’t run it on the primary node, or else set the run > schedule to 5 seconds or something instead of 0. > > To find out if this is the case – well, this is what I did. It worked, and > wasn’t that hard, though isn’t exactly elegant. > > Back up your flowfile (flow.xml.gz) > Stop all your processors and see what your CPU does > Start half of them and watch your CPU – basically do a binary search. If your > CPU stays reasonable, then whatever group you started is fine. If not, then > start stopping things until your CPU becomes reasonable. > Eventually you’ll find a processor that spikes your CPU when you start it and > then you can figure out what to do about that processor. Record which > processor it is and how you altered it to bring CPU down. > Repeat, as there may be more than one processor spiking CPU. > Stop NiFi and restore your flowfile by copying it back in place – since you > were running around stopping things, this just makes sure everything is > correctly back to where it should be > > Then use the list of processors and fixes to make changes. > > --- > > Evan Reynolds > e...@usermind.com <mailto:e...@usermind.com> > > > From: Jon Logan > Reply-To: "users@nifi.apache.org" > Date: Sunday, October 13, 2019 at 6:12 PM > To: "users@nifi.apache.org" > Subject: Re: High CPU consumption > > That isn't logback, that lists all jars on your classpath, the first of which > happens to be logback. > > If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread > stacks to stdout (probably the bootstrap log)...but that's just for one > instant, and may or may not be helpful. > > On Sun, Oct 13, 2019 at 8:58 PM Luis Carmona <mailto:lcarm...@openpartner.cl>> wrote: > hi Aldrin, > > thanks a lot, by now I'm trying to learn how to make the profiling you > mentioned. > > One more question: Is it normal that the father java process has very low > consumption while the child process related to logback jar is the one that is > eating up all the CPU ? > Please take a look at the attached image. > > Thanks, > > LC > > From: "Aldrin Piri" mailto:aldrinp...@gmail.com>> > To: "users" mailto:users@nifi.apache.org>> > Sent: Sunday, October 13, 2019 9:30:47 PM > Subject: Re: High CPU consumption > > Luis, please feel free to give us some information on your flow so we can > help you track down problematic areas as well. > > On Sun, Oct 13, 2019 at 3:56 PM Jon Logan <mailto:jmlo...@buffalo.edu>> wrote: > You should put a profiler on it to be sure. > Just because your processors aren't processing data doesn't mean they are > idle though -- many have to poll for new data, especially sources -- ex. > connecting to Kafka, etc, will itself consume some CPU. > > But again, you should attach a profiler before participating in a wild goose > chase of performance issues. > > On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona <mailto:lcarm...@openpartner.cl>> wrote: > HI, > > I've struggling to reduce my nifi installation CPU consumption. Even in idle > state, all processors running but no data flowing, it is beyond 60% CPU > consumption, with peaks of 200%. > > What I've done so far > - Read and followed every instruction/post about tuning NIFI I've found > googling. > - Verify scheduling is 1s for most consuming processor
Re: Implement heartbeat from remote servers
If the NiFi nodes are all in a cluster, the heartbeating is taken care of by the cluster communication. If you are talking about two distinct instances, you could put a GenerateFlowFile processor with arbitrary “heartbeat” data and send it via Site to Site or InvokeHTTP to the “listening” NiFi instance. There you could set a value in the Distributed Map Cache for each server which has pinged in and the timestamp, and then use another processor to check on a determined schedule for the presence/absence of certain values in the cache. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 3, 2019, at 2:46 AM, Jeremy Pemberton-Pigott > wrote: > > Does anyone have an idea of how to implement a heartbeat message posted from > remote Nifi servers to another server running Nifi? This is so that I can > tell if within say 1 hour there was no heartbeat received from a known list > it could generate a log message with an error message of the server that no > heartbeat was received from. > > Thanks, > > Jeremy
Re: Nifi Cluster Untrusted Proxy Error
We are working on additional documentation to explain the details of TLS-based trust, but this is a complicated topic and not specifically a core area of existing NiFi docs as it is an independent topic. I would recommend this post [1] for an understanding of the actual TLS process, and this high-level overview of how Java keystores work [2]. For NiFi, each node needs to present a certificate via the keystore which identifies the node and is signed by a certificate in (or explicitly present in) the truststore of every other node. [1] https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work/20847#20847 <https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work/20847#20847> [2] https://dzone.com/articles/ssl-in-java <https://dzone.com/articles/ssl-in-java> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 5, 2019, at 12:08 AM, Dweep Sharma wrote: > > Sure, could you please share resources on how to enable trust between ssl > certs on two nodes > > authorizations.xml > > > > > > resource="/flow" action="R"> > > > > > resource="/restricted-components" action="W"> > > > > resource="/tenants" action="R"> > > > > resource="/tenants" action="W"> > > > > resource="/policies" action="R"> > > > > resource="/policies" action="W"> > > > > resource="/controller" action="R"> > > > > resource="/controller" action="W"> > > > > resource="/provenance" action="R"> > > > resource="/site-to-site" action="R"> > > > resource="/system" action="R"> > > > resource="/proxy" action="W"> > > > resource="/counters" action="R"> > > > resource="/counters" action="W"> > > > resource="/policies/process-groups/a97c370b-016c-1000-87c2-2ed45eaf0b48" > action="R"/> > resource="/process-groups/a97c370b-016c-1000-87c2-2ed45eaf0b48" action="R"/> > resource="/process-groups/a97c370b-016c-1000-87c2-2ed45eaf0b48" action="W"> > > > resource="/operation/processors/b3c48d49-016c-1000-8396-950d03ad5e07" > action="W"> > > > resource="/operation/processors/b3c8228a-016c-1000-8e36-f4315d3da34c" > action="W"> > > > resource="/operation/process-groups/b3c41e61-016c-1000-b40f-21bbefe6599c" > action="W"> > > > resource="/processors/b3c48d49-016c-1000-8396-950d03ad5e07" action="R"> > > > resource="/process-groups/b3c41e61-016c-1000-b40f-21bbefe6599c" action="R"> > > > resource="/process-groups/b3ce9097-016c-1000-fbbe-c5f148d3d5bc" action="R"> > > > > > > > On Tue, Sep 3, 2019 at 7:15 PM Bryan Bende <mailto:bbe...@gmail.com>> wrote: > Please show authorizations.xml, thank you. > > Also, you shouldn't really be using wildcard certs - > https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#wildcard_certificates > > <https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#wildcard_certificates> > > On Tue, Sep 3, 2019 at 5:32 AM Dweep Sharma <mailto:dweep.sha...@redbus.com>> wrote: > > > > Can someone take a peek at this - what could be wrong? Thanks > > > > -Dweep > > > > On Fri, Aug 30, 2019 at 4:52 PM Dweep Sharma > <mailto:dweep.sha...@redbus.com>> wrote: > >> > >> Hi All, > >> > >> I am receiving an error while setting up a 2 node cluster (
Re: My nifi no more serve admin interface
I think in general it’s hard for us to know when a bad keystore is provided
until a connection tries to come in because a lot of that is delegated to
Jetty. There was talk previously about a “keystore checker” toolkit feature
which would look at the complete provided configuration for TLS and try to
verify it was correct / diagnose any issues. Unfortunately with time &
availability the way they are, I don’t think anyone has been able to work on
it.
I know there is a lot of documentation around configuring TLS for NiFi and
diagnosing various problems manually but it’s not collected seamlessly in a
canonical format and location. I am slowly working with some other community
members to improve this as well. For now we generally try to push users who
don’t have a strong grasp of the TLS generation/configuration process to the
toolkit to offload a lot of that process, and suggest they read the
documentation proactively to have a sense of the “good path” forward rather
than exploring/experimenting and going down a rabbit hole.
We always value contributions to the process/documentation and even something
as minimal as sharing the process you took so we can identify the best
wording/approach to intercept at the exact point where the wrong decision
seemed like the right one is helpful to the entire community. Thanks.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 14, 2019, at 7:31 AM, Edward Armes wrote:
>
> Hmm, I wonder if there's a change that could be made to expose this error so
> its a bit more obvious, maybe one for the Dev mailing list?
>
> Edward
>
> On Wed, Aug 14, 2019 at 3:12 PM Pierre Villard <mailto:pierre.villard...@gmail.com>> wrote:
> Glad you sorted it out and thanks for letting us know!
> In case you missed it, you might be interested by the NiFi toolkit [1]
> containing a TLS toolkit to help you with certificates [2].
>
> [1] https://nifi.apache.org/download.html
> <https://nifi.apache.org/download.html>
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit
> <https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit>
> Le mer. 14 août 2019 à 15:54, Nicolas Delsaux <mailto:nicolas.dels...@gmx.fr>> a écrit :
> Oh damn
>
> It appeared (after a long search) that my keystore was incorrectly built.
>
> Indeed, it contained the server certificate as a trusted certificate, where
> it should had been a key pair (with both private and public keys in) as is
> explained in Jetty documentation
> (https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys
>
> <https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys>
> - see part Layout of keystore and truststore). And this happened because I'm
> really bad at certificates.
>
> Sorry to have consumed some of your time, you all.
>
> Le 13/08/2019 à 16:21, Nicolas Delsaux a écrit :
>> oh, sorry, I forgot to mention i use the nifi docker image, with
>> configuration
>>
>> services:
>> nifi-runner:
>> hostname: nifi-psh.adeo.com <http://nifi-psh.adeo.com/>
>> image: apache/nifi:1.9.2
>> ports:
>> - "38080:8443"
>> - "5000:8000"
>> volumes:
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
>>
>> And port 8443 is standard http port, I guess (the port 8000 is the standard
>> debug one)
>>
>>
>>
>> Le 13/08/2019 à 16:10, Pierre Villard a écrit :
>>> Might be a dumb question but I'm wondering why you're trying with port
>>> 38080? Did you change the configuration to use that specific port with a
>>> secured instance?
>>>
>>> Pierre
>>>
>>> Le mar. 13 août 2019 à 16:00, Nicolas Delsaux >> <mailto:nicolas.dels...@gmx.fr>> a écrit :
>>> To go a little further, a test with openssl s_client gives the following
>>>
>>> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>>> $ openssl s_client -host localhost -port 38080
>>> CONNECTED(0164)
>>> 416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
>>> failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
>>&
Re: Certificates in Truststore
Joseph, Joe provided a number of good links to help provide context around this. I will be working with a couple other committers next week to improve our documentation around this task. Hopefully when we have that complete, you can take a look and let us know if it would have helped you in this effort and any changes you suggest. The short answer to your request is to import the public certificate of the certificate authority (CA) which is used to sign the individual users’ client certificates into the truststore, which is then provided to NiFi. As the CA public cert does not change, you can use it (actually the corresponding private key) to sign as many user certificates as you want without requiring any changes to the deployed truststore (truststores if in a clustered environment). Please let me know if anything above is not clear. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 25, 2019, at 9:09 AM, Joe Witt wrote: > > Joseph > > I'd make sure to read about the keystore/truststore model and high level bits > of PKI. A site like [1] can help with that but the first key is > understanding the client cert, server cert, CA, and general trust model. > > With that basis in mind setting up NiFi for mutual auth with certificates > both on the client side and nifi server side and proper trust mechanism is > much easier. The docs in NiFi on this topic should then be really helpful > [2,3,4]. > > [1] > http://www.robinhowlett.com/blog/2016/01/05/everything-you-ever-wanted-to-know-about-ssl-but-were-afraid-to-ask/ > > <http://www.robinhowlett.com/blog/2016/01/05/everything-you-ever-wanted-to-know-about-ssl-but-were-afraid-to-ask/> > [2] > http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_configuration > > <http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_configuration> > [3] > http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication > > <http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication> > [4] > http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls_generation_toolkit > > <http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls_generation_toolkit> > > Thanks > Joe > > On Thu, Jul 25, 2019 at 11:58 AM Joe Witt <mailto:joe.w...@gmail.com>> wrote: > Joseph > > You are absolutely right that it would be terrible to have to edit the > truststore on the nifi server(s) each time you wanted to add a client cert. > You're also right that there is a way to never do this. I'll poke around for > some links to help send you in the right direction. > > Thanks > > On Thu, Jul 25, 2019 at 11:45 AM Joseph Wheeler <mailto:jwhee...@innovasi.com>> wrote: > Hello, > > > > I apologize if this is a simple/stupid question, but reading through the > administration guide and copious amounts of googling have returned very > little regarding this. > > > > I’m looking into utilizing only client certificates for authentication to our > Apache NiFi server. I want to avoid having to add another software package > (e.g. LDAP, Kerberos, etc.) to the server. After spending the last few days > working on this and getting an understanding of how to get new users created, > I’m running into an issue: a user’s client certificate has to be added to the > truststore on the server in order for it to be allowed to access the NiFi web > server, and NiFi doesn’t seem to recognize changes to the truststore while > it’s running. While I don’t expect to need to add a ton of new users, I am > imagining a scenario where my program managers need a new user added > immediately while one of our lead developers is in the process of doing > something in the web app that he can’t lose due to a service restart. Is > there a way to make NiFi recognize changes to the truststore without > requiring the service to be restarted? If not, is there a way to have NiFi > trust all certs from a certain CA? They still wouldn’t actually be able to > access anything without having a user account tied to their cert’s DN… > > > > Thanks! > > > > r/ > > > > Joseph Wheeler >
Re: flow.xml sync
Placing a flow.xml.gz file on each node before starting the cluster should not cause any problems as long as the flow.xml.gz files are identical. If the flows differ, the cluster will not be able to start up (superficial discrepancies can be resolved automatically, but any value differences will cause problems). A (healthy) cluster will always sync the flow between all nodes. This is because multiple users can interact with the flow on any node’s web server, and the flow must reflect the most accurate state (this is handled with two-phase commits on flow changes). If a primary node is lost or disconnected, ZK performs a new election to determine the replacement primary node. Processors which only run on the primary node are configured via the processor configuration, which you can update via the UI or API. You cannot explicitly determine the primary node; it is the result of the ZK election at runtime. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 24, 2019, at 3:08 AM, Clay Teahouse wrote: > > According to the NiFi documentation, zookeeper decides on the primary node, > when the system starts. So that answers my question. Now my question is how > to designate where to run the isolated processes without the NiFi UI? thanks. > > On Wed, Jul 24, 2019 at 4:50 AM Clay Teahouse <mailto:clayteaho...@gmail.com>> wrote: > Hi Everyone, > I have a cluster of multiple nodes. If I place flow.xml on all of them before > starting the nodes, would that cause a conflict? If yes, can I disable > syncing between the nodes? I do not want to have to decide on a primary node > at the start up. > In other words, can the primary node be decided on the run time? > > thank you > Clay.
Re: Docker nifi doesn't support OpenID Connect ?
I believe the wording on the Docker Hub page simply means the environment variables are not mapped for OpenID Connect configuration through Docker. If you mount a pre-configured NiFi.properties file with OpenID Connect provider configured, I am not aware of any reason that would not work as expected. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 3, 2019, at 8:09 AM, Nicolas Delsaux wrote: > > Hi, > > I've read on Docker hub that nifi docker container doesn't support > OpenID Connect. > > But if I mount the nifi.properties file using a volume, is it possible > to have openID Connect working ? or is it replaced by the Docker > start.sh script (which invoke secure.sh only for LDAP or two-way SSL) ? >
Re: Comments about installing Secured NiFi
Thanks for documenting this Ken. You’re right that this is challenging and not user-friendly, especially for first-time users. The point about DN spacing is especially well-taken. I’m working on new docs for this, and I’ll share them soon and hope your feedback will be helpful to make this process much easier for users. Thanks. If anyone has more info to add for difficult use cases or unexpected problems, please add it here. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 30, 2019, at 23:45, Ken Danniswara wrote: > > Hello, > > Couple days ago I talked with Andy(@yolopey) over twitter about me > experiencing a hard part when installing secure NiFi 1.9.2. Before I forgot > (as usual) I thought if I put it somewhere else it could be somehow helpful. > > First, not getting used with LDAP DN creates a long confusion. I have hard > time following example which DN tree to use on different parts of the guide. > While the LDAP tutorial is outside the scope, maybe having consistent DN tree > throughout the guide could be helpful. For example between File-based (LDAP > Authentication) and LDAP-based Users/Groups Referencing User DN, also when > generating initial admin cert with TLS-Toolkit. > > Other problem with DN it is spaced-sensitive. I created the person > certificate without space: tls-toolkit.sh -C > 'cn=nifiadmin,ou=users,dc=exa,dc=local', then copied the same string to the > "initial admin identity" properties. Apparently the certificate > auto-generated the space and my 'not-spaced' version authorization became > failed in login time. In the end I tried with changing the initial admin + > deleting users.xml or simply change the name inside users.xml file directly > both works. > > Last part which is my mistake. I did un-comment the legacy FileAuthorizer > class at the bottom of the authorizer.xml file. I thought it will be the same > procedure to do like enabling ldap-provider in the > local-identity-provider.xml. I am not sure how easy other fall to this > mistake. > > These are my main challenges over building the secured NiFi. The problem > maybe would happen for person without LDAP experience like me. Otherwise > there are no big problem. I haven't tried the Kerberos one which I'd love to > try other time. > > Best Regards, > Ken
Re: feature suggestion
Wyllys, It may help to think of NiFi processors the same way you think about *nix command line tools — each tool has a very specific and limited responsibility and focus. Rather than continue to add generic features to each in order to cover all use cases, you combine the right tools in the necessary order to achieve your goal (i.e. cat, grep, sed, cut, etc.). No individual tool cares where the input data is coming from or where the output data is going; a critical idea which is part of the Flow Based Programming [1] design philosophy that NiFi models. Adding features increases complexity, configuration, and the opportunity for bugs or unexpected behavior, especially in edge cases. The InvokeHTTP processor is concerned with transmitting flowfile attributes & content to a remote HTTP endpoint. Rather than include additional functionality to massage the content, we would recommend using an additional preceding processor to form the content into the expected format. I definitely understand the desire to add “just one thing” to make a specific use case easier, but we have to balance that with the maintainability and supportability of the framework as a whole moving forward. Hope this helps. Thanks. [1] https://nifi.apache.org/docs/nifi-docs/html/overview.html#the-core-concepts-of-nifi <https://nifi.apache.org/docs/nifi-docs/html/overview.html#the-core-concepts-of-nifi> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 19, 2019, at 7:23 AM, Wyllys Ingersoll > wrote: > > OK, that helps. I put AttributesToJSON and a ReplaceText processor in front > of my invokeHTTP and was able to get it to work as I wanted. > > One of the confusing issues is that InvokeHTTP does not have any > configuration that allows me to modify the flowfile content, which forces me > to use those additional processors to make sure the content is correct. It > might be a little more convenient if invokeHTTP had a couple of parameters to > allow for setting the content format so that the extra processors are not > necessary. > > > > On Tue, Jun 18, 2019 at 11:01 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > Wyllys, > > I am sorry but I am having a different experience when I try to reproduce the > scenario you describe. > > I set up a sample flow which uses an InvokeHTTP processor to send a flowfile > to a remote HTTP endpoint. That endpoint happens to be a disparate flow in > NiFi because it was easy to configure on the fly, but it could be any remote > web server. > > I can verify that when I send flowfile content, it is not transmitting the > flowfile attributes unless I explicitly configure it to. Please see below an > annotated excerpt from the nifi-app.log file and a link to the template I > exported with this flow. > > I think the issue may be one of terminology? Attributes and content are > separate in NiFi and flowfiles distinguish very cleanly between the two. Are > you referring to elements of the JSON body as “attributes”? In that case, you > would need to use a ReplaceText, EvaluateJsonPath, JoltTransformJson, or > other processor as Jerry suggested to extract/manipulate the JSON in the > flowfile content to the exact data you wish to send over HTTP before using > InvokeHTTP. If you want to do this but not lose the other content, you can > split the flow by dragging an additional “success” relationship from the > preceding processor to handle the two different flow behaviors. > > Template for flow (tested on NiFi 1.9.2): > https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702 > <https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702> > > Log output from flow: > https://gist.github.com/alopresto/b596372005032b9c80063a0652a955ea > <https://gist.github.com/alopresto/b596372005032b9c80063a0652a955ea> > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 18, 2019, at 7:47 PM, Jerry Vinokurov > <mailto:grapesmo...@gmail.com>> wrote: >> >> Hi Wyllys, >> >> One way that I solve this problem in my work is to use the AttributesToJSON >> processor to form the body of the POST before sending it. Granted, that does >> override the body of the flowfile, so I'm not sure if that works for your >> specific case, but it does allow you to select which attributes you want to >> turn into JSON key/value pairs. For more complex formations I would suggest >> checking out the JOLT Transform processor, which c
Re: feature suggestion
Wyllys,
I am sorry but I am having a different experience when I try to reproduce the
scenario you describe.
I set up a sample flow which uses an InvokeHTTP processor to send a flowfile to
a remote HTTP endpoint. That endpoint happens to be a disparate flow in NiFi
because it was easy to configure on the fly, but it could be any remote web
server.
I can verify that when I send flowfile content, it is not transmitting the
flowfile attributes unless I explicitly configure it to. Please see below an
annotated excerpt from the nifi-app.log file and a link to the template I
exported with this flow.
I think the issue may be one of terminology? Attributes and content are
separate in NiFi and flowfiles distinguish very cleanly between the two. Are
you referring to elements of the JSON body as “attributes”? In that case, you
would need to use a ReplaceText, EvaluateJsonPath, JoltTransformJson, or other
processor as Jerry suggested to extract/manipulate the JSON in the flowfile
content to the exact data you wish to send over HTTP before using InvokeHTTP.
If you want to do this but not lose the other content, you can split the flow
by dragging an additional “success” relationship from the preceding processor
to handle the two different flow behaviors.
Template for flow (tested on NiFi 1.9.2):
https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702
<https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702>
Log output from flow:
https://gist.github.com/alopresto/b596372005032b9c80063a0652a955ea
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 18, 2019, at 7:47 PM, Jerry Vinokurov wrote:
>
> Hi Wyllys,
>
> One way that I solve this problem in my work is to use the AttributesToJSON
> processor to form the body of the POST before sending it. Granted, that does
> override the body of the flowfile, so I'm not sure if that works for your
> specific case, but it does allow you to select which attributes you want to
> turn into JSON key/value pairs. For more complex formations I would suggest
> checking out the JOLT Transform processor, which can be quite powerful, if
> somewhat painful to work with.
>
> Jerry
>
> On Tue, Jun 18, 2019 at 9:49 PM Wyllys Ingersoll
> mailto:wyllys.ingers...@keepertech.com>>
> wrote:
> Andy -
>Yes, Im referring to the PUT and POST methods, in which case the processor
> just sends the entire flowfile as a JSON object in the message body. I'd
> prefer to either have the option to exclude some of the flow attributes or
> (even better) have the ability to craft my own message body. There are lots
> of instances where the receiver of the PUT or POST expects a particular
> structure that doesn't easily work with just a flat JSON-ified set of flow
> attributes.
>
> One example:
> We have a flow that has an authentication token as one of the attributes
> and a bunch of other key/value pairs used for other purposes. In the
> InvokeHTTP processor, I use the auth token attribute in an Authorization
> header by creating a dynamic attribute with the correct format
> ("Authorization: Bearer ${token}"). However, the recipient of the PUT/POST
> also expects the body of the request to be formatted a specific way and does
> not expect or want to see the auth token or some of the other unrelated
> information that ends up getting transmitted as part of the message body
> simply because they are flow attributes. So, if InvokeHTTP were able to
> exclude certain fields from the message body AND also allow the body of the
> message to be configurable into a structure other than just a flat dictionary
> of flow attributes, it would be much more powerful and useful. As it stands,
> I'm thinking I may have to develop a custom processor to get past this issue,
> which is not ideal at all.
>
> Thanks!
> Wyllys Ingersoll
>
>
>
>
> On Tue, Jun 18, 2019 at 8:34 PM Andy LoPresto <mailto:alopre...@apache.org>> wrote:
> Hi Wyllys,
>
> Sorry to hear you are having trouble with this processor. Can you please
> provide a more detailed example of an incoming flowfile and what your
> expected output is compared to what is currently happening? Based on my
> understanding, the flowfile attributes are only sent as request headers, and
> that can be controlled using the regular expression value of “Attributes to
> Send”. I believe only the flowfile content is sent as the request body when
> using PUT or POST. Thanks.
>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 B
