Re: Refreshing HTML displayed for View usage
Ive got a wks/offline server scenario where reverse DNS doesn't work and things like the online Usage help and processor state don't update in a similar way. If I was to guess it'd be something like a recent jetty upgrade behaving differently; but just a guess. When the 30s timeout occurs, things render as expected. It's also a pain trying to drag connectors into a funnel, that processor type in particular has difficulty turning green. On Mon, 21 Aug 2023, 23:41 Russell Bateman, wrote: > Thanks, Matt. This is happening locally during development where I'm not > using DNS (just localhost). I find this problem less acute on older > versions of NiFi (1.1.3.2) than more recent ones (1.23.0, 1.19.1). > > Thanks, > Russ > > On 8/20/23 19:03, Matthew Hawkins wrote: > > Hi Russell, > > > > Something I've noticed myself, and it's easily reproducible on the Usage > > pages for processors, is that Nifi is doing a reverse DNS lookup when > > loading the page. If your DNS is broken, it can take 10-30 seconds for > the > > page content to appear. > > > > Kr, > > > > On Fri, 18 Aug 2023, 07:29 Russell Bateman, > wrote: > > > >> Matt, > >> > >> I experimented with the Open in New Window button, but the new window, > >> which probably isn't an iframe, nevertheless doesn't respond to attempts > >> to get around the cache. > >> > >> You're probably right about the version not changing being at the root > >> of the browser's inability to respond to a change. Sadly, the > >> edit-display-debug-edit cycle doesn't excite me into changing versions > >> just so I can see the result. > >> > >> I'm certainly not going to clamor for work to be done to get around > >> this. It's annoying, but not crippling. > >> > >> Thanks, > >> > >> Russ > >> > >> > >> On 8/17/23 13:54, Matt Gilman wrote: > >>> Russell, > >>> > >>> Thanks for the response. The suggestion to open a new tab was for the > >>> generated documentation, not the NiFi canvas itself. The generated > >>> documentation is shown in an iframe which will offer you the menu item > in > >>> the context menu for opening in a new tab. IIRC correctly, the path to > >> the > >>> generated documentation contains the version of the component. For > folks > >>> simply using a released version of NiFi this works great since versions > >>> will differ and any browser caching will only optimize page loading for > >> the > >>> user. If you're a developer, however, you're likely making changes to > >> your > >>> component and the version is not changing (x.y.z-SNAPSHOT). This is > where > >>> the browser cache could result in the behavior your seeing. > >>> > >>> Matt > >>> > >>> On Thu, Aug 17, 2023 at 3:21 PM Russell Bateman > >>> wrote: > >>> > Thanks, Matt, > > (Since I use Chrome 99% of the time, I'm using Chrome here.) > > Clicking on the reload control next to the address bar, while holding > down Shift, reloads the canvas, but the NiFi Documentation page > disappears and I have to reload it using the View usage context menu > of > my processor. Once reloaded, nothing has changed. > > This is also the behavior of the Configure Processor dialog. As I > reload > View usage, ... > > In Chrome, there's no effective content-menu option to right-click in > the page contents and open in a new tab. I can... > > Back > Forward > Reload > --- > Save As... > Print > Cast > Search images with Google > - > Send to your devices > Create QR code for this page > > Translate > - > View page source > View frame source > Reload frame > Inspect > > > If I right-click the current tab itself, I'm able to choose to > Duplicate., but that doesn't seem to lead to anything more helpful. > > By the way, I'm doing this using NiFi 1.23.0. I have also verified the > date of my NAR to ensure that I'm not repeatedly working using a NAR > with old content. Also, the version displayed for the custom processor > is that of the various /pom.xml/ files I'm building with. > > I usually develop on the latest NiFi version. However, my company has > NARs that won't load beginning 1.14.0. I just brought up 1.13.2, what > most of our customers run, and tried my latest NAR which *does* > display > my changes.Same with 1.1.2, which represents the oldest platform > executing at any custom site. Exceptionally, I set up 1.19.1 where it > *also works*. > > This is unexpected; maybe it does point, as unlikely as it seems, to > something changed in 1.23.0 (instead of being caused by anything > between > the dossier of my chair and the space bar of my keyboard as I have > been > expecting to lea
Re: JoltTransformJSON EL when using file
Hi Matt,
Many of the deployments I get involved with are not in a Hadoop ecosystem,
so building in resilience becomes part of the challenge, especially when
data input is out of your control and often has bad data. In these
situations, one thing I like doing is a test flow using DuplicateFlowFile
and DistributeLoad configured as a valve so the main queue isn't
backlogged. You can then use live data to work on massaging bad input.
Manually creating schemas for record based processors and JoltTransformJSON
becomes a large part of the job.
In one such experiment I was either trying to work around GenerateRecord
not having capability for some data types (e.g. Boolean) and faking it with
a jolt transform, or I was putting parent key/values back into a content
after splitting on a sub-arrray. Either way, I ended up with a jolt
transform inserting a timestamp.
Current time is pretty useful for downstream systems to understand latency
between an event and it getting to that system, I often ensure Nifi is
dropping it into content.
I'm not near my test system to get the exact spec file but will endeavour
to at the next opportunity.
On Fri, 18 Aug 2023, 07:13 Matt Burgess, wrote:
> Matthew,
>
> What was your second case trying to use ${now():toNumber()} ? My unit
> test evaluated the expression into an integer and it worked (versus
> putting the expression in quotes which would make it a string).
>
> Regards,
> Matt
>
> On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins
> wrote:
> >
> > Hi Matt,
> >
> > The tag will be ${firstname}, the spec is correct.
> > (well, I also get lastname, but it's unimportant)
> >
> > Weird thing is this failed similarly with a file based input on the
> Record
> > based jolt processor as well on a completely different system.
> >
> > Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> > source with
> include-graph,include-media,include-rules,include-sql-reporting
> >
> > In the second case I was trying to get ${now():toNumber()} into a json
> > record using either default or modify-overwrite-beta (tried both). The
> > following success processor failed on reading schema as the literal
> > ${now...} was a string not a long and the JVM refused the type cast. If I
> > insert the transform directly then it still fails with a type cast
> problem
> > from generic Object :/ (that was midnight last night, and I tossed my
> hands
> > in the air and turned the server off)
> >
> > I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> > postgresql. Trying to be a good nifi citizen and use record based
> > processing where possible. Since I don't run Hadoop at home, it's all
> > manual schemas and using postgres as the perm data store. I'm using json
> as
> > mqtt does it intrinsically and I can jolt transform other data into json
> > form and then store it easily in postgres. Well, that was the idea 😁
> >
> > Kr,
> >
> > On Sun, 13 Aug 2023, 10:44 Matt Burgess, wrote:
> >
> > > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > > worked. I noticed you referred to "attrname" in your post but your
> > > spec refers to "firstname", is that a typo?
> > >
> > > Regards,
> > > Matt
> > >
> > > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess
> wrote:
> > > >
> > > > I added file support to JoltTransformJSON under NIFI-4957 [1], a
> first
> > > > glance at the code seems like it should work, but I'll try to
> > > > reproduce it and follow up, thanks for bringing this to our
> attention!
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > > >
> > > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins >
> > > wrote:
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > Using 1.23.0 I have a simple transform adding some flowfile
> attributes
> > > into
> > > > > the content.
> > > > > When putting the spec directly to the processor it works fine. If I
> > > have
> > > > > the spec in an external file however it seems to put in the literal
> > > string
> > > > > ${attrname} - ie the EL didn't process.
> > > > >
> > > > > Have I done something incorrectly or should I be now asking for an
> > > account
> > > > > to log a bug?
> > > > >
> > > > > Sample spec run using Chain DSL:
> > > > >
> > > > > [{
> > > > > "operation": "shift",
> > > > > "spec": {
> > > > > "@": "values"
> > > > > },
> > > > > {
> > > > > "operation": "default",
> > > > > "spec": {
> > > > > "firstname": "${firstname}"
> > > > > }
> > > > > }
> > > > > ]
> > > > >
> > > > > Kind regards,
> > > > > Matthew
> > >
>
Re: FileMaker and Apache NiFi
Samuel, Where do you see that DBCPConnectionPool uses JNDI? As far as I know it just uses Apache DBCP and JDBC DataSources using the supplied driver. Regards, Matt On Wed, Aug 23, 2023 at 9:14 AM Namazi, Samuel wrote: > > To whom it may concern, > > I am trying to connect my FileMaker database to NiFi as a data source. > However that ist not so easily done, as NiFi's DBCPConnectionPool (1.22.0) > uses JNDI, but FileMaker does not support the JNDI standard for JDBC. It > probably is possible to use a JDBC-ODBC Bridge or create a custom processor > and controller to make it work, but those options aren't really considerable > for me. Am I overlooking something and there is an easy solution to this ? If > there is something else that I can do, I would greatly appreciate your help. > > Kind regards > > Samuel Namazi
Re: NiFi 1.19.1 MSSQL issue with RH9 when using crypto policies = DEFAULT
Martin,
Thanks for providing the detailed background.
Based on the error message and configuration, it sounds like the MS
SQL server has a certificate signed with SHA-1.
SHA-1 is not secure for cryptographic operations, provisioning a new
database server certificate and restoring the default Java security
policy is highly recommended.
Regards,
David Handermann
On Wed, Aug 23, 2023 at 10:34 AM Martin Fong wrote:
>
> We have an MS SQL connection that worked fine with RH7.
>
> Once we upgraded to RH9 and the crypto policies=DEFAULT
> (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_security_considerations-in-adopting-rhel-9)
>
> We are getting the following errors:
>
> Failed to establish Database Connection: java.sql.SQLException: Cannot create
> PoolableConnectionFactory ("encrypt" property is set to "true" and
> "trustServerCertificate" property is set to "true" but the driver could not
> establish a secure connection to SQL Server by using Secure Sockets Layer
> (SSL) encryption: Error: Certificates do not conform to algorithm
> constraints. ClientConnectionId:b844ea35-c351-43e7-8645-5c676d2b3cce)
>
> From the log java trace got this at the end:
> Caused by: java.security.cert.CertPathValidatorException: Algorithm
> constraints check failed on signature algorithm: SHA1withRSA
>
> We have searched many areas and found similar issue:
> https://github.com/keycloak/keycloak/issues/19185
>
> People are saying to put SHA1 back to
> /etc/crypto-policies/back-ends/java.config
>
> When we set crypto policies = LEGACY, NiFi SQL connection worked again.
> Meaning SHA1 is back.
>
> We have set the following in NiFi:
> nifi.web.https.ciphersuites.include=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
>
> Also verified the keystore/truststore that NiFi uses is SHA256.
>
> The SQL driver we are using: mssql-jdbc-12.4.0.jre8.jar
>
> Is there a way to find out where that SHA1withRSA is coming from?
>
> Please advise,
> Martin Fong
> Enterprise Technical Support Specialist, Infrastructure & Platform (IAG)
> Technology Services Division, Technology Infrastructure Services
> City of Toronto
> 703 Don Mills Road, 2nd Floor
> Toronto, ON
> M3C 3N3
> Tel: 416-397-7565
> e-mail: martin.f...@toronto.ca
>
> This e-mail message is confidential and subject to copyright. Any
> unauthorized use or disclosure is prohibited. If you have received this email
> and are not the intended recipient, please advise and delete it. Thank you.
>
NiFi 1.19.1 MSSQL issue with RH9 when using crypto policies = DEFAULT
We have an MS SQL connection that worked fine with RH7.
Once we upgraded to RH9 and the crypto policies=DEFAULT
(https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_security_considerations-in-adopting-rhel-9)
We are getting the following errors:
Failed to establish Database Connection: java.sql.SQLException: Cannot create
PoolableConnectionFactory ("encrypt" property is set to "true" and
"trustServerCertificate" property is set to "true" but the driver could not
establish a secure connection to SQL Server by using Secure Sockets Layer (SSL)
encryption: Error: Certificates do not conform to algorithm constraints.
ClientConnectionId:b844ea35-c351-43e7-8645-5c676d2b3cce)
>From the log java trace got this at the end:
Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints
check failed on signature algorithm: SHA1withRSA
We have searched many areas and found similar issue:
https://github.com/keycloak/keycloak/issues/19185
People are saying to put SHA1 back to /etc/crypto-policies/back-ends/java.config
When we set crypto policies = LEGACY, NiFi SQL connection worked again.
Meaning SHA1 is back.
We have set the following in NiFi:
nifi.web.https.ciphersuites.include=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Also verified the keystore/truststore that NiFi uses is SHA256.
The SQL driver we are using: mssql-jdbc-12.4.0.jre8.jar
Is there a way to find out where that SHA1withRSA is coming from?
Please advise,
Martin Fong
Enterprise Technical Support Specialist, Infrastructure & Platform (IAG)
Technology Services Division, Technology Infrastructure Services
City of Toronto
703 Don Mills Road, 2nd Floor
Toronto, ON
M3C 3N3
Tel: 416-397-7565
e-mail: martin.f...@toronto.ca
This e-mail message is confidential and subject to copyright. Any unauthorized
use or disclosure is prohibited. If you have received this email and are not
the intended recipient, please advise and delete it. Thank you.
FileMaker and Apache NiFi
To whom it may concern, I am trying to connect my FileMaker database to NiFi as a data source. However that ist not so easily done, as NiFi's DBCPConnectionPool (1.22.0) uses JNDI, but FileMaker does not support the JNDI standard for JDBC. It probably is possible to use a JDBC-ODBC Bridge or create a custom processor and controller to make it work, but those options aren't really considerable for me. Am I overlooking something and there is an easy solution to this ? If there is something else that I can do, I would greatly appreciate your help. Kind regards Samuel Namazi
