Re: Nifi on RockyLinux

[e-sociaux]

 


Hello Deepak,

 

We have migrated all our nifi instances to Rocky Linux 8 and until now no got any issue.

 

Regards

 

Minh

 

Envoyé: vendredi 14 juin 2024 à 19:59
De: "Joe Witt" 
À: users@nifi.apache.org
Objet: Re: Nifi on RockyLinux


Deepak
 

In Apache we don't have mechanisms to verify/validate that but indications are that it should work just fine.

 

Thanks

 


On Fri, Jun 14, 2024 at 10:47 AM Chirthani, Deepak Reddy  wrote:





Hi,

 

Writing this email to get a confirmation for if Apache Nifi can be supported on the Linux Distribution System RockyLinux?

 

Thanks

Deepak

The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.








 

 

Accessing File Size in ExecuteGroovyScript script

[James McMahon]

I am trying to use the file size. On the DEtails tab for my flowfile in
queue, I see that my File Size is 8.01 GB.

I log the following from this section of a Groovy script, running in an
ExecuteGroovyScript processor:

def ff = session.get()
if (!ff) return

def jsonFactory = new JsonFactory()
log.info("JsonFactory created successfully.")

def numberOfObjects = 0
def lineCounter = 0  // Initialize a counter to track the number of lines

// Ensure file_size is not null and cast to Integer
def file_size = (ff.size != null) ? (ff.size as Integer) : 0

if (file_size == 0) {
log.error("file_size is undefined or zero, which prevents division.")
session.transfer(ff, REL_FAILURE)
return
}

log.info("File size: ${file_size}")


I want file_size to always be the number of bytes in the file so that I am
always working with a consistent representation.

But in my log, the result is this:
ExecuteGroovyScript[id=1110134d-1ea1-1565-f962-eee47a3fc654] File size:
15408597

That isn't 8.01 GB. Where am I making my error?

Re: Accessing File Size in ExecuteGroovyScript script

[James McMahon]

I should mention that I have also tried to access fileSize like this
without success:

//def file_size = (ff.size != null) ? (ff.size as Integer) : 0
def file_size = ff.getAttribute('fileSize')

This returns null.

I tried this approach based on a post in cloudera, which said this:

|The default FlowFile attributes include:

entryDate

lineageStartDate

fileSize

filename

path

uuid


On Mon, Jun 17, 2024 at 7:50 AM James McMahon  wrote:

> I am trying to use the file size. On the DEtails tab for my flowfile in
> queue, I see that my File Size is 8.01 GB.
>
> I log the following from this section of a Groovy script, running in an
> ExecuteGroovyScript processor:
>
> def ff = session.get()
> if (!ff) return
>
> def jsonFactory = new JsonFactory()
> log.info("JsonFactory created successfully.")
>
> def numberOfObjects = 0
> def lineCounter = 0  // Initialize a counter to track the number of lines
>
> // Ensure file_size is not null and cast to Integer
> def file_size = (ff.size != null) ? (ff.size as Integer) : 0
>
> if (file_size == 0) {
> log.error("file_size is undefined or zero, which prevents division.")
> session.transfer(ff, REL_FAILURE)
> return
> }
>
> log.info("File size: ${file_size}")
>
>
> I want file_size to always be the number of bytes in the file so that I am
> always working with a consistent representation.
>
> But in my log, the result is this:
> ExecuteGroovyScript[id=1110134d-1ea1-1565-f962-eee47a3fc654] File size:
> 15408597
>
> That isn't 8.01 GB. Where am I making my error?
>
>

RE: [EXTERNAL] Accessing File Size in ExecuteGroovyScript script

[Greene (US), Geoffrey N via users]

I believe you are casting ff.size as an Integer.

8 GB is signifincantly larger than can be held in a 32 bit integer, which is 
just 2 GBs.

Suggest you use a double or a big decimal.

Whats happening is you are truncating the high order bits.

From: James McMahon 
Sent: Monday, June 17, 2024 7:51 AM
To: users 
Subject: [EXTERNAL] Accessing File Size in ExecuteGroovyScript script

EXT email: be mindful of links/attachments.


I am trying to use the file size. On the DEtails tab for my flowfile in queue, 
I see that my File Size is 8.01 GB.

I log the following from this section of a Groovy script, running in an 
ExecuteGroovyScript processor:

def ff = session.get()
if (!ff) return

def jsonFactory = new JsonFactory()
log.info("JsonFactory created successfully.")

def numberOfObjects = 0
def lineCounter = 0  // Initialize a counter to track the number of lines

// Ensure file_size is not null and cast to Integer
def file_size = (ff.size != null) ? (ff.size as Integer) : 0

if (file_size == 0) {
log.error("file_size is undefined or zero, which prevents division.")
session.transfer(ff, REL_FAILURE)
return
}

log.info("File size: ${file_size}")


I want file_size to always be the number of bytes in the file so that I am 
always working with a consistent representation.

But in my log, the result is this:
ExecuteGroovyScript[id=1110134d-1ea1-1565-f962-eee47a3fc654] File size: 15408597

That isn't 8.01 GB. Where am I making my error?

Re: Accessing File Size in ExecuteGroovyScript script

[James McMahon]

It appears the best way to do this is to apply the java getSize() method to
the ff object, like this:
def ffSize = ff.getSize()

This seems to output the file size to the log in bytes.

def file_size = (ffSize != null) ? ffSize :0

if (file_size == 0) {
log.error("file_size is undefined or zero, which prevents division.")
session.transfer(ff, REL_FAILURE)
return
}

log.info("File size: ${file_size}")



On Mon, Jun 17, 2024 at 8:31 AM James McMahon  wrote:

> I should mention that I have also tried to access fileSize like this
> without success:
>
> //def file_size = (ff.size != null) ? (ff.size as Integer) : 0
> def file_size = ff.getAttribute('fileSize')
>
> This returns null.
>
> I tried this approach based on a post in cloudera, which said this:
>
> |The default FlowFile attributes include:
>
> entryDate
>
> lineageStartDate
>
> fileSize
>
> filename
>
> path
>
> uuid
>
>
> On Mon, Jun 17, 2024 at 7:50 AM James McMahon 
> wrote:
>
>> I am trying to use the file size. On the DEtails tab for my flowfile in
>> queue, I see that my File Size is 8.01 GB.
>>
>> I log the following from this section of a Groovy script, running in an
>> ExecuteGroovyScript processor:
>>
>> def ff = session.get()
>> if (!ff) return
>>
>> def jsonFactory = new JsonFactory()
>> log.info("JsonFactory created successfully.")
>>
>> def numberOfObjects = 0
>> def lineCounter = 0  // Initialize a counter to track the number of lines
>>
>> // Ensure file_size is not null and cast to Integer
>> def file_size = (ff.size != null) ? (ff.size as Integer) : 0
>>
>> if (file_size == 0) {
>> log.error("file_size is undefined or zero, which prevents division.")
>> session.transfer(ff, REL_FAILURE)
>> return
>> }
>>
>> log.info("File size: ${file_size}")
>>
>>
>> I want file_size to always be the number of bytes in the file so that I
>> am always working with a consistent representation.
>>
>> But in my log, the result is this:
>> ExecuteGroovyScript[id=1110134d-1ea1-1565-f962-eee47a3fc654] File size:
>> 15408597
>>
>> That isn't 8.01 GB. Where am I making my error?
>>
>>

Re: Accessing File Size in ExecuteGroovyScript script

[Rafael Fracasso]

if fileSize is not visible in ExecuteGroovyScript, why don't you try to
updateattribute before the ExecuteGroovyScript and get fileSize attribute
into another attribute, like fileSize_readable = ${fileSize}

in your code you read like this: def file_size =
ff.getAttribute('fileSize_readable')

On Mon, Jun 17, 2024 at 9:59 AM James McMahon  wrote:

> It appears the best way to do this is to apply the java getSize() method
> to the ff object, like this:
> def ffSize = ff.getSize()
>
> This seems to output the file size to the log in bytes.
>
> def file_size = (ffSize != null) ? ffSize :0
>
> if (file_size == 0) {
> log.error("file_size is undefined or zero, which prevents division.")
> session.transfer(ff, REL_FAILURE)
> return
> }
>
> log.info("File size: ${file_size}")
>
>
>
> On Mon, Jun 17, 2024 at 8:31 AM James McMahon 
> wrote:
>
>> I should mention that I have also tried to access fileSize like this
>> without success:
>>
>> //def file_size = (ff.size != null) ? (ff.size as Integer) : 0
>> def file_size = ff.getAttribute('fileSize')
>>
>> This returns null.
>>
>> I tried this approach based on a post in cloudera, which said this:
>>
>> |The default FlowFile attributes include:
>>
>> entryDate
>>
>> lineageStartDate
>>
>> fileSize
>>
>> filename
>>
>> path
>>
>> uuid
>>
>>
>> On Mon, Jun 17, 2024 at 7:50 AM James McMahon 
>> wrote:
>>
>>> I am trying to use the file size. On the DEtails tab for my flowfile in
>>> queue, I see that my File Size is 8.01 GB.
>>>
>>> I log the following from this section of a Groovy script, running in an
>>> ExecuteGroovyScript processor:
>>>
>>> def ff = session.get()
>>> if (!ff) return
>>>
>>> def jsonFactory = new JsonFactory()
>>> log.info("JsonFactory created successfully.")
>>>
>>> def numberOfObjects = 0
>>> def lineCounter = 0  // Initialize a counter to track the number of lines
>>>
>>> // Ensure file_size is not null and cast to Integer
>>> def file_size = (ff.size != null) ? (ff.size as Integer) : 0
>>>
>>> if (file_size == 0) {
>>> log.error("file_size is undefined or zero, which prevents division.")
>>> session.transfer(ff, REL_FAILURE)
>>> return
>>> }
>>>
>>> log.info("File size: ${file_size}")
>>>
>>>
>>> I want file_size to always be the number of bytes in the file so that I
>>> am always working with a consistent representation.
>>>
>>> But in my log, the result is this:
>>> ExecuteGroovyScript[id=1110134d-1ea1-1565-f962-eee47a3fc654] File size:
>>> 15408597
>>>
>>> That isn't 8.01 GB. Where am I making my error?
>>>
>>>

Re: NiFi CLI in secure environment

[David Handermann]

Hi Brant,

Thanks for describing the environment and including the version information.

NiFi 2.0.0-M3 introduced support for the Client Credentials Grant Type in
conjunction with OpenID Connect authentication, but did not include changes
to the NiFi CLI. Support for Client Credentials could be added to the NiFi
CLI now that the REST API supports it, and that would involve a new feature
request.

As far as a solution that could be implemented with existing versions, one
common pattern is to provision a client certificate that is specific to the
NiFi CLI, and use that for automated REST API requests. When configured
with OpenID Connect, NiFi still supports mutual TLS with client
certificates, so that is another way forward.

Regards,
David Handermann

On Wed, Jun 12, 2024 at 3:54 PM Brant Gardner 
wrote:

> We have NiFi 2.0.0-M2 installed in a secure environment (OIDC) and we’re
> trying to utilize the CLI (running on the same machine, so against
> localhost).  We’re finding it nearly impossible to make any calls against
> the server due to 403 Forbidden errors.  It **does** seem to work with
> registry commands, just not nifi commands.
>
>
>
> Is there any documentation for how to contact the server with the CLI in
> this type of configuration?
>
>
>
> *Brant Gardner*
>
> Software Developer – BI & Analytics
>
> Time: GMT -6:00
>
> bcgard...@solventum.com
> [image: A black background with green text Description automatically
> generated]
>
>
>

RE: Re: NiFi CLI in secure environment

[Brant Gardner]

I appreciate the reply David,

Are you aware of documentation for how to provision the client certificate for 
the CLI actions?  We’ve tried this route with limited success, any docs that 
might help in this regard would be greatly appreciated.

Thank you,

Brant Gardner
Software Developer – BI & Analytics
Time: GMT -6:00
Office: +1 651 467 3620 | Mobile: +1 402 470 7895 | 
bcgard...@solventum.com
[A black background with green text  Description automatically generated]

From: David Handermann 
Sent: Monday, June 17, 2024 10:22
To: users@nifi.apache.org
Subject: [EXTERNAL] Re: NiFi CLI in secure environment

Hi Brant, Thanks for describing the environment and including the version 
information. NiFi 2. 0. 0-M3 introduced support for the Client Credentials 
Grant Type in conjunction with OpenID Connect authentication, but did not 
include changes to

Hi Brant,

Thanks for describing the environment and including the version information.

NiFi 2.0.0-M3 introduced support for the Client Credentials Grant Type in 
conjunction with OpenID Connect authentication, but did not include changes to 
the NiFi CLI. Support for Client Credentials could be added to the NiFi CLI now 
that the REST API supports it, and that would involve a new feature request.

As far as a solution that could be implemented with existing versions, one 
common pattern is to provision a client certificate that is specific to the 
NiFi CLI, and use that for automated REST API requests. When configured with 
OpenID Connect, NiFi still supports mutual TLS with client certificates, so 
that is another way forward.

Regards,
David Handermann

On Wed, Jun 12, 2024 at 3:54 PM Brant Gardner 
mailto:bcgard...@solventum.com>> wrote:
We have NiFi 2.0.0-M2 installed in a secure environment (OIDC) and we’re trying 
to utilize the CLI (running on the same machine, so against localhost).  We’re 
finding it nearly impossible to make any calls against the server due to 403 
Forbidden errors.  It *does* seem to work with registry commands, just not nifi 
commands.

Is there any documentation for how to contact the server with the CLI in this 
type of configuration?

Brant Gardner
Software Developer – BI & Analytics
Time: GMT -6:00
bcgard...@solventum.com
[A black background with green textDescription automatically generated]