Re: [ANNOUNCE] New Apache NiFi Committer Chris Sampson

[Otto Fowler]

Awesome!  Congratulations!

> On May 13, 2021, at 16:25, Joe Witt  wrote:
> 
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Chris has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate the contributions from Chris
> on the mailing list, PR reviews, code contributions, and especially rich
> communications in the Apache NiFi slack channels.
> 
> Welcome and congratulations!

Re: [ANNOUNCE] New Apache NiFi Committer Chris Sampson

[David Handermann]

Congratulations Chris!  Looking forward to your continued contributions!

Regards,
David Handermann

On Thu, May 13, 2021 at 3:38 PM Matt Burgess  wrote:

> Congratulations and welcome aboard Chris!
>
> On Thu, May 13, 2021 at 4:25 PM Joe Witt  wrote:
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Chris has accepted the PMC's invitation to become a committer on the
> > Apache NiFi project. We greatly appreciate the contributions from Chris
> > on the mailing list, PR reviews, code contributions, and especially rich
> > communications in the Apache NiFi slack channels.
> >
> > Welcome and congratulations!
>

Re: [ANNOUNCE] New Apache NiFi Committer Chris Sampson

[Matt Burgess]

Congratulations and welcome aboard Chris!

On Thu, May 13, 2021 at 4:25 PM Joe Witt  wrote:
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Chris has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate the contributions from Chris
> on the mailing list, PR reviews, code contributions, and especially rich
> communications in the Apache NiFi slack channels.
>
> Welcome and congratulations!

[ANNOUNCE] New Apache NiFi Committer Chris Sampson

[Joe Witt]

On behalf of the Apache NiFI PMC, I am very pleased to announce that
Chris has accepted the PMC's invitation to become a committer on the
Apache NiFi project. We greatly appreciate the contributions from Chris
on the mailing list, PR reviews, code contributions, and especially rich
communications in the Apache NiFi slack channels.

Welcome and congratulations!

Re: PutFile to mounted directory

[Phil H]

Thanks Shawn and Eduardo - two great ideas to pursue!

Cheers,
Phil

On Thu, 13 May 2021 at 22:45, Shawn Weeks  wrote:

> You could set the permissions on the unmounted folder where NiFi can't
> write to them since the permissions when mounted come from the NFS share.
>
> For example assume we're mounting to /mnt/nifi_dropoff, before mounting to
> that mount point create the directory set the permissions to 555 or
> something but after mounting set it to 755 or whatever permissions you
> want. Then you'll get a permission denied if the folder isn't mounted.
>
> Thanks
> Shawn
>
> -Original Message-
> From: Eduardo Fontes 
> Sent: Thursday, May 13, 2021 5:12 AM
> To: dev@nifi.apache.org
> Subject: Re: PutFile to mounted directory
>
> Hi,
>
> You can use PutSMBFile if it is a Cifs/Smb/Windows share.
>
> If it is a NFS share, the only way is to mount the share on the nifi nodes.
>
> Regards,
>
> Eduardo Fontes
>
> Em qui., 13 de mai. de 2021 01:11, Phil H  escreveu:
>
> > Hi there,
> >
> > I want to use PutFile write some flow files out to a network drive on
> > my Unix system, but fail if the disk is not mounted. I can’t see any
> > way to do this - any ideas?
> >
> > Cheers,
> > Phil
> >
>

Re: Streaming Splitting Processor

[Eric Secules]

Hello,

Sorry to revive an older thread. I just created a feature ticket
for using non-heap
collections in the StandardProcessSession based on our conversation here. I
think it would be beneficial if the session behaved similarly to the nifi
queues by having the ability to store at least some of the FlowFiles in the
session off-heap.

Thanks,
Eric

On Tue, Mar 9, 2021 at 10:25 AM Eric Secules  wrote:

> Hi Mark,
>
> The NPE Came after trying to retrieve the parent flowfile from the
> outSession's records map. The parent flowfile was not migrated to
> outSession. I don't think it's right to migrate the parent flowfile to the
> output session as well.
>
> Thanks,
> Eric
>
> On Tue, Mar 9, 2021 at 10:02 AM Eric Secules  wrote:
>
>> Hi Mark,
>>
>> Thanks for the details! With that in mind I can also make the operation
>> more transactional by using wait and notify (won't solve the duplication on
>> restart issue).
>>
>> I tried the example code and I got an NPE inside commit(). I initially
>> thought this was because I was running version 1.11.4. I  upgraded to NiFi
>> 1.13.0 and the problem still persists where repoRecord is Null on line 786
>> in StandardProcessSession.java
>>
>> Stack Trace:
>>
>>> 2021-03-09 17:41:44,013 ERROR [Timer-Driven Process Thread-7]
>>> c.m.p.p.MyProcessor MyProcessor[id=f3439a7a-cb5a-3c76-121e-60c448d1f910]
>>> Failure FF: null: java.lang.NullPointerException
>>> java.lang.NullPointerException: null
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.updateEventContentClaims(StandardProcessSession.java:786)
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:600)
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:353)
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:332)
>>>
>>> ... My code calling outSession.comit()
>>>
>>
>> This may be my issue since my code doesn't look exactly like the example,
>> but it would be good to have a better exception than NPE.
>>
>> Thanks,
>> Eric
>>
>>
>> On Mon., Mar. 8, 2021, 1:30 p.m. Mark Payne, 
>> wrote:
>>
>>> Eric,
>>>
>>> The session holds onto all ‘active flowfiles’ as well as other
>>> information about them, such as which relationship they’ve been transferred
>>> to, which attributes have been added, removed, etc. So if you have a
>>> separate session for each outbound FlowFile, each time that session is
>>> committed, all those FlowFiles can be removed from the session and placed
>>> in the next queue. This is important because the session holds all
>>> FlowFiles it knows about in memory, but the queues will swap the flowfiles
>>> out - writing them to disk and then dropping them from memory/heap. This
>>> allows us to hold many millions of FlowFiles within NiFi at a time.
>>> However, if you’re trying to hold a million FlowFiles in heap, you’re going
>>> to use a huge amount of heap. This is predominantly due to the HashMap that
>>> is used to store attributes. Those can quickly use up a huge amount of heap.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> > On Mar 8, 2021, at 12:53 PM, Eric Secules  wrote:
>>> >
>>> > Hi Mark,
>>> >
>>> > Thanks for the example code and the considerations. I see the error I
>>> made
>>> > before.
>>> > If I were to use the standard way and put this all in one session what
>>> are
>>> > the memory characteristics? What is tracked in memory for a session
>>> and is
>>> > this actually any different (in overall memory use) from my approach of
>>> > using a separate session for output?
>>> >
>>> > Thanks,
>>> > Eric
>>> >
>>> > On Sat, Mar 6, 2021 at 1:22 PM Mark Payne 
>>> wrote:
>>> >
>>> >> Hi Eric,
>>> >>
>>> >> We should definitely throw a better Exception there, rather than
>>> >> NullPointerException. But what you’re looking to do can be done. It’s
>>> >> slightly more nuanced than that, though. What you’ll need to do, in
>>> order
>>> >> to maintain the lineage, is to create the child FlowFile in the
>>> session
>>> >> that owns the parent. You can then “migrate” the child FlowFile to a
>>> new
>>> >> session. Something to the effect of:
>>> >>
>>> >> -
>>> >> ProcessSession inSession = sessionFactory.create();
>>> >> FlowFile flowFile = inSession.get();
>>> >> if (flowFile == null) {
>>> >>  return;
>>> >> }
>>> >>
>>> >> ProcessSession outSession = sessionFactory.create();
>>> >>
>>> >> try (InputStream in = inSession.read(flowFile)) {
>>> >>  Record record;
>>> >>  while ((record = getRecord(in)) {
>>> >>  FlowFile child = inSession.create(flowFile);
>>> >>  try (OutputStream out = inSession.write(child)) {
>>> >>// write some data.
>>> >>  }
>>> >>
>>> >>  inSession.migrate(outSession, Collections.singleton(child));
>>> >>  outSession.transfer(child, REL_SUCCESS);
>>> >>  outSession.commit();
>>> 

Re: Jira contributor access

[Mark Payne]

Hi Daniel,

I’ve added you as a contributor.

Thanks!
-Mark


> On May 13, 2021, at 9:46 AM, Dániel Urbán  wrote:
> 
> Hi everyone,
> 
> I would like to ask for Jira contributor access.
> My username is durban.
> 
> Thanks in advance,
> Daniel

Jira contributor access

[Dániel Urbán]

Hi everyone,

I would like to ask for Jira contributor access.
My username is durban.

Thanks in advance,
Daniel

RE: PutFile to mounted directory

[Shawn Weeks]

You could set the permissions on the unmounted folder where NiFi can't write to 
them since the permissions when mounted come from the NFS share.

For example assume we're mounting to /mnt/nifi_dropoff, before mounting to that 
mount point create the directory set the permissions to 555 or something but 
after mounting set it to 755 or whatever permissions you want. Then you'll get 
a permission denied if the folder isn't mounted.

Thanks
Shawn

-Original Message-
From: Eduardo Fontes  
Sent: Thursday, May 13, 2021 5:12 AM
To: dev@nifi.apache.org
Subject: Re: PutFile to mounted directory

Hi,

You can use PutSMBFile if it is a Cifs/Smb/Windows share.

If it is a NFS share, the only way is to mount the share on the nifi nodes.

Regards,

Eduardo Fontes

Em qui., 13 de mai. de 2021 01:11, Phil H  escreveu:

> Hi there,
>
> I want to use PutFile write some flow files out to a network drive on 
> my Unix system, but fail if the disk is not mounted. I can’t see any 
> way to do this - any ideas?
>
> Cheers,
> Phil
>

Re: PutFile to mounted directory

[Eduardo Fontes]

Hi,

You can use PutSMBFile if it is a Cifs/Smb/Windows share.

If it is a NFS share, the only way is to mount the share on the nifi nodes.

Regards,

Eduardo Fontes

Em qui., 13 de mai. de 2021 01:11, Phil H  escreveu:

> Hi there,
>
> I want to use PutFile write some flow files out to a network drive on my
> Unix system, but fail if the disk is not mounted. I can’t see any way to do
> this - any ideas?
>
> Cheers,
> Phil
>