Re: New Apache NiFi Website Design Launched
Looks Great! Well Done! Jörg Am 08.01.2024 um 19:17 schrieb David Handermann: Team, Thanks to a collaborative effort from several designers and developers, the Apache NiFi project website has a new look, with more prominent links to downloads, documentation, and source code! https://nifi.apache.org There is more work to be done in particular areas like generated documentation, but as with the project itself, the website is open for collaborative input through Jira [1] and GitHub [2]. Regards, David Handermann Apache NiFi PMC Member [1] https://issues.apache.org/jira/browse/NIFI [2] https://github.com/apache/nifi-site
Re: New Apache NiFi Website Design Launched
It looks awesome. Great work. When time permits, please add some more projects and update the videos to match the current version since it's 7 years old. Thanks On Mon, Jan 8, 2024 at 1:19 PM David Handermann wrote: > Team, > > Thanks to a collaborative effort from several designers and > developers, the Apache NiFi project website has a new look, with more > prominent links to downloads, documentation, and source code! > > https://nifi.apache.org > > There is more work to be done in particular areas like generated > documentation, but as with the project itself, the website is open for > collaborative input through Jira [1] and GitHub [2]. > > Regards, > David Handermann > Apache NiFi PMC Member > > [1] https://issues.apache.org/jira/browse/NIFI > [2] https://github.com/apache/nifi-site > -- PROPRIETARY / CONFIDENTIALITY NOTICE This message (including any attachments) contains information that is proprietary and confidential to PRIORITY TECHNOLOGY HOLDINGS, INC. and its affiliates and subsidiaries, including the sender hereof, and is for the sole use of the intended recipients. If you are not an intended recipient, you may not read, print, retain, use, copy, distribute, forward or disclose to anyone this message or any information contained in this message (including any attachments). If you have received this message in error, please advise the sender of this error by reply e-mail, and please destroy all copies of this message (including any attachments).
Re: New Apache NiFi Website Design Launched
Agreed! It looks awesome. Great work. Brandon From: Steven Matison Sent: Monday, January 8, 2024 1:42:36 PM To: users@nifi.apache.org Subject: Re: New Apache NiFi Website Design Launched So awesome to see this! Thanks David and everyone who made this a reality! You all are true rock stars. Happy new nifi year, Steven On Mon, Jan 8, 2024 at 1:19 PM David Handermann mailto:exceptionfact...@apache.org>> wrote: Team, Thanks to a collaborative effort from several designers and developers, the Apache NiFi project website has a new look, with more prominent links to downloads, documentation, and source code! https://nifi.apache.org There is more work to be done in particular areas like generated documentation, but as with the project itself, the website is open for collaborative input through Jira [1] and GitHub [2]. Regards, David Handermann Apache NiFi PMC Member [1] https://issues.apache.org/jira/browse/NIFI [2] https://github.com/apache/nifi-site
Re: New Apache NiFi Website Design Launched
So awesome to see this! Thanks David and everyone who made this a reality! You all are true rock stars. Happy new nifi year, Steven On Mon, Jan 8, 2024 at 1:19 PM David Handermann wrote: > Team, > > Thanks to a collaborative effort from several designers and > developers, the Apache NiFi project website has a new look, with more > prominent links to downloads, documentation, and source code! > > https://nifi.apache.org > > There is more work to be done in particular areas like generated > documentation, but as with the project itself, the website is open for > collaborative input through Jira [1] and GitHub [2]. > > Regards, > David Handermann > Apache NiFi PMC Member > > [1] https://issues.apache.org/jira/browse/NIFI > [2] https://github.com/apache/nifi-site >
Re: New Apache NiFi Website Design Launched
Changes look great! Nice job! On Mon, Jan 8, 2024 at 1:24 PM Joe Witt wrote: > Thanks so much to all involved and David for driving this to completion! > > It looks great > > On Mon, Jan 8, 2024 at 11:17 AM David Handermann < > exceptionfact...@apache.org> wrote: > >> Team, >> >> Thanks to a collaborative effort from several designers and >> developers, the Apache NiFi project website has a new look, with more >> prominent links to downloads, documentation, and source code! >> >> https://nifi.apache.org >> >> There is more work to be done in particular areas like generated >> documentation, but as with the project itself, the website is open for >> collaborative input through Jira [1] and GitHub [2]. >> >> Regards, >> David Handermann >> Apache NiFi PMC Member >> >> [1] https://issues.apache.org/jira/browse/NIFI >> [2] https://github.com/apache/nifi-site >> >
Re: New Apache NiFi Website Design Launched
Thanks so much to all involved and David for driving this to completion! It looks great On Mon, Jan 8, 2024 at 11:17 AM David Handermann < exceptionfact...@apache.org> wrote: > Team, > > Thanks to a collaborative effort from several designers and > developers, the Apache NiFi project website has a new look, with more > prominent links to downloads, documentation, and source code! > > https://nifi.apache.org > > There is more work to be done in particular areas like generated > documentation, but as with the project itself, the website is open for > collaborative input through Jira [1] and GitHub [2]. > > Regards, > David Handermann > Apache NiFi PMC Member > > [1] https://issues.apache.org/jira/browse/NIFI > [2] https://github.com/apache/nifi-site >
New Apache NiFi Website Design Launched
Team, Thanks to a collaborative effort from several designers and developers, the Apache NiFi project website has a new look, with more prominent links to downloads, documentation, and source code! https://nifi.apache.org There is more work to be done in particular areas like generated documentation, but as with the project itself, the website is open for collaborative input through Jira [1] and GitHub [2]. Regards, David Handermann Apache NiFi PMC Member [1] https://issues.apache.org/jira/browse/NIFI [2] https://github.com/apache/nifi-site
Re: Hardware requirement for NIFI instance
Hello Mark,
I shared the screenshot about ExecuteSQL properties.
This 3 properties together work very well for PostGRESQL in the NIFI with 2CPU/8Gb.
Without the properties "Set Auto Commit" to FALSE, the NIFI stuck and crash.
Thanks a lot for help
Minh
Envoyé: vendredi 5 janvier 2024 à 15:52
De: "Mark Payne"
À: "users@nifi.apache.org"
Objet: Re: Hardware requirement for NIFI instance
Thanks for following up. That actually makes sense. I don’t think Output Batch Size will play a very big role here. But Fetch Size, if I understand correctly, is essentially telling the JDBC Driver “Here’s how many rows you should pull back at once.” And so it’s going to buffer all of those rows into memory until it has written out all of them.
So if you set Fetch Size = 0, it’s going to pull back all rows in your database into memory. To be honest, I cannot imagine a single scenario where that’s desirable. We should probably set the default to something reasonable like 1,000 or 10,000 at most. And in 2.0, where we have the ability to migrate old configurations we should automatically change any config that has Fetch Size of 0 to the default value.
@Matt Burgess, et al., any concerns with that?
Thanks
-Mark
On Jan 5, 2024, at 9:45 AM, e-soci...@gmx.fr wrote:
So after some tests, here the result perhaps could help someone.
With nifi (2CPU / 8Go Ram)
I have tested with these couples properties :
> 1 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 10
> 2 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 20
> 2 executeSQL with "select * from table"
Output Batch Size : 1
Fetch Size : 40
and started 5 executeSQL in the same time
The 5 processors work perfectly and receive 5 avro files with same size.
And during the test, the memory is stable and the Web UI works perfectly
FAILED TEST "OUT OF MEMORY" if the properties are :
> 1 executeSQL with "select * from table"
Output Batch Size : 0
Fetch Size : 0
Regards
Envoyé: vendredi 5 janvier 2024 à 08:12
De: "Matt Burgess"
À: users@nifi.apache.org
Objet: Re: Hardware requirement for NIFI instance
You may not need to merge if your Fetch Size is set appropriately. For
your case I don't recommend setting Max Rows Per Flow File because you
still have to wait for all the results to be processed before the
FlowFile(s) get sent "downstream". Also if you set Output Batch Size
you can't use Merge downstream as ExecuteSQL will send FlowFiles
downstream before it knows the total count.
If you have a NiFi cluster and not a standalone instance you MIGHT be
able to represent your complex query using GenerateTableFetch and use
a load-balanced connection to grab different "pages" of the table in
parallel with ExecuteSQL. Those can be merged later as long as you get
all the FlowFiles back to a single node. Depending on how complex your
query is then it's a long shot but I thought I'd mention it just in
case.
Regards,
Matt
On Thu, Jan 4, 2024 at 1:41 PM Pierre Villard
wrote:
>
> You can merge multiple Avro flow files with MergeRecord with an Avro Reader and an Avro Writer
>
> Le jeu. 4 janv. 2024 à 22:05, a écrit :
>>
>> And the important thing for us it has only one avro file by table.
>>
>> So it is possible to merge avro files to one avro file ?
>>
>> Regards
>>
>>
>> Envoyé: jeudi 4 janvier 2024 à 19:01
>> De: e-soci...@gmx.fr
>> À: users@nifi.apache.org
>> Cc: users@nifi.apache.org
>> Objet: Re: Hardware requirement for NIFI instance
>>
>> Hello all,
>>
>> Thanks a lot for the reply.
>>
>> So for more details.
>>
>> All the properties for the ExecuteSQL are set by default, except "Set Auto Commit: false".
>>
>> The sql command could not be more simple than "select * from ${db.table.fullname}"
>>
>> The nifi version is 1.16.3 and 1.23.2
>>
>> I have also test the same sql command in the another nifi (8 cores/ 16G Ram) and it is working.
>> The result is the avro file with 1.6GB
>>
>> The detail about the output flowfile :
>>
>> executesql.query.duration
>> 245118
>> executesql.query.executiontime
>> 64122
>> executesql.query.fetchtime
>> 180996
>> executesql.resultset.index
>> 0
>> executesql.row.count
>> 14961077
>>
>> File Size
>> 1.62 GB
>>
>> Regards
>>
>> Minh
>>
>>
>> Envoyé: jeudi 4 janvier 2024 à 17:18
>> De: "Matt Burgess"
>> À: users@nifi.apache.org
>> Objet: Re: Hardware requirement for NIFI instance
>> If I remember correctly, the default Fetch Size for Postgresql is to
>> get all the rows at once, which can certainly cause the problem.
>> Perhaps try setting Fetch Size to something like 1000 or so and see if
>> that alleviates the problem.
>>
>> Regards,
>> Matt
>>
>> On Thu, Jan 4, 2024 at 8:48 AM Etienne Jouvin wrote:
>> >
>> > Hello.
>> >
>> > I also think the problem is more about the processor, I guess ExecuteSQL.
>> >
>> > Should play with batch configuration and commit flag to commit
