CompressContent hadoop-snappy

[Noe Detore]

Hello

CompressContent ver 1.9 uses snappy-java. Is there an easy way to change it
to hadoop-snappy? Or a custom processor needs to be created?

thank you
Noe

Re: AutoAck false AMQ

[Mark Payne]

Dweep,

The difference that NiFi offers between Auto Acknowledge and Client Acknowledge 
is that Auto Acknowledge is much faster but at the expense of possibly losing 
messages (if it is acknowledge, then NiFi is shutdown before persisting the 
data). With Client Acknowledge, the ConsumeAMQP Processor will persist the data 
to its repositories and only then acknowledge the message. As a result, if NiFi 
is restarted, the data will not be lost but could perhaps be duplicated, if the 
data is persisted and then NiFi restarts before the message is acknowledged. 
But on restart, NiFi will have the data wherever it was in the flow and 
automatically resume processing it.

There is also an effort underway known as Stateless NiFi [1], which would offer 
semantics more like what you're proposing, where the entire flow would be 
executed and only then the message would be acknowledged. This would require 
that the flow be stored in the NiFi Registry and then Stateless NiFi could be 
launched. But this effort is still pretty well in its infancy and as such 
doesn't have quite the level of documentation that you may be accustomed to 
with NiFi yet.

[1] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-stateless/README.md

On Nov 22, 2019, at 4:10 AM, Dweep Sharma 
mailto:dweep.sha...@redbus.com>> wrote:

Hi,

I want to use the ConsumeAMQ processor to consume events from rabbitmq.

The second step is to make an API call to acquire more data and finally merge 
contents

and store it as Parquet in S3.

My question is, since there are so many steps after the consumeAMQ, how can i 
acknowledge the message only after all these steps are complete to prevent any 
loss of data

The AutoAck false property will not acknowledge the message by default but when 
does the message get dequeued if this is set to false?

-Dweep

::DISCLAIMER::


The contents of this e-mail and any attachments are confidential and intended 
for the named recipient(s) only.E-mail transmission is not guaranteed to be 
secure or error-free as information could be intercepted, corrupted,lost, 
destroyed, arrive late or incomplete, or may contain viruses in transmission. 
The e mail and its contents(with or without referred errors) shall therefore 
not attach any liability on the originator or redBus.com. 
Views or opinions, if any, presented in this email are solely those of the 
author and may not necessarily reflect the views or opinions of 
redBus.com. Any form of reproduction, dissemination, 
copying, disclosure, modification,distribution and / or publication of this 
message without the prior written consent of authorized representative of 
redbus.com is strictly prohibited. If you have received this 
email in error please delete it and notify the sender immediately.Before 
opening any email and/or attachments, please check them for viruses and other 
defects.