pkelly-nifi commented on PR #11430: URL: https://github.com/apache/nifi/pull/11430#issuecomment-4972944874
Hi @exceptionfactory, thank you for your input. Admittedly this is a narrow use case, but it can be important. To answer your question, setting the visibility timeout in PutSQS processor can't be done. There is no option in the SQS API for setting the visibility timeout on the message during the SendMessage operation, so it's not something we could incorporate into PutSQS even if we wanted to. You can set a message delay when sending a message, but not a visibility timeout. The difference is that a delay affects the time between when you send the message and when a client can first receive that message, whereas the visibility timeout is the interval between when a client first receives the message (but does not delete it) and when the message is considered to be a failure and SQS automatically replays it to another client. It will continue to be replayed at the visibility timeout interval until it is deleted by a client or considered to be a dead letter by the service, based on the configuration of the queue. You can set a default visibility timeout for the queue, but that doesn't really affect NiFi since the default visibility timeout is overwritten by the GetSQS processor when the message is retrieved. This is done during GetSQS's ReceiveMessage operation, before you know anything about the message, so all messages retrieved by a single GetSQS use a common visibility timeout. We could allow GetSQS's visibility timeout to be empty, in which case SQS would fallback to the visibility timeout set on the queue, but that still doesn't help the use case here. I think there is one important use case for this processor: being able to set a per-message visibility timeout for S3 event notifications, which something NiFi cannot currently do that this new processor allows. Let's say you have an S3 bucket that normally receives small files (on the order of a few MBs), but sometimes receives very large files (hundreds of GBs or into TBs), and you have S3 set up to send you event notifications to SQS for objects that are uploaded. You drive the flow for processing those files using a GetSQS processor to receive these event notifications. You want to be able to very quickly retry any small files that fail, but you need to allow a lot of time for processing the big files. NiFi cannot do this currently. Your GetSQS would need to use a visibility timeout large enough to cover your slowest file, which for this case would be several hours. Small files that fail would need to wait this maximum interval until their notifications are replayed. Ev en if we allowed the visibility timeout in GetSQS to be empty, in which case you would use the queue's default visibility timeout, that wouldn't help because S3 event notifications cannot set different visibility timeouts based on the object size. Doing so is up to the receiving client. What you could do instead given this new processor is set the visibility timeout in GetSQS to only a few minutes to cover the expected small files. You would then feed the event notification messages into EvaluateJsonPath to extract the object's size from the message, then evaluate the size and optionally use this new processor to set a new visibility timeout for messages for large objects (SQS supports up to 12 hours) in order to give you enough time to process them. Messages for small objects would replay on the shorter interval while allowing you enough time to process the large files without their messages getting replayed. There are several other cases that are similar, but they are all derived from this same concept: you want to be able to set different replay intervals based on some characteristic of the message, which NiFi cannot currently do. There are two alternatives to this processor: 1) Using the longest expected interval as your visibility timeout as previously described, which neglects the ability to replay messages for smaller files. 2) Setting up a more much more complicated flow with multiple SQS queues for different object sizes. Because S3 event notifications cannot filter based on object size, you'd have a single queue receive your initial S3 event notification, then either use NiFi or a lambda function to evaluate the message and pass it onto a different queues based on the size. A different GetSQS for each follow-on queue would be set up to use a different visibility timeout for each queue, and you'd need to keep track of which queue a given message came from so that you can later delete it from the proper queue. This leads to a lot of processor duplication and complication throughout the NiFi flow. We have always been using alternative #1, but we started migrating towards #2 recently, and decided this processor would greatly help to simplify the flow and infrastructure. AWS's documentation has a pretty good description of their recommended practices for changing the visibility timeout of a message: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html I'm happy to discuss further, but I think this could be a handy, if not critical, feature for some flows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
