[
https://issues.apache.org/jira/browse/SDAP-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751774#comment-17751774
]
Riley Kuttruff commented on SDAP-410:
-------------------------------------
Recent work with RMQ has led me to 2 potential causes of this:
# Heartbeat frames from remote aren't being returned by client due to the
thread that owns the BlockingConnection being busy. Solvable by handling
received messages in a worker thread then waiting for it to complete, calling
the connection's sleep method while waiting.
# (Most likely) Message processing times are exceeding the default consumer
ACK timeout (30 minutes). Unfortunately, there's no easy (read: in code) fix
for this; it requires manually setting the value in /etc/rabbitmq/rabbitmq.conf
. The Bitnami image/chart provides no way to do this through environment
variables; it must be mounted to the container as a volume.
These were found while working with pika instead of aiopika.
> Granule Ingester fails on ack-ing RMQ message after pipeline completion
> -----------------------------------------------------------------------
>
> Key: SDAP-410
> URL: https://issues.apache.org/jira/browse/SDAP-410
> Project: Apache Science Data Analytics Platform
> Issue Type: Bug
> Components: granule-ingester
> Reporter: Riley Kuttruff
> Priority: Minor
>
> Seems to happen only when the pipeline runs for a very long time.
>
> Traceback:
>
> 2022-11-10 00:57:02,974 [ERROR] [__main__::197] Shutting down because of an
> unrecoverable error:
> 'NoneType' object has no attribute 'write'
> Traceback (most recent call last):
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/MessageConsumer.py",
> line 80, in _received_message
> await message.ack()
> File
> "/opt/conda/lib/python3.7/site-packages/aio_pika-6.7.1-py3.7.egg/aio_pika/message.py",
> line 582, in ack
> delivery_tag=self.delivery_tag, multiple=multiple,
> File
> "/opt/conda/lib/python3.7/site-packages/aiormq-3.3.1-py3.7.egg/aiormq/channel.py",
> line 436, in basic_ack
> self.writer.write(
> AttributeError: 'NoneType' object has no attribute 'write'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/sdap/granule_ingester/main.py", line 191, in main
> await consumer.start_consuming(args.max_threads)
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/MessageConsumer.py",
> line 113, in start_consuming
> raise e
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/MessageConsumer.py",
> line 105, in start_consuming
> pipeline_max_concurrency)
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/MessageConsumer.py",
> line 92, in _received_message
> await message.reject(requeue=True)
> File
> "/opt/conda/lib/python3.7/site-packages/aio_pika-6.7.1-py3.7.egg/aio_pika/message.py",
> line 611, in reject
> delivery_tag=self.delivery_tag, requeue=requeue,
> File
> "/opt/conda/lib/python3.7/site-packages/aiormq-3.3.1-py3.7.egg/aiormq/channel.py",
> line 468, in basic_reject
> self.writer.write(
> AttributeError: 'NoneType' object has no attribute 'write'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
