Re: [python-committers] miss-islington backport pipeline is stalled?
I'm seeing more of this today, just heads up in case you see miss-islington
not working.
Seems like this is a known issue with celery + kombu
https://github.com/celery/kombu/issues/1019
ᐧ
On Wed, May 8, 2019 at 10:02 AM Mariatta wrote:
> There was an error from Redis. I think this is the first time I've seen
> it, so I don't have any resolution on how to fix it right now.
> I will look into handling the error and have miss-islington leave a
> comment in the PR when there is such error.
>
> log:
>
> at=info method=POST path="/" host=miss-islington.herokuapp.com
> request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15"
> dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https
> May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
> 600, in send_packed_command
> May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item)
> May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110]
> Connection timed out
> May 08 09:35:15 miss-islington app/web.1: During handling of the above
> exception, another exception occurred:
> May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line
> 431, in _reraise_as_library_errors
> May 08 09:35:15 miss-islington app/web.1: yield
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line
> 744, in send_task
> May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P,
> task_id)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
> line 265, in on_task_call
> May 08 09:35:15 miss-islington app/web.1:
> self.result_consumer.consume_from(task_id)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
> line 126, in consume_from
> May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
> line 132, in _consume_from
> May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096,
> in subscribe
> May 08 09:35:15 miss-islington app/web.1: ret_val =
> self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009,
> in execute_command
> May 08 09:35:15 miss-islington app/web.1: self._execute(connection,
> connection.send_command, *args)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013,
> in _execute
> May 08 09:35:15 miss-islington app/web.1: return command(*args)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
> 620, in send_command
> May 08 09:35:15 miss-islington app/web.1:
> self.send_packed_command(self.pack_command(*args))
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
> 613, in send_packed_command
> May 08 09:35:15 miss-islington app/web.1: (errno, errmsg))
> May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError:
> Error 110 while writing to socket. Connection timed out.
>
>
>
> ᐧ
>
> On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith wrote:
>
>> Yesterday it failed to produce a backport or tell me that it couldn't
>> (after the "i'm now working on ..." message was left on the master PR). I
>> waited a couple hours just in case and ran cherry_picker myself instead.
>> Same thing today apparently on
>> https://github.com/python/cpython/pull/13192 assuming the usual backport
>> latency is no more than a minute or two.
>>
>> -gps
>> ___
>> python-committers mailing list
>> python-committers@python.org
>> https://mail.python.org/mailman/listinfo/python-committers
>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>
>
___
python-committers mailing list
python-committers@python.org
https://mail.python.org/mailman/listinfo/python-committers
Code of Conduct: https://www.python.org/psf/codeofconduct/
Re: [python-committers] miss-islington backport pipeline is stalled?
We don't have sentry (I will ask Ernest about it), but such errors are
already sent to Zulip (
https://python.zulipchat.com/#narrow/stream/142203-workflow.2Fbot-api.20rate.20limit/topic/alerts/near/165174773)
and core-workflow
https://github.com/python/core-workflow/issues/257#issuecomment-490541831
But posting back to the PR itself will still be useful.
ᐧ
On Wed, May 8, 2019 at 10:04 AM Alex Gaynor wrote:
> Would it make sense to work with the PSF infra staff so that
> miss-isslington is hooked up to the PSF Sentry account so folks can get
> email notifications and similar on unhandled exceptions?
>
> Alex
>
> On Wed, May 8, 2019 at 1:02 PM Mariatta wrote:
>
>> There was an error from Redis. I think this is the first time I've seen
>> it, so I don't have any resolution on how to fix it right now.
>> I will look into handling the error and have miss-islington leave a
>> comment in the PR when there is such error.
>>
>> log:
>>
>> at=info method=POST path="/" host=miss-islington.herokuapp.com
>> request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15"
>> dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https
>> May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
>> 600, in send_packed_command
>> May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item)
>> May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110]
>> Connection timed out
>> May 08 09:35:15 miss-islington app/web.1: During handling of the above
>> exception, another exception occurred:
>> May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line
>> 431, in _reraise_as_library_errors
>> May 08 09:35:15 miss-islington app/web.1: yield
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line
>> 744, in send_task
>> May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P,
>> task_id)
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
>> line 265, in on_task_call
>> May 08 09:35:15 miss-islington app/web.1:
>> self.result_consumer.consume_from(task_id)
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
>> line 126, in consume_from
>> May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id)
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
>> line 132, in _consume_from
>> May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key)
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line
>> 3096, in subscribe
>> May 08 09:35:15 miss-islington app/web.1: ret_val =
>> self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line
>> 3009, in execute_command
>> May 08 09:35:15 miss-islington app/web.1: self._execute(connection,
>> connection.send_command, *args)
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line
>> 3013, in _execute
>> May 08 09:35:15 miss-islington app/web.1: return command(*args)
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
>> 620, in send_command
>> May 08 09:35:15 miss-islington app/web.1:
>> self.send_packed_command(self.pack_command(*args))
>> May 08 09:35:15 miss-islington app/web.1: File
>> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
>> 613, in send_packed_command
>> May 08 09:35:15 miss-islington app/web.1: (errno, errmsg))
>> May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError:
>> Error 110 while writing to socket. Connection timed out.
>>
>>
>>
>> ᐧ
>>
>> On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith wrote:
>>
>>> Yesterday it failed to produce a backport or tell me that it couldn't
>>> (after the "i'm now working on ..." message was left on the master PR). I
>>> waited a couple hours just in case and ran cherry_picker myself instead.
>>> Same thing today apparently on
>>> https://github.com/python/cpython/pull/13192 assuming the usual
>>> backport latency is no more than a minute or two.
>>>
>>> -gps
>>> ___
>>> python-committers mailing list
>>> python-committers@python.org
>>>
Re: [python-committers] miss-islington backport pipeline is stalled?
Would it make sense to work with the PSF infra staff so that
miss-isslington is hooked up to the PSF Sentry account so folks can get
email notifications and similar on unhandled exceptions?
Alex
On Wed, May 8, 2019 at 1:02 PM Mariatta wrote:
> There was an error from Redis. I think this is the first time I've seen
> it, so I don't have any resolution on how to fix it right now.
> I will look into handling the error and have miss-islington leave a
> comment in the PR when there is such error.
>
> log:
>
> at=info method=POST path="/" host=miss-islington.herokuapp.com
> request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15"
> dyno=web.1 connect=1ms service=1005ms status=200 bytes=168 protocol=https
> May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
> 600, in send_packed_command
> May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item)
> May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110]
> Connection timed out
> May 08 09:35:15 miss-islington app/web.1: During handling of the above
> exception, another exception occurred:
> May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py", line
> 431, in _reraise_as_library_errors
> May 08 09:35:15 miss-islington app/web.1: yield
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py", line
> 744, in send_task
> May 08 09:35:15 miss-islington app/web.1: self.backend.on_task_call(P,
> task_id)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
> line 265, in on_task_call
> May 08 09:35:15 miss-islington app/web.1:
> self.result_consumer.consume_from(task_id)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
> line 126, in consume_from
> May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
> line 132, in _consume_from
> May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3096,
> in subscribe
> May 08 09:35:15 miss-islington app/web.1: ret_val =
> self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3009,
> in execute_command
> May 08 09:35:15 miss-islington app/web.1: self._execute(connection,
> connection.send_command, *args)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/client.py", line 3013,
> in _execute
> May 08 09:35:15 miss-islington app/web.1: return command(*args)
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
> 620, in send_command
> May 08 09:35:15 miss-islington app/web.1:
> self.send_packed_command(self.pack_command(*args))
> May 08 09:35:15 miss-islington app/web.1: File
> "/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py", line
> 613, in send_packed_command
> May 08 09:35:15 miss-islington app/web.1: (errno, errmsg))
> May 08 09:35:15 miss-islington app/web.1: redis.exceptions.ConnectionError:
> Error 110 while writing to socket. Connection timed out.
>
>
>
> ᐧ
>
> On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith wrote:
>
>> Yesterday it failed to produce a backport or tell me that it couldn't
>> (after the "i'm now working on ..." message was left on the master PR). I
>> waited a couple hours just in case and ran cherry_picker myself instead.
>> Same thing today apparently on
>> https://github.com/python/cpython/pull/13192 assuming the usual backport
>> latency is no more than a minute or two.
>>
>> -gps
>> ___
>> python-committers mailing list
>> python-committers@python.org
>> https://mail.python.org/mailman/listinfo/python-committers
>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>
> ___
> python-committers mailing list
> python-committers@python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
--
All that is necessary for evil to succeed is for good people to do nothing.
___
python-committers mailing list
Re: [python-committers] miss-islington backport pipeline is stalled?
There was an error from Redis. I think this is the first time I've seen it,
so I don't have any resolution on how to fix it right now.
I will look into handling the error and have miss-islington leave a comment
in the PR when there is such error.
log:
at=info method=POST path="/" host=miss-islington.herokuapp.com
request_id=8de611a2-6112-4039-a425-97850ac989b0 fwd="140.82.115.15"
dyno=web.1 connect=1ms service=1005ms status=200 bytes=168
protocol=https
May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py",
line 600, in send_packed_command
May 08 09:35:15 miss-islington app/web.1: self._sock.sendall(item)
May 08 09:35:15 miss-islington app/web.1: TimeoutError: [Errno 110]
Connection timed out
May 08 09:35:15 miss-islington app/web.1: During handling of the above
exception, another exception occurred:
May 08 09:35:15 miss-islington app/web.1: Traceback (most recent call last):
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/kombu/connection.py",
line 431, in _reraise_as_library_errors
May 08 09:35:15 miss-islington app/web.1: yield
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/celery/app/base.py",
line 744, in send_task
May 08 09:35:15 miss-islington app/web.1:
self.backend.on_task_call(P, task_id)
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
line 265, in on_task_call
May 08 09:35:15 miss-islington app/web.1:
self.result_consumer.consume_from(task_id)
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
line 126, in consume_from
May 08 09:35:15 miss-islington app/web.1: self._consume_from(task_id)
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/celery/backends/redis.py",
line 132, in _consume_from
May 08 09:35:15 miss-islington app/web.1: self._pubsub.subscribe(key)
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/redis/client.py",
line 3096, in subscribe
May 08 09:35:15 miss-islington app/web.1: ret_val =
self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/redis/client.py",
line 3009, in execute_command
May 08 09:35:15 miss-islington app/web.1:
self._execute(connection, connection.send_command, *args)
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/redis/client.py",
line 3013, in _execute
May 08 09:35:15 miss-islington app/web.1: return command(*args)
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py",
line 620, in send_command
May 08 09:35:15 miss-islington app/web.1:
self.send_packed_command(self.pack_command(*args))
May 08 09:35:15 miss-islington app/web.1: File
"/app/.heroku/python/lib/python3.6/site-packages/redis/connection.py",
line 613, in send_packed_command
May 08 09:35:15 miss-islington app/web.1: (errno, errmsg))
May 08 09:35:15 miss-islington app/web.1:
redis.exceptions.ConnectionError: Error 110 while writing to socket.
Connection timed out.
ᐧ
On Wed, May 8, 2019 at 9:44 AM Gregory P. Smith wrote:
> Yesterday it failed to produce a backport or tell me that it couldn't
> (after the "i'm now working on ..." message was left on the master PR). I
> waited a couple hours just in case and ran cherry_picker myself instead.
> Same thing today apparently on
> https://github.com/python/cpython/pull/13192 assuming the usual backport
> latency is no more than a minute or two.
>
> -gps
> ___
> python-committers mailing list
> python-committers@python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
python-committers mailing list
python-committers@python.org
https://mail.python.org/mailman/listinfo/python-committers
Code of Conduct: https://www.python.org/psf/codeofconduct/
[python-committers] miss-islington backport pipeline is stalled?
Yesterday it failed to produce a backport or tell me that it couldn't (after the "i'm now working on ..." message was left on the master PR). I waited a couple hours just in case and ran cherry_picker myself instead. Same thing today apparently on https://github.com/python/cpython/pull/13192 assuming the usual backport latency is no more than a minute or two. -gps ___ python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
