Re: AssertionError without traceback?
> On Jan 14, 2019, at 10:40 PM, dieter wrote: > > Israel Brewster writes: >> I have a flask application deployed on CentOS 7 using Python 3.6.7 and uwsgi >> 2.0.17.1, proxied behind nginx. uwsgi is configured to listed on a socket in >> /tmp. The app uses gevent and the flask_uwsgi_websockets plugin as well as >> various other third-party modules, all installed via pip in a virtualenv. >> The environment was set up using pip just a couple of days ago, so >> everything should be fully up-to-date. The application *appears* to be >> running properly (it is in moderate use and there have been no reports of >> issues, nor has my testing turned up any problems), however I keep getting >> entries like the following in the error log: >> >> AssertionError >> 2019-01-14T19:16:32Z failed with >> AssertionError > > I would try to find out where the log message above has been generated > and ensure it does not only log the information above but also the > associated traceback. > > I assume that the log comes from some framework -- maybe "uwsgi" > or "gevent". It is a weakness to log exceptions without the > associated traceback. After extensive debugging, it would appear the issue arrises due to a combination of my use of gevent.spawn to run a certain function, and the portion of that function that sends web socket messages. If I remove either the gevent.spawn and just call the function directly, or keep the gevent.spawn but don't try to send any messages via web sockets, the error goes away. With the combination, I *occasionally* get the message - most of the time it works. So I guess I just run everything synchronously for now, and as log as the performance isn't hurt noticeably, call it good. I still find it strange that this never happened on CentOS 6, but whatever. The gevent.spawn calls were probably pre-mature optimization anyway. > >> There is no additional information provided, just that. I was running the >> same app (checked out from a GIT repository, so exact same code) on CentOS 6 >> for years without issue, it was only since I moved to CentOS 7 that I've >> seen the errors. I have not so far been able to correlate this error with >> any specific request. Has anyone seen anything like this before such that >> you can give me some pointers to fixing this? As the application *appears* >> to be functioning normally, it may not be a big issue, but it has locked up >> once since the move (no errors in the log, just not responding on the >> socket), so I am a bit concerned. >> --- >> Israel Brewster >> Systems Analyst II >> 5245 Airport Industrial Rd >> Fairbanks, AK 99709 >> (907) 450-7293 >> --- >> >> [cid:05a3a602-0c27-4749-91b8-096a5857d984@flyravn.com] >> >> >> >> [cid:bbc82752-6db4-44cf-b919-421ed304e1d1@flyravn.com] > > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: AssertionError without traceback?
--- Israel Brewster Systems Analyst II 5245 Airport Industrial Rd Fairbanks, AK 99709 (907) 450-7293 --- [cid:bfa5c323-b100-481d-96b1-fc256ef2eb39@flyravn.com] [cid:8c891973-9e67-47b3-aa14-5f58b9b93607@flyravn.com] On Jan 14, 2019, at 10:40 PM, dieter mailto:die...@handshake.de>> wrote: Israel Brewster mailto:ibrews...@flyravn.com>> writes: I have a flask application deployed on CentOS 7 using Python 3.6.7 and uwsgi 2.0.17.1, proxied behind nginx. uwsgi is configured to listed on a socket in /tmp. The app uses gevent and the flask_uwsgi_websockets plugin as well as various other third-party modules, all installed via pip in a virtualenv. The environment was set up using pip just a couple of days ago, so everything should be fully up-to-date. The application *appears* to be running properly (it is in moderate use and there have been no reports of issues, nor has my testing turned up any problems), however I keep getting entries like the following in the error log: AssertionError 2019-01-14T19:16:32Z failed with AssertionError I would try to find out where the log message above has been generated and ensure it does not only log the information above but also the associated traceback. Any tips as to how? I tried putting in additional logging at a couple places where I called gevent.spawn() to see if that additional logging lined up with the assertions, but no luck. I guess I could just start peppering my code with logging commands, and hope something pops, but this seems quite...inelegant. I have not been able to reproduce the error, unfortunately. I assume that the log comes from some framework -- maybe "uwsgi" or "gevent". It is a weakness to log exceptions without the associated traceback. Fully agreed on both points. The reference to the callback for some reason puts me in mind of C code, but of course AssertionError is python, so maybe not. For what it's worth, the issue only seems to happen when the server is under relatively heavy load. During the night, when it is mostly idle, I don't get many (if any) errors. And this has only been happening since I upgraded to CentOS7 and the latest versions of all the frameworks. Hopefully it isn't a version incompatibility... There is no additional information provided, just that. I was running the same app (checked out from a GIT repository, so exact same code) on CentOS 6 for years without issue, it was only since I moved to CentOS 7 that I've seen the errors. I have not so far been able to correlate this error with any specific request. Has anyone seen anything like this before such that you can give me some pointers to fixing this? As the application *appears* to be functioning normally, it may not be a big issue, but it has locked up once since the move (no errors in the log, just not responding on the socket), so I am a bit concerned. --- Israel Brewster Systems Analyst II 5245 Airport Industrial Rd Fairbanks, AK 99709 (907) 450-7293 --- [cid:05a3a602-0c27-4749-91b8-096a5857d984@flyravn.com] [cid:bbc82752-6db4-44cf-b919-421ed304e1d1@flyravn.com] -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: AssertionError without traceback?
Israel Brewster writes: > I have a flask application deployed on CentOS 7 using Python 3.6.7 and uwsgi > 2.0.17.1, proxied behind nginx. uwsgi is configured to listed on a socket in > /tmp. The app uses gevent and the flask_uwsgi_websockets plugin as well as > various other third-party modules, all installed via pip in a virtualenv. The > environment was set up using pip just a couple of days ago, so everything > should be fully up-to-date. The application *appears* to be running properly > (it is in moderate use and there have been no reports of issues, nor has my > testing turned up any problems), however I keep getting entries like the > following in the error log: > > AssertionError > 2019-01-14T19:16:32Z failed with > AssertionError I would try to find out where the log message above has been generated and ensure it does not only log the information above but also the associated traceback. I assume that the log comes from some framework -- maybe "uwsgi" or "gevent". It is a weakness to log exceptions without the associated traceback. > There is no additional information provided, just that. I was running the > same app (checked out from a GIT repository, so exact same code) on CentOS 6 > for years without issue, it was only since I moved to CentOS 7 that I've seen > the errors. I have not so far been able to correlate this error with any > specific request. Has anyone seen anything like this before such that you can > give me some pointers to fixing this? As the application *appears* to be > functioning normally, it may not be a big issue, but it has locked up once > since the move (no errors in the log, just not responding on the socket), so > I am a bit concerned. > --- > Israel Brewster > Systems Analyst II > 5245 Airport Industrial Rd > Fairbanks, AK 99709 > (907) 450-7293 > --- > > [cid:05a3a602-0c27-4749-91b8-096a5857d984@flyravn.com] > > > > [cid:bbc82752-6db4-44cf-b919-421ed304e1d1@flyravn.com] -- https://mail.python.org/mailman/listinfo/python-list
