Found that this may be related: https://github.com/kennethreitz/requests/issues/2422
On Fri, Mar 24, 2017 at 10:36 AM Jorge Alpedrinha Ramos < [email protected]> wrote: > I just read the stacktrace with more attention and celery is not the > culprit, I thought that logs were served by celery but it's the serve_logs > app from airflow that is responsible for this. This sounds like some > configuration on the max buffer size for responses, but no being an expert > in flask this is just a wild guess. > > Nicholas is this the behavior you're seing? If this isn't the case you may > have a situation where your webserver container is not able to comunicate > with the worker. > > As promised here is the stacktrace: > > worker_1 | [2017-03-24 10:29:39,574] {_internal.py:87} INFO - > 172.18.0.8 - - [24/Mar/2017 10:29:39] "GET > /log/rates.ticker-to-analytics/parse-syslog/2017-01-12T07:45:00 HTTP/1.1" > 200 - > worker_1 | Traceback (most recent call last): > worker_1 | File "/usr/local/bin/airflow", line 28, in <module> > worker_1 | args.func(args) > worker_1 | File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 858, in > serve_logs > worker_1 | host='0.0.0.0', port=WORKER_LOG_SERVER_PORT) > worker_1 | File > "/usr/local/lib/python2.7/site-packages/flask/app.py", line 843, in run > worker_1 | run_simple(host, port, self, **options) > worker_1 | File > "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 736, in > run_simple > worker_1 | inner() > worker_1 | File > "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 699, in > inner > worker_1 | srv.serve_forever() > worker_1 | File > "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 536, in > serve_forever > worker_1 | HTTPServer.serve_forever(self) > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 233, in serve_forever > worker_1 | self._handle_request_noblock() > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 292, in _handle_request_noblock > worker_1 | self.handle_error(request, client_address) > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 290, in _handle_request_noblock > worker_1 | self.process_request(request, client_address) > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 318, in process_request > worker_1 | self.finish_request(request, client_address) > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 331, in finish_request > worker_1 | self.RequestHandlerClass(request, client_address, > self) > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 654, in __init__ > worker_1 | self.finish() > worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line > 713, in finish > worker_1 | self.wfile.close() > worker_1 | File "/usr/local/lib/python2.7/socket.py", line 283, > in close > worker_1 | self.flush() > worker_1 | File "/usr/local/lib/python2.7/socket.py", line 307, > in flush > worker_1 | > self._sock.sendall(view[write_offset:write_offset+buffer_size]) > worker_1 | socket.error: [Errno 32] Broken pipe > > On Fri, Mar 24, 2017 at 10:27 AM Jorge Alpedrinha Ramos < > [email protected]> wrote: > > Hi, > > I have run into this on a specific scenario where the log file is huge and > the worker throws some error when responding to the webserver. You'll > probably see some issue with writing to buffer if you check the worker > logs. I'll send a second email with a stacktrace of this, I believe this is > somehow related to celery, but I'll need to do some more digging. > > Another thing that could be a solution would be to make airflow webserver > to use the remote logs location for fetching the log instead of always > relying on the worker (which may be decommissioned, after a scaling > operation for handling a spike, and no longer available). > > On Fri, Mar 24, 2017 at 9:37 AM Gael Magnan <[email protected]> wrote: > > Hi, > > we have encountered the same problem on some machines but not all of them. > > One of our developer can't access log on his mac since the move to airflow > 1.8.0, but our production machine on Ubuntu doesn't have the problem. > For him it seems like the log file name the UI try to access is not the one > the worker created (IE different date format). > > We haven't found out the cause of the problem so if you find something I'm > interested. > > Regards > Gael > > Le ven. 24 mars 2017 à 01:26, Nicholas Hodgkinson < > [email protected]> a écrit : > > > So I'm running my scheduler and webserver in different Docker container > on > > the same host, everything seems to be working fine with the exception of > > access to logs from the UI. When doing so I get this: > > > > *** Log file isn't local. > > *** Fetching here: > > http://30751ef88b7a:8793/log/dag-id/task-id/2017-03-23T23:00:00 > > *** Failed to fetch log file from worker. > > > > *** Reading remote logs... > > *** Unsupported remote log location. > > > > However both containers have the same log directory mounted as a volume > > inside the container, which is specified correctly as a environment > > variable. Resources on this problem are scarce and I'm not sure how to > > solve it. Thoughts? > > > > -Nik > > [email protected] > > > > -- > > > > > > Read our founder's story. > > <https://collectivehealth.com/blog/started-collective-health/> > > > > *This message may contain confidential, proprietary, or protected > > information. If you are not the intended recipient, you may not review, > > copy, or distribute this message. If you received this message in error, > > please notify the sender by reply email and delete this message.* > > > >
