[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift (Ubuntu) Status: Confirmed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to swift in Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/python-swiftclient/+bug/959221/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift (Ubuntu) Status: Confirmed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/python-swiftclient/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Fix releases in Havana release of OpenStack. ** Changed in: python-swiftclient Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to swift in Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/python-swiftclient/+bug/959221/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Fix releases in Havana release of OpenStack. ** Changed in: python-swiftclient Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/python-swiftclient/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Project changed: swift = python-swiftclient -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/python-swiftclient/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Fixed in: https://github.com/openstack/python-swiftclient/commit/3196daf9929eef25d69d47592beef4cd31573b80 For some reason, Jenkins and Gerrit did not close the bug. ** Changed in: swift Status: Confirmed = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Patch added: Fix as committed https://bugs.launchpad.net/ubuntu/+source/swift/+bug/959221/+attachment/3675321/+files/python-swiftclient-1.4-lp959221-2.diff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift Assignee: Flavio Percoco (flaper87) = Pete Zaitcev (zaitcev) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
see also https://review.openstack.org/22569 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
I've filed a bug[0] for this so we can get some feedback from eventlet folks as well. https://bitbucket.org/eventlet/eventlet/issue/138/greenssl-consumes- over-100-of-cpu -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift Assignee: (unassigned) = Flavio Percoco Premoli (flaper87) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
--- ssl.py.fcs 2012-04-10 08:54:43.257885841 -0600 +++ ssl.py 2012-04-10 08:56:27.282585355 -0600 @@ -129,7 +129,10 @@ count = 0 while (count amount): v = self.send(data[count:]) - count += v + if v: + count += v + else: + time.sleep(0.01) return amount else: while True: I think you can just: count += v + time.sleep() And it will yield some CPU time. I think it will be impossible to do a parallel upload without green threads, e.g. swift --object-threads 10 or swift --segment-threads 10 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Just to clarify: my patch does not remove SSL at all. SSL is there and works as before. The only difference is that it's not trampolined into eventlet anymore. The root cause is the conflict between eventlet's green threads, which require faking of blocking sockets, and SSL, which requires looping (to dispatch opposing-read events and to send little chunks). There is nothing wrong with SSL or SSL performance, just the violation of assumptions about the way sockets behave. The whole problem is on the client. It has absolutely nothing with having SSL terminated on the proxy. In fact in my tests SSL was terminated by Pound. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Roman, please try this and let us know if it works for you. ** Patch added: patch 2, test workaround https://bugs.launchpad.net/swift/+bug/959221/+attachment/3041102/+files/swift-lp959221-2.diff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
This works, too: [root@lembas green]# pwd /usr/lib/python2.7/site-packages/eventlet/green [root@lembas green]# diff -u ssl.py.fcs ssl.py --- ssl.py.fcs 2012-04-10 08:54:43.257885841 -0600 +++ ssl.py 2012-04-10 08:56:27.282585355 -0600 @@ -129,7 +129,10 @@ count = 0 while (count amount): v = self.send(data[count:]) -count += v +if v: +count += v +else: +time.sleep(0.01) return amount else: while True: Still, I do not understand quite why this is a problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
The attachment patch 2, test workaround of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report. [This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Pete, yes, with plain httplib swift manages to upload the file without significant CPU usage, in my test it is around 7-10% which is still high compared to the curl version (around 1%) but I guess one can't get better performance from httplib. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 959221] Re: swift consumes over 100% of cpu during upload
FYI from the source for eventlet/green/ssl.py: Python nonblocking ssl objects don't give errors when the other endof the socket is closed (they do notice when the other end is shutdown,though). Any write/read operations will simply hang if the socket isclosed from the other end. There is no obvious fix for this problem;it appears to be a limitation of Python's ssl object implementation.A workaround is to set a reasonable timeout on the socket usingsettimeout(), and to close/reopen the connection when a timeout occurs at an unexpected juncture in the code. In my opinion, SSL should never be run in the python for any production system and would advocate removing SSL from swift all together to prevent future headaches. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Although I don't think that SSL should be completely removed from swift, I do agree with Andrew that it should never be used in production on the proxy server. Python's SSL handling also causes the ssl sockets to block and severely limits concurrent connections. SSL support in the proxy is useful for testing purposes. I would support adding warnings if the proxy is started with ssl. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Profile shows that eventlet/green/ssl.py is the culprit. Look at call numbers and cumulative time. ** Attachment added: profile 1 https://bugs.launchpad.net/swift/+bug/959221/+attachment/3035280/+files/lp959221.prof1.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
It looks like the problem may be looping. If I do the following slew of hand, CPU burn dissipates: --- /usr/lib/python2.7/site-packages/eventlet/green/ssl.py.fcs 2012-04-09 19:32:04.516730732 -0600 +++ /usr/lib/python2.7/site-packages/eventlet/green/ssl.py 2012-04-09 21:49:38.823537699 -0600 @@ -128,7 +128,7 @@ amount = len(data) count = 0 while (count amount): -v = self.send(data[count:]) +v = self.write(data[count:]) count += v return amount else: But I do not see a simple way to do anything about it. The client invokes the ssl.py indirectly, through httplib (there's also bufferedhttp, but it changes nothing in SSL case). The httplib hardcodes using of sendall() on the socket object, while only send() can be overrided easily by HTTPConnection. Since sendall just calls send repeatedly as seen above, one may be devious and try to turn send() into write(), but I cannot write my head around the possible implications of that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
This reproduces easily on my laptop, running Fedora 16, using git tip swift client (1.4.10). The problem definitely has something with SSL. I created a 2-node Swift+Keystone to be accessible through Pound, and configured Pound to allow both plain HTTP and SSL. Uploads with HTTP only use up a worrying amount of CPU - 25..30% but not everything. Uploads with HTTP+SSL peg at 105%. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift (Ubuntu) Status: New = Confirmed ** Changed in: swift (Ubuntu) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
In the screnshot - swift is actively uploading through https to hpcloud consuming 99% of CPU. ** Attachment added: Screenshot of the terminal with top while swift is uploading a file https://bugs.launchpad.net/swift/+bug/959221/+attachment/2909011/+files/swift-upload.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
I am having the same problem with 1.4.7 on precise -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Also affects: swift Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
