Re: [Openstack] [Swift] 404 re-reading just created container
On 08/06/16 19:07, Mark Kirkwood wrote: changing this servers line to include the other proxy's memcache too (e.g proxy 192.168.122.21 again): $ cat /etc/swift/proxy-server.conf ... [filter:cache] use = egg:swift#memcache memcache_servers = 192.168.122.21:11211,192.168.122.24:11211 I no longer see any 404s with my test. Reading the install docs (a bit more carefully this time it seems), I see that they are leaning us towards doing that - am I understanding that correctly? A completely different approach is to remove the 'cache' entry from the proxy pipeline. Obviously this is not ideal from a performance/scalability point of view, but we currently have a very small object storage archive so - for now - performance is not that critical. It will be later for sure, but that gives us time to plan for (centralized etc) memcache resources that Swift can be easily amended to use. Thoughts - is it crazy to switch off caching? regards Mark ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] 404 re-reading just created container
On 08/06/16 10:50, Mark Kirkwood wrote: On 08/06/16 07:06, Pete Zaitcev wrote: The proxies are load balanced behind Haproxy (which I'm guessing is causing the 404 - see below) HAproxy is often troublesome, but I don't expect it is at fault in this instance. The logs that you quoted show that the proxy-server returned the 404. Do proxies happen to talk to separate memcached instances by any chance? Yeah - good point, each proxy has its own memcache. Following up this point, I can reproduce this issue by putting haproxy in front of some development proxies I have running as VMs on my workstation (easier to amend config in this case): So I have 2 Swift proxies, each using itself as a memcache server (e.g proxy 192.168.122.21), other one (192.168.122.24) defined similarly: $ cat /etc/swift/proxy-server.conf ... [filter:cache] use = egg:swift#memcache memcache_servers = 192.168.122.21:11211 changing this servers line to include the other proxy's memcache too (e.g proxy 192.168.122.21 again): $ cat /etc/swift/proxy-server.conf ... [filter:cache] use = egg:swift#memcache memcache_servers = 192.168.122.21:11211,192.168.122.24:11211 I no longer see any 404s with my test. Reading the install docs (a bit more carefully this time it seems), I see that they are leaning us towards doing that - am I understanding that correctly? regards Mark ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] 404 re-reading just created container
On 08/06/16 07:06, Pete Zaitcev wrote: On Fri, 3 Jun 2016 16:09:06 +1200 Mark Kirkwood wrote: This is a Swift 2.7.0.1 system with 2 proxies and 3 storage nodes (each the latter with 6 devices). What is your replica count? 3 replicas for objects and containers - sorry forgot mention! The proxies are load balanced behind Haproxy (which I'm guessing is causing the 404 - see below) HAproxy is often troublesome, but I don't expect it is at fault in this instance. The logs that you quoted show that the proxy-server returned the 404. Do proxies happen to talk to separate memcached instances by any chance? Yeah - good point, each proxy has its own memcache. What you really need in this instance is to capture logs that have the same tx-id on _all_ nodes. Then we can construct the scenario. Thx, will do a bit more digging. regards Mark ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] 404 re-reading just created container
On Fri, 3 Jun 2016 16:09:06 +1200 Mark Kirkwood wrote: > This is a Swift 2.7.0.1 system with 2 proxies and 3 storage nodes (each > the latter with 6 devices). What is your replica count? > The proxies are load balanced behind Haproxy (which I'm guessing is > causing the 404 - see below) HAproxy is often troublesome, but I don't expect it is at fault in this instance. The logs that you quoted show that the proxy-server returned the 404. Do proxies happen to talk to separate memcached instances by any chance? What you really need in this instance is to capture logs that have the same tx-id on _all_ nodes. Then we can construct the scenario. -- Pete ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] 404 re-reading just created container
On 03/06/16 16:09, Mark Kirkwood wrote: I'll dig up the Haproxy config and post. However any thoughts in the meantime? We have: defaults log global maxconn 8000 option redispatch retries 3 stats enable timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout check 10s ... listen swift-pxy00 bind :::8843 ssl crt /var/lib/puppet/ssl/certs/host.crt.pem ca-file /var/lib/puppet/ssl/certs/ca.pem no-sslv3 balance roundrobin mode http option httpchk /info server tst-wgtn-opxy1 10.16.0.39:8843 check ca-file /var/lib/puppet/ssl/certs/ca.pem crt /var/lib/puppet/ssl/certs/host.crt.pem ssl server tst-wgtn-opxy2 10.16.0.49:8843 check ca-file /var/lib/puppet/ssl/certs/ca.pem crt /var/lib/puppet/ssl/certs/host.crt.pem ssl Which is missing a few options compared to https://swiftstack.com/blog/2013/06/10/configuring-haproxy-as-a-swiftstack-load-balancer/ . We will amend and retest. ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] [Swift] 404 re-reading just created container
Hi, I did previously mention that I would start a new thread if I could reproduce 'unexplained 404s'. Well, I can - however not for the exact case I mentioned in the previous thread. I have some python code using python-swiftclient that does: for a number of containers: create the container create a number of (e.g 1000) objects in it I'm seeing: container con3 ..object con3/obj0 Traceback (most recent call last): File "./swiftclient-creatread.py", line 72, in content_type='text/plain') File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1709, in put_object response_dict=response_dict) File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1281, in put_object http_response_content=body) swiftclient.exceptions.ClientException: Object PUT failed: https://api.ostst.wgtn.cat-it.co.nz:8843/v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3/obj0 404 Not Found [first 60 chars of response] Not FoundThe resource could not be found.< However the container has in fact been created ok - rerunning the program works fine (for the container where it failed prev, subsequent iterations *may* fail adding objects to just created containers). This is a Swift 2.7.0.1 system with 2 proxies and 3 storage nodes (each the latter with 6 devices). The proxies are load balanced behind Haproxy (which I'm guessing is causing the 404 - see below) The variables recheck_account_existence, recheck_container_existence, request_node_count are all default. Looking at the proxy logs, it seems that the request to create the container went to proxy number 1, and the request to create the 1st object in the container went to proxy number 2: Jun 3 15:06:04 tst-wgtn-opxy1 proxy-server: 10.16.0.41 127.0.0.1 03/Jun/2016/03/06/04 PUT /v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3 HTTP/1.0 201 - python-swiftclient-3.0.0 af45fc08a94d4097... - - - tx6aaf7d3208724079b1b92-005750f41c - 0.0444 - - 1464923164.878182888 1464923164.922573090 0 Jun 3 15:06:04 tst-wgtn-opxy2 proxy-server: 10.16.0.41 127.0.0.1 03/Jun/2016/03/06/04 PUT /v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3/obj0 HTTP/1.0 404 - python-swiftclient-3.0.0 af45fc08a94d4097... - 70 - tx3ed4012d74e94eafa6ca1-005750f41c - 0.0007 - - 1464923164.943866968 1464923164.944570065 0 I'll dig up the Haproxy config and post. However any thoughts in the meantime? regards Mark ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
