Slawek, did you change Swift proxy-server.conf file too?
On Wed, Oct 1, 2014 at 11:26 PM, Sławek Kapłoński <sla...@kaplonski.pl> wrote: > Hello, > > Thanks for Your help but it not helps. I checked that for sure on each > swift > node there is a lot of free space. What can confirm that is fact that when > I > try to create image with size about 1.7GB and I have > swift_store_large_object_size set to 1GB than there is error (always after > send first chunk to swift (200MB). When I only change > swift_store_large_object_size to 2GB and restart glance-api than the same > image is created correctly (it is then in one big object). > > --- > Best regards > Sławek Kapłoński > sla...@kaplonski.pl > > Dnia wtorek, 30 września 2014 22:28:11 Timur Nurlygayanov pisze: > > Hi Slawek, > > > > we faced the same error and this is issue with Swift. > > We can see 100% disk usage on the Swift node during the file upload and > > looks like Swift can't send info about status of the file loading in > time. > > > > On our environments we found the workaround for this issue: > > 1. Set swift_store_large_object_size = 200 in glance.conf. > > 2. Add to Swift proxy-server.conf: > > > > [DEFAULT] > > ... > > node_timeout = 90 > > > > Probably we can set this value as default value for this parameter > instead > > of '30'? > > > > > > Regards, > > Timur > > > > > > On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński <sla...@kaplonski.pl> > > > > wrote: > > > Hello, > > > > > > I can't find that upload from was previous logs but I now try to upload > > > same image once again. In glance there was exactly same error. In swift > > > logs I have: > > > > > > Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y > > > 30/Sep/2014/15/35/10 HEAD > /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance > > > HTTP/1.0 204 > > > Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y > > > 30/Sep/2014/15/35/16 PUT /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dcee > > > bd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-00001 HTTP/1.0 201 - - > > > > > > Best regards > > > Slawek Kaplonski > > > > > > W dniu 2014-09-30 17:03, Kuo Hugo napisał(a): > > >> Hi , > > >> > > >> Could you please post the log of related requests in Swift's log ??? > > >> > > >> Thanks // Hugo > > >> > > >> 2014-09-30 22:20 GMT+08:00 Sławek Kapłoński <sla...@kaplonski.pl>: > > >> Hello, > > >> > > >>> I'm using openstack havana release and glance with swift backend. > > >>> Today I found that I have problem when I create image with url in > > >>> "--copy-from" when image is bigger than my > > >>> "swift_store_large_object_size" because then glance is trying to > > >>> split image to chunks with size given in > > >>> "swift_store_large_object_chunk_size" and when try to upload first > > >>> chunk to swift I have error: > > >>> > > >>> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error > > >>> during chunked upload to backend, deleting stale chunks > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback > > >>> (most recent call last): > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File > > >>> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384, > > >>> in add > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift > > >>> > > >>> content_length=content_length) > > >>> > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File > > >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234, > > >>> in put_object > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift > > >>> > > >>> response_dict=response_dict) > > >>> > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File > > >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143, > > >>> in _retry > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift > > >>> > > >>> reset_func(func, *args, **kwargs) > > >>> > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File > > >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215, > > >>> in _default_reset > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift % > > >>> (container, obj)) > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift > > >>> ClientException: put_object('glance', > > >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no > > >>> ability to reset contents for reupload. > > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift > > >>> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed > > >>> to add object to Swift. > > >>> Got error from Swift: put_object('glance', > > >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no > > >>> ability to reset contents for reupload. > > >>> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-] > > >>> Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056 > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> Traceback (most recent call last): > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> > > >>> File > > >>> > > >>> "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", > > >>> line 101, in upload_data_to_store > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> > > >>> store) > > >>> > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> > > >>> File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", > > >>> > > >>> line 333, in store_add_to_backend > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> > > >>> (location, size, checksum, metadata) = store.add(image_id, data, > > >>> > > >>> size) > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> > > >>> File "/usr/lib/python2.7/dist-packages/glance/store/swift.py", > > >>> > > >>> line 447, in add > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> > > >>> raise glance.store.BackendException(msg) > > >>> > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils > > >>> BackendException: Failed to add object to Swift. > > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got > > >>> error from Swift: put_object('glance', > > >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no > > >>> ability to reset contents for reupload. > > >>> > > >>> Does someone of You got same error and know what is solution of it? > > >>> I was searching about that in google but I not found anything what > > >>> could solve my problem. > > >>> > > >>> -- > > >>> Best regards > > >>> Sławek Kapłoński > > >>> sla...@kaplonski.pl > > >>> > > >>> _______________________________________________ > > >>> Mailing list: > > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1] > > >>> Post to : openst...@lists.openstack.org > > >>> Unsubscribe : > > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1] > > >> > > >> Links: > > >> ------ > > >> [1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > > -- > > > Pozdrawiam > > > Sławek Kapłoński > > > sla...@kaplonski.pl > > > > > > _______________________________________________ > > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > > > openstack > > > Post to : openst...@lists.openstack.org > > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > > > openstack > > > _______________________________________________ > > > OpenStack-dev mailing list > > > OpenStack-dev@lists.openstack.org > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Timur, QA Engineer OpenStack Projects Mirantis Inc
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev