Re: [openstack-dev] [SWIFT] Delete operation problem
*FYI more concrete logs * *Client Node* Request failed for objkey objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 org.jclouds.http.HttpResponseException: command: DELETE http://www.example.com:80/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615HTTP/1.1 failed with response: HTTP/1.1 409 Conflict; content: [ConflictThere was a conflict when trying to complete your request.] *Proxy Node * Apr 23 12:00:27 proxy1 swift-proxy allow user with role admin as account admin (txn: txe45b336b0ab44aee8a698-0053572ccb) (client_ip: 10.41.54.27) *Storage Node* Apr 23 12:00:26 object1* container-server* 10.41.54.9 - - [23/Apr/2014:03:00:26 +] "PUT /vdb1/955/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615" 201 - "tx3ba78bc8924349d18f7af-0053572ccc" "PUT http://www.example.com/vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615"; "obj-server 17908" 0.0006 Apr 23 12:00:26 object1 *object-server* 10.41.54.21 - - [23/Apr/2014:03:00:26 +] "PUT /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615" 201 - "*PUT * http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615"; "tx3ba78bc8924349d18f7af-0053572ccc" "proxy-server 23927" 0.0113 Apr 23 12:00:26 object1 *object-server *10.41.54.19 - - [23/Apr/2014:03:00:26 +] "DELETE /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615" 409 - "*DELETE * http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615"; "txe45b336b0ab44aee8a698-0053572ccb" "proxy-server 10071" 0.0003 On Wed, Apr 23, 2014 at 5:09 AM, Clay Gerrard wrote: > 409 on DELETE (object?) is a pretty specific error. That should mean that > the timestamp assigned to the delete is earlier than the timestamp of the > data file. > > Most likely mean that you're getting some time-drift on your proxies (but > that assumes multi-node) or maybe that you're reusing names between threads > and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be > a pretty tight race... > > Should all be logged - try and find a DELETE that went 409 and trace the > transaction id. > > > On Mon, Apr 21, 2014 at 5:54 AM, taurus huang wrote: > >> Please provide the log file： /var/log/swift/swift.log AND >> /var/log/keystone/keystone.log >> >> >> On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur wrote: >> >>> Hi >>> I using jclouds lib integrated with Openstack Swift+ keystone >>> combination. Things are working fine except stability test. After 20-30 >>> hours of test jclouds/SWIFT start degrading in TPS and keep going down over >>> the time. >>> >>> 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. >>> 2) I am getting a lot of 409 and DEL failure for the response too from >>> SWIFT. >>> >>> >>> Can sombody help me what is going wrong here ? >>> >>> Thanks >>> sumit >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
Further to above details please find related logs ... *object server* 4:07:21:20 +] "DELETE /vdb1/844/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b" 409 - "DELETE http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b"; "txcb3b4b47278e483ca560e-005350d271" "proxy-server 9619" 0.0004 or [22/Apr/2014:08:30:07 +] "DELETE /vdb1/238/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14" 409 - "DELETE http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14"; "tx0bc79d62fec445d28dfab-005356288f" "proxy-server 2142" 0.0005 On Wed, Apr 23, 2014 at 8:20 AM, Sumit Gaur wrote: > Hi clay, > Thanks for responding , first about setup , it is multi node env, and > problem is with delete object . > > I checked proxy and storage nodes clock, they are in sync. Also storage > node run only main daemons I.e. no auditor, and others. > > I traced the 409 call and it is coming from object server. > I am generating random id for key so no reuse of same name. But after one > put(t1) next is del(t1).is this could be a problem...I think they are > sync calls. > > Regards > Sumit > > On Apr 23, 2014 5:14 AM, "Clay Gerrard" wrote: > > > > 409 on DELETE (object?) is a pretty specific error. That should mean > that the timestamp assigned to the delete is earlier than the timestamp of > the data file. > > > > Most likely mean that you're getting some time-drift on your proxies > (but that assumes multi-node) or maybe that you're reusing names between > threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but > that'd be a pretty tight race... > > > > Should all be logged - try and find a DELETE that went 409 and trace the > transaction id. > > > > > > On Mon, Apr 21, 2014 at 5:54 AM, taurus huang > wrote: > >> > >> Please provide the log file： /var/log/swift/swift.log AND > /var/log/keystone/keystone.log > >> > >> > >> On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur > wrote: > >>> > >>> Hi > >>> I using jclouds lib integrated with Openstack Swift+ keystone > combination. Things are working fine except stability test. After 20-30 > hours of test jclouds/SWIFT start degrading in TPS and keep going down over > the time. > >>> > >>> 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. > >>> 2) I am getting a lot of 409 and DEL failure for the response too from > SWIFT. > >>> > >>> > >>> Can sombody help me what is going wrong here ? > >>> > >>> Thanks > >>> sumit > >>> > >>> ___ > >>> OpenStack-dev mailing list > >>> OpenStack-dev@lists.openstack.org > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>> > >> > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
Hi clay, Thanks for responding , first about setup , it is multi node env, and problem is with delete object . I checked proxy and storage nodes clock, they are in sync. Also storage node run only main daemons I.e. no auditor, and others. I traced the 409 call and it is coming from object server. I am generating random id for key so no reuse of same name. But after one put(t1) next is del(t1).is this could be a problem...I think they are sync calls. Regards Sumit On Apr 23, 2014 5:14 AM, "Clay Gerrard" wrote: > > 409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file. > > Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race... > > Should all be logged - try and find a DELETE that went 409 and trace the transaction id. > > > On Mon, Apr 21, 2014 at 5:54 AM, taurus huang wrote: >> >> Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log >> >> >> On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur wrote: >>> >>> Hi >>> I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. >>> >>> 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. >>> 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT. >>> >>> >>> Can sombody help me what is going wrong here ? >>> >>> Thanks >>> sumit >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file. Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race... Should all be logged - try and find a DELETE that went 409 and trace the transaction id. On Mon, Apr 21, 2014 at 5:54 AM, taurus huang wrote: > Please provide the log file： /var/log/swift/swift.log AND > /var/log/keystone/keystone.log > > > On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur wrote: > >> Hi >> I using jclouds lib integrated with Openstack Swift+ keystone >> combination. Things are working fine except stability test. After 20-30 >> hours of test jclouds/SWIFT start degrading in TPS and keep going down over >> the time. >> >> 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. >> 2) I am getting a lot of 409 and DEL failure for the response too from >> SWIFT. >> >> >> Can sombody help me what is going wrong here ? >> >> Thanks >> sumit >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur wrote: > Hi > I using jclouds lib integrated with Openstack Swift+ keystone combination. > Things are working fine except stability test. After 20-30 hours of test > jclouds/SWIFT start degrading in TPS and keep going down over the time. > > 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. > 2) I am getting a lot of 409 and DEL failure for the response too from > SWIFT. > > > Can sombody help me what is going wrong here ? > > Thanks > sumit > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [SWIFT] Delete operation problem
Hi I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT. Can sombody help me what is going wrong here ? Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
