Which object store do you use? Some providers only guarantee eventual consistency which manifests as stale reads:
https://github.com/andrewgaul/are-we-consistent-yet On Thu, Mar 23, 2017 at 01:35:00AM +0530, Arunagiri Rajasekaran wrote: > I ran put blob operation of 1 KB and received 200 OK response but when I > queried > for that blob after some time, it gave me 404 not found error. It looks > like the IOPS-4 transfer was not at all successful at first hand even > though it reported 200 OK. > > This issue is not seen for every run but happens very infrequently at > times. > > What could be the problem here? > > jclouds debug logs are pasted below. > > *2017-03-14@07:41:54.372 D [validate-34 ] > pCommandExecutorService:logDebug - Sending request -341731354: PUT > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 > <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4> > HTTP/1.1* > 2017-03-14@07:41:54.374 D [validate-34 > ] wire:logDebug - >> > "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" > 2017-03-14@07:41:54.374 D [validate-34 ] > headers:logDebug - >> PUT > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 > HTTP/1.1 > 2017-03-14@07:41:54.374 D [validate-34 ] > headers:logDebug - >> Host: > mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com > 2017-03-14@07:41:54.374 D [validate-34 ] > headers:logDebug - >> x-amz-meta-ics.stack-description: > ICStore.1(BlobStoreConnection.1) > 2017-03-14@07:41:54.375 D [validate-34 ] > headers:logDebug - >> x-amz-meta-ics.meta-digest: > 8bd279c4f237299a539cef43a02f3cfc > 2017-03-14@07:41:54.375 D [validate-34 ] > headers:logDebug - >> x-amz-meta-ics.meta-version: 1 > 2017-03-14@07:41:54.375 D [validate-34 ] > headers:logDebug - >> Date: Tue, 14 Mar 2017 11:41:41 GMT > 2017-03-14@07:41:54.375 D [validate-34 ] > headers:logDebug - >> Authorization: AWS > AKIAJFXGJAH3NFZLJMWQ:DO1PzrgSpA5aK+5ogXIfocl+qkY= > 2017-03-14@07:41:54.375 D [validate-34 ] > headers:logDebug - >> Content-Type: application/unknown > 2017-03-14@07:41:54.375 D [validate-34 ] > headers:logDebug - >> Content-Length: 1024 > 2017-03-14@07:41:54.464 D [validate-34 ] > pCommandExecutorService:logDebug - Receiving response -2056154209: HTTP/1.1 > 307 Temporary Redirect > 2017-03-14@07:41:54.464 D [validate-34 ] > headers:logDebug - << HTTP/1.1 307 Temporary Redirect > 2017-03-14@07:41:54.464 D [validate-34 ] > headers:logDebug - << Transfer-Encoding: chunked > 2017-03-14@07:41:54.464 D [validate-34 ] > headers:logDebug - << Server: AmazonS3 > 2017-03-14@07:41:54.465 D [validate-34 ] > headers:logDebug - << Connection: close > 2017-03-14@07:41:54.465 D [validate-34 ] > headers:logDebug - << x-amz-request-id: 9A843CB77D2F5B90 > 2017-03-14@07:41:54.465 D [validate-34 ] > headers:logDebug - << x-amz-id-2: > JCga+UZjgKmtvTTEzOXSaeOqRO3Spf3K2tcxBstqXWIkVVDKUMXSRH4cUbAiY+3H > 2017-03-14@07:41:54.465 D [validate-34 ] > headers:logDebug - << Date: Tue, 14 Mar 2017 11:41:52 GMT > 2017-03-14@07:41:54.465 D [validate-34 ] > headers:logDebug - << x-amz-bucket-region: ap-northeast-1 > 2017-03-14@07:41:54.466 D [validate-34 ] > headers:logDebug - << Location: > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 > 2017-03-14@07:41:54.466 D [validate-34 ] > headers:logDebug - << Content-Type: application/xml > 2017-03-14@07:41:54.467 D [validate-34 > ] wire:logDebug - << "<?xml version="1.0" > encoding="UTF-8"?>[\n]" > 2017-03-14@07:41:54.468 D [validate-34 > ] wire:logDebug - << > "<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request > to the specified temporary endpoint. Continue to use the original request > endpoint for future > requests.</Message><Bucket>mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918</Bucket><Endpoint> > mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com > </Endpoint><RequestId>9A843CB77D2F5B90</RequestId><HostId>JCga+UZjgKmtvTTEzOXSaeOqRO3Spf3K2tcxBstqXWIkVVDKUMXSRH4cUbAiY+3H</HostId></Error>" > *2017-03-14@07:41:54.470 D [validate-34 ] > pCommandExecutorService:logDebug - Sending request -593084728: PUT > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 > <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4> > HTTP/1.1* > 2017-03-14@07:41:54.472 D [validate-34 > ] wire:logDebug - >> > "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" > 2017-03-14@07:41:54.473 D [validate-34 ] > headers:logDebug - >> PUT > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 > HTTP/1.1 > 2017-03-14@07:41:54.473 D [validate-34 ] > headers:logDebug - >> x-amz-meta-ics.stack-description: > ICStore.1(BlobStoreConnection.1) > 2017-03-14@07:41:54.473 D [validate-34 ] > headers:logDebug - >> x-amz-meta-ics.meta-digest: > 8bd279c4f237299a539cef43a02f3cfc > 2017-03-14@07:41:54.473 D [validate-34 ] > headers:logDebug - >> x-amz-meta-ics.meta-version: 1 > 2017-03-14@07:41:54.474 D [validate-34 ] > headers:logDebug - >> Host: > mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com > 2017-03-14@07:41:54.474 D [validate-34 ] > headers:logDebug - >> Date: Tue, 14 Mar 2017 11:41:41 GMT > 2017-03-14@07:41:54.474 D [validate-34 ] > headers:logDebug - >> Authorization: AWS > AKIAJFXGJAH3NFZLJMWQ:DO1PzrgSpA5aK+5ogXIfocl+qkY= > 2017-03-14@07:41:54.474 D [validate-34 ] > headers:logDebug - >> Content-Type: application/unknown > 2017-03-14@07:41:54.475 D [validate-34 ] > headers:logDebug - >> Content-Length: 1024 > 2017-03-14@07:41:54.710 D [validate-34 ] > pCommandExecutorService:logDebug - Receiving response 1571963586: HTTP/1.1 > 200 OK > *2017-03-14@07:41:54.710 D [validate-34 > ] headers:logDebug - << HTTP/1.1 200 OK* > 2017-03-14@07:41:54.711 D [validate-34 ] > headers:logDebug - << Server: AmazonS3 > 2*017-03-14@07:41:54.711 D [validate-34 > ] headers:logDebug - << ETag: > "c9a34cfc85d982698c6ac89f76071abd"* > 2017-03-14@07:41:54.711 D [validate-34 ] > headers:logDebug - << x-amz-request-id: FEAF745B7A6ADCCA > 2017-03-14@07:41:54.711 D [validate-34 ] > headers:logDebug - << x-amz-id-2: > PoXZwaaoyJdgN4IcWBPFindUVfFBUl51zbQVQ2iL8YHDosyVJksqE3JotpiwYtMmUQDxjADzBVA= > 2017-03-14@07:41:54.711 D [validate-34 ] > headers:logDebug - << Date: Tue, 14 Mar 2017 11:41:54 GMT > 2017-03-14@07:41:54.712 D [validate-34 ] > headers:logDebug - << Content-Type: application/unknown > 2017-03-14@07:41:54.712 D [validate-34 ] > headers:logDebug - << Content-Length: 0 > 2017-03-14@07:47:44.733 D [validate-41 ] > pCommandExecutorService:logDebug - Sending request -225160499: GET > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 > HTTP/1.1 > 2017-03-14@07:47:44.733 D [validate-41 ] > headers:logDebug - >> GET > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 > HTTP/1.1 > 2017-03-14@07:47:44.733 D [validate-41 ] > headers:logDebug - >> Host: > mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com > 2017-03-14@07:47:44.733 D [validate-41 ] > headers:logDebug - >> Date: Tue, 14 Mar 2017 11:47:42 GMT > 2017-03-14@07:47:44.733 D [validate-41 ] > headers:logDebug - >> Authorization: AWS > AKIAJFXGJAH3NFZLJMWQ:iAI8UP3HnovXs3/4F6vqJWn/DQw= > 2017-03-14@07:47:44.785 D [validate-41 ] > pCommandExecutorService:logDebug - Receiving response -225160499: HTTP/1.1 > 307 Temporary Redirect > 2017-03-14@07:47:44.785 D [validate-41 ] > headers:logDebug - << HTTP/1.1 307 Temporary Redirect > 2017-03-14@07:47:44.785 D [validate-41 ] > headers:logDebug - << Transfer-Encoding: chunked > 2017-03-14@07:47:44.785 D [validate-41 ] > headers:logDebug - << Server: AmazonS3 > 2017-03-14@07:47:44.785 D [validate-41 ] > headers:logDebug - << x-amz-request-id: F76A3D3964A41BA2 > 2017-03-14@07:47:44.786 D [validate-41 ] > headers:logDebug - << x-amz-id-2: > 7FrHX+IorCrvzJ4JKNi7DzT6JorA/mYcDnxmgJKLAtoDtwxB888AiYyGFE2Y6/PjOameoA38YvI= > 2017-03-14@07:47:44.786 D [validate-41 ] > headers:logDebug - << Date: Tue, 14 Mar 2017 11:47:42 GMT > 2017-03-14@07:47:44.786 D [validate-41 ] > headers:logDebug - << x-amz-bucket-region: ap-northeast-1 > 2017-03-14@07:47:44.786 D [validate-41 ] > headers:logDebug - << Location: > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 > 2017-03-14@07:47:44.786 D [validate-41 ] > headers:logDebug - << Content-Type: application/xml > 2017-03-14@07:47:44.786 D [validate-41 > ] wire:logDebug - << "<?xml version="1.0" > encoding="UTF-8"?>[\n]" > 2017-03-14@07:47:44.786 D [validate-41 > ] wire:logDebug - << > "<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request > to the specified temporary endpoint. Continue to use the original request > endpoint for future > requests.</Message><Bucket>mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918</Bucket><Endpoint> > mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com > </Endpoint><RequestId>F76A3D3964A41BA2</RequestId><HostId>7FrHX+IorCrvzJ4JKNi7DzT6JorA/mYcDnxmgJKLAtoDtwxB888AiYyGFE2Y6/PjOameoA38YvI=</HostId></Error>" > *2017-03-14@07:47:44.787 D [validate-41 ] > pCommandExecutorService:logDebug - Sending request -476513873: GET > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 > <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4> > HTTP/1.1* > *2017-03-14@07:47:44.787 D [validate-41 > ] headers:logDebug - >> GET > https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 > <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4> > HTTP/1.1* > 2017-03-14@07:47:44.787 D [validate-41 ] > headers:logDebug - >> Host: > mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com > 2017-03-14@07:47:44.787 D [validate-41 ] > headers:logDebug - >> Date: Tue, 14 Mar 2017 11:47:42 GMT > 2017-03-14@07:47:44.787 D [validate-41 ] > headers:logDebug - >> Authorization: AWS > AKIAJFXGJAH3NFZLJMWQ:iAI8UP3HnovXs3/4F6vqJWn/DQw= > 2017-03-14@07:47:45.260 D [validate-41 ] > pCommandExecutorService:logDebug - Receiving response -476513873: HTTP/1.1 > 404 Not Found > 2017-03-14@07:47:45.260 D [validate-41 ] > headers:logDebug - << HTTP/1.1 404 Not Found > 2017-03-14@07:47:45.260 D [validate-41 ] > headers:logDebug - << Transfer-Encoding: chunked > 2017-03-14@07:47:45.260 D [validate-41 ] > headers:logDebug - << Server: AmazonS3 > 2017-03-14@07:47:45.261 D [validate-41 ] > headers:logDebug - << x-amz-request-id: 5A08FE7C9A9FDADA > 2017-03-14@07:47:45.261 D [validate-41 ] > headers:logDebug - << x-amz-id-2: > gTbzpFjMIhD/P0QO6AcEAiqKo8WKHAs5olmHz5fegMCbfMnSCbGvnuCcmHmx8qrFF6fkdKRI38o= > 2017-03-14@07:47:45.261 D [validate-41 ] > headers:logDebug - << Date: Tue, 14 Mar 2017 11:47:42 GMT > 2017-03-14@07:47:45.261 D [validate-41 ] > headers:logDebug - << Content-Type: application/xml > 2017-03-14@07:47:45.261 D [validate-41 > ] wire:logDebug - << "<?xml version="1.0" > encoding="UTF-8"?>[\n]" > > *2017-03-14@07:47:45.261 D [validate-41 > ] wire:logDebug - << > "<Error><Code>NoSuchKey</Code><Message>The specified key does not > exist.</Message><Key>IOPS-4</Key><RequestId>5A08FE7C9A9FDADA</RequestId><HostId>gTbzpFjMIhD/P0QO6AcEAiqKo8WKHAs5olmHz5fegMCbfMnSCbGvnuCcmHmx8qrFF6fkdKRI38o=</HostId></Error>"* > Thanks, > Arunagiri -- Andrew Gaul http://gaul.org/
