Hi Ignasi https://github.com/sumitkgaur/test
1) Example8.java is original programme and required all jclouds libs. 2) Worker.java is the delayed delete programme. Thanks sumit On Apr 25, 2014 3:45 PM, "Ignasi Barrera" <[email protected]> wrote: > Hi Sumit, > > Could you share the entire code of both programs in a git or pastie so we > can understand better how your benchmark works, and reproduce it locally? > El 25/04/2014 02:39, "Sumit Gaur" <[email protected]> escribió: > > > Hi Shri, > > After all experiments I feel there is a problem in Jclouds. > > > > 1) I tried retires for every 409 error. After successful retry Jclouds > > started reporting that blob is no more exists but in real it is still > there > > in SWIFT storage. > > 2) I try delaying the delete after 100 puts and voila there are no 409 in > > 24 hours. That exactly says there are some race situation in jclouds if > we > > do immediate Delete after PUT. > > 3) I know 409 Errors are coming all the way from SWIFT object server but > > same is not happening even if I generate much higher "concurrent" load > from > > curl (PUT- GET-DEL) cycle. I was getting TPS of 150. > > 4) I have run SWIFT without any extra daemon like auditor and others to > > avoid conflicts because of them. Storage node run only > > object/container/account server. > > 5) To generate concurrent curl load I am sending curl commands in the > > background. I ran this test for 48 hours and not even a single 409 error. > > 6) For an idea of sequence of client code > > > > static BlobStoreContext getSwiftClientView() { > > return ContextBuilder.newBuilder("swift-keystone") > > .credentials("test:tester", "test123") > > .endpoint("http://a.x.y.z.:5000/v2.0/";) > > .buildView(BlobStoreContext.class); > > } > > > > BlobStoreContext context = getSwiftClientView(); > > blobStore = context.getBlobStore(); > > blobStore.createContainerInLocation(null, containerName); > > blobStore.blobBuilder(key).payload(file).build(); > > > > blobStore.putBlob(containerName, blob); > > getBlob(containerName, key); > > blobStore.removeBlob(containerName, key); > > > > Let me know if you still see any gaps. > > Thanks > > sumit > > > > > > > > > > > > On Apr 23, 2014 2:36 AM, "Shrinand Javadekar" <[email protected]> > > wrote: > > > > > So there are two problems: > > > > > > 1) 409 when deleting objects. > > > 2) Transactions taking longer after 24-48 hours. > > > > > > For (1), it looks like the request reached the Swift cluster but the > > > Swift cluster itself wasn't able to fulfill it. This could be because > > > of the "eventual consistency" semantics of blobstores. When the delete > > > request reached Swift, it could have been in the middle of some > > > operation on the object itself (e.g. reading the object for > > > replicating it, auditing it, etc). Jclouds did it's job of actually > > > sending the request. So not sure what else can be done here. Maybe we > > > could add retries if the blobstore returns 409. But the main problem > > > lies on the Swift side. The Openstack mailing list would be a better > > > place for asking this question. There are many more Swift experts > > > there. > > > > > > For (2), from the curl example code, it looks like you're creating > > > multiple processes, each doing a put or a delete (no get). This is > > > different from jclouds spawning multiple threads. It would be great if > > > the experiments count the number of transactions they're doing and > > > whether they both reach the same number of transactions in the given > > > amount of time. If they do and yet there are less txns via jclouds > > > compared to the shell script, we can conclude that jclouds is the > > > cause of the problem. > > > > > > Now, answering some of the questions below. > > > > > > > It would be great if someone let me know how jcloud delete works. Is > > > there > > > > any internal queue while put or delete ? I saw if I put a small sleep > > of > > > > 300ms between put n del call, it works fine. > > > > > > I presume the blobstore object you're using in Example9.blobStore is > > > of type "BlobStore" and not "AsyncBlobStore". AsyncBlobStore is > > > deprecated. The BlobStore object is synchronous. There is no queue. > > > When you call removeBlob, the request gets created and sent to the > > > Swift cluster. > > > > > > > Also I assume that jclouds calls are synchronous one n put could not > > come > > > > out till object get saved in swift. > > > > > > For the BlobStore type, yes, it is sync. > > > > > > There are some jvm level settings that might also be at play here > > > related to the amount of memory you're allocating to the heap. You > > > could change the memory given to the jvm using the -Xms and -Xmx > > > options. > > > > > > -Shri > > > > > > > On Apr 22, 2014 11:59 AM, "Sumit Gaur" <[email protected]> > wrote: > > > > > > > >> Hi > > > >> Please find my answer below > > > >> > > > >> On Apr 22, 2014 10:49 AM, "Jasdeep Hundal" < > > > [email protected]> > > > >> wrote: > > > >> > > > > >> > Hey Sumit, > > > >> > > > > >> > I have a couple more questions that might help clarify the > > situation: > > > >> > > > > >> > 1. Are you running the stability test as a single long running > Java > > > >> process > > > >> > (that just keeps cycling through the 10 uploads/gets/deletes)? > > > >> > > > > >> > > > >> Yes. But this process has threads. > > > >> > > > >> > 2. Are you always running the test in the same container, or are > you > > > >> > creating new containers for each test iteration? > > > >> > > > > >> No, I am doing roundrobin in 1000 containers > > > >> > > > >> > 3. If the answer to #2 is is that the test runs in a single > > container, > > > >> how > > > >> > many objects does that container currently have? > > > >> > > > > >> > > > >> 0 in ideal case. But as I m facing 409 delete fail also... so there > > are > > > >> some objects on each container in hundreds only. > > > >> > > > >> > It may also help to time each of the individual blobstore actions > as > > > you > > > >> > run the test to see if any particular one is slowing down. > > > >> > > > > >> > > > >> Even indivitual put and del time increase over the time. > > > >> > > > >> > Jasdeep > > > >> > > > > >> > > > > >> > On Mon, Apr 21, 2014 at 6:21 PM, Sumit Gaur <[email protected] > > > > > >> wrote: > > > >> > > > > >> > > hi Shri, > > > >> > > Please find answers below > > > >> > > > > > >> > > On Tue, Apr 22, 2014 at 9:23 AM, Shrinand Javadekar < > > > >> > > [email protected] > > > >> > > > wrote: > > > >> > > Few more questions to try and understand this better: > > > >> > > > > > >> > > 1) On the Swift instance you are using, how many replicas do you > > > have? > > > >> > > > > > >> > > 3 replica > > > >> > > > > > >> > > 2) Also, how are you using the curl command in the shell script? > > > >> > > > > > >> > > send below command in backgroud for 10 iterations and wait > > similiar > > > to > > > >> the > > > >> > > 10 threads in jclouds. > > > >> > > > > > >> > > curl -X PUT -i -T 100k -H "X-Auth-Token: > > $OS_AUTH_TOKEN" > > > >> > > http:// > > > >> > > > > > >> > > > > > >> > > > > > > $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt > > > >> > > curl -X DELETE -i -H "X-Auth-Token: $OS_AUTH_TOKEN" > > > http:// > > > >> > > > > > >> > > > > > >> > > > > > > $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt > > > >> > > > > > >> > > I > > > >> > > think the shell script and jclouds-with-10-parallel-threads may > > not > > > be > > > >> > > doing the same amount of work. In 20 hours jclouds might be > doing > > > much > > > >> > > more work than the shell script. If you let the shell script > also > > go > > > >> > > upto that point, it might see failures too. Do you know how many > > > >> > > PUT-GET-DEL operations have been performed when you start seeing > > the > > > >> > > 409 errors. > > > >> > > > > > >> > > Actually 409 errors are coming since the start of the test but > TPS > > > >> start > > > >> > > degrading after 24-48 hours. > > > >> > > On Apr 22, 2014 9:23 AM, "Shrinand Javadekar" < > > > [email protected] > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > > > Few more questions to try and understand this better: > > > >> > > > > > > >> > > > 1) On the Swift instance you are using, how many replicas do > you > > > >> have? > > > >> > > > 2) Also, how are you using the curl command in the shell > > script? I > > > >> > > > think the shell script and jclouds-with-10-parallel-threads > may > > > not > > > >> be > > > >> > > > doing the same amount of work. In 20 hours jclouds might be > > doing > > > >> much > > > >> > > > more work than the shell script. If you let the shell script > > also > > > go > > > >> > > > upto that point, it might see failures too. Do you know how > many > > > >> > > > PUT-GET-DEL operations have been performed when you start > seeing > > > the > > > >> > > > 409 errors. > > > >> > > > > > > >> > > > -Shri > > > >> > > > > > > >> > > > > > > >> > > > On Mon, Apr 21, 2014 at 4:55 PM, Sumit Gaur < > > [email protected] > > > > > > > >> > > wrote: > > > >> > > > > FYI ..This is block of code ..... also I am using jclouds > > > 1.7.1 > > > >> > > (Stable > > > >> > > > > branch) > > > >> > > > > try { > > > >> > > > > String key = "objkey" + UUID.randomUUID(); > > > >> > > > > Blob blob = > > > >> > > > > > > > Example9.blobStore.blobBuilder(key).payload(Example9.file).build(); > > > >> > > > > > > > >> > > Example9.blobStore.putBlob(Example9.containerName+count, > > > >> > > > > blob); > > > >> > > > > > > > >> > > Example9.blobStore.getBlob(Example9.containerName+count, > > > >> > > > > key); > > > >> > > > > > > > >> > > > Example9.blobStore.removeBlob(Example9.containerName+count, > > > >> > > > > key); > > > >> > > > > } catch (Exception ace) { > > > >> > > > > System.out.println("Request failed for > objkey > > " > > > + > > > >> key > > > >> > > + " > > > >> > > > > " + ace); > > > >> > > > > } > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > On Tue, Apr 22, 2014 at 8:32 AM, Sumit Gaur < > > > [email protected]> > > > >> > > > wrote: > > > >> > > > > > > > >> > > > >> Hi Shri, > > > >> > > > >> Thanks for paying attention to it, Please find my answers > > > below:- > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On Tue, Apr 22, 2014 at 2:31 AM, Shrinand Javadekar < > > > >> > > > >> [email protected]> wrote: > > > >> > > > >> > > > >> > > > >>> Sumit, > > > >> > > > >>> > > > >> > > > >>> I realize that you had sent out a similar email sometime > ago > > > >> about > > > >> > > > >>> performance degradation. I'm not sure if anyone has run > > these > > > >> types > > > >> > > of > > > >> > > > >>> long running experiments with jclouds. So this may be a > > first. > > > >> > > > >>> > > > >> > > > >> Tried to debug it in last 2 weeks without success. Want to > > > >> understand > > > >> > > > more > > > >> > > > >> how jclouds code handle this use case or any pointers that > > this > > > >> is a > > > >> > > > >> problematic use case would help > > > >> > > > >> > > > >> > > > >>> > > > >> > > > >>> The 409 status is returned because of a conflict [1]. Are > > you > > > >> sure > > > >> > > you > > > >> > > > >>> didn't have two or more threads trying to delete the same > > > object? > > > >> > > > >>> > > > >> > > > >> No two threads share the same object key in my programme > > > (String > > > >> key = > > > >> > > > >> "objkey" + UUID.randomUUID();). It is some kind of race > > between > > > >> PUT > > > >> > > and > > > >> > > > >> DEL call . If I put say 10 ms sleep between call then there > > is > > > no > > > >> 409 > > > >> > > > error. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >>> Also, I see that that 409 is returned by Swift if you try > to > > > >> delete a > > > >> > > > >>> container that isn't empty[2]. Is that something your test > > > code > > > >> > > > >>> could've tried? > > > >> > > > >>> > > > >> > > > >> I am trying to delete objects .. not containers. > > > >> > > > >> > > > >> > > > >>> > > > >> > > > >>> When you say there was a similar test you're trying with > > curl, > > > >> are > > > >> > > you > > > >> > > > >>> using the curl command-line utility or the libcurl > library? > > > >> > > > >> > > > >> > > > >> curl command in shell script with for loops. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >>> How are > > > >> > > > >>> you specifying the number of threads to use and what > object > > > each > > > >> > > > >>> thread should get/put/delete? > > > >> > > > >>> > > > >> > > > >> > > > >> > > > >> It is a java test programme using ThreadPoolExecutor. > > Somthing > > > >> > > similiar > > > >> > > > as > > > >> > > > >> here > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > >> > > > > > >> > > > > > > http://www.javacodegeeks.com/2013/01/java-thread-pool-example-using-executors-and-threadpoolexecutor.html > > > >> > > > >> > > > >> > > > >> Object is a 5KB file. with key = "objkey" + > > UUID.randomUUID(); > > > >> with > > > >> > > > Pool > > > >> > > > >> of 10 threads. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Hope this would give a good inside. Let me know if you get > > any > > > >> problem > > > >> > > > >> here. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >>> > > > >> > > > >>> Thanks. > > > >> > > > >>> -Shri > > > >> > > > >>> > > > >> > > > >>> [1] > http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html > > > >> > > > >>> [2] https://bugs.launchpad.net/horizon/+bug/1096084 > > > >> > > > >>> > > > >> > > > >>> On Sun, Apr 20, 2014 at 5:55 PM, Sumit Gaur < > > > >> [email protected]> > > > >> > > > wrote: > > > >> > > > >>> > Hi > > > >> > > > >>> > I using jclouds lib integrated with Openstack Swift+ > > > keystone > > > >> > > > >>> combinaiton. > > > >> > > > >>> > Things are working fine except stability test. After > 20-30 > > > >> hours of > > > >> > > > test > > > >> > > > >>> > jclouds/SWIFT start degrading in TPS and keep going down > > > over > > > >> the > > > >> > > > time. > > > >> > > > >>> > > > > >> > > > >>> > 1) I am running the (PUT-GET-DEL) cycle in 10 parallel > > > threads. > > > >> > > > >>> > 2) I am getting a lot of 409 and DEL failure for the as > > > >> response > > > >> > > too > > > >> > > > >>> from > > > >> > > > >>> > SWIFT. > > > >> > > > >>> > 3) Direct similiar test from curl does not show much > > impact > > > >> and TPS > > > >> > > > >>> remain > > > >> > > > >>> > constant. > > > >> > > > >>> > > > > >> > > > >>> > Can sombody help me wht is going wrong here ? > > > >> > > > >>> > > > > >> > > > >>> > Thanks > > > >> > > > >>> > sumit > > > >> > > > >>> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > >> > > > > > >> > > > > > >
