Sumit, >> > After all experiments I feel there is a problem in Jclouds.
There may be a problem with jclouds. I was trying to understand the test and the environment better so as to get to the bottom of this. On Fri, Apr 25, 2014 at 12:39 AM, Sumit Gaur <[email protected]> wrote: > Hi Ignasi > https://github.com/sumitkgaur/test > > 1) Example8.java is original programme and required all jclouds libs. > 2) Worker.java is the delayed delete programme. > > Thanks > sumit > > > > > > On Apr 25, 2014 3:45 PM, "Ignasi Barrera" <[email protected]> wrote: > >> Hi Sumit, >> >> Could you share the entire code of both programs in a git or pastie so we >> can understand better how your benchmark works, and reproduce it locally? >> El 25/04/2014 02:39, "Sumit Gaur" <[email protected]> escribió: >> >> > Hi Shri, >> > After all experiments I feel there is a problem in Jclouds. >> > >> > 1) I tried retires for every 409 error. After successful retry Jclouds >> > started reporting that blob is no more exists but in real it is still >> there >> > in SWIFT storage. >> > 2) I try delaying the delete after 100 puts and voila there are no 409 in >> > 24 hours. That exactly says there are some race situation in jclouds if >> we >> > do immediate Delete after PUT. >> > 3) I know 409 Errors are coming all the way from SWIFT object server but >> > same is not happening even if I generate much higher "concurrent" load >> from >> > curl (PUT- GET-DEL) cycle. I was getting TPS of 150. >> > 4) I have run SWIFT without any extra daemon like auditor and others to >> > avoid conflicts because of them. Storage node run only >> > object/container/account server. >> > 5) To generate concurrent curl load I am sending curl commands in the >> > background. I ran this test for 48 hours and not even a single 409 error. >> > 6) For an idea of sequence of client code >> > >> > static BlobStoreContext getSwiftClientView() { >> > return ContextBuilder.newBuilder("swift-keystone") >> > .credentials("test:tester", "test123") >> > .endpoint("http://a.x.y.z.:5000/v2.0/";) >> > .buildView(BlobStoreContext.class); >> > } >> > >> > BlobStoreContext context = getSwiftClientView(); >> > blobStore = context.getBlobStore(); >> > blobStore.createContainerInLocation(null, containerName); >> > blobStore.blobBuilder(key).payload(file).build(); >> > >> > blobStore.putBlob(containerName, blob); >> > getBlob(containerName, key); >> > blobStore.removeBlob(containerName, key); >> > >> > Let me know if you still see any gaps. >> > Thanks >> > sumit >> > >> > >> > >> > >> > >> > On Apr 23, 2014 2:36 AM, "Shrinand Javadekar" <[email protected]> >> > wrote: >> > >> > > So there are two problems: >> > > >> > > 1) 409 when deleting objects. >> > > 2) Transactions taking longer after 24-48 hours. >> > > >> > > For (1), it looks like the request reached the Swift cluster but the >> > > Swift cluster itself wasn't able to fulfill it. This could be because >> > > of the "eventual consistency" semantics of blobstores. When the delete >> > > request reached Swift, it could have been in the middle of some >> > > operation on the object itself (e.g. reading the object for >> > > replicating it, auditing it, etc). Jclouds did it's job of actually >> > > sending the request. So not sure what else can be done here. Maybe we >> > > could add retries if the blobstore returns 409. But the main problem >> > > lies on the Swift side. The Openstack mailing list would be a better >> > > place for asking this question. There are many more Swift experts >> > > there. >> > > >> > > For (2), from the curl example code, it looks like you're creating >> > > multiple processes, each doing a put or a delete (no get). This is >> > > different from jclouds spawning multiple threads. It would be great if >> > > the experiments count the number of transactions they're doing and >> > > whether they both reach the same number of transactions in the given >> > > amount of time. If they do and yet there are less txns via jclouds >> > > compared to the shell script, we can conclude that jclouds is the >> > > cause of the problem. >> > > >> > > Now, answering some of the questions below. >> > > >> > > > It would be great if someone let me know how jcloud delete works. Is >> > > there >> > > > any internal queue while put or delete ? I saw if I put a small sleep >> > of >> > > > 300ms between put n del call, it works fine. >> > > >> > > I presume the blobstore object you're using in Example9.blobStore is >> > > of type "BlobStore" and not "AsyncBlobStore". AsyncBlobStore is >> > > deprecated. The BlobStore object is synchronous. There is no queue. >> > > When you call removeBlob, the request gets created and sent to the >> > > Swift cluster. >> > > >> > > > Also I assume that jclouds calls are synchronous one n put could not >> > come >> > > > out till object get saved in swift. >> > > >> > > For the BlobStore type, yes, it is sync. >> > > >> > > There are some jvm level settings that might also be at play here >> > > related to the amount of memory you're allocating to the heap. You >> > > could change the memory given to the jvm using the -Xms and -Xmx >> > > options. >> > > >> > > -Shri >> > > >> > > > On Apr 22, 2014 11:59 AM, "Sumit Gaur" <[email protected]> >> wrote: >> > > > >> > > >> Hi >> > > >> Please find my answer below >> > > >> >> > > >> On Apr 22, 2014 10:49 AM, "Jasdeep Hundal" < >> > > [email protected]> >> > > >> wrote: >> > > >> > >> > > >> > Hey Sumit, >> > > >> > >> > > >> > I have a couple more questions that might help clarify the >> > situation: >> > > >> > >> > > >> > 1. Are you running the stability test as a single long running >> Java >> > > >> process >> > > >> > (that just keeps cycling through the 10 uploads/gets/deletes)? >> > > >> > >> > > >> >> > > >> Yes. But this process has threads. >> > > >> >> > > >> > 2. Are you always running the test in the same container, or are >> you >> > > >> > creating new containers for each test iteration? >> > > >> > >> > > >> No, I am doing roundrobin in 1000 containers >> > > >> >> > > >> > 3. If the answer to #2 is is that the test runs in a single >> > container, >> > > >> how >> > > >> > many objects does that container currently have? >> > > >> > >> > > >> >> > > >> 0 in ideal case. But as I m facing 409 delete fail also... so there >> > are >> > > >> some objects on each container in hundreds only. >> > > >> >> > > >> > It may also help to time each of the individual blobstore actions >> as >> > > you >> > > >> > run the test to see if any particular one is slowing down. >> > > >> > >> > > >> >> > > >> Even indivitual put and del time increase over the time. >> > > >> >> > > >> > Jasdeep >> > > >> > >> > > >> > >> > > >> > On Mon, Apr 21, 2014 at 6:21 PM, Sumit Gaur <[email protected] >> > >> > > >> wrote: >> > > >> > >> > > >> > > hi Shri, >> > > >> > > Please find answers below >> > > >> > > >> > > >> > > On Tue, Apr 22, 2014 at 9:23 AM, Shrinand Javadekar < >> > > >> > > [email protected] >> > > >> > > > wrote: >> > > >> > > Few more questions to try and understand this better: >> > > >> > > >> > > >> > > 1) On the Swift instance you are using, how many replicas do you >> > > have? >> > > >> > > >> > > >> > > 3 replica >> > > >> > > >> > > >> > > 2) Also, how are you using the curl command in the shell script? >> > > >> > > >> > > >> > > send below command in backgroud for 10 iterations and wait >> > similiar >> > > to >> > > >> the >> > > >> > > 10 threads in jclouds. >> > > >> > > >> > > >> > > curl -X PUT -i -T 100k -H "X-Auth-Token: >> > $OS_AUTH_TOKEN" >> > > >> > > http:// >> > > >> > > >> > > >> > > >> > > >> >> > > >> > >> $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt >> > > >> > > curl -X DELETE -i -H "X-Auth-Token: $OS_AUTH_TOKEN" >> > > http:// >> > > >> > > >> > > >> > > >> > > >> >> > > >> > >> $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt >> > > >> > > >> > > >> > > I >> > > >> > > think the shell script and jclouds-with-10-parallel-threads may >> > not >> > > be >> > > >> > > doing the same amount of work. In 20 hours jclouds might be >> doing >> > > much >> > > >> > > more work than the shell script. If you let the shell script >> also >> > go >> > > >> > > upto that point, it might see failures too. Do you know how many >> > > >> > > PUT-GET-DEL operations have been performed when you start seeing >> > the >> > > >> > > 409 errors. >> > > >> > > >> > > >> > > Actually 409 errors are coming since the start of the test but >> TPS >> > > >> start >> > > >> > > degrading after 24-48 hours. >> > > >> > > On Apr 22, 2014 9:23 AM, "Shrinand Javadekar" < >> > > [email protected] >> > > >> > >> > > >> > > wrote: >> > > >> > > >> > > >> > > > Few more questions to try and understand this better: >> > > >> > > > >> > > >> > > > 1) On the Swift instance you are using, how many replicas do >> you >> > > >> have? >> > > >> > > > 2) Also, how are you using the curl command in the shell >> > script? I >> > > >> > > > think the shell script and jclouds-with-10-parallel-threads >> may >> > > not >> > > >> be >> > > >> > > > doing the same amount of work. In 20 hours jclouds might be >> > doing >> > > >> much >> > > >> > > > more work than the shell script. If you let the shell script >> > also >> > > go >> > > >> > > > upto that point, it might see failures too. Do you know how >> many >> > > >> > > > PUT-GET-DEL operations have been performed when you start >> seeing >> > > the >> > > >> > > > 409 errors. >> > > >> > > > >> > > >> > > > -Shri >> > > >> > > > >> > > >> > > > >> > > >> > > > On Mon, Apr 21, 2014 at 4:55 PM, Sumit Gaur < >> > [email protected] >> > > > >> > > >> > > wrote: >> > > >> > > > > FYI ..This is block of code ..... also I am using jclouds >> > > 1.7.1 >> > > >> > > (Stable >> > > >> > > > > branch) >> > > >> > > > > try { >> > > >> > > > > String key = "objkey" + UUID.randomUUID(); >> > > >> > > > > Blob blob = >> > > >> > > > > >> > > Example9.blobStore.blobBuilder(key).payload(Example9.file).build(); >> > > >> > > > > >> > > >> > > Example9.blobStore.putBlob(Example9.containerName+count, >> > > >> > > > > blob); >> > > >> > > > > >> > > >> > > Example9.blobStore.getBlob(Example9.containerName+count, >> > > >> > > > > key); >> > > >> > > > > >> > > >> > > > Example9.blobStore.removeBlob(Example9.containerName+count, >> > > >> > > > > key); >> > > >> > > > > } catch (Exception ace) { >> > > >> > > > > System.out.println("Request failed for >> objkey >> > " >> > > + >> > > >> key >> > > >> > > + " >> > > >> > > > > " + ace); >> > > >> > > > > } >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > On Tue, Apr 22, 2014 at 8:32 AM, Sumit Gaur < >> > > [email protected]> >> > > >> > > > wrote: >> > > >> > > > > >> > > >> > > > >> Hi Shri, >> > > >> > > > >> Thanks for paying attention to it, Please find my answers >> > > below:- >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >> On Tue, Apr 22, 2014 at 2:31 AM, Shrinand Javadekar < >> > > >> > > > >> [email protected]> wrote: >> > > >> > > > >> >> > > >> > > > >>> Sumit, >> > > >> > > > >>> >> > > >> > > > >>> I realize that you had sent out a similar email sometime >> ago >> > > >> about >> > > >> > > > >>> performance degradation. I'm not sure if anyone has run >> > these >> > > >> types >> > > >> > > of >> > > >> > > > >>> long running experiments with jclouds. So this may be a >> > first. >> > > >> > > > >>> >> > > >> > > > >> Tried to debug it in last 2 weeks without success. Want to >> > > >> understand >> > > >> > > > more >> > > >> > > > >> how jclouds code handle this use case or any pointers that >> > this >> > > >> is a >> > > >> > > > >> problematic use case would help >> > > >> > > > >> >> > > >> > > > >>> >> > > >> > > > >>> The 409 status is returned because of a conflict [1]. Are >> > you >> > > >> sure >> > > >> > > you >> > > >> > > > >>> didn't have two or more threads trying to delete the same >> > > object? >> > > >> > > > >>> >> > > >> > > > >> No two threads share the same object key in my programme >> > > (String >> > > >> key = >> > > >> > > > >> "objkey" + UUID.randomUUID();). It is some kind of race >> > between >> > > >> PUT >> > > >> > > and >> > > >> > > > >> DEL call . If I put say 10 ms sleep between call then there >> > is >> > > no >> > > >> 409 >> > > >> > > > error. >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >>> Also, I see that that 409 is returned by Swift if you try >> to >> > > >> delete a >> > > >> > > > >>> container that isn't empty[2]. Is that something your test >> > > code >> > > >> > > > >>> could've tried? >> > > >> > > > >>> >> > > >> > > > >> I am trying to delete objects .. not containers. >> > > >> > > > >> >> > > >> > > > >>> >> > > >> > > > >>> When you say there was a similar test you're trying with >> > curl, >> > > >> are >> > > >> > > you >> > > >> > > > >>> using the curl command-line utility or the libcurl >> library? >> > > >> > > > >> >> > > >> > > > >> curl command in shell script with for loops. >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >>> How are >> > > >> > > > >>> you specifying the number of threads to use and what >> object >> > > each >> > > >> > > > >>> thread should get/put/delete? >> > > >> > > > >>> >> > > >> > > > >> >> > > >> > > > >> It is a java test programme using ThreadPoolExecutor. >> > Somthing >> > > >> > > similiar >> > > >> > > > as >> > > >> > > > >> here >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >> > > >> > > >> > > >> >> > > >> > >> http://www.javacodegeeks.com/2013/01/java-thread-pool-example-using-executors-and-threadpoolexecutor.html >> > > >> > > > >> >> > > >> > > > >> Object is a 5KB file. with key = "objkey" + >> > UUID.randomUUID(); >> > > >> with >> > > >> > > > Pool >> > > >> > > > >> of 10 threads. >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >> Hope this would give a good inside. Let me know if you get >> > any >> > > >> problem >> > > >> > > > >> here. >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >>> >> > > >> > > > >>> Thanks. >> > > >> > > > >>> -Shri >> > > >> > > > >>> >> > > >> > > > >>> [1] >> http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html >> > > >> > > > >>> [2] https://bugs.launchpad.net/horizon/+bug/1096084 >> > > >> > > > >>> >> > > >> > > > >>> On Sun, Apr 20, 2014 at 5:55 PM, Sumit Gaur < >> > > >> [email protected]> >> > > >> > > > wrote: >> > > >> > > > >>> > Hi >> > > >> > > > >>> > I using jclouds lib integrated with Openstack Swift+ >> > > keystone >> > > >> > > > >>> combinaiton. >> > > >> > > > >>> > Things are working fine except stability test. After >> 20-30 >> > > >> hours of >> > > >> > > > test >> > > >> > > > >>> > jclouds/SWIFT start degrading in TPS and keep going down >> > > over >> > > >> the >> > > >> > > > time. >> > > >> > > > >>> > >> > > >> > > > >>> > 1) I am running the (PUT-GET-DEL) cycle in 10 parallel >> > > threads. >> > > >> > > > >>> > 2) I am getting a lot of 409 and DEL failure for the as >> > > >> response >> > > >> > > too >> > > >> > > > >>> from >> > > >> > > > >>> > SWIFT. >> > > >> > > > >>> > 3) Direct similiar test from curl does not show much >> > impact >> > > >> and TPS >> > > >> > > > >>> remain >> > > >> > > > >>> > constant. >> > > >> > > > >>> > >> > > >> > > > >>> > Can sombody help me wht is going wrong here ? >> > > >> > > > >>> > >> > > >> > > > >>> > Thanks >> > > >> > > > >>> > sumit >> > > >> > > > >>> >> > > >> > > > >> >> > > >> > > > >> >> > > >> > > > >> > > >> > > >> > > >> >> > > >> > >>
