POST - run multiple upload batches at once. each tile is probably stored independently so there should be no write contention. uploading 10 tiles at the same time will (mostly) give you 10x performance increase. in Arc2Earth, our bulk tile uploader works on multiple folders and we have not seen a problem yet
what is the duration of each of your POST uploads? that sounds like a lot of time for 5500 requests. I'm seeing ~70ms per tile request (256x256, 5kb-25kb PNGs) On Sep 4, 9:16 am, Alper <[EMAIL PROTECTED]> wrote: > Hi, > > I want to ask another thing about tile hosting. > > I have been using datastore for storing tiles for more than 2 weeks > and there is no problem about datastore, but the problem is upload > tiles to datastore. > > Sample tile set is composed of approximately 5500 tiles with 27 MBs. > > I have tried 2 different methods to load : > > 1.Method : (take 8 minutes to fetch) > I have uploaded all tiles to internet and GAE fetch it from the web. > This method is faster than 2.Method but I don't want to do a second > work as uploading to the internet before fetching. > > 2.Method: (take 10 hours to POST) > > I have uploaded tiles via POST with Curl from my computer but it is > very very slow compared to the 1.Method which can be seen above. > > My question is that is there a way to accelerate the uploading via > POST method? > > Thanks. > > Alper. > > On Sep 2, 9:36 pm, Fred <[EMAIL PROTECTED]> wrote: > > > Thanks all, the verdict then is to try the datastore first. The tiles > > will be refreshed on a regular basis (probably every 24 hours) using > > an application outside the GAE, so the reliability and speed of the > > bulk uploader will be an important factor. The last time I tried the > > bulk uploader I found it would fail occassionally - perhaps it's more > > reliable now. > > > Otherwise, I'll give Amazon's S3 a try. > > > On Sep 2, 6:16 pm, Wooble <[EMAIL PROTECTED]> wrote: > > > > On Sep 2, 12:09 pm, Fred <[EMAIL PROTECTED]> wrote: > > > > > Tim: > > > > > > why aren't you storing your tiles as entities in the store rather than > > > > > as files? > > > > > I was concerned about CPU quotas - each tile request would hit the > > > > datastore. > > > > If the information in Issue 161 is correct and the reason for the 1000 > > > file limit is that the files themselves are being stored in the > > > datastore, I imagine you'll get a similar CPU hit whether you're > > > accessing "files" or pulling them out of the datastore directly, and > > > I'd also imagine that the CPU time involved in loading a zip file of > > > thousands of tiles, unarchiving it, and then finding the tile(s) you > > > want would be even worse. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
