That is similar to what I am seeing. Writing to the datastore is VERY expensive. 130K items for me consumes nearly 6.5 hours of CPU. Not very efficient
On Aug 16, 5:36 pm, Stakka <henrik.lindqv...@gmail.com> wrote: > I implemented a rough version of my solution, and it seems to work up > to ~15k entities. Above that I hit the undocumented transaction write > limit you mention when trying to commit 36408 entities serialized into > 24 blobs of 999960 bytes: > > java.lang.IllegalArgumentException: datastore transaction or write too > big. > > Well, the datastore seems fast enough for large dataset writes, but > all the limitations really makes it troublesome to implement. Also the > potentiallossofdataintegrity while processing in multiple requests/ > tasks without transactions is risky. Costly too, an 15k entities > "upload" comsumes about 30 minutes of CPU quota. > > On 15 Aug, 19:57, Juraj Vitko <juraj.vi...@gmail.com> wrote: > > > I agree with everything you said. Just one thing to consider: by first > > storing the uploadeddata, then retrieving thatdatafor reprocessing > > and then storing the processeddataagain will consume additional > > resources / quotas of your app. > > > GAE really appears to be designed for apps with very high read to > > write ratio. I would say, if you don't need to handle more than > > thousand of concurrent users, then you'd be better off renting a > > server. Into this I've factored additional hassles you may not know > > about yet, like index size and count limits, single entity group write > > limits, transaction limitations. All of these are possible to work > > around, but I have yet to see if those workarounds are feasible in > > terms of the final price I will be paying to run the app. > > > On Aug 14, 9:24 pm, Stakka <henrik.lindqv...@gmail.com> wrote: > > > > Thanks for the tip, but why write a web app when Java Applets are > > > required, that whouldn't be a good solution. Also, the uploaded file > > > needs to be parsed in it's entirety (CRC check, value references, > > > etc.), and it's not XML. > > > > I think I have to parse the file server-side, populate (Java) Entity > > > objects and serialize as many I can into 1 MB blobs. When that is > > > done, start a task that put the de-serialized entities in batches of > > > 500 into the datastore. The response for the file upload request will > > > have to contain some unique task URL that the browser can (AJAX) poll > > > to display the progress. > > > > Before I commit to such a elaborate solution, I'll have to test the > > > batch-put performance to see if GAE is even suitable for this kind of > > > app.http://groups.google.com/group/google-appengine/browse_thread/thread/... > > > > Users of an online apps shouldn't have to wait hours for a simpledata > > > import just because it's hosted at GAE. If the app where using an SQL > > > database this would only take a minute. > > > > On Aug 14, 4:48 pm, Juraj Vitko <juraj.vi...@gmail.com> wrote: > > > > > I think you need to write your own Flash or Java Applet based chunked > > > > uploader. Or use an existing one and let us know, so that we can use > > > > it too. > > > > > On Aug 12, 11:36 pm, Stakka <henrik.lindqv...@gmail.com> wrote: > > > > > > I'am working on an browser based accounting app which has a feature to > > > > > import ledger transactions through file uploads. I'am currently only > > > > > running on the local dev server, but from what I've read datastore > > > > > puts -- even batch -- is very slow and CPU (quota) intensive when > > > > > deployed live. > > > > > > How do I overcome this problem if the user uploads a large file with > > > > > thousands transaction? > > > > > > I've seen solutions where you batch put entities in chunks of 500. > > > > > That only works if you run a custom upload tool on your computer, not > > > > > from a browser since the request is limited to 30 seconds. Am I forced > > > > > to use the Task Queue? But where do I store the raw uploaded file or > > > > > the preferably parsed interim transaction entities when the task isn't > > > > > executing? > > > > > > Funny App Engine has a 10 megabyte request (file upload) size limit > > > > > when storing 10 megabyte worth of entities seems to be so hard. > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
