[appengine-java] Implementing custom login
Anyone have tips on how to best implement custom login. Not forcing users to have an Google account. I've thought of two possible solutions: * Implement custom login in a Servlet Filter, thus replacing the standard servlet security model. * Implement an OpenID provider, then use Federated Login with only that provider. This will let me use the standard servlet security model and GAE build-in UserServices. Discuss... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
One feature I see useful for a layer atop the Low-Level API is asynchronization. Since you'll never know how much time a Low-Level API call will take it should automatically create Tasks for them, writes atleast. On Oct 22, 10:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Massive datastore batch put, how to?
I wonder how Google thinks the app providers should bill their customers? A flat fee isn't feasible when a single user action costs soo much of the app profit. Limiting the number of actions (imports) is a possibility, but thats bad for the app's productiveness. Google should remove the cost of CPU for internal API calls. It might be more fair if they billed for the number of API calls in combination with the megabyte size of data in each call. That would also encurage good app design; Batch put whould be cheaper than many single puts. Small entities with few indices cheaper than large entities with many indices, etc,. Google say App Engine is great for small startups and scale to millions of users. But how will a startup afford to reach the scale where GAE shines. With regular hosting you atleast know the cost in advance, and during heavy load it's just slower. I really like GAE, I'd like to use it! But with all the missing JRE classes, many limits and the cost of datastore CPU I probably can't for my current project. ;( On 21 Aug, 16:57, Mark Jones mark0...@gmail.com wrote: That is similar to what I am seeing. Writing to the datastore is VERY expensive. 130K items for me consumes nearly 6.5 hours of CPU. Not very efficient On Aug 16, 5:36 pm, Stakka henrik.lindqv...@gmail.com wrote: I implemented a rough version of my solution, and it seems to work up to ~15k entities. Above that I hit the undocumented transaction write limit you mention when trying to commit 36408 entities serialized into 24 blobs of 60 bytes: java.lang.IllegalArgumentException: datastore transaction or write too big. Well, the datastore seems fast enough for large dataset writes, but all the limitations really makes it troublesome to implement. Also the potentiallossofdataintegrity while processing in multiple requests/ tasks without transactions is risky. Costly too, an 15k entities upload comsumes about 30 minutes of CPU quota. On 15 Aug, 19:57, Juraj Vitko juraj.vi...@gmail.com wrote: I agree with everything you said. Just one thing to consider: by first storing the uploadeddata, then retrieving thatdatafor reprocessing and then storing the processeddataagain will consume additional resources / quotas of your app. GAE really appears to be designed for apps with very high read to write ratio. I would say, if you don't need to handle more than thousand of concurrent users, then you'd be better off renting a server. Into this I've factored additional hassles you may not know about yet, like index size and count limits, single entity group write limits, transaction limitations. All of these are possible to work around, but I have yet to see if those workarounds are feasible in terms of the final price I will be paying to run the app. On Aug 14, 9:24 pm, Stakka henrik.lindqv...@gmail.com wrote: Thanks for the tip, but why write a web app when Java Applets are required, that whouldn't be a good solution. Also, the uploaded file needs to be parsed in it's entirety (CRC check, value references, etc.), and it's not XML. I think I have to parse the file server-side, populate (Java) Entity objects and serialize as many I can into 1 MB blobs. When that is done, start a task that put the de-serialized entities in batches of 500 into the datastore. The response for the file upload request will have to contain some unique task URL that the browser can (AJAX) poll to display the progress. Before I commit to such a elaborate solution, I'll have to test the batch-put performance to see if GAE is even suitable for this kind of app.http://groups.google.com/group/google-appengine/browse_thread/thread/... Users of an online apps shouldn't have to wait hours for a simpledata import just because it's hosted at GAE. If the app where using an SQL database this would only take a minute. On Aug 14, 4:48 pm, Juraj Vitko juraj.vi...@gmail.com wrote: I think you need to write your own Flash or Java Applet based chunked uploader. Or use an existing one and let us know, so that we can use it too. On Aug 12, 11:36 pm, Stakka henrik.lindqv...@gmail.com wrote: I'am working on an browser based accounting app which has a feature to import ledger transactions through file uploads. I'am currently only running on the local dev server, but from what I've read datastore puts -- even batch -- is very slow and CPU (quota) intensive when deployed live. How do I overcome this problem if the user uploads a large file with thousands transaction? I've seen solutions where you batch put entities in chunks of 500. That only works if you run a custom upload tool on your computer, not from a browser since the request is limited to 30 seconds. Am I forced to use the Task Queue? But where do I store the raw uploaded
[google-appengine] Re: Transactionally updating multiple entities over 1MB
Wierd, I just hit a limit on the size of a transaction when commiting: java.lang.IllegalArgumentException: datastore transaction or write too big. All (23) entities in the transaction where in the same entity group, not using batch put and ~990k in size. On 23 Juli, 12:30, Nick Johnson (Google) nick.john...@google.com wrote: Hi Juraj, No, there's no limit to the size of an entity group - only on the maximum rate at which you can update entities in a single entity group. -Nick Johnson On Fri, Jul 17, 2009 at 4:03 PM, Juraj Vitko juraj.vi...@gmail.com wrote: Nick, just one clarification (I can't find in docs) - is there a limit on the total size of an entity group? On Jun 29, 12:28 pm, Nick Johnson (Google) nick.john...@google.com wrote: On Sat, Jun 27, 2009 at 4:14 PM, Andy Freemanana...@earthlink.net wrote: Does that mean that db.put((e1, e2, e3,)) where all of the entities are 500kb will fail? Yes. Thanks. I'll take this opportunity to promote a couple of related feature requests. (1) We need a way to estimate entity sizes http://code.google.com/p/googleappengine/issues/detail?id=1084 The 1MB limit is on the API call, rather than the entity itself, per-se, so index size doesn't count in the 1MB limit. You can always serialize the entity yourself and check its size, though that requires touching datastore-internal methods. (2) We need a way to help predict when datastore operations will fail http://code.google.com/p/googleappengine/issues/detail?id=917 I assume that db.get((k1, k2,)) can fail because of size reasons when db.get(k1) followed by db.get(k2) will succeed. Does db.get((k1, k2,)) return at least one entity in that case? No, the operation will simply fail. Given that it's an invariant that the returned list has the same length as the passed list, there's no sensible way to return partial results without implying that certain entities didn't exist when they actually do. -Nick Johnson On Jun 26, 9:36 am, Nick Johnson (Google) nick.john...@google.com wrote: On Fri, Jun 26, 2009 at 4:42 PM, Andy Freeman ana...@earthlink.net wrote: the 1MB limit applies only to single API calls Does that mean that db.put((e1, e2, e3,)) where all of the entities are 500kb will fail? Yes. Where are limits on the total size per call documented? http://code.google.com/appengine/docs/python/datastore/overview.html#... only mentions a limit on the size of individual entities and the total number of entities for batch methods. The batch method documentation ( http://code.google.com/appengine/docs/python/datastore/functions.html andhttp:// code.google.com/appengine/docs/python/memcache/functions.html) does not mention any limits. You're right - we need to improve our documentation in that area. The 1MB limit applies to _all_ API calls. Is there a documented limit on the number of entities per memcache call? No. BTW - There is a typo in http://code.google.com/appengine/docs/python/memcache/overview.html#Q... . It says In addition to quotas, the following limits apply to the use of the Mail service: instead of Memcache service Thanks for the heads-up. -Nick Johnson On Jun 26, 7:28 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi tav, Batch puts aren't transactional unless all the entities are in the same entity group. Transactions, however, _are_ transactional, and the 1MB limit applies only to single API calls, so you can make multiple puts to the same entity group in a transaction. -Nick Johnson On Fri, Jun 26, 2009 at 8:53 AM, tavt...@espians.com wrote: Hey guys and girls, I've got a situation where I'd have to transactionally update multiple entities which would cumulatively be greater than the 1MB datastore API limit... is there a decent solution for this? For example, let's say that I start off with entities E1, E2, E3 which are all about 400kb each. All the entities are specific to a given User. I grab them all on a remote node and do some calculations on them to yield new computed entities E1', E2', and E3'. Any failure of the remote node or the datastore is recoverable except when the remote node tries to *update* the datastore... in that situation, it'd have to batch the update into 2 separate .put() calls to overcome the 1MB limit. And should the remote node die after the first put(), we have a messy situation =) My solution at the moment is to: 1. Create a UserRecord entity which has a 'version' attribute corresponding to the latest versions of the related entities for any given User. 2. Add a 'version' attribute to all the entities.
[google-appengine] Re: Massive datastore batch put, how to?
I implemented a rough version of my solution, and it seems to work up to ~15k entities. Above that I hit the undocumented transaction write limit you mention when trying to commit 36408 entities serialized into 24 blobs of 60 bytes: java.lang.IllegalArgumentException: datastore transaction or write too big. Well, the datastore seems fast enough for large dataset writes, but all the limitations really makes it troublesome to implement. Also the potential loss of data integrity while processing in multiple requests/ tasks without transactions is risky. Costly too, an 15k entities upload comsumes about 30 minutes of CPU quota. On 15 Aug, 19:57, Juraj Vitko juraj.vi...@gmail.com wrote: I agree with everything you said. Just one thing to consider: by first storing the uploaded data, then retrieving that data for reprocessing and then storing the processed data again will consume additional resources / quotas of your app. GAE really appears to be designed for apps with very high read to write ratio. I would say, if you don't need to handle more than thousand of concurrent users, then you'd be better off renting a server. Into this I've factored additional hassles you may not know about yet, like index size and count limits, single entity group write limits, transaction limitations. All of these are possible to work around, but I have yet to see if those workarounds are feasible in terms of the final price I will be paying to run the app. On Aug 14, 9:24 pm, Stakka henrik.lindqv...@gmail.com wrote: Thanks for the tip, but why write a web app when Java Applets are required, that whouldn't be a good solution. Also, the uploaded file needs to be parsed in it's entirety (CRC check, value references, etc.), and it's not XML. I think I have to parse the file server-side, populate (Java) Entity objects and serialize as many I can into 1 MB blobs. When that is done, start a task that put the de-serialized entities in batches of 500 into the datastore. The response for the file upload request will have to contain some unique task URL that the browser can (AJAX) poll to display the progress. Before I commit to such a elaborate solution, I'll have to test the batch-put performance to see if GAE is even suitable for this kind of app.http://groups.google.com/group/google-appengine/browse_thread/thread/... Users of an online apps shouldn't have to wait hours for a simple data import just because it's hosted at GAE. If the app where using an SQL database this would only take a minute. On Aug 14, 4:48 pm, Juraj Vitko juraj.vi...@gmail.com wrote: I think you need to write your own Flash or Java Applet based chunked uploader. Or use an existing one and let us know, so that we can use it too. On Aug 12, 11:36 pm, Stakka henrik.lindqv...@gmail.com wrote: I'am working on an browser based accounting app which has a feature to import ledger transactions through file uploads. I'am currently only running on the local dev server, but from what I've read datastore puts -- even batch -- is very slow and CPU (quota) intensive when deployed live. How do I overcome this problem if the user uploads a large file with thousands transaction? I've seen solutions where you batch put entities in chunks of 500. That only works if you run a custom upload tool on your computer, not from a browser since the request is limited to 30 seconds. Am I forced to use the Task Queue? But where do I store the raw uploaded file or the preferably parsed interim transaction entities when the task isn't executing? Funny App Engine has a 10 megabyte request (file upload) size limit when storing 10 megabyte worth of entities seems to be so hard. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Datastore put performance?
Has anyone benchmarked the live datastore, especially the low-level batch put operation? How many entities per second can I expect? I know it all depends on number of properties, ancestors, indices, etc,. but any estimate would be appreciated. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Massive datastore batch put, how to?
Thanks for the tip, but why write a web app when Java Applets are required, that whouldn't be a good solution. Also, the uploaded file needs to be parsed in it's entirety (CRC check, value references, etc.), and it's not XML. I think I have to parse the file server-side, populate (Java) Entity objects and serialize as many I can into 1 MB blobs. When that is done, start a task that put the de-serialized entities in batches of 500 into the datastore. The response for the file upload request will have to contain some unique task URL that the browser can (AJAX) poll to display the progress. Before I commit to such a elaborate solution, I'll have to test the batch-put performance to see if GAE is even suitable for this kind of app. http://groups.google.com/group/google-appengine/browse_thread/thread/41276f470e12037c# Users of an online apps shouldn't have to wait hours for a simple data import just because it's hosted at GAE. If the app where using an SQL database this would only take a minute. On Aug 14, 4:48 pm, Juraj Vitko juraj.vi...@gmail.com wrote: I think you need to write your own Flash or Java Applet based chunked uploader. Or use an existing one and let us know, so that we can use it too. On Aug 12, 11:36 pm, Stakka henrik.lindqv...@gmail.com wrote: I'am working on an browser based accounting app which has a feature to import ledger transactions through file uploads. I'am currently only running on the local dev server, but from what I've read datastore puts -- even batch -- is very slow and CPU (quota) intensive when deployed live. How do I overcome this problem if the user uploads a large file with thousands transaction? I've seen solutions where you batch put entities in chunks of 500. That only works if you run a custom upload tool on your computer, not from a browser since the request is limited to 30 seconds. Am I forced to use the Task Queue? But where do I store the raw uploaded file or the preferably parsed interim transaction entities when the task isn't executing? Funny App Engine has a 10 megabyte request (file upload) size limit when storing 10 megabyte worth of entities seems to be so hard. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Massive datastore batch put, how to?
I'am working on an browser based accounting app which has a feature to import ledger transactions through file uploads. I'am currently only running on the local dev server, but from what I've read datastore puts -- even batch -- is very slow and CPU (quota) intensive when deployed live. How do I overcome this problem if the user uploads a large file with thousands transaction? I've seen solutions where you batch put entities in chunks of 500. That only works if you run a custom upload tool on your computer, not from a browser since the request is limited to 30 seconds. Am I forced to use the Task Queue? But where do I store the raw uploaded file or the preferably parsed interim transaction entities when the task isn't executing? Funny App Engine has a 10 megabyte request (file upload) size limit when storing 10 megabyte worth of entities seems to be so hard. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] How to handle a file upload containing thousands of entities?
I'am working on an browser based accounting app which has a feature to import ledger transactions through file uploads. Currently it's only running on the local dev server, but from what I've read datastore puts -- even batch -- is very slow and CPU (quota) intensive when deployed live. So, how do I overcome this problem when a user uploads a file with thousands of transactions? I've seen solutions where you batch put entities in chunks of 500. That only works if you run a custom upload tool on your computer, not from a browser since the request is limited to 30 seconds. Am I forced to use the Task Queue? If so, where do I store the raw uploaded file or preferably the parsed interim transaction entities when the task isn't executing? It's funny that App Engine has a 10 megabytes request (file upload) size limit when storing 10 megabytes worth of entities is such a hazzle. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: How to handle a file upload containing thousands of entities?
Sorry for the dup posting, 1st post screening took ages. http://groups.google.com/group/google-appengine/browse_thread/thread/b11a07e4a71566b6# --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
