[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
Great! Since someone else is now using this besides me, I've added more comments. Also, I've added some testcases and fixed an issue when invoking put() with entities that contain partial keys. If your entities don't have complete keys when invoking put(), you should go get the latest code. Let me know if you run into any problems, or have suggestions for improvements. Vince On Mon, Nov 2, 2009 at 10:47 PM, Roy Smith roy.smith@googlemail.com wrote: Hi Vince Thanks for sharing. I've modified my persistence framework to incorporate your class and it works very well. kind regards Roy --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
One feature I see useful for a layer atop the Low-Level API is asynchronization. Since you'll never know how much time a Low-Level API call will take it should automatically create Tasks for them, writes atleast. On Oct 22, 10:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
You might be interested in my CachingDatastoreService class: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlanta/appengine/datastore/CachingDatastoreService.java It has the following features: - Implements the com.google.appengine.api.datastore.DatastoreService, so it's a plug-in replacement for the standard implementation. - Automatically caches entities in memcache for datastore reads and writes. - Has a write-behind option (the default) that queues all datastore writes as background tasks (except for transactions, which are always write-through directly to the datastore). - A watchdog task makes sure the write-behind task is always available. - If the write-behind task isn't available, defaults to write-through to insure no loss of data. - Supports configurable expiration of memcache entities (the default is no expiration). In order to use the CachingDatastoreService, first configure the write-behind task in web.xml: servlet servlet-nameCachingDatastoreService/servlet-name servlet-classcom.newatlanta.appengine.datastore.CachingDatastoreService/servlet-class /servlet servlet-mapping servlet-nameCachingDatastoreService/servlet-name url-pattern/_ah/queue/write-behind-task/url-pattern /servlet-mapping Then configure the write-behind-task queue in queue.xml (use whatever rate you want): queue namewrite-behind-task/name rate5/s/rate /queue Then replace the following code: DatastoreService ds = DatastoreServiceFactory().getDatastoreService(); with this code, and then use the DatastoreService methods as you normally would: DatastoreService ds = new CachingDatastoreService(); The default CachingDatastoreService constructor enables the CacheOptions.WRITE_BEHIND option and sets the expiration to null (no expiration). There are additional constructors that allow you to specify CacheOptions.WRITE_THROUGH and/or specify a memcache expiration value. Vince On Mon, Nov 2, 2009 at 2:21 PM, Stakka henrik.lindqv...@gmail.com wrote: One feature I see useful for a layer atop the Low-Level API is asynchronization. Since you'll never know how much time a Low-Level API call will take it should automatically create Tasks for them, writes atleast. On Oct 22, 10:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
Hi Vince Thanks for sharing. I've modified my persistence framework to incorporate your class and it works very well. kind regards Roy On Mon, Nov 2, 2009 at 8:15 PM, Vince Bonfanti vbonfa...@gmail.com wrote: P.S. Because of issue 2097 (http://code.google.com/p/googleappengine/issues/detail?id=2097), the write-behind task doesn't work on the development server, so it always defaults to write-through (which is a nice test of the watchdog mechanism). The write-behind task works fine on the production servers. On Mon, Nov 2, 2009 at 3:12 PM, Vince Bonfanti vbonfa...@gmail.com wrote: You might be interested in my CachingDatastoreService class: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlanta/appengine/datastore/CachingDatastoreService.java It has the following features: - Implements the com.google.appengine.api.datastore.DatastoreService, so it's a plug-in replacement for the standard implementation. - Automatically caches entities in memcache for datastore reads and writes. - Has a write-behind option (the default) that queues all datastore writes as background tasks (except for transactions, which are always write-through directly to the datastore). - A watchdog task makes sure the write-behind task is always available. - If the write-behind task isn't available, defaults to write-through to insure no loss of data. - Supports configurable expiration of memcache entities (the default is no expiration). In order to use the CachingDatastoreService, first configure the write-behind task in web.xml: servlet servlet-nameCachingDatastoreService/servlet-name servlet-classcom.newatlanta.appengine.datastore.CachingDatastoreService/servlet-class /servlet servlet-mapping servlet-nameCachingDatastoreService/servlet-name url-pattern/_ah/queue/write-behind-task/url-pattern /servlet-mapping Then configure the write-behind-task queue in queue.xml (use whatever rate you want): queue namewrite-behind-task/name rate5/s/rate /queue Then replace the following code: DatastoreService ds = DatastoreServiceFactory().getDatastoreService(); with this code, and then use the DatastoreService methods as you normally would: DatastoreService ds = new CachingDatastoreService(); The default CachingDatastoreService constructor enables the CacheOptions.WRITE_BEHIND option and sets the expiration to null (no expiration). There are additional constructors that allow you to specify CacheOptions.WRITE_THROUGH and/or specify a memcache expiration value. Vince On Mon, Nov 2, 2009 at 2:21 PM, Stakka henrik.lindqv...@gmail.com wrote: One feature I see useful for a layer atop the Low-Level API is asynchronization. Since you'll never know how much time a Low-Level API call will take it should automatically create Tasks for them, writes atleast. On Oct 22, 10:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
With JPA+RDBMS a relationship between two objects (say, Invoice and Customer) could be implemented by a customer attribute inside the Invoice class. In the RDBMS, this is translated to a CUSTOMER_ID column in the INVOICE table. Datanucleus does not support this (the relationship must be owned, that is, the Invoice class must be nested inside the Customer instance), otherwise you must do this by hand, managing the Key instance yourself (if I understood correctly). What I don't get is the rationale behind this. With DataNucleus (AccessPlatform) you have the full range of JDO and JPA relations for RDBMS, since it passes the TCKs for both, and would be impossible to do that if it didn't support something so basic. So in the situation you describe as not supported yes you do get a CUSTOMER_ID FK in the INVOICE table. This has been supported since the very first version of JPOX ... back in 2003. If instead you're talking about GAE/J specifics, as per the link that you provided, then that is a different issue ... dependent on Googles plugin for DataNucleus. Only they can comment on that. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I didn't use Spring, but I found detach/attach annoying and irrelevant in GAE context because: 1. Without detaching, when you update a field, and later on close the PM, the object will be committed regardless. This makes no sense in GAE as 99.99% of time you want to control what you commit or not. Worse yet if you persist other object and have a transaction, it might fail because the objects are in different entity group. 2. To get around #1, you might want to detachCopy the object once it's fetched, so that it won't be accidentally committed. However, when the object have a child object (or fields that's not in default fetch group), detach won't work (when you get child will throw error). You need to first get that child, detach it, then use the child as is. It's very messy when detaching a object with different types of children. 3. Putting object (that has children) in memcache is even more annoying. You can't just put it in memcache, because when it's read, you can't get the child. I am not sure if I get the child first then I will be able to read it later. I just couldn't make it work. 4. When serialized (happens when you memcache put), the size is actually bigger then necessary because some datastore connection related information is also serialized. I don't remember exactly, but a minimal JDO object is like ~500byte. If I serialize its fields with Hessian it's ~50 bytes. Probably not a issue when your object is big. The object I want to cache is some Session info, which is very small, but there might be lots of them. Also, if you really want to cache a JDO object as is, try to make it transient first, it will be smaller. Overall I just feel like detach/attach create more problems then its worth. I just can't think of a scenario that the JDO states are actually beneficial within a GAE context. Here's what I just found out with JDO today: If you change a type of a field, say from Long to Double, you can never delete the entries with JDO. You need to fetch the object first, then delete it, but you can't fetch it because a ClassCast exception will happen. I end up writing low level keys only query and using low level API to delete the entities with keys. I can go on but I am tired. :) On Oct 22, 7:21 pm, Rusty Wright rwright.li...@gmail.com wrote: Peter, it was gratifying to hear you say detach/attach is also problematic when dealing with caching and transactions because I've been banging my head against the wall trying to write integration tests with GAE's datastore, using JDO and Spring's transactions. I either get the is managed by a different Object Manager error or my objects don't get persisted or other errors I can't remember now. I can't figure out if I'm getting bit by Spring, GAE's datastore, or JDO, and it's very frustrating. What has been your experience, war stories, etc.? Peter Liu wrote: Awesome! I recently played with the lower level API as well. Some of the features are not available in JDO, like reserving a key before committing a new object. The detach/attach is also problematic when dealing with caching and transaction. I also noticed small JDO objects are much bigger than it needs to be when serialized. Just wondering do you have any performance profiling stats? I also feel like JDO is overkill, but I couldn't justify porting all JDO objects to another simple type without knowing how much cpu_ms it will save. Also, do you keep track of which field is dirty and avoid committing unmodified fields? I believe this is important to minimize api_cpu to avoid unnecessary index updates. On Oct 22, 2:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. *
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
1. Without detaching, when you update a field, and later on close the PM, the object will be committed regardless. This makes no sense in GAE as 99.99% of time you want to control what you commit or not. Worse yet if you persist other object and have a transaction, it might fail because the objects are in different entity group. Which is why there are transactions in JDO, and the use of rollback(). How GAE/J want to make use of these is down to them, but this is not a limitation of JDO. 2. To get around #1, you might want to detachCopy the object once it's fetched, so that it won't be accidentally committed. However, when the object have a child object (or fields that's not in default fetch group), detach won't work (when you get child will throw error). You need to first get that child, detach it, then use the child as is. It's very messy when detaching a object with different types of children. Which is why there are fetch groups in JDO and fetch depth so you, the user, has full control over what gets detached. You can detach whatever you need. 3. Putting object (that has children) in memcache is even more annoying. You can't just put it in memcache, because when it's read, you can't get the child. I am not sure if I get the child first then I will be able to read it later. I just couldn't make it work. JDO supports Level2 Cache. This is turned off by default but you could enable it, to use the memcached feature of GAE/J. This will mean objects are put into the L2 cache for you, and taken out when you need them. If you change a type of a field, say from Long to Double, you can never delete the entries with JDO. You need to fetch the object first, then delete it, but you can't fetch it because a ClassCast exception will happen. I end up writing low level keys only query and using low level API to delete the entities with keys. This is again not a limitation of JDO. You get the same issue with RDBMS; you have to migrate your schema. Google could provide a schema upgrade facility to take an existing schema definition and migrate to the updated form. I can go on but I am tired. :) I'm tired too ;-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I think most complains about JDO in this group is not saying that JDO has issues. It just that GAE is so different then traditional environments that old frameworks aren't 100% suitable. If there's a modify version of JDO that strip out irrelevant features, and put in some important low level api features, it will be much much better. I believe the users who are writing there own frameworks are doing exactly that, from the ground. On Oct 23, 1:19 am, datanucleus andy_jeffer...@yahoo.com wrote: 1. Without detaching, when you update a field, and later on close the PM, the object will be committed regardless. This makes no sense in GAE as 99.99% of time you want to control what you commit or not. Worse yet if you persist other object and have a transaction, it might fail because the objects are in different entity group. Which is why there are transactions in JDO, and the use of rollback(). How GAE/J want to make use of these is down to them, but this is not a limitation of JDO. 2. To get around #1, you might want to detachCopy the object once it's fetched, so that it won't be accidentally committed. However, when the object have a child object (or fields that's not in default fetch group), detach won't work (when you get child will throw error). You need to first get that child, detach it, then use the child as is. It's very messy when detaching a object with different types of children. Which is why there are fetch groups in JDO and fetch depth so you, the user, has full control over what gets detached. You can detach whatever you need. 3. Putting object (that has children) in memcache is even more annoying. You can't just put it in memcache, because when it's read, you can't get the child. I am not sure if I get the child first then I will be able to read it later. I just couldn't make it work. JDO supports Level2 Cache. This is turned off by default but you could enable it, to use the memcached feature of GAE/J. This will mean objects are put into the L2 cache for you, and taken out when you need them. If you change a type of a field, say from Long to Double, you can never delete the entries with JDO. You need to fetch the object first, then delete it, but you can't fetch it because a ClassCast exception will happen. I end up writing low level keys only query and using low level API to delete the entities with keys. This is again not a limitation of JDO. You get the same issue with RDBMS; you have to migrate your schema. Google could provide a schema upgrade facility to take an existing schema definition and migrate to the updated form. I can go on but I am tired. :) I'm tired too ;-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I think most complains about JDO in this group is not saying that JDO has issues. It just that GAE is so different then traditional environments that old frameworks aren't 100% suitable. If there's a modify version of JDO that strip out irrelevant features, and put in some important low level api features, it will be much much better. Totally agree there are features of the ***ORM part of the JDO specification*** that aren't applicable to BigTable, but the whole point of JDO is that those should only be used where they are applicable to the datastore. To give an example, we (DataNucleus) support persistence to ODF documents, yet such a thing as @SecondaryTable is simply ignored there since it makes no sense. The same idea should be applied to GAE/J. There is absolutely nothing in the JDO ***API*** that is irrelevant to GAE/J and BigTable. When somebody actually defines what are these low level features that can't be handled by the JDO API then there would be a basis for comment, so if someone wants to define that then there can be meaningful discussion. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
There is absolutely nothing in the JDO ***API*** that is irrelevant to GAE/J and BigTable. I disagree, and that's the main reason why we developed our own framework. These are just some random thoughts about this subject: * Transactions in JDO is a global thing tied to the persistence store (one database = one transaction), but for GAE it's one transaction per entity group. It's perfectly reasonable to execute two transactions at the same time, which is hard to fit into the traditional development model with spring. * The current way of generating a primary key with a parent is not natural IMHO. My key attribute already has a parent, why should I add _another_ attribute with the parent PK? * Entity retrieval can be done just by providing the primary key, but JDO and JPA are not prepared for it - you must provide the entity class as well. This is redundant. * There are a lot of use cases (structures etc) supported by datanucleus that can be optimized out. This is important in a system where I pay per CPU cycle. The list goes on. The point is that you cannot develop a standard framework that works fine with a RDBMS and expect it to be the optimal solution for a storage that is entirely different. For me, there are enough differences that justify going for a specific solution. My .02 euro. P.S.: Please keep in mind that I have absolutely nothing against datanucleus, which is indeed a nice implementation. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I recently played with the lower level API as well. Some of the features are not available in JDO, like reserving a key before committing a new object. Yes, we wanted to get access to those. Specially, the create several keys at once and persist several entities at once are great, we combined those to implement the put(IterableObject) method. Just wondering do you have any performance profiling stats? I also feel like JDO is overkill, but I couldn't justify porting all JDO objects to another simple type without knowing how much cpu_ms it will save. Nope. I can tell you that deployments are much faster and the typical stack trace has dropped from 10-15 lines to 3. And, of course, class enhancements are no longer necessary. Also, do you keep track of which field is dirty and avoid committing unmodified fields? I believe this is important to minimize api_cpu to avoid unnecessary index updates. I don't think you can do that (please someone correct me if I'm wrong). If you omit any field while putting an entity, you lose those fields in the persisted entity because it's schema-less. What you can do is disable field checks (required fields while saving, existing fields while querying) to make it faster in production, if you feel confident in your QA. Regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
Do you have some more doc/java doc ? I was browsing through your page and found nothing. As far as I caught you implemented simple set of CRUD operations on items and simple query mechanism. Am I right ? The decision to get rid of relationships is very sound because this GAE/J implementation is rather like a bargain - it brings more troubles than advantages. The only advantage of owned relationship is to persist it under transactions - but you can achieve transactional behavior in more straightforward way. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I have uploaded the generated javadoc here: http://code.google.com/p/simpleds/downloads/list I haven't had time to review it yet, so take it with a grain of salt. The list of features you should look for are: * CRUD operations at the EntityManager interface * SimpleQuery * PagedQuery That's it. The generated javadoc also makes reference to some classes that are not yet included in the library, such as @MultivaluedIndex which is a feature we are playing with to avoid the hassle of implementing Relation Index Entities. Regards, Nacho. On Fri, Oct 23, 2009 at 4:34 PM, leszek leszek.ptokar...@gmail.com wrote: Do you have some more doc/java doc ? I was browsing through your page and found nothing. As far as I caught you implemented simple set of CRUD operations on items and simple query mechanism. Am I right ? The decision to get rid of relationships is very sound because this GAE/J implementation is rather like a bargain - it brings more troubles than advantages. The only advantage of owned relationship is to persist it under transactions - but you can achieve transactional behavior in more straightforward way. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
* Transactions in JDO is a global thing tied to the persistence store (one database = one transaction), but for GAE it's one transaction per entity group. It's perfectly reasonable to execute two transactions at the same time, which is hard to fit into the traditional development model with spring. One txn per PM, at any one time. You can have multiple PMs open, hence multiple txns. How Spring handles that is for Spring. * The current way of generating a primary key with a parent is not natural IMHO. My key attribute already has a parent, why should I add _another_ attribute with the parent PK? That is the problem of Google for exposing their Key. A user doesn't care about internal handling. When we provided support for db4o we didn't expose db4o's key, instead providing a clean long/String/int type etc, and doing conversions internally ... aka transparent ... the whole point of JDO. * Entity retrieval can be done just by providing the primary key, but JDO and JPA are not prepared for it - you must provide the entity class as well. This is redundant. pm.getObjectById(Object id) It's an id, a JDO identity (pm.getObjectId(obj)). Can't get simpler. JPA insists on the class too but then that's JPA. * There are a lot of use cases (structures etc) supported by datanucleus that can be optimized out. This is important in a system where I pay per CPU cycle. Yes, but when the plugin doing the work (the GAE/J DN plugin) doesn't (shouldn't) touch the majority of those then that is minimal The list goes on. Please do continue the list, because I'm sure Max et al will be interested to know what you think are problem areas so they can cater for such things in their DN plugin. P.S.: Please keep in mind that I have absolutely nothing against datanucleus, which is indeed a nice implementation. Not a problem, neither do I have anything against any alternative, just wanting to understand the motivations, and to see if there is something that can be adapted. The fact is that the GAE/J DN plugin is open source too, and Max et al are open to input on it --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
On Fri, Oct 23, 2009 at 6:06 PM, datanucleus andy_jeffer...@yahoo.comwrote: * Transactions in JDO is a global thing tied to the persistence store (one database = one transaction), but for GAE it's one transaction per entity group. It's perfectly reasonable to execute two transactions at the same time, which is hard to fit into the traditional development model with spring. One txn per PM, at any one time. You can have multiple PMs open, hence multiple txns. How Spring handles that is for Spring. Injecting one PM per transaction seems cumbersome to me. It's also not documented anywhere AFAIK (at least, not in the transactions page). * The current way of generating a primary key with a parent is not natural IMHO. My key attribute already has a parent, why should I add _another_ attribute with the parent PK? That is the problem of Google for exposing their Key. A user doesn't care about internal handling. When we provided support for db4o we didn't expose db4o's key, instead providing a clean long/String/int type etc, and doing conversions internally ... aka transparent ... the whole point of JDO. I would say that AppEngine users care a big deal about the internals, specially when considering sharding. I understand that Google low-level interface does not fit well with JDO key conversion, but again that is my whole point - that maybe the standard is not the best option for this concrete case. Please do continue the list, because I'm sure Max et al will be interested to know what you think are problem areas so they can cater for such things in their DN plugin. Now that we are discussing it, I am curious: why does datanucleus only support owned relationships? There is not that much difference between the typical Foreign Key field and a GAE Key property. The fact is that the GAE/J DN plugin is open source too, and Max et al are open to input on it I am really grateful for all the insight about the subject. Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I believe there are a lot of reasons and use cases not to go the JDO/ JPA way and look for a light weight solution. I came up with a similar idea like Nacho and created a simple class to persist Java objects using the low-level API. However I consider my code not being in a state yet for publising it. I also agree that this initiatives/ideas should not claim to be a replacement for JPA/JDO. It is simply an alternative. In the situation where I not even can write a file something as simple like a file might be attractive. Also it seems to me that JPA/JDO is tight to the relational model and the datastore is not. Java is not either. So going from Java to the relational model and the mapping back to a non-relational somehow seems to be odd. JPA/ JDO would be attractive to hide if there is a non-relational or relational store to a Java developer. Just provide the annotations/ configuration and do not care about the characteristics of the store. But looking at all the limitations I doubt that it will ever work like this. I am eager to follow this discussion and as you said: at the end we will benefit all from sharing thoughts and ideas how we can improve things. Jens --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
Sounds good. I also think I'll focus much of my attention on the low- level API, so this could be very useful! Regards, Richard On Oct 22, 11:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
I did more or less the same thing for the same reasons, and with the same happy result.The only difference for me was instead of annotations, I generate my model from an RDBMS. This way I know I can port my app to an RDBMS world should GAE ever go the same way as Google Notebook. On Thu, Oct 22, 2009 at 11:43 AM, Richard richard.wat...@gmail.com wrote: Sounds good. I also think I'll focus much of my attention on the low- level API, so this could be very useful! Regards, Richard On Oct 22, 11:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
My goal wasn't to economise on api _ms so I haven't done any comparisons. afaik, commit is done at an entity level, so I don't monitor individual fields for changes. If I'm wrong then it wouldn't be too difficult to build dirty flags into my DTO setters. My advice to anybody building apps for GAE is to have your own abstraction layer between JDO/JPA/LLAPI so any porting between regimes is painless. This is why I took the approach of modelling my data in an RDBMS and then generating my object data model automatically. At first it sounds like a retrograde step, but the benefit is that my data model is independent of any specific persistence framework (and the design tools are more mature). On Thu, Oct 22, 2009 at 11:37 PM, Peter Liu tinyee...@gmail.com wrote: Awesome! I recently played with the lower level API as well. Some of the features are not available in JDO, like reserving a key before committing a new object. The detach/attach is also problematic when dealing with caching and transaction. I also noticed small JDO objects are much bigger than it needs to be when serialized. Just wondering do you have any performance profiling stats? I also feel like JDO is overkill, but I couldn't justify porting all JDO objects to another simple type without knowing how much cpu_ms it will save. Also, do you keep track of which field is dirty and avoid committing unmodified fields? I believe this is important to minimize api_cpu to avoid unnecessary index updates. On Oct 22, 2:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
The name initially confused me because it made me think of http://www.opends.org/ Nacho Coloma wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here: http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: SimpleDS: an alternative for Datastore persistence
Peter, it was gratifying to hear you say detach/attach is also problematic when dealing with caching and transactions because I've been banging my head against the wall trying to write integration tests with GAE's datastore, using JDO and Spring's transactions. I either get the is managed by a different Object Manager error or my objects don't get persisted or other errors I can't remember now. I can't figure out if I'm getting bit by Spring, GAE's datastore, or JDO, and it's very frustrating. What has been your experience, war stories, etc.? Peter Liu wrote: Awesome! I recently played with the lower level API as well. Some of the features are not available in JDO, like reserving a key before committing a new object. The detach/attach is also problematic when dealing with caching and transaction. I also noticed small JDO objects are much bigger than it needs to be when serialized. Just wondering do you have any performance profiling stats? I also feel like JDO is overkill, but I couldn't justify porting all JDO objects to another simple type without knowing how much cpu_ms it will save. Also, do you keep track of which field is dirty and avoid committing unmodified fields? I believe this is important to minimize api_cpu to avoid unnecessary index updates. On Oct 22, 2:37 am, Nacho Coloma icol...@gmail.com wrote: Hi all, We have been developing a persistence framework for the AppEngine Datastore based on the raw DatastoreService API. For our (simple) persistence case, both JDO and JPA were a bit overkill as we were spending a significant amount of time jumping through hoops to make our application roll, but at the same time the Datastore API was a too low-level solution to be usable. So we reinvented our wheel. In two days. SimpleDS is a light wrapper around the DatastoreService APIs that provide a simple interface for java persistent classes. It does not include fancy stuff or any super features, it's just the DatastoreService ported to a world where Java entities can be persisted directly (using a subset of JPA annotations). This is _not_ a JPA/JDO replacement, and will never be. But we have been using it for some weeks and are quite happy with it. Any kind of feedback from the AppEngine community would be welcome. Before calling the typical but we already have JPA/JDO! argument, please notice the following: * There are lots of considerations in a relational database that do not apply to AppEngine. This allows a specific solution to be simplified big time. Just see the depth of your typical stack trace to understand what I am talking about. * Solutions can be designed for specific cases that are common practice in AppEngine but do not apply to a relational database. See, for example, saving entities with a parent instance. * Transactions also behave a bit differently, where a one size fits all approach would probably not be the best solution. To better ilustrate with an example, these are some typical tasks performed with SimpleDS: Retrieve instance: FooBar bar = entityManager.get(key); Transform from Google Datastore Entity to java and viceversa: Entity entity = entityManager.datastoreToJava(bar); Save generating a primary key, with a parent instance: FooBar bar = new FooBar(); entityManager.put(parentKey, bar); More can be seen here:http://code.google.com/p/simpleds/wiki/SimpleTasks Any discussion about the current API state is welcome. This entire thing was rolled in two days and tested in a couple of weeks so there should be some bugs in between. It is important to keep in mind the current list of limitations: * Only the Key class is a supported primary key. * IN and != are not supported (yet). I have big concerns about supporting this, performance-wise. * Relationships are not supported. You can use Keys and collections of Keys for that purpose. * Transactions are not yet included. We are not yet sure about how to proceed here. As I said, this is not conceived to become a feature-complete JPA replacement, so please don't treat it like that. Best regards, Nacho. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
