Re: [google-appengine] Re: Is the bulkloader ready for production use ?
This isn't completely true. If you use java and have Long primary keys, the Long keys get duplicated as string versions of those longs. So a bulk-download and then bulk-upload will actually duplicate all of your data (and sucks if you find out at the wrong time). On Sat, Sep 11, 2010 at 6:47 PM, Tim Hoffman zutes...@gmail.com wrote: Keys won't change if you restore into the appid. They must change if you restore to a new appid as the appid is part of the key. T On Sep 11, 11:59 pm, Tom Wu service.g2...@gmail.com wrote: Is any possible without key changing after restoring my data in same app ? Best Regards Tom Wu 2010/9/11 Tim Hoffman zutes...@gmail.com Hi In theory reference properties should be ok, (list properties holding a list of keys - especially as strings would have a problem). Have you compared the key in the reference property and the key of the target before and after the restore. It may be because you using id's rather than key names? But I am not sure without running some tests. I would work out what is being changed and you may find a custom loader for bulkloader may be required to move your dataset between appids. T On Sep 11, 10:24 pm, Francois MASUREL masu...@mably.com wrote: Yep, I was restoring to another appId. Does that mean that when you're storing entity keys as a reference property you can't backup/restore to another appId ? What is the best practice then ? Should we create some kind of generated UUID and use it as key for our entities then ? François On Sat, Sep 11, 2010 at 16:17, Tim Hoffman zutes...@gmail.com wrote: As to the keys changing, where you restoring to an instance with the same appid ? If not the keys have to change, and you might find problems where you store keys in list properties. T On Sep 11, 10:12 pm, Tim Hoffman zutes...@gmail.com wrote: Hi I am actually using appcfg.py download_data and upload_data with great success with 65,000 entities. Which is pretty much the same thing. I have found you may need to play around with the batch size settings if you have particularly large entities. T On Sep 11, 10:01 pm, Francois Masurel fm2...@mably.com wrote: Hello everybody, I've tried to do a full bulkloader --dump and --restore of one app to another one and it failed miserably. First, I had a quite a few Backing off errors during export which took more than 30 minutes for about 6000 small entities. Then when I restored it, I lost about 1000 entities it said it couldn't restore without explanations. And then I realized that all entities keys had been changed. So all my relations between entities were lost. Did I do something wrong ? Thanx for your help. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Index build stuck Server Error 500 when trying to view Datastore Indexes page in admin console
Couldn't you create two task queue's? That should fix the starvation problem. I suppose the trick would be to determine which queue to stick it on. On Fri, Sep 10, 2010 at 1:15 PM, Ikai L (Google) ika...@google.com wrote: I think one thing we can do better is to schedule some index builds for apps with no data ahead of apps with a lot of data, since they are essentially no-ops. Of course, one wonders if this means large index builds will be delayed forever? *rubs hands together for Computer Science fun* On Thu, Sep 9, 2010 at 7:11 PM, Jeff Schwartz jefftschwa...@gmail.comwrote: Soon after I posted the issue it was fixed. I don't know how or by whom but somehow it got resolved. I waited a few hours for the index to build even though there was no data to which the index was referring to. Thank you, Ikai. On Thu, Sep 9, 2010 at 9:39 PM, Ikai L (Google) ika...@google.comwrote: The indexes all look to be serving to me. On Thu, Sep 9, 2010 at 2:07 PM, Jeff Schwartz jefftschwa...@gmail.comwrote: I have 1 index that was building for quite a while though there was no data associated with the index. When I clicked on the Datastore Indexes link in the admin console I am directed to a error page that says: Server Error (500) A server error has occurred. Return to Applications screen » http://appengine.google.com/ My application's id is i-emote and its version is 1. Can anyone fix this for me and as soon as posible? Thanks in advance. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- -- Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Using OR statement in datastore
http://code.google.com/appengine/docs/java/datastore/queriesandindexes.html#Introducing_Queries See Query Filters. Should answer your how-to question. In the backend it actually does two separate queries, but in parallel (then does any merging/union-ing based on your entire query after retrieving the results). There are some google tech talks on how the datastore works which should give you a really good idea why it is done this way. On Sun, Sep 5, 2010 at 10:34 PM, killer barney ajcha...@gmail.com wrote: This is something I never figured out how to do. How do you supplement an OR statement in datastore? So let's say I have a user who sells Dogs and Cats. How do I get a list of all of the dogs and cats that a user sells? I thought of 2 ways, query both tables and combine the results in the backend or create an additional generic table Animals that tabulates all of the selling action into a list and search the Animals entries for that user. I created this scenario just so you can have an idea of what I am looking for, but in reality, my cats table may consist of hundreds of users. I'm not sure if creating a combined list property of Dog and Cat users is the way to go. And doing two queries just doesn't seem right either. Is there a better way to do this? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Using OR statement in datastore
An entity must match all filters to be a result. In the JDOQL string syntax, you can separate multiple filters with || (logical or) and (logical and), although keep in mind that || can only be employed when the filters it separates all have the same field name. In other words, || is only legal in situations where the filters it separates can be combined into a single contains() filter: // legal, all filters separated by || are on the same field Query query = pm.newQuery(Employee.class, (lastName == 'Smith' || lastName == 'Jones') + firstName == 'Harold'); // not legal, filters separated by || are on different fields Query query = pm.newQuery(Employee.class, lastName == 'Smith' || firstName == 'Harold'); On Sun, Sep 5, 2010 at 10:53 PM, killer barney ajcha...@gmail.com wrote: I'm a dunce. how does query filters help in the OR process? So do I do a query with contains dog and cat? On Sep 5, 8:44 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: http://code.google.com/appengine/docs/java/datastore/queriesandindexe... See Query Filters. Should answer your how-to question. In the backend it actually does two separate queries, but in parallel (then does any merging/union-ing based on your entire query after retrieving the results). There are some google tech talks on how the datastore works which should give you a really good idea why it is done this way. On Sun, Sep 5, 2010 at 10:34 PM, killer barney ajcha...@gmail.com wrote: This is something I never figured out how to do. How do you supplement an OR statement in datastore? So let's say I have a user who sells Dogs and Cats. How do I get a list of all of the dogs and cats that a user sells? I thought of 2 ways, query both tables and combine the results in the backend or create an additional generic table Animals that tabulates all of the selling action into a list and search the Animals entries for that user. I created this scenario just so you can have an idea of what I am looking for, but in reality, my cats table may consist of hundreds of users. I'm not sure if creating a combined list property of Dog and Cat users is the way to go. And doing two queries just doesn't seem right either. Is there a better way to do this? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Order of data
I don't know the solution in python. Perhaps someone else can lend a hand
here.
The solution in Java is simple, see this as an example:
import javax.jdo.annotations.Index;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;
@PersistenceCapable
public class U {
@PrimaryKey
@Persistent
private Long i;
@Index(unique = true)
@Persistent
private String n;
public U(Long id, String name) {
this.i = id;
this.n= name;
}
@Override
public String toString() { return [+n+:+i+]; }
public Long id() { return i; }
public String name() { return n; }
}
Then creating instances and persisting them is all that's left to do.
2010/8/25 Martynas Mickevičius mmarty...@gmail.com
Could you explain more how to use longs as a keys?
On 16 Lie, 19:10, Nate Bauernfeind nate.bauernfe...@gmail.com wrote:
Repeated calls will return items in a deterministic order ascending by
key.
So... yes it is deterministic, though it may not be the order you want.
Additionally, you can use Long's as your keys instead of making a counter
if
you need them to return in inserted order.
On Fri, Jul 16, 2010 at 11:07 AM, Blixt andreasbl...@gmail.com wrote:
Okay, so data will be returned in a non-determinate order unless you
specify keys yourself (since ids aren't guaranteed to be in order).
I'll make my own counter.
Thanks!
On Jul 16, 6:01 pm, Nate Bauernfeind nate.bauernfe...@gmail.com
wrote:
They will be ordered by key. You should consider watching one of the
datastore tech talks.
On Fri, Jul 16, 2010 at 3:54 AM, Blixt andreasbl...@gmail.com
wrote:
Imagine this query:
Person.all().fetch(100)
Is the order of data in the datastore guaranteed, or can the order
of
the entities returned change as data is inserted?
--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to
google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib
e...@googlegroups.comgoogle-appengine%2Bunsubscrib
e...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib
e...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Too much data makes Appengine Fail
You should do a couple of searches on this thread. There have been many people who analyze how much data is being used within their application and why it is not counted as part of the actual entity sizes (within the datastore viewer). If you can't do a search I highly recommend at least reading my post on this: http://groups.google.com/group/google-appengine/browse_thread/thread/67dcafbc9bb8811e/f75178d08eff55ee On Sun, Aug 8, 2010 at 11:12 AM, Matt H matt2...@gmail.com wrote: Maybe the statistics were slightly out of date? They aren't live. On Aug 8, 4:50 pm, NiceGuy devassyh...@gmail.com wrote: I made a new app in appengine which deals with a lot of data First i wrote a crawling using task queues and i got some 117k records tallying up to 418MB according to Datastore statistics. But my App quota was showing 78% usage which means 780MB !!! I could not understand this large disparity between Datastore statistics and later i added an issuehttp:// code.google.com/p/googleappengine/issues/detail?id=3555 , Since i didnot get any replies to issue i decided to delete all the entities that i had and i wrote another task queue based deleting in which will clear 10 entities per queue. It went smoothly well but after some two hours quotas showed that i have used up 100% of it and i cannot proceed furthur deletion. Now i see that at this time my App total stored data is 48% . But now i am not able to take any of Datastore statistics page or Datastore viewer page , When i click on those pages i see a page telling Server Error occurred . So i am not able to delete my records now . Now i will have to wait till tomorrow and do something i guess. As of now we cannot delete an App , But will it be possible to provide an option for clearing the app , I mean clearing all the entities of an App in one shot ? Please some form Google tell me Why DataStore statistics showed 418MB , while App Quota was showing 78% ( 780MB ). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Total Stored Data statistics not updated after datastore delete
It takes about 24 hours to update the real stored data counter. (The indexes are slow to update their portion of total stored data.) On Thu, Jul 22, 2010 at 9:57 AM, ivarconr ivarc...@gmail.com wrote: I just deleted most of the old data for my application and the size of all entities are now reported to be 44 MBytes. I have 6 indexes (where 3 are over 99% of the data). Shouldn't the total stored data counter be updated? It says I'm still using 280 MB, which sounds a lot to me, even with additional data for indexes. Application id: bikeoslo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Total Stored Data statistics not updated after datastore delete
I have a type that has 1.45 million instances using a total of 27mb of raw data but uses a total of 340mb in the datastore including *only* the default indexes (on the key and on a string field). On Thu, Jul 22, 2010 at 3:10 PM, ivarconr ivarc...@gmail.com wrote: I have been waiting a few days. Would expect it to be updated by now. On Jul 22, 9:21 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: It takes about 24 hours to update the real stored data counter. (The indexes are slow to update their portion of total stored data.) On Thu, Jul 22, 2010 at 9:57 AM, ivarconr ivarc...@gmail.com wrote: I just deleted most of the old data for my application and the size of all entities are now reported to be 44 MBytes. I have 6 indexes (where 3 are over 99% of the data). Shouldn't the total stored data counter be updated? It says I'm still using 280 MB, which sounds a lot to me, even with additional data for indexes. Application id: bikeoslo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Total Stored Data statistics not updated after datastore delete
If you want to use less space you will need to obfuscate your type's names. I'm using Java/Scala and my type's name is U and the parameter is n (in scala I can easily import U as User). This made a 200mb+ difference by reducing the names to one character. You may want to watch some of the How the Datastore works Google IO videos. All of the indexes include full parameter names on every entry, so it might be worth investigating if you're already in the process of repopulating your datastore. On Thu, Jul 22, 2010 at 3:34 PM, ivarconr ivarc...@gmail.com wrote: Ok, sounds like mine. I got 153k entities taking about 45MB raw data with 6 different indexes. The total stored data is reported to be 280MB, which sounds a bit high to me, but might be correct depending on how much overhead you get on each index. On Jul 22, 10:15 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: I have a type that has 1.45 million instances using a total of 27mb of raw data but uses a total of 340mb in the datastore including *only* the default indexes (on the key and on a string field). On Thu, Jul 22, 2010 at 3:10 PM, ivarconr ivarc...@gmail.com wrote: I have been waiting a few days. Would expect it to be updated by now. On Jul 22, 9:21 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: It takes about 24 hours to update the real stored data counter. (The indexes are slow to update their portion of total stored data.) On Thu, Jul 22, 2010 at 9:57 AM, ivarconr ivarc...@gmail.com wrote: I just deleted most of the old data for my application and the size of all entities are now reported to be 44 MBytes. I have 6 indexes (where 3 are over 99% of the data). Shouldn't the total stored data counter be updated? It says I'm still using 280 MB, which sounds a lot to me, even with additional data for indexes. Application id: bikeoslo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com google-appengine%252bunsubscr...@googlegroups.comgoogle-appengine%25252bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Order of data
They will be ordered by key. You should consider watching one of the datastore tech talks. On Fri, Jul 16, 2010 at 3:54 AM, Blixt andreasbl...@gmail.com wrote: Imagine this query: Person.all().fetch(100) Is the order of data in the datastore guaranteed, or can the order of the entities returned change as data is inserted? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Order of data
Repeated calls will return items in a deterministic order ascending by key. So... yes it is deterministic, though it may not be the order you want. Additionally, you can use Long's as your keys instead of making a counter if you need them to return in inserted order. On Fri, Jul 16, 2010 at 11:07 AM, Blixt andreasbl...@gmail.com wrote: Okay, so data will be returned in a non-determinate order unless you specify keys yourself (since ids aren't guaranteed to be in order). I'll make my own counter. Thanks! On Jul 16, 6:01 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: They will be ordered by key. You should consider watching one of the datastore tech talks. On Fri, Jul 16, 2010 at 3:54 AM, Blixt andreasbl...@gmail.com wrote: Imagine this query: Person.all().fetch(100) Is the order of data in the datastore guaranteed, or can the order of the entities returned change as data is inserted? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Hoping to understand the difference between CPU Time and Datastore CPU Time
I have been. On Wed, Jul 14, 2010 at 5:37 PM, Greg Tracy gtr...@gmail.com wrote: Does this imply that I can go over the CPU Time quota and not pay for it? On Jul 14, 5:26 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: The DataStore CPU time is included in total CPU time. To take advantage of free datastore cpu time you need to increase your cpu-usage quota. When I loaded a bunch of initial test data into the datastore I got tons of deadline exceeded errors, though I assumed it was cause I was trying to stuff a lot of data in there all at once. Nate On Wed, Jul 14, 2010 at 5:24 PM, Greg Tracy gtr...@gmail.com wrote: I've been adding some new datastore-intensive features to an app and am blowing through the CPU Time quota. While my understanding is that the Datastore CPU Time quotas have been lifted while the App Engine team continues to work on the performance, I can't figure out why these are mutually exclusive metrics in my app. When I measure the cpu cycles being consumed in the new features, it is almost exclusively in the db.put() call. In fact, 50% of the time, I'm getting DeadlineExceeded errors before the put call even returns. Is the Datastore CPU Time also counted in the CPU Time? If so, are there plans to extend the quota limits there as well? I'm interested in finding some resources that track performance metrics on App Engine. I don't know what to expect in terms of performance when I store, for example, 400 entities in the data store. Are there resources where folks are contributing their metrics for others to see? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Hoping to understand the difference between CPU Time and Datastore CPU Time
You pay for cpu_time - datastore_cpu_time. I was making the assumption that most of your cpu time was in the datastore (99% of my time was). On Wed, Jul 14, 2010 at 6:24 PM, Greg Tracy gtr...@gmail.com wrote: Not for me... I set a budget and immediately started paying. Not thrilled about this and may need to start a new thread asking for advice on datastore contention... Thanks. On Jul 14, 5:57 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: I have been. On Wed, Jul 14, 2010 at 5:37 PM, Greg Tracy gtr...@gmail.com wrote: Does this imply that I can go over the CPU Time quota and not pay for it? On Jul 14, 5:26 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: The DataStore CPU time is included in total CPU time. To take advantage of free datastore cpu time you need to increase your cpu-usage quota. When I loaded a bunch of initial test data into the datastore I got tons of deadline exceeded errors, though I assumed it was cause I was trying to stuff a lot of data in there all at once. Nate On Wed, Jul 14, 2010 at 5:24 PM, Greg Tracy gtr...@gmail.com wrote: I've been adding some new datastore-intensive features to an app and am blowing through the CPU Time quota. While my understanding is that the Datastore CPU Time quotas have been lifted while the App Engine team continues to work on the performance, I can't figure out why these are mutually exclusive metrics in my app. When I measure the cpu cycles being consumed in the new features, it is almost exclusively in the db.put() call. In fact, 50% of the time, I'm getting DeadlineExceeded errors before the put call even returns. Is the Datastore CPU Time also counted in the CPU Time? If so, are there plans to extend the quota limits there as well? I'm interested in finding some resources that track performance metrics on App Engine. I don't know what to expect in terms of performance when I store, for example, 400 entities in the data store. Are there resources where folks are contributing their metrics for others to see? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib e...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Task Queue - Stored Task Count - When will it be reset to 0?
I have observed this exact behavior and was wondering the same thing. On Sat, Jul 10, 2010 at 2:24 AM, Jagan ksja...@gmail.com wrote: Hi All the tasks in all my queues are done with. But why is the Admin Console showing the Stored Task Count as around 2000: --- Tasks Storage Quota Task Queue Stored Task Count 0% 2,819 of 1,000,000 Task Queue Stored Task Bytes 0% 430,337 of 1,073,741,824 When will this be reset? It has been weeks since all the tasks have run. Thanks Jagan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] how would i add multimedia to bigtable?
Consider using the blobstore. http://code.google.com/appengine/docs/java/blobstore/overview.html On Thu, Jul 8, 2010 at 10:50 AM, Rene Veerman rene7...@gmail.com wrote: Hi. How would i store multimedia in bigtable or another appengine facility? I mean large video's, large pictures, long audio, etc. -- - Greetings from Rene7705, My free open source webcomponents: http://code.google.com/u/rene7705/ http://mediabeez.ws/downloads (and demos) My music (i'm DJ firesnake) http://mediabeez.ws/music http://www.facebook.com/rene7705 - -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] how would i add multimedia to bigtable?
Yep. http://code.google.com/appengine/docs/python/blobstore/overview.html On Thu, Jul 8, 2010 at 11:59 AM, Rene Veerman rene7...@gmail.com wrote: ok cool. but can i use that thing from python as well? On Thu, Jul 8, 2010 at 6:42 PM, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: Consider using the blobstore. http://code.google.com/appengine/docs/java/blobstore/overview.html On Thu, Jul 8, 2010 at 10:50 AM, Rene Veerman rene7...@gmail.com wrote: Hi. How would i store multimedia in bigtable or another appengine facility? I mean large video's, large pictures, long audio, etc. -- - Greetings from Rene7705, My free open source webcomponents: http://code.google.com/u/rene7705/ http://mediabeez.ws/downloads (and demos) My music (i'm DJ firesnake) http://mediabeez.ws/music http://www.facebook.com/rene7705 - -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- - Greetings from Rene7705, My free open source webcomponents: http://code.google.com/u/rene7705/ http://mediabeez.ws/downloads (and demos) My music (i'm DJ firesnake) http://mediabeez.ws/music http://www.facebook.com/rene7705 - -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] 5 transactional tasks
I have noticed that batch datastore calls run within a single transaction. The maximum number of entities that can be added or deleted (and probably modified, though I have not tried) was 500. I'm betting you could probably wrap them around a single transaction. Though, from my experience, I wouldn't really recommend doing this (since trying to commit two batches of 500 to the datastore within the same call tended to time out for me). Also, don't confuse transactional tasks with the task queue. Transactional tasks are the those that will all happen simultaneously or will graciously fail without any partial commits. Due to the implementation details you probably want to avoid intentionally doing lots of transaction updates on the same few objects, since it is possible (if you are kicking lots of simultaneous jobs off with the task queue) to shoot yourself in the foot and have a very small success rate. Transactions work with the task queue in such a way that the task will only be added to the queue if no other piece of your commits fail. For example, you wouldn't really want your app to run the new user code if you couldn't create the new user account for that user because someone else registered that user name at the same time. I.E. Things on the task queue do not continue running within the same transaction if they were created within one. On Wed, Jul 7, 2010 at 4:22 AM, hawkett hawk...@gmail.com wrote: Hi, This page - http://code.google.com/appengine/docs/python/datastore/transactions.html - says that we cannot raise more than 5 transactional tasks in a single transaction, and I wanted to check if this was a limit that you were hoping to raise, or if this is likely to be a long term restriction? Does this restriction limit API calls to the task queue or actual number of tasks - e.g. could I raise more than 5 by doing them in batch with a single call to the task queue API? Cheers, Colin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the datastore not good for lots of small sized objects?
The statistic displayed went from 88% metadata down to 69%. Pre-index average size went from 100 bytes to 64 bytes. Again, the only modification was 12 char class name down to 1 and 4 char property name down to 1. On Fri, Jul 2, 2010 at 11:02 AM, djidjadji djidja...@gmail.com wrote: A nice reduction. How much percent is the Meta Data now? It would be a nice addition to GAE if we can have a section in index.yaml where we make all the needed indices explicit. No more implicit indices. To be compatible with the current implementation of GAE this section should be optional. The developer chooses if he wants implicit property indices that he partially can disable with the indexed argument of the Property. Or that all the indices are explicit. Maybe then we don't need an index on a property if we only use it in a complex query. The development server will create the explicit index definitions if needed, and the explicit-section is specified in index.yaml. GAE team is this possible? If yes, I will make an issue for this feature. 2010/7/2 Nate Bauernfeind nate.bauernfe...@gmail.com: For those who were following this thread, After reuploading all of my data but reducing the class name from 12 characters to 1 and from reducing the non-key field from 4 characters to 1, I was able to reduce my total footprint from 530mb to 340mb for 1.45M entities. Yielding about 245 bytes per entity (including the default indexes). Certainly wish I could get rid of the DESC index on my one property (for this entity type). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the datastore not good for lots of small sized objects?
For those who were following this thread, After reuploading all of my data but reducing the class name from 12 characters to 1 and from reducing the non-key field from 4 characters to 1, I was able to reduce my total footprint from 530mb to 340mb for 1.45M entities. Yielding about 245 bytes per entity (including the default indexes). Certainly wish I could get rid of the DESC index on my one property (for this entity type). On Mon, Jun 28, 2010 at 2:54 PM, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: Hmm. It looks like I should've read through the datastore specs prior to uploading so much data. I will give this another run with exactly the same data but with an obfuscated class definition. On Mon, Jun 28, 2010 at 2:06 PM, Geoffrey Spear geoffsp...@gmail.comwrote: On Jun 28, 1:49 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: Oh and does anyone know if the 500 mb in the quota usage is the 140mb of data replicated redundantly? I'm trying to figure out where that is coming from. Or is it that the 108mb of metadata is *not* the primary indexes? You don't pay extra for replication of your data. I believe the metadata figure is the space taken up in the protocol buffers by things like your application name and property names. If you have lots tiny entities and relatively long names for things, this can certainly be a significant proportion of your storage. The space used by the indexes themselves is, as far as I can tell, not reported anywhere, and only appears in the totals. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Is datastore still slow?
So far in my experience (over the past two days and then on the 26th) the data store has been slow for bulk inserts and deletes. I think that the bulk stuff is still iterating over each element and making single calls to the datastore. I don't know how possible it is to affect other users with my app's behavior... but I am in the process of trying to delete 1.5M entries. I have about 500k left, but my task has stopped deleting things due to timeouts. I don't know if it's because my primary indexes are behind or if the entire datastore is currently troubled. And I apologize if I am the culprit of issues (though from what I understand there are plenty of real-production-like apps implying that my behavior shouldn't affect anyone). I'm really not sure why there isn't a deleteEverythingBetweenKeys(startKey, endKey) functionality in the datastore. It sounds like BigTable should easily be able to support that. It is a serious deficiency of the datastore to not be able to bulk delete efficiently. On Wed, Jun 30, 2010 at 10:10 AM, Khookie chris.k...@gmail.com wrote: I'm working on an app at the moment, and it sometimes drops HTTP requests, especially ones that make calls to the datastore. There are periods during the day where the app is absolutely unusable because most AJAX calls just stalls and timeouts. At other times, the same queries work like an absolute champion - super fast. Are datastore issues still ongoing? Chris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: appcfg.py --dump or download_data
You may need to run: appcfg.py create_bulkloader_config --url= http://appid.appspot.com/remote_api --filename=bulkloader.yaml Prior to downloading the data. 2010/6/29 Luís Marques luismarq...@gmail.com http://stackoverflow.com/questions/3066934/appcfg-py-error-no-such-option-dump-on-google-app-engine The docs are wrong. Use: appcfg.py download_data --application=app_id --url=http://etc -- filename=file -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Problem with IdGeneratorStrategy.INCREMENT in JDO
Per: http://code.google.com/appengine/docs/java/datastore/creatinggettinganddeletingdata.html I think you want this: @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) private Long id; On Mon, Jun 28, 2010 at 9:40 PM, shookie10 chade.sh...@gmail.com wrote: Hi I am getting the following error when using the INCREMENT strategy. It show as available in the API but DataN. returns the error below There is no available value generator for strategy increment for this datastore. Relevant code : @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.INCREMENT) private Long id; It should create the id at runtime with no sequence. Is there something I am missing? Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: Bulkupload int key?
FWIW, I've been looking for a solution to this myself. I generally need to automate and bulk uploads to my app on a daily basis. Having any manual interaction to move the data will prevent my goal from ever being reached. So, the solution that I have found that seems to work is to create your own bulk uploader via the FileUpload example from here: http://code.google.com/appengine/kb/java.html#fileforms And you can then use python's poster library to post files to your app. You will still need to figure out how you want to handle administrative privileges. On May 21, 10:21 pm, Craig Berry cdbe...@gmail.com wrote: I have an xml file containing data which I wish to bulkupload to GAE. I am using the preview version of the new bulk uploader, if that matters. How do I include an int value for the __key__ derived from a field in the xml data? If I specify no import transform, like so - property: __key__ external_name: id export_transform: transform.key_id_or_name_as_string it loads the key as a string 'name', so it sorts lexicographically rather than numerically. I want an int id instead. But I can't find what to put in as an import_transform to make this happen. Thanks in advance for any help on this. -- Craig Berry -http://lapidum.org/home.html Magicians lie to the universe, and the universe believes them. -- Lenore Berry -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Java AppEngine App w/BulkUploader with key type == Long
I have a very simple app engine program. I'm following all of the JDO
instructions and have a class like this:
@PersistenceCapable
public class ExampleUser {
@PrimaryKey
@Persistent
private Long id;
@Persistent
private String name;
public ExampleUser(Long id, String name) {
this.id = id;
this.name = name;
}
@Override
public String toString() { return [+name+:+id+]; }
public Long id() { return id; }
public String name() { return name; }
}
I programmatically inserted one item for this and waiting for the
datastore statistics to exist before pulling down the config via
create_bulkloader_config. This is the yaml property_map:
50 - kind: ExampleUser
51 connector: csv # TODO: Choose a connector here: csv, simplexml,
etc...
52 connector_options:
53 # TODO: Add connector options here--these are specific to each
connector.
54 property_map:
55 - property: __key__
56 external_name: key
57 export_transform: transform.key_id_or_name_as_string
58
59 - property: name
60 external_name: name
61 # Type: String Stats: 11 properties of this type in this
kind.
62
Here's what a sample download looks like (with the names intentionally
truncated):
1 name,key
2 Nef,1
3 clu,2
4 Guy,1
5 edg,2
Now in this example lines 2 3 were inserted programmatically
specifying the id as a Long. Where as lines 4 5 were inserted with
the bulk uploader.
When I re-upload this file I lose users on lines 4 5 and users on 2
3 are there twice. This is the view from the datastore admin panel:
id=1 Nef
id=2 clu
name=1 Nef
name=2 clu
I really want to use the Long id's as ID's in other JDO objects, but
don't think I will be able to if I can't upload with explicit Id's. Is
there any way I can modify the yaml file to insert with id= and not
name=?
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Is the datastore not good for lots of small sized objects?
I would post to one of my other posts, except that they're still in moderator-limbo. So I apologize for creating a new thread. To briefly summarize my project has lots of very small objects. In particular I've uploaded about 1.45 Million objects with a Long id and a String name. On average, the strings are about 12 characters in length. I plan(ned) on adding about 84 Million other objects that consisted of 4 Longs (including the id). (And this is just for a subset of my data... which is probably on the order of 2x to 3x larger in total). Also it seems that I could estimate adding in about 200k new records every day. So I'm having several issues. One is that inserting 500 items takes about 30 seconds, which turns out to be a lot of CPU-api time. The second is that after loading up the 1.45 Million objects the datastore usage has ballooned to 500 megabytes. The raw data is only on the order of 30 megabytes, this is about the size that I can store it and the size that the datastore statistics says it takes up. The datastore statistics also says that about 108mb exists as metadata. I imagine this is the default index on key, and I imagine it is a reasonable value. However, on the main app of my page it says I'm taking up about 500 megabytes when the datastore statistics only account for about 140mb. Is this normal? Should I expect the rest of my data to balloon about the same? (i.e. 2.5 gigabytes of raw data to require 65 gigabytes pre-indexes) If so... Well that really sucks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] I'm seeing 30 second delays when inserting 500 small objects in bulk.
I've uploaded about 1.4 million objects made of a Long and a String (12 characters on average) today over the span of 6/7 hours. I could not use the bulk uploader since I need my Java objects to use real Long id's and not string's as id's. The upload process would bulk up 500 data items and make one POST request sending the data gzipped. The servlet unzips the data, builds up a collection (fwiw, I'm using an ArrayList) and then submitting all 500 objects in one fell swoop. So far, I think that this process is normal; if not please do let me know. Now my observations are as such: 1. Inserting 500 objects is taking about 30+ seconds (per the logs in the data store). 2. Nearly 3,000 page loads occurred using up 24.97 CPU/hours (24.76 CPU/hours worth in the datastore). Now this is not a lot of data. After everything has loaded into the datastore it's on the order of 20 megabytes. (Note that no indexes exist for this data.) Are these observations relatively common? Or is this only because of the recent issues of the datastore? If this is not common, does anyone know what I should expect in terms of cpu usage for something as simple as saving this small amount of data? Nate P.S. Is there a convenient way to query for something that matches one condition or another? (like query * where __key__ = KEY(..., 1) or __key__ = KEY(..., 2)) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the datastore not good for lots of small sized objects?
Is it possible to force it to make only the Ascending index and not the reverse too? I would like to be able to lookup the id by name, but I will never need reverse order. On Mon, Jun 28, 2010 at 12:28 PM, Darien Caldwell darien.caldw...@gmail.com wrote: The reason for the ballooning is, by default two indexes are created for every property in your Entity model. All of these indexes add quite a bit to the necessary storage space. If you have any properties in your model that don't require an index, it's a good idea to specify indexed=False for that property in the model. More info here: http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html Bulk uploading does take time, especially for so many entities, but hopefully it's something you only have to do once. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the datastore not good for lots of small sized objects?
Oh and does anyone know if the 500 mb in the quota usage is the 140mb of data replicated redundantly? I'm trying to figure out where that is coming from. Or is it that the 108mb of metadata is *not* the primary indexes? On Mon, Jun 28, 2010 at 12:47 PM, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: Is it possible to force it to make only the Ascending index and not the reverse too? I would like to be able to lookup the id by name, but I will never need reverse order. On Mon, Jun 28, 2010 at 12:28 PM, Darien Caldwell darien.caldw...@gmail.com wrote: The reason for the ballooning is, by default two indexes are created for every property in your Entity model. All of these indexes add quite a bit to the necessary storage space. If you have any properties in your model that don't require an index, it's a good idea to specify indexed=False for that property in the model. More info here: http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html Bulk uploading does take time, especially for so many entities, but hopefully it's something you only have to do once. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Java AppEngine App w/BulkUploader with key type == Long
BTW. If you haven't seen my other posts, I have gotten around this by
serving up a servlet that takes gzipped data and builds objects (in
the JVM) to stick into the datastore by the bulk. I found that sending
data for more than one datastore operation would often cause it to
fail. So I ended up sending only 500 objects per request and most of
them went through (though my stuff was not resilient enough to keep
track of which uploads failed which was on the order of 0.2% of them;
so if anyone does this I'd recommend making sure you have something in
place).
On Jun 26, 9:25 am, Nate Bauernfeind nate.bauernfe...@gmail.com
wrote:
I have a very simple app engine program. I'm following all of the JDO
instructions and have a class like this:
@PersistenceCapable
public class ExampleUser {
@PrimaryKey
@Persistent
private Long id;
@Persistent
private String name;
public ExampleUser(Long id, String name) {
this.id = id;
this.name = name;
}
@Override
public String toString() { return [+name+:+id+]; }
public Long id() { return id; }
public String name() { return name; }
}
I programmatically inserted one item for this and waiting for the
datastore statistics to exist before pulling down the config via
create_bulkloader_config. This is the yaml property_map:
50 - kind: ExampleUser
51 connector: csv # TODO: Choose a connector here: csv, simplexml,
etc...
52 connector_options:
53 # TODO: Add connector options here--these are specific to each
connector.
54 property_map:
55 - property: __key__
56 external_name: key
57 export_transform: transform.key_id_or_name_as_string
58
59 - property: name
60 external_name: name
61 # Type: String Stats: 11 properties of this type in this
kind.
62
Here's what a sample download looks like (with the names intentionally
truncated):
1 name,key
2 Nef,1
3 clu,2
4 Guy,1
5 edg,2
Now in this example lines 2 3 were inserted programmatically
specifying the id as a Long. Where as lines 4 5 were inserted with
the bulk uploader.
When I re-upload this file I lose users on lines 4 5 and users on 2
3 are there twice. This is the view from the datastore admin panel:
id=1 Nef
id=2 clu
name=1 Nef
name=2 clu
I really want to use the Long id's as ID's in other JDO objects, but
don't think I will be able to if I can't upload with explicit Id's. Is
there any way I can modify the yaml file to insert with id= and not
name=?
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the datastore not good for lots of small sized objects?
Hmm. It looks like I should've read through the datastore specs prior to uploading so much data. I will give this another run with exactly the same data but with an obfuscated class definition. On Mon, Jun 28, 2010 at 2:06 PM, Geoffrey Spear geoffsp...@gmail.comwrote: On Jun 28, 1:49 pm, Nate Bauernfeind nate.bauernfe...@gmail.com wrote: Oh and does anyone know if the 500 mb in the quota usage is the 140mb of data replicated redundantly? I'm trying to figure out where that is coming from. Or is it that the 108mb of metadata is *not* the primary indexes? You don't pay extra for replication of your data. I believe the metadata figure is the space taken up in the protocol buffers by things like your application name and property names. If you have lots tiny entities and relatively long names for things, this can certainly be a significant proportion of your storage. The space used by the indexes themselves is, as far as I can tell, not reported anywhere, and only appears in the totals. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
