[google-appengine] Re: HELP! 7 GB of ghost data - billable but not in datastore, blobs, or versions.
I know I probably shouldn't reply to myself, but this issue seems to be a bit more serious than your average newbie question. This is something approaching a 1000% inflation from my raw data to my billable data (actually, the csv data, without redundant field names, is even smaller, so it would be more like 2000%). Also, I've tried to google this, and I can't find any reference to similar problems. Sure, in the end it's probably partly my fault. For instance, my cron jobs might be considered overactive. But I still deserve some documentation that would help me understand where the heck my (boss's) money is (and will be?) going. At the moment, unless my boss wants to pay for this ghost data forever, I can only advise him to start fresh with a new app id... and with no guarantee the problem won't repeat. That is not going to make me popular, and I'd really like to avoid it. On Mar 21, 5:19 pm, homunq wrote: > My app is showing only 739 MB of datastore data, only 1 version > (<150MB), no blobs, a tiny amount of index and memcache use (<2K in > each case) - and yet 7.7GB of billable data! What the heck is going > on? How can I fix it? > > Last updated Total number of entities Size of all entities > 3:27:42 ago 279,689 739 MBytes > > Total Stored Data $0.005/GByte-day 86% 7.70 of 9.00 > GBytes > $0.04 / $0.04 > > This situation has persisted for well over 24 hours now, it's not just > a figment of the update period. Also, there's negligible non-default > indexes, and probably a total of around 2 GB if you count all my > uploaded code versions ever (but since they all had the same version > string, they should have overwritten). I probably have a fair amount > of data in the logs - I have a number of cron jobs, collectively they > run just under twice a minute - but supposedly logs are not billable > data. > > This is an app I'm developing for a client. Until this issue is fixed, > I'm certainly not going to bill for my work, which is otherwise done. > So I'm anxious to fix this ASAP. If there's somewhere else I should/ > could be taking this question (aside from IRC, where I've brought it > up twice), I'd be happy to learn it. If there's any further info which > could help resolve this, I'd be happy to share it, too. (My app id is > vulahealth) > > Thanks, > Jameson > > (Second post, I think the first post was moderated into oblivion - it > was my first post on this list.) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: HELP! 7 GB of ghost data - billable but not in datastore, blobs, or versions.
Thanks for responding. I've starred the bug - it seems basic. More answers below. On Mar 21, 7:06 pm, Eli Jones wrote: > Depending on how your Models are defined (how many properties and what the > property types are there) and depending on custom indexes, 10 to 1 is > reasonable. Even if I have essentially no custom indexes? (Only one ever, on a few dozen entities with two integers each which handle cron housekeeping). > > For my app, I aggressively set most properties to Indexed=false and I have > no custom indexes. I'm still at 2 to 1 for Datastore usage versus Size of > All Entities. When you say "datastore usage", you mean "total data", right? Or is there some other place this number is reported which I've missed? > > To fully explore what is going on, you'd need to post what your Model > definitions are.. and then post what custom indexes are defined on any of > the properties. I have no custom indexes. I have only two entities. 99.9% of the data is in the Expando which has about 45 stringproperties (initally all indexed, now around 30 are; many frequently empty string); 4 integer properties; 4 dates; 4 lists (about 4-8 members typically); from 3-18 expando string lists (of 20 types total; an average of 6 lists with 2 members each); and 2 expando strings (mostly but not always present). > > If you've been creating and then later on deleting custom indexes or > changing your Models around.. running vacuum indexes might help (but that's > just a shot in the dark). I have done nothing with custom indexes, except the one miniscule housekeeping one on a few entities. For the main entity kind, is there any way to delete a default index without re-touching all the entities one-by-one? > > On Sun, Mar 21, 2010 at 7:19 PM, homunq wrote: > > My app is showing only 739 MB of datastore data, only 1 version > > (<150MB), no blobs, a tiny amount of index and memcache use (<2K in > > each case) - and yet 7.7GB of billable data! What the heck is going > > on? How can I fix it? > > > Last updated Total number of entities Size of all entities > > 3:27:42 ago 279,689 739 MBytes > > > Total Stored Data $0.005/GByte-day 86% 7.70 of > > 9.00 GBytes > > $0.04 / $0.04 > > > This situation has persisted for well over 24 hours now, it's not just > > a figment of the update period. Also, there's negligible non-default > > indexes, and probably a total of around 2 GB if you count all my > > uploaded code versions ever (but since they all had the same version > > string, they should have overwritten). I probably have a fair amount > > of data in the logs - I have a number of cron jobs, collectively they > > run just under twice a minute - but supposedly logs are not billable > > data. > > > This is an app I'm developing for a client. Until this issue is fixed, > > I'm certainly not going to bill for my work, which is otherwise done. > > So I'm anxious to fix this ASAP. If there's somewhere else I should/ > > could be taking this question (aside from IRC, where I've brought it > > up twice), I'd be happy to learn it. If there's any further info which > > could help resolve this, I'd be happy to share it, too. (My app id is > > vulahealth) > > > Thanks, > > Jameson > > > (Second post, I think the first post was moderated into oblivion - it > > was my first post on this list.) > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: HELP! 7 GB of ghost data - billable but not in datastore, blobs, or versions.
On Mar 22, 4:20 am, homunq wrote: > I have no custom indexes. I have only two entities. 99.9% of the data > is in the Expando which has about 45 stringproperties (initally all > indexed, now around 30 are; many frequently empty string); 4 integer > properties; 4 dates; 4 lists (about 4-8 members typically); from 3-18 > expando string lists (of 20 types total; an average of 6 lists with 2 > members each); and 2 expando strings (mostly but not always present). For each indexed property, there will be 2 index entries (one in the forward index and one in the reverse index), each of which contains the actual data, the name of the property, the name of your application, and if there's a parent entity information about the parent will be in the key as well. This quickly adds up. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: HELP! 7 GB of ghost data - billable but not in datastore, blobs, or versions.
Sorry, this thread is a duplicate of http://groups.google.com/ group/google-appengine/browse_thread/thread/2a116e341b97c6fd/ 6ad663cd210032b2?lnk=gst">that other one. I got impatient and reposted before the other one got moderated. Useful word from Google along with my response is over there - basically, I ask, is there any way to bulk delete default indexes. On 22 mar, 05:44, Wooble wrote: > On Mar 22, 4:20 am, homunq wrote: > > > I have no custom indexes. I have only two entities. 99.9% of the data > > is in the Expando which has about 45 stringproperties (initally all > > indexed, now around 30 are; many frequently empty string); 4 integer > > properties; 4 dates; 4 lists (about 4-8 members typically); from 3-18 > > expando string lists (of 20 types total; an average of 6 lists with 2 > > members each); and 2 expando strings (mostly but not always present). > > For each indexed property, there will be 2 index entries (one in the > forward index and one in the reverse index), each of which contains > the actual data, the name of the property, the name of your > application, and if there's a parent entity information about the > parent will be in the key as well. This quickly adds up. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
