[google-appengine] Re: Cost of mapreduce was $6,500 to update a ListProperty on 14.1 million entities

2012-01-05 Thread Yohan
Hi, I feel your pain. it cost me a few thousand dollars to delete my millions enities from the datastore after a migration job (ikai never replied my post though...) and im still paying since the deletion is not completed yet (spending 100-300$ a day for the past 2 weeks now!!). Not doing much ju

[google-appengine] Re: Where to start optimizing cost?

2012-01-02 Thread Yohan
s if your write frequency is very high. Or switch to memcache counters of your write frequency is super high (much faster than the queue api but risk of losing data). On Jan 2, 8:02 am, jon wrote: > Yohan your suggestion regarding the use of pull queue for batch writes > is interesting. In t

[google-appengine] Re: Where to start optimizing cost?

2011-12-31 Thread Yohan
10,000$ a month in gae that's a lot and then 10,000$ in consulting fees with brandon haha anyway back to business: 1) are you using python or java 2) how many apps are generating that much traffic 3) check the admin panel maybe even post a screenshot here (or link to it or something) it will alrea

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-29 Thread Yohan
Yep the blobstore approach is a better solution IMHO but costs will still be prohibitive. Im afraid nothong much you can do about that. Initially i thought that we could get an export of the datastore, download it as a binary file and process it locally (like the local .bin used for devs) but i'm d

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-29 Thread Yohan
0th of a CTO, 1/10 of a product manager 2 test > engineers 1/5 of a release manager, and 1/5 of a performance engineer. That > is 2.5 support staff for every programmer.  If you are just writing code you > are working in a vacuum that makes it hard to plan, test, debug, and run > sca

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-29 Thread Yohan
ot;mistakes" grow to the point we can't control them. > > > > > > > > -Original Message- > From: google-appengine@googlegroups.com > > [mailto:google-appengine@googlegroups.com] On Behalf Of Raymond > Sent: Thursday, December 29, 2011 12:13 AM > To: Go

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-29 Thread Yohan
price of deletion a bit. On Dec 29, 4:13 pm, Raymond wrote: > Dear Yohan, > > On my side I thank you for sharing your experience, I am beginning > with GAE and know that whatever the time I will put on this project I > will be making beginner mistakes and this kind of info is precio

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-29 Thread Yohan
ing you've done and start from scratch. Very frustrating... On Dec 29, 1:25 pm, jon wrote: > Yohan I agree that there should be an easy and cheap way to get your > data out. I think it's a little unfair that leaving GAE is made that > hard. > > How much did you spend on yo

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-29 Thread Yohan
other tools... and a lot of things that are easier to > build on app engine.  And some things that are best hybrids of GAE and > something else. > > Jeff > > > > > > > > > > On Wed, Dec 28, 2011 at 5:26 AM, Yohan wrote: > > Hi Brandon, > > &

[google-appengine] Re: Cautionary Tale: Abusive price for data migration and deletion

2011-12-28 Thread Yohan
Hi Brandon, Although i agree with you that the original dataset wasnt fully optimized (that was over 2 years ago), i believe that i have a good understanding of datatore vs SQL, caching etc. Im not building public facing website im dealing with private apis and I am already stretching memcache and

[google-appengine] Cautionary Tale: Abusive price for data migration and deletion

2011-12-27 Thread Yohan
Hi fellow developers, just a cautionary tale for the new members out there and people building up large datasets. We already know that the difference in reported datastore size between the actual data and the total size is due to the indexes and various voodoo stuff that the datastore is doing to

[google-appengine] Export Data in bulk WITH blobs and text

2011-11-07 Thread Yohan
Hi, I've been trying to export the entities in my datastore for offline processing. I tried the python bulkloader but it wont export the binary data (im serializing stuff in blobs and text fields i need this data as well - maybe b64 encoded or something). And the java remote api is just a remote

[google-appengine] Re: Generate valid datastore key manually

2011-06-19 Thread Yohan
hy not just generate a GUID and use that as the > key name?  Or, setup an API in your app that you make an AJAX call to that > returns a key for you? > > If you are still interested in the > code:http://code.google.com/p/googleappengine/source/browse/#svn%2Ftrunk%2... > > Robert &

[google-appengine] Generate valid datastore key manually

2011-06-19 Thread Yohan
Hi Guys, I've been trying to generate a valid datastore Key manually in javascript. For some reason, i need the javascript to know the keys of the objects the server will generate before sending the request to the server (as long as both JS and Servlet generate the keys the same way). I couldn't

[google-appengine] Re: Is GAE expected to get faster in the future?

2011-05-02 Thread Yohan
Hi Brandon, So how do you explain that Google serves pages in 50ms when GAE serves pages at 300ms ? And what to say about Google instant. These guys can do better than 300ms for sure. I built my own config on a AWS instance relying on memcache+mysql+php it would server under 50ms. If you look at

[google-appengine] Re: GAE Hauling the Freight

2011-05-02 Thread Yohan
I used to have an API app running at 1700 req/s with spikes about 2000+ running in multithreaded mode with about 50 instances in parallel. Not bad but costs where spiking for some reason. After investigation it seemed that memcache wasnt holding the data long enough between requests and the app had

[google-appengine] Billing Disabled

2010-12-17 Thread yohan
Hi, I'm new gae developer, now I'm developing adlinkshopsafe.appspot.com I have a problem, about the billing setting, I had disable the billing, my status now is must wait 7 day, now I want to re-enable the billing, is there any way to re-enable imediately? -- You received this message because y

[google-appengine] Over Quota for over 6 hours and Green Dashboard !

2010-09-06 Thread yohan radioactive
Hi there, Simple one : the app has been displaying an 503 Over Quota error for the last 6 hours although the dashboard shows green everywhere. I suspect the app has been "disabled" from your side since the previous deployment was killing the quota real fast. I fixed it in latest release now can I