Re: [google-appengine] Simple query times out repeatedly for hours!

Thanks Stephen. Good point about the possibility of background splitting. But then again, the app has been running for a year without problems, and suddenly last week that query started to timeout. I didn't do any app updates recently to cause this. And when the query times-out, it tends to keep t

Re: [google-appengine] Memcache proper use?

Hi, So to clarify on the memcache principal, High reads - low writes, memcache is a good option. High reads - high writes, might as well only use the datastore? Thanks, Rohan -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this

[google-appengine] Re: Poor performance since the past 2-3 days

Back to normal here, too. Thanks. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegrou

[google-appengine] Re: Poor performance since the past 2-3 days

Same here. No more issues. Thanks Google team! Sarang myContactID: **2010-1399-1401* -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googl

[google-appengine] Re: Up loading jpg files to google docs area

You should have no problem storing any type of document in the datastore. If you want to upload to google docs, I would check out oauth authentication and for getting the docs there use one of the apis like Robert is referring to. I have some notes here that I think are relevant to what you mi

[google-appengine] Re: data structure size

Your correct so far the entity limit is 1M. I've heard talk into looking into moving limit but haven't seen anything yet. http://code.google.com/appengine/docs/java/datastore/overview.html#Quotas_and_Limits What I do when I hit the limit is shard the data into multiple entities, and/or I use th

[google-appengine] Re: Remove self from app permissions

I can't. I was given 'developer' permission which has restricted access. On May 25, 12:24 pm, Robert Kluin wrote: > It is probably a little "rude", but you could always delete the application... > > > > > > > > On Wed, May 25, 2011 at 10:32, Backpack wrote: > > I've been invited to collaborate

[google-appengine] Re: data structure size

I don't mean entity size, I mean actual variable size in your code. Like, x = 1 There used to be a limit on the size of x, but now it appears it has been lifted... Just looking for a confirmation On May 26, 8:42 am, Brandon Donnelson wrote: > Your correct so far the entity limit is 1M. I've hea

[google-appengine] Re: data structure size

Yes properties can = 1024 bytes. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroup

[google-appengine] Re: data structure size

my bad, variables can be much larger, you'll have to worry about the memory limit then. My bad -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this

[google-appengine] Re: data structure size

I guess the "data structure size" confused me. Usually that refers to stored data. Then you said variable, which JDO has. Are you talking about class variables? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send e

[google-appengine] Re: data structure size

I'm actually talking about python variables. Ints, Strings, Lists, Dicts, etc.. On May 26, 8:55 am, Brandon Donnelson wrote: > I guess the "data structure size" confused me. Usually that refers to stored > data. Then you said variable, which JDO has. > > Are you talking about class variables? -

Re: [google-appengine] Number of entities or size of indexes would affect searching speed?

Hi, The number of entities should have effectively no impact on query performance; the number of results returned by a query will of course impact performance (fetching more takes longer). I'm not sure what you're asking about index size, but the number of indexes you have will negatively impact

[google-appengine] App response times very poor with high pending_ms time

Hi, We are developing a Java app for Google App Engine. We did some testing today using "ab" to hit a particular URL. We had 1000 concurrent threads in ab hitting the URL 1 million times. We were surprised by the response times. A lot of the responses were taking well over a second to return with

[google-appengine] Indexing Issues

After 3 or 4 uninstall and new installations, my Google Desktop will not index. I have left my computer idle for days between each installation and tried every hint offered online with no result. I'm using Windows 7 and Office 2007. I would appreciate any new ideas. thanks -- You received th

[google-appengine] Re: Detect mobile phone

The easiest way I can think of is to check to 'User-Agent' header. When you get a connection from a web browser on an Android phone, "Android" will appear somewhere in the user agent header. See http://www.zytrax.com/tech/web/mobile_ids.html for a good list of mobile user agent headers. -- You r

[google-appengine] Can data deletion be made cheaper?

Sometimes, you need data deleted immediately so that it doesn't return in queries. Other times, the nature of your data doesn't have this constraint. Think of account related data: you can delete the parent account and orphan all of the child data and clean it up whenever. Synchronous data deleti

[google-appengine] Re: Poor performance since the past 2-3 days

I am seeing the same problems on my app. My request graph is extremely spiky and almost flat near the bottom in some parts. It seemed to clear up a little near midnight PST last night, but the problems are definitely back. My tasks are also backlogged around 25k tasks and not running. Please he

[google-appengine] Re: Can data deletion be made cheaper?

You could add a boolean "deleted" flag to your entities and have a task queue task or cron job very casually delete a few flagged entities at a time. You can also add a filter so that entities with deleted=True don't get returned in queries. Unfortunately this doesn't help you with the existin

[google-appengine] Re: Can data deletion be made cheaper?

That wouldn't make and difference to the overall cost. What I'm looking for is something deeper in the stack, provided by Google, that would make data deletion fundamentally cheaper. j On May 26, 11:43 am, Calvin wrote: > You could add a boolean "deleted" flag to your entities and have a task >

[google-appengine] Re: Can data deletion be made cheaper?

It would make a difference because you could do the deletions within the daily free quota. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this gro

[google-appengine] Federated login session times out after few hours

My java app is setup to use Federated Login with cookie expiration 1 Day. It works fine, however, after few hours (sometimes 1, sometimes 3) session is invalidated for some reason and user has to login again. It does not matter if user is being using the application all this time - authenticated

[google-appengine] Something up with cron and task services last couple days?

Since at least yesterday, all of my cron jobs have been failing with "on time - Failed" up to half the time they are [supposedly] run. This is on all my apps, some of which have been updated to 1.5 and others still running 1.4.X. >From my logs, it looks like the handler servlet (I'm using java) is

Re: [google-appengine] Memcache proper use?

On Thu, May 26, 2011 at 2:32 AM, Rohan Chandiramani wrote: > Hi, > So to clarify on the memcache principal, > High reads - low writes, memcache is a good option. > High reads - high writes, might as well only use the datastore? That's the vague principle, but without knowing more about your appli

Re: [google-appengine] Re: Can data deletion be made cheaper?

On Thu, May 26, 2011 at 10:58 AM, Calvin wrote: > It would make a difference because you could do the deletions within the > daily free quota. It's probably fair to assume that his application is already billed way beyond the free quota. Deleting data does seem painfully slow and expensive. I w

[google-appengine] Re: Where (geographically) do App Engine apps run?

Hi, Looking into GAE log gives your the time zone On May 25, 8:54 pm, Craig wrote: > I understand that GAE apps are deployed across multiple data centers, > but I would like a better idea of where those are for the purpose of > latency expectations for non-US traffic.  If everything is only

[google-appengine] Re: Tasks are not running from the last one hour!

Can someone please look into this. My tasks run intermittently. My users have been complaining :( -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from t

Re: [google-appengine] Re: Poor performance since the past 2-3 days

My tasks are not running either. Or they run once a while and then stop/ AppID: justunfollow On 26 May 2011 23:12, David wrote: > I am seeing the same problems on my app. My request graph is > extremely spiky and almost flat near the bottom in some parts. It > seemed to clear up a little near

[google-appengine] Re: What do you want to see answered in Greg's pricing FAQ?

On May 25, 1:33 pm, Emanuele Ziglioli wrote: > On May 26, 8:13 am, Mars wrote: > > > I understand that AlwaysOn is still under discussion, but I'd just > > like to point out that the old pricing model suggested that having an > > idle instance costs around 0.4c/hr (0.3 / 3 / 24) while in the new

[google-appengine] Re: Can data deletion be made cheaper?

There's free quota? ;) Yes, we're way beyond that. On May 26, 12:54 pm, Jeff Schnitzer wrote: > On Thu, May 26, 2011 at 10:58 AM, Calvin wrote: > > It would make a difference because you could do the deletions within the > > daily free quota. > > It's probably fair to assume that his applicati

Re: [google-appengine] Re: Can data deletion be made cheaper?

> It's probably fair to assume that his application is already billed > way beyond the free quota. > True, but assuming there are days when traffic decreases, like weekends and holidays, having the delete task running slowly forever could end up being (slightly) cheaper than doing it all in one

[google-appengine] Re: Can data deletion be made cheaper?

Jeff, good point on the new pricing model. From what I understand, we'd get charged on an API call (e.g., each call to db.delete()), regardless of the number of indexes (which make up a good portion of the overall CPU costs in our case). There likely is a fairly substantial difference (reduction)

[google-appengine] Admin Console Unresponsive

Did the Admin console just become really unresponsive for anyone else? -- Kyle Mulka Co-Founder, DealSavant http://www.dealsavant.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@goo

[google-appengine] Re: Admin Console Unresponsive

Yes. I can not see my MapReduce or custom admin pages specifically -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google

Re: [google-appengine] Memcache proper use?

Alright, let's give a concrete example. I'm still working on my game application. Each gamesession holds 4 players and each player updates the shared gamession entity once every 2 sec. Basically every time a player requests the gamesession entity, he/she needs the most up2date entity. I don't qu

Re: [google-appengine] Memcache proper use?

This is unlikely to work. You will need to transactionally update each game session entity every 0.5 second. You might get that kind of throughput, but with collisions you may not. The docs officially say don't expect more than 1 write per second per entity group. Memcache can flush at any mome

Re: [google-appengine] Re: Can data deletion be made cheaper?

Everything I've heard from the GAE team indicates that each index change would be a separate "datastore operation" - so if you write 100 entities with 3 indexes each, that's 300 operations. I'm just curious to know if a delete is the same as a write - my impression (unscientific) is that deletes u

Re: [google-appengine] Re: Can data deletion be made cheaper?

On Thu, May 26, 2011 at 12:22 PM, Calvin wrote: > >> It's probably fair to assume that his application is already billed >> way beyond the free quota. > > True, but assuming there are days when traffic decreases, like weekends and > holidays, having the delete task running slowly forever could end

[google-appengine] Re: Something up with cron and task services last couple days?

My M/S app has been affected by this the last few days as well. This cron failure stuff used to happen regularly in 2010. I was already beginning to think that those days are over now, but unfortunately that is not the case. I moved the more critical parts to task queue a long time ago and now it

[google-appengine] Re: What do you want to see answered in Greg's pricing FAQ?

> I think it's 3 instances for Always On when it was first released in > 1.4.0:http://groups.google.com/group/google-appengine/browse_thread/thread/... > > Not sure if it's still the case. yes, it's 3 instances, currently > > you sound like a kiwi :-) > > lol I am one indeed. he he, are you on

RE: [google-appengine] Memcache proper use?

If you are going to build what you are talking about, you are going to have to have 4 collector entities, and a writer entity, and do clean up. > Player Updates data Base <<><><><><> Get Player Number, Select bucket to write to, (any of the following) Collector Player 1: Turn 1 OBJECT

[google-appengine] Re: Can data deletion be made cheaper?

"my impression (unscientific) is that deletes up to this point are considerably more expensive than writes" I concur, also unscientifically. j On May 26, 2:22 pm, Jeff Schnitzer wrote: > Everything I've heard from the GAE team indicates that each index > change would be a separate "datastore op

Re: [google-appengine] HR migration - should I?

Yep - one entity group per user sounds good. You are probably not expecting more than 1 write/second per user. In practice the throughput is a bit higher, but you should *engineer* for 1 write/second. Robert's answer is dead on: if you write an entity group all at once in a single API call, that c

Re: [google-appengine] Re: Can data deletion be made cheaper?

If your app's traffic is spiky enough that you don't use all 6.5 free CPU hours 365 days a year, then on those days you'd get at least some deletes for free. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send emai

Re: RE: [google-appengine] Memcache proper use?

@Brandon That's a genius idea, i did not look at the architecture from such a perspective. But if it's expensive, the app will not be able to sustain itself in the end. @Jeff Understood. ... Perhaps in the end AppEngine is just not made for this kind of usage. I'l have to rethink my strategy

[google-appengine] Re: What do you want to see answered in Greg's pricing FAQ?

> I think it's 3 instances for Always On when it was first released in > 1.4.0:http://groups.google.com/group/google-appengine/browse_thread/thread/... > > Not sure if it's still the case. yes, it's 3 instances, currently > > you sound like a kiwi :-) > > lol I am one indeed. he he, are you on

[google-appengine] Task Speed during High Volume Periods

The question that I have concerns the speed of tasks running and completing when there are a large number of tasks in different task queues. Can the task load of one queue affect the performance of tasks in a different task queue? -- You received this message because you are subscribed to the

[google-appengine] Re: data structure size

On May 26, 10:11 am, JH wrote: > I'm actually talking about python variables.  Ints, Strings, Lists, > Dicts, etc.. I don't believe there was ever such a limit. The 1MB limit is on the API calls, which makes it impossible to have a datastore entity larger than 1MB (and, thus, impossible to have

[google-appengine] Re: Can data deletion be made cheaper?

On May 26, 2:54 pm, Jeff Schnitzer wrote: > Deleting data does seem painfully slow and expensive.  I wonder, does > this change at all with the new pricing model? I think this is really just the nature of he datastore, and possibly the underlying BigTable; it's very much optimized for reads rat

[google-appengine] Migrating away from App Engine but keeping Google Account integration

Hi guys, (this might not be the best place to ask this. Let me know if not) I've got an app currently running on App Engine that takes advantage of the close integration with Google Accounts so that it is the only way users sign in. The sign-in process was this: 1. User would click sign

Re: [google-appengine] Redirection problems after enabling OpenID

Clearing your browser cookies and / or restarting the browser often help resolve this issue. On Fri, May 20, 2011 at 19:11, Eric Kolotyluk wrote: > After enabling the OpenID Federated Authentication on my app, when I > go to my app (via Firefox) I get the following > > The page isn't redirec

[google-appengine] Re: Migrating away from App Engine but keeping Google Account integration

it's not that hard Google is openid provider use one of the libs http://openid.net/developers/libraries/ On May 26, 9:57 pm, Jordon Wii wrote: > Hi guys, > > (this might not be the best place to ask this.  Let me know if not) > >  I've got an app currently running on App Engine that takes advanta

Re: [google-appengine] The ways of uploading data to GAE datastore

Hey Dave, You can also upload the CSV to the blobstore then process it with your own code or using the mapreduce lib. Ikai has a blog post explaining the process. http://ikaisays.com/2010/08/11/using-the-app-engine-mapper-for-bulk-data-import/ I've used a similar approach several times

Re: [google-appengine] App response times very poor with high pending_ms time

Hi, Did you try ramping up your request ratea little more slowly? If your app has a long startup time (and it liekly does since it is Java) and services most requests very fast, then I've sometimes seen the scheduler be a bit too resistent to spin up new instances. In that situation I sometimes

Re: [google-appengine] Indexing Issues

I'm guessing you've sent this to the wrong group. This group is for Google App Engine. http://code.google.com/appengine/ On Thu, May 26, 2011 at 11:36, Dian S wrote: > After 3 or 4 uninstall and new installations, my Google Desktop will > not index.   I have left my computer idle for day

Re: [google-appengine] Federated login session times out after few hours

Hi Anatoliy, I was recently having exactly the same problem. It was extremely annoying and made the app basically unusable; the worst part was, as you said, even if you were *using* the app it would suddenly log you out. My solution: increase the cookie expiration time in the Application Set

Re: [google-appengine] Re: Tasks are not running from the last one hour!

I'd submit a production issue: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue On Thu, May 26, 2011 at 15:12, nischalshetty wrote: > Can someone please look into this. My tasks run intermittently. My users > have been complaining :( > > -- > You received th

Re: [google-appengine] Task Speed during High Volume Periods

Hi Dan, From what I've seen the queues run largely independent from each other. Of course, there are events that can impact your app as a whole. If you've got one queue that is pounding very hard on the datastore and it starts causing errors, that would naturally impact your other queues. Also

Re: [google-appengine] Re: Can data deletion be made cheaper?

Your interpretation of the operation count is in line with what I've heard as well. Although, it is sure nice to assume it refers to API calls. I think Greg actually posted something to the groups stating this. In the end, the new datastore pricing model is actually (exactly) the same as the cur

Re: [google-appengine] Memcache proper use?

I'd say memcache generally has a much higher value when you're very read heavy. But, one thing to consider is that even if you're writing to the datastore in most requests, memcache might can still save you time fetching entities from the datastore. However, as Jeff noted, keeping your memcache a

Re: [google-appengine] Re: Remove self from app permissions

You should submit an issue about this. That would be pretty annoying to not be able to get junk apps off your list. In the mean time, maybe submitting a billing support issue is the way to go? http://code.google.com/support/bin/request.py?contact_type=AppEngineBillingSupport Robert On Thu

Re: [google-appengine] Re: Tasks are not running from the last one hour!

@Robert It's back to normal now. Thanks for the link. Is that even for times when Task Queue does not work for a few hours (because that happens often). I thought the issue tracker was for reproducible bugs and feature request. This seems more like some temporary glitch in the service. On 27 May 2

Re: [google-appengine] Re: Tasks are not running from the last one hour!

Hey Nischal, The production issues template in the issue tracker is for problems in the production environment, like serving issues and index problems. Personally, I'd say your tasks not executing for hours at a time is a production issue. I guess if they close it and refer you back here we'll

Re: [google-appengine] Re: Tasks are not running from the last one hour!

Oh, then I'll definitely do that the next time my tasks stop. This wasn't the first time and I'm guessing it wouldn't be the last. We shall know soon ;) On 27 May 2011 10:49, Robert Kluin wrote: > Hey Nischal, > The production issues template in the issue tracker is for problems in > the produ

Re: [google-appengine] Simple query times out repeatedly for hours!

I had the same thought as Stephen about the tablet splitting, but that wouldn't last for hours and hours unless your adding new data at a very high rate durring that time. Also, I'd expect the datastore viewer to not work correctly if your in code queries were failing because of that. How do you

[google-appengine] Re: Something up with cron and task services last couple days?

I've been having problems with the Task Queue yesterday. Seems to have corrected now though. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this g