[google-appengine] Re: Does free quota still allow for "roughly 5 million pageviews a month for an efficient application"?

[hyperflame]

On Feb 27, 7:29 pm, Mark Ivey  wrote:
> I'm mostly worried about outgoing bandwidth, since that's seems like
> the hardest quota to stay within.

> Am I missing something? Is there a trick to this or is the FAQ just
> out of date?

Both actually. The FAQ is out of date and you could probably pull a
few tricks to lower your load times.

The basic question you need to ask is, what is in my website that
requires GAE? If it's just a simple blog (or can be translated to a
blog format) you can just move your site to WordPress, Blogger, or a
similar free blogging site. It's pretty much impossible to slashdot
them. Other than that, just try to offload as much as possible, for
example, jquery files can be loaded off other CDN services.

> I'd appreciate hints here too. For a site that is within the free
> quota except for the rare slashdotting, what's the best approach?

After the AppEngine billing change (and lowering of the free tier's
limits) I split my site into two parts (my site is an open source
project based on JSP). One part is hosted by GitHub Pages - entirely
static content such as documentation, examples, and so forth, and the
second, dynamic demonstration part is hosted by Google App Engine. You
may want to consider a similar arrangement.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Jul 18, 2:38 pm, Michael Hermus  wrote:
> I dont believe that you (or anyone) has sufficiently explained how sending 
> user requests to cold instances is ever better than warming it up first.. 
> Said request can ALWAYS be pulled right off the pending queue and sent to the 
> instance as soon as it is ready.

Let me take a shot at this. Brandon Wirtz touched on it before, when
he said "I agree that the Queue is sub optimal, but it is more sub
optimal the smaller you are.  When you get to 50 instances it is
amazing how well the load balancing works. "

Suppose you have a massive store (we'll call it MegaWalmart).
MegaWalmart has 100 staffed checkout lanes. Suppose all of these lanes
have a single queue of people lined up, and a supervisor which sends
customers to open checkout lanes (roughly analogous to your preferred
way of handling GAE request queuing). That one line would be huge,
would block traffic around the store, etc. It's far better, from
MegaWalmart's POV, to have multiple check out lines, one line per each
lane.

Now suppose you have another checkout lane open up. (remember, now we
have one line per lane) That's an additional 1% capacity. Now if the
checkout clerk is drunk/hungover/whatever, that additional lane will
take extra time to open, annoying the customers lined up in that lane.
>From MegaWalmart's POV, who cares? Less than 1% of your customers were
inconveniced. 99% of people still had a decent time checking out.

Let's apply this to the scheduler. Suppose there was one single queue
of requests. At Google-scale, that queue of requests could easily
exceed millions of entries, possibly billions. And God help you if the
machine hosting the queue gets a hiccup, or an outright failure. Don't
you agree that, at least at Google-scale, requests should immediately
be shunted to instance-level queues? Even if a single instance takes
forever, or fails, we don't have to care: such a failure would only
affect 0.01% of users.

This leads me to my final point: My understanding, from reading the
documentation and blog/news posts about GAE, is that the the core of
GAE is ripped pretty much directly from production Google services.
The problem with this is, the scheduler is intended to work at very
high scale, not at low scale. And frankly, this makes sense when you
consider a lot of the finer points of the GAE ecosystem.

So, to fix this: GAE needs to have a good relook at scheduler code,
and rewrite it so that it has two different rules for apps at less
than 50 instances, and more than 50 instances. Additionally, perhaps
the GAE should look at making the scheduler smarter; perhaps it could
measure the startup time of instances, and in the future, not send
requests to cold instances until that startup time has elapsed.

Personal thoughts: I have admined a corporate GAE app that has
exceeded 100 instances, and I use GAE for personal apps that use, at
max, 3-4 instances. When you use GAE at these two extremes, you really
get an understanding of how GAE scales. For instance, a personal
anecdote: for my low end apps, I occasionally notice that GAE starts
up a new idle instance. I'm not charged for it, it doesn't do any
work, but it is counted in the "current instances" counter. My guess
is that, during non-peak times, the GAE scheduler will load into
memory additional instances of low end apps, to try and be ready for
quick scaling.  So I believe the GAE team tries to handle low-end
instances, but it does need more work.

TLDR: the scheduler needs more work, and MegaWalmart is the same thing
as Google's scheduler.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Jul 18, 5:11 pm, Michael Hermus  wrote:
> If that is true, I suppose I would be quite surprised, for the following
> reasons:
>
> a) Google's entire infrastructure is designed for EVERYTHING to scale
> massively and still work well.

"Massively" being the key word there. Every Google service is massive.
Even abandoned or low use Google services (i.e. Wave, Buzz) are going
to use the equivalent of a minimum 10,000 F4 instances. The GAE
scheduler does inefficient scheduling at less than 50 instances. It
works wonderfully once you hit scale at 100+ machines(correct me if
I'm wrong, but most of the people seeing problems here are probably
less than 50 instances).

> b) By waiting for instances to warm up first, I don't think you would
> really increase the maximum depth of the pending queue by a whole lot.

I would have to disagree with you on this. My experience with big
websites tells me that requests can very quickly pile up if you're not
handling them expeditiously.

> c) I don't think the pending queue is 'hosted' on a single machine; I am
> pretty sure it relies on a resilient queue infrastructure designed to
> tolerate failures and scale well.

My analogy, like all analogies, breaks down if you apply it literally.
Even if you (hypothetically) built a datacenter with 100,000 machines
solely dedicated to hosting a single request queue, that datacenter
can still go down (earthquake, power, hurricane, etc). Far better to
simply dump requests into instance level queues and be done with it.

Just to be clear, I am agreeing with you in that the GAE scheduler
needs work; it is currently optimizing for high-scale apps, and not
apps that are using double-digit or lower instances.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Jul 18, 7:01 pm, Jeff Schnitzer  wrote:
> This is an interesting discussion.  How many apps do you think are on
> GAE that run 100+ instances?  1000+?  While these are certainly going
> to be the "important" apps for Google, I suspect they are fairly few
> in number.  With a couple exceptions, the owners are very quiet on
> this list.

As I noted before, I have admined a 100+ instance GAE app, and know a
few companies that have or currently do run at that scale. To be
completely honest, many of those (that I know personally) have
migrated to AWS or similar cloud firms. 8 cents per F1 instance hour
is a budget breaker; AWS is much cheaper and (this is far more
important to managers) has a track record of periodically lowering
prices.

Also, you might be surprised at who reads this list. I have seen
firsthand several executives and corporate strategists read this list,
or get forwarded excerpts by their engineers. They might be "very
quiet" but they are watching (cue Jaws the movie music... )

> "Less than 1% of your customers were inconveniced. 99% of people still
> had a decent time checking out." is not very comforting.  If 1% of
> Google searches took 8+ seconds to respond, the Googleplex alarms
> would deafen people as far away as Oakland.

Just as a purely practical issue, I routinely experience long load
times for certain Google services. Search is always fast, but Gmail
can take some time to load on my iPad: when i go to gmail.com, the url
box will bounce between several different urls before loading, at
least 6-8 seconds. Is there an "alarm" form that I can fill out? :-)

> I fear that GAE is optimized for single-threaded Python2.5 apps which
> take 2s to spin up, an environment that is becoming less relevant as
> time goes on.  Given an app that takes 20+ seconds to start, would a
> high-traffic app work any better than a low-traffic app?

My experience is in Java, so I have no reference to single threaded
Python 2.5 apps. With that said, the main problem is the scheduler.

Essentially, the GAE people have to rewrite the scheduler to reflect
the following rules:
1. For apps that use 50+ instances, use the current rules for
scheduling.
2. For apps that are fewer than 50 instances, change to the following
rules:
2A: Record the startup times for the past 20 instance startups.
2B: Instances that are started up do not receive requests until the
average of the startup times + 20% has elapsed.
2C: In nonpeak hours, idle instances are free or heavily  discounted
2D: During times when GAE is experiencing high latency, the scheduler
is more aggressive in spooling up more instances. Idle instances
spooled up during this time are cheaper (30% discount seems fair...)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Thursday, July 19, 2012 2:57:28 AM UTC-5, Richard Watson wrote:
>
>
> That's fine, but all we're asking for is "don't dump those requests into 
> an instance that isn't able to serve the request immediately".  If there 
> are three instances running and one being started up, don't consider the 
> latter in the population of instances to send requests to. Dump it into an 
> instance level queue that is already running. Everything else remains 
> constant.
>

Let's back up a minute here. To the GAE scheduler, as it is right now, 
there is no concept of "starting up." Either there is an instance, or there 
is no instance. A "cold" instance is just as good as an instance that has 
been active for hours. 

If you read Mr. Matsuo of the GAE team's posting, he wrote, "If you have an 
app with average +50s loading time, I totally understand that you strongly 
want to avoid sending requests to cold instances. On the other hand, there 
are also many well-behaved apps with <5 secs loading/warming time. Please 
understand that for those apps, it is still acceptable if we send requests 
to cold instances, so it's likely we can not prioritize the feature bellow 
over other things, however..."

I read that as (and correct me if I'm wrong) that the GAE scheduler expects 
cold instances to be ready in less than 5 seconds (notice the 
"well-behaved" remark). If that is so, there is no reason to categorize 
instances as being in "starting up" phase. An instance is an instance is an 
instance, regardless of whether or not it was just cold-started.

Let's go back to the MegaWalmart example. MegaWalmart is guaranteeing that 
if it opens a new checkout lane, that checkout lane will be open quickly, 
that the clerk will not be drunk, the scanner will be working, etc. Is that 
guarantee reasonable? That's the point of this thread.
 

On Thursday, July 19, 2012 2:57:28 AM UTC-5, Richard Watson wrote: 
> I stand to be corrected, but I doubt that Google searches are dumped onto 
> cold instances.
>

The GAE scheduler expects hosted apps to cold start in less than 5 seconds. 
If the GAE scheduler expects that of us, I don't see why Google wouldn't 
hold itself to that same commitment.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/nwb8Q4Bxx7cJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

I did some thinking about this problem, and it occurred to me that we
might be going at this the completely wrong way. Could we reverse-
bloom instance startups?

Here's the basic idea. If apps need to load in/process a bunch of
stuff at startup, why don't we give them the processing power to do
that, but only at the time of startup? Suppose we start up an
instance. For the first 30 seconds of that instance, we give it F16
ranking (that is, 4 times the power of a current F4 instance). For the
next 30 seconds, the instance gets F12 ranking (3 times F4 power). The
next 30 seconds, the instance gets F8 ranking (2 times F4 power).
After that, the instance gets lowered to F4 and remains there for the
rest of its lifespan.

For the first 90 seconds of the instance's lifespan, it gets
turbocharged, and should be able to handle all startup processes and a
few initial requests. The scheduler doesn't need to change; it can
continue feeding requests into cold instances, because even cold
instances will tear through their startup and handle the requests.

Even if this turboboosted time was billed at standard rate, it
wouldn't cost that much; F4 costs 32 cents per instance hour, so F16
would (theoretically) cost 4 times as much, so an F16 instance hour
would cost $1.28 per instance hour. We only need 90 seconds at that
rate, so 3.2 cents. I think paying 3 cents extra per instance startup
is very reasonable, especially if it prevents user-facing cold
instance startups.

It's entirely doable on a technical level too. Amazon AWS has a
similar system where Micro servers can temporarily boost to 2 compute
units, then get lowered to their allotted 1 ECU. Linux has the ability
built-in: see nice/renice http://en.wikipedia.org/wiki/Nice_(Unix)

I'd be interested in hearing if this system is possible within the GAE
hosting system.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

Discussing theoretical startup times is great, but I'd like to see
some real-world startups. Does anyone with high startup times ( say,
30+ seconds) want to share the results of a code profiler/appstats?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

I just looked over the logs of a corporate GAE application. It's a very 
simple "heartbeat" application; essentially our production applications 
have to send a message to it periodically. The message sent is about 500 
characters; the application does some checksum-ing and sends back a JSON 
string about 100 characters long. The only external libraries are jars of 
twitter4j and the org.json parser. It's really not much more complex than a 
Hello World app.

Looking over the logs, the fastest start time from cold-instance to 
first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5 
seconds to cold-start) on a F1 instance. Once the instance is warm, 
subsequent requests take, on average, ~150ms to process and send back a 
reply. I could probably get that lower if I upgraded to F4.

On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote:
>
> I use getRessourceAsStream an Stream-copy - but you miss the point...it's 
> a Hello World, done in 5 minutes as test for your post, stripping away all 
> excuses.
> The calls after startup are answered in <80 ms.
> That mans that I'm still far above 3 seconds for initializing of an empty 
> project. I could simply write Hello World into the Stream.
>
> I could use static...yes...but thats not the point. Some people have 
> dynamic and user dependand data on each page.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

Are you using a free or paid app to do your testing? My understanding
is that paid apps are treated preferentially by the scheduler.

On Jul 21, 3:49 pm, André Pankraz  wrote:
> May be it's my language barrier...don't know. Whats hard to understand in
> NewProject->HelloWorld-Servlet->3 seconds startup time->1,5 s isn't true
> and cannot be true
> If I take 500 ms away for the request because i'm so terribad it's still >3
> s startup time, for HelloWorld. Even if it's 2.5 s in a good timeslice,
> it's far off 1.5 and it has to load 1 (!) servlet without depencies

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Jul 21, 5:17 pm, "Drake"  wrote:
> unicorn land where you traded having nice friendly donkey's who will put up
> with what ever shit you feed them and still pull your wagon, for a flying
> unicorn that shoots rainbows out of its ass but requires that you don't just
> give it a sugar cube every now and then, but that it is raw organic sugar
> fed to it by beautiful naked virgins.  In exchange you can go farther,
> faster, and you don't have to clean up poo off the floor.  These are the
> trades you make for living in Unicorn land.

I like this visual. If someone puts it on a coffee cup or t-shirt as a
GAE promotion, I'd buy it. Seriously, someone do this.

On Jul 21, 9:21 pm, Prashant Hegde  wrote:
> Here is a counter example. We are a small app with peak traffic of 1 request 
> per second. We use java. No frameworks. We use jdo, guice. Startup time 20 
> seconds. Average request latency under 500 ms. We have been on appengine 
> close to 2 years. Right now we are able to serve our users with one instance 
> without exposing them to loading requests ( almost zero )

This isn't a counterexample, for the reason that you mentioned in the
first sentence: you can serve everything off one instance. The
original poster needs multiple instances, and to be able to scale as
load changes. If you're not loading new instances, then startup time
is pretty much irrelevant; you can start an instance once, then your
users will never see a cold instance. This entire discussion is about
cold instances.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Jul 21, 4:50 pm, Tomas  wrote:
> I use spring in work for other projects which runs on full java stack (but
> thats just Tomcat/Jetty) and when I run the "new" librarist equivalent code
> on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be
> wrong here a little bit), the startup is 2-3 secs and I can serve all my
> traffic with it with load at max 0.5.
>
> When I run the similar code on GAE, I get 45+ secs startup...
>
> Okay thats cool, I can understand the  GAE is different and basically it
> could be used just for small projects where you write everything by
> yourself and don't use any normal stuff like Dependency Injection /
> Proxying and similar. But my biggest question for GAE guys would be this -
> why the hell the startup takes so long simply I don't get what is actually
> the issue here:
>
> - is it loading of the jars from the network (virtual drive) when the gae
> loads each class as separate file
> - is it the scanning of the classes (annotations etc)
> - is it java reflections
> - is it creating of proxies

One (small) reason why GAE is slower is just that instances are
smaller than comparable AWS instances. An AWS Micro is 613mb RAM and
can temporarily use 2 ECU of CPU (looking at Amazon docs, they claim
that 1 ECU is equal to a 1 ghz 2007 opteron). A GAE F1 is 128 mb of
RAM and 600mhz.

I bet that the major reason is just network I/O, for the GAE servers
to find an available server, transfer a copy of the application
+libraries to that server, and start up the servlet runner. It would
explain why even simple apps have long startup times: it's not that
the frameworks and libraries are taking a long time, it's just the
overhead of moving all those libraries into place initially.

This also suggests a simple solution to the problem: just integrate a
copy of commonly used libraries (spring, guice, twitter4j, etc) into
the servlet runner.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Jul 22, 5:12 am, Aleksei Rovenski 
wrote:
> Regarding min idle instances, I must admit there is something strange
> going on in my Instance tab in Dashboard.
> I have settings like this: idle instances min=auto, max=1, pending
> queue min and max=15s. I have some working instances that I get
> charged for and then few more that are anyway created by google just
> waiting out there for hours and even days and I do not seem to pay for
> them. If I kill them, they are coming back anyway. This looks nice
> indeed, but could someone explain me what is behind that. I'm worried
> because it may be a drug that google will suddenly stop providing
> me :)

I get that as well. My theory is that at non-peak times, the Google
scheduler opens up extra instances to be ready for spikes.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Monday, July 23, 2012 10:38:12 AM UTC-5, Paul v wrote:
>
> Well I love the who's-a-better-guru argument.  I've learned a ton reading 
> this thread, I hope they keep going at it.


Just out of sheer curiosity, what exactly have you learned from reading 
this read? Really, the entire discussion between Mr. Wirtz and Mr. 
Schnitzer distills down to optimizing your code as much as possible. Which 
is a really good idea regardless of what hosting platform you pick. 

And frankly, this discussion hinges entirely on instance cold start issues. 
If you have enough money to keep spare idle instances sitting around, or 
your traffic is steady and constant (as opposed to spiky and unpredictable) 
you really won't hit this problem very much.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/V4-frxZVYSEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Frameworks on GAE

[hyperflame]

+1 to this reply.

Frameworks are OK, sometimes. The real question that you have to ask 
yourself is, what are you using from the framework, and does it justify 
copying the entire framework library in. Sometimes you absolutely need the 
framework, but you only need a portion of it. Fine, tear out the parts you 
don't need, and leave the rest in. 

Other times, you're using the framework for services that App Engine 
already provides. A lot of people will point to authentication and security 
for reasons why they need a framework. While I understand that, it's also 
important to understand that using AppEngine's authentication capabilities 
are frequently faster and (depending on your application) simpler. App 
Engine can authenticate through Google Accounts (pretty much everybody has 
at least a Gmail account) or Federated authentication via OpenID. I have an 
apps that authenticate through one or the other service, and it only 
requires a few lines of code.



On Sunday, July 22, 2012 4:43:25 AM UTC-5, Brandon Wirtz wrote:
>
> In an argument we often polarize beyond what is entirely true…
>
>  
>
> Libraries… More than Frameworks are “OK” 
> Some Frameworks are closer to libraries, and some Libraries are closer to 
> frameworks. 
>
*From:* google-appengine@googlegroups.com [mailto:
> google-appengine@googlegroups.com] *On Behalf Of *glimmung
>
> *Sent:* Sunday, July 22, 2012 2:33 AM
> *To:* google-appengine@googlegroups.com
> *Subject:* [google-appengine] Frameworks on GAE
>
>  
>
> Hi All,
>
>  
>
> I've been reading, initially with amusement but more recently with 
> concern, the "dialogue" (for want of a better word) between Brandom Wirtz 
> and Jeff Schnitzer re. startup time/optimisation. Brandom has now made 
> the following very strong statement: "NO FRAMEWORKS. NONE. Deal with it." 
>
> This leads me to ask the Google team for their position on this: Is it 
> your position that GAE is an unsuitable platform for framework-driven apps?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/VJ_eFuwZ_OUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Google App Engine, rogue crawlers, and PageSpeed Insights

[hyperflame]

Alternately, you could institute a rate limiting mechanism. If a user
asks for more than X pages over a specified time period, serve up a
HTTP 429 error code (stands for Too Many Requests). Legitimate bots
such as GoogleBot will slow down their requests, while poorly-written
bots will most likely fail.

You could also put a trap into your robots.txt. List a url in your
robots.txt that goes to a servlet, if a user hits that servlet, their
IP is banned for some amount of time.

On Jul 26, 7:47 pm, jswap  wrote:
> Thanks, Jeff, but how do I block requests by header and not by IP?  I
> usually use iptables to block the requests, but cannot do so in this
> situation because then I block access to Google's PageSpeed Insights tool
> too.
>
>
>
> On Thursday, July 26, 2012 5:27:27 PM UTC-4, Jeff Schnitzer wrote:
>
> > Every fetch request from GAE includes the appid as a header... you
> > obviously see it yourself, which is how you know the appid of the
> > crawler.  This is how Google enables you to block applications; just
> > block all requests with that particular header.
>
> > Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Differentiated service with Google App Engine

[hyperflame]

Correct me if I'm on the wrong track, but I thought two applications
could share the same datastore via the Remote API:
https://developers.google.com/appengine/docs/java/tools/remoteapi


On Jul 26, 2:11 pm, rerngvit yanggratoke  wrote:
> The only way I could think of is creating two
> separated applications and letting the premium users access the one that
> enables billing, while free users access the one that doesn't. However, my
> free and premium users need to share the datastore because they are also
> have to communicate within the application. Unfortunately, it seems that
> sharing a datastore across applications is not possible at the moment (feel
> free to correct me if I am wrong).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: My site bombarded with thousands of URL requests from different IPs - how to stop them

[hyperflame]

In the interim, it would be a good idea to put up a static page
stating that your website is under emergency maintenance or similar,
just so your customers have some information.

On Aug 1, 6:01 pm, Joshua Smith  wrote:
> http://code.google.com/p/googleappengine/issues/list
>
> New Issue
>
> Production Issue template
>
> Follow the instructions in the template
>
> On Aug 1, 2012, at 6:56 PM, Kate  wrote:
>
>
>
> > Thanks Joshue,
> > How do I file a production issue?
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To view this discussion on the web 
> > visithttps://groups.google.com/d/msg/google-appengine/-/xPmxxyiQfPsJ.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to solve the aggregate query with condition in Google App Engine

[hyperflame]

Well, you could just use Google Cloud SQL if you really want a SQL
environment.

Otherwise ( this is just a quick example, obviously there are some
optimizations you could do). Also i'm typing this out on my phone, so
forgive me if I miss a semicolon or something.

To add a string to the datastore:
Entity record = new Entity("record");
record.setProperty("some_input", input_string);
record.setProperty("add_date", new Date());
datastore.put(record);

To search for strings in a given time period:
Query q = new Query("record");
Q.setFilter("add_date", FilterOperator.LESS_THAN,
some_date_that_you're_searching_for);
PreparedQuery pq = datastore.prepare(q);

And then you can pull out the top x entities out of pq as a List, then
extract their properties to find the string.



On Aug 1, 1:29 am, Neo  wrote:
> Suppose, In my website, I ask users to input some string. A user can input
> string multiple times. Whenever any user inputs a string, I log it in the
> database along with the time. Many strings can be same, even though
> inputted by different users. In the home page, I need to give the interface
> such that any user can query for top n (say 50) strings in any time period
> (say 10 Jan 2012 to 30 Jan 2012). If it was SQL, I could have written query
> like:
>
> select string, count(*)
> from userStrings where day >= d1 and day <= d2
> group by string
> order by count(*) desc
> limit n
>
> How do I solve it in GAE environment? For each such user query, I can't
> process the record at query time - there can be millions of records.
>
> I am using JDO. My obvious goal is to minimize the app engine cost : CPU +
> data.
>
> Thanks,

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How can I block curl requests

[hyperflame]

Return a HTTP 429 error code ( stands for too many requests ). See
https://developers.google.com/appengine/docs/python/tools/webapp/redirects

On Aug 1, 8:58 pm, Kate  wrote:
> Yes Brandon, the user agent is the same, but how do I issue the denied 
> message?
>
> Thanks,
>
> Kate

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to move data reliably from front to backends ?

[hyperflame]

Just to be clear, the GAE datastore returns stale results ONLY at a
specific time, and only when load is low? Is it because of both of
those factors together or just a single one? Seems odd that GAE is
reacting slowly just when load is the lowest. Can you post some load
graphs?

If the staleness is due to low load, you could always run some fake
users to add to the load, then have your backend subtract those fake
users when it builds the scoreboards. It wouldn't be hard; you can
capture some real users data, save it to a log, then replay it against
production servers.

On Aug 1, 1:27 pm, Richard  wrote:
> Summary so far:
>
> "Sometimes" (at the same time of day.. when load is lowest),  GAE will do a
> put() and 5-10 seconds later when I do a query all to create leaderboards
> for that game round, I get stale results. for some of the entries.
> This is because the put() has completed, but not the (internal) commit().

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to move data reliably from front to backends ?

[hyperflame]

It's just odd that GAE is having trouble not when load is high, but
when load is low. I doubt tablet issues are the cause here. Can you
run some fake load against the datastore? Just build a B1 that adds
some fake scores, and have your tabulator backend delete those fake
scores when it builds the leaderboard.

Also, i'm wondering why it just started 5 days ago. Did your load
profile change? Try this: change your instances to F2, then back to
F1. Perhaps your resident instance got moved to a faulty server 5 days
ago; this will force GAE to open a new instance on a new server.

On Aug 2, 9:50 am, Richard  wrote:
> Kinda.  By 'low load', I mean 150-200 simultaneous users.  Oh... and it
> never used to have this problem.  This started 5 days ago.
>
>
>
> On Thursday, August 2, 2012 12:19:51 AM UTC-4, hyperflame wrote:
>
> > Just to be clear, the GAE datastore returns stale results ONLY at a
> > specific time, and only when load is low? Is it because of both of
> > those factors together or just a single one? Seems odd that GAE is
> > reacting slowly just when load is the lowest. Can you post some load
> > graphs?
>
> > If the staleness is due to low load, you could always run some fake
> > users to add to the load, then have your backend subtract those fake
> > users when it builds the scoreboards. It wouldn't be hard; you can
> > capture some real users data, save it to a log, then replay it against
> > production servers.
>
> > On Aug 1, 1:27 pm, Richard  wrote:
> > > Summary so far:
>
> > > "Sometimes" (at the same time of day.. when load is lowest),  GAE will
> > do a
> > > put() and 5-10 seconds later when I do a query all to create
> > leaderboards
> > > for that game round, I get stale results. for some of the entries.
> > > This is because the put() has completed, but not the (internal)
> > commit().

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How can I block curl requests

[hyperflame]

You're trying to find curl requests by looking for a "win" string?
IIRC, curl uses "curl" as its default user-agent, you have to look for
that. Also, it's a bad idea to look for "Win", as legitimate requests
(users using WINdows) will be blocked.

Also, it should be noted (and I believe a number of people have
already mentioned) that curl allows a person to change their user-
agent; just looking for a "curl" user-agent may not stop the problem.

On Aug 2, 1:55 pm, Kate  wrote:
> Well I tried this by testing user agent but it passes the test and the page
> loads correctly, which it shouldn't in this example.
> Am I meant to pass 'app' as such? I am not sure of this parameter.
>
> Here is my code
> from webob import Response
>
> class AntiCurlMiddleware(object):
>     def __init__(self, app):
>         self.app = app
>
>     def __call__(self, environ, start_response):
>         ua = os.environ.get('HTTP_USER_AGENT', "unknown")
> sindex = string.find(ua,'Win',0)
> if sindex > 0:
>       resp = Response('Too many requests!')
>       resp.status_code = 423
>       return resp(environ, start_response)
>         return self.app(environ, start_response)
>
> def webapp_add_wsgi_middleware(app):
>     return AntiCurlMiddlewarey(app)
>
>
>
>
>
>
>
> On Wednesday, August 1, 2012 4:43:58 PM UTC-4, Kate wrote:
>
> > I am getting tens of thousands of curl requests - many thousands time
> > browser requests and I want to block them. I'm using python. They coming
> > from many different IPs most in Europe.  If I can't stop them I will have
> > to close my site or go to a new provider.
>
> > Thank in advance,
>
> > Kate

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: trying to send HTTP ERROR 429 for a certain user agent

[hyperflame]

Did you spell this correctly:

>     return AntiCurlMiddlewarey(app)

There's an extra "y" there. Looks like the real function is named
AntiCurlMiddleware without the y.

On Aug 2, 2:53 pm, Kate  wrote:
> How can I trap a certain user agent and send a 429 response.
>
> I have the following in my appengine_config.py and it always defaults to a
> 200.
>
> (Note I do not intend to block all Windows user agent requests bit have the
> code "sindex = string.find(ua,'Win',0)" there for testing only).
>
> from webob import Response
> import os
>
> class AntiCurlMiddleware(object):
>     def __init__(self, app):
>         self.app = app
>
>     def __call__(self, environ, start_response):
>         ua = os.environ.get('HTTP_USER_AGENT', "unknown")
> sindex = string.find(ua,'Win',0)
>
> if sindex > 0:
>       resp = Response('Too many requests!')
>       resp.status_code =429
>       return resp(environ, start_response)
>         else:
>            return self.app(environ, start_response)
>
> def webapp_add_wsgi_middleware(app):
>     return AntiCurlMiddlewarey(app)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to move data reliably from front to backends ?

[hyperflame]

Richard,

I did some testing overnight, and I have some good news, and some bad
news.

Good news, I can give you a system that stores 1,000 users and scores
in roughly 1 second. In less than a second, I can pull out all 1,000
scores, sort the scores numerically, and print out the score list.
Bad news: It depends on memcache.

Details: Last night, I wrote an application to generate 1000 users and
1000 scores randomly, and store them in memcache. On average, this
operation takes roughly 1 - 1.3 seconds, although I suspect the
slowness is due to the random number generator, not the memcache. I'll
test this more.

Then a task is enqueued, to call another F1 instance in three seconds.

The next instance pulls out all 1000 scores, sorts them using a
treemap, and prints out the sorted data into GAE logging in less than
a second. Then, the memcache is cleared for the next iteration of the
test, so we don't get old data.

A cron job repeats this test every 2 minutes.

Here is my memcache viewer screen: http://i.imgur.com/oypAm.png . As
you can see, this service ran overnight, and didn't drop a single user/
score. Over 330,000 scores were posted and accessed in total. Is this
good enough performance for your game?



On Aug 3, 11:19 am, Richard  wrote:
> Thanks Alex, VERY much appreciated, since I can't test this myself without
> buying a shell account somewhere.
>
> Luckily, the backend crashed due to being unable to reuse the connection
> for the delete.  So I added some exception handling :)
>
> Can I ask some more people to try this link:  
> http://sven-anagramhero.appspot.com/client/loadtest
>
> Please ping it once from a web browser just before you hit it.  This will
> ensure the DB is up :)
>
> I would like to see results for loads of n > 1000 with c => 500.
>
> The server clears out results every 3 minutes (synchronized to NTP time) on
> the minute boundary, so please try to avoid doing it exactly on that
> boundary (in which case the results will be spread and it makes it more
> difficult to ensure we did not 'lose' any).
>
> NOTE:  It seems we can store at least 1k users within 10 seconds . I
> really don't like the 6.8 second response (I would prefer 300 msec).
> viable ?  y/n ?
>
> Thanks !
>
> -R
>
>
>
>
>
>
>
> On Friday, August 3, 2012 10:02:09 AM UTC-4, alex wrote:
>
> > From Rackspace (London):
>
> > ab -n 1000 -c 200  http://sven-anagramhero.appspot.com/client/loadtest
> > This is ApacheBench, Version 2.3 <$Revision: 655654 $>
>
> > Server Software:        Google
> > Server Hostname:        sven-anagramhero.appspot.com
> > Server Port:            80
>
> > Document Path:          /client/loadtest
> > Document Length:        2 bytes
>
> > Concurrency Level:      200
> > Time taken for tests:   15.694 seconds
> > Complete requests:      1000
> > Failed requests:        0
> > Write errors:           0
> > Total transferred:      171000 bytes
> > HTML transferred:       2000 bytes
> > Requests per second:    63.72 [#/sec] (mean)
> > Time per request:       3138.712 [ms] (mean)
> > Time per request:       15.694 [ms] (mean, across all concurrent requests)
> > Transfer rate:          10.64 [Kbytes/sec] received
>
> > Connection Times (ms)
> >               min  mean[+/-sd] median   max
> > Connect:        8    8   1.5      8      22
> > Processing:   139 2827 1197.6   2910    8487
> > Waiting:      139 2827 1197.6   2910    8487
> > Total:        147 2835 1197.6   2918    8494
>
> > Percentage of the requests served within a certain time (ms)
> >   50%   2918
> >   66%   3341
> >   75%   3620
> >   80%   3874
> >   90%   4257
> >   95%   4700
> >   98%   5900
> >   99%   6131
> >  100%   8494 (longest request)
>
> > ab -n 1000 -c 500  http://sven-anagramhero.appspot.com/client/loadtest
> > This is ApacheBench, Version 2.3 <$Revision: 655654 $>
>
> > Server Software:        Google
> > Server Hostname:        sven-anagramhero.appspot.com
> > Server Port:            80
>
> > Document Path:          /client/loadtest
> > Document Length:        2 bytes
>
> > Concurrency Level:      500
> > Time taken for tests:   6.879 seconds
> > Complete requests:      1000
> > Failed requests:        0
> > Write errors:           0
> > Total transferred:      171000 bytes
> > HTML transferred:       2000 bytes
> > Requests per second:    145.37 [#/sec] (mean)
> > Time per request:       3439.463 [ms] (mean)
> > Time per request:       6.879 [ms] (mean, across all concurrent requests)
> > Transfer rate:          24.28 [Kbytes/sec] received
>
> > Connection Times (ms)
> >               min  mean[+/-sd] median   max
> > Connect:        8   17   9.0     11      28
> > Processing:   144 2210 1535.1   1885    6831
> > Waiting:      144 2210 1535.2   1885    6831
> > Total:        152 2227 1539.7   1894    6853
>
> > Percentage of the requests served within a certain time (ms)
> >   50%   1894
> >   66%   2410
> >   75%   3100
> >   80%   3225
> >   90%   4492
> >   95%   5628
> >   98%   6418
> >   99%   6484
> >  100%  

[google-appengine] Re: How to move data reliably from front to backends ?

[hyperflame]

On Aug 3, 1:13 pm, Richard  wrote:
> Sounds interesting. but how do you handle write contention to the
> memcache datastorage structure from multiple F1's serving client side score
> submissions ?

I'm sure it could be done, I have some ideas regarding that (perhaps
vary the key structure depending on the instance/user?) but I really
don't want to pay the cost of multiple F1s, B1s, etc to test my
theory. I might mock up something on my local dev server if I have
time over the weekend, but I don't know how memcache works on the
local development eclipse plugin.

On Aug 3, 1:13 pm, Richard  wrote:
> Also, I thought memcache had a size limit ?  I store a lot more than just
> username + score (including a full stream of all actions the user takes in
> the UI to prevent cheating).

How much do you store? My general rule of thumb is that I depend on
memcache to store 1 GB of data before it starts force-expiring objects
(this is for enterprise-level, paid apps). I'm trying to Google around
for some documentation regarding the memcache limit, but it seems that
there is very little documentation regarding memcache. Frankly, I
think shooting for a 100 MB self-imposed-limit should be fine. This is
something you really should ask Takashi.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to move data reliably from front to backends ?

[hyperflame]

I'm assuming you need storage space to log past user actions so you
can prevent cheating, correct? If so, couldn't you just log, say, the
past 5 (or some relatively small number) of actions and check those
for cheating?

It's difficult to talk hypothetically about these issues without a
diagram or flowchart of what is actually happening in your
application.

On Aug 3, 1:38 pm, Richard  wrote:
> Sorry, I should have been more explicit.
>
> I thought memcache had a size limit on a single object (1MB).  Now imagine
> I have 2000 people submitting data for a game.  i don't think I will be
> fitting all that into 1MB.  Which means I need to store multiple objects
> and fan out/fan in results into memory from memcache (assuming I solve the
> write contention problem... WITHOUT making clients timeout waiting for a
> write lock!).
>
>
>
> On Friday, August 3, 2012 2:29:20 PM UTC-4, hyperflame wrote:
>
> > On Aug 3, 1:13 pm, Richard  wrote:
> > > Sounds interesting. but how do you handle write contention to the
> > > memcache datastorage structure from multiple F1's serving client side
> > score
> > > submissions ?
>
> > I'm sure it could be done, I have some ideas regarding that (perhaps
> > vary the key structure depending on the instance/user?) but I really
> > don't want to pay the cost of multiple F1s, B1s, etc to test my
> > theory. I might mock up something on my local dev server if I have
> > time over the weekend, but I don't know how memcache works on the
> > local development eclipse plugin.
>
> > On Aug 3, 1:13 pm, Richard  wrote:
> > > Also, I thought memcache had a size limit ?  I store a lot more than
> > just
> > > username + score (including a full stream of all actions the user takes
> > in
> > > the UI to prevent cheating).
>
> > How much do you store? My general rule of thumb is that I depend on
> > memcache to store 1 GB of data before it starts force-expiring objects
> > (this is for enterprise-level, paid apps). I'm trying to Google around
> > for some documentation regarding the memcache limit, but it seems that
> > there is very little documentation regarding memcache. Frankly, I
> > think shooting for a 100 MB self-imposed-limit should be fine. This is
> > something you really should ask Takashi.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to move data reliably from front to backends ?

[hyperflame]

Congrats!

Let's see some graphs afterwards, I'd be interested in seeing how
Cloud SQL holds up.

On Aug 3, 2:34 pm, Richard  wrote:
> Just moved the scoring over to CloudSQL . and got Featured on Google
> Play Store 30 min ago.
>
> Let's PRAY that Cloud SQL saves our ass or else I am screwed.
>
>
>
> On Friday, August 3, 2012 2:38:32 PM UTC-4, Richard wrote:
>
> > Sorry, I should have been more explicit.
>
> > I thought memcache had a size limit on a single object (1MB).  Now imagine
> > I have 2000 people submitting data for a game.  i don't think I will be
> > fitting all that into 1MB.  Which means I need to store multiple objects
> > and fan out/fan in results into memory from memcache (assuming I solve the
> > write contention problem... WITHOUT making clients timeout waiting for a
> > write lock!).
>
> > On Friday, August 3, 2012 2:29:20 PM UTC-4, hyperflame wrote:
>
> >> On Aug 3, 1:13 pm, Richard  wrote:
> >> > Sounds interesting. but how do you handle write contention to the
> >> > memcache datastorage structure from multiple F1's serving client side
> >> score
> >> > submissions ?
>
> >> I'm sure it could be done, I have some ideas regarding that (perhaps
> >> vary the key structure depending on the instance/user?) but I really
> >> don't want to pay the cost of multiple F1s, B1s, etc to test my
> >> theory. I might mock up something on my local dev server if I have
> >> time over the weekend, but I don't know how memcache works on the
> >> local development eclipse plugin.
>
> >> On Aug 3, 1:13 pm, Richard  wrote:
> >> > Also, I thought memcache had a size limit ?  I store a lot more than
> >> just
> >> > username + score (including a full stream of all actions the user takes
> >> in
> >> > the UI to prevent cheating).
>
> >> How much do you store? My general rule of thumb is that I depend on
> >> memcache to store 1 GB of data before it starts force-expiring objects
> >> (this is for enterprise-level, paid apps). I'm trying to Google around
> >> for some documentation regarding the memcache limit, but it seems that
> >> there is very little documentation regarding memcache. Frankly, I
> >> think shooting for a 100 MB self-imposed-limit should be fine. This is
> >> something you really should ask Takashi.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: What are the pros and cons of using Google App engine for my startup?

[hyperflame]

It would be a good fit for most of those subjects, but keep in mind
that GAE cannot access email through IMAP or POP3 because it doesn't
support sockets, only standard url fetch. If you want to include email
in your service, you'll have to base a proxy somewhere else, such as
Rackspace, Heroku/AWS, etc.

On Aug 3, 4:37 am, Levi Campbell  wrote:
> I'm building a startup, and I'm considering GAE as the platform, however
> I've been having a hard time finding information on why a startup might
> consider GAE instead of the many cloud providers out there. Let me explain
> what I'm working on.
>
> I'm a big fan of David Allen's Getting Things Done: The Art of Stress-Free
> Productivity,
> and after trying several tools and online services that (claim to)
> implement the GTD methodology, I couldn't find anything that I loved, so I
> decided to build my own and make it available as a SaaS offering. This app
> will allow users to pull in their info_crap from email, facebook, twitter
> (and yes, I do have plans to add support for more social networks.), and
> RSS feeds and organize it by relationship to the sender (i.e. family,
> work colleague, vendor, and the like), project (i.e. planning a family
> vacation.), and context (Either the when or where something should
> happen.).\
>
> Would GAE be a good fit for the application I'm developing? Why?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Google app engine using Python- Emails having the same instance not being delivered.

[hyperflame]

Using the same instance is fine, the second email was most likely spam-
filtered (emails from GAE get incorrectly caught in spam filters a
lot). You're better off using a email service like Amazon SES.

On Aug 4, 11:18 am, Tribblehunter  wrote:
> Hi.
>
> Using app engine free quota to forward information in an email which is
> sent from android app.
>
> Information is parsed in app engine using python then mailed from google
> app engine.
>
> At times emails have not arrived at the recipient. I took a long look at
> the last time this happened and found that the emails had the same
> instance. The first one (sent from 1 user) was recieved, but the second
> (sent from another user 30 mins later) didnt.
>
> Can someone confirm that having the same instance is correct/incorrect? all
> the other emails have different instances.
>
> This is a critical part of the application and without it working everytime
> the package i have created is as useful as a rubber hammer for breaking
> rocks
>
> Thanks in advance for any help.
>
> Brian

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to solve the aggregate query with condition in Google App Engine

[hyperflame]

Instead of counting the number of times a string has appeared at
querytime, just change to counting the number of times the string has
been input at input-time.

E.g. When your user puts in a string, search the datastore to see if
that string already appears in an entity. If so, set the count
property on that entity +1. If not, then add a new entity and set the
count property to 1. Then at query time, you can sort by the count
property, in addition to the date limitations.

On Aug 5, 6:20 am, Neo  wrote:
> Hi Hyperflame. Thanks for your reply. The problem is, between any given
> time period, there can be millions of such records. For each of such query,
> I will have to process those millions of records to get the top x entities.
> This is definitely very inefficient and not acceptable.
>
> -Neo
>
>
>
> On Thursday, August 2, 2012 7:12:12 AM UTC+5:30, hyperflame wrote:
>
> > Well, you could just use Google Cloud SQL if you really want a SQL
> > environment.
>
> > Otherwise ( this is just a quick example, obviously there are some
> > optimizations you could do). Also i'm typing this out on my phone, so
> > forgive me if I miss a semicolon or something.
>
> > To add a string to the datastore:
> > Entity record = new Entity("record");
> > record.setProperty("some_input", input_string);
> > record.setProperty("add_date", new Date());
> > datastore.put(record);
>
> > To search for strings in a given time period:
> > Query q = new Query("record");
> > Q.setFilter("add_date", FilterOperator.LESS_THAN,
> > some_date_that_you're_searching_for);
> > PreparedQuery pq = datastore.prepare(q);
>
> > And then you can pull out the top x entities out of pq as a List, then
> > extract their properties to find the string.
>
> > On Aug 1, 1:29 am, Neo  wrote:
> > > Suppose, In my website, I ask users to input some string. A user can
> > input
> > > string multiple times. Whenever any user inputs a string, I log it in
> > the
> > > database along with the time. Many strings can be same, even though
> > > inputted by different users. In the home page, I need to give the
> > interface
> > > such that any user can query for top n (say 50) strings in any time
> > period
> > > (say 10 Jan 2012 to 30 Jan 2012). If it was SQL, I could have written
> > query
> > > like:
>
> > > select string, count(*)
> > > from userStrings where day >= d1 and day <= d2
> > > group by string
> > > order by count(*) desc
> > > limit n
>
> > > How do I solve it in GAE environment? For each such user query, I can't
> > > process the record at query time - there can be millions of records.
>
> > > I am using JDO. My obvious goal is to minimize the app engine cost : CPU
> > +
> > > data.
>
> > > Thanks,

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Under DDoS attack

[hyperflame]

Try Cloudflare ( they have a free plan, check out 
https://www.cloudflare.com/plans
). They should protect you from some of the DDOS.

On Aug 6, 10:04 am, Mahron  wrote:
> Dynamic blacklisting would be nice. Some kind of service with an
> optimized ip blocker.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Unique Integer ID for a non primary key field for Entities in Google App Engine

[hyperflame]

This sounds a lot like a url sharing service like tinyurl, etc.
Perhaps investigate how they create their shortened urls.

Could you use String.hashcode()? I haven't worked with it before, but
it might generate the ints that you need.

On Aug 1, 1:32 am, Neo wrote:
> I have an Entity type say URLInfo which keeps info about URLs. The primary
> key of this entity is URL itself ( that makes sure that I always have
> unique URLs in the datastore). I also want unique integer id for each url
> so that sharing the id becomes easier. Though, I can use GUIDs, but that is
> not a preferred thing for me. How can I achieve this requirement? Integer
> Ids need not be sequential ( preferred, if they are). Entities can be
> generated at a faster rate (that means I can't keep a common counter to
> update each time I generate a new URL record). This is what I have tried so
> far - In the URLInfo class, I defined a field - Long Id and annotate it
> with @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) in the hope
> that it will get automatically generated with the unique value. But when I
> save the new entity (with id set as null), it saves the entity in the
> datastore but doesn't assign any value to this field.
>
> I am trying all this on a local machine. I am using Java/JDO.
>
> Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How to solve the aggregate query with condition in Google App Engine

[hyperflame]

On Aug 6, 1:14 am, Steve James  wrote:
> I'd say horses for courses on this one.
>
> IMHO the Datastore may not be the best fit to solving your requirement.
>
> Perhaps Cloud SQL or BigQuery might be better suited?


Have to agree with Steve, this is increasingly sounding like a storage
schema that the datastore is not appropriate for.


On Aug 6, 4:49 am, Joakim  wrote:
> I'd solve this by having one unique entity per combination of date and
> string, storing the string's total for that day in said entity. You can
> achieve this uniqueness either by formatting special entity names (string
> ids) to include both the date and the string. Aggregate the day's totals in
> an entity keyed on the date, holding a map/dictionary of string->count.
> Your query would become something like this:
> SELECT * FROM DayCount WHERE __key__ >= Key('DayCount', startDate) AND
> __key__ <= Key('DayCount', endDate)

Nice one, Joakim!

Just to turn this into code:

To insert an Entity into the datastore (this is not production-quality
code, just showing an example):

try {
Key key = KeyFactory.createKey("someentityname", unique_string +
CURRENT_MONTH + CURRENT_DAY + CURRENT_YEAR);
Entity entity = datastore.get(key);
//If that operation succeeded, then there is an entity already with
that unique string, and was added today. Increase the count by one.
int count = (Integer)entity.getProperty("count");
entity.setProperty("count", count + 1);
entity.setProperty("add_date", new Date());
datastore.put(entity);
}
catch (EntityNotFoundException e) {
//this means that the unique string has not been added so far today
Entity entity = new Entity("someentityname", unique_string +
CURRENT_MONTH + CURRENT_DAY + CURRENT_YEAR);
entity.setProperty("count", 1);
entity.setProperty("add_date", new Date());
datastore.put(entity);
}

And then at query time, you can do a standard search using the Query
class. Of course, if your app is being accessed a lot, you need to
shard, or do some memcaching of data.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Uploading to google.com account on appengine.

[hyperflame]

Daniel,

I usually get that problem when I'm not signed into Google (within the
Eclipse GAE plugin) or when the application id hasn't been created/is
incorrect. Maybe double check that your application id is spelled
correctly, and that there is an available, registered appid in GAE.

Also, you do realize that this is a public list, not an internal
Google mailing list, correct? :-)

On Aug 5, 5:37 am, iDaniel  wrote:
> I have the same problem. Any clues how to fix it?
>
> Thanks,
>
> Daniel
>
> On Monday, December 26, 2011 6:08:51 AM UTC+1, aruncg wrote:
>
> > Hi,
>
> > I am not able to upload a very basic app to appengine.  I just created
> > a web application project in Eclipse.  I then set the appId to be one
> > that is hosted on googleplex.com.  I then tried to upload the app, but
> > I get the following error:
> > --- 
> > --- 
> > --- 
> > --
>
> > Unable to update app: Error posting to URL:
>
> >https://appengine.google.com/api/appversion/create?app_id=temp-aruncg...
> > 404 Not Found
> > This application does not exist (app_id=u'temp-aruncg-1').
>
> > See the deployment console for more details
> > Unable to update app: Error posting to URL:
>
> >https://appengine.google.com/api/appversion/create?app_id=temp-aruncg...
> > 404 Not Found
> > This application does not exist (app_id=u'temp-aruncg-1').
> > --- 
> > --- 
> > --- 
> > --
>
> > Any clues?
> > Arun.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Wildcard in web.xml creates infinite directories?

[hyperflame]

If crawlers are requesting deals/deals/deal-name instead of deals/deal-
name, that means that someone is linking to your site ( or you're
linking to your own site ) with the first method. You might want to
double check that your internal links, and any links that you're
promoting, are correctly written. It's easy to accidentally do this,
especially when you're using relative links.

If you can't find anything wrong, then you can write some code into
your Deal servlet to correctly route the request. Use getRequestURI
( 
http://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html#getRequestURI()
) to get the path the request used. If you find "deals" twice in the
path, you can issue a 301 Moved Permanently, then redirect the client
to deals/deal-name appropriately. Crawlers won't penalize you for a
301.

On Aug 7, 8:04 pm, Phil McDonnell  wrote:
> Apparently, I am creating infinite sub-directories from a web crawler's
> perspective due to a wildcard I have in my web.xml.  I have a single jsp
> file that dynamically handles all requests to the directory
> mysite.com/deals/*. The goal here was for the jsp to handle
> mysite.com/deals/some-deal-name-here.  However, crawlers apparently are
> also picking up on mysite.com/deals/deals/some-deal-name-herewhich
> unfortunately looks like duplicate content from a web crawler
> perspective.  This is not good for search ranking. Is there a way to
> configure my web.xml in order to stop this from happening?
>
> *Here's what I have in my web.xml:*
> 
>         deal
>         */deals/**
> 
> 
>     deal
>     /deal.jsp
> 
>
> *Here's what I tried to fix it:*
> 
>         deal
>         */deals/[a-z0-9_A-Z-]**
> 
> 
>      deal
>      /deal.jsp
> 
>
> Any ideas for other ways to potentially configure this?
>
> Thanks,
> Phil

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Backend performance, compared

[hyperflame]

On Aug 13, 4:43 pm, Hernan Liendo  wrote:
> MemCache.getAll(keys) method could work if you know the keys. Using sharded
> keys you could reach them all.

I wrote a simple test application for memcache earlier. I managed to
collect 1k keys reliably for my simple test application, and I would
bet that 10k keys would be no problem. The problem is the amount of
data in the value. If you just wanted to store scores, that would be
fine, but IIRC the original poster needed to store a lot more data
than just scores.

On Aug 13, 4:43 pm, Hernan Liendo  wrote:
> On Monday, August 13, 2012 3:10:17 AM UTC-3, Jeff Schnitzer wrote:
> > The problem is pretty easily stated:  Collect 10k score submissions in
> > 5s and be able to provide a sorted leaderboard 5s later.  GAE does not
> > offer any practical facility capable of accomplishing this.
>
> Yeah, that's right :(

Personally, I'd like to see memcache made more reliable, and more
guarantees added to it (and I would have no problem paying more).
Right now memcache is "Well, it's there, but it could go out at any
time."

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: logs are not showing in the appengine console

[hyperflame]

On Aug 16, 12:40 am, Jeff Schnitzer  wrote:
> Also:  I am unable to expand log msgs by clicking the '+'.  It
> registers my click and changes the icon to '-' but doesn't actually
> expand the logs - I still can't see an important stacktrace.  Probably
> related:  it chops off the length of these un-expanded lines too.
>
> Jeff

+1

I started experiencing this problem yesterday. I thought it was
because of malfunctioning browser addons, it's good to see that it's
not just me.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Next Time you think the scheduler is Broken...

[hyperflame]

Perhaps the 1st, 3rd, and 4th checkout clerks were drinking on the
job? Complain to Google's HR department :-)




On Aug 18, 3:23 am, "Drake"  wrote:
> I'm pretty sure the most efficient check out system is not to have one
> cashier handle 21k customers in 8 hours and the register that was open 15
> minutes sooner handle only 25.
>
> Or to have a 4th lane open for 57 requests.
>
> No combination of this was optimal for the 15 minute billing increment.
> Even if those 25 requests are 14 min 59 sec apart, 25 /4 is only 6 hours and
> change.
>
> From: google-appengine@googlegroups.com
> [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Friday, August 17, 2012 11:58 PM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Next Time you think the scheduler is
> Broken...
>
> What are you trying to say here?
>
> Jeff
>
> On Sat, Aug 18, 2012 at 2:08 AM, Drake  wrote:
>
> So the next time you think the Check out line analogy is flawed.. I think it
> is accurate but I'm not sure the scheduler does what we all expect..
>
> This is about 100 Gigs worth of xfer in 70k Requests a day.
>
> 0.000
>
> 0.0 ms
>
> 57
>
> 0
>
> 1:30:37
>
> 32.4 MBytes
>
>   c
> 61b117c8ba9543aab3ecfc850baacd6dd4dcf&filter_type=regex&date_type=now&limit =
> 20&view=Search> View Logs
>
> Description: Dynamic IconDynamic
>
> 1.683
>
> 21.2 ms
>
> 21023
>
> 0
>
> 7:50:49
>
> 51.0 MBytes
>
>   c
> 61b117cceee87a5c8b3c26bfdef9393196486&filter_type=regex&date_type=now&limit =
> 20&view=Search> View Logs
>
> Description: Dynamic IconDynamic
>
> 0.000
>
> 0.0 ms
>
> 266
>
> 0
>
> 8:04:53
>
> 31.6 MBytes
>
>   c
> 61b117c5937b082ee32869e0155dd6dd48ffb&filter_type=regex&date_type=now&limit =
> 20&view=Search> View Logs
>
> Description: Dynamic IconDynamic
>
> 0.000
>
> 0.0 ms
>
> 25
>
> 0
>
> 8:11:32
>
> 31.0 MBytes
>
>   c
> 61b117c320e688017ba4d38f95222d9430596&filter_type=regex&date_type=now&limit =
> 20&view=Search> View Logs
>
> Description: Dynamic IconDynamic
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
>  .
> For more options, visit this group 
> athttp://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group 
> athttp://groups.google.com/group/google-appengine?hl=en.
>
>  image001.png
> 1KViewDownload

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: joining the App Engine G+ hangouts

[hyperflame]

+1 me.

On Sep 7, 7:30 pm, Amy Unruh  wrote:
> hi all,
>
> We (the App Engine Developer Relations team) are putting together a list of
> developers who are interested in joining our App Engine Office Hours and
> other hangout events. (That is, interested in joining the actual hangouts,
> rather than just viewing the broadcasts).
> If you're interested, you can +1 or comment on this G+ post:
>
> https://plus.google.com/117105793163182226623/posts/725CiYA3381
>
> .. or reply to this thread. Or, if you prefer, you can send me your email
> address directly.
>
>  -Amy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

[hyperflame]

Christina,

Let me take a wild guess at what happened internally. A few months
ago, there was a 100+ post thread on this mailing list complaining
about long instance startup times (specifically, that classloading was
slow). You guys had an internal discussion, and produced some code to
streamline classloading, and it's failing because the streamlined
classloading process made some assumptions that don't hold up in
production.

Let me make an educated guess at why the failure is happening. Jeff
Schnitzer reported that the base exception was
"java.lang.ClassNotFoundException: java.io.FileOutputStream". But GAE
shouldn't need to load FileOutputStream, since the GAE JRE whitelist
(located at https://developers.google.com/appengine/docs/java/jrewhitelist
) doesn't even include FileOutputStream. There is no legitimate reason
to load or make available FileOutputStream if we're not allowed to use
it. So why is GAE attempting to load a non-whitelisted class?

My guess is that GAE deployed with the wrong copy of the SDK; perhaps
there was a reference to FileOutputStream in one of the internal GAE
classes, and everything went kaput when the servers failed to find the
reference. Or perhaps there was a last minute change by a programmer
who forgot that FileOutputStream wasn't available in GAE.

Am I close or completely wrong?

On Sep 12, 4:56 pm, Christina Ilvento  wrote:
> *Hi All,
>
> Beginning yesterday, September 11, Google App Engine experienced two
> periods of serving degradation for a subset of Java applications due to a
> gradual roll-out of a new version of the Java runtime. Affected
> applications would have seen errors related to class loading. We have
> resolved the first issue by fixing the underlying bug. We are still
> investigating the cause of the second issue but have rolled back the
> problematic update and all affected applications should now be returned to
> normal serving behavior.
>
> No changes to your code or application configuration are needed at this
> time. We apologize for any inconvenience this issue has caused, and we’ll
> follow up with more details on the underlying incident and resolution soon.
>
> Regards,
> Christina Ilvento, App Engine PM*
>
>
>
> On Wed, Sep 12, 2012 at 1:34 PM, Kaan Soral  wrote:
> > I wonder what happened, subscribed to this topic to be updated, hope
> > someone explains what happened and additionally hope this never happens to
> > me (python) :)
>
> > On Wednesday, September 12, 2012 11:00:49 PM UTC+3, Jeff Schnitzer wrote:
>
> >> We are back up and running now after 2 hrs of downtime.
>
> >> To whoever fixed it:  THANK YOU!!!
>
> >> To whoever broke it in the first place:  SPANKINGS!!!
>
> >> Jeff
>
> >> On Wed, Sep 12, 2012 at 12:11 PM, Jeff Schnitzer 
> >> wrote:
> >> > More information:
>
> >> >  * The failure began at 10:54am (pacific).
> >> >  * Same app on different appid has the same problem.
>
> >> > (as listed in the stacktraces, the appid is voost0)
>
> >> > Jeff
>
> >> > On Wed, Sep 12, 2012 at 11:53 AM, Jeff Schnitzer 
> >> wrote:
> >> >> HELP!
>
> >> >> We have tried everything at this point.  Shut down instances, tried to
> >> >> deploy a new version, even tried old versions.  We've reported a
> >> >> production issue.  Something is broken inside of GAE.  The Guice error
> >> >> must be a symptom; the smoking gun seems to be:
>
> >> >> java.lang.**ClassNotFoundException: java.io.FileOutputStream
>
> >> >> We've been down for 30 mins and getting complaints from our clients.
> >> >> This looks really, really bad.  It's my nightmare scenario - an outage
> >> >> in GAE that is small enough not to raise major alarm bells, but
> >> >> nevertheless cripples my business.  It is not the first time this has
> >> >> happened.  It is shaking my faith in GAE.
>
> >> >> Jeff
>
> >> >> On Wed, Sep 12, 2012 at 11:11 AM, Jeff Schnitzer 
> >> wrote:
> >> >>> Our app has been running fine on the same version, no code deploys
> >> >>> (our versions are timestamped so this is 100% certain), for two days.
> >> >>> All of a sudden (10 mins ago) our app stopped running.  Every attempt
> >> >>> to run a request produces this cryptic message:
>
> >> >>> 
>
> >> >>> 2012-09-12 10:56:21.791
> >> >>> com.google.inject.servlet.**GuiceFilter setPipeline: Multiple
> >> Servlet
> >> >>> injectors detected. This is a warning indicating that you have more
> >> >>> than one GuiceFilter running in your web application. If this is
> >> >>> deliberate, you may safely ignore this message. If this is NOT
> >> >>> deliberate however, your application may not work as expected.
> >> >>> D 2012-09-12 10:56:21.792
> >> >>> st.voo.tick.util.cambridge.**CambridgeSetup : Establishing
> >> >>> cambridge view resolver
> >> >>> I 2012-09-12 10:56:21.792
> >> >>> st.voo.tick.GuiceConfig contextInitialized: Guice initialization took
> >> 514 millis
> >> >>> W 2012-09-12 10:56:21.888
> >> >>> Failed startup of context
> >> >>> com.google.apphosting.utils.**jetty.**RuntimeAppEngin

[google-appengine] [Feature Request] Friendlier Error Pages

[hyperflame]

Would it be possible for GAE to display user-friendly error pages to
users?

Right now, if a hosted app fails to execute (for whatever reason) GAE
sends out a HTTP 500 error. This error page consists of the following
words: "Server Error: The server encountered an error and could not
complete your request. If the problem persists, please report (http://
code.google.com/appengine/community.html) your problem and mention
this error message and the query that caused it."

While developers understand that message, it is not user-friendly, and
doesn't report the correct action for the user to take. If my
organization's app fails, I want the user to report the failure
directly to my organization, not to a general Google mailing list.

What I'd like to see is a more user-friendly error page; perhaps you
could write something along the lines of "We're sorry, but your
request has encountered a problem. Please email us (email address
here) to let us know." And then put the app owner's email address (or
Twitter/Facebook/etc account name) as a clickable link.

Or you could allow us to specify a static page to serve up whenever a
failure occurs. Amazon S3 does something similar, where you can
specify a static page for 404 errors. Or even just allow us to put in
a redirect url; if a 500 error occurs, GAE could reply with a Moved
Temporarily code and redirect to the redirect url.

While we all hope and aim to build web sites with no downtime, errors
do happen occasionally, and this would be a huge help in the UI/UX
department.

Thanks for your time.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Friendlier Error Pages

[hyperflame]

Oops. Clearly I'm didn't see that section. Thanks a bunch all.

On Sep 18, 9:05 am, Simon Knott  wrote:
> Doesn't this functionality already exist?
>
> https://developers.google.com/appengine/docs/java/config/appconfig#Cu...
>
> https://developers.google.com/appengine/docs/python/config/appconfig#...
>
> Cheers,
> Simon
>
>
>
> On Tuesday, 18 September 2012 14:47:36 UTC+1, hyperflame wrote:
>
> > Would it be possible for GAE to display user-friendly error pages to
> > users?
>
> > Right now, if a hosted app fails to execute (for whatever reason) GAE
> > sends out a HTTP 500 error. This error page consists of the following
> > words: "Server Error: The server encountered an error and could not
> > complete your request. If the problem persists, please report (http://
> > code.google.com/appengine/community.html) your problem and mention
> > this error message and the query that caused it."
>
> > While developers understand that message, it is not user-friendly, and
> > doesn't report the correct action for the user to take. If my
> > organization's app fails, I want the user to report the failure
> > directly to my organization, not to a general Google mailing list.
>
> > What I'd like to see is a more user-friendly error page; perhaps you
> > could write something along the lines of "We're sorry, but your
> > request has encountered a problem. Please email us (email address
> > here) to let us know." And then put the app owner's email address (or
> > Twitter/Facebook/etc account name) as a clickable link.
>
> > Or you could allow us to specify a static page to serve up whenever a
> > failure occurs. Amazon S3 does something similar, where you can
> > specify a static page for 404 errors. Or even just allow us to put in
> > a redirect url; if a 500 error occurs, GAE could reply with a Moved
> > Temporarily code and redirect to the redirect url.
>
> > While we all hope and aim to build web sites with no downtime, errors
> > do happen occasionally, and this would be a huge help in the UI/UX
> > department.
>
> > Thanks for your time.- Hide quoted text -
>
> - Show quoted text -

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Receiving Mail - DKIM Signature

[hyperflame]

Does GAE verify DKIM signatures on incoming mail? In other words, if a 
@gmail.com account sends my app mail, and a DKIM header appears on that 
email, can I count on that email actually being from gmail.com?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/z6tHIfoSzsIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Update on our GAE Experience

[hyperflame]

On Friday, November 23, 2012 4:40:07 PM UTC-6, Kristopher Giesing wrote:
>
> I haven't heard a credible refutation of anything Brandon wrote in the 
> original post.  My own (admittedly limited) experience is very consistent 
> with Brandon's description.


+1, to you and to Brandon's list.

I'll add to Brandon's list: the bug tracker. There are issues there that 
have been outstanding for literally years, and there is no clear roadmap of 
when issues will be fixed. Quick examples (that I'm picking because I'd 
really like to see them fixed):

Issue 2314 ( http://code.google.com/p/googleappengine/issues/detail?id=2314 
): Making inbound email work for custom domains. This shouldn't be too 
hard, custom domains already get hosted Gmail, why can't App Engine access 
the inbound mail services of Gmail? Yet this issue was filed in 2009 and 
acknowledged by Google engineers in 2010. It's almost 2013, and no 
followup. I'd like some details on this; will this be fixed soon, or do I 
need to buy a subscription to context.io/other external mail services and 
use their incoming mail parsing services?

Issue 2145 ( http://code.google.com/p/googleappengine/issues/detail?id=2145 
): Same idea, make custom domains work via the XMPP service. Filed in 2009, 
acknowledged in 2010. No roadmap for implementation. This is pretty much 
required for using XMPP professionally; Jabber IDs with @appspot.com simply 
don't look professional. 

Issue 739 ( http://code.google.com/p/googleappengine/issues/detail?id=739 
): URLFetch operations don't follow no-caching headers. This issue has been 
marked Fixed, but I still get cached and stale urlfetches even after 
setting cache control headers, setUseCaches(), etc. I'm not the only one 
having problems, just check the issue.

Issue 1741 ( http://code.google.com/p/googleappengine/issues/detail?id=1741 
): The ability to remove a named task from a queue. Filed in 2009, accepted 
in 2010, absolutely no feedback since then. More fine grained control over 
task queues in general would be nice.


Leaving issues open for years is insane, give us a roadmap to when you'll 
fix these, or close them as Won'tFix. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/SCXgXmZivIMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Would you rather have 1GB or 26GB of RAM?

[hyperflame]

To be fair, you need to compare equivalent machines. n1-highmem-4-d has 
1770 GB of local hard disk storage, while a B8 doesn't even have local 
storage, it has to write to Google Cloud, or to the datastore which are 
billed separately.

The best comparison is to the n1-highmem-4 instance, which has no local 
storage either. That costs $0.508 an hour (even cheaper!) compared to a 
B8's $0.64 an hour.

So basically, 26GB of RAM in Compute Engine is 13 cents an hour cheaper 
than 1GB of RAM in AppEngine. Something feels off here



On Monday, November 26, 2012 12:44:33 PM UTC-6, pdknsk wrote:
>
> Google adds new instances to Google Compute Engine. 
>
> n1-highmem-4-d26GB $0.636 
>
> https://cloud.google.com/pricing/compute-engine 
>
> B8 1024MB $0.64 
>
> https://developers.google.com/appengine/docs/python/backends/overview 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/tDh5rutQ-FAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] GAE Java Upload failures

[hyperflame]

Please explain how a link to the Java "Getting Started" guide in any way 
addresses OP's question about SLA uptime.

On Friday, November 30, 2012 3:21:09 PM UTC-6, Shilendra Sharma wrote:
>
> Hello Saurabh
>
> It can be helpful to you 
>
> https://developers.google.com/appengine/docs/java/gettingstarted/installing
>
> Thanks,
> Shilendra 
>
>
>
>
>
> On Sat, Dec 1, 2012 at 12:14 AM, Saurabh Gupta 
> 
> > wrote:
>
>> Hi all,
>>
>> Anyone else seeing failures in uploading to the Java runtime (2012-11-30 
>> 13:40)? The system status dashboard shows a spike in latency for the Java 
>> runtime. I've been unable to deploy my app for the past hour and its 
>> incredibly frustrating: queues had failed a few weeks ago, and now the Java 
>> runtime is having issues as a whole.
>>
>> How is this consistent with the GAE SLA for paid apps?
>>
>> - Saurabh
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msg/google-appengine/-/Jbyc689awj8J.
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> To unsubscribe from this group, send email to 
>> google-appengi...@googlegroups.com .
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/zNqhbZqqvS4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: What does a good downtime report look like?

[hyperflame]

Another interesting downtime report is the Microsoft Azure downtime
from leap day earlier this year:
http://blogs.msdn.com/b/windowsazure/archive/2012/03/09/summary-of-windows-azure-service-disruption-on-feb-29th-2012.aspx

Just a fascinating read, and shows how just a few problems can spiral
into a disaster,

On Dec 5, 2:15 pm, Jon Stevens  wrote:
> https://github.com/blog/1346-network-problems-last-friday
>
> When a community asks for information about what is going on when downtime
> occurs (especially with AppEngine), I think that it helps to provide an
> example of what a good report would look like. After reading this story,
> I'm really impressed with the level of detail and transparency that GitHub
> is sharing. In no way did they expose any top secret information, but they
> certainly made me feel like they've gone above and beyond what they needed
> to and that is really appreciated.
>
> I think that over the last year or two, the AppEngine staff has really
> improved their transparency and reporting of issues. Thank you, it hasn't
> gone unnoticed! That said, it can always get better, which is why I think
> sharing this link is important.
>
> best,
>
> jon

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Console issues?

[hyperflame]

On Tuesday, January 15, 2013 11:58:11 AM UTC-6, Janusz Skonieczny wrote: 
>
> The issue may be wider. My application started to show default error and 
> timeout error static pages (those setup in app.yamls error_handlers)
>
 
+1.
 
Seeing a massive spike in latency on my end. Servlets that normally handle 
requests in 0.5 seconds are taking 4+ seconds to complete. Instance startup 
times are up 30%. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/fysAMTX5NdQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Twitter Streaming API with Tweepy on GAE

[hyperflame]

Twitter Streams don't work with Google App Engine because GAE doesn't 
support sockets. 
 
What you can do is set up a cron to periodically search Twitter for 
relevant tweets. Look up the documentation for since_id; you can tell 
Twitter API search to only retrieve tweets after a specific tweet, so you 
can process just new tweets and not repeatedly process old tweets.

On Saturday, January 26, 2013 3:08:09 AM UTC-6, Spencer Lim wrote:

>
> Hi, Can somebody please guide me on how could i consume Twitter Live 
> Stream through GAE? 
> http://stackoverflow.com/questions/14495868/twitter-streaming-api-with-tweepy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] GAE Instance Startup Similarities

[hyperflame]

In this mailing list, there has been a lot of discussion on instance 
startup times, request queues, etc on the Java runtime (to note a few: 
https://groups.google.com/forum/#!msg/google-appengine/sA3o-PTAckc/T2eA64xZ1m0J
  https://groups.google.com/forum/#!topic/google-appengine/Y6i9BYQG0Ws ).

A YCombinator News discussion popped up yesterday discussing many of the 
same topics on Heroku, and it's very interesting reading for those 
performance tuners out there: *Heroku's Ugly Secret: The story of how the 
cloud-king turned its back on Rails 
http://news.ycombinator.com/item?id=5215884  *
*
*
The TLDR of the discussion is basically: long request times aren't a GAE/J 
only problem, Heroku on Rails is having some of the same difficulties.

-hyperflame

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.