[google-appengine] calling out the app engine team on ssl for custom domains

[Amir More]

Hi

(I apologize, it's a long one).

I'm calling out the app engine team on how they've handled issue 
792
 - 
ssl/https support on google apps domains, and would like a straightforward 
answer to the current status of this issue.

Before I get into how the team has dealt with the issue and my problem with 
it, I'd like to make a few observations:
1. The engineers on the GAE team have known about the lack of SNI on IE in 
XP for quite some time (see comment 
#48 - 
quoting an IRC session).  We can safely regard August 2009 as a point of 
time at which GAE engineers knew about this problem.
2. The issue was market as Started in October 2010.  The GAE team and/or 
it's decision makers have clearly chosen to work on this issue.  This is 
non-trivial given the success of GAE which, as usual, has lead to a queue of 
dev tasks far larger than the capacity of the team.  In other words, this 
issue is decidedly important enough to work on that dev resources are 
prioritized to work on it.
3. In the last two Google I/O events, the issue of SSL on google apps / 
custom domains has been prominently displayed as a feature which will be 
available 'soon', whether as part of GAE for Business (available by EOY 
2010) or as a feature that's on deck for general availability.
4. Despite the issue of SNI and IE/XP, there appear to be a handful of 
possible solutions to overcome the limitations it presents.  More on this 
later.
5. Other features which have been taken on by the team and were expected to 
take a relatively long time to bring to fruition have been declared as such. 
 For example, if we look at the Full-Text Search issue, it was marked as 
Started back in October 2009, but with the following text:
"I'm marking this issue as "Started" but I want to set expectations 
appropriately: 
This is a major undertaking and this will not happen soon, even by the most 
generous definition of soon."

Given these observations - it is understandable and acceptable that there 
are factors beyond the control of the team; namely that the lack of support 
for SNI in IE on Windows XP presents a technical obstacle.  On the other 
hand, the recurring assurances that this issue is being worked on, and in 
some cases promised to land about a year ago, leads one to believe that 
regardless of the SNI problem the feature will be generally available in the 
near term future.

Case in point: "SSL access on custom domains" has been on the 
roadmap since 
May 2010, updated around the time of I/O 2010.  Given the heading "..Most of 
these features here are intended to be launched within the following six 
months. .." it should come as no surprise that a sizable number of 
developers are quite irate over the fact that we are now well into September 
2011 and yet no sign of SSL for custom domains, for better or for worse.
At no less than a keynote speech of I/O 
2010, 
given in front of thousands of developers, Kevin Gibbs says in regards to 
App Engine for Business (@113:12): "..That means we're putting our money 
where our mouth is when it comes to the reliability of your business.  And 
finally, it also comes with two additional key features that we had heard 
almost every enterprise needs: SSL for your company's domain and SQL 
databases.  That's right.  You heard me right."
On the other hand, a full year later, at a fireside chat from I/O 
2011 the 
SSL issue comes up a few times and at some point Peter Mckenzie, who claims 
responsibility for SSL, says (@26:35):
"We are actively working on it and making good progress.  We can't give a 
date, but it's certainly high priority.  To be honest, we would have liked 
to have launched it by now, but it's certainly foremost in my thoughts, and 
in the thoughts of several others of my team."
.. immediately followed by a moment of candor by Brett Slatkin, who jokingly 
suggests we get everyone to install a browser with SNI and/or move to IPv6.

I'm calling out the app engine over what has happened over the course of 2-3 
years on this issue.  First, the issue is accepted in 2009 and marked as 
'started', despite the known SNI problem.  Next, in 2010 it's added to the 
roadmap, announced in a keynote and given an approximate ETA of EOY 2010. 
 Yet a year later, all of a sudden they don't have a date and are throwing 
around remarks about how they would have liked to launch it by now.

The duplicitous and shady behavior on the issue of SSL on custom domains is, 
in googley terms, a form of evil.

On a personal note: You'll have to take my word for it that I had attended 
I/O 2011 and brought up the issue at App Engine offi

Re: [google-appengine] calling out the app engine team on ssl for custom domains

[Amir More]

Hi Brandon,

Regarding your resolution of setting up a squid/proxy that forwards requests 
to https://*.appspot.com somewhere:
Having all your traffic go through a single server on AWS or other VPS to 
proxy requests for google app engine is equivalent to buying a top of the 
line super-car, ripping out it's four-digit horse power engine and strapping 
it to a moped with the goal of using said turbo-charged moped for a road 
trip from new york to california, all because the super-car is too wide for 
some of the roads you might encounter.  It might work but you're doing it 
wrong.

Let's try to go with the flow: I'll see your "proxy/squid set up somewhere" 
and raise you a multinational dns provider with name servers on most 
continents with geolocation rules to forward dns requests to the CNAME of a 
multi-zone load balancer at the nearest AWS region out of 5 worldwide; each 
region's ELB configured to forward to auto-scaling health-checked linux 
instances balanced across all availability zones, each instance running a 
modified linux kernel optimized for the sole purpose of being a web proxy, 
running either varnish or nginx as a front end, using a SPDY library for SSL 
requests to app engine (yeah, the app engine front end has SPDY - GAE team 
didn't even know about it for a while).  It would be best to have the AWS 
ELB use SSL with SPDY to talk to the instances but we'll let that slide.

Yet, still, this solution is sub par considering:
1. There's hardly any point to serving static files from GAE, might as well 
serve them from your proxy.  Yay, more locations to push to :) Or you could 
have your proxy be a literal cache and time out files but then you'd have to 
either push a cache flush or use cache busters for your content.  IMO both 
reek of an equally vile stench.
2. It's likely the Channel API won't work with all those middle men.  If the 
GAE team starts using websockets (or regular tcp/ip sockets while we're at 
it) we'd be back to the drawing board.  Also various other APIs might be 
negatively affected or require some tinkering (blobstore upload, XMPP and 
task queue API)
3. Consistently added latency in the best case - you have the added latency 
of initiating an SSL handshake with the proxy.  Given a solution with a load 
balancer you'd have two handshakes, and depending on how they're set up 
either one or both of them will be full SSL handshakes (not just TCP 
handshakes)
4. Additional highly varying latency in the worst case - what happens when 
the GAE or production teams move the instances serving GAE requests to 
another data center?  Your users would see a sudden jump in latency since 
the proxy they're working with takes longer to reach the GAE domains/ips 
it's been working with.  It'll take a while for new geolocation rules at the 
dns level meant to counter the effect to kick in due to dns caching.
5. It's expensive.  The cheapest single server solution is more expensive 
than the GAE $9/app/month under the new pricing model.  The full-blown AWS 
solution is somewhere around $1k a month.  Not including the labor and 
maintenance required for the custom linux image.

There are probably more issues but those are the best I could think of off 
the top of my head.

I think there is no reason to believe that internal politics at google are 
holding this issue back.  Also I can't find a reason to believe that google 
is an organization set up such that the GAE team and/or their 
product/project managers need to give financial incentives to other teams 
just to get a major feature out the door.  I'm not saying there aren't 
internal politics because there probably are.  I'm just saying that a 
feature of this type seems like something that, at worst, would be 
arbitrated quickly by a manager high enough on the corporate ladder to see 
the detrimental effect of not finishing this feature a full year after it 
was announced at a major company event.

Regarding 2014 - I agree.. in that MS and their products will be synonymous 
with myspace by then to developers.  But you can't deny the size of MS and 
their existing install base.  EOL for XP doesn't necessarily mean that 
bureaucratic organizations like government offices will have upgraded by 
2014.

IPv6.. the jury's still out on that one.

The question remains - are we waiting for 2014 or whichever decade will 
bring IPv6 / end of IE on XP or is there something in the works?
I don't take issue with which decision the GAE team made, but with the fact 
that they're zigzagging on the availability of the feature.

Amir

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/9Rs5wZemfCEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://

[google-appengine] Re: Appstats traceback's need some love

[Amir More]

your can enlarge the depth of the appstats traceback by adding an 
appengine_config.py file at the root of your project
add the following parameter
appstats_MAX_STACK = 

On Thursday, April 12, 2012 6:56:55 PM UTC+3, Jason Collins wrote:
>
> We used to have really helpful tracebacks in appstats that pointed to 
> our actual source. 
>
> Somewhere along the way, we lost them, leaving us with things like 
> this: 
>
> Stack: 
>   /google/appengine/datastore/datastore_rpc.py:1165 
> make_rpc_call() 
>   /google/appengine/datastore/datastore_query.py:2381 
> _make_query_result_rpc_call() 
>   /google/appengine/datastore/datastore_query.py:2190 
> create_async() 
>   /google/appengine/datastore/datastore_rpc.py:93 
> positional_wrapper() 
>   /google/appengine/datastore/datastore_query.py:1787 
> run_async() 
>   /google/appengine/datastore/datastore_query.py:1657 run() 
>   /google/appengine/api/datastore.py:1595 GetBatcher() 
>   /google/appengine/api/datastore.py:1616 Run() 
>   /google/appengine/ext/db/__init__.py:2010 run() 
>   /google/appengine/ext/db/__init__.py:2091 fetch() 
> [This is the bottom of the traceback.] 
>
> Any possible way to get back the helpful tracebacks? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/YW5HujPxTyUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.