@Robert and Eli. Thanks so much for your feedback. @Robert. The 10x more complex refers to numbers of users and different classes of users. Just for example we have "animators" now, other classes of users would likely include "lighters", "modelers", and other CG artists. I'm painfully aware that I won't be able to get any serious number crunching on image/video data from GAE. The questions become even more interesting with rumors of AWS offering its own transcoding service, and implementing virtualized GPUs. So it is highly likely that I won't be able to avoid some kind of AWS engagement in the long run. (Their new "micro" instances also make the development hardware cost issue moot.)
@Eli. Yes. It sounds I'll need to budget substantially for defensive programming. I don't have any top guns in Boise (that I know of), but I don't want to lose the talented kid from Chennai either. Specifically I'll need to use the "Task Queue / Cron Job" trick to keep at least one instance alive, and I'll need to check upload file sizes and retry if necessary, and probably a bunch of other stuff. The biggest pain though is probably going to be defensive programming for any off-GAE services (transcoding for example). On the other hand I am indeed anticipating AWS system admin for our main application would be a drag (especially if we stick with Windows 2003 servers). One problem is that we have usage spikes at the beginning of our projects, and I doubt we could get away with a single server, since we are serving lots of videos from a central database. Scaling to a handful of servers is almost as much trouble as scaling to a hundred. I also like not having to think too hard about edge caching, latency, or managing instances across EC2 regions. I'm not worried about GAE going away at all (applause for Ikai, Nick, et. al.) John On Nov 1, 10:17 pm, Eli Jones <eli.jo...@gmail.com> wrote: > I suppose if you foresee system admin tasks becoming a big pain in the ass, > then Appengine would alleviate that. > > Also, you can really keep your costs low during initial development which > can be good if every single penny counts. > > But, it sounds like your user base will be fairly small (you mentioned > thousands to tens of thousands), and you don't expect any big time > scalability issues... so you won't get much benefit from Appengine's > abilities there. And, a well written application with good DB design > running on EC2 should be able to serve out reads to 10,000 users with no > problem at all (especially if you slathered in a little memcache). > > If you have a small scale app that will frequently have to cold start (say, > 5 minutes goes by without anyone using the app), then you will probably run > into issues with any 3rd party imports timing out or with the app failing to > get an instance allocated within the 10 second limit. This is not a big > deal if the user doesn't mind reloading a page.. but it can be somewhat > annoying to end users who just want the site to work. > > For example, I am the only user of my app. It is all back end processing > that takes place once every 5 minutes over a 6 hour and 35 minute period > (so, this scheduled process should successfully run 80 times). Sadly, the 5 > minute mark is just enough time for all hot instances to go cold. So, it > cold starts every time. Twice today it threw DeadlineExceeded errors while > trying to import a 3rd party module that is included with my app. From week > to week I will see a number of 500 errors with one of these messages: > > "A serious problem was encountered with the process that handled this > request, causing it to exit. This is likely to cause a new process to be > used for the next request to your application. If you see this message > frequently, you should contact the App Engine team. (Error code 203)" > or > "Request was aborted after waiting too long to attempt to service your > request. This may happen sporadically when the App Engine serving cluster is > under unexpectedly high or uneven load. If you see this message frequently, > please contact the App Engine team." > > I saw two 500 errors on Thursday and Wednesday of last week. (2 may not seem > like a lot, but this is for a process that runs only 80 times a day. So, > that's 2.5% failure.) > > For me this doesn't matter, since it just retries its task after the failure > and succeeds. I expect potential failure and just work around it.. it's > much better than having to pay to have an EC2 instance running for 6 hours > every day (not to mention the chicken and the egg problem of what would > start up this EC2 process in the first place [guess you could use Appengine, > but then.. why not just do it all on Appengine eh?]).. or needing to bother > with administering the EC2 machine. > > So, if the apps you're developing might have frequent cold starts, make sure > you're able to deal with the occasional instance timeout or deadlineexceeded > error on an import. If cold starts won't happen much or if your users won't > mind a 1% to 2.5% retry rate on cold page loading (or well, they won't raise > all hell because of it), then Appengine can work very well for you. > > Of course, there's always the concern of Appengine going the way of Google > Wave.. or maybe all the Appengine Developers going to work at Facebook or > Twitter instead.. but, you should have a 3 year lead time on that.. and > there are a number of projects that aim to run Appengine apps in other > environments (like Appscale and TyphoonAE). > > There are other bonuses... like the Map Reduce API and the upcoming Channel > API and the Matcher API.. that are great if you have a use for them. > > I guess this is all just a long winded way of saying, if you like Python and > want to be able to iterate the hell out of your development projects at 100 > miles an hour into the future and want to be able to scale out from 1 to 130 > instances in 60 seconds and want some wildin' out APIs in your bag of tricks > and can stand the bumps in the road along the way.. do it.. port to > Appengine. But, if you have a top gun client in Boise who will lose > patience with your app because he has to reload something and might drop you > because of this and you can't guarantee 99.9% uptime because you aren't in > control of what your app is running on... and topgun is a big cash cow for > you.. then you might do better to stick to EC2 and deal with administering a > rock solid service there. Everything is a trade off... I guess that's what > I'm trying to say. > > On Mon, Nov 1, 2010 at 10:14 PM, John McLaughlin < > > > > > > > > johnmclaugh...@massanimation.com> wrote: > > Hi Eli, > > > Thanks for asking. > > > The EC2 app was originally written more than 4 years ago using a .NET > > MySQL tech stack. It has had fairly minor updates since, but the > > original developer is mostly unavailable for further development. The > > app is currently running on a single EC2 instance and it allows us to > > manage a single project at a time. We will we need a more nimble > > environment because we will be running multiple projects at a time, > > that are up to 10 times more complex, with more varied user > > engagement. > > > There is still a fair possibility that we would stick with our current > > EC2 methodology and put off the inevitable tech rewrite. But we are > > currently between projects, and have the opportunity to switch > > technologies now and avoid the "changing tires while speeding down the > > freeway" syndrome that we might have a year from now. > > > John > > > On Nov 1, 3:17 pm, Eli Jones <eli.jo...@gmail.com> wrote: > > > What are your reasons for leaving Amazon's EC2? > > > > Knowing the primary reasons you have for leaving EC2 may help determine > > if > > > migrating to Appengine is worthwhile. > > > > On Mon, Nov 1, 2010 at 6:08 PM, John McLaughlin < > > > > johnmclaugh...@massanimation.com> wrote: > > > > Hi All, > > > > I’m looking for a reality check before diving into developing (in > > > > python) a major application for our web site (technically this is a > > > > ground-up rewrite of an aging EC2 based app). There are a lot of > > > > things that I like about GAE: easy development and deployment, > > > > (mostly) worry free data storage, world wide response caching, etc. > > > > And I’ve already programmed some smaller test apps with no major > > > > problems. However now its “rubber hits the road” time and the > > > > decision is not yet a slam dunk. Here are my concerns: > > > > 1. As far as I can tell there are few people using the the blobstore > > > > to serve video. I haven’t had a problem with it, but I don’t feel I > > > > have the safety in numbers factor either. > > > > 2. Video transcoding is an essential component. Currently I’m going > > > > off-GAE to Zencoder -- but this gives me one extra point of failure, > > > > and an awkward data flow to round-trip back to the blobstore. There > > > > could be other off-GAE processing requirements later such as 3D > > > > rendering. > > > > 3. I don’t yet know how to serve Flash content from GAE. I assume > > > > PyAMF is my tool, but I don’t know enough about it to assess how much > > > > effort it will take to get up-to-speed and implement my Flash to > > > > Datastore connections. > > > > 4. My app traffic and database size are probably on the small side > > > > compared to most commercial sites that would use GAE. This could be > > > > good -- I benefit from all the big guys hammering the platform, or bad > > > > -- my particular needs don't get Google love. > > > > > Opinions? Is GAE the right platform for my app? > > > > John > > > > > PS: Appoximate app parameters are: > > > > 1. Rapid and flexible development is a high priority. Therefore we > > > > could trade operational dollars to reduce development expenses (within > > > > reason). > > > > 2. Up to 2000 world wide "member" users. These users would upload > > > > short videos (~10 seconds, ~10MB) and other data at least a couple > > > > times a day. > > > > 3. About 5 main pages that are member accessible which serve mostly > > > > moderately complex Flash content. > > > > 4. Facebook Connect with up to 100,000 worldwide fans. Fans will > > > > access the site and view a few short videos daily. 10% of this access > > > > will probably be from mobile devices. > > > > 5. Primary database tables are: “Members” (<2000 entries), > > > > “Tasks” (<3000 entries), and “Uploads” (<100000 entries) > > > > 6. Only moderate security demands: (No financial transactions, uses > > > > Facebook or OpenID login, low profile target for bot or DDOS threats.) > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to google-appengine@googlegroups.com > > . > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > > > e...@googlegroups.com><google-appengine%2Bunsubscrib > > e...@googlegroups.com> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > e...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
