[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Lets look at it from a performance perspective. 1- 1MB datastructure - which unit of data (leaving files outside) should be bigger than 1mb? IMO that's a badly design datastore. 2- 1000 query limit, which user is going to want 1000 results? 3- Short CPU, it is common knowledge that a user will go away from a page after 3 seconds of loading. so in order to eliminate this bottleneck you use catching, after all if it's intensive to compute it's worth catching. 4- Quotas in general, not even google has enough machines for us to waste. 5- Admin, a django junkie complaining for the lack of UI Let's stop the slogans, and get down to a real discussion here. Discussions are useful. This post misrepresents the original article needlessly and this does not add to the discussion at all. Use-cases for 1 MB were given, and there are plenty more. It is not sufficient that you can use S3, because the over 1 MB file may well be autogenerated (like a downloadable PDF or Excel or XML file). The query limit was given specifically in combination with the lack of expression power to select the records. Nobody wants to return 10.000 records to the browser, but you have to be able to get the 50 records you do want to return. True, some applications know upfront what the exact key will be, but some applications need more dynamic querying. Also, the limit is hurtful because currently MapReduce can only be implemented as a series of successive calls. Likewise, the CPU was not discussed in the context of rendering a standard user page but in the context of background processing and/or report building. Quota's were discussed with lot of understanding, but also a lot of nuance. How do you actually respond to the remarks made? Let's get down to real applications. Admin again was not about being able to use Django, but about how to do data transformation on your database. Do you think Google would be able to rebuild its index, or do any other part of its magic, without MapReduce? Admin is about the ability to prepare things before the user needs them, so that yes you can respond in subsecond turnarounds. The only concern where I agree is file upload, we do need a facility to uplod videos or pdf or images or whatever we want, but that is being worked out, same with SSL. Look, Google has declined to provide forward looking data. Yes, I was at Google IO and they ARE good guys and I believe them when they express their best intentions to meet specific targets (back then, at the beginning of the year). But not providing an official calendar means they are not putting their foot down, essentially that they don't know themselves. Google is asking us to use what is there, and only what is there (not that I wouldn't like a calendar, and that would reframe this discussion completely - but it simply isn't there). And there is little tangible evidence that warrant the faith that they will in fact have a big bang improvement within what has now become a really very short time frame. So my guess is that we'll see substantial delays relative to what was said back at Google IO. And I like Google App Engine very much. Really. And yes, I do build real applications on top of it. And I do believe that many of their limitations stem from the need to scale. I feel that still leaves plenty of room for discussion, and especially for this blog post that makes a lot of very good arguments. Plus the author clearly uses and understands GAE, uses Python, isn't complaining about the designers philosophy. The good news is that by the end of *2009*, the world might be really interesting with GAE, and with competing platforms driving each others features. Filip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
On Thu, Oct 9, 2008 at 6:09 AM, Filip [EMAIL PROTECTED] wrote: Lets look at it from a performance perspective. 1- 1MB datastructure - which unit of data (leaving files outside) should be bigger than 1mb? IMO that's a badly design datastore. 2- 1000 query limit, which user is going to want 1000 results? 3- Short CPU, it is common knowledge that a user will go away from a page after 3 seconds of loading. so in order to eliminate this bottleneck you use catching, after all if it's intensive to compute it's worth catching. 4- Quotas in general, not even google has enough machines for us to waste. 5- Admin, a django junkie complaining for the lack of UI Let's stop the slogans, and get down to a real discussion here. Discussions are useful. This post misrepresents the original article needlessly and this does not add to the discussion at all. Use-cases for 1 MB were given, and there are plenty more. It is not sufficient that you can use S3, because the over 1 MB file may well be autogenerated (like a downloadable PDF or Excel or XML file). Ok that seems like a valid use case, I'll give you that point when you are generating a file on the fly you may get stuck with the 1MB limit. But keep in mind the big file serving Guido talks about still apllies to this use case. The query limit was given specifically in combination with the lack of expression power to select the records. Nobody wants to return 10.000 records to the browser, but you have to be able to get the 50 records you do want to return. True, some applications know upfront what the exact key will be, but some applications need more dynamic querying. Also, the limit is hurtful because currently MapReduce can only be implemented as a series of successive calls. will you post an example where you need 1000 results to then narrow it down to 50? this seems to me like a joins design which is something you shouldn't be doing in datastore, it has been discuss several times that you shouldn't use datastore as a relation database. You may disagree if you are strong on SQL but denormalization is the first step of scalability. Likewise, the CPU was not discussed in the context of rendering a standard user page but in the context of background processing and/or report building. Which are facilities that aren't out on the engine, so you can't critize an interface that was build to serve pages as a bad background processing tool, it will be like complaining about how bad that Italian restaurant is at making Hamburgers. Quota's were discussed with lot of understanding, but also a lot of nuance. How do you actually respond to the remarks made? Let's get down to real applications. I'm a minimalistic at heart, and I strongly believe that any app running that is getting over the head of it's quota is because it isn't efficient enough, with the sole exception of the max page views in which case google has and will increase that bar for the applications that demand it. Again from the talk, the top 5 applications running on GAE have had their quotas raised because they really need it not because some script kiddie is loading half his datastore into memory on every request. Admin again was not about being able to use Django, but about how to do data transformation on your database. Do you think Google would be able to rebuild its index, or do any other part of its magic, without MapReduce? Admin is about the ability to prepare things before the user needs them, so that yes you can respond in subsecond turnarounds. what? the Admin (in a django concept) is a way to fix error in the data and sometimes input data by advanced users, you shouldn't be manipulating a db structure from a GUI, that is why we have migration scripts and so many project dedicated to them. The only concern where I agree is file upload, we do need a facility to uplod videos or pdf or images or whatever we want, but that is being worked out, same with SSL. Look, Google has declined to provide forward looking data. Yes, I was at Google IO and they ARE good guys and I believe them when they express their best intentions to meet specific targets (back then, at the beginning of the year). But not providing an official calendar means they are not putting their foot down, essentially that they don't know themselves. Google is asking us to use what is there, and only what is there (not that I wouldn't like a calendar, and that would reframe this discussion completely - but it simply isn't there). And there is little tangible evidence that warrant the faith that they will in fact have a big bang improvement within what has now become a really very short time frame. So my guess is that we'll see substantial delays relative to what was said back at Google IO. Well that's a social issue, and I wasn't there so I don't have the first hand experience. But do keep in mind the same thing could be said about any product. But I'm not the person to answer this,
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Davew said it way back at the top - appengine's killer feature is scalability. That is what sets it apart from the other cloud systems out there, and it is also the root cause of most complaints (except the quotas, which will disappear when you get to pay for the service). For the application I'm working on, I'm happy to trade off lack of a relational database for the future gain of scalability. My guess is that most of you haven't had the nightmare of an application that suddenly became popular, and you had to become an expert at database replication, load balancing and multi-system maintenance overnight. It's a very stressful situation. So my advice is that if you don't need scalability, get a normal hosting account or EC3. Then you can have PHP, Ruby, MySQL, cron jobs, anything you want - problem solved. Oh, yes you are going to have to shell out a few buck a month. But if you do need scalability, then appengine is a godsend. The limitations are there to make it safe and scalable, not because Google wants to annoy you. You spend a little more time now working around the limitations, and save endless time later managing systems and capacity. And lastly, I believe that many of the complaints come from people just wanting a free hosting service, and not finding what they are used to. It would be a crying shame if Google listened to these people and turned appengine into a vanilla PHP/MySQL hosting service. Appengine is so much more... On Oct 7, 3:54 pm, Ross Ridge [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: One thing you have to remember it is not what Guido or the engineers want. If Google App Engine is to succeed it is what the customers want. If it is designed as you have stated it will never recoup what Google has spent so far let alone down the road. Google App Engine has so many many limitations. Regardless if the limitations are by design or not it is virtually unusable by 99% of all developers. Can Google make a business off the remaining 1%? The question of whether Google can turn Google App Engine into a profitable business doesn't depend on what percentage of developers find it useful, but whether Google exploit a competive advantage. Google could've started up a tradtional web hosting service using popular SQL databases and other techonologies and created something that would have had a much broader appeal. Any one could. That's the problem. Google might be able to grab market share, but without anything to distiguish themselves from their competitors, a best they only get a marginal return on their investment. We can only speculate on what Google business plan for GAE is, but it seems pretty obvious to me that leveraging Google's own internal technologies is at the heart of it. A number of limitations and problems with GAE stem from technologies like Big Table, Google Frontend and Google Apps. Another part of their plan appears to be keeping support costs low, so you're not given much rope to hang yourself (or others). If, in the long term, Google can't make a business following this plan, if it doesn't give them enough a competive advanage, then there's probably no way they can make the kind profits from a hosting service that Google's investors expect. (While it's not terribly relevent to this discussion, I suspect Google has some other goals for GAE that don't deal directly with its viability as a business. One is to educate programmers in the Google way of doing things. I'm sure Google has been fustrated with tons of amazing job applicants with advanced degrees, 10+ years of WWW experience, and the inability work with anything but PHP and SQL. Another is that they want to make even easier for people to create WWW sites, the sort of small little sites that through AdWords/AdSense, Google has made billions.) Ultimately, what matters is what you want and what Google is willing give you. It doesn't matter what 99% developers want. The are number of problems and limitations with GAE that will be fixed. You can look at the issue database to get and idea of what these are. However, there are no timelines, so don't plan anything being fixed tommorow or even a year from now. Many limitations will always be there. You're never going to get all the functionality of an SQL database, nor will GAE be suitable for computationally intensive tasks. Look at a GAE, and see if it offers it what you want as it is now. If it's close but not quite there maybe play around with it, maybe go so far as making a proof of concept of something. On the other hand, if GAE is far away from what you want, then walk away. GAE isn't for you, and probably won't ever be. Maybe check back in a year or so, but now you should be looking for another hosting solution. Ross Ridge --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Assuming you can actually work around all of the limitation simultaneously, considering the more things you work around the close you come to the CPU time quota. It also is not completely accurate to state that the limitations are there to make apps safe and scalable. For example transactions being limited to a single entity group makes it very complicated (i.e. not safe) to write code that needs to reliably update entities from different groups (it isn't always possible to structure entity groups such that everything that is needs to be updated for a request can be all be in one entity group). Also given the roundness of the quotas, I find it very unlikely that the quota numbers were choosen based on an in depth analysis or a broad sampling of data, rather the being choosen fairly arbitrarily (possibly based to some extent on what works for google's own apps). On Oct 7, 4:16 pm, Greg [EMAIL PROTECTED] wrote: Davew said it way back at the top - appengine's killer feature is scalability. That is what sets it apart from the other cloud systems out there, and it is also the root cause of most complaints (except the quotas, which will disappear when you get to pay for the service). For the application I'm working on, I'm happy to trade off lack of a relational database for the future gain of scalability. My guess is that most of you haven't had the nightmare of an application that suddenly became popular, and you had to become an expert at database replication, load balancing and multi-system maintenance overnight. It's a very stressful situation. So my advice is that if you don't need scalability, get a normal hosting account or EC3. Then you can have PHP, Ruby, MySQL, cron jobs, anything you want - problem solved. Oh, yes you are going to have to shell out a few buck a month. But if you do need scalability, then appengine is a godsend. The limitations are there to make it safe and scalable, not because Google wants to annoy you. You spend a little more time now working around the limitations, and save endless time later managing systems and capacity. And lastly, I believe that many of the complaints come from people just wanting a free hosting service, and not finding what they are used to. It would be a crying shame if Google listened to these people and turned appengine into a vanilla PHP/MySQL hosting service. Appengine is so much more... On Oct 7, 3:54 pm, Ross Ridge [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: One thing you have to remember it is not what Guido or the engineers want. If Google App Engine is to succeed it is what the customers want. If it is designed as you have stated it will never recoup what Google has spent so far let alone down the road. Google App Engine has so many many limitations. Regardless if the limitations are by design or not it is virtually unusable by 99% of all developers. Can Google make a business off the remaining 1%? The question of whether Google can turn Google App Engine into a profitable business doesn't depend on what percentage of developers find it useful, but whether Google exploit a competive advantage. Google could've started up a tradtional web hosting service using popular SQL databases and other techonologies and created something that would have had a much broader appeal. Any one could. That's the problem. Google might be able to grab market share, but without anything to distiguish themselves from their competitors, a best they only get a marginal return on their investment. We can only speculate on what Google business plan for GAE is, but it seems pretty obvious to me that leveraging Google's own internal technologies is at the heart of it. A number of limitations and problems with GAE stem from technologies like Big Table, Google Frontend and Google Apps. Another part of their plan appears to be keeping support costs low, so you're not given much rope to hang yourself (or others). If, in the long term, Google can't make a business following this plan, if it doesn't give them enough a competive advanage, then there's probably no way they can make the kind profits from a hosting service that Google's investors expect. (While it's not terribly relevent to this discussion, I suspect Google has some other goals for GAE that don't deal directly with its viability as a business. One is to educate programmers in the Google way of doing things. I'm sure Google has been fustrated with tons of amazing job applicants with advanced degrees, 10+ years of WWW experience, and the inability work with anything but PHP and SQL. Another is that they want to make even easier for people to create WWW sites, the sort of small little sites that through AdWords/AdSense, Google has made billions.) Ultimately, what matters is what you want and what Google is willing give you. It doesn't matter what 99% developers want. The are number of
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
The documentation says that an entity group could be as large as a single user's data. It seems to me that to avoid the problem you are describing that it would be a good idea to do so. I have a feeling that the number of cases where you would need to update more than one user's data in a single transaction would be limited. 2008/10/8 Josh Heitzman [EMAIL PROTECTED] Assuming you can actually work around all of the limitation simultaneously, considering the more things you work around the close you come to the CPU time quota. It also is not completely accurate to state that the limitations are there to make apps safe and scalable. For example transactions being limited to a single entity group makes it very complicated (i.e. not safe) to write code that needs to reliably update entities from different groups (it isn't always possible to structure entity groups such that everything that is needs to be updated for a request can be all be in one entity group). Also given the roundness of the quotas, I find it very unlikely that the quota numbers were choosen based on an in depth analysis or a broad sampling of data, rather the being choosen fairly arbitrarily (possibly based to some extent on what works for google's own apps). On Oct 7, 4:16 pm, Greg [EMAIL PROTECTED] wrote: Davew said it way back at the top - appengine's killer feature is scalability. That is what sets it apart from the other cloud systems out there, and it is also the root cause of most complaints (except the quotas, which will disappear when you get to pay for the service). For the application I'm working on, I'm happy to trade off lack of a relational database for the future gain of scalability. My guess is that most of you haven't had the nightmare of an application that suddenly became popular, and you had to become an expert at database replication, load balancing and multi-system maintenance overnight. It's a very stressful situation. So my advice is that if you don't need scalability, get a normal hosting account or EC3. Then you can have PHP, Ruby, MySQL, cron jobs, anything you want - problem solved. Oh, yes you are going to have to shell out a few buck a month. But if you do need scalability, then appengine is a godsend. The limitations are there to make it safe and scalable, not because Google wants to annoy you. You spend a little more time now working around the limitations, and save endless time later managing systems and capacity. And lastly, I believe that many of the complaints come from people just wanting a free hosting service, and not finding what they are used to. It would be a crying shame if Google listened to these people and turned appengine into a vanilla PHP/MySQL hosting service. Appengine is so much more... On Oct 7, 3:54 pm, Ross Ridge [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: One thing you have to remember it is not what Guido or the engineers want. If Google App Engine is to succeed it is what the customers want. If it is designed as you have stated it will never recoup what Google has spent so far let alone down the road. Google App Engine has so many many limitations. Regardless if the limitations are by design or not it is virtually unusable by 99% of all developers. Can Google make a business off the remaining 1%? The question of whether Google can turn Google App Engine into a profitable business doesn't depend on what percentage of developers find it useful, but whether Google exploit a competive advantage. Google could've started up a tradtional web hosting service using popular SQL databases and other techonologies and created something that would have had a much broader appeal. Any one could. That's the problem. Google might be able to grab market share, but without anything to distiguish themselves from their competitors, a best they only get a marginal return on their investment. We can only speculate on what Google business plan for GAE is, but it seems pretty obvious to me that leveraging Google's own internal technologies is at the heart of it. A number of limitations and problems with GAE stem from technologies like Big Table, Google Frontend and Google Apps. Another part of their plan appears to be keeping support costs low, so you're not given much rope to hang yourself (or others). If, in the long term, Google can't make a business following this plan, if it doesn't give them enough a competive advanage, then there's probably no way they can make the kind profits from a hosting service that Google's investors expect. (While it's not terribly relevent to this discussion, I suspect Google has some other goals for GAE that don't deal directly with its viability as a business. One is to educate programmers in the Google way of doing things. I'm sure Google has been fustrated with tons of
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Not all applications' data can be modeled such that that each piece of data belongs to one and only one user. On Oct 7, 10:24 pm, Ian Lewis [EMAIL PROTECTED] wrote: The documentation says that an entity group could be as large as a single user's data. It seems to me that to avoid the problem you are describing that it would be a good idea to do so. I have a feeling that the number of cases where you would need to update more than one user's data in a single transaction would be limited. 2008/10/8 Josh Heitzman [EMAIL PROTECTED] Assuming you can actually work around all of the limitation simultaneously, considering the more things you work around the close you come to the CPU time quota. It also is not completely accurate to state that the limitations are there to make apps safe and scalable. For example transactions being limited to a single entity group makes it very complicated (i.e. not safe) to write code that needs to reliably update entities from different groups (it isn't always possible to structure entity groups such that everything that is needs to be updated for a request can be all be in one entity group). Also given the roundness of the quotas, I find it very unlikely that the quota numbers were choosen based on an in depth analysis or a broad sampling of data, rather the being choosen fairly arbitrarily (possibly based to some extent on what works for google's own apps). On Oct 7, 4:16 pm, Greg [EMAIL PROTECTED] wrote: Davew said it way back at the top - appengine's killer feature is scalability. That is what sets it apart from the other cloud systems out there, and it is also the root cause of most complaints (except the quotas, which will disappear when you get to pay for the service). For the application I'm working on, I'm happy to trade off lack of a relational database for the future gain of scalability. My guess is that most of you haven't had the nightmare of an application that suddenly became popular, and you had to become an expert at database replication, load balancing and multi-system maintenance overnight. It's a very stressful situation. So my advice is that if you don't need scalability, get a normal hosting account or EC3. Then you can have PHP, Ruby, MySQL, cron jobs, anything you want - problem solved. Oh, yes you are going to have to shell out a few buck a month. But if you do need scalability, then appengine is a godsend. The limitations are there to make it safe and scalable, not because Google wants to annoy you. You spend a little more time now working around the limitations, and save endless time later managing systems and capacity. And lastly, I believe that many of the complaints come from people just wanting a free hosting service, and not finding what they are used to. It would be a crying shame if Google listened to these people and turned appengine into a vanilla PHP/MySQL hosting service. Appengine is so much more... On Oct 7, 3:54 pm, Ross Ridge [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: One thing you have to remember it is not what Guido or the engineers want. If Google App Engine is to succeed it is what the customers want. If it is designed as you have stated it will never recoup what Google has spent so far let alone down the road. Google App Engine has so many many limitations. Regardless if the limitations are by design or not it is virtually unusable by 99% of all developers. Can Google make a business off the remaining 1%? The question of whether Google can turn Google App Engine into a profitable business doesn't depend on what percentage of developers find it useful, but whether Google exploit a competive advantage. Google could've started up a tradtional web hosting service using popular SQL databases and other techonologies and created something that would have had a much broader appeal. Any one could. That's the problem. Google might be able to grab market share, but without anything to distiguish themselves from their competitors, a best they only get a marginal return on their investment. We can only speculate on what Google business plan for GAE is, but it seems pretty obvious to me that leveraging Google's own internal technologies is at the heart of it. A number of limitations and problems with GAE stem from technologies like Big Table, Google Frontend and Google Apps. Another part of their plan appears to be keeping support costs low, so you're not given much rope to hang yourself (or others). If, in the long term, Google can't make a business following this plan, if it doesn't give them enough a competive advanage, then there's probably no way they can make the kind profits from a hosting service that Google's investors expect. (While it's not terribly relevent to this
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Jorge, One thing you have to remember it is not what Guido or the engineers want. If Google App Engine is to succeed it is what the customers want. If it is designed as you have stated it will never recoup what Google has spent so far let alone down the road. Google App Engine has so many many limitations. Regardless if the limitations are by design or not it is virtually unusable by 99% of all developers. Can Google make a business off the remaining 1%? I think Google should pay much more attention to what the posters here are asking for and less on what they have designed App Engine for. Failure to listen to your customers is Always a Mistake! Defend it if you like. App Engine is going to be a huge money pit for Google. Engineers are terrible Businessmen and even worse Marketers Guido said as much in his talk. In the end Google is a publicly owned corporation and not a University and ultimately Google must answer to its customers and stockholders. The goal is to increase shareholder value through offering products and services the broad market demands and not basic research. App Engine in it's current form seems more like someone's thesis project more than a marketable computing platform. On Oct 6, 5:13 am, Jorge Vargas [EMAIL PROTECTED] wrote: On Sun, Oct 5, 2008 at 4:47 AM, Aral Balkan [EMAIL PROTECTED] wrote: Hi Jorge, You clearly don't get must of the stuff you are talking about. ALL the quotas are there for you to be efficient they the whole purpose, they are not hard coded to make your live bad they are in there to make the system better. I feel that you have missed the point of my post. Rest assured that I get the stuff I'm talking about. Please re-read the post and you will see that it is not an attempt to bash Google App Engine. Quite the contrary, it is a simple summary of the issues encountered by one developer while developing a real-world application. The reason for the post is to highlight these issues so that they can be fixed and thus, ultimately, to improve Google App Engine. As you can see from the replies here, and on the post, I am not the only one who has encountered these issues either. We provide constructive criticism for the things that we care about so that they can evolve. I invite you to add value to the conversation and help make Google App Engine better. did you watch the video I posted? they are aware of anything you propose, and half of them are non-fixable specially because they are impositions made so the system won't crumble with bad code, the other half are being worked on and as Guido suggested will probably be out before the end of the year. Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
On Oct 6, 11:20 am, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Regardless if the limitations are by design or not it is virtually unusable by 99% of all developers. From which hat are you pulling that number? Source? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
On Sun, Oct 5, 2008 at 4:47 AM, Aral Balkan [EMAIL PROTECTED] wrote: Hi Jorge, You clearly don't get must of the stuff you are talking about. ALL the quotas are there for you to be efficient they the whole purpose, they are not hard coded to make your live bad they are in there to make the system better. I feel that you have missed the point of my post. Rest assured that I get the stuff I'm talking about. Please re-read the post and you will see that it is not an attempt to bash Google App Engine. Quite the contrary, it is a simple summary of the issues encountered by one developer while developing a real-world application. The reason for the post is to highlight these issues so that they can be fixed and thus, ultimately, to improve Google App Engine. As you can see from the replies here, and on the post, I am not the only one who has encountered these issues either. We provide constructive criticism for the things that we care about so that they can evolve. I invite you to add value to the conversation and help make Google App Engine better. did you watch the video I posted? they are aware of anything you propose, and half of them are non-fixable specially because they are impositions made so the system won't crumble with bad code, the other half are being worked on and as Guido suggested will probably be out before the end of the year. Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
On Oct 6, 11:20 am, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: In the end Google is a publicly owned corporation and not a University and ultimately Google must answer to its customers and stockholders. The goal is to increase shareholder value through offering products and services the broad market demands and not basic research. The goal is whatever the people actually running the company want it to be. If the stockholders don't like it they'll try to replace them. I'm sure Porsche could increase shareholder value by introducing a $15,000 subcompact, too. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Hi Jorge, You clearly don't get must of the stuff you are talking about. ALL the quotas are there for you to be efficient they the whole purpose, they are not hard coded to make your live bad they are in there to make the system better. I feel that you have missed the point of my post. Rest assured that I get the stuff I'm talking about. Please re-read the post and you will see that it is not an attempt to bash Google App Engine. Quite the contrary, it is a simple summary of the issues encountered by one developer while developing a real-world application. The reason for the post is to highlight these issues so that they can be fixed and thus, ultimately, to improve Google App Engine. As you can see from the replies here, and on the post, I am not the only one who has encountered these issues either. We provide constructive criticism for the things that we care about so that they can evolve. I invite you to add value to the conversation and help make Google App Engine better. Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
On Oct 4, 6:20 pm, Nash [EMAIL PROTECTED] wrote: Great article Aral, My strong suggestion at this stage to anyone considering GAE for a production, business use DO NOT USE GAE. I just replied to this same post in a different thread, not realizing it had been posted to this one as well. http://groups.google.com/group/google-appengine/msg/ced273b3bc1a0fa0 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
From near the bottom of http://codecrafter.wordpress.com/2008/09/16/light-web-strategy-game-data-model-smacks-into-gae-limitations/ : After some thought I decided my requirements for an enhanced GAE data model layer were: 1. Transactions spanning entity groups 2. [redacted - more of a nice to have] 3. Schema version management And from earlier tonight http://codecrafter.wordpress.com/2008/10/03/google-app-engine-scalability-that-doesnt-just-work/ * [redacted - not really broken if other issues are fixed] * [redacted - not really broken if other issues are fixed] * [redacted - not really broken if other issues are fixed] * GAE can’t not tell you how many total entities there are of a specific type and it can’t count more then 1000 entities. * GAE limits the entities that can be processed in a single transaction to those in the same entity group and only one request processing instance can write to an entire entity group at a time. * [redacted - not really broken if other issues are fixed] * [redacted - not really broken if other issues are fixed] * GAE’s version of Python does not include marshall or cpickle, only the slow pickle Of these issues, the two I'm finding most annoying is that transactions can't span entity groups (combined with needing to keep entity groups small to avoid put contention so the app will actually scale) and the lack of cpickle (or something more peformant then pickle) so I could efficiently serialize collections of small data structures associated with another entity for which I have need to query. Josh Heitzman On Oct 3, 11:52 am, Aral Balkan [EMAIL PROTECTED] wrote: I just wrote up a blog post summarizing the biggest issues I have with App Engine:http://aralbalkan.com/1504 If you are developing real-world/commercial apps with App Engine, please add your thoughts to the discussion. Thanks, Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Aral, Great article. I think it summarize very well GAE and its limitations I think you can add other issues : - GQL limitation and particularly with these : * Inequality Filters Are Allowed On One Property Only * Properties In Inequality Filters Must Be Sorted Before Other Sort Orders * missing keyword like 'distinct', 'average', ... and so you have to spend time to recreate these basic features and during this time you don't focus on your app. All these features recreated in python (sort(),...) will consume CPU (- CPU quota) - Datasore Timeout/Exception : Each day, i've about 10 timeouts on a simple put() and it seems that these Timeout will continue after the release. Regards @davew : If you don't know what causes the problems, then find out? Is it in your code? Is it a datastore put timeout? Profile it? I've spent a lot of time to understand it too. And I realize that you can raise this Quota Warning in 2 lines : myModel.all(), myModel.fetch(100). So here, there is nothing to optimize or to profile, particularly when it is random. On 4 oct, 10:00, Josh Heitzman [EMAIL PROTECTED] wrote: If sharded entity counters can be implemented at the application level then they can be implemented at the data store level as well. Sharded counters implemented at the application level in python code have scalability limits (see referenced blog post for details) that are lower when then they would be if they implemented directly by the data store (i.e. this is a common enough problem to be pushed down into the platform). As far as transactions go, the global coordination is already there at the entity group level (if it wasn't, then there wouldn't be any transactions at all). Sure it will be more costly at the entity level then at the group level, but it will still be less costly then layering the equivalent on top of the data store API in the python application code. Allowing transactions spanning entity groups, doesn't mean that entity groups and non-spanning transactions need be eliminated. Leave it the application developer to choose the appropriate transaction type as needed. Josh On Oct 3, 11:25 pm, David Symonds [EMAIL PROTECTED] wrote: On Sat, Oct 4, 2008 at 4:06 PM, Josh Heitzman [EMAIL PROTECTED] wrote: * GAE can't not tell you how many total entities there are of a specific type and it can't count more then 1000 entities. * GAE limits the entities that can be processed in a single transaction to those in the same entity group and only one request processing instance can write to an entire entity group at a time. I fail to see how either of these could be implemented in a truly scalable and distributed system such as App Engine. The data is potentially spread over thousands of machines, so any kind of global co-ordination (needed for both entity counts and transactions) would require a single machine to be in charge, and hence the scaling would be limited to that machine's capacity. If you care about counting the number of a particular kind or all kinds, simply increment a sharded counter each time you add an entity. That would scale very well, and give you quick access to totals. Dave. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
@davew: As Sylvain points out, below, the simplest things can cause this. I've had straight-to-template calls cause high CPU errors and, to tell you the truth, even if I could meaningfully profile Django templates, I don't think there's much I could do about it. Especially, as Sylvain states, it appears to be random. Aral On Oct 4, 9:30 am, Sylvain [EMAIL PROTECTED] wrote: @davew : If you don't know what causes the problems, then find out? Is it in your code? Is it a datastore put timeout? Profile it? I've spent a lot of time to understand it too. And I realize that you can raise this Quota Warning in 2 lines : myModel.all(), myModel.fetch(100). So here, there is nothing to optimize or to profile, particularly when it is random. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Hi David, The point of my post is that the scalable and distributed system should be just one mode of operation of Google App Engine as it ignores the other 50% use case which is admin and long-running processes. Those do _not_ have to scale. But they are essential. Aral I fail to see how either of these could be implemented in a truly scalable and distributed system such as App Engine. The data is potentially spread over thousands of machines, so any kind of global co-ordination (needed for both entity counts and transactions) would require a single machine to be in charge, and hence the scaling would be limited to that machine's capacity. snip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
On Sat, Oct 4, 2008 at 6:00 PM, Josh Heitzman [EMAIL PROTECTED] wrote: As far as transactions go, the global coordination is already there at the entity group level (if it wasn't, then there wouldn't be any transactions at all). Sure it will be more costly at the entity level then at the group level, but it will still be less costly then layering the equivalent on top of the data store API in the python application code. Allowing transactions spanning entity groups, doesn't mean that entity groups and non-spanning transactions need be eliminated. Leave it the application developer to choose the appropriate transaction type as needed. You're right, application-wide transactions don't have to subsume entity-group transactions; you could, after all, put every entity in your application in a single entity group. They are, however, still inherently unscalable. Entity groups mean that global coordination is *not* required, only that there is a home for each entity group. As long as the entity groups are small, this responsibility can be split across machines quite easily. Dave. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
The business is the main issues. Search the word obfuscate http://evolvingtrends.wordpress.com/2008/07/23/google-app-engine-threat-or-opportunity/ On Oct 3, 8:52 pm, Aral Balkan [EMAIL PROTECTED] wrote: I just wrote up a blog post summarizing the biggest issues I have with App Engine:http://aralbalkan.com/1504 If you are developing real-world/commercial apps with App Engine, please add your thoughts to the discussion. Thanks, Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
If the entity-groups have a (single) home and there is not any global coordination for transactions that means that there is a single machine in charge of that entity group. Allowing cross entity group transactions does not require that a single machine be in charge of all entities, it just means that the request processing process will need to coordinate with the machines in charge of each entity or that the machines in charge of each entity need would need to coordinate with each other as peers. That said I'm doubtful that entity groups have a single home as they would need to be mirrored around to actually scale, so it seems likely to me there is already coordination between the nodes mirroring the same entity group. On Oct 4, 2:53 am, David Symonds [EMAIL PROTECTED] wrote: On Sat, Oct 4, 2008 at 6:00 PM, Josh Heitzman [EMAIL PROTECTED] wrote: As far as transactions go, the global coordination is already there at the entity group level (if it wasn't, then there wouldn't be any transactions at all). Sure it will be more costly at the entity level then at the group level, but it will still be less costly then layering the equivalent on top of the data store API in the python application code. Allowing transactions spanning entity groups, doesn't mean that entity groups and non-spanning transactions need be eliminated. Leave it the application developer to choose the appropriate transaction type as needed. You're right, application-wide transactions don't have to subsume entity-group transactions; you could, after all, put every entity in your application in a single entity group. They are, however, still inherently unscalable. Entity groups mean that global coordination is *not* required, only that there is a home for each entity group. As long as the entity groups are small, this responsibility can be split across machines quite easily. Dave. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
This is a good point. This isn't something I care about for the web app I writing at the moment, but I've already there are other apps I've though about doing that I definitely would not want be willing upload non-obfuscated source code for, and I was figuring I'd just have to host those elsewhere. On Oct 4, 2:48 am, Davide Rognoni [EMAIL PROTECTED] wrote: The business is the main issues. Search the word obfuscatehttp://evolvingtrends.wordpress.com/2008/07/23/google-app-engine-thre... On Oct 3, 8:52 pm, Aral Balkan [EMAIL PROTECTED] wrote: I just wrote up a blog post summarizing the biggest issues I have with App Engine:http://aralbalkan.com/1504 If you are developing real-world/commercial apps with App Engine, please add your thoughts to the discussion. Thanks, Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Great article Aral, My strong suggestion at this stage to anyone considering GAE for a production, business use DO NOT USE GAE. GAE has significant flaws; these are basic flaws and the time spent writing a work-around to these problems is far too great for very short internet times. Let me add some basic very necessary items (I am going to do a blog entry as well on these issues) 1. No Bulk Delete/Update: If you ever create a one-to-many or many-to- many relation; you will inevitably come to a point when you have to remove an object. In doing so, if that object was being referred by 400 objects; you have to read, change and write those objects back. GAE does not allow you to change that this many objects. A user might leave your service, you would want to remove all their data or mark it as unavailable. A large tweet comes in, 20,000 followers need to be updated. A group gets deleted, all the contacts referring to that group need to be updated. 2. Random Datastore timeouts: The most annoying issue is that a single object read/writeback can sometimes be 1000mc or can be more than 5000mc. This random behavior cripples the app and makes it impossible to optimize 3. The Google Quota Bad: if your app exceeded it's short-term quota the punishment is very excessive: a 24 hour ban. This is the worst possible action you can take on a dot com: You take it offline. 4. No sorting: When using lists, inequalities etc you can't sort on multiple properties. You just can't. 5. Limited Datastore functionality and very poor workarounds: Want to use OR? Sorry, you can't. What to simulate OR in memory? Sorry, your process will be killed either because of high quota or long response time. Even if you get your app to do it in memory, it is a ticking timebomb, it will explode when more users come in. Very unsalable in that regard. Want to use two inequalities? Sorry. 6. Magic Exploding Indexes: Yes, Lists are a great concept but they can cut you in half. You cannot mix multiple lists in a single WHERE clause, or suffer explosion. You cannot create too many (20 if that sounds like too many) indexes, you will have them explode. Lists are a double-edged sword which cut you a lot more than help you. 7. In 2008, GAE keeps on making you reinvent the wheel: As a webapplication/startup, the most important thing is feature velocity. How fast can you deliver features? With GAE, some very common functionality has to be reinvented over and over. To the point where it consumes so much time that the cost-time benefits are completely lost. 8. No HTTPS. Toy apps aside (apologies to wordle and buddypoke), if google wants serious applications it NEEDS to add HTTPS support. In this day and age of trust building, colored address-bar to peace of mind; you cannot leave this feature out. 9. Dev Server is broken. The local test server doesn't work on half our development systems. Its broken. Its results do not reflect the behavior of GAE itself. It won't do simple things like load static files. 10. No support: Ofcourse, this is a preview, if you get in a mess and need a Googler's attention; it's up to their discretion and leisure time do they respond to you. Nothing is binding at this time; you're not paying them anything yet. 11. GAE Admin is NO replacement for the Django Admin. But that's how it is portrayed in the GAE's documentation. You think that Google's Admin is a replacement for django's admin. Boy, are you wrong. The GAE admin is a very limited app; on dev server it will keep throwing errors at you. The app looks more like a backend for programmers where as the whole point of the django admin was to allow for admin USERS to access it and make changes to the data. In its current form, it can't be used by non-tech-savy users. 12. Very slow GAE upgrades: The GAE team is very slow on introducing changes to appengine. For something that's targeted for release by the end of the year, this is not at all going to the pace required. 13. No roadmap shared: We'd all shutup on the features if Google said we're working on it, it'll be out; Google won't even say it's working on it or that there is work being done I'm a big fan of Django and when Google announced a scalable web application framework with Django, I was thrilled! But I have been very disappointed. This preview is not up to the level of Google's previews. Google has shown us that it has high quality standards so much so that we trust it with our private email, docs etc whereas all these services are in Beta. My software shop had a team of 6 GAE developers, but until GAE can get it's act together, we're pulling away from it. The time and money wasted on getting simple things to work is atrocious and the light at the end of the tunnel is just way too far away. WS -Nash On Oct 3, 11:52 pm, Aral Balkan [EMAIL PROTECTED] wrote: I just wrote up a blog post summarizing the biggest issues I have with App Engine:http://aralbalkan.com/1504 If you are developing
[google-appengine] Re: Why Google App Engine is broken and what Google must do to fix it.
Hi Aral, Your 25% ready for primetime certainly depends on what criteria you are basing your decisions on. I personally disagree, but my needs are possibly very different. My number one requirement is scalability. For that I give it two thumbs up, and for everything else I'm willing to adapt to achieve that one goal. If I wanted to put a 2MB file on App Engine and couldn't, I certainly wouldn't knock App Engine 25% for it. (Although I think I can. Simply by splitting the file and concatenating the blobs as I write the response.) To me it also sounds like you're knocking App Engine on some points without profiling live code with real world usage. You say: In my tests, I've found that the high CPU limit can be randomly triggered even in calls that return within a second. To tell you the truth, I don't actually know what causes these. If you don't know what causes the problems, then find out? Is it in your code? Is it a datastore put timeout? Profile it? Also, make sure you run tests that are accurate representations of real world usage patterns and curves. Typically the guard mechanisms kick in when devs run tests that jump from 0 calls to 1000 calls per second, without any kind of ramp up. That's not a good representation of real world usage. In terms of the offsets, etc. Personally it's not a big deal for me. Once I had a better understanding of the architecture it became more obvious why certain limitations are in place. Those limitations are the reason why this platform can scale so beautifully. -Dave On Oct 3, 11:52 am, Aral Balkan [EMAIL PROTECTED] wrote: I just wrote up a blog post summarizing the biggest issues I have with App Engine:http://aralbalkan.com/1504 If you are developing real-world/commercial apps with App Engine, please add your thoughts to the discussion. Thanks, Aral --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
