[google-appengine] How to do full table scans for analysis on App Engine?
hello everyone, i want to provide detailed reports in my app. those reports require data aggregated across full tables. in the client i want to plot the change of values over time. this then requires to select very huge data sets. however, in the app engine there is the restriction to get 1000 items in one request. so what is the best approach to do queries with big result sets? i can imagine that the client could split the big request into 10 smaller ones to collect, merge and display the data. but i think this is complex, error prone and slow. i would rather like to do this on the server side. but do i have to split the requests to work around the 1000 items limit or is there another pattern that developers should use to handle huge datasets. basically i want to know two things: 1) how can i retrieve huge amounts of data regarding the get 1000 items limit? 2) is the app engine performance good enough to allow showing reports on the client side i.e. can app engine retrieve and analyse (avg,min,max,grouping) huge amounts of data (10k - 1million rows) fast enough? kind regards, ingo -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: How to do full table scans for analysis on App Engine?
hello again, one more thing: is it possible to reduce the amount of data that is returned dramatically by doing all the aggregations (e.g. by grouping and summing) completely within the datastore? currently i am experimenting with doing full table scans (but i can only retrieve 1000 items in one request..) and aggregating the data within the application (on server side). kind regards, ingo 2010/7/16 Ingo Jaeckel : > hello everyone, > > i want to provide detailed reports in my app. those reports require > data aggregated across full tables. in the client i want to plot the > change of values over time. this then requires to select very huge > data sets. however, in the app engine there is the restriction to get > 1000 items in one request. so what is the best approach to do queries > with big result sets? > > i can imagine that the client could split the big request into 10 > smaller ones to collect, merge and display the data. but i think this > is complex, error prone and slow. i would rather like to do this on > the server side. but do i have to split the requests to work around > the 1000 items limit or is there another pattern that developers > should use to handle huge datasets. > > basically i want to know two things: > 1) how can i retrieve huge amounts of data regarding the get 1000 items limit? > 2) is the app engine performance good enough to allow showing reports > on the client side i.e. can app engine retrieve and analyse > (avg,min,max,grouping) huge amounts of data (10k - 1million rows) fast > enough? > > kind regards, > ingo > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Boost Extensibility and Simplify Maintainability of Web Applications: Proposals for Google Web Toolkit Plug-in Architectures
Hello everyone I wrote a short article proposes a plug-in management architecture for GWT/AppEngine applications, see http://honeycrm.googlecode.com/files/BoostExtensibilityAndSimplifyMaintainabilityOfWebApplicationsProposalsForGoogleWebToolkitPlugInArchitectures.pdf. Now, I would like to invite the community to discuss my proposal. Thanks in advance to everyone for reading and discussing the proposal. Abstract: The extensibility of Google Web Toolkit (GWT) applications running on the Google App Engine platform is currently limited due to the absence of a plug-in infrastructure. Even though for desktop, server, and web applications, plug-ins have been existing for decades, they are not prevalent in GWT today. In this article, I outline the development environment restrictions that make it a challenge to provide dynamic plug-in loading support. I assume that the introduction of a plug-in architecture would dramatically enhance the extensibility and the maintainability of GWT applications. Therefore, this article proposes approaches for statically and dynamically loaded plug-ins for GWT applications. Kind regards Ingo -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Boost Extensibility and Simplify Maintainability of Web Applications: Proposals for Google Web Toolkit Plug-in Architectures
Hello everyone after discussing the topic with a college, I realized I have to explain the context of my proposal. There is no application independent mechanism in GWT to dynamically load extensions. Consequently, GWT applications either do not support it or implement plug-in loading on their own. I am convinced this has to be changed, mainly because of two obervations: The dramatically increasing popularity of both (1) AppStores, e.g. Apple App Store, Android Market, and (2) browser/web applications. To support AppStore concepts in web applications, the GWT/AppEngine community would profit from having an application independent framework for the management of application extensions, i.e. plug-ins, apps, or add-ons. Kind regards Ingo 2011/1/17 Ingo Jaeckel : > Hello everyone > > I wrote a short article proposes a plug-in management architecture for > GWT/AppEngine applications, see > http://honeycrm.googlecode.com/files/BoostExtensibilityAndSimplifyMaintainabilityOfWebApplicationsProposalsForGoogleWebToolkitPlugInArchitectures.pdf. > Now, I would like to invite the community to discuss my proposal. > Thanks in advance to everyone for reading and discussing the proposal. > > Abstract: The extensibility of Google Web Toolkit (GWT) applications > running on the Google App Engine platform is currently limited due to > the absence of a plug-in infrastructure. Even though for desktop, > server, and web applications, plug-ins have been existing for decades, > they are not prevalent in GWT today. In this article, I outline the > development environment restrictions that make it a challenge to > provide dynamic plug-in loading support. I assume that the > introduction of a plug-in architecture would dramatically enhance the > extensibility and the maintainability of GWT applications. Therefore, > this article proposes approaches for statically and dynamically loaded > plug-ins for GWT applications. > > Kind regards > > Ingo > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
