Re: [google-appengine] Re: Large data volume for live reports
I use GWT with Appengine. We use No 3 mentioned by Klaus in our project
extensively. I can poll the task completion with GWT timer.
Here are the steps approximately:
Create the following Objectify class
class MyTask {
@Id long id;
int status;
}
- GWT client uses rpc to initiate the task for your business logic
- On the server, do a new MyTask with status = 0;
- Pass the MyTask also into the Deferred task.
- Return MyTask key back to the client
- In the Deferred task,
- do the business logic. After your logic is complete, store the
results in the memcache with the MyTask key as the value.
- update the MyTask status value to 1 and do put().
- Client keeps polling using RPC for the value of MyTask key.
- Client checks the value of the MyTask status. If its value is 1, then
- do RPC, to get the results from the memcache.
-Aswath
www.accounting-guru.net.
On Mon, Dec 16, 2013 at 10:01 PM, Anil S sadinenia...@gmail.com wrote:
Thank you very much for all your responses.
On Sunday, December 15, 2013 3:52:22 PM UTC-5, Klaus Post wrote:
Hi!
You should probably take a general look at what you are dealing with, and
consider some which common options are available.
Some of these could apply to your project:
1) Segment data into smaller chunks, and deliver partial data if possible.
2) Have precomputed a summary, if you are summing up large amount of
items, that are updated on write, or as a cron job every 5 minutes if
changes are flowing in all the time.
3) Start a background task doing the report, send an update to the client
when the result is ready, so the user can be notified.
4) If your app is calculation heavy - and not database (datastore) heavy,
consider using the Compute Engine.
5) I haven't tried it myself, but mapreduce could help you summarising
big amounts of data. Just google app engine mapreduce.
Either way your design should have a maximum execution time of only a
fraction of the timeout limit - 10 seconds is an acceptable *worst case*
target. My rule of thumbs is - if you cannot promise that, you should
redesign what you are doing.
I hope this helps a bit.
Regards,
Klaus
On Sunday, December 15, 2013 12:01:19 PM UTC+1, Anil S wrote:
HI,
I have been using Google app engine for a while and faced some timeout
issues while handling large data sets. MY application developed on Java
Platform.
Problem:
When customer selects a date duration from UI, request goes to app
engine to fetch the data from data store and run through custom logic on
results and sends summarized view of data back to display in chats. Due to
large volume of data, I am getting DeadlineExceededException.
I can't really use task queues as I need to show the data on view for
user request.
Can anyone suggest how to resolve this issue?
Note: I am using Objectfy 4
Thanks in advance,
Anil.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an
email to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Large data volume for live reports
+1 Klaus On Mon, Dec 16, 2013 at 2:22 AM, Klaus Post klausp...@gmail.com wrote: Hi! You should probably take a general look at what you are dealing with, and consider some which common options are available. Some of these could apply to your project: 1) Segment data into smaller chunks, and deliver partial data if possible. 2) Have precomputed a summary, if you are summing up large amount of items, that are updated on write, or as a cron job every 5 minutes if changes are flowing in all the time. 3) Start a background task doing the report, send an update to the client when the result is ready, so the user can be notified. 4) If your app is calculation heavy - and not database (datastore) heavy, consider using the Compute Engine. 5) I haven't tried it myself, but mapreduce could help you summarising big amounts of data. Just google app engine mapreduce. Either way your design should have a maximum execution time of only a fraction of the timeout limit - 10 seconds is an acceptable *worst case* target. My rule of thumbs is - if you cannot promise that, you should redesign what you are doing. I hope this helps a bit. Regards, Klaus On Sunday, December 15, 2013 12:01:19 PM UTC+1, Anil S wrote: HI, I have been using Google app engine for a while and faced some timeout issues while handling large data sets. MY application developed on Java Platform. Problem: When customer selects a date duration from UI, request goes to app engine to fetch the data from data store and run through custom logic on results and sends summarized view of data back to display in chats. Due to large volume of data, I am getting DeadlineExceededException. I can't really use task queues as I need to show the data on view for user request. Can anyone suggest how to resolve this issue? Note: I am using Objectfy 4 Thanks in advance, Anil. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- *Jayakrishnan Damodaran* *Uroleaf Technologies* Skype: jayakrishnand88 Phone: 8281853554 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Large data volume for live reports
Thank you very much for all your responses. On Sunday, December 15, 2013 3:52:22 PM UTC-5, Klaus Post wrote: Hi! You should probably take a general look at what you are dealing with, and consider some which common options are available. Some of these could apply to your project: 1) Segment data into smaller chunks, and deliver partial data if possible. 2) Have precomputed a summary, if you are summing up large amount of items, that are updated on write, or as a cron job every 5 minutes if changes are flowing in all the time. 3) Start a background task doing the report, send an update to the client when the result is ready, so the user can be notified. 4) If your app is calculation heavy - and not database (datastore) heavy, consider using the Compute Engine. 5) I haven't tried it myself, but mapreduce could help you summarising big amounts of data. Just google app engine mapreduce. Either way your design should have a maximum execution time of only a fraction of the timeout limit - 10 seconds is an acceptable *worst case* target. My rule of thumbs is - if you cannot promise that, you should redesign what you are doing. I hope this helps a bit. Regards, Klaus On Sunday, December 15, 2013 12:01:19 PM UTC+1, Anil S wrote: HI, I have been using Google app engine for a while and faced some timeout issues while handling large data sets. MY application developed on Java Platform. Problem: When customer selects a date duration from UI, request goes to app engine to fetch the data from data store and run through custom logic on results and sends summarized view of data back to display in chats. Due to large volume of data, I am getting DeadlineExceededException. I can't really use task queues as I need to show the data on view for user request. Can anyone suggest how to resolve this issue? Note: I am using Objectfy 4 Thanks in advance, Anil. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Large data volume for live reports
if something is taking longer than 20-30 sec to generate, it is not good to make user to wait. One minute limit for requests is there for a reason. Either you have to prepare document in advance, or use bigquery, or send email once doc is ready. On Sunday, 15 December 2013 11:01:19 UTC, Anil S wrote: HI, I have been using Google app engine for a while and faced some timeout issues while handling large data sets. MY application developed on Java Platform. Problem: When customer selects a date duration from UI, request goes to app engine to fetch the data from data store and run through custom logic on results and sends summarized view of data back to display in chats. Due to large volume of data, I am getting DeadlineExceededException. I can't really use task queues as I need to show the data on view for user request. Can anyone suggest how to resolve this issue? Note: I am using Objectfy 4 Thanks in advance, Anil. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Large data volume for live reports
Hi! You should probably take a general look at what you are dealing with, and consider some which common options are available. Some of these could apply to your project: 1) Segment data into smaller chunks, and deliver partial data if possible. 2) Have precomputed a summary, if you are summing up large amount of items, that are updated on write, or as a cron job every 5 minutes if changes are flowing in all the time. 3) Start a background task doing the report, send an update to the client when the result is ready, so the user can be notified. 4) If your app is calculation heavy - and not database (datastore) heavy, consider using the Compute Engine. 5) I haven't tried it myself, but mapreduce could help you summarising big amounts of data. Just google app engine mapreduce. Either way your design should have a maximum execution time of only a fraction of the timeout limit - 10 seconds is an acceptable *worst case* target. My rule of thumbs is - if you cannot promise that, you should redesign what you are doing. I hope this helps a bit. Regards, Klaus On Sunday, December 15, 2013 12:01:19 PM UTC+1, Anil S wrote: HI, I have been using Google app engine for a while and faced some timeout issues while handling large data sets. MY application developed on Java Platform. Problem: When customer selects a date duration from UI, request goes to app engine to fetch the data from data store and run through custom logic on results and sends summarized view of data back to display in chats. Due to large volume of data, I am getting DeadlineExceededException. I can't really use task queues as I need to show the data on view for user request. Can anyone suggest how to resolve this issue? Note: I am using Objectfy 4 Thanks in advance, Anil. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
