Hi, Thanx for your reply.But i dont know how to use task/Queues.can u plz send me some example and steps i need to follow. Thanx Sandeep
On Mon, Aug 2, 2010 at 7:13 PM, John Patterson <jdpatter...@gmail.com>wrote: > There is no way around the 30 second limit yet. You must break up your > work into tasks - probably start one task for each for each URL. The > Deferred library linked to below makes this easy to manage. > > > On 2 Aug 2010, at 23:13, Sandeep Ghosh wrote: > > Hi, > Twig is good but the problem i am facing is that from the browser i am > passing a load command as parameter like this > *http://localhost:8888/sbphishlock?load* > which tells my code that i need to load the data. > now in the logic i have written few logic to process in a loop > 1)Reads a csv file containg a list of URL's(may be 3000) > 2)downloads each URL HTML pages and tokenize it > 3)Creating a Object and storing in the database > > Problem is first 2 points itself takes a lot of time > I think i need to use Task/Queues so that i can start a task for each url > > But i dont know what i need to do for task/Queues > moreover is there any other way of doing this > Can u help me regarding this > > Thanx > Sandeep > > On Sun, Aug 1, 2010 at 1:25 AM, John Patterson <jdpatter...@gmail.com>wrote: > >> I mean "bulk put" - that is using DatastoreService.put(Iterable) or JDO's >> makePersistentAll(...) >> >> If you do one call with many instances instead of many calls with one >> instance it will be a lot faster. >> >> On 31 Jul 2010, at 14:49, Sandeep Ghosh wrote: >> >> Hi, >> can i do bulk load in Java. >> Can you Please give me a sample code. >> >> Thanx >> Sandeep >> >> On Fri, Jul 30, 2010 at 6:22 PM, John Patterson <jdpatter...@gmail.com>wrote: >> >>> >>> On 30 Jul 2010, at 18:10, Sandeep Ghosh wrote: >>> >>> Hi, >>>> I am facing promlem of entering many data into the Google App >>>> engine database as its taking more then 30 sec time limit. >>>> >>>> PersistenceManager pm = PMF.get().getPersistenceManager(); >>>> for(;;) >>>> { >>>> EnterData data = new EnterData(key,value,new Text(totaData));// >>>> EnterData is my class Entity >>>> pm.makePersistent(data); >>>> } >>>> >>>> How can i use the Tasks/Queues in this scenario >>>> >>> >>> You could try using bulk put rather than making a datastore call for >>> every iteration. >>> >>> You can also do many store operations in parallel if you use Twig like >>> this: >>> >>> for (...) >>> { >>> datastore.store().instance(data).later(); >>> } >>> >>> this is an non-blocking async call that returns immediately and lets you >>> run all your operations in parallel. >>> >>> Otherwise, to use task queues I would recommend using the Deferred >>> library to chain tasks: >>> >>> >>> http://groups.google.com/group/google-appengine-java/browse_thread/thread/6f736624cc820d5b >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To post to this group, send email to >>> google-appengine-j...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >>> >> >> >> -- >> "Kind words can be short and easy to speak, >> but their echoes are truly endless." >> Mother Teresa >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > > > -- > "Kind words can be short and easy to speak, > but their echoes are truly endless." > Mother Teresa > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- "Kind words can be short and easy to speak, but their echoes are truly endless." Mother Teresa -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
