Hi, On Sat, Dec 19, 2009 at 2:14 AM, killer barney <ajcha...@gmail.com> wrote:
> I'm definately going to start comparing the load times. I think you're > right, reading a file seems to be a much faster idea. > > There's one problem, however, since App Engine doesn't allow users to > write to the disk, I won't be able to update the file when I do need > to update it. So that means I have to do it manually when I do need > to update it, right? That seems like a pretty tedious task. > Yes, you'd need to deploy a new app version in order to update the data. > > This got me thinking, instead of storing it in a file, what about just > storing it in some huge blob entity. Won't that be essentially be > doing the same thing, except with the benefit of being able to write > and update to it when I need to? It would seem if I stored all the > keywords in a json blob that could solve my problem as well. What are > your thoughts on that? > If the data is less than 1MB in size, that's possible too. Remember to cache it in local instance memory between requests, though! -Nick > > thanks for your help Nick > > > > On Dec 18, 10:23 am, "Nick Johnson (Google)" <nick.john...@google.com> > wrote: > > On Fri, Dec 18, 2009 at 6:12 PM, killer barney <ajcha...@gmail.com> > wrote: > > > You know I thought about that, is it really better to open a file and > > > read the contents then just querying the database? > > > > If it'll fit in the file, and you can read it efficiently (eg, you're not > > loading lots of data you don't need), absolutely! Files are stored on the > > local filesystem, so access to them is much faster than datastore access. > > > > > > > > > Maybe for a table of my size, opening files would be faster, so > > > definately a consideration. > > > > > So considering this solution, how would you load the keywords from a > > > local file on a disk? If I am not wrong, you have to just download it > > > via http because there is no direct local file load right? If so, then > > > won't that also compound the loading times of a local file? > > > > No, you can store the data in a local file, and load it directly from > disk > > the same way you load templates. Simply store it in whatever format you > > need, such as comma-delimited, and load it as you normally would. > > > > -Nick > > > > > > > > > > > > > > > > > On Dec 18, 2:34 am, "Nick Johnson (Google)" <nick.john...@google.com> > > > wrote: > > > > Hi, > > > > > > On Thu, Dec 17, 2009 at 10:48 PM, killer barney <ajcha...@gmail.com> > > > wrote: > > > > > sorry, hopefully I can explain in better detail :) > > > > > > > I have a list of keywords in a table that's the size of thousands. > It > > > > > changes, but not very often, so the list doesn't have to be updated > > > > > all that often. > > > > > > > When I receive a body of text, I want to highlight the keywords in > the > > > > > body of text. So I have to constantly compare the body of text to > my > > > > > keywords. Because of that, I think querying would be too large a > hit > > > > > on the database. So I have a working version where I store in > > > > > memcache keywords sharded by letter, for example storing all > keywords > > > > > starting with the letter 'A', and so forth. This is a temporary > fix > > > > > that will work for now but will cause problems later as my keywords > > > > > exceed 1000 per letter. Not only that, when the memcache gets > > > > > disposed, the user will have to query to populate that memcache, > which > > > > > could take a really long time for that unlucky person who has to > > > > > populate the memcache for everybody else. > > > > > > > So looking for a better version of what I'm currently doing. > > > > > > Why not simply load the keywords from a local file on disk? > > > > > > -Nick Johnson > > > > > > > On Dec 16, 5:12 am, "Nick Johnson (Google)" < > nick.john...@google.com> > > > > > wrote: > > > > > > Hi, > > > > > > > > It's difficult to comment usefully without more information. > Where do > > > you > > > > > > get your list from? How is it calculated? How large is it? > > > > > > > > -Nick Johnson > > > > > > > > On Tue, Dec 15, 2009 at 11:11 PM, killer barney < > ajcha...@gmail.com> > > > > > wrote: > > > > > > > I have a question that perhaps someone can give me some insight > on. > > > > > > > > > I have a list of thousands of keywords in my website that I am > > > going > > > > > > > to constantly need to query and iterate through. So I thought > > > rather > > > > > > > than querying for every keyword in the datastore everytime, I > > > should > > > > > > > store it in the memcache. But even this doesn't seem like a > very > > > good > > > > > > > solution as the memcache gets eliminated and I'm going to have > to > > > > > > > somehow query the whole list and store it back into memcache > > > everytime > > > > > > > it gets disposed of. > > > > > > > > > I thought about having the list stored as sharded arrays into > > > > > > > memcache, so when it does get disposed of, then I only have to > > > update > > > > > > > the keywords that start with "N", for example, but this still > > > doesn't > > > > > > > seem like the best solution. > > > > > > > > > Is there a better way to do this? > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the > Google > > > > > Groups > > > > > > > "Google App Engine" group. > > > > > > > To post to this group, send email to > > > google-appengine@googlegroups.com > > > > > . > > > > > > > To unsubscribe from this group, send email to > > > > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com><google-appengine%252Bunsub > scr...@googlegroups.com> > > > ><google-appengine%2Bunsubscrib > > > > > e...@googlegroups.com> > > > > > > > . > > > > > > > For more options, visit this group at > > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > > > -- > > > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, > Registration > > > > > Number: > > > > > > 368047 > > > > > > > > -- > > > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, > Registration > > > > > Number: > > > > > > 368047 > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Google App Engine" group. > > > > > To post to this group, send email to > google-appengine@googlegroups.com > > > . > > > > > To unsubscribe from this group, send email to > > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com><google-appengine%252Bunsub > scr...@googlegroups.com> > > > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > > Number: > > > > 368047 > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appengine@googlegroups.com > . > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > Number: > > 368047 > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
