Re: [google-appengine] Re: Keeping an updated dictionary list of over 1000 in App Engine
Hi, On Sat, Dec 19, 2009 at 2:14 AM, killer barney wrote: > I'm definately going to start comparing the load times. I think you're > right, reading a file seems to be a much faster idea. > > There's one problem, however, since App Engine doesn't allow users to > write to the disk, I won't be able to update the file when I do need > to update it. So that means I have to do it manually when I do need > to update it, right? That seems like a pretty tedious task. > Yes, you'd need to deploy a new app version in order to update the data. > > This got me thinking, instead of storing it in a file, what about just > storing it in some huge blob entity. Won't that be essentially be > doing the same thing, except with the benefit of being able to write > and update to it when I need to? It would seem if I stored all the > keywords in a json blob that could solve my problem as well. What are > your thoughts on that? > If the data is less than 1MB in size, that's possible too. Remember to cache it in local instance memory between requests, though! -Nick > > thanks for your help Nick > > > > On Dec 18, 10:23 am, "Nick Johnson (Google)" > wrote: > > On Fri, Dec 18, 2009 at 6:12 PM, killer barney > wrote: > > > You know I thought about that, is it really better to open a file and > > > read the contents then just querying the database? > > > > If it'll fit in the file, and you can read it efficiently (eg, you're not > > loading lots of data you don't need), absolutely! Files are stored on the > > local filesystem, so access to them is much faster than datastore access. > > > > > > > > > Maybe for a table of my size, opening files would be faster, so > > > definately a consideration. > > > > > So considering this solution, how would you load the keywords from a > > > local file on a disk? If I am not wrong, you have to just download it > > > via http because there is no direct local file load right? If so, then > > > won't that also compound the loading times of a local file? > > > > No, you can store the data in a local file, and load it directly from > disk > > the same way you load templates. Simply store it in whatever format you > > need, such as comma-delimited, and load it as you normally would. > > > > -Nick > > > > > > > > > > > > > > > > > On Dec 18, 2:34 am, "Nick Johnson (Google)" > > > wrote: > > > > Hi, > > > > > > On Thu, Dec 17, 2009 at 10:48 PM, killer barney > > > wrote: > > > > > sorry, hopefully I can explain in better detail :) > > > > > > > I have a list of keywords in a table that's the size of thousands. > It > > > > > changes, but not very often, so the list doesn't have to be updated > > > > > all that often. > > > > > > > When I receive a body of text, I want to highlight the keywords in > the > > > > > body of text. So I have to constantly compare the body of text to > my > > > > > keywords. Because of that, I think querying would be too large a > hit > > > > > on the database. So I have a working version where I store in > > > > > memcache keywords sharded by letter, for example storing all > keywords > > > > > starting with the letter 'A', and so forth. This is a temporary > fix > > > > > that will work for now but will cause problems later as my keywords > > > > > exceed 1000 per letter. Not only that, when the memcache gets > > > > > disposed, the user will have to query to populate that memcache, > which > > > > > could take a really long time for that unlucky person who has to > > > > > populate the memcache for everybody else. > > > > > > > So looking for a better version of what I'm currently doing. > > > > > > Why not simply load the keywords from a local file on disk? > > > > > > -Nick Johnson > > > > > > > On Dec 16, 5:12 am, "Nick Johnson (Google)" < > nick.john...@google.com> > > > > > wrote: > > > > > > Hi, > > > > > > > > It's difficult to comment usefully without more information. > Where do > > > you > > > > > > get your list from? How is it calculated? How large is it? > > > > > > > > -Nick Johnson > > > > > > > > On Tue, Dec 15, 2009 at 11:11 PM, killer barney < > ajcha...@gmail.com> > > > > > wrote: > > > > > > > I have a question that perhaps someone can give me some insight > on. > > > > > > > > > I have a list of thousands of keywords in my website that I am > > > going > > > > > > > to constantly need to query and iterate through. So I thought > > > rather > > > > > > > than querying for every keyword in the datastore everytime, I > > > should > > > > > > > store it in the memcache. But even this doesn't seem like a > very > > > good > > > > > > > solution as the memcache gets eliminated and I'm going to have > to > > > > > > > somehow query the whole list and store it back into memcache > > > everytime > > > > > > > it gets disposed of. > > > > > > > > > I thought about having the list stored as sharded arrays into > > > > > > > memcache, so when it does get disposed of, then I only have to > > > update > > > > > > > the keywords that start with
[google-appengine] Eclipse: no Google menu options after update to 1.3.0
Hi,
I am running an Eclipse EE 3.5 on Windows Vista (Build id:
20090920-1017) and I have successfully used the GAE plugin 1.2.6 and
GWT plugin 1.7.1. My application is running on GAE.
Yesterday I updated to GAE 1.3.0 and GWT 2.0.0 via "Help->Install new
software". After the update my project showed errors on all Google
related classes ("import cannot be resolved"). The links to the google
libs are shown as "com.google.appengine.eclipse.core.GAE_CONTAINER"
and "com.google.gwt.eclipse.core.GWT_CONTAINER". Furthermore, the
Google menu in the preferences dialog is gone and in the "New
project..." dialog there is no option to create a new web application
project, the complete Google category is missing. But: if I go to Help-
>About Eclipse->Installation details->Installed software, all three
plugins are listed correctly with the newest version (GAE plugin is
dated 20091214, GWT and Google plugins are dated 20091206).
I have already tried to uninstall the Google plugin, the GAE plugin
and the GWT plugin and to reinstall them - no success. I have
uninstalled them again, deleted all Google related entries in C:\Users
\Delo\.eclipse\org.eclipse.platform_3.5.0_1709980481\features and C:
\Users\Delo\.eclipse/org.eclipse.platform_3.5.0_1709980481\plugins,
deleted the configuration directory, started Eclipse with the "-clean"
option and installed again, but no success. Still the same: plugins
are shown as correctly installed but no Google options anywhere.
Does anybody have similar problems or can anybody give me a hint what
I could try next?
Thanks in advance,
Delo
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Datastore won't delete final two Entities; is stuck
I tried to clear out my datastore by deleting what was in there, but I got down to two final entities and nothing I do will delete them. I'm tried a few different methods, but I'm completely stuck. App name is keystodermatology -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Blank page response
What steps will reproduce the problem? 1. Page working ok(displaying content) 2. Code change in py-file 3. Refresh the page in the browser What is the expected output? What do you see instead? Expect: the page content See: blank page, firebug reports "200 Good to go" What version of the product are you using? On what operating system? It happens on my development machine (windows vista + eclipse + app engine patch 1.1RC + SDK 1.3) and on app spot, same behavior in every browser Please provide any additional information below. It passes trough main.py, call real_main() function, everything seems fine, no error and exceptions -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 2 memcache.incr questions
I found the answer.
I have set the default parent to the object in context. Now they
became same entity group.
On Dec 18, 11:57 am, SivaTumma wrote:
> I am taking this opportunity to ask one more question in this context.
> ( Ikai L, Pls suggest. )
>
> I am having an object which has a reference property - Many-to-One to
> an object for which I would like to know the feelings of the users.
>
> I am using memcache to increment a feeling name
> object = object()
> object.reference = db.get(self.request.get('key-stored-in-
> html')
> object.updateFeeling(self.request.get('feeling'))
>
> This updateFeeling does check the memcache and does like this:
> memcache.incr (FeelingShardName)
>
> I am getting an error saying that a transaction should not be done
> with different entity groups.(FeelingShardName, objectEntity,
> referenceEntity).
>
> I know it is complex to think into other's code, but shall look this
> thread for a while if I can find a suggestion.
>
> On Dec 11, 11:04 pm, "Ikai L (Google)" wrote:
>
>
>
> > INCR is atomic. An application should not be connected to multiple memcache
> > servers because key sharding will cause certain keys to have an affinity for
> > certain instances of memcache.
>
> > Expiration works in Memcache lazily if items have not been evicted from the
> > cache in according with LRU (Least Recently Used): on a GET of a value, the
> > client checks the expiration to see if the item has expired or not. If so,
> > the value is not returned to the user and is explicitly expired. Incremented
> > values are stored in Memcache as Strings, so in your example, what *should*
> > happen is that an expiration is set such that when you do a GET, if the
> > value is past its expiration time, it will not return. Have you been seeing
> > different behavior in your testing?
>
> > On Thu, Dec 10, 2009 at 12:26 PM, N Panar
> > wrote:
>
> > > memcache.incr() is supposed to increment a key transactionally, so
> > > that two clients can't clobber each other ... but can the application
> > > expect to be connected to ever be connected to more than one memcache
> > > server that each have different values? (it is expected that is
> > > possible that it will be connected to zero memcache servers).
>
> > > Also, how does expiration interact with incr()? There isn't a time
> > > flag for incr(), but what happens if I add the key with another call
> > > rather than using the initial_value flag, like so:
>
> > > memcache.add('rlcnt-%s-%s' % (op, ip_h), 0, time=60*90)
> > > newcnt = memcache.incr('rlcnt-%s-%s' % (op, ip_h))
>
> > > --
>
> > > You received this message because you are subscribed to the Google Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to google-appeng...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > google-appengine+unsubscr...@googlegroups.com > > e...@googlegroups.com>
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > Ikai Lan
> > Developer Programs Engineer, Google App Engine
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Does request_logs count towards bandwidth quota?
Hey all, There's a bug in the request_logs command on Windows, where --append will download all logs, ever, and append them to the file (so it fills up with duplicates), so for now I've changed it to num_days=0 and let it just overwrite the file. It just occured to me that this might count towards my quota, in which case I probably shouldn't. Does anyone know if this is the case (and/or if the Windows --append bug is likely to be fixed?): http://code.google.com/p/googleappengine/issues/detail?id=1731 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Checking if and entity exists
I think i solved the problem. The Entry.exists call was inside an
async urlfetch callback. I had this :
http://groups.google.com/group/google-appengine-python/browse_thread/thread/0230d030d8407de7
problem with Datastore puts inside callbacks. But the gets seemed to
work because they didnt raise any Exceptions, instead they were
returning inconsistent results. It got fixed when i rearranged some of
the stuff so that the put can be done outside callback.
On Dec 19, 10:04 am, johnP wrote:
> z33m - just looked at your thread, and see that the query in question
> is similar to
> info = cls.all().get()
>
> My situation was similar. Before allowing a person to be deleted, I
> check if the person has any data, by doing something similar to:
> has_data = person.data_set.get()
> if has_data:
> return error message saying clear data before deleting
> else:
> delete person
>
> So one case was where a person got deleted, but there was data.
> Understood - there can be something in my code. But after tracing the
> logs, speaking with the user to be sure he acted as expected, and
> looking carefully enough to be pretty sure that the code is generally
> OK led me to a preliminary conclusion to wait and see if something
> similar reoccurs. btw - this incident occurred pretty close to an
> Appengine deployment period.
>
> In general, these kinds of cases are fairly rare. And experience
> shows that whenever I'm sure that the datastore's at fault, I
> eventually find the real bug in my code :)
>
> johnP
>
> On Dec 18, 7:32 pm, z33m wrote:
>
>
>
> > This is actually not the first time for me either. I've had a similar
> > problem some months back
> > :http://groups.google.com/group/google-appengine/browse_thread/thread/...
> > . Not sure if they are related or not. But in either case the
> > Datastore seemed to return inconsistent data.
>
> > On Dec 19, 1:24 am, johnP wrote:
>
> > > That's an interesting question. I've had two or three instances in
> > > the past few months where *it appeared* that a query returned None
> > > when it should have returned something (or an error, at least). I'm
> > > not sure if this was the case, and do not claim it to be the case:)
> > > But in analyzing chains of events, it seemed that this could have been
> > > the root cause for certain situations.
>
> > > That's why I too am interested the answer to this question.
>
> > > johnP
>
> > > On Dec 18, 12:05 pm, z33m wrote:
>
> > > > Hi,
>
> > > > Im writing a simple feed crawler using tasks. Feed entries are stored
> > > > in Entry model. I use guids to check whether an entry exists or not,
> > > > to avoid entries from begin crawled more than once.. But my checking
> > > > mechanism seems to be failing..
>
> > > > Here is the check function and Entry model that i use..
>
> > > > class Entry(db.Model):
> > > > guid = db.StringProperty()
> > > > title = db.TextProperty()
>
> > > > @classmethod
> > > > def exists(cls, guid):
> > > > query = db.Query(cls, keys_only = True)
> > > > key = query.filter('guid =', guid).get()
> > > > if key:
> > > > logging.info('True : %s' % guid)
> > > > return True
> > > > else:
> > > > logging.info('False : %s' % guid)
> > > > return False
>
> > > > and i get strange logs like this
>
> > > > I 12-18 11:29AM 06.720 True :http://www.techcrunch.com/?p=129969
> > > > I 12-18 11:29AM 06.721 False :http://www.techcrunch.com/?p=129937
> > > > I 12-18 11:29AM 06.722 False :http://www.techcrunch.com/?p=129963
> > > > I 12-18 11:29AM 06.723 False :http://www.techcrunch.com/?p=129883
> > > > all other entries are False
>
> > > > I 12-18 11:19AM 00.745 True :http://www.techcrunch.com/?p=129969
> > > > I 12-18 11:19AM 00.754 True :http://www.techcrunch.com/?p=129937
> > > > I 12-18 11:19AM 00.763 True :http://www.techcrunch.com/?p=129963
> > > > I 12-18 11:19AM 00.770 True :http://www.techcrunch.com/?p=129883
> > > > all other entries are True
>
> > > > am i doing something wrong? .. how is it possible that the same query
> > > > that returned a key 10min ago return None now?
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Noob: Python or Java???
bump -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: how to delete a table/entity?
it is simple i know.
but what i am concerning is about statistics
to count the different fields for different usage,
it means we must count all the data, get the statistics info at once
query.
and more over, this statistics info may be more than one fields and
have different orders between fields.
for the time being, appengine can not handler this no only that we can
not count an entity but also
we can not count indexed fields.
so for a real and fairly big website which need statistics info to see
the conditions of the site.
how can it be achieved by using appengine?
On Dec 17, 8:43 pm, Grant wrote:
> I do bulk operations (including delete) using the queue.
>
> Here is some psuedo code
>
> List itemsToDelete = loadOneHundredItems();
> for (Item i in itemsToDelete) {
> delete(i);
>
> }
>
> //this will stop it when it gets to the end of the queue
> if (itemsTodelete.size() > 0) {
> QueueFactory.getDefaultQueue().add("/admin/delete-all");
>
>
>
> }
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: how to delete a table/entity?
obviously, if you have to page a data set more than 5 items which is not ordered by __key__, you may find that the __key__ is of no use, because the filtered data is ordered not by key. but by the fields value, and for that reason you need to loop query as you may like to do. but you will encounter a timeout exception before you really finished the action. On Dec 19, 8:26 am, Andy Freeman wrote: > > > if the type of data is larger than 1 items, you need reindexing > > for this result. > > and recount each time for getting the proper item. > > What kind of reindexing are you talking about. > > Global reindexing is only required when you change the indices in > app.yaml. It doesn't occur when you add more entities and or have big > entities. > > Of course, when you change an entity, it gets reindexed, but that's a > constant cost. > > Surely you're not planning to change all your entities fairly often, > are you? (You're going to have problems if you try to maintain > sequence numbers and do insertions, but that doesn't scale anyway.) > > > > it seems you have not encountered such a problem. > > on this situation, the indexes on the fields helps nothing for the > > bulk of data you have to be sorted is really big. > > Actually I have. I've even done difference and at-least-# > (intersection and union are special cases - at-least-# also handles > majority), at-most-# (binary xor is the only common case that I came > up with), and combinations thereof on paged queries. > > Yes, I know that offset is limited to 1000 but that's irrelevant > because the paging scheme under discussion doesn't use offset. It > keeps track of where it is using __key__ and indexed data values. > > On Dec 16, 7:56 pm, ajaxer wrote: > > > > > of course the time is related to the type data you are fetching by one > > query. > > > if the type of data is larger than 1 items, you need reindexing > > for this result. > > and recount each time for getting the proper item. > > > it seems you have not encountered such a problem. > > on this situation, the indexes on the fields helps nothing for the > > bulk of data you have to be sorted is really big. > > > On Dec 17, 12:20 am, Andy Freeman wrote: > > > > > it still can result in timout if the data is really big > > > > How so? If you don't request "too many" items with a page query, it > > > won't time out. You will run into runtime.DeadlineExceededErrors if > > > you try to use too many page queries for a given request, but > > > > > of no much use to most of us if we really have big data to sort and > > > > page. > > > > You do know that the sorting for the page queries is done with the > > > indexing and not user code, right? Query time is independent of the > > > total amount of data and depends only on the size of the result set. > > > (Indexing time is constant per inserted/updated entity.) > > > > On Dec 16, 12:13 am, ajaxer wrote: > > > > > it is too complicated for most of us. > > > > and it still can result in timout if the data is really big > > > > > of no much use to most of us if we really have big data to sort and > > > > page. > > > > > On Dec 15, 11:35 pm, Stephen wrote: > > > > > > On Dec 15, 8:04 am, ajaxer wrote: > > > > > > > also 1000 index limit makes it not possible to fetcher older data on > > > > > > paging. > > > > > > > for if we need an indexed page more than 1 items, > > > > > > it would cost us a lot of cpu time to calculate the base for GQL > > > > > > to fetch the data with index less than 1000. > > > > > >http://code.google.com/appengine/articles/paging.html-Hidequotedtext - > > > > > - Show quoted text -- Hide quoted text - > > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
