[google-appengine] Re: FAQ for out of preview pricing changes

2011-05-18 Thread Luís Marques
On May 19, 12:05 am, "Brandon Wirtz"  wrote:
> $9  is less than I spend on Mountain Dew a Day.  This shouldn't be a big
> deal.  You can't get any hosting for $9 a month short of Dreamhost.  If you
> want cheaper than DreamHost pricing likely you don't need GAE's
> infrastructure.

By that rationale the free service would not be needed.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: FAQ for out of preview pricing changes

2011-05-18 Thread Luís Marques
One problem people seem to have with the new pricing is the sudden
transition between 0$ and 9$.
The change to "max(9$, used billable quota)" from the earlier "$9 +
used billable quota" is nice, but still makes the gap somewhat
jarring.
I was thinking, can't there be a middle ground between:

1) small fixed quotas, min $0, max 0$
2) very large fixed quotas, min 9$, max unlimited$

What is the rationale for the $9? To recoup the losses from the $0
loss leader? To support the cost of the free quota that paid apps
receive (e.g. 24 free IHs)? To provide guarantees for serious apps?
(how so?).

Consider offering a billing class which doesn't need to be
particularly generous regarding fixed quotas, but is more elastic than
"max $0 billable quota". That would allow for, say, a web service
which has very low traffic but needs 600 MB of HR storage.

BTW: looking at the price of reserved front-end instances, if one had
to allocate all of the provided free front-end instances, that would
cost 0.05$ * 24h * 30d = 36$ per month. So the 9$ now seems cheap. But
why bundle all that and charge the 9$? You are offering 720 free
monthly instance hours. Many people won't be needing that, just the
elasticity that comes with billing. So just let people buy a whatever
amount they need. Some will need less, and will pay less. Others will
buy more and pay proportionally.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Charging for memory using Megabyte.hours

2011-05-15 Thread Luís Marques
That is indeed an interesting suggestion, but I suspect it might be
unworkable at that granularity. If you have an idle instance occupying
20 MB of a 128 MB instance class, GAE can't make use of the remaining
108 MB without restrictions. If your instance suddenly needs more than
20 MB to process an incoming request then it could not have it
available (without paging) because some other process would be using
it.

Perhaps a clever combination of scheduling and non-guaranteed QoS
instances would allow such finer granularity. A statistical analysis
could allow various apps with intermittent usage spikes to be combined
on the same servers / instance class partitions, in such a way that
you would known with a given probability that no trashing (massive
paging) would occur. Different probabilistic QoS / performance classes
could be worked out.

BTW, some QoS questions:

1) regarding backend classes: I suspect the B8 4.8 GHz is a 2.4 + 2.4
GHz dual core. Is it so? shouldn't that be advertised? It's not the
same for some apps.

2) What exactly is, anyway, the provided QoS of the various instances?
A B1 with 600 Mz is surely a server being partitioned in various class
configurations. But Linux is not a real-time OS, so the other
processes probably can have a significant worst-case impact on your
apps' performance. What that impact can be should probably be better
disclosed.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: mapreduce multi-entity datastore input reader

2010-07-09 Thread Luís Marques
On Jul 9, 7:51 pm, "Ikai L (Google)"  wrote:
> This is a case where I'd suggest that you scratch your own itch:
> DatastoreInputFormat is open source, so you can build the feature yourself:

Yeah, I thought about that, took a quick look at the source some days
ago, and though: "meh, I bet those google guys could just tweak this
in 1/10 the time it would take me" :-) I'll see what I can do.

BTW: how's work going on the reduce step? Do you think it might be
finished in less than a year? I realized today that one one the
planned batch jobs needs a reduce step, so its not possible with the
current mapreduce.

(so, no mapreduce mailing list, do you confirm that?)

Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] mapreduce multi-entity datastore input reader

2010-07-09 Thread Luís Marques
Hello Googlers,

(I didn't find a mailing list for mapreduce, so I'm posting this here.
Is there one?)

Would you please consider adding a new datastore input reader (or
extending existing ones) to allow specifying more than one entity
kind? Or an input reader which processes all entity kinds? (perhaps
being intelligent enough to skip over mapreduce control entities, if
necessary)

The main use I'm thinking about is re-putting entities, to regenerate
indexes, but there are other uses.

Thanks,
Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Upcoming maintenance notification

2010-06-30 Thread Luís Marques
Same here, in the spam folder marked as fishing.

Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: App updates are taking forever...

2010-06-29 Thread Luís Marques
On Jun 28, 7:20 am, Robert Kluin  wrote:
>    I am not sure this would be a good idea, since indexes are global
> to the app.  How would you handle the case when you are just uploading
> an index and not a version?  Perhaps your rule would only apply when
> uploading indexes with a new version?

Yes. It would just be an automatization on the manual process of
uploading a new version, waiting for the indexes to build and then
setting that version as the default.
If there was no new version then the indexes would just be updated.

Being an internal process, perhaps it could be done with the internal
minor version number, so that no visible version change would occur.
In any case, the dashboard would have to be updated to show that a new
version was ready to be deployed as soon as the indexes were fully
built.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: appcfg.py --dump or download_data

2010-06-29 Thread Luís Marques
http://stackoverflow.com/questions/3066934/appcfg-py-error-no-such-option-dump-on-google-app-engine

The docs are wrong. Use:

appcfg.py download_data --application=app_id --url=http://etc --
filename=file

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Datastore design question

2010-06-29 Thread Luís Marques
On Jun 29, 8:41 am, Phil  wrote:
> It sounds like the list property can't be appended to without a
> transaction?  If so, it seems it'll be better to create two tables,
> one for Users and one for Items.

The list property can be appended without an explicit transaction. It
just might not be wise, since between a read of the entity and the
write with the appended list another request can update the entity,
and that update would be lost. The entity write itself is atomic
though. You won't have some of the properties written from the
original request and some written from the other concurrent request.
So just be careful of race conditions, if you can be sure there aren't
concurrent requests or that they are idempotent then I guess you
should be OK without a transaction.

> Just to answer the above questions... I expect many reads of the data
> and few writes.  I also expect the normal case to have few items (1-2
> on avg) and it won't change often.  The key to the Items table would
> be user_id-item_name.  Are prefix scans in GQL efficient?  Also, in
> terms of data I'm storing about the items, I have ~5-10 fields of
> metadata.

If you know the key then that's the fastest way to retrieve the
entity, instead of using a query. I don't know the Java syntax by
heart, but it's easy to find.

> For now I'm assuming that the right thing to do here is to split the
> data between tables.  Even if it seems to be an unlikely problem now,
> I don't want to put myself in a situation where I'm limited by the QPS
> that transactions can support.

Someone more experienced might be able to elucidate, but on your
scenario I don't know what you'd gain with that, other than smaller
RPC requests for writing to the user items (at least performance-wise,
but it might be a cleaner design). If you have to make a transaction
then the User and UserItems must be part of a transaction group (items
have the user as a parent), so the transaction locks out the entire
group anyway. That's the same as locking out a group made of a single
entity, a user with a list of items property.

See if you can avoid transactions with your design, if you ever run
into write performance problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Need help in building index

2010-06-29 Thread Luís Marques
Could it be a  syntax error in index.yaml?

Luís

On Jun 29, 9:08 am, Ujj  wrote:
> Hi Robert,
>
> Yes I do not see the required index in the Datastore indexes in the
> admin console. I get the NeedIndexError while doing certain a action
> on my live app that requires that index. The index (as prompted by the
> NeedIndexError) is present in my index.yaml and I have tried deleting
> & recreating the file. Still no luck.
> I have even tried updating a new version of the app but that does not
> help either.
>
> Thanks
> Ujj
>
> On Jun 29, 10:33 am, Robert Kluin  wrote:
>
>
>
> > Hi Ujj,
> >    After appcfg.py updates the application it will upload the indexes
> > in index.yaml.  So, if the index is present in index.yaml when you run
> > appcfg.py update / it should be uploaded.
>
> >    So on the appspot.com admin console you do not even see the index
> > listed?  To rule out a admin console issue, have you tried the live
> > app to see if you get a missing index error?  Also, have you tried
> > clearing and recreating your index.yaml before uploading?
>
> > Robert
>
> > On Mon, Jun 28, 2010 at 11:55 PM, Ujj  wrote:
> > > Hi Robert,
>
> > > Thanks for the response. Yes I ran the update_indexes. The index still
> > > does not show up.
> > > On a related note, is it required that we run the 'appcfg.py
> > > update_indexes' even though the index is present in the index.yaml
> > > file and we update the app via 'appcfg.py update /'?
>
> > > Thanks
> > > Ujj
>
> > > On Jun 29, 8:42 am, Robert Kluin  wrote:
> > >> Hi Ujj,
> > >>   Did you try running "appcfg.py update_indexes"?
>
> > >> Robert
>
> > >> On Mon, Jun 28, 2010 at 12:20 PM, Ujj  wrote:
> > >> > Hi,
>
> > >> > For some reason one of my indexes for my app  'mypockit' (version:
> > >> > minimal) is not showing up. On localhost the app works fine and the
> > >> > index is present in the index.yaml file. Even after uploading, the
> > >> > indexes have not been updated for over 24 hours.
>
> > >> > In the dashboard I don't even see the index as 'Building'. It simply
> > >> > does not appear. Could you please let me know if I am doing anything
> > >> > wrong.
>
> > >> > Thanks
> > >> > Ujj
>
> > >> > --
> > >> > You received this message because you are subscribed to the Google 
> > >> > Groups "Google App Engine" group.
> > >> > To post to this group, send email to google-appeng...@googlegroups.com.
> > >> > To unsubscribe from this group, send email to 
> > >> > google-appengine+unsubscr...@googlegroups.com.
> > >> > For more options, visit this group 
> > >> > athttp://groups.google.com/group/google-appengine?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups 
> > > "Google App Engine" group.
> > > To post to this group, send email to google-appeng...@googlegroups.com.
> > > To unsubscribe from this group, send email to 
> > > google-appengine+unsubscr...@googlegroups.com.
> > > For more options, visit this group 
> > > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: What's the best way to enable blobstore without getting charged?

2010-06-27 Thread Luís Marques
On Jun 25, 4:46 pm, Mark Ivey  wrote:
> (I want to use blobstore for its API. The files I want to serve are
> actually small so I can't imagine going over the 1GB free quota. Given
> this it seem strange that I have to enable billing to use blobstore.)

Is there any disadvantage to using a db BlobProperty? They are not
indexed, and if the files are small they should be within the
datastore RPC limits.

Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: App updates are taking forever...

2010-06-27 Thread Luís Marques
On Jun 26, 11:19 am, Jody Belka  wrote:
> What you can do of course, is deploy the new version but don't make it live
> until the indexes are ready

That is true, but perhaps a feature proposal should be made for the
following. Add an option for clients not to be served the updated
version until all the new indexes are built. Make that option on by
default.

Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Datastore design question

2010-06-27 Thread Luís Marques
Hello Phil,

Anyone correct me if I'm wrong but you use one of the following
options, among others.

You can have a "User" model, with a "items" ListProperty. The pros: if
you have the key for the user (e.g. you can use the email), then with
a simple get (faster than a query) you can retrieve the entity
including the items. Writing a new object is transactional. The cons:
you must always get a set the entire user entity. The entity might get
large if there are a lot of items. If you use custom indexes with the
"items" property the index might get large quickly, especially if you
use more than one ListProperty (exploding indexes).

You can have a "User" model, and a "UserItem" whose parent is the
User, making it part of a transaction group. Cons: you have limited
QPS writes to the transaction group (about 1-10 QPS), but that's the
same limitation you have to a standalone User with items. You have to
query (seek) the items, although I wouldn't call that a problem
without profile data. Pros: you can write individual UserItems. You
can have transactions for the user and the items s/he has.

You can have a "User" model and a "UserItem" with a User key
reference. Cons: you can no longer have a transaction depending on the
user and the items s/he has. Pros: you can write individual items
without write collisions to the User.

You can have a "User" and a flat list (e.g. text, non-indexed) of the
corresponding items. Cons: limits the queries you can make, have to
update all items at once. Pros: better index performance.

Etc. I hope it was helpful and correct?

Best regards,
Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: AppEngineJS base on python ?

2010-06-27 Thread Luís Marques
Hello Tom,

As good as the language is, Python isn't the best one to implement
most other languages, due to its speed, so you don't see many
languages or DSLs implemented on top of Python (PyPy being a notable
exception). So I wouldn't much expect to find a Python JS
implementation, upon which an App Engine JS framework could be built.

Best regards,
Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Investigating usage costs, profiling

2010-06-27 Thread Luís Marques
Hi Lenny,

If you find the answer to this one please post it here, since it seems
quite a few interesting questions go unanswered in this group.

I'm no expert on this matter, but there's something hard to understand
indeed. The CPU cost (without API) makes sense if you think about the
following. In the middle of the graph, between 20 ms / 21 ms get / set
memcache calls there is a gap, about 600 ms in length. That is your
could executing, without RPCs, so that explains about 0.6s of the 2s
of CPU usage. Presumably, the remaining 1.4s would be code executing
after the last memcache get RPC, and so wouldn't be visible on the
timeline (perhaps you've got something slow there?).

Now, what doesn't make sense to me is the CPU + API grand total. There
are 66 ms of RPC API summed up at the RPC total. That makes sense,
it's the two datastore calls, a get and a query. But where does the
remaining API time come from? I thought API times only came from RPC
calls. It seems something else must be responsible for that. I'm
curious what.

BTW, can't you merge the memcache gets, sets and deletes in only three
RPC calls? E.g. memcache.get_multi()?



On Jun 27, 11:54 pm, Lenny Rachitsky  wrote:
> I'm having trouble understanding where the usage costs are coming
> from, specifically arround the CPU/API usage. I'm looking at the
> following app stats example:
>
> http://img.skitch.com/20100627-x6s7s988y2kg2at3kg3t6984ig.png
>
> To me this looks rather well optimized, except when you get to the
> "Grand Total" which has a huge cpu+api cost. Can anyone help me
> understand what could be causing this? My costs are growing quickly,
> and I'm considering some major revamping to deal with this situtation
> I can't explain.
>
> Any help would be much appreaciated,
> Lenny

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] How do you analyze your sites' usage

2010-06-25 Thread Luís Marques
Hello,

What techniques are you using for analyzing the usage of your sites?

Are you downloading and analyzing the logs? Do you add log statements
that track (user) actions?
Are you writing to the datastore directly to keep a history of
actions? Per user? With sharding?
Are you writing to the memcache only, AppStats like?
Are you writing to the memcache, and then periodically writing those
to the datastore? Periodically downloading off site?
Are you using Google analytics?
Or do you just don't analyze user behavior / site usage and hope for
the best?

I would like to hear your opinions on the pros and cons of different
techniques, their impact on performance and ability to understand what
is going on with your sites and what your users do / seem to want.

Thanks for your feedback,
Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Module paths for MapReduce (using Django)

2010-06-12 Thread Luís Marques
On Jun 12, 8:40 am, gops  wrote:
> you don't have to link it via app.yaml, you can map urlmap via
> django , just make sure that it points to the right page.

I used app.yaml because I needed to specify "login: admin". Is there
another way?

The mapreduce/main.py does its own initialization. How would I use
that through Django? It doesn't have a Django page handler, doesn't
return a Django HttpResponse and initializes its own GAE request
handling stuff. So I'm not sure how I could use Django to point the
right page.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Module paths for MapReduce (using Django)

2010-06-11 Thread Luís Marques
Hello,

OK, it seems that the problem that is making this difficult is the
following.

If the mapreduce tries to import module "myapp.foo.A", and module A
tries and fails to import module "myapp.foo.B", mapreduce reports that
A could not be found in path "myapp", instead of the import error that
occurs in A during its import.

In my specific case, the error was importing Django, even though this
was not exactly obvious, given the misleading error. So, I suggesting
fixing this usability error. Perhaps I should open a bug report?

Another question though is the following. In my main.py I have a
standard django initialization...

from appengine_django import InstallAppengineHelperForDjango
InstallAppengineHelperForDjango()
(etc)

...followed by a main() execution which uses Django. When mapreduce is
finally included in GAE, I will not have control to modify the
mapreduce/main.py. So, how should I do to (efficiently) import/install
Django for use with mapreduce?

I suppose an efficient technique will not repear the "install" inside
of the main(), but only do so once for each new instance? How do I do
this? I got problems creating a new main and calling mapreduce from
there, while trying to avoid patching mapreduce/main.py.

Thanks,
Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Module paths for MapReduce (using Django)

2010-06-11 Thread Luís Marques
Hi,

I'm having some problems with the paths when using mapreduce.
It seems that mapreduce works well out of the box in a flat project,
but I'm having problems when using Django, which puts the actual
Django app inside a subdir.

So, simplifying a bit, I have the following files...

myapp/main.py
myapp/mapreduce/main.py
myapp/myapp/views.py  (imports models)
myapp/myapp/models.py

...and an entry on app.yaml
- url: /mapreduce(/.*)?
  script: mapreduce/main.py

What do I have to change to be able to launch a mapreduce job through
the mapreduce URL? Currently, I get the error:

Error -- ImportError: Could not find 'views' on path 'myapp'

I have tried several approaches but so far I always get one path error
or another. Have any of you been able to setup mapreduce with such a
filesystem structure?

Thanks for your insights,
Luís

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: Order problem or index wrongly computed?

2010-06-11 Thread Luís Marques
Hello Robert,

It seems, indeed, that this is a manifestation of issue 2481. I have
stared that issue.

It worries me very much that such a critical issue has lingered for
more than a year without resolution, with no visible progress, while
the GAE team has continued to work on new features. The same goes for
issue 2396.

Luís

On Jun 10, 6:40 am, Robert Kluin  wrote:
> Hi Luís,
>   If it is possible, you might want to iterate over all entities of
> that kind and re-put them.
>
>   Also, you might want to see, and star, issue 2481.
>      http://code.google.com/p/googleappengine/issues/detail?id=2481
>
> Robert

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Re: mapreduce, with a very strange warning message in logs

2010-06-10 Thread Luís Marques
On Jun 9, 2:12 pm, "Nick Johnson (Google)" 
wrote:
> No, that's not necessary. Keys are (of course) indexed, so no re-putting is
> required. re-putting your data is only necessary when you want to move from
> an unindexed property to an indexed one, or to add a new indexed property.

I'm not sure I understood. Changing an index from asc. to desc. does
not require a re-put, because the asc. index already exists?

There's also related question I never understood. If the keys already
have an asc. index, why can't that index be used in reverse for desc.
queries? Due to the CPU / disk caches being optimized for forward
reading?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



[google-appengine] Order problem or index wrongly computed?

2010-06-09 Thread Luís Marques
Hello,

I'm changing the names a bit, for privacy reasons (if you need more
details please say so), but I have the following models:

class Event(BaseModel):
name = db.StringProperty(required=True)
...

class EventObservation(BaseModel):
event = db.ReferenceProperty(Event,
collection_name='observations')
accuracy = db.FloatProperty(required=True)
...

For every Event instance in the datastore I have at least one
EventObservation. I also have the following index:

- kind: EventObservation
  properties:
  - name: event
  - name: accuracy
direction: desc

The dashboard for the app says that the index is computed ("Serving").
But in a scheduled task which goes through the EventObservation's of
some Event's, the app got an exception because no EventObservation was
returned for one of the Event's, even though there is one. The
relevant code is:

q = db.Query(EventObservation).filter('event =', event).order("-
accuracy")
obs = q.get()
if obs.accuracy > 0.0: # assumed there always is an EventObservation
...

Using the remote API and a local shell, I found the following two
behaviors:

# with order("-accuracy") there are no results
my-app> q2 = db.Query(EventObservation).filter('event =',
a_specific_event).order("-accuracy")
my-app> q2.count()
0L

# without the order the single observation is correctly returned
my-app> q2 = db.Query(EventObservation).filter('event =',
a_specific_event)
my-app> q2.count()
1L

# the EventObservation has good data
my-app> obs = q2.get()
my-app> obs
EventObservation(**{'event': datastore_types.Key.from_path(u'Event',
138001L, _app=u'my-app'), 'accuracy': 89.0})

My question is the following. Given that I have an index for "event /
accuracy", and that the index is "Serving", what can be the reason
that a particular EventObservation is not being returned, since it
exists? Could it be that the indexing process missed it? Is it because
there is a single EventObservation for that Event? Am I relying on a
wrong type of consistency? Am I doing something else wrong?

(BTW, the index was generated by uploading a new index.yaml. Since
then no more Event's or EventObservations were added to the datastore,
which makes it even stranger. I haven't tried regenerating the
indexes, since if this happened to be an app engine bug I guess you
might want to try ascertaining the cause?)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.