16:02 < scudder_google> Hi all, I'd like to start another installment
of App Engine chat time, where Google engineers are on hand to answer
your questions and listen to concerns
16:03 < nickjohnson> *tumbleweed*
16:03 < knoonan> Hi guys, for Ruby on App Engine, I think it would be
very useful to have a page with a whitelist of gems (and plugins for
Rails) which have been found to work on GAE/J.
16:05 < maxr_google> That sounds useful. I think I see ribrdb logged
in here.
16:05 < scudder_google> knoonan: interesting idea, the best way to
start might be to create your own list somewhere and allow others to
collaborate with you on it
16:05 < maxr_google> He's done a lot of the GAE Ruby work.
16:05 < nickjohnson> knoonan: That seems like a good idea, but
probably best on the appengine-jruby wiki?
16:05 < Wooble> Is there any truth to the rumor that henceforth people
spamming the main GAE google group with requests to activate their
Java accounts will be redirected to the Java group and moved to the
back of the queue to be
activated?
16:06 < bmckinlay> I have an issue with URLFetch on Java. My
application contacts a web service that sometimes needs more time to
respond to my queries than the ~4.5 seconds that app engine allows. In
the Python SDK, there is an
option to increase this limit. Any possibility of a
similar option being added in Java?
16:06 < nickjohnson> Wooble: If there is, nobody's told me. ;)
16:06 < knoonan> Thanks guys, the App Engine JRuby wiki sounds like
the place.
16:06 < schwardo_google> bmckinlay: it's definitely something that we
plan to support. do you know if it's been filed in the issue tracker
yet?
16:07 < ribrdb> only a couple of people can edit the wiki right now
though.
16:07 < bmckinlay> schwardo_google: yes...
http://code.google.com/p/googleappengine/issues/detail?id=1559
16:07 < ribrdb> knoonan: do you know of a list of gems anywhere that
do or do not work?
16:08 < knoonan> @ribrdb No, I don't know of an existing list. That's
why I think it's time to start one...!
16:08 < schwardo_google> bmckinlay: ok great, i'll accept it to
indicate that it's something we plan to support, and we'll rely on the
# of stars to help prioritize it
16:09 < knoonan> ((I did a lightning talk at the Euruko conference in
Barcelona, Spain, on May 10th, about App Engine. I hope I got some
Rubyists interested.)
16:09 < bmckinlay> schwardo_google: thanks! This would be very useful
to me.
16:10 < drawlabs> any plan for monthly pricing? Sometimes I wish I
could over pay on slow months with the insurance of not breaking too
high into billing on large months. or is this just a dev chat?
16:10 < brett-google> fine to talk about billing too, of course
16:11 < brett-google> we've heard that request from other people, too;
definitely considering that option going forward
16:11 < mcxx> how's the XMPP API going? for me, that's the feature I'm
looking forward to most
16:11 < brett-google> drawlabs: one question is, do you mean you want
your budget to carry over, or your actual payment?
16:12 < brett-google> mcxx: going good. =) nothing to announce but it
should be soon according to the roadmap
16:12 < drawlabs> thanks, yes more like an insurance cloud. Every
indie developer I know is scared of the cloud on that high end one day
or process that ends up costing a few grand.
16:12 < drawlabs> monthly, no carry over to make it economical
16:12 < brett-google> hrm
16:12 < brett-google> yeah because right now when you set your budget
that's a maximum spend, it's not an actual paid amount
16:12 < mcxx> brett-google: superb :)
16:12 < brett-google> so you can at any time raise your budget to fit
a big day
16:13 < brett-google> but then the next day drop your budget again in
case you're worried about costs
16:13 < brett-google> does that make sense? i'm confused about how
they're scared about one day costing a few grand
16:13 < drawlabs> true, but what I am saying is some part of big days
would be shared cost because people pay like an insurance for that.
16:13 < brett-google> ah okay so this is more getting into 95%-like
billing
16:13 < brett-google> where you don't have ot pay for extreme peaks
16:14 < drawlabs> yep
16:14 < nickjohnson> It sounds like you're asking for "slashdot
insurance"? :)
16:14 < drawlabs> hehe yes
16:14 < brett-google> well traditional hosting providers effectively
have that, nick, with 95% metered billing
16:14 < nickjohnson> right
16:15 < drawlabs> I know it would be useful for some people that have
had fears, so they stay at trad hosting for that.
16:15 < brett-google> in my personal opinion, though 95% can help with
peaks in some cases, it's actually bad for customers because it
overcharges the rest of the time
16:15 < brett-google> if you have a single 95% peak, then you're
billed at that level for the whole month
16:15 < brett-google> even if your average is zero
16:15 < brett-google> so our current billing model lets you change
your budget on a daily basis depending on your current needs
16:15 < drawlabs> true, it might be one way to get people though, I
know it would help me moving people onto it.
16:15 < Wooble> For a service where the cost varies so much depending
on what your app does, I can't imagine how you'd begin to even
calculate the monthly costs to charge...
16:16 < brett-google> yeah that's a fair point
16:16 < drawlabs> I think what you have is good or better than most,
sry to overtake with billing
16:16 < rogerh> are there any plans to allow access to the timestamped
fields inside bigtable?
16:16 < brett-google> wooble: indeed it's a complex idea
16:16 < brett-google> drawlabs: well i worked on billing so i'm happy
to talk about it =)
16:16 < brett-google> rogerh: of previous cell version snapshots?
16:16 < nickjohnson> rogerh: What do you mean exactly?
16:17 < brett-google> nick: i think he means the version axis
16:17 < nickjohnson> ah
16:17 < Ashar__> did i missed the "session" ?!
16:17 < rogerh> brett/nick: yeah, trying to figure out how best to be
able to keep track of previous verisons of data we store
16:17 < nickjohnson> Ashar__: No, you're in it.
16:17 < Ashar__> nice
16:17 < oizo> hi, can we expect java local data viewer in the next
SKD? Really, its very necessary for many developers..
16:17 < maxr_google> ozio: Yes, you can expect a java local data
viewer in the next SDK.
16:18 < nickjohnson> rogerh: Even if it were available, Bigtable's
model is such that you can't rely on it sticking around. If you need
to be able to reliably access old versions of data, you're best to
implement it in user code.
16:18 < maxr_google> It was supposed to have made the update that went
out last week but we missed the cutoff by just a little bit.
16:18 < drawlabs> python question, any way in the future plans to
specify a python version you want to target? Or Google supporting
versions of python that you can specify per app?
16:18 < AsharLohmar> i had a problem these days building a data
structure in java
16:18 < AsharLohmar> using JDO
16:18 < brett-google> rogerh: indeed nick's right; especially
considering the sugar we put on top of bigtable to support multi-row
transactions, it gets more complex
16:19 < brett-google> but it would be cool to have built-in support
for Datastore snapshotting
16:19 < brett-google> which we coudl do under the covers more
efficiently than just user code
16:19 < maxr_google> yes, built-in support for Datastore snapshotting
would be super cool
16:19 < AsharLohmar> i had a situation like classroom -> students->
grades, built with 1-n bidirectional relations
16:19 < maxr_google> AsharLohmar: Tell me more about your JDO problem.
16:20 < AsharLohmar> on the parrent classes i've used the "mabedBy"
16:20 < rogerhu> max/brett/nick: great, thanks. indeed that would be
a great feature to have
16:20 < scudder_google> drawlabs: if we were to add additional
versions of python, you would likely specify a different runtime
version. However different Python versions are not on our roadmap at
this point
16:20 < dieterk> I submitted this problem a while ago:
http://code.google.com/p/googleappengine/issues/detail?id=1378
16:20 < brett-google> drawlabs: we've heard requests for python3k; it
would have to be treated as a different runtime version (so in
app.yaml you'd specify py3k instead of python)
16:20 < AsharLohmar> th problem was in this situation ... i had the
classrom and the students defined
16:20 < oizo> maxr_google: good, thanks! And when it will be released
if not secret?
16:20 < AsharLohmar> i could add grades to students
16:20 < brett-google> so yes there's support for doing something liek
this, but there's no py3k right now
16:21 < drawlabs> brett: thanks good to know there is support for it,
hoping for 3000 soon
16:21 < AsharLohmar> but i didn't manage to query the grades for a
student
16:21 < knoonan> With GAE/J and Ruby, does anyone have a neat solution
for zipping files to overcome the limit of 1000 *static* files? (Ola
Bini and others showed how to tackle the code-file limit.)
16:21 < dieterk> I have two questions: 1. can somebody state if there
is at least a will to fix this? 2. why did somebody move this issue to
Java. My appengine is running on python.
16:21 < maxr_google> oizo: The next release isn't a secret but we
don't have a good estimate of when it will be available.
16:22 < Wooble> knoonan: I don't believe that's possible; you almost
certainly need to serve them out of the datastore.
16:22 < scudder_google> dieterk: looks like there was some confusion,
as you are serving Java applet code, I'll recategorize
16:22 < maxr_google> AhsarLohmar: Did your query return 0 results?
16:22 < maxr_google> Or was there some other problem with it?
16:22 < AsharLohmar> i recived an message saying that the student
field from tha grade that i tryed to query is maped to Student, and
tha paramater i've sent has key from tha classroom
16:22 < knoonan> @wobble, there already are well-documented solutions
for Python and for Java itself, using Zip libraries.
16:22 < dieterk> scudder_google: yes. but Content-Encoding is all
about the HTTP Headers and has nothing to do with the appengine at all
16:23 < AsharLohmar> my real classes war Country->city->taxiCompany
16:23 < maxr_google> AsharLohmar: You're passing a Key in as a
parameter to the query?
16:23 < Wooble> the files aren't static anymore if your code can unzip
them.
16:23 < AsharLohmar> no, i use strings
16:23 < AsharLohmar> gae.encoded-pk or something
16:23 < maxr_google> Right.
16:23 < dieterk> scudder_google: also...may I ask somebody from the
appengine team to comment on the issue - so I get to know the mind of
the appengine team about this issue.
16:23 < maxr_google> So, there's a good chance that error message is
accurate.
16:24 < mib_5nyoz6qq> Any comments why I'm getting
ApiDeadlineExceededException for querying one entity?
16:24 < schwardo_google> dieterk: yeah, it's actually not relevant
whether it's marked as python or java -- it's the core infrastructure
that would need to be changed
16:24 < bmckinlay> [A[B[B[B[B[B[B[B[B[B/quit
16:24 < maxr_google> The way to check is to do
System.out.println(KeyFactory.stringToKey(theKeyString))
16:24 < AsharLohmar> i know that the message was telling the truth, i
didn't find anty workarounds thought
16:24 < nickjohnson> mib_5nyoz6qq: Occasional timeouts will happen,
even for simple queries. They should only account for a very small
fraction of your total requests, though
16:24 < maxr_google> That will tell you if the String you're passing
in identifies a Country or a city.
16:25 < schwardo_google> dieterk: I believe we were only aware of gzip
and compress being used as Content-Encodings, -- I wasn't aware that
custom encodings were being used in this way
16:25 < AsharLohmar> exactly, i know that he is right, my real problem
was how can i query then
16:25 < knoonan> @wobble, sure: the code is unzipping the content from
zip-files stored in the file-system: it works for Python and Java, I
wonder if someone's done it successfully for JRuby...
16:25 < schwardo_google> will need to read up to see if that's even
allowed in the spec
16:25 < mib_5nyoz6qq> nickjohnson: no, I'm getting this error for more
than 15 hours constantly
16:25 < dieterk> schwardo_google: core change? So it is unlikely to
happen in the near future or ever? Please comment on the issue if you
do not want to allow it since it is a security risk or else?
16:25 < nickjohnson> mib_5nyoz6qq: For every fetch of a particular
entity, or of all entities?
16:25 < maxr_google> If you want to filter by city you'll need a
String that identifies a city, not a country.
16:25 < maxr_google> If you want to filter by city you'll need a
String that identifies a city, not a country.
16:25 < scudder_google> dieterk: and schwardo, we do not currently
allow the content-encoding header to be set/modified
16:26 < AsharLohmar> because i save the city as chil of country
ofcource it has country in it
16:26 < AsharLohmar> also inside the taxi object
16:26 < AsharLohmar> the city has the same key
16:26 < nickjohnson> And have you determined what your failure rate
is? The logs page only shows errors by default, so it can give the
impression everything is failing when only a very small proportion
are.
16:26 < ribrdb> knoonan: you can jar your code or your gem files, and
put it in WEB-INF/lib
16:26 < mib_5nyoz6qq> nickjohnson: for simple query with range set for
1 entity, I described my problem here:
http://groups.google.com/group/google-appengine-java/browse_thread/thread/9720cb2b5cf45947
16:26 < maxr_google> City and country should different keys since they
are different objects.
16:27 < TheBashar> Quick question. If I create a new entity with a
key_name MyEntity(key_name="abc") and I put() it, if there is already
an entity of the same kind with the same key_name in the datastore,
will it get overwritten?
16:27 < schwardo_google> dieterk: i don't think there are any
insurmountable security implications, but yes, it's a lower level
change than most and would probably take longer to roll out
16:27 < maxr_google> This might be easier to debug if you post your
model classes and your query code to ghe group.
16:27 < nickjohnson> mib_5nyoz6qq: As I said - what proportion of the
time does this happen? And is it only for a specific entity/query?
16:27 < nickjohnson> TheBashar: Yes.
16:27 < maxr_google> Then I can drop it in to my test framework and
see exactly what is going on.
16:27 < TheBashar> Thank you Nick!
16:27 < Aaron_10> I need some help, I uploaded a app to google which
has django. I need to use the admin area but I can't for some reason.
16:28 < Aaron_10> the site is : dev.mobileconsultinginc.com/admin
16:28 < AsharLohmar> thanks maxr_google , i'll add them on the
appengine group
16:28 < dieterk> schwardo_google: scudder_google: also I don't
understand the argument about core change since you allow to set some
http headers? I really would need Content-Encoding for only a single
file. I didn't think this to be
an issue since I do set my Content-Encoding in my
many other hosting setups on a daily basis. Seems to be no security
risk for them.
16:28 < drawlabs> datatype question: any plans for a UUID type or
UUIDProperty for datamodels or as the key maybe? Currently we store
them in strings but lots of apps are distributed and linked via UUID.
Or do you recommend using
the name property of the db.Key field? What is the
best way to use UUID when ints as keys aren't needed?
16:29 < knoonan> @ribrdb I know you can JAR your code and gem files
with JRuby, but what about zipping web-content files to exceed the
static-file limit of 1000...?
16:29 < gvanrossum> Hey all, Guido van Rossum here
16:29 < nickjohnson> drawlabs: There's no plans to introduce a
UUIDProperty, but you can write your own. If you're using UUIDs as the
primary key for your entity, certainly, using it in the key name makes
sense
16:29 < nickjohnson> As long as you make sure the first character
isn't numeric
16:30 < nickjohnson> Hi Guido. Do you have an App Engine question to
ask? ;)
16:30 < moraes> hehehe. yes, this is the place.
16:30 < rmalik> hello Guido!
16:30 < schwardo_google> dieterk: i'll respond with more detail on the
issue itself -- sorry about mislabelling it Language-Java
16:30 < drawlabs> nickjohnson: thanks
16:30 < dieterk> gvanrossum: hi...I am Python fan for ever. Thanks!
16:30 < gvanrossum> Hi Nick, no, I thought I'd just make some
waves :-)
16:30 < nickjohnson> You seem to be doing that. ;)
16:30 < ribrdb> knoonan:ah. well you could write some ruby code to
serve the files out of a jar or zip.
16:30 < rmalik> is there a way to delete an existing application that
you are no longer developing?
16:31 < rogerhu> right now a major limitation is the 1 inequality
property, which makes it hard to do date range scan types. noticed in
google finance you can graphically view stock performances
http://www.google.com/finance?q=INDEXDJX:.DJI,INDEXSP:.INX,INDEXNASDAQ:.IXIC
does this mean that multiple table scans are required to implement
this type of feature?
16:31 < nickjohnson> rmalik: Not currently, no.
16:31 < drawlabs> guido came to support python 3 on GAE :)
16:31 < drawlabs> guido came to support python 3 on GAE :)
16:31 < nickjohnson> drawlabs: Single-handedly, while blindfolded, in
his sleep! ;)
16:31 < yowgi> Hello. Are there any plans to provide support for the
delivery of static files that have been uploaded or created by the
app? Putting images in the database is always quite inefficient...
16:31 < nickjohnson> rmalik: If you just want it off the list of apps,
you can create a dummy account and make it the sole administrator of
the app, but it'll still count towards your 10 app limit
16:31 < gvanrossum> sorry, no py3k support planned yet
16:32 < maxr_google> rogerhu: You can do a 'between' query for a
single property though.
16:32 < TheBashar> Are there any plans to support 2.6 on GAE in the
near future?
16:32 < maxr_google> rogerhu: What sort of query are you looking to
write?
16:32 < dieterk> schwardo_google: no problem. I really like appengine
in every way but this really is somehow a big issue. With my other
hosting services if a feature is missing I just pay more and they fix
it. I guess we cannot
settle this with cash?
16:32 < dieterk> :)
16:32 < nickjohnson> yowgi: Static files are already served directly,
not from the datastore.
16:32 < rmalik> nickjohnson: yeah i'm just trying to take it off from
my 10 app limit
16:32 < drawlabs> gvanrossum: yeh I asked, the mechanism is there but
maybe soon.
16:32 < nickjohnson> rmalik: If you run out of apps, you can ask on
the group for more.
16:32 < rogerhu> max: like trying to pull up a list of more than 1000
people who have registered to vote between certain dates or signed up
online between a few months.
16:33 < yowgi> nickjohnson, even user-uploaded images? i mean HTTP
upload, think blog post attachments.
16:33 < rmalik> nickjohnson: the problem with one app is that i
originally set authentication against all google accounts versus a
google apps domain i created later
16:33 < nickjohnson> rmalik: What do you want to do with >1000 records
in a single query?
16:33 < AsharLohmar> (when )will we be available to delete apps from
our account ?!
16:33 < rmalik> nickjohnson: and now i want to switch that
authentication
16:33 < gvanrossum> TheBashar: no 2.6 support planned either, though
it will be a lot easier to pull off. I'm waiting for the rest of
Google to adopt 2.6 first though.
16:33 < knoonan> @ribrdb I believe it should be feasible with JRuby to
serve web-content out of jars / zip files, I'm just wondering if
anyone has done it. I'm especially concerned about how it might be
implemented with Rails (or
Merb)...
16:34 < TheBashar> @GvR: Thanks!
16:34 < gvanrossum> drawlabs: I don't understand your remark
16:34 < nickjohnson> yowgi: Oh, right. That is coming - see the
roadmap 'service for storing and serving large files'
16:34 < maxr_google> select from Person where registrationDate < x and
registrationDate > y
16:34 < rmalik> nickjohnson: I think you mistaken the >1000 records
concern with someone else's question
16:34 < yowgi> nickjohnson, thanks.
16:34 < maxr_google> that should work fine since you're only filtering
on a single property.
16:34 < brett-google> rmalik: indeed we've heard people say they want
this before; right now it's not that easy for a variety of reasons,
but it's on our radar
16:34 < nickjohnson> rmalik: Unfortunately, the only way to change the
authentication mode is by creating a new app, currently
16:34 < nickjohnson> rmalik: my apologies
16:34 < dieterk> schwardo_google: scudder_google: thank you for our
help! please also contact me by email if you have any questions - I
could provide much more information on why not havin Content-Encoding
is such a big problem for me.
16:35 < nickjohnson> I meant to address that at rogerhu
has joined #appengine
16:35 < DocSavage> Hello
16:35 < drawlabs> gvanrossum: i asked about py3k on gae. dev
responded with a reply that said they had the ability to do versioning
but no plans on adding python 3 yet.
16:35 < rmalik> nickjohnson: i'm glad it's on the radar and i
understand it might not be a high priority at the moment. p.s. i'm
REALLY glad you guys added cron jobs
16:35 < rogerhu> max: what if there are more than 1000 records that
get pulled out? then i need to pull out the next 1000 with a __key__
query
16:36 < nickjohnson> rogerhu: What do you want to do with >1000
records in a single query, though? Surely not display them all to the
user at once?
16:36 < drawlabs> architecture question: say you have an app that runs
on GAE. Then you have 3-4 services that it uses (say a user, stat and
content service). Is that allowed under the TOS to have the site, and
each service on a
different app or do all services also have to be
hosted under the same app? the services would be shared among many
apps not just one. But is that seen as getting out of billing?
16:37 < gvanrossum> drawlabs: the versioning capability is for the
library, not for the language version; py3k language support (or even
2.6) requires much more work and would probably require you to say
"runtime: python3.0" or
"runtime: python2.6".
16:37 < rogerhu> nope, but what if we want to export it to a .csv
file?
16:37 < nickjohnson> rogerhu:
http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_Data_from_App_Engine
16:37 < brett-google> drawlabs: that's a difficult question because it
really depends on how it's implemented
16:37 < nickjohnson> That's the built-in support; you can also write
your own using remote_api and __key__ queries if you want.
16:37 < brett-google> for example, if you have a generic stats service
that's used by your app and many other people's apps, then yeah,
that's fine
16:37 < drawlabs> gvanrossum: thanks, yes I imagine it is quit a bit
of work to support on this scale. I hope for it in the future.
16:37 < mun> does bdbdatastore understand ORDER BY <attr> DESC?
16:38 < nickjohnson> mun: It certainly ought to
16:38 < brett-google> but if you have one app for storing data and
another app that just does URLfetch calls to retrieve that data, then
that's more of a single use; that would fall under section 4.4
16:38 < mun> somehow i can get it to print by 'ORDER BY...' by not
'ORDER BY ... DESC'
16:38 < nickjohnson> If it doesn't, that's a bug
16:38 < rogerhu> nickjohnson: thanks . just trying to figure out a
good way to provide tiered-level permission based access to the data
set. the remote_api makes it hard to do that
16:38 < nickjohnson> mun: Might want to take this offline or wait
until the end of the dev chat to avoid too much clutter. :)
16:39 < DocSavage> I had a stub-level question on how a cookbook
recipe works -- http://tinyurl.com/rbgfq9
16:39 < picalolabu> good day. python question: do i need to install
PyCrypto separately for my dev server?
16:39 < mun> nickjohnson, yeah sure
16:39 < nickjohnson> rogerhu: Yes, you definitely don't want to use
remote_api for anything other than administrators. If you want your
users to be able to download CSV files, I would suggest using __key__
queries, with multiple
requests if necessary.
16:39 < drawlabs> brett: yes that is the item I am talking about 4.4,
we have multiple games for instance that report data. Each in
different places. Some use one or more of the services. But it isn't
a single app. This is where
differnet billing would help maybe, to do stuff like
this without seeing as getting out of billing?
16:39 < gvanrossum> picalolabu: yes
16:39 < oizo> bdbdatastore question: if this db emulate production db,
may i estimate datastore size with indexes, (for example 10000
entites), based on the size of bdfile bdbdatastore on the local disk
(with this 10k entities)?
16:39 < nickjohnson> picalolabu: If you want to use PyCrypto, yes.
16:40 < gvanrossum> DocSavage: what's the question about that page?
16:40 < nickjohnson> oizo: I would not rely on BDBDatastore's index
size reflecting space consumption on production App Engine
16:40 < brett-google> drawlabs: we'd like to provide better native
support for 'multitenant' applications, so you can run multiple siloed
datastores and applications under a single app_id
16:40 < rogerhu> nickjohnson/maxr: then does it become difficult in
app engine to do query date ranges & dumping out more than 1000
people? (i guess rather than using the __key__ > last key name i could
retrieve the next 1000 entries
by using an offset)?
16:40 < picalolabu> nickjohnson: gvanrossum: thanks - but don't upload
the PyCrypto libraries with my app. Correct?
16:40 < nickjohnson> It may - it's probably representative - but don't
rely on it.
16:40 < nickjohnson> picalolabu: Correct.
16:40 < brett-google> this would help a lot with cases like this where
you have lots of code and DB sharing but some fundamental differences
16:41 < DocSavage> The cookbook recipe gives "seamless memcaching" of
puts/gets/queries. But I'm trying to figure out how it expires query
caches after a put on an entity.
16:41 < nickjohnson> rogerhu: You can filter by date range instead of
key range, as long as you make sense to exclude duplicates with the
exact same timestamp.
16:41 < picalolabu> nickjohnson: thx
16:41 < nickjohnson> DocSavage: It doesn't need to expire caches;
memcache does that
16:41 < drawlabs> brett: that would be good. However as long as it
doesn't add overhead. Let's say for architectural reasons you'd want
them under different locations or virtual endpoints. Like right now
we have services hosted for
user, content, game, and stats all in differnet
places and can get data async/parallel.
16:42 < DocSavage> Oh, you mean there's some amount of time the cache
is stale after I do an entity put?
16:42 < brett-google> sure well, you can host an app_id on many
domains and then use the 'host' header to differentiate the task to do
16:42 < brett-google> i've seen people do this
16:42 < nickjohnson> drawlabs, brett: It seems to me that drawlabs is
talking about horizontal separation; what we see abuse around mostly
is vertical separation, with sharding to different backends
16:43 < brett-google> well nick i think he's talking about both
16:43 < nickjohnson> DocSavage: No, it does the memcache put when you
do the datastore put
16:43 < brett-google> like each game would be horizontally separated
16:43 < brett-google> but then the stats and user services would be
vertical
16:43 < rogerhu> nickj: but ultimately the number of queries that get
dumped back is 1000. so i'm trying to figure out how to retrieve the
next batch and still doing a date range query (inequality property) +
getting the next 1000
batch
16:43 < drawlabs> brett: I hope to see more on this soon maybe,
because since it is unclear. Yes talkign more about horizontal not
just breaking it up for getting out of billing.
16:43 < picalolabu> python and gdata: i added a function to read
blogger via gdata (news on my home page). I cache the results but I
have started to see very inconsistent processing times (i.e. some
requests take 3000ms others 60) -
is this normal?
16:43 < rogerhu> number of results that get dumped back that is
16:44 < nickjohnson> rogerhu: If you do a "SELECT * FROM MyEntity
ORDER BY date", you can take the last date returned and do "SELECT *
FROM MyEntity WHERE date > :1 ORDER BY DATE"
16:44 < brett-google> drawlabs: i guess the way i see it is it makes
sense to have one app_id for al the common stuff
16:44 < brett-google> content, users, stats
16:44 < brett-google> and then one app_Id per game
16:44 < DocSavage> If I do a Model put, then ask in a query for all
entities in a Model that meet some criteria, how is that query
memcached? Do I have a misunderstanding in how queries are handled at
the stub-level?
16:44 < brett-google> but i think it makes less sense to have one
app_Id for each individual common service
16:44 < scudder_google> picalolabu: are those times for the urlfetch
request to the Blogger API?
16:44 < brett-google> does that distinction make sense?
16:44 < nickjohnson> DocSavage: Queries aren't memcached, only the
entities themselves. So you'll see the most benefit from puts and
gets.
16:45 < jcgregorio> rogerhu:
http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt
16:45 < lidaobing> [java] if I want to count the rows with JDO, is
this also limited to 1000?
16:45 < nickjohnson> lidaobing: Yes
16:46 < jcgregorio> rogerhu: That writeup from Ryan should cover the
use case you are talking about, using a bookmark to resume the query
for the next 1000 items
16:46 < rogerhu> jcgregorio: great thanks i remember seeing that link,
thanks for the reminder
16:47 < drawlabs> brett: thanks that helps but it would be great if
this was more clear on the ethics there in the future. We are
actually discussing what to do now and some are suggesting mt grid
service at $20 a pop for each. The
limit is 1TB but the drop off is a scary 2.30 per
GB. So you can see how billing is kindof dictating how things are
architected in many cloud projects probably. ok enough billing :)
16:47 < picalolabu> scudder_google: i don't know, they are the times
that appear in the log files with the "... high CPU ..." warning -
when i test from my computer it is always under 200ms
16:47 < DocSavage> nick: Oh, jeez. Queries *aren't* memcached. OK,
wasn't clear because the recipe says "the cache is updated through -
queries", but I see this meant the individual entities are memcached
on query.
16:47 < nickjohnson> right
16:47 < knoonan> Has there been any progress on using certs other than
Google's, or domains other than XYZ.appspot.com, for HTTPS
connections? (I remember talk of a workaround using the latest
browsers...)
16:48 < nickjohnson> It could probably be improved now that key-only
query support is available to do a key-only query and retrieve
entities from memcache and the datastore as appropriate, but I'm not
sure how much that would improve
things.
16:48 < brett-google> drawlabs: yeah makes sense
16:48 < brett-google> knoonan: the issue here is still HTTPS's support
for SNI
16:48 < brett-google> "server name identification"
16:48 < oizo> if app_id is the part of all primary keys, is more
shortly app_id for very big db (billions entities) saves my budget for
data store?
16:49 < lidaobing> is there any progress (or any possibility) on issue
1269: unique static ip per app/account
http://code.google.com/p/googleappengine/issues/detail?id=1269
16:49 < brett-google> the issue is that HTTPS requires the reverse IP
address of the host to match the certificate hostname
16:49 < brett-google> so each website with HTTPS must have a unique IP
address
16:49 < nickjohnson> oizo: In theory, yes. In practice, I think you'd
be hard pressed to tell the difference.
16:50 < brett-google> SNI fixes this by putting the hostname in the
SSL handshake; so then virtual hosting providers can use a single IP
range to serve multiple hosts
16:50 < nickjohnson> brett-google: I thought the real problem was that
you have to establish the SSL session _before_ specifying the
hostname, but you can't tell what hostname they want until _after_, in
the virtual hosting scenario.
16:50 < brett-google> that's the technical problem
16:50 < brett-google> but the only way to get around that is to assign
a separate IP for each hostname you want to serve HTTPS on
16:50 < nickjohnson> right
16:50 < brett-google> the good news is that TLS 3.1 (which is in
safari, firefox, ie8, and opera) supports SNI
16:50 < scudder_google> picalolabu: it sounds like the CPU usage
warnings are from a longer than expected wait time for the response to
get back from the Blogger API, which occasionally happens
16:50 < nickjohnson> I didn't know SSL certs encoded the reverse-ip,
though.
16:50 < brett-google> that's how they work
16:51 < brett-google> the certs odn't have it, but the policy of
browsers is to do the reverrse lookup
16:51 < nickjohnson> Oh, right.
16:51 < brett-google> the recent ie6 forced upgrade to ie8 has helped
a lot
16:52 < brett-google> so maybe we'll have more wide-spread SNI support
16:52 < brett-google> but until then it's likely unrealistic to get
HTTPS support beyond https://app_id.appspot.com
16:52 < brett-google> the good news is that now you can use any Google
Apps domain to log in on appspot.com
16:52 < brett-google> so all app engine apps can use it
16:52 < picalolabu> scudder_google: could the inconsistency be that
some users see the cache and others don't, hence the need to hit
blogger again? or would the cache be seen by everybody until it is
destroyed?
16:53 < brett-google> lidaobing: nothing to report now; this bug falls
into a similar realm as HTTPS support actually
16:53 < brett-google> we'd need a separate IP per app_id
16:53 < rogerhu> am i correct to assume also that there's no current
way to have multiple versions of a datastore for one application? (i
know you can have different app versions)
16:53 < knoonan> @brett-google and @nickjohnson: thanks! I guess we
just have to wait for IE8 to become the norm...
16:53 < brett-google> that's very costly and significantly increases
the complexity of our network infrastructure
16:53 < nickjohnson> rogerhu: That's correct.
16:53 < nickjohnson> Don't thank me, I'm learning from brett-google
too ;)
16:53 < mun> nickjohnson, i'm now using bdb and it contains about 150k
entities. but when i run a .count() on it, it still times out. is it
typical?
16:54 < nickjohnson> mun: Yes, because BDBDatastore takes the same
approach to counting queries as App Engine does, and thus has the same
limitation
16:55 < lidaobing> brett-google, bad news
16:55 < picalolabu> brett-google: could you please expand on "... any
Google Apps domain to log in on appspot" - im not sure I understand
the implication
16:55 < DocSavage> nick: After installing bdbdatastore, I'm getting
the following error in socket_apiproxy_stub.py, line 63, in
closeSession: self._sock.close() ... was wondering if there's a
Google group or someplace bdbdatastore
users could exchange info or ask questions.
16:55 < mun> nickjohnson, you mean they take the same approach as the
app engine on the dev server or on the production server as well?
16:55 < rogerhu> nick: great thanks. this has all been very helpful
16:55 < nickjohnson> mun: On the production server.
16:55 < scudder_google> picalolabu: I'm not sure what your design
looks like, are you memcaching the results of the Blogger API query?
Do you set an expiration, etc? An infrequent long request could also
be a new instance of your app
being spun up, imports need to be done again,
etc.
16:55 < DocSavage> nick: The error was 'NoneType' object has no
attribute 'close'
16:56 < brett-google> picalolabu: if your app has auth restricted to a
single google apps domain (i.e., users @example.com), then you can
serve content to them over HTTPS on app_id.appspot.com
16:56 < nickjohnson> Counting the results of a query requires
iterating over them all. This is actually the case for the vast
majority of queries in an RDBMS, too, they just hide the work.
16:56 < brett-google> and they can log in thorugh appspot.com
16:56 < brett-google> before it was the case that these users could
not log in through appspot.com
16:56 < mun> nickjohnson, so does that mean if my queries time out on
bdb, then they'll time out on the production server as well?
16:56 < brett-google> they could only log in on domains that were
under example.com
16:56 < nickjohnson> DocSavage: Known bug that I'm fixing this
evening. It happens when there were no datastore requests in a
request. You can fix it by adding an "if self._socket:" guard clause.
16:56 < mun> nickjohnson, well, i've set the limit of the count to a
small value-- like 10.
16:56 < DocSavage> nick: thanks!
16:57 < nickjohnson> mun: Not for certain, but they're definitely
correlated
16:57 < nickjohnson> mun: And it times out when the limit is set to
10?
16:57 < mun> nickjohnson, yeah that's right
16:57 < mun> i'm doing .count(10)
16:57 < nickjohnson> mun: I'll look into it.
16:57 < mun> nickjohnson, i see. thanks
16:59 < picalolabu> scudder_google: yes, i was referring to the
blogger api results being memcached - but i had not thought about the
new creating a new instance. thx
16:59 < picalolabu> brett-google: got it - thx
17:00 < DocSavage> Regarding different apps providing services -- does
each app_id get its own allocation of memcache and memory because each
app has its own instances?
17:00 < nickjohnson> mun: Just checked the source - stupidly, I'm not
giving up if it reaches the count specified. Easy fix. :)
17:01 < Aaron_10> I went to the django channel and the told me that
the django admin area uses django models which google apps dosen't
allow, so they said to ask here for a work around the problem.
17:01 < Aaron_10> cause I am having trouble making a admin user for
the website django admin area.
17:01 < Aaron_10> which is called superuser.
17:02 < mun> nickjohnson, you mean it keeps searching even beyond n
many have found?
17:02 < gvanrossum> Aaron_10: have you tried the Google app engine
helper for Django?
17:02 < nickjohnson> mun: Yes.
17:02 < mun> nickjohnson, does that mean fetch has that behaviour too?
17:02 < Aaron_10> ya, it didn't do much.
17:02 < gvanrossum> Aaron_10:
the connection]
17:02 < Aaron_10> are you saying that should be the work around?
17:02 < gvanrossum> Aaron_10: IIRC it supports the Django admin
interface
17:03 < gvanrossum> Aaron_10: yes
17:03 < knoonan> @Aaron_10 try AEP (App Engine Patch): the sample
application includes an admin interface out-of-the-box.
17:03 < jago1> I have a problem with wordle.net hosted on the Google
appengine
17:03 < nickjohnson> mun: No, fetch only returns the requested number
of entities
17:03 < Aaron_10> ok, I will look more into it. I tried it but I might
of done something wrong.
17:03 < nickjohnson> gvanrossum: Aaron is trying to use the Django
admin interface, which apparrently uses Django models
17:04 < mun> nickjohnson, okay
17:04 < nickjohnson> I'm not aware of anything that lets you use
unmodified Django models in GAE, though
17:04 < jago1> visiting the site works...but when I open the applet it
blocks for over 30 sec. before the applet starts. It seems it is
confused finding the appengine:
http://img166.imagevenue.com/img.php?image=38755_http_traffic_122_43lo.jpg
17:04 < scudder_google> I'd like to close the official chat hour, but
some of us will still be here after, so don't hold back on questions.
<gvanrossum> Aaron_10: nickjohnson: I take it back, the article says
it does not use the Django admin site
<gvanrossum> So, try app-engine-patch :-)
Thanks all!
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
