[google-appengine] Datastore bug?
I'm running this query (.toString()) in my Java code:
SELECT * FROM FactWeekMediumKeyword WHERE __key__ =
DimAccount(d001200RlItfAAF)/FactWeekMediumKeyword(O27kBI-kCV==
JqKWRaCc73KiOq_iOF== ) AND __key__
DimAccount(d001200RlItfAAF)/FactWeekMediumKeyword(O27kBI-kCV==
JqKWRaCc73KiOq_iOV== ) AND __ancestor__ is
DimAccount(d001200RlItfAAF) ORDER BY __key__
which is equalivalent to this GQL query for the Datastore Viewer in
the appengine website:
SELECT * FROM FactWeekMediumKeyword WHERE __key__ =
Key('DimAccount','d001200RlItfAAF', 'FactWeekMediumKeyword',
'O27kBI-kCV== JqKWRaCc73KiOq_iOF== ') AND __key__
Key('DimAccount','d001200RlItfAAF',
'FactWeekMediumKeyword','O27kBI-kCV== JqKWRaCc73KiOq_iOV== ') AND
ancestor is Key('DimAccount', 'd001200RlItfAAF') ORDER BY __key__
This query (and my production code) return 23 records. If I remove
the ancestor is part and query:
SELECT * FROM FactWeekMediumKeyword WHERE __key__ =
Key('DimAccount','d001200RlItfAAF', 'FactWeekMediumKeyword',
'O27kBI-kCV== JqKWRaCc73KiOq_iOF== ') AND __key__
Key('DimAccount','d001200RlItfAAF',
'FactWeekMediumKeyword','O27kBI-kCV== JqKWRaCc73KiOq_iOV== ') ORDER BY
__key__
I get what I want, all 49 records.
Looking at two records as an example,
name=O27kBI-kCV== JqKWRaCc73KiOq_iOF==
NMKoQqpWS5KZ75OjRaKs76GmNLGdQaR=
which is returned in both queries, and
name=O27kBI-kCV== JqKWRaCc73KiOq_iOF== Nb8jPqKm75OjRaKs76GmNLGdQaR=
which is returned only in the 49 result query, there is something
wrong with the ancestor I believe.
The keys (and internal entity keys) are:
DimAccount: name=d001200RlItfAAF FactWeekMediumKeyword:
name=O27kBI-kCV== JqKWRaCc73KiOq_iOF==
NMKoQqpWS5KZ75OjRaKs76GmNLGdQaR=
entity key prefix:
ag1idHNlYXJjaHNtYXJ0cnwLEgpEaW1BY2NvdW50IhNkMDAxMjAwMDAwMF
DimAccount: name=d001200RlItfAAF FactWeekMediumKeyword:
name=O27kBI-kCV== JqKWRaCc73KiOq_iOF==
NMKoQqpWS5KZ75OjRaKs76GmNLGdQaR=
entity key prefix:
ag1idHNlYXJjaHNtYXJ0coABCxIKRGltQWNjb3VudCITZDAwMTIwMDAwMD
The second entity key prefix, I imagine, should match the first since
they have the same ancestor.
Two questions:
1) If I query on a range of keys, is there any performance gain by
filtering on the ancestor as well?
2) Can you confirm that this is some internal Datastore bug?
Last bit of information, I believe this bug started when I created
new __key__ indexes, which I didn't think I needed, but they get
generated anyways from the auto-generate-indexes code.
Thanks,
Clay
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore bug?
I've posted this to the issue tracker, which is probably more appropriate place. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What are your top three issues in appengine?
I was one of the ones who starred C# support since C# has LINQ. LINQ gives you joins and aggregations, two features that are really needed in the app engine. C# is a serious request. On Oct 6, 10:04 am, Kenneth goo...@kmacleod.ie wrote: I was looking at the issue list the other day. There is a lot rubbish in there (support c#!) but what do people who are actively using app engine want fixed? If you had to pick three issues what would they be? Link to issues only, don't give a why app engine sucks rant. :-) Here are my top three: 1) More granular accounting of how datastore space is usedhttp://code.google.com/p/googleappengine/issues/detail?id=1396 This one should be easy, the information is being pulled from somewhere already. Right now storage usage is a total black hole. 2) SSL/HTTPS Support on Google Apps domainshttp://code.google.com/p/googleappengine/issues/detail?id=792 I know it's a hard problem, but it is really holding gae back. 3) SLAhttp://code.google.com/p/googleappengine/issues/detail?id=501 Release it already, and give us a paid SLA, and a pony. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: normalisation in datastore
Whoops, this posted twice. I posted once, and didn't see it for hours, so I posted basically the same message. On Sep 29, 2:55 pm, clay clay.lenh...@searchlatitude.com wrote: The death of normalization is premature. I wouldn't drink the denormalization coolaid yet. Sure joins are difficult in GAE, and disk space is cheap, however when you change one atomic value in a denormalized database, you'll have to change many rows, which itself can be slow, especially in GAE. On Sep 28, 6:06 pm, Wooble geoffsp...@gmail.com wrote: The datastore isn't a relational database, so articles about relational databases for the most part don't apply. On Sep 28, 7:54 am, jerry ramphisa jere...@gmail.com wrote: Hi there, I hear normalizing the database is bad if you are using google datastore. On other other hand, most articles mention people should normalize databases, it doesn't matter what kind. Can someone give me light here.- Hide quoted text - - Show quoted text -- Hide quoted text - - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: normalisation in datastore
The death of normalization is premature. I wouldn't drink the denormalization coolaid yet. Sure joins are difficult in GAE, and disk space is cheap, however when you change one atomic value in a denormalized database, you'll have to change many rows, which itself can be slow, especially in GAE. On Sep 28, 6:06 pm, Wooble geoffsp...@gmail.com wrote: The datastore isn't a relational database, so articles about relational databases for the most part don't apply. On Sep 28, 7:54 am, jerry ramphisa jere...@gmail.com wrote: Hi there, I hear normalizing the database is bad if you are using google datastore. On other other hand, most articles mention people should normalize databases, it doesn't matter what kind. Can someone give me light here.- Hide quoted text - - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: normalisation in datastore
This is said because the datastore has cheap diskspace and there isn't good support for joins here. However, we'll relearn why normalization is good. Denormalization means that what would normally modify a single cell will have to modify many rows. This isn't ideal in a relational database, nor in the datastore. I wouldn't drink the normalization isn't for the datastore coolaid too much. ;) On Sep 28, 6:06 pm, Wooble geoffsp...@gmail.com wrote: The datastore isn't a relational database, so articles about relational databases for the most part don't apply. On Sep 28, 7:54 am, jerry ramphisa jere...@gmail.com wrote: Hi there, I hear normalizing the database is bad if you are using google datastore. On other other hand, most articles mention people should normalize databases, it doesn't matter what kind. Can someone give me light here.- Hide quoted text - - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: My recommendation: Use Low-Level API instead of JDO/JPA
+1 We're having a similar discussion here: http://groups.google.com/group/google-appengine-java/browse_thread/thread/e406e4f77548f803?hl=en Andy, My view is that there is no productivity difference between the two. It might be good to show us what is difficult to do in the datastore but easy in JDO. Cheers, Clay On Sep 23, 11:02 am, datanucleus andy_jeffer...@yahoo.com wrote: BigTable simply is not a relational db, so it is highly misleading to fake a relational wrapper and to me it caused lots of pain. Apart from the fact that JDO is not a relational wrapper. The API and metadata are object-based; only a subset of metadata is specific to mapped datastores and this is marked clearly in the (DataNucleus) docs. As ever, if you have some specific issue to raise why not define it - e.g something that can be done in the low level API that can't be done via a generic API. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Java Low Level commit timeout
Jason, Just to clarify, the exception happens *during* the commit, and then succeeds. There isn't a way to rollback at this point. I do believe there is a bug in the commit() method that you guys might want to look at. (FYI, my issue is resolved by other means). Cheers, Clay On Sep 21, 7:07 pm, Jason (Google) apija...@google.com wrote: Hi Clay. The datastore timeout limit is fixed. If you're using transactions, you should be able to rollback your changes if any of the writes (in this case, deletes) failed. But this will only work if all entities that you're deleting are in the same entity group. Since bulk updates to entities in the same group are performed sequentially and not in parallel, you will likely see these errors consistently if you pass in the maximum of 500. You should try lowering this number until this particular exception stops being triggered consistently. - Jason --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: what's the bigger/complex application Java/Python running on GAE
A Google guy responded to a similar question: We have no limits :) Are they talking about 1TB, 10TB, 100s TB? We would want to work with them to ensure their data moves on smoothly and that their app has enough storage allocated, but there are no practical limits. Based on the documentation, you'll want to pay attention to how you set your ancestors when you have a large amount of data. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Java Low Level commit timeout
I received an error that the commit timed out using the Java low-level API. The transaction actually succeeded, which was unexpected. The operation deleted around 1000 to 2000 entities (in batches of 500). The biggest issue is when the timeout occurs, we are not sure whether the transaction will succeed or not. This uncertainty goes against one of the reasons we'd want to use transactions (think: transfer money from one bank account to another where one bank account is in GAE). Can you give us the opportunity to specify our own timeout as we'll know what is best for our apps? Or set a larger default timeout? Also, can you rollback on timeout? Cheers, Clay --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Java Low Level commit timeout
For the curious, below is the stacktrace: It only takes about 3 seconds to report failure (but succeed): Uncaught exception from servlet com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown at com.google.appengine.api.datastore.DatastoreApiHelper.translateError (DatastoreApiHelper.java:38) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall (DatastoreApiHelper.java:56) at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall (TransactionImpl.java:39) at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall (TransactionImpl.java:51) at com.google.appengine.api.datastore.TransactionImpl.commit (TransactionImpl.java:61) at mynamespace.MyClass.doPost(MyClass.java:61) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1093) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle (ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle (SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle (ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 405) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java:237) at org.mortbay.jetty.handler.HandlerWrapper.handle (HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 506) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete (HttpConnection.java:830) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest (JettyServletEngineAdapter.java:139) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java:235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:4950) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:4948) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java:24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 359) at com.google.net.rpc.impl.Server$2.run(Server.java:823) at com.google.tracing.LocalTraceSpanRunnable.run (LocalTraceSpanRunnable.java:56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan (LocalTraceSpanBuilder.java:516) at com.google.net.rpc.impl.Server.startRpc(Server.java:778) at com.google.net.rpc.impl.Server.processRequest(Server.java:351) at com.google.net.rpc.impl.ServerConnection.messageReceived (ServerConnection.java:437) at com.google.net.rpc.impl.RpcConnection.parseMessages (RpcConnection.java:319) at com.google.net.rpc.impl.RpcConnection.dataReceived (RpcConnection.java:290) at com.google.net.async.Connection.handleReadEvent(Connection.java: 428) at com.google.net.async.EventDispatcher.processNetworkEvents (EventDispatcher.java:762) at com.google.net.async.EventDispatcher.internalLoop (EventDispatcher.java:207) at com.google.net.async.EventDispatcher.loop(EventDispatcher.java: 101) at com.google.net.rpc.RpcService.runUntilServerShutdown (RpcService.java:251) at com.google.apphosting.runtime.JavaRuntime$RpcRunnable.run (JavaRuntime.java:392) at java.lang.Thread.run(Unknown Source) Unexpected exception from servlet: com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en
[appengine-java] Re: I broke something using Eclipse plugin
That did it!! Thanks Jason! On Sep 16, 11:24 pm, Jason (Google) apija...@google.com wrote: Try adding the JARs to your build path without copying them to your lib directory. Just use Add external JARs in the Eclipse Build Path dialog. - Jason --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: querying using ANCESTOR IS
Thanks Jason! If you ever want to return to JDO ... --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] querying using ANCESTOR IS
I'm getting an error when trying to filter on ANCESTOR IS locally in
Eclipse. Anyone know what might be the problem? Does anyone have a
working example? Am I forced to use the Low-Level API?
The code is:
PersistenceManager pm =
PMF.get().getPersistenceManager();
Transaction tx = pm.currentTransaction();
try {
tx.begin();
Query query = pm.newQuery(ChildClass.class,
WHERE ANCESTOR IS pAnc +
parameters
com.google.appengine.api.datastore.Key pAnc);
try {
Key ancesterKey =
ParentClass.generateKey(accountId)
.getKey();
IterableChildClass queryResults =
(IterableChildClass) query
.execute(ancesterKey,
startDate, endDate);
...
It errors on the execute() method. I currently have no data stored
yet.
Can Google update the docs with a full example?
Portion of expression could not be parsed: ANCESTOR IS pAnc parameters
com.google.appengine.api.datastore.Key pAnc
My code returns the error as JSON to the browser:
{errorMessage:Portion of expression could not be parsed: ANCESTOR
IS pAnc parameters com.google.appengine.api.datastore.Key
pAnc,errorClass:JDOUserException,hasErrors:true,stackTrace:\r
\norg.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException
(NucleusJDOHelper.java:375)\r\norg.datanucleus.jdo.JDOQuery.execute
(JDOQuery.java:299)\r\n}
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: querying using ANCESTOR IS
I got this working using the low-level API. Actually, I'm not sure what JDO gives you over this. I wrap the low- level Entity with my class and give it getters and setters. It doesn't feel any more or less verbose. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
