[google-appengine] Re: Dev workflow for Modules + Maven

[AndyD]

I haven't found the EAR project to be necessary.  I work on individual 
modules separately; I just do a "mvn package" to build a module's WAR file 
and then do testing/debugging by running the devserver against that WAR 
file.  I also deploy those module WAR files individually when I need to via 
"mvn appengine:update"; I don't deploy the EAR file.

-Andy

On Sunday, April 20, 2014 7:27:46 PM UTC-4, aloo wrote:
>
> I'm curious how others have setup their dev workflow when using appengine 
> modules + maven?
>
> In order to run the local dev server I need to rebuild my entire project 
> and then run mvn appengine:devserver in my ear directory. This typically 
> takes a minute for a simple code change.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How to tell if I'm enabling edge caching correctly?

[AndyD]

Thanks, I've seen that and used that info for my implementation, yet it 
doesn't seem to be working for me.  That info is from 2011, maybe things 
have changed?  Or maybe it's necessary but not sufficient; i.e. the Edge 
Cache won't cache unless you do those things, but it may still use other 
criteria to decide whether to cache something.

Has anybody implemented edge caching recently, and have any insight as to 
whether there are additional factors involved?

-Andy

On Friday, January 24, 2014 12:06:47 PM UTC-5, Joshua Smith wrote:
>
> Brandon cracked this nut a long time ago, and documented it here:
>
>
> http://www.xyhd.tv/2011/11/industry-news/setting-cache-control-headers-in-python-to-take-advantage-of-google-appengines-edgecache/
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] How to tell if I'm enabling edge caching correctly?

[AndyD]

OK, I've got a handler running on a test URL, and it's returning the 
following response headers.

Cache-Control: public, max-age=31449600  (that's 364 days)
Pragma: Public

I've got biling enabled.  My domain is an appspot.com domain (it would 
appear that at some point in this past, this didn't work, but then as of 
2012, it was supposedly working, according to 
this: http://www.keepitcloud.com/2012/05/caching/).

However, I can hit my test URL it over and over with curl, and every 
request gets handled as a dynamic request (there are no Cache-control 
headers in these requests).  So, either I'm doing something wrong, or the 
edge cache doesn't deem this URL's response worthy of caching.  How can I 
tell?

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] controlling cache control headers for images served through ImagesService

[AndyD]

I'm storing images in GCS and serving them via a serving URL generated by 
the ImagesService.  One problem I've noticed is that although I've 
specified cache control metadata on the GCS objects (e.g. max-age), the 
cache control headers in the response from the image serving URL seem to be 
ignoring them.  For example, I set the max-age to 31449600 in GCS (364 
days), and if I fetch the image directly from GCS, the Cache-control 
headers reflects that, but if I fetch the image from the ImagesServices 
serving URL, the Cache-control header is always "public, max-age=86400, 
no-transform".

Am I doing something wrong? Is there a way to specify or influence the 
Cache-control header provided by the ImagesService?

Note: I filed an issue in the issue tracker for this: 
https://code.google.com/p/googleappengine/issues/detail?id=10524

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] BlobInfo entities in datastore for files in Google Cloud Storage?

[AndyD]

My GAE application allows clients to upload files.  The app implements the 
upload functionality using upload URLs generated by the BlobstoreService, 
configured to store the files in GCS (not the Blobstore).  When my app 
receives the post-upload callback request from the BlobstoreService, I 
extract the filename of the uploaded file, and save it in an entity in the 
datastore, which allows later access to the file in GCS.  

Since I'm not storing the files in the Blobstore, I'm surprised to find 
that each file stored in GCS has a corresponding BlobInfo entity in the 
datastore.  Since these seem to be averaging about 2KB in size, it's 
turning out to be quite a bit of storage for something I'm not using.

Is there a way to store files in GCS using the BlobstoreService's upload 
URL without having these BlobInfo entities getting created in the 
datastore?  If not, do I actually need them (can I delete them)?

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] GAE cost projection calculator - anybody got a good spreadsheet template they could share?

[AndyD]

I'm working up a spreadsheet to help project GAE costs for an application. 
 Basically I want to do stuff like:

   - enumerate all the kinds of requests the service will be receiving, and 
   for each one identify the number of datastore small/read/write operations 
   per request, the number of requests/day, then factor in the daily free 
   quota and the cost per small/read/write op over the quota.  
   - For each request type, estimate the frontend/backend CPU time per 
   request, add it all up per day, apply the quotas, cost per unit time over 
   quota, etc.
   - enumerate all the datastore entity kinds I'll be storing, how big each 
   is on average, how many of them there will be, then factor in the free 
   quota and the monthly charge per GB over that quota
   - and so on for things like network trafffic, all the other APIs that 
   have free quotas and over-quota costs, you get the idea...


So... obviously I'm not the first person to ever do this.  If you've put 
together a useful spreadsheet template, how about sharing?

-Andy Dennie


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Can't create application "invalid name" no matter what I choose..

[AndyD]

My experience has been that it doesn't matter what name you pick, if it 
doesn't have a hyphen, it won't be accepted.  I'm sure there are other 
magic characters, but generally speaking, a name like 
jknvajkdhnilutysljknfvbkljhsldfkjtysuiehlvks won't work, while 
a-jknvajkdhnilutysljknfvbkljhsldfkjtysuiehlvks is just fine.

-Andy

On Friday, October 11, 2013 9:15:06 PM UTC-4, Vinny P wrote:
>
> On Fri, Oct 11, 2013 at 5:27 AM, Ben Ritchie 
> 
> > wrote:
>
>> No matter what I choose as the app name it says "invalid"
>>
> The application name "asdfafasdf" shown in your screenshot seems to be 
> legitimately invalid, as I'm getting the same error: 
> http://imgur.com/oilppL5
>
> You just have to play around a little with names to get a valid 
> application ID. Remember that app IDs can't share names with an 
> already-existing Gmail account, or previously-deleted application IDs.
> -
> -Vinny P
>
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Transfer application to other account

[AndyD]

OK, it's been 3 years.  Anybody know if this policy is still in effect? 
 I'm working with a startup and I want to get them off the ground with GAE 
by creating the application for them and then transferring it to them 
later, but I don't want to "use up" one of my allotted 10 applications 
permanently.

-Andy

On Tuesday, October 19, 2010 11:39:36 AM UTC-4, Nick Johnson (Google) wrote:
>
> Hi Nikolay,
>
> You are able to create 10 apps per account. What you do with those apps 
> afterwards - including transferring them to other accounts - is immaterial 
> to your app creation quota.
>
> -Nick Johnson
>
> On Sun, Oct 10, 2010 at 9:14 PM, Nikolay Tenev 
> 
> > wrote:
>
>>  Hi
>>
>> I was created application in app engine and today I want to give 
>> ownership of application to another account. I invite the other developer, 
>> he accept invitation and then delete my account as application developer 
>> but ... in my app engine account I still view "You have 7 applications 
>> remaining." (I have 2 other working) and the other developer, in dashboard, 
>> have transfered application but the text is "You have 10 applications 
>> remaining." It is true that I create the application but is it normal to 
>> still count on me even I'm not developer/administrator on application any 
>> more ?
>>
>> Regards !
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> To unsubscribe from this group, send email to 
>> google-appengi...@googlegroups.com .
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
> -- 
> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. 
> :: Registered in Dublin, Ireland, Registration Number: 368047
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 
> 368047
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Cannot read data from datastore that has just been persisted

[AndyD]

This might 
help: 
http://stackoverflow.com/questions/12358372/how-to-junit-test-entity-persistence-with-hrd-w-o-parent-relationship

If you're using Eclipse, the "unapplied job percentage" is specified in the 
App Engine tab of the Run Configurations dialog.

-Andy

On Wednesday, January 9, 2013 10:36:47 AM UTC-5, Dirk Vranckaert wrote:
>
> I'm building some kind of synchronization service on AE (I'm still in my 
> first month of GA experience) and the first step in my sync process is to 
> sync a bunch of projects.
> I do checks if I have to persist the project, update the project or just 
> ignore the incoming project.
> Next step is to sync tasks. But the tasks have a reference to a project 
> (not necessarily the one of the projects synced before, but it can). And 
> there the problem happens.
>
> So in my test case I have persisted a project, then I do a lookup to find 
> the project for which I want to sync a task. But the project is not found 
> because it has just been created in step 1 and thus not comitted yet. At 
> least I think that is the reason.
>
> If I do another test-case in which a task is synced for a project that is 
> already available in the database, the test case succeeds. So I'm not 
> exactly sure what the exact problem is, but if I just persisted an object I 
> cannot load it a second later...
> So for me it must be or caching related or the transaction because it's 
> not yet comitted...
>
> Any help would be appreciated.. :-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/U312CcVNnykJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: will a read-only datastore transaction fail on commit if the entity group is modified during the transaction?

[AndyD]

On Tuesday, October 23, 2012 5:13:28 PM UTC-4, Andrew Mackenzie wrote:
>
> After reading the documentation that, yes you will get an exception (or if 
> it is the write that has arrived "late" then it will get the exception on 
> committing...) - otherwise, what would be the point of using an exception 
> around any data store reads?


I'm not positive, but I doubt the write transaction ("B" above) would get 
an exception if it committed last, since the read transaction didn't write 
anything.  The point of using a read transaction is to get a consistent 
view of the data store over the course of multiple reads, instead of 
getting the state as each individual read is made.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ruJBrP2Lr5oJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] is it possible to generate web.xml and appengine-web.xml from app.yaml locally?

[AndyD]

I've got a GAE/Java app an I'm experimenting with using app.yaml instead of 
manually maintaining web.xml and appengine-web.xml.  However, I can't 
figure out what I need to do to generate the xml files from the yaml file 
in my build.  The GPE seems to really want the xml files for local 
execution.  What's the secret?

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/SvxhpOu3R8MJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Which task queue exceptions raised from Queue.add() make sense to retry?

[AndyD]

Thanks, Robert.

FYI, you're right, a little code spelunking 
in
 
QueueApiHelper.java shows that in Java, the tombstone error becomes a 
TaskAlreadyExistsException:

case TASK_ALREADY_EXISTS:

return new TaskAlreadyExistsException("Task name already exists : " + 
detail);  case TOMBSTONED_TASK:return new 
TaskAlreadyExistsException("Task name is tombstoned : " + detail);
The same file also shows that QueueFailureException is basically a generic 
catch-all for anything not handled explicitly:

 default:return new QueueFailureException("Unspecified error (" + 
errorCode + ") : " + detail);

So that seems worth a retry if it pops up.

Regarding TransactionalTaskException, it's used for any Datastore errors 
that arise.  I'm not sure if those would only arise when the task is part 
of an enclosing transaction or not, but based on the name, I'm going to 
guess that's the case.  I got a little lost trying to trace back to the 
possible sources where that could be raised.

Similarly I wasn't able to track down the source of an 
InternalFailureException.  If anybody has an additional insight into those, 
please chime in.

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/60AaEXo1XrQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Which task queue exceptions raised from Queue.add() make sense to retry?

[AndyD]

[hmm, this 
questiondidn't
 attract any interest on StackOverflow..., thought I'd give it a try 
here]

I'm looking to add some defensive exception handling/retry logic around my 
(Java) code that enqueues tasks in App Engine. In reviewing the 
documentation for potential exceptions raised, I see several candidates 
that might make sense to catch and handle with a retry, but the 
documentation is a bit vague, so I thought I'd see what others have found 
to be worthwhile.

Here are the ones I'm looking at, and their descriptions from the docs:

   - InternalFailureException - Internal task queue error.
   - TransientFailureException - Intermittent failure.
   - QueueFailureException - Unspecified queue failure.
   - TransactionalTaskException - Queue operation failure caused by 
   Datastore exception.

The only one for which the docs explicitly suggest that retrying might be 
helpful is TransientFailureException.

Also, regarding TransactionalTaskException, is this something that would 
only arise when enlisting a task in an enclosing datastore transaction, or 
could this arise when enqueuing a standalone task?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/JV4cR4o2BNYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: App Engine SDK 1.6.2 is out!

[AndyD]

Personally, I find Brandon's input here to be useful.  The lengthy "tails" 
are not my favorite thing, but I'd rather have them than his absence.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/h9z4VdjTshAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] will a read-only datastore transaction fail on commit if the entity group is modified during the transaction?

[AndyD]

Example case:

time 0: transaction A begins
time 1: transaction B begins
time 2: transaction A reads entity 123
time 3: transaction B writes entity 123 (or entity 456, in the same entity 
group as 123)
time 4: transaction B is committed
time 5: transaction A is committed

There are no write operations in transaction A.  Does the commit of A fail 
with a ConcurrentModificationException?  

-AndyD

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/75vJbRtn4mYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] datastore: key properties vs. Long ID properties for storing references to other entities

[AndyD]

What are the pros/cons of using the datastore native "key" property type 
for storing a reference to another entity, vs. using the Long property type 
to hold the other object's ID?  

A few differences come to mind immediately, but I'm not sure if they're 
significant.

   1. keys are longer and take up more space on disk, in memory, on the 
   wire, etc.
   2. With Long properties, whenever you want to fetch the related object, 
   you first need to generate its Key using a KeyFactory.  I haven't timed it, 
   but my assumption is that the key generation performance overhead is 
   trivial, since it's just a string formatting exercise.
   3. With Long properties, if the referenced object is parented and the 
   parent ID of the referenced object is not implied or otherwise available 
   from context  (e.g. known to be the same parent as the referencing object), 
   it would be necessary for the referencing object to store the parent ID of 
   the referenced object also.  So in that case, two properties would be 
   needed instead of one.  Is that significant (assuming they're not indexed)?

In all my use cases, I know the type of the related object (and thus its 
kind), so generating the key is trivial and can be pushed out of sight into 
helper methods in base classes, so I'm not concerned about the boilerplate. 
 Parented object references don't seem to present a problem, either, since 
you could just as easily store the parent Id in a Long field also, if it is 
not implied by the context of the referencing object.

In a nutshell, it seems to me that a Key is just a formatted string that 
encapsulates the reference information, but there's no magic there.

Are there benefits to using the Key property type that I'm overlooking?

-AndyD

P.S. some other discussion on this subject:
http://stackoverflow.com/questions/5312904/key-or-long-for-id-in-google-app-engine-jdo
 
http://stackoverflow.com/questions/5553702/app-engine-identifier-key-vs-id 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/d-OUIqZ7ua0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Once-only transactions

[AndyD]

Yes, I see what you're saying.  But in looking at both your and Robert's 
approaches, I still have questions.

Jeff, how does your approach avoid executing the operation twice?  Let's 
say there's a web page with a submit button that triggers the process, and 
the user clicks that button twice.  The algorithm you outlined...

Create TxnID entity instance
Repeat until success {
 * start txn
 * load TxnID (if null, return success immediately)
 * debit $5 from AccountA
 * credit $5 to AccountB
 * delete TxnID
 * commit txn
} 
 
...if run serially, would do the operation twice, wouldn't it?  The first 
execution would create the TxnID entity, then in the loop, it would load 
that entity successfully, perform the operation, then delete the TxnID 
entity.  Then, when the 2nd button click is handled, the operation would be 
be performed again.

Robert, I didn't quite follow your algorithm.  Does your marker indicate 
that the operation has not yet been performed?  If so, shouldn't 2b say "if 
the marker is *not* found, abort?  Also, is there an implied retry 
mechanism (e.g. a task) encompassing step 2?

It seems to me there are multiple challenges:
1) eliminate the potential of concurrency
2) retry on failure
3) only do the operation once

If we don't eliminate concurrency, then two parallel threads, in the same 
or different instances, could find the operation not yet completed and both 
perform it.   If we don't indicate (by the presence or absence of a marker) 
that the operation has been completed, then two serial executions would 
both perform it.  And if we don't have a retry mechanism, there is a risk 
that the operation won't be performed at all. 

It still seems to me that the task queue approach addresses 1 (by not 
dequeing a task to two handlers simultaneously and also not accepting a 
duplicate task into the queue) and 2 (by automatically retrying on 
failure), but as Jeff pointed out, not 3.  If you add an "operation not 
performed yet" marker entity to the picture that gets deleted in the same 
transaction as the run-once operation, does that cover all the bases? 

Just to make it interesting, what if the operation to be performed once was 
*not* a datastore operation?  What if it was a call to an external system 
(like a credit card processing service)?  You could successfully make the 
external call, but fail to write the "operation complete" entity.

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/YU3N7fnt8rAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Once-only transactions

[AndyD]

Yes, I see what you're saying.  But in looking at both your and Robert's 
approaches, I still have questions.

Jeff, how does your approach avoid executing the operation twice?  Let's 
say there's a web page with a submit button that triggers the process, and 
the user clicks that button twice.  The algorithm you outlined...

Create TxnID entity instance
Repeat until success {
 * start txn
 * load TxnID (if null, return success immediately)
 * debit $5 from AccountA
 * credit $5 to AccountB
 * delete TxnID
 * commit txn
} 
 
...if run serially, would do the operation twice, wouldn't it?  The first 
execution would create the TxnID entity, then in the loop, it would load 
that entity successfully, perform the operation, then delete the TxnID 
entity.  Then, when the 2nd button click is handled, the operation would be 
be performed again.

Robert, I didn't quite follow your algorithm.  Does your marker indicate 
that the operation has not yet been performed?  If so, shouldn't 2b say "if 
the marker is *not* found, abort?  Also, is there an implied retry 
mechanism (e.g. a task) encompassing step 2?

It seems to me there are multiple challenges:
1) eliminate the potential of concurrency
2) retry on failure
3) only do the operation once

If we don't eliminate concurrency, then two parallel threads, in the same 
or different instances, could find the operation not yet completed and both 
perform it.  If we don't indicate (by the presence or absence of a marker) 
that the operation has been completed, then two serial executions would 
both perform it.  And if we don't have a retry mechanism, there is a risk 
that the operation won't be performed at all.

It still seems to me that the task queue approach addresses 1 (by not 
dequeing a task to two handlers simultaneously and also not accepting a 
duplicate task into the queue) and 2 (by automatically retrying on 
failure), but as Jeff pointed out, not 3.  If you add an "operation 
complete" marker entity to the picture that gets written in the same 
transaction as the run-once operation, does that cover all the bases?  The 
marker would still need to be cleaned up at some later point, of course.

Just to make it interesting, what if the operation to be performed once was 
*not* a datastore operation?  What if it was a call to an external system 
(like a credit card processing service)?  You could successfully make the 
external call, but fail to write the "operation complete" entity.

-Andy

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Mamb76pc7akJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Once-only transactions

[AndyD]

Yes, I see what you're saying.  But in looking at both your and Robert's 
approaches, I still have questions.

Jeff, how does your approach avoid executing the operation twice?  Let's 
say there's a web page with a submit button that triggers the process, and 
the user clicks that button twice.  The algorithm you outlined...

Create TxnID entity instance
Repeat until success {
 * start txn
 * load TxnID (if null, return success immediately)
 * debit $5 from AccountA
 * credit $5 to AccountB
 * delete TxnID
 * commit txn
} 
 
...if run serially, would do the operation twice, wouldn't it?  The first 
execution would create the TxnID entity, then in the loop, it would load 
that entity successfully, perform the operation, then delete the TxnID 
entity.  Then, when the 2nd button click is handled, the operation would be 
be performed again.

Robert, I didn't quite follow your algorithm.  Does your marker indicate 
that the operation has not yet been performed?  If so, shouldn't 2b say "if 
the marker is *not* found, abort?  Also, is there an implied retry 
mechanism (e.g. a task) encompassing step 2?

It seems to me there are multiple challenges:
1) eliminate the potential of concurrency
2) retry on failure
3) only do the operation once

If we don't eliminate concurrency, then two parallel threads, in the same 
or different instances, could find the operation not yet completed and both 
perform it.  If we indicate (by the presence or absence of a marker) that 
the operation has been completed, then two serial executions would both 
perform it.  And if we don't have a retry mechanism, there is a risk that 
the operation won't be performed at all.

It still seems to me that the task queue approach addresses 1 (by not 
dequeing a task to two handlers simultaneously and also not accepting a 
duplicate task into the queue) and 2 (by automatically retrying on 
failure), but as Jeff pointed out, not 3.  If you add an "operation 
complete" marker entity to the picture that gets written in the same 
transaction as the run-once operation, does that cover all the bases?  The 
marker would still need to be cleaned up at some later point, of course.

Just to make it interesting, what if the operation to be performed once was 
*not* a datastore operation?  What if it was a call to an external system 
(like a credit card processing service)?  You could successfully make the 
external call, but fail to write the "operation complete" entity.

-Andy


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/aKzPhfSgF0AJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Once-only transactions

[AndyD]

Would a task queue work here?  You could queue a task with a name that is 
unique to the operation (e.g. leveraging some payment ID, in the example 
case).  If a task with the same name is already in the queue, or was in the 
queue in the last 7 days, GAE won't queue it again.  This gives you the 
run-once, retry, and pruning of old tombstones (in this case, the task 
name) automatically.

Giving credit where due, this approach is taken from Dan Sanderson's 
"Programming Google App Engine" book (O'Reilly).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/VVq0iD0escIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Once-only transactions

[AndyD]

Would a task queue work here?  You could queue a task with a name that is 
unique to the operation (e.g. leveraging some payment ID, in the example 
case).  If a task with the same name is already in the queue, or was in the 
queue in the last 7 days, GAE won't queue it again.  This gives you the 
run-once, retry, and pruning of old 7tombstones (in this case, the task 
name) automatically.

Giving credit where due, this approach is taken from Dan Sanderson's 
"Programming Google App Engine" book (O'Reilly).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/TR3ww5dxVE8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to override Content-Type header when submitting task to task queue?

[AndyD]

That did the trick, thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/U1vQZq3Zlm8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] How to override Content-Type header when submitting task to task queue?

[AndyD]

First off, I'm new to App Engine task queues, and so I may just be going 
against the grain here.  Feel free to redirect me if that's the case.

What I'd like to do is queue a task and supply a JSON payload in the HTTP 
request to the handler

Here's what I'm trying...

TaskOptions task = TaskOptions.Builder
.withUrl("/mytaskhandler")
.header("Content-Type", "application/json")
.payload(jsonBody);
QueueFactory.getDefaultQueue().add(task);

What I'm seeing is that this results in a request that has **both** of the 
following headers:
Content-Type:application/json 
and 
Content-Type:text/plain

That seems bogus, and it's confusing the REST library (Restlet) that I'm 
using to handle the request content negotiation.  Bug?

-AndyD

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/fEC1iFfb2T0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.