[google-appengine] Re: Best practicies for making GAE microservices idempotent

[Attila-Mihaly Balazs]

Just a quick note: while GAE supports named tasks and transactional 
enqueuing of tasks, it does not support the transactional enqueing of named 
tasks.

Just a small detail to be aware of when architecting your application.

Attila

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/08ac4874-f095-478b-9f6e-c2ce7c2d3d24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: gae php mail quota too low, error?

['Kenworth (Google Cloud Platform)' via Google App Engine]

Google Groups discussion forum is meant for open-ended discussions. This 
issue is leading to an increasingly specific situation and may potentially 
qualify as a defect on the platform. As such, I would recommend you submit 
a public issue tracker  which most likely 
ask for your project details to further investigate the issue. You can 
refer to this thread as a background to the issue tracker and feel free to 
post here any updates that may help community users experiencing same issue.

I look forward to your issue report.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1bac0c6e-0fd8-4b80-a04c-ce28352f288d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: gae php mail quota too low, error?

[Jukka Hautakorpi]

Please Google check this problem out! It must be a bug in the quota limits! 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/bfbffa76-d61b-4209-9019-06534b7d480d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Firewall - Allowing Access for PubSub Push Subscriptions?

['George (Cloud Platform Support)' via Google App Engine]

Hello JP, 

The architecture for push subscriptions does not seem to require such a 
magic URL as the in case of UrlFetch, as there is no user-configurable 
firewall involved in the delivery path. "The Pub/Sub server sends any 
messages for your subscription to the webhook address you have configured. 
" In other words the firewall you have in mind would not affect your 
webhook directly. You can read more related detail on the "Push Subscriber 
Guide" documentation page , 
"Receiving Push Messages" subtitle. 

To directly reply to your question, the architectural use of a webhook 
makes extra firewall configuration unnecessary, in this case.  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5846d492-af90-4ff0-90cb-a5da88af9c92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Best practicies for making GAE microservices idempotent

[Jason Collins]

Sorry, I don't have operational experience with Cloud Spanner. It didn't 
exist when Fantasm was written.


On Monday, 9 October 2017 07:32:13 UTC-7, Nilson Pontello wrote:
>
> Thanks for the clarification, It looks like I will have to keep hitting 
> datastore for my use case.
>
> BTW: Can Fantasm benefit from making lock transactions using cloud spanner 
>  instead of datastore? or  is 
> datastore faster for small transactions like the ones performed by lock.py?
>
> Thanks
>
> On Monday, 9 October 2017 10:28:04 UTC-3, Jason Collins wrote:
>>
>> No. You can't make that assumption about task queue delivery, though it's 
>> very, very good. This is why Fantasm had to go the extra distance with 
>> Datastore transactions. 
>>
>> BTW, Fantasm allows you to turn off the Datastore run-once check and rely 
>> solely on task queue because in practice, task queue does a great job. Just 
>> not a perfect one. So you have to choose between the small performance hit 
>> of Datastore transaction, or very small chance of duplicate task delivery. 
>>
>> BTW I can't recall the named task deduplication ever failing when 
>> enqueuing a task. 
>> On Mon, Oct 9, 2017 at 4:48 AM Nilson Pontello  
>> wrote:
>>
>>> Thanks Jason/George,
>>>
>>> My code does a job very similar to lock.py 
>>>  
>>> (this is what I am trying to avoid).
>>>
>>> So it looks like the secret is around task queues. Can I assume that 
>>> enqueued tasks will be delivered *just once *(if my endpoint returns 
>>> 200 OK)? 
>>>
>>> If yes then my problem is solved. But if it behaves similar to pubsub 
>>> which guarantees  "At-Least-Once Delivery" then it will be useless for my 
>>> case.
>>>
>>> BTW: If those named tasks are hitting datastore for deduplication then 
>>> lock.py's approach is still valid and its performance will be as good as 
>>> named tasks.
>>>
>>> Thanks
>>>
>>>
>>> On Saturday, 7 October 2017 23:45:25 UTC-3, Jason Collins wrote:

 Fantasm! That's a blast from the past!

 I'm one of the original authors of that package. It does indeed help to 
 make things idempotent, but at the end of the day, it relies on a 
 datastore 
 transaction (fronted by memcache for some performance gains): 
 https://github.com/iki/fantasm/blob/master/src/fantasm/lock.py#L150

 Datastore transactions are one of the few things you can "count on" in 
 a distributed system like App Engine.

 Another tool that is often helpful are named tasks, which can help 
 ensure that you don't requeue an already queued task (e.g., in the event 
 of 
 a retry): 
 https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-tasks#naming_a_task

 Finally, keep in mind that you can also enqueue a task transactionally 
 with a Datastore transaction: 
 https://cloud.google.com/appengine/docs/standard/python/ndb/transactions#python_Transactional_task_enqueuing

 With some creativity, you can get a long way with these basic building 
 blocks.
  

 On Friday, 6 October 2017 16:39:14 UTC-7, George (Cloud Platform 
 Support) wrote:
>
> Hello Nilson, 
>
> Among other sources, you may benefit from reading the "Implementing 
> Workflows on Google App Engine with Fantasm" article 
> . Fantasm is an 
> excellent launch pad for building an understanding of how to decompose a 
> workflow into the appropriate chunk size as well as gaining a solid 
> footing 
> in how to build idempotent states.
>
> -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Google App Engine" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/google-appengine/p8Xcap4eaKU/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to 
>>> google-appengi...@googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/4ccbccfa-e90a-4ac0-a125-503ba343e2ab%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at 

Re: [google-appengine] Re: Best practicies for making GAE microservices idempotent

[Nilson Pontello]

Thanks for the clarification, It looks like I will have to keep hitting 
datastore for my use case.

BTW: Can Fantasm benefit from making lock transactions using cloud spanner 
 instead of datastore? or  is datastore 
faster for small transactions like the ones performed by lock.py?

Thanks

On Monday, 9 October 2017 10:28:04 UTC-3, Jason Collins wrote:
>
> No. You can't make that assumption about task queue delivery, though it's 
> very, very good. This is why Fantasm had to go the extra distance with 
> Datastore transactions. 
>
> BTW, Fantasm allows you to turn off the Datastore run-once check and rely 
> solely on task queue because in practice, task queue does a great job. Just 
> not a perfect one. So you have to choose between the small performance hit 
> of Datastore transaction, or very small chance of duplicate task delivery. 
>
> BTW I can't recall the named task deduplication ever failing when 
> enqueuing a task. 
> On Mon, Oct 9, 2017 at 4:48 AM Nilson Pontello  > wrote:
>
>> Thanks Jason/George,
>>
>> My code does a job very similar to lock.py 
>>  
>> (this is what I am trying to avoid).
>>
>> So it looks like the secret is around task queues. Can I assume that 
>> enqueued tasks will be delivered *just once *(if my endpoint returns 200 
>> OK)? 
>>
>> If yes then my problem is solved. But if it behaves similar to pubsub 
>> which guarantees  "At-Least-Once Delivery" then it will be useless for my 
>> case.
>>
>> BTW: If those named tasks are hitting datastore for deduplication then 
>> lock.py's approach is still valid and its performance will be as good as 
>> named tasks.
>>
>> Thanks
>>
>>
>> On Saturday, 7 October 2017 23:45:25 UTC-3, Jason Collins wrote:
>>>
>>> Fantasm! That's a blast from the past!
>>>
>>> I'm one of the original authors of that package. It does indeed help to 
>>> make things idempotent, but at the end of the day, it relies on a datastore 
>>> transaction (fronted by memcache for some performance gains): 
>>> https://github.com/iki/fantasm/blob/master/src/fantasm/lock.py#L150
>>>
>>> Datastore transactions are one of the few things you can "count on" in a 
>>> distributed system like App Engine.
>>>
>>> Another tool that is often helpful are named tasks, which can help 
>>> ensure that you don't requeue an already queued task (e.g., in the event of 
>>> a retry): 
>>> https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-tasks#naming_a_task
>>>
>>> Finally, keep in mind that you can also enqueue a task transactionally 
>>> with a Datastore transaction: 
>>> https://cloud.google.com/appengine/docs/standard/python/ndb/transactions#python_Transactional_task_enqueuing
>>>
>>> With some creativity, you can get a long way with these basic building 
>>> blocks.
>>>  
>>>
>>> On Friday, 6 October 2017 16:39:14 UTC-7, George (Cloud Platform 
>>> Support) wrote:

 Hello Nilson, 

 Among other sources, you may benefit from reading the "Implementing 
 Workflows on Google App Engine with Fantasm" article 
 . Fantasm is an 
 excellent launch pad for building an understanding of how to decompose a 
 workflow into the appropriate chunk size as well as gaining a solid 
 footing 
 in how to build idempotent states.

 -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "Google App Engine" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/google-appengine/p8Xcap4eaKU/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to 
>> google-appengi...@googlegroups.com .
>> To post to this group, send email to google-a...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/4ccbccfa-e90a-4ac0-a125-503ba343e2ab%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/03b67df6-f61d-41db-9847-1f839d201ee7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Best practicies for making GAE microservices idempotent

[Jason Collins]

No. You can't make that assumption about task queue delivery, though it's
very, very good. This is why Fantasm had to go the extra distance with
Datastore transactions.

BTW, Fantasm allows you to turn off the Datastore run-once check and rely
solely on task queue because in practice, task queue does a great job. Just
not a perfect one. So you have to choose between the small performance hit
of Datastore transaction, or very small chance of duplicate task delivery.

BTW I can't recall the named task deduplication ever failing when enqueuing
a task.
On Mon, Oct 9, 2017 at 4:48 AM Nilson Pontello 
wrote:

> Thanks Jason/George,
>
> My code does a job very similar to lock.py
> 
> (this is what I am trying to avoid).
>
> So it looks like the secret is around task queues. Can I assume that
> enqueued tasks will be delivered *just once *(if my endpoint returns 200
> OK)?
>
> If yes then my problem is solved. But if it behaves similar to pubsub
> which guarantees  "At-Least-Once Delivery" then it will be useless for my
> case.
>
> BTW: If those named tasks are hitting datastore for deduplication then
> lock.py's approach is still valid and its performance will be as good as
> named tasks.
>
> Thanks
>
>
> On Saturday, 7 October 2017 23:45:25 UTC-3, Jason Collins wrote:
>>
>> Fantasm! That's a blast from the past!
>>
>> I'm one of the original authors of that package. It does indeed help to
>> make things idempotent, but at the end of the day, it relies on a datastore
>> transaction (fronted by memcache for some performance gains):
>> https://github.com/iki/fantasm/blob/master/src/fantasm/lock.py#L150
>>
>> Datastore transactions are one of the few things you can "count on" in a
>> distributed system like App Engine.
>>
>> Another tool that is often helpful are named tasks, which can help ensure
>> that you don't requeue an already queued task (e.g., in the event of a
>> retry):
>> https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-tasks#naming_a_task
>>
>> Finally, keep in mind that you can also enqueue a task transactionally
>> with a Datastore transaction:
>> https://cloud.google.com/appengine/docs/standard/python/ndb/transactions#python_Transactional_task_enqueuing
>>
>> With some creativity, you can get a long way with these basic building
>> blocks.
>>
>>
>> On Friday, 6 October 2017 16:39:14 UTC-7, George (Cloud Platform Support)
>> wrote:
>>>
>>> Hello Nilson,
>>>
>>> Among other sources, you may benefit from reading the "Implementing
>>> Workflows on Google App Engine with Fantasm" article
>>> . Fantasm is an
>>> excellent launch pad for building an understanding of how to decompose a
>>> workflow into the appropriate chunk size as well as gaining a solid footing
>>> in how to build idempotent states.
>>>
>>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/p8Xcap4eaKU/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/4ccbccfa-e90a-4ac0-a125-503ba343e2ab%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAL_dBeZE9viV%3D2%2BnP2TqRdk9n-1TBYfGWzv3ST%3D0rkkd%2BzcE_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Best practicies for making GAE microservices idempotent

[Nilson Pontello]

Thanks Jason/George,

My code does a job very similar to lock.py 
 (this 
is what I am trying to avoid).

So it looks like the secret is around task queues. Can I assume that 
enqueued tasks will be delivered *just once *(if my endpoint returns 200 
OK)? 

If yes then my problem is solved. But if it behaves similar to pubsub which 
guarantees  "At-Least-Once Delivery" then it will be useless for my case.

BTW: If those named tasks are hitting datastore for deduplication then 
lock.py's approach is still valid and its performance will be as good as 
named tasks.

Thanks


On Saturday, 7 October 2017 23:45:25 UTC-3, Jason Collins wrote:
>
> Fantasm! That's a blast from the past!
>
> I'm one of the original authors of that package. It does indeed help to 
> make things idempotent, but at the end of the day, it relies on a datastore 
> transaction (fronted by memcache for some performance gains): 
> https://github.com/iki/fantasm/blob/master/src/fantasm/lock.py#L150
>
> Datastore transactions are one of the few things you can "count on" in a 
> distributed system like App Engine.
>
> Another tool that is often helpful are named tasks, which can help ensure 
> that you don't requeue an already queued task (e.g., in the event of a 
> retry): 
> https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-tasks#naming_a_task
>
> Finally, keep in mind that you can also enqueue a task transactionally 
> with a Datastore transaction: 
> https://cloud.google.com/appengine/docs/standard/python/ndb/transactions#python_Transactional_task_enqueuing
>
> With some creativity, you can get a long way with these basic building 
> blocks.
>  
>
> On Friday, 6 October 2017 16:39:14 UTC-7, George (Cloud Platform Support) 
> wrote:
>>
>> Hello Nilson, 
>>
>> Among other sources, you may benefit from reading the "Implementing 
>> Workflows on Google App Engine with Fantasm" article 
>> . Fantasm is an 
>> excellent launch pad for building an understanding of how to decompose a 
>> workflow into the appropriate chunk size as well as gaining a solid footing 
>> in how to build idempotent states.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4ccbccfa-e90a-4ac0-a125-503ba343e2ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Help me mail

[timh]

You could set up a relay proxy for SMTP running on a high port that 
connects to your customers SMTP server and connect to that.

T

On Friday, October 6, 2017 at 9:05:36 PM UTC+8, Karl Tinawi wrote:
>
> Hi George/all,
>
> We're a small startup and we've invested in Google Cloud infrastructure 
> for hosting our platform. We offer a B2B service and it is imperative for 
> our customers to send emails to their clients using our platform (thus 
> using their SMTP).
>
> We're at a critical juncture in our early days and this is a showstopper 
> for us. Our users need to be able to send emails from their mail servers 
> using SMTP, whether it be Gmail, Yahoo, FastMail, or whoever. Using 
> SendGrid or MailGun doesn't allow us to fulfil this requirement.
>
> There was a post from Paul Nash, here:
>
> https://googlecloudplatform.uservoice.com/forums/302595-compute-engine/suggestions/10079937-send-and-receive-email-using-tcp-smtp-imap-ports
> that suggested that we could request SMTP and IMAP ports to be unblocked?
>
> I'd be very grateful for any advice as I'm currently looking at all 
> possible alternatives as we need to go live with this pretty soon.
>
> Thank you kindly,
> Karl
>
> On Tuesday, August 8, 2017 at 3:12:20 PM UTC+1, George (Cloud Platform 
> Support) wrote:
>>
>> Google Compute Engine does not allow outbound connections on ports 25, 
>> 465, and 587. By default, these outbound SMTP ports are blocked because of 
>> the large amount of abuse these ports are susceptible to. You can find more 
>> information on how to send email from a VM and what services to use on the 
>> "Sending Email from an Instance" documentation page 
>> . 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6aec80ee-e6a6-4a60-9168-b8cc840aabfa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.