[google-appengine] Re: Cloud Functions - safe for a key?

[Andrew Davies]

Thanks @Elliot!

To answer those questions:
- It's entirely possible that I'm using Cloud Functions for a purpose they 
weren't designed for. There are Python scripts I want to run on a scheduled 
basis (once a day) via PubSub which access some data, parse it to a table, 
then load to BigQuery. Using service accounts is an easy way to manage 
authorisation and seem designed for this sort of task. 
- I'm still learning and not sure of the best way to convert a locally 
running script to being cloud based. Security is less of a consideration 
when running locally but I'm aware that storing private keys in plain text 
in a public cloud isn't a great idea, which is why I was curious as to what 
I might need to consider or if my concerns were unfounded.
- Security concerns are that someone may be able to access the plain text 
key file. Again, not that this service account has access to highly 
confidential information, but I like taking reasonable precautions.

I think the method you described should work fine though! Thank you. :)
On Tuesday, 12 January 2021 at 6:20:03 am UTC+10 Elliott (Cloud Platform 
Support) wrote:

> Hello,
>
> I’d like to address your questions but before I do, I would like you to 
> know that there would be no by and large answer on security. It depends on 
> your requirements. Depending on the answers to some questions, I would be 
> guiding you to a point where you may benefit.
>
> To begin, you mentioned that you wish to use a service key to your Cloud 
> Function.
>
> Is there a reason why you chose this approach in that a Cloud Function has 
> a service account?
> Why are you using a Cloud Function to store your key?
> What specific security concerns do you have? 
>
> I was able to look further and was suggested this thread here 
> 
>  
> that has several views that may assist you. Perhaps it may provide you with 
> further insight.
>
> Encrypt your service account credential json file using Cloud KMS/vault 
> and upload it to Cloud Storage.
> Fetch service account credential json file from Cloud Storage and decrypt 
> it using a Cloud KMS service account which has encrypt/decrypt permission.
> Parse service account credential json file at runtime and get private_key, 
> client_email and ProjectId.
> Pass these three secret variables to the client library
> We must not store secret things like plain text, e.g cloud function 
> environment variables.
>
> I hope this helps.
>
>
> On Monday, January 11, 2021 at 9:16:31 AM UTC-5 ajdav...@gmail.com wrote:
>
>> Hi there,
>>
>> I'm pretty new so please pardon if this is a stupid question. I seem to 
>> get led astray when looking for security info too. 
>>
>> Is it safe to store a key as part of a package published to Google Cloud 
>> Functions? Specifically this would be a key used for a Google service 
>> account as part of a GCP project. 
>>
>> If not natively, is there anything I could do to make it 'safe' or is it 
>> generally a bad idea all around?
>>
>> Nothing super sensitive to be done by the function but still would prefer 
>> it to run with minimal risk of interference. 
>>
>> Thanks!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/48f68a9d-7c6a-402f-ba4e-6c6a1febb808n%40googlegroups.com.

[google-appengine] Re: BS Limits of Cloud Firestore - And the Inability to Horizontally Scale

[Kaan]

Thanks, I guess this issue I created sums it up 
then: https://issuetracker.google.com/issues/177236402

On Tuesday, January 12, 2021 at 12:48:59 AM UTC+3 Elliott (Cloud Platform 
Support) wrote:

> Hello,
>
> I understand that you are concerned about 10,000/s writes per second limit 
> to the entire Cloud Firestore database. I was able to confirm this behavior 
> here  
> and I do apologize for the frustration this is causing you. All projects 
> grow and because of this, limits will be reached as time goes on. I have 
> consulted the information I have and found an opportunity provide you with 
> an existing feature request  
> but please note that there is no ETA no guarantee that it will be 
> implemented.
>
> You have suggested other improvements you wish to make and the best way 
> forward is to use Google Issue Tracker 
>  to initiate the 
> process.
>
> Again, I know you put a lot of thought when you reached out to us. If you 
> require any further assistance please let us know.
>
>
>
> On Saturday, January 9, 2021 at 10:40:54 AM UTC-5 kaan...@gmail.com wrote:
>
>> A bit constructive criticism, my solution approach would be
>>
>> *1)* Make Datastore Mode default, don't confuse anyone
>> *2)* Make it easier to create and connect to other databases
>> *3)* Make it possible to change the default database (People could use 
>> Datastore Mode, create a Native one, migrate their data within their app, 
>> switch the Native to be the default, easy peasy, or they could just use 
>> them both)
>> *4)* (Comes with 2) Make sure new developers have a development database 
>> too, a live database they use for development, and maybe set 
>> the GOOGLE_APPLICATION_CREDENTIALS etc. automatically, or make things more 
>> practical
>>
>> I currently created another project for development, and using it's 
>> Datastore mode for the time being - it's my step *(4)* - so it was 
>> needed anyway
>>
>> *(4) *Goes into the territory of the currently "you're on your own" 
>> local development approach - just hold their hands, most integrations are 
>> there anyway, back in the day they were local, with more complicated 
>> products, it's obviously separate  live databases, buckets etc. to be used 
>> for development, that they can access locally (when online)
>>
>> Still, it's* really sad* that I can't use a new product, maybe I'm 
>> autistic and I can't get over this fact, but *is there really a 
>> database-wide limit of 10,000/s writes?* If I commit to the old 
>> Datastore Mode, and if this limit gets removed in the future, I'll 
>> literally cry
>>
>> On Saturday, January 9, 2021 at 5:40:57 PM UTC+3 Kaan wrote:
>>
>>> I guess if we could just use multiple databases, and also easily change 
>>> the default database, this kind of heartbreak would be prevented
>>>
>>> As I see, I made a mistake by choosing the Native option, my approach is 
>>> to have scalability, and from experience, that 10,000/s limit just makes 
>>> the native mode a good fit for highly agile / practical but small apps, 
>>> rather than anything that processes data or scales
>>>
>>> For example, one could create 10 firestore databases, and just use them 
>>> to broadcast data to 10 million people, for that use case, it's very 
>>> practical
>>>
>>> But for an actual heart of a project, it will just result in a heart 
>>> attack and early death
>>>
>>> Since my databases are empty, I hope I can easily fix the mistake, I 
>>> guess I'll open a ticket, maybe delete/re-create the apps etc.
>>>
>>> On Saturday, January 9, 2021 at 5:25:38 PM UTC+3 Kaan wrote:
>>>
 I probably originally read this article, and since it's always best to 
 use new technologies, I picked the native mode: 
 https://cloud.google.com/datastore/docs/firestore-or-datastore

 In the above article, you can't see the 10,000/s limit, which is 
 basically a *MAJOR* bottleneck, I mean, even in my 10 year old app, I 
 do more than 10,000/s writes throughput-wise, with mapreduce-like tasks ...

 On Saturday, January 9, 2021 at 5:20:00 PM UTC+3 Kaan wrote:

> In this age, 1,000,000 concurrent connections is the dream, so let's 
> ignore that
>
> But, *10,000/s writes per second to the entire database* - what kind 
> of a funny limit is that?
>
> I mean it's really agony having to make binding database decisions 
> like this, and after making these decisions, discovering (or in my case, 
> I 
> assume re-discovering, I don't recall seeing this absurd limit when I 
> first 
> selected Cloud Firestore) these limits
>
> For get 1,000,000 concurrent users, with 100,000 active 
> users/players/clients-whatever - that *10,000/s* limit just takes App 
> Engine from infinitely scalable to a joke
>
> We can't horizontally scale 

[google-appengine] Re: BS Limits of Cloud Firestore - And the Inability to Horizontally Scale

['Elliott (Cloud Platform Support)' via Google App Engine]

Hello,

I understand that you are concerned about 10,000/s writes per second limit 
to the entire Cloud Firestore database. I was able to confirm this behavior 
here  
and I do apologize for the frustration this is causing you. All projects 
grow and because of this, limits will be reached as time goes on. I have 
consulted the information I have and found an opportunity provide you with 
an existing feature request  but 
please note that there is no ETA no guarantee that it will be implemented.

You have suggested other improvements you wish to make and the best way 
forward is to use Google Issue Tracker 
 to initiate the 
process.

Again, I know you put a lot of thought when you reached out to us. If you 
require any further assistance please let us know.



On Saturday, January 9, 2021 at 10:40:54 AM UTC-5 kaan...@gmail.com wrote:

> A bit constructive criticism, my solution approach would be
>
> *1)* Make Datastore Mode default, don't confuse anyone
> *2)* Make it easier to create and connect to other databases
> *3)* Make it possible to change the default database (People could use 
> Datastore Mode, create a Native one, migrate their data within their app, 
> switch the Native to be the default, easy peasy, or they could just use 
> them both)
> *4)* (Comes with 2) Make sure new developers have a development database 
> too, a live database they use for development, and maybe set 
> the GOOGLE_APPLICATION_CREDENTIALS etc. automatically, or make things more 
> practical
>
> I currently created another project for development, and using it's 
> Datastore mode for the time being - it's my step *(4)* - so it was needed 
> anyway
>
> *(4) *Goes into the territory of the currently "you're on your own" local 
> development approach - just hold their hands, most integrations are there 
> anyway, back in the day they were local, with more complicated products, 
> it's obviously separate  live databases, buckets etc. to be used for 
> development, that they can access locally (when online)
>
> Still, it's* really sad* that I can't use a new product, maybe I'm 
> autistic and I can't get over this fact, but *is there really a 
> database-wide limit of 10,000/s writes?* If I commit to the old Datastore 
> Mode, and if this limit gets removed in the future, I'll literally cry
>
> On Saturday, January 9, 2021 at 5:40:57 PM UTC+3 Kaan wrote:
>
>> I guess if we could just use multiple databases, and also easily change 
>> the default database, this kind of heartbreak would be prevented
>>
>> As I see, I made a mistake by choosing the Native option, my approach is 
>> to have scalability, and from experience, that 10,000/s limit just makes 
>> the native mode a good fit for highly agile / practical but small apps, 
>> rather than anything that processes data or scales
>>
>> For example, one could create 10 firestore databases, and just use them 
>> to broadcast data to 10 million people, for that use case, it's very 
>> practical
>>
>> But for an actual heart of a project, it will just result in a heart 
>> attack and early death
>>
>> Since my databases are empty, I hope I can easily fix the mistake, I 
>> guess I'll open a ticket, maybe delete/re-create the apps etc.
>>
>> On Saturday, January 9, 2021 at 5:25:38 PM UTC+3 Kaan wrote:
>>
>>> I probably originally read this article, and since it's always best to 
>>> use new technologies, I picked the native mode: 
>>> https://cloud.google.com/datastore/docs/firestore-or-datastore
>>>
>>> In the above article, you can't see the 10,000/s limit, which is 
>>> basically a *MAJOR* bottleneck, I mean, even in my 10 year old app, I 
>>> do more than 10,000/s writes throughput-wise, with mapreduce-like tasks ...
>>>
>>> On Saturday, January 9, 2021 at 5:20:00 PM UTC+3 Kaan wrote:
>>>
 In this age, 1,000,000 concurrent connections is the dream, so let's 
 ignore that

 But, *10,000/s writes per second to the entire database* - what kind 
 of a funny limit is that?

 I mean it's really agony having to make binding database decisions like 
 this, and after making these decisions, discovering (or in my case, I 
 assume re-discovering, I don't recall seeing this absurd limit when I 
 first 
 selected Cloud Firestore) these limits

 For get 1,000,000 concurrent users, with 100,000 active 
 users/players/clients-whatever - that *10,000/s* limit just takes App 
 Engine from infinitely scalable to a joke

 We can't horizontally scale either, not possible to create more 
 databases, but even if you could, the 10,000/s limit would need you to 
 scale everything horizontally, even just authenticating the users etc.

 Also there's the attack possibilities, just 2,000 people could come 
 together, just make regular requests to your 

[google-appengine] Re: Cloud Functions - safe for a key?

['Elliott (Cloud Platform Support)' via Google App Engine]

Hello,

I’d like to address your questions but before I do, I would like you to 
know that there would be no by and large answer on security. It depends on 
your requirements. Depending on the answers to some questions, I would be 
guiding you to a point where you may benefit.

To begin, you mentioned that you wish to use a service key to your Cloud 
Function.

Is there a reason why you chose this approach in that a Cloud Function has 
a service account?
Why are you using a Cloud Function to store your key?
What specific security concerns do you have? 

I was able to look further and was suggested this thread here 

 
that has several views that may assist you. Perhaps it may provide you with 
further insight.

Encrypt your service account credential json file using Cloud KMS/vault and 
upload it to Cloud Storage.
Fetch service account credential json file from Cloud Storage and decrypt 
it using a Cloud KMS service account which has encrypt/decrypt permission.
Parse service account credential json file at runtime and get private_key, 
client_email and ProjectId.
Pass these three secret variables to the client library
We must not store secret things like plain text, e.g cloud function 
environment variables.

I hope this helps.


On Monday, January 11, 2021 at 9:16:31 AM UTC-5 ajdav...@gmail.com wrote:

> Hi there,
>
> I'm pretty new so please pardon if this is a stupid question. I seem to 
> get led astray when looking for security info too. 
>
> Is it safe to store a key as part of a package published to Google Cloud 
> Functions? Specifically this would be a key used for a Google service 
> account as part of a GCP project. 
>
> If not natively, is there anything I could do to make it 'safe' or is it 
> generally a bad idea all around?
>
> Nothing super sensitive to be done by the function but still would prefer 
> it to run with minimal risk of interference. 
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/91624061-208d-4652-93bb-541c5348e061n%40googlegroups.com.

[google-appengine] Re: What's the best way to keep Requirements up-to-date?

['Andrew Cooper' via Google App Engine]

You're welcome. If you have any other questions let us know,
On Monday, January 11, 2021 at 2:49:20 PM UTC+1 Alex wrote:

> Very interesting thank you!
>
> On Friday, January 8, 2021 at 5:08:04 PM UTC+1 anco...@google.com wrote:
>
>> Managing dependencies can be tricky! You might want to check Pipenv as 
>> suggested above as it acts as a package and virtual environment management 
>> tool. It also uses a [Pipfile][1] and a [Pipfile.lock][2] rather than a 
>> requirments.txt file. The advantage of a Pipfile is it will be more 
>> accommodating semantically regarding how you declare your versions. For 
>> example, `pipenv install requests~=1.2` will install version 1.2 and any 
>> minor update, but not 2.0. You can also specify a hash to the 
>>
>> ```
>> $ pipenv install "requests>=1.4"   # install a version equal or larger 
>> than
>> $ pipenv install "requests<=2.13"  # install a version equal or lower
>> $ pipenv install "requests>2.19"   # install 2.19.1 but not 2.19.0
>> ```
>>
>> You can specify `pipenv install -r requirements.txt` to import a 
>> requirements file and create the Pipfile. If you want to upgrade all 
>> packages, you can just run `pipenv update`.
>>
>> [1]: https://github.com/pypa/pipfile
>> [2]: 
>> https://pipenv.pypa.io/en/latest/basics/#example-pipfile-pipfile-lock
>>
>> On Friday, January 8, 2021 at 10:24:00 AM UTC+1 Alex wrote:
>>
>>> I'm on GAE standard but I don't think that makes a difference. The 
>>> thread you share is interesting but goes into the direction of having 
>>> versioned packages in the requirements file for even your dev environment. 
>>>
>>> Trying to be more specific on the workflow I imagine, I would describe 
>>> the following: 
>>> 1- continue to build and deploy my Dev project without versioned 
>>> requirements
>>> 2- when I have a stabilized Dev project build, be able to gather the 
>>> actual package version it was built with. How could I do that even manually?
>>> 3- build staging with a versioned requirements file - versions being 
>>> those gathered at step 2
>>> 4- perform full QA on staging
>>> 5- build prod with a versioned requirements file - versions being those 
>>> gathered at step 2 and those staging QA was performed with
>>>
>>> So a key point seems to be about how can I perform step 2? 
>>>
>>> On Thursday, January 7, 2021 at 4:04:58 PM UTC+1 Elliott (Cloud Platform 
>>> Support) wrote:
>>>
 Hello Alex,

 From your description, you’ve decided not to version your 
 requirements.txt in Python. This is not a crazy approach but there are 
 alternatives. I’m assuming you’re using App Engine Flex. I was able to 
 find 
 a document here 
 
  
 that describes that you do not need to specify a version but indicates why 
 it would work in some cases. The following information that I found highly 
 suggests using a virtual environment in Python.

 It reads:

 Specifying a version is not a requirement though it does help a lot in 
 the future. Some versions of packages will not work well with other 
 packages and their respective versions. It is hard to predict how changes 
 in the future will affect these interrelationships. This is why it is very 
 beneficial to create a snapshot in time (in your requirements.txt) showing 
 which version interrelationships do work.

 There is an approach for best practices in this thread regarding your 
 concern of breaking change in a package update and explains virtual 
 environments:

 it's a good idea to get into the habit of using virtual environments to 
 avoid dependency issues, especially when developing your own stuff. 
 Anaconda offers a simple solution with the conda create command, and 
 virtualenv works great with virtualenvwrapper for a lighter-weight 
 solution. Another solution, pipenv, is quite popular.

 There is further explanation:

 A tip - you should aim to be using a virtual environment for each 
 individual project that you'll be working on. This creates a 'bubble' for 
 you to work within and to install specific package versions in, without it 
 affecting your other projects. It will save you a lot of headaches in the 
 future as your packages and versions will be kept project specific. 

 There is a way in this thread 
 
  
 on how to check if the packages in your requirements.txt are up to date. 
 You may choose to act on them as described.

 $ pip list --outdated

 You have described how you are working with three projects to implement 
 your non-versioning scenario on Google Cloud Platform but I am unclear 
 about your question. Can you tell me more about how you are validating the 
 application and 

[google-appengine] Re: Issues accessing backend API deployed in App Engine

['Elliott (Cloud Platform Support)' via Google App Engine]

Hello,

I understand that when you are placing a request using postman, it remains 
in “sending request” for a long time.

I am unable to find a common issue at this time that would cause the 
behavior you are describing. Also, Google Groups is meant for general 
discussions.

To move this along and narrow down the focus, I would suggest while sending 
your request, open Postman Console (View > Show Postman Console), resend 
the request and check for any error logs in the console. 

This way, we will be able to get extra information that may help isolate 
the problem.


On Monday, January 11, 2021 at 9:16:40 AM UTC-5 song...@gmail.com wrote:

> Hi All,
>
> I have a set of Backend springboot application deployed successfully 
> deployed in the APP engine. However, when i tried to access the one of the 
> backend API in the postman, the request does not return, the postman 
> remains in "sending request" for a long time.
>
> As you can see from the GCP logs below, my backed service is running.
>
> May i know if anyone encounter the same issue before whereby the service 
> is deployed successfully in the APP engine, but request cannot be made and 
> no timeout is thrown either. 
>
> Thanks in advance!
>
> Regards
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b5762c09-14ee-4b19-a2bd-f33f2d4cabben%40googlegroups.com.

Re: [google-appengine] Re: Environment Variables inside cloud_build.yaml + PHP

['Daniel (Google Cloud Support)' via Google App Engine]

The environment variables defined on the cloudbuild.yaml file will be 
available within the instances running your Cloud Build steps. For the 
variables to persist on your App Engine application and be used at runtime 
they'd need to be defined on the app.yaml file. So, as explained on this 
stackoverflow posts [1], [2] you'd need to compile the app.yaml at runtime 
within a Cloud Build step. Additionally, you could always use other 
alternatives to hide your secrets, such as Secret Manager [3] if your 
environment variables contain sensible information you don't feel 
comfortable to be exposed within a file in source control.

[1] 
https://stackoverflow.com/questions/52840187/how-to-set-environment-variables-using-google-cloud-build-or-other-method-in-goo
[2] 
https://stackoverflow.com/questions/31122337/does-google-app-engine-support-environment-variables
[3] https://cloud.google.com/secret-manager/docs/accessing-the-api

On Friday, January 8, 2021 at 3:14:52 PM UTC+1 Bill wrote:

> The documentation says use options to define a list of global env 
> variables to be used within all build steps.
>
> Are these env variables going to persist when Laravel app (PHP) runs and 
> reads the system environment variables? That's the tricky part.
>
> On Fri, Jan 8, 2021 at 12:26 PM 'jstsetinin' via Google App Engine <
> google-a...@googlegroups.com> wrote:
>
>> Hello,
>> I think it is possible but you have to use "options" field and "env" 
>> subfield. According to documentation [1] in "env" subfield you can specify 
>> "a list of global environment variable definitions that will exist for all 
>> build steps in this build. If a variable is defined in both globally and in 
>> a build step, the variable will use the build step value." 
>>
>> [1] https://cloud.google.com/cloud-build/docs/build-config#options
>>
>>
>> On Friday, 8 January 2021 at 00:43:23 UTC+1 Bill wrote:
>>
>>> Hi,
>>> The environment variables set inside the cloud_build.yaml file, are they 
>>> read by PHP? I mean, are they treated as system environment variables?
>>>
>>> I am considering using environment variables inside the cloud_build.yaml 
>>> file (via substitution) this way I can drop the use of app.yaml and not 
>>> expose my environment variables inside a file on source control.
>>>
>>> Thanks
>>> Bil 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/4643414b-7557-450b-b8ce-c6bd1f09b421n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0447c0e7-31f2-4b36-8285-603f66db74cbn%40googlegroups.com.

[google-appengine] Issues accessing backend API deployed in App Engine

[ks songs]

Hi All,

I have a set of Backend springboot application deployed successfully 
deployed in the APP engine. However, when i tried to access the one of the 
backend API in the postman, the request does not return, the postman 
remains in "sending request" for a long time.

As you can see from the GCP logs below, my backed service is running.

May i know if anyone encounter the same issue before whereby the service is 
deployed successfully in the APP engine, but request cannot be made and no 
timeout is thrown either. 

Thanks in advance!

Regards

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/3973bb56-d0cc-49ad-a520-e3023326748cn%40googlegroups.com.

[google-appengine] Cloud Functions - safe for a key?

[Andrew Davies]

Hi there,

I'm pretty new so please pardon if this is a stupid question. I seem to get 
led astray when looking for security info too. 

Is it safe to store a key as part of a package published to Google Cloud 
Functions? Specifically this would be a key used for a Google service 
account as part of a GCP project. 

If not natively, is there anything I could do to make it 'safe' or is it 
generally a bad idea all around?

Nothing super sensitive to be done by the function but still would prefer 
it to run with minimal risk of interference. 

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e83a0ffc-547f-49fb-8cd0-dc890aaac948n%40googlegroups.com.

Re: [google-appengine] Question about dispatch & backends

[Joshua Smith]

FYI, I've solved this puzzle.

The problem wasn't that the B8 instance was timing out in less than 60 seconds.

The problem was that I had the B8 set to max-instance = 1, so if two concurrent 
API requests came in, the dispatcher had nowhere to send the second request. It 
would hang around awhile, waiting for the B8 to free up, and eventually time 
out.

So the only "bug" here is that the error message from the dispatcher could be 
improved, by mentioning that the *reason* it timed out isn't that the server 
took too long, but rather that max-instances is too low.

-Joshua

> On Dec 14, 2020, at 11:26 AM, Linus Larsen  wrote:
> 
> I think I have seen some documentation way back that it was not possible to 
> serve user facing requests from a B instance, instead you
> were supposed to dispatch the request to a TaskQueue and use some sort of 
> client polling in order to get notified when the work on B
> instance was completed. 
> 
> Cannot find it anymore, the documentation has changed so much you don't 
> really know what's the truth anymore.
> 
> / Linus  
> 
> On Mon, Dec 14, 2020 at 3:13 PM Joshua Smith  > wrote:
> I'm thinking you didn't read my initial message carefully, so let me try 
> again.
> 
> Yes, I'm using the legacy runtime with the 1 minute limit on frontends.
> 
> Yes, I'm using the B8 with basic scaling for my API calls, which should not 
> have that limit.
> 
> Yes, I'm using URL routing via dispatch.yaml, and have confirmed the B8 
> instance is handling the request by checking the logs.
> 
> The request is nonetheless failing because it's taking more than one minute. 
> 
> That should not be happening, according to the documentation. Unless there's 
> another 1 minute limit somewhere else that I'm not aware of.
> 
> -Joshua
> 
>> On Dec 13, 2020, at 8:58 PM, 'Manpreet Sidhu (Google Cloud Support)' via 
>> Google App Engine > > wrote:
>> 
>> Even though your frontend uses automatic scaling, the work is being done on 
>> the B8 instance that is responsible for running your API service.
>> 
>> Based on the screenshot that you attached, the B8 instance has to return in 
>> 24 hours. This is due to the fact that the B8 instance is only capable of 
>> Manual and Basic scaling[0].
>> 
>> Can it be safe to assume that you are using a Legacy Runtime as that is 
>> where the limit of 1 minute is enforced. As an alternative, you can try 
>> routing with URLs[1] or look into implementing the use of Task Queues as 
>> that has a timeout of 10 minutes.
>> 
>> [0]: https://cloud.google.com/appengine/docs/standard#instance_classes 
>> 
>> [1]: 
>> https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls
>>  
>> 
>> On Saturday, December 12, 2020 at 10:11:28 AM UTC-5 Joshua Smith wrote:
>> https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout
>>  
>> 
>> 
>> My frontend service uses automatic scaling, so it has a 1 minute limit to 
>> respond. But my API service is a B8 with this:
>> 
>> 
>> 
>> which means that it should not be subjected to that limit. Yet when I send 
>> requests to it via the dispatch, it seems to be enforcing at 1 minute limit 
>> regardless.
>> 
>> 
>> 
>> 
>>> On Dec 11, 2020, at 4:44 PM, 'Elliott (Cloud Platform Support)' via Google 
>>> App Engine > wrote:
>>> 
>> 
>>> Hello Joshua,
>>> 
>>> I understand that your program uses APIs that need more than 60 seconds to 
>>> complete but even though you are using B8 instances in App Engine, your 
>>> program times out past the 60 seconds. I’m assuming you are using App 
>>> Engine Standard.
>>> 
>>> To find supporting documentation for you, can you tell me more about the 
>>> "gotta get it done in 60 seconds" rule? Can you please give examples of 
>>> this?
>>> 
>>> 
>>> On Friday, December 11, 2020 at 12:01:48 PM UTC-5 Joshua Smith wrote:
>>> I have an app with a frontend UX that queries some backend APIs for 
>>> information. Some of these APIs need more than 60 seconds to complete their 
>>> work.
>>> 
>>> I *thought* I could solve this by doing the following:
>>> 
>>> 1. Run the code on a B8 instance with a different service name
>>> 2. Set up dispatch.yaml to direct the incoming API requests to that service:
>>> 
>>> dispatch:
>>> 
>>> - url: "*/api/*"
>>>   service: reporting
>>> 
>>> - url: "*/*"
>>>   service: default
>>> 
>>> The dispatching is working correctly. I see this in my logs for an /api hit:
>>> 
>>> 
>>> However, it's timing out:
>>> 
>>> 
>>> 
>>> My understanding was that a B8 instance didn't have the "gotta get it done 
>>> in 60 seconds" rule that frontend instance types did.
>>> 
>>> What am I missing?
>>> 
>>> -Joshua
>>> 
>>> 
>> 

[google-appengine] Re: What's the best way to keep Requirements up-to-date?

[Alex]

Very interesting thank you!

On Friday, January 8, 2021 at 5:08:04 PM UTC+1 anco...@google.com wrote:

> Managing dependencies can be tricky! You might want to check Pipenv as 
> suggested above as it acts as a package and virtual environment management 
> tool. It also uses a [Pipfile][1] and a [Pipfile.lock][2] rather than a 
> requirments.txt file. The advantage of a Pipfile is it will be more 
> accommodating semantically regarding how you declare your versions. For 
> example, `pipenv install requests~=1.2` will install version 1.2 and any 
> minor update, but not 2.0. You can also specify a hash to the 
>
> ```
> $ pipenv install "requests>=1.4"   # install a version equal or larger than
> $ pipenv install "requests<=2.13"  # install a version equal or lower
> $ pipenv install "requests>2.19"   # install 2.19.1 but not 2.19.0
> ```
>
> You can specify `pipenv install -r requirements.txt` to import a 
> requirements file and create the Pipfile. If you want to upgrade all 
> packages, you can just run `pipenv update`.
>
> [1]: https://github.com/pypa/pipfile
> [2]: https://pipenv.pypa.io/en/latest/basics/#example-pipfile-pipfile-lock
>
> On Friday, January 8, 2021 at 10:24:00 AM UTC+1 Alex wrote:
>
>> I'm on GAE standard but I don't think that makes a difference. The thread 
>> you share is interesting but goes into the direction of having versioned 
>> packages in the requirements file for even your dev environment. 
>>
>> Trying to be more specific on the workflow I imagine, I would describe 
>> the following: 
>> 1- continue to build and deploy my Dev project without versioned 
>> requirements
>> 2- when I have a stabilized Dev project build, be able to gather the 
>> actual package version it was built with. How could I do that even manually?
>> 3- build staging with a versioned requirements file - versions being 
>> those gathered at step 2
>> 4- perform full QA on staging
>> 5- build prod with a versioned requirements file - versions being those 
>> gathered at step 2 and those staging QA was performed with
>>
>> So a key point seems to be about how can I perform step 2? 
>>
>> On Thursday, January 7, 2021 at 4:04:58 PM UTC+1 Elliott (Cloud Platform 
>> Support) wrote:
>>
>>> Hello Alex,
>>>
>>> From your description, you’ve decided not to version your 
>>> requirements.txt in Python. This is not a crazy approach but there are 
>>> alternatives. I’m assuming you’re using App Engine Flex. I was able to find 
>>> a document here 
>>> 
>>>  
>>> that describes that you do not need to specify a version but indicates why 
>>> it would work in some cases. The following information that I found highly 
>>> suggests using a virtual environment in Python.
>>>
>>> It reads:
>>>
>>> Specifying a version is not a requirement though it does help a lot in 
>>> the future. Some versions of packages will not work well with other 
>>> packages and their respective versions. It is hard to predict how changes 
>>> in the future will affect these interrelationships. This is why it is very 
>>> beneficial to create a snapshot in time (in your requirements.txt) showing 
>>> which version interrelationships do work.
>>>
>>> There is an approach for best practices in this thread regarding your 
>>> concern of breaking change in a package update and explains virtual 
>>> environments:
>>>
>>> it's a good idea to get into the habit of using virtual environments to 
>>> avoid dependency issues, especially when developing your own stuff. 
>>> Anaconda offers a simple solution with the conda create command, and 
>>> virtualenv works great with virtualenvwrapper for a lighter-weight 
>>> solution. Another solution, pipenv, is quite popular.
>>>
>>> There is further explanation:
>>>
>>> A tip - you should aim to be using a virtual environment for each 
>>> individual project that you'll be working on. This creates a 'bubble' for 
>>> you to work within and to install specific package versions in, without it 
>>> affecting your other projects. It will save you a lot of headaches in the 
>>> future as your packages and versions will be kept project specific. 
>>>
>>> There is a way in this thread 
>>> 
>>>  
>>> on how to check if the packages in your requirements.txt are up to date. 
>>> You may choose to act on them as described.
>>>
>>> $ pip list --outdated
>>>
>>> You have described how you are working with three projects to implement 
>>> your non-versioning scenario on Google Cloud Platform but I am unclear 
>>> about your question. Can you tell me more about how you are validating the 
>>> application and building to production so I may research?
>>>
>>>
>>> On Wednesday, January 6, 2021 at 11:32:13 AM UTC-5 Alex wrote:
>>>
 Hi all, 

 Would love to hear your views on this. I run a Python app on GAE. I 
 have quite a few 

Re: [google-appengine] Re: Deploy app into two environments: Staging and Production.

[Bilal Haidar]

Thanks a lot.

I’m a bit confused here. When under App Engine, build takes place on one server 
and then deployment happens on another server? All this under the umbrella of 
App Engine? 

> On 11 Jan 2021, at 3:41 PM, 'Denis T (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
> 
> 
> Hello Bill,
> 
> Specifying the environment variables (env) under options[1] will make them 
> global for your build steps.
> However, if you want to define the environment variables that would be 
> available to your app, you can do that in the "app.yaml" - [2].
> 
> Regards,
> 
> Denis T.
> ___
> [1]: https://cloud.google.com/cloud-build/docs/build-config#options
> [2]: 
> https://cloud.google.com/appengine/docs/standard/python3/config/appref#environment_variables
>> On Friday, January 8, 2021 at 9:35:49 PM UTC+1 Bill wrote:
>> The only difference is the env variables.
>> 
>> I am considering defining this inside build_cloud.yaml file under "options" 
>> to make them global ones. However, I am not sure Laravel/PHP will pick up 
>> those env variables. I still need to try it out.
>> 
>>> On Fri, Jan 8, 2021 at 9:24 PM Kaan  wrote:
>>> My .02 but keep in mind that I don't use cloud build
>>> 
>>> Build your apps to be identity aware, app.yaml doesn't even include the app 
>>> id anymore, the app can know whether it's in staging or production
>>> 
>>> So basically, in theory, previously you only needed a build step that 
>>> renamed app.yaml's project ID, now you don't even need it, just deploy to 
>>> staging or production
>>> 
 On Friday, January 8, 2021 at 1:07:43 PM UTC+3 Bill wrote:
 Thanks a lot Elliot! That is really so much helpful!
 
> On Fri, Jan 8, 2021 at 12:01 AM 'Elliott (Cloud Platform Support)' via 
> Google App Engine  wrote:
> Hello Bill,
> 
> I was able to find the information you need at least to get started. The 
> following document describes what a project is and how you may use one or 
> more than one depending on your use case. Since you want two urls, each 
> environment has its own URL.
> 
> “You control the scope of your projects. A single project might contain 
> multiple separate apps, or conversely a single app might include several 
> projects. Projects can contain resources spread across multiple regions 
> and geographies.”
> 
> The article goes on and provides a recommendation…
> 
> “to have one project per application per environment. For example, if you 
> have two applications, "app1" and "app2", each with a development and 
> production environment, you would have four projects: app1-dev, 
> app1-prod, app2-dev, app2-prod.”
> 
> The reason is that the environments will be isolated from each other to 
> prevent changes from one application affecting the other.
> 
> The article gives an example that you may “grant all developers access to 
> development projects but restrict production access to your CI/CD 
> pipeline.”
> 
> The article goes on that your project structure depends on your business 
> case and requirements.
> 
> There is an example structure here.
> 
> 
> You also asked about having separate app.yaml and cloud_build.yaml files. 
> From my understanding, it is possible to have separate configuration 
> files in each project. 
> 
> I hope I provided the information you need.
> 
> 
> 
> 
>>> On Thursday, January 7, 2021 at 9:52:54 AM UTC-5 Bill wrote:
>>> Hello,
>>> 
>>> I have a Laravel app with app.yaml and cloud_build.yaml files. I need 
>>> to deploy the master branch to production and dev branch to staging. 
>>> Such that each environment has its own URL.
>>> 
>>> Do I need a separate Cloud Project for each environment? Or I can use 
>>> the same App Engine to host both the Production and Staging?
>>> 
>>> If I need two separate projects, how can I have two app.yaml and two 
>>> cloud_build.yaml files?
>>> 
>>> 
>>> Regards
>>> Bilal Haidar
>> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengi...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/c4760782-abf3-4722-8951-afab3c577fa5n%40googlegroups.com.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to google-appengi...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/c7dd2ad4-b4ef-494a-aeff-c382250cfa44n%40googlegroups.com.
> 
> -- 
> You received this message because 

Re: [google-appengine] Re: Deploy app into two environments: Staging and Production.

['Denis T (Google Cloud Platform Support)' via Google App Engine]

Hello Bill,

Specifying the environment variables (env) under options[1] will make them 
global for your build steps.
However, if you want to define the environment variables that would be 
available to your app, you can do that in the "app.yaml" - [2].

Regards,

Denis T.
___
[1]: https://cloud.google.com/cloud-build/docs/build-config#options
[2]: 
https://cloud.google.com/appengine/docs/standard/python3/config/appref#environment_variables
On Friday, January 8, 2021 at 9:35:49 PM UTC+1 Bill wrote:

> The only difference is the env variables.
>
> I am considering defining this inside build_cloud.yaml file under 
> "options" to make them global ones. However, I am not sure Laravel/PHP will 
> pick up those env variables. I still need to try it out.
>
> On Fri, Jan 8, 2021 at 9:24 PM Kaan  wrote:
>
>> My .02 but keep in mind that I don't use cloud build
>>
>> Build your apps to be identity aware, app.yaml doesn't even include the 
>> app id anymore, the app can know whether it's in staging or production
>>
>> So basically, in theory, previously you only needed a build step that 
>> renamed app.yaml's project ID, now you don't even need it, just deploy to 
>> staging or production
>>
>> On Friday, January 8, 2021 at 1:07:43 PM UTC+3 Bill wrote:
>>
>>> Thanks a lot Elliot! That is really so much helpful!
>>>
>>> On Fri, Jan 8, 2021 at 12:01 AM 'Elliott (Cloud Platform Support)' via 
>>> Google App Engine  wrote:
>>>
 Hello Bill,

 I was able to find the information you need at least to get started. 
 The following document 
 
  
 describes what a project is and how you may use one or more than one 
 depending on your use case. Since you want two urls, each environment has 
 its own URL.

 “You control the scope of your projects. A single project might contain 
 multiple separate apps, or conversely a single app might include several 
 projects. Projects can contain resources spread across multiple regions 
 and 
 geographies.”

 The article goes on and provides a recommendation…

 “to have one project per application per environment. For example, if 
 you have two applications, "app1" and "app2", each with a development and 
 production environment, you would have four projects: app1-dev, app1-prod, 
 app2-dev, app2-prod.”

 The reason is that the environments will be isolated from each other to 
 prevent changes from one application affecting the other.

 The article gives an example that you may “grant all developers access 
 to development projects but restrict production access to your CI/CD 
 pipeline.”

 The article goes on that your project structure depends on your 
 business case and requirements.

 There is an example structure here 
 
 .


 You also asked about having separate app.yaml and cloud_build.yaml 
 files. From my understanding, it is possible to have separate 
 configuration 
 files in each project. 

 I hope I provided the information you need.




 On Thursday, January 7, 2021 at 9:52:54 AM UTC-5 Bill wrote:

> Hello,
>
> I have a Laravel app with app.yaml and cloud_build.yaml files. I need 
> to deploy the master branch to production and dev branch to staging. Such 
> that each environment has its own URL.
>
> Do I need a separate Cloud Project for each environment? Or I can use 
> the same App Engine to host both the Production and Staging?
>
> If I need two separate projects, how can I have two app.yaml and two 
> cloud_build.yaml files?
>
>
> Regards
> Bilal Haidar
>
 -- 
 You received this message because you are subscribed to the Google 
 Groups "Google App Engine" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to google-appengi...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-appengine/c4760782-abf3-4722-8951-afab3c577fa5n%40googlegroups.com
  
 
 .

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/c7dd2ad4-b4ef-494a-aeff-c382250cfa44n%40googlegroups.com
>>  
>> 

Re: [google-appengine] Re: Deploy app into two environments: Staging and Production.

['Denis T (Google Cloud Platform Support)' via Google App Engine]

Hello Bill,

Specifying the environment variables (env) under options[1] will make them 
global for your build steps.
However, if you want to define the environment variables that would be 
available to your app, you can do that in the "app.yaml" - [2].

Regards,

Deniss
___
[1]: https://cloud.google.com/cloud-build/docs/build-config#options
[2]: 
https://cloud.google.com/appengine/docs/standard/python3/config/appref#environment_variables

On Friday, January 8, 2021 at 9:35:49 PM UTC+1 Bill wrote:

> The only difference is the env variables.
>
> I am considering defining this inside build_cloud.yaml file under 
> "options" to make them global ones. However, I am not sure Laravel/PHP will 
> pick up those env variables. I still need to try it out.
>
> On Fri, Jan 8, 2021 at 9:24 PM Kaan  wrote:
>
>> My .02 but keep in mind that I don't use cloud build
>>
>> Build your apps to be identity aware, app.yaml doesn't even include the 
>> app id anymore, the app can know whether it's in staging or production
>>
>> So basically, in theory, previously you only needed a build step that 
>> renamed app.yaml's project ID, now you don't even need it, just deploy to 
>> staging or production
>>
>> On Friday, January 8, 2021 at 1:07:43 PM UTC+3 Bill wrote:
>>
>>> Thanks a lot Elliot! That is really so much helpful!
>>>
>>> On Fri, Jan 8, 2021 at 12:01 AM 'Elliott (Cloud Platform Support)' via 
>>> Google App Engine  wrote:
>>>
 Hello Bill,

 I was able to find the information you need at least to get started. 
 The following document 
 
  
 describes what a project is and how you may use one or more than one 
 depending on your use case. Since you want two urls, each environment has 
 its own URL.

 “You control the scope of your projects. A single project might contain 
 multiple separate apps, or conversely a single app might include several 
 projects. Projects can contain resources spread across multiple regions 
 and 
 geographies.”

 The article goes on and provides a recommendation…

 “to have one project per application per environment. For example, if 
 you have two applications, "app1" and "app2", each with a development and 
 production environment, you would have four projects: app1-dev, app1-prod, 
 app2-dev, app2-prod.”

 The reason is that the environments will be isolated from each other to 
 prevent changes from one application affecting the other.

 The article gives an example that you may “grant all developers access 
 to development projects but restrict production access to your CI/CD 
 pipeline.”

 The article goes on that your project structure depends on your 
 business case and requirements.

 There is an example structure here 
 
 .


 You also asked about having separate app.yaml and cloud_build.yaml 
 files. From my understanding, it is possible to have separate 
 configuration 
 files in each project. 

 I hope I provided the information you need.




 On Thursday, January 7, 2021 at 9:52:54 AM UTC-5 Bill wrote:

> Hello,
>
> I have a Laravel app with app.yaml and cloud_build.yaml files. I need 
> to deploy the master branch to production and dev branch to staging. Such 
> that each environment has its own URL.
>
> Do I need a separate Cloud Project for each environment? Or I can use 
> the same App Engine to host both the Production and Staging?
>
> If I need two separate projects, how can I have two app.yaml and two 
> cloud_build.yaml files?
>
>
> Regards
> Bilal Haidar
>
 -- 
 You received this message because you are subscribed to the Google 
 Groups "Google App Engine" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to google-appengi...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-appengine/c4760782-abf3-4722-8951-afab3c577fa5n%40googlegroups.com
  
 
 .

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/c7dd2ad4-b4ef-494a-aeff-c382250cfa44n%40googlegroups.com
>>  
>>