Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Indeed, there is no such account with any permission on that bucket.

So there you have it. That’s probably what happened. (Doesn’t matter anymore 
for me, since I’m now using cloudstorage for uploads, not blobstore.)

Google really botched this transition from appcfg to gcloud.

-Joshua

> On Oct 9, 2020, at 4:42 AM, 'Jose V' via Google App Engine 
>  wrote:
> 
> Hello,
> 
> I have seen previous occurrences in which deploying with "gcloud app deploy" 
> led to 50X errors when uploading to blobstore. It was indeed due to the App 
> Engine Service account lacking permissions on the newly created Cloud Storage 
> bucket.
> Note that by default, the Service account used by App Engine has a name like 
> "[PROJECT-ID]@appspot.gserviceaccount.com" and the Cloud Storage Bucket will 
> have a name like "[PROJECT-ID].appspot.com "
> 
> Could you confirm that the App Engine service account has write permissions 
> for the bucket? See Cloud Storage Roles [1] for reference.
> 
> Regards
> [1]: https://cloud.google.com/storage/docs/access-control/iam-roles 
> 
> 
> On Thursday, October 8, 2020 at 7:23:05 PM UTC+2 Will Reiher wrote:
> I don't remember what the error code was anymore. I just knew that something 
> was f'd. It was really a hunch that that new bucket was involved and that if 
> my app didn't have permission it might be the answer. I was prepared for a 
> very long day but after the permission change it all just started working 
> again - with the blobs showing up in that new bucket.
> 
> Most of my stuff is using Cloud Storage except for user uploads. I'm working 
> to replace that with a UUID based storage name so that I can maintain the 
> ability to allow for same-named files to be uploaded.
> 
> Of course today Cloud Task UI is not available and it wants me to enable the 
> API (I've been using it for months now) Nervous that my queues will be erased 
> if I enable it.
> 
> On Thursday, October 8, 2020 at 10:03:22 AM UTC-7 Joshua Smith wrote:
> Do you recall if you got 500 errors? I’d expect that to generate 403’s.
> 
> Regardless, that does sound like the kind of thing that happened.
> 
> Switching to use cloudstorage instead of blobstore wasn’t too hard, so that’s 
> the workaround I’d recommend for anyone else who hits this issue.
> 
> I’m mostly just annoyed that google has switched to a support model where 
> they won’t provide timely support unless you pay them $100/month. That’s 
> greedy and stupid.
> 
> -Joshua
> 
> 
>> On Oct 8, 2020, at 12:58 PM, Will Reiher > 
>> wrote:
>> 
> 
>> I had an issue a few weeks back. It was due to my transition to gcloud 
>> deploy. It created a new app-id.appspot.com  
>> bucket but my app engine service account did not have permissions. Once I 
>> gave it permission it started working again with the blobstore items showing 
>> up in the new "default" bucket.
>> 
>> On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:
>> 
>> Hi,
>> 
>> I have deployed this Blobstore API sample 
>> 
>>  service and it works well and allows me to successfully upload images. You 
>> could try to add further logging statements in your app to know what is 
>> causing your issue. 
>> 
>> On the other hand, I recommend you to fully migrate you app to Python3 since 
>> Python 2 is no longer supported by the community and use Cloud Storage 
>> instead of Blobstore.
>> 
>> And also note that appcfg tooling was shuted down 
>>  on August 30th, 2020.
>> 
>> 
>> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
>> Unfortunately, I used gcloud app deploy because appcfg.py stopped working a 
>> couple weeks ago.
>> 
>> I did use curl and it revealed nothing. Just a terse 500 with no content 
>> from the server.
>> 
>> 
>>> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus > 
>>> wrote:
>>> 
>> 
>>> Hey Joshua,
>>> 
>>> I don’t think I’m gonna be of much help here, but wanted to mention: if you 
>>> think that running “gcloud app deploy” introduced the issue, can you try 
>>> redeploying the application using appcfg.py? 
>>> 
>>> You could also use curl commands to troubleshoot the upload to blobstore.. 
>>> 
>>> Thanks,
>>> 
>>> Char 
>>> 
>>> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith > wrote:
>>> The problem persists.
>>> 
>>> It occurs to me that last week was the first time I had to make a small 
>>> update to the app in a while, and I did that using “gcloud app deploy”  
>>> instead of appcfg.py for the first time on that project.
>>> 
>>> Is it possible that pushing the app using gcloud introduced a problem 
>>> uploading to blobstore? I wouldn’t think they’d be related, but the timing 
>>> is suspicious.
>>> 
>>> Downloading from blobstore still works. It’s only uploads that are failing.

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

['Jose V' via Google App Engine]

Hello,

I have seen previous occurrences in which deploying with "gcloud app 
deploy" led to 50X errors when uploading to blobstore. It was indeed due to 
the App Engine Service account lacking permissions on the newly created 
Cloud Storage bucket.
Note that by default, the Service account used by App Engine has a name 
like "[PROJECT-ID]@appspot.gserviceaccount.com" and the Cloud Storage 
Bucket will have a name like "[PROJECT-ID].appspot.com"

Could you confirm that the App Engine service account has write permissions 
for the bucket? See Cloud Storage Roles [1] for reference.

Regards
[1]: https://cloud.google.com/storage/docs/access-control/iam-roles

On Thursday, October 8, 2020 at 7:23:05 PM UTC+2 Will Reiher wrote:

> I don't remember what the error code was anymore. I just knew that 
> something was f'd. It was really a hunch that that new bucket was involved 
> and that if my app didn't have permission it might be the answer. I was 
> prepared for a very long day but after the permission change it all just 
> started working again - with the blobs showing up in that new bucket.
>
> Most of my stuff is using Cloud Storage except for user uploads. I'm 
> working to replace that with a UUID based storage name so that I can 
> maintain the ability to allow for same-named files to be uploaded.
>
> Of course today Cloud Task UI is not available and it wants me to enable 
> the API (I've been using it for months now) Nervous that my queues will be 
> erased if I enable it.
>
> On Thursday, October 8, 2020 at 10:03:22 AM UTC-7 Joshua Smith wrote:
>
>> Do you recall if you got 500 errors? I’d expect that to generate 403’s.
>>
>> Regardless, that does sound like the kind of thing that happened.
>>
>> Switching to use cloudstorage instead of blobstore wasn’t too hard, so 
>> that’s the workaround I’d recommend for anyone else who hits this issue.
>>
>> I’m mostly just annoyed that google has switched to a support model where 
>> they won’t provide timely support unless you pay them $100/month. That’s 
>> greedy and stupid.
>>
>> -Joshua
>>
>> On Oct 8, 2020, at 12:58 PM, Will Reiher  
>> wrote:
>>
>> I had an issue a few weeks back. It was due to my transition to gcloud 
>> deploy. It created a new app-id.appspot.com bucket but my app engine 
>> service account did not have permissions. Once I gave it permission it 
>> started working again with the blobstore items showing up in the new 
>> "default" bucket.
>>
>> On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:
>>
>>>
>>> Hi,
>>>
>>> I have deployed this Blobstore API sample 
>>> 
>>>  service 
>>> and it works well and allows me to successfully upload images. You could 
>>> try to add further logging statements in your app to know what is causing 
>>> your issue. 
>>>
>>> On the other hand, I recommend you to fully migrate you app to Python3 
>>> since Python 2 is no longer supported by the community and use Cloud 
>>> Storage instead of Blobstore.
>>>
>>> And also note that appcfg tooling was shuted down 
>>>  on August 30th, 
>>> 2020.
>>>
>>>
>>> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
>>>
 Unfortunately, I used gcloud app deploy because appcfg.py stopped 
 working a couple weeks ago.

 I did use curl and it revealed nothing. Just a terse 500 with no 
 content from the server.

 On Oct 6, 2020, at 4:14 PM, Charounson Saintilus  
 wrote:

 Hey Joshua,

 I don’t think I’m gonna be of much help here, but wanted to mention: if 
 you think that running “gcloud app deploy” introduced the issue, can 
 you try redeploying the application using appcfg.py? 

 You could also use curl commands to troubleshoot the upload to 
 blobstore.. 

 Thanks,

 Char 

 On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith  
 wrote:

> The problem persists.
>
> It occurs to me that last week was the first time I had to make a 
> small update to the app in a while, and I did that using “gcloud app 
> deploy”  instead of appcfg.py for the first time on that project.
>
> Is it possible that pushing the app using gcloud introduced a problem 
> uploading to blobstore? I wouldn’t think they’d be related, but the 
> timing 
> is suspicious.
>
> Downloading from blobstore still works. It’s only uploads that are 
> failing.
>
> My users are getting upset, and your lack of feedback or an ETA is not 
> helping.
>
> -Joshua
>
> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <
> google-a...@googlegroups.com> wrote:
>
> Hi, Joshua
>
> Thank you for reporting this issue. It seems you may be affected by an 
> internal issue reported about App Engine standard applications returning 
> elev

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Will Reiher]

I don't remember what the error code was anymore. I just knew that 
something was f'd. It was really a hunch that that new bucket was involved 
and that if my app didn't have permission it might be the answer. I was 
prepared for a very long day but after the permission change it all just 
started working again - with the blobs showing up in that new bucket.

Most of my stuff is using Cloud Storage except for user uploads. I'm 
working to replace that with a UUID based storage name so that I can 
maintain the ability to allow for same-named files to be uploaded.

Of course today Cloud Task UI is not available and it wants me to enable 
the API (I've been using it for months now) Nervous that my queues will be 
erased if I enable it.

On Thursday, October 8, 2020 at 10:03:22 AM UTC-7 Joshua Smith wrote:

> Do you recall if you got 500 errors? I’d expect that to generate 403’s.
>
> Regardless, that does sound like the kind of thing that happened.
>
> Switching to use cloudstorage instead of blobstore wasn’t too hard, so 
> that’s the workaround I’d recommend for anyone else who hits this issue.
>
> I’m mostly just annoyed that google has switched to a support model where 
> they won’t provide timely support unless you pay them $100/month. That’s 
> greedy and stupid.
>
> -Joshua
>
> On Oct 8, 2020, at 12:58 PM, Will Reiher  
> wrote:
>
> I had an issue a few weeks back. It was due to my transition to gcloud 
> deploy. It created a new app-id.appspot.com bucket but my app engine 
> service account did not have permissions. Once I gave it permission it 
> started working again with the blobstore items showing up in the new 
> "default" bucket.
>
> On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:
>
>>
>> Hi,
>>
>> I have deployed this Blobstore API sample 
>> 
>>  service 
>> and it works well and allows me to successfully upload images. You could 
>> try to add further logging statements in your app to know what is causing 
>> your issue. 
>>
>> On the other hand, I recommend you to fully migrate you app to Python3 
>> since Python 2 is no longer supported by the community and use Cloud 
>> Storage instead of Blobstore.
>>
>> And also note that appcfg tooling was shuted down 
>>  on August 30th, 
>> 2020.
>>
>>
>> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
>>
>>> Unfortunately, I used gcloud app deploy because appcfg.py stopped 
>>> working a couple weeks ago.
>>>
>>> I did use curl and it revealed nothing. Just a terse 500 with no content 
>>> from the server.
>>>
>>> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus  
>>> wrote:
>>>
>>> Hey Joshua,
>>>
>>> I don’t think I’m gonna be of much help here, but wanted to mention: if 
>>> you think that running “gcloud app deploy” introduced the issue, can 
>>> you try redeploying the application using appcfg.py? 
>>>
>>> You could also use curl commands to troubleshoot the upload to 
>>> blobstore.. 
>>>
>>> Thanks,
>>>
>>> Char 
>>>
>>> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith  
>>> wrote:
>>>
 The problem persists.

 It occurs to me that last week was the first time I had to make a small 
 update to the app in a while, and I did that using “gcloud app deploy” 
  instead of appcfg.py for the first time on that project.

 Is it possible that pushing the app using gcloud introduced a problem 
 uploading to blobstore? I wouldn’t think they’d be related, but the timing 
 is suspicious.

 Downloading from blobstore still works. It’s only uploads that are 
 failing.

 My users are getting upset, and your lack of feedback or an ETA is not 
 helping.

 -Joshua

 On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <
 google-a...@googlegroups.com> wrote:

 Hi, Joshua

 Thank you for reporting this issue. It seems you may be affected by an 
 internal issue reported about App Engine standard applications returning 
 elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
 instances in us-central1 region. This issue is externalized on the GCP 
 Status Page as you can find more information on the Status board[1].

 I apologize for the delay or the impact this issue may be causing. For 
 swifter responses or help on issues that are urgent to you, I think it is 
 best to reach out directly to the GCP Support Engineers[2]. 

 Let us know if your application is not running in the us-central1 
 region.  

 Thank you. 

 [1]https://status.cloud.google.com/incident/appengine/20007
 [2]https://cloud.google.com/support-hub

 On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:

> The silence here is deafening, but my users are howling.
>
> *Google: Please address thi

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Do you recall if you got 500 errors? I’d expect that to generate 403’s.

Regardless, that does sound like the kind of thing that happened.

Switching to use cloudstorage instead of blobstore wasn’t too hard, so that’s 
the workaround I’d recommend for anyone else who hits this issue.

I’m mostly just annoyed that google has switched to a support model where they 
won’t provide timely support unless you pay them $100/month. That’s greedy and 
stupid.

-Joshua

> On Oct 8, 2020, at 12:58 PM, Will Reiher  wrote:
> 
> I had an issue a few weeks back. It was due to my transition to gcloud 
> deploy. It created a new app-id.appspot.com bucket but my app engine service 
> account did not have permissions. Once I gave it permission it started 
> working again with the blobstore items showing up in the new "default" bucket.
> 
> On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:
> 
> Hi,
> 
> I have deployed this Blobstore API sample 
> 
>  service and it works well and allows me to successfully upload images. You 
> could try to add further logging statements in your app to know what is 
> causing your issue. 
> 
> On the other hand, I recommend you to fully migrate you app to Python3 since 
> Python 2 is no longer supported by the community and use Cloud Storage 
> instead of Blobstore.
> 
> And also note that appcfg tooling was shuted down 
>  on August 30th, 2020.
> 
> 
> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
> Unfortunately, I used gcloud app deploy because appcfg.py stopped working a 
> couple weeks ago.
> 
> I did use curl and it revealed nothing. Just a terse 500 with no content from 
> the server.
> 
> 
>> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus > 
>> wrote:
>> 
> 
>> Hey Joshua,
>> 
>> I don’t think I’m gonna be of much help here, but wanted to mention: if you 
>> think that running “gcloud app deploy” introduced the issue, can you try 
>> redeploying the application using appcfg.py? 
>> 
>> You could also use curl commands to troubleshoot the upload to blobstore.. 
>> 
>> Thanks,
>> 
>> Char 
>> 
>> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith > wrote:
>> The problem persists.
>> 
>> It occurs to me that last week was the first time I had to make a small 
>> update to the app in a while, and I did that using “gcloud app deploy”  
>> instead of appcfg.py for the first time on that project.
>> 
>> Is it possible that pushing the app using gcloud introduced a problem 
>> uploading to blobstore? I wouldn’t think they’d be related, but the timing 
>> is suspicious.
>> 
>> Downloading from blobstore still works. It’s only uploads that are failing.
>> 
>> My users are getting upset, and your lack of feedback or an ETA is not 
>> helping.
>> 
>> -Joshua
>> 
>>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>>> > wrote:
>>> 
>> 
>>> Hi, Joshua
>>> 
>>> Thank you for reporting this issue. It seems you may be affected by an 
>>> internal issue reported about App Engine standard applications returning 
>>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>>> instances in us-central1 region. This issue is externalized on the GCP 
>>> Status Page as you can find more information on the Status board[1].
>>> 
>>> I apologize for the delay or the impact this issue may be causing. For 
>>> swifter responses or help on issues that are urgent to you, I think it is 
>>> best to reach out directly to the GCP Support Engineers[2]. 
>>> 
>>> Let us know if your application is not running in the us-central1 region.  
>>> 
>>> Thank you. 
>>> 
>>> [1]https://status.cloud.google.com/incident/appengine/20007 
>>> 
>>> [2]https://cloud.google.com/support-hub 
>>> 
>>> 
>>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>>> The silence here is deafening, but my users are howling.
>>> 
>>> Google: Please address this issue immediately! It’s been down at least 24 
>>> hours!
>>> 
>>> I created an issue in public issue tracker with the same info you see here: 
>>> 170195261 
>>> 
>>> I highly doubt I’m the only one experiencing this.
>>> 
>>> -Joshua
>>> 
>>> 
 On Oct 5, 2020, at 11:50 AM, Joshua Smith > wrote:
 
 There is a production issue in my app. (I’m happy to create a private 
 production issue ticket, but I couldn’t figure out where to do that.)
 
 My app is still on Python 2.7 and uses the legacy blobstore.
 
 I use blobstore.create_upload_url('/upload’) to populate a form. This has 
 been working fine for several years.
 
 The last successful one was: 02/Oct/2020:11:37:23 -0400
 
 As of this morning, attempts to upload to the blobstore are resulting in 
 error 5

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Will Reiher]

I had an issue a few weeks back. It was due to my transition to gcloud 
deploy. It created a new app-id.appspot.com bucket but my app engine 
service account did not have permissions. Once I gave it permission it 
started working again with the blobstore items showing up in the new 
"default" bucket.

On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:

>
> Hi,
>
> I have deployed this Blobstore API sample 
> 
>  service 
> and it works well and allows me to successfully upload images. You could 
> try to add further logging statements in your app to know what is causing 
> your issue. 
>
> On the other hand, I recommend you to fully migrate you app to Python3 
> since Python 2 is no longer supported by the community and use Cloud 
> Storage instead of Blobstore.
>
> And also note that appcfg tooling was shuted down 
>  on August 30th, 
> 2020.
>
>
> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
>
>> Unfortunately, I used gcloud app deploy because appcfg.py stopped working 
>> a couple weeks ago.
>>
>> I did use curl and it revealed nothing. Just a terse 500 with no content 
>> from the server.
>>
>> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus  
>> wrote:
>>
>> Hey Joshua,
>>
>> I don’t think I’m gonna be of much help here, but wanted to mention: if 
>> you think that running “gcloud app deploy” introduced the issue, can you 
>> try redeploying the application using appcfg.py? 
>>
>> You could also use curl commands to troubleshoot the upload to 
>> blobstore.. 
>>
>> Thanks,
>>
>> Char 
>>
>> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith  wrote:
>>
>>> The problem persists.
>>>
>>> It occurs to me that last week was the first time I had to make a small 
>>> update to the app in a while, and I did that using “gcloud app deploy” 
>>>  instead of appcfg.py for the first time on that project.
>>>
>>> Is it possible that pushing the app using gcloud introduced a problem 
>>> uploading to blobstore? I wouldn’t think they’d be related, but the timing 
>>> is suspicious.
>>>
>>> Downloading from blobstore still works. It’s only uploads that are 
>>> failing.
>>>
>>> My users are getting upset, and your lack of feedback or an ETA is not 
>>> helping.
>>>
>>> -Joshua
>>>
>>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <
>>> google-a...@googlegroups.com> wrote:
>>>
>>> Hi, Joshua
>>>
>>> Thank you for reporting this issue. It seems you may be affected by an 
>>> internal issue reported about App Engine standard applications returning 
>>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>>> instances in us-central1 region. This issue is externalized on the GCP 
>>> Status Page as you can find more information on the Status board[1].
>>>
>>> I apologize for the delay or the impact this issue may be causing. For 
>>> swifter responses or help on issues that are urgent to you, I think it is 
>>> best to reach out directly to the GCP Support Engineers[2]. 
>>>
>>> Let us know if your application is not running in the us-central1 
>>> region.  
>>>
>>> Thank you. 
>>>
>>> [1]https://status.cloud.google.com/incident/appengine/20007
>>> [2]https://cloud.google.com/support-hub
>>>
>>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>>>
 The silence here is deafening, but my users are howling.

 *Google: Please address this issue immediately! It’s been down at least 
 24 hours!*

 I created an issue in public issue tracker with the same info you see 
 here: 170195261 

 I highly doubt I’m the only one experiencing this.

 -Joshua


 On Oct 5, 2020, at 11:50 AM, Joshua Smith  wrote:

 There is a production issue in my app. (I’m happy to create a private 
 production issue ticket, but I couldn’t figure out where to do that.)

 My app is still on Python 2.7 and uses the legacy blobstore.

 I use blobstore.create_upload_url('/upload’) to populate a form. This 
 has been working fine for several years.

 The last successful one was: 02/Oct/2020:11:37:23 -0400

 As of this morning, attempts to upload to the blobstore are resulting 
 in error 500 and I don’t see the request in my logs.

 I simulated one of these uploads using curl, but that produced no 
 insight:

 * Using HTTP2, server supports multi-use
 * Connection state changed (HTTP/2 confirmed)
 * Copying HTTP/2 data in stream buffer to connection buffer after 
 upgrade: len=0
 * Using Stream ID: 1 (easy handle 0x7fa064808200)
 > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
 > Host: —redacted— 
 > User-Agent: curl/7.64.1
 > Accept: */*
 > Content-Length: 12881
 > Content-Type: multipa

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

['barrado' via Google App Engine]

Hi,

I have deployed this Blobstore API sample 

 service 
and it works well and allows me to successfully upload images. You could 
try to add further logging statements in your app to know what is causing 
your issue. 

On the other hand, I recommend you to fully migrate you app to Python3 
since Python 2 is no longer supported by the community and use Cloud 
Storage instead of Blobstore.

And also note that appcfg tooling was shuted down 
 on August 30th, 2020.


On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:

> Unfortunately, I used gcloud app deploy because appcfg.py stopped working 
> a couple weeks ago.
>
> I did use curl and it revealed nothing. Just a terse 500 with no content 
> from the server.
>
> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus  
> wrote:
>
> Hey Joshua,
>
> I don’t think I’m gonna be of much help here, but wanted to mention: if 
> you think that running “gcloud app deploy” introduced the issue, can you 
> try redeploying the application using appcfg.py? 
>
> You could also use curl commands to troubleshoot the upload to blobstore.. 
>
> Thanks,
>
> Char 
>
> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith  wrote:
>
>> The problem persists.
>>
>> It occurs to me that last week was the first time I had to make a small 
>> update to the app in a while, and I did that using “gcloud app deploy” 
>>  instead of appcfg.py for the first time on that project.
>>
>> Is it possible that pushing the app using gcloud introduced a problem 
>> uploading to blobstore? I wouldn’t think they’d be related, but the timing 
>> is suspicious.
>>
>> Downloading from blobstore still works. It’s only uploads that are 
>> failing.
>>
>> My users are getting upset, and your lack of feedback or an ETA is not 
>> helping.
>>
>> -Joshua
>>
>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <
>> google-a...@googlegroups.com> wrote:
>>
>> Hi, Joshua
>>
>> Thank you for reporting this issue. It seems you may be affected by an 
>> internal issue reported about App Engine standard applications returning 
>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>> instances in us-central1 region. This issue is externalized on the GCP 
>> Status Page as you can find more information on the Status board[1].
>>
>> I apologize for the delay or the impact this issue may be causing. For 
>> swifter responses or help on issues that are urgent to you, I think it is 
>> best to reach out directly to the GCP Support Engineers[2]. 
>>
>> Let us know if your application is not running in the us-central1 
>> region.  
>>
>> Thank you. 
>>
>> [1]https://status.cloud.google.com/incident/appengine/20007
>> [2]https://cloud.google.com/support-hub
>>
>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>>
>>> The silence here is deafening, but my users are howling.
>>>
>>> *Google: Please address this issue immediately! It’s been down at least 
>>> 24 hours!*
>>>
>>> I created an issue in public issue tracker with the same info you see 
>>> here: 170195261 
>>>
>>> I highly doubt I’m the only one experiencing this.
>>>
>>> -Joshua
>>>
>>>
>>> On Oct 5, 2020, at 11:50 AM, Joshua Smith  wrote:
>>>
>>> There is a production issue in my app. (I’m happy to create a private 
>>> production issue ticket, but I couldn’t figure out where to do that.)
>>>
>>> My app is still on Python 2.7 and uses the legacy blobstore.
>>>
>>> I use blobstore.create_upload_url('/upload’) to populate a form. This 
>>> has been working fine for several years.
>>>
>>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>>>
>>> As of this morning, attempts to upload to the blobstore are resulting in 
>>> error 500 and I don’t see the request in my logs.
>>>
>>> I simulated one of these uploads using curl, but that produced no 
>>> insight:
>>>
>>> * Using HTTP2, server supports multi-use
>>> * Connection state changed (HTTP/2 confirmed)
>>> * Copying HTTP/2 data in stream buffer to connection buffer after 
>>> upgrade: len=0
>>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>>> > Host: —redacted— 
>>> > User-Agent: curl/7.64.1
>>> > Accept: */*
>>> > Content-Length: 12881
>>> > Content-Type: multipart/form-data; 
>>> boundary=c1bce33de7c355c9
>>> > 
>>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>>> * We are completely uploaded and fine
>>> < HTTP/2 500 
>>> < x-guploader-uploadid: 
>>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>>> < content-length: 0
>>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>>> < server: UploadServer
>>> < content-type: text/html; charset=UTF-8
>>> < 
>>> * Connection #0 to host mytowngovernment.o

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Unfortunately, I used gcloud app deploy because appcfg.py stopped working a 
couple weeks ago.

I did use curl and it revealed nothing. Just a terse 500 with no content from 
the server.

> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus  
> wrote:
> 
> Hey Joshua,
> 
> I don’t think I’m gonna be of much help here, but wanted to mention: if you 
> think that running “gcloud app deploy” introduced the issue, can you try 
> redeploying the application using appcfg.py? 
> 
> You could also use curl commands to troubleshoot the upload to blobstore.. 
> 
> Thanks,
> 
> Char 
> 
> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith  > wrote:
> The problem persists.
> 
> It occurs to me that last week was the first time I had to make a small 
> update to the app in a while, and I did that using “gcloud app deploy”  
> instead of appcfg.py for the first time on that project.
> 
> Is it possible that pushing the app using gcloud introduced a problem 
> uploading to blobstore? I wouldn’t think they’d be related, but the timing is 
> suspicious.
> 
> Downloading from blobstore still works. It’s only uploads that are failing.
> 
> My users are getting upset, and your lack of feedback or an ETA is not 
> helping.
> 
> -Joshua
> 
>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>> > > wrote:
>> 
> 
>> Hi, Joshua
>> 
>> Thank you for reporting this issue. It seems you may be affected by an 
>> internal issue reported about App Engine standard applications returning 
>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>> instances in us-central1 region. This issue is externalized on the GCP 
>> Status Page as you can find more information on the Status board[1].
>> 
>> I apologize for the delay or the impact this issue may be causing. For 
>> swifter responses or help on issues that are urgent to you, I think it is 
>> best to reach out directly to the GCP Support Engineers[2]. 
>> 
>> Let us know if your application is not running in the us-central1 region.  
>> 
>> Thank you. 
>> 
>> [1]https://status.cloud.google.com/incident/appengine/20007 
>> 
>> [2]https://cloud.google.com/support-hub 
>> 
>> 
>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>> The silence here is deafening, but my users are howling.
>> 
>> Google: Please address this issue immediately! It’s been down at least 24 
>> hours!
>> 
>> I created an issue in public issue tracker with the same info you see here: 
>> 170195261 
>> 
>> I highly doubt I’m the only one experiencing this.
>> 
>> -Joshua
>> 
>> 
>>> On Oct 5, 2020, at 11:50 AM, Joshua Smith > wrote:
>>> 
>>> There is a production issue in my app. (I’m happy to create a private 
>>> production issue ticket, but I couldn’t figure out where to do that.)
>>> 
>>> My app is still on Python 2.7 and uses the legacy blobstore.
>>> 
>>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>>> been working fine for several years.
>>> 
>>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>>> 
>>> As of this morning, attempts to upload to the blobstore are resulting in 
>>> error 500 and I don’t see the request in my logs.
>>> 
>>> I simulated one of these uploads using curl, but that produced no insight:
>>> 
>>> * Using HTTP2, server supports multi-use
>>> * Connection state changed (HTTP/2 confirmed)
>>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>>> len=0
>>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>>> > Host: —redacted— 
>>> > User-Agent: curl/7.64.1
>>> > Accept: */*
>>> > Content-Length: 12881
>>> > Content-Type: multipart/form-data; 
>>> > boundary=c1bce33de7c355c9
>>> > 
>>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>>> * We are completely uploaded and fine
>>> < HTTP/2 500 
>>> < x-guploader-uploadid: 
>>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>>> < content-length: 0
>>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>>> < server: UploadServer
>>> < content-type: text/html; charset=UTF-8
>>> < 
>>> * Connection #0 to host mytowngovernment.org  
>>> left intact
>>> * Closing connection 0
>>> 
>>> Although perhaps that uploadid might help you diagnose?
>>> 
>>> Please advise.
>>> 
>>> -Joshua
>>> 
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengine+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://grou

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Charounson Saintilus]

Hey Joshua,

I don’t think I’m gonna be of much help here, but wanted to mention: if you
think that running “gcloud app deploy” introduced the issue, can you try
redeploying the application using appcfg.py?

You could also use curl commands to troubleshoot the upload to blobstore..

Thanks,

Char

On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith 
wrote:

> The problem persists.
>
> It occurs to me that last week was the first time I had to make a small
> update to the app in a while, and I did that using “gcloud app deploy”
>  instead of appcfg.py for the first time on that project.
>
> Is it possible that pushing the app using gcloud introduced a problem
> uploading to blobstore? I wouldn’t think they’d be related, but the timing
> is suspicious.
>
> Downloading from blobstore still works. It’s only uploads that are failing.
>
> My users are getting upset, and your lack of feedback or an ETA is not
> helping.
>
> -Joshua
>
> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <
> google-appengine@googlegroups.com> wrote:
>
> Hi, Joshua
>
> Thank you for reporting this issue. It seems you may be affected by an
> internal issue reported about App Engine standard applications returning
> elevated HTTP 500 errors. This issue is observed for App Engine  Standard
> instances in us-central1 region. This issue is externalized on the GCP
> Status Page as you can find more information on the Status board[1].
>
> I apologize for the delay or the impact this issue may be causing. For
> swifter responses or help on issues that are urgent to you, I think it is
> best to reach out directly to the GCP Support Engineers[2].
>
> Let us know if your application is not running in the us-central1 region.
>
> Thank you.
>
> [1]https://status.cloud.google.com/incident/appengine/20007
> [2]https://cloud.google.com/support-hub
>
> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>
>> The silence here is deafening, but my users are howling.
>>
>> *Google: Please address this issue immediately! It’s been down at least
>> 24 hours!*
>>
>> I created an issue in public issue tracker with the same info you see
>> here: 170195261 
>>
>> I highly doubt I’m the only one experiencing this.
>>
>> -Joshua
>>
>>
>> On Oct 5, 2020, at 11:50 AM, Joshua Smith  wrote:
>>
>> There is a production issue in my app. (I’m happy to create a private
>> production issue ticket, but I couldn’t figure out where to do that.)
>>
>> My app is still on Python 2.7 and uses the legacy blobstore.
>>
>> I use blobstore.create_upload_url('/upload’) to populate a form. This has
>> been working fine for several years.
>>
>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>>
>> As of this morning, attempts to upload to the blobstore are resulting in
>> error 500 and I don’t see the request in my logs.
>>
>> I simulated one of these uploads using curl, but that produced no insight:
>>
>> * Using HTTP2, server supports multi-use
>> * Connection state changed (HTTP/2 confirmed)
>> * Copying HTTP/2 data in stream buffer to connection buffer after
>> upgrade: len=0
>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>> > Host: —redacted—
>> > User-Agent: curl/7.64.1
>> > Accept: */*
>> > Content-Length: 12881
>> > Content-Type: multipart/form-data;
>> boundary=c1bce33de7c355c9
>> >
>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>> * We are completely uploaded and fine
>> < HTTP/2 500
>> < x-guploader-uploadid:
>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>> < content-length: 0
>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>> < server: UploadServer
>> < content-type: text/html; charset=UTF-8
>> <
>> * Connection #0 to host mytowngovernment.org left intact
>> * Closing connection 0
>>
>> Although perhaps that uploadid might help you diagnose?
>>
>> Please advise.
>>
>> -Joshua
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com
> 
> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/9EC69CDD-FDB4-4CEA-AFA2-BC03919C300A%40gmail.com
> 

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Sigh.

I’ve gone ahead and re-coded my app to use cloudstorage instead of blobstore 
for new uploads. So this is no longer critical.

(Thankfully, downloads have continued to work. If you break blobstore 
downloads, that’ll be a real problem for me.)

The idea of using a cloud provider is they:

1. Keep the infrastructure working, and
2. Respond quickly when there are outages.

If I have to spend $100/month to get that level of service, it changes the 
economics completely.

I’m very disappointed in GCS right now.

-Joshua

> On Oct 6, 2020, at 3:39 PM, Joshua Smith  wrote:
> 
> The problem persists.
> 
> It occurs to me that last week was the first time I had to make a small 
> update to the app in a while, and I did that using “gcloud app deploy”  
> instead of appcfg.py for the first time on that project.
> 
> Is it possible that pushing the app using gcloud introduced a problem 
> uploading to blobstore? I wouldn’t think they’d be related, but the timing is 
> suspicious.
> 
> Downloading from blobstore still works. It’s only uploads that are failing.
> 
> My users are getting upset, and your lack of feedback or an ETA is not 
> helping.
> 
> -Joshua
> 
>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>> > > wrote:
>> 
>> Hi, Joshua
>> 
>> Thank you for reporting this issue. It seems you may be affected by an 
>> internal issue reported about App Engine standard applications returning 
>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>> instances in us-central1 region. This issue is externalized on the GCP 
>> Status Page as you can find more information on the Status board[1].
>> 
>> I apologize for the delay or the impact this issue may be causing. For 
>> swifter responses or help on issues that are urgent to you, I think it is 
>> best to reach out directly to the GCP Support Engineers[2]. 
>> 
>> Let us know if your application is not running in the us-central1 region.  
>> 
>> Thank you. 
>> 
>> [1]https://status.cloud.google.com/incident/appengine/20007 
>> 
>> [2]https://cloud.google.com/support-hub 
>> 
>> 
>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>> The silence here is deafening, but my users are howling.
>> 
>> Google: Please address this issue immediately! It’s been down at least 24 
>> hours!
>> 
>> I created an issue in public issue tracker with the same info you see here: 
>> 170195261 
>> 
>> I highly doubt I’m the only one experiencing this.
>> 
>> -Joshua
>> 
>> 
>>> On Oct 5, 2020, at 11:50 AM, Joshua Smith >> > wrote:
>>> 
>>> There is a production issue in my app. (I’m happy to create a private 
>>> production issue ticket, but I couldn’t figure out where to do that.)
>>> 
>>> My app is still on Python 2.7 and uses the legacy blobstore.
>>> 
>>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>>> been working fine for several years.
>>> 
>>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>>> 
>>> As of this morning, attempts to upload to the blobstore are resulting in 
>>> error 500 and I don’t see the request in my logs.
>>> 
>>> I simulated one of these uploads using curl, but that produced no insight:
>>> 
>>> * Using HTTP2, server supports multi-use
>>> * Connection state changed (HTTP/2 confirmed)
>>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>>> len=0
>>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>>> > Host: —redacted— 
>>> > User-Agent: curl/7.64.1
>>> > Accept: */*
>>> > Content-Length: 12881
>>> > Content-Type: multipart/form-data; 
>>> > boundary=c1bce33de7c355c9
>>> > 
>>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>>> * We are completely uploaded and fine
>>> < HTTP/2 500 
>>> < x-guploader-uploadid: 
>>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>>> < content-length: 0
>>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>>> < server: UploadServer
>>> < content-type: text/html; charset=UTF-8
>>> < 
>>> * Connection #0 to host mytowngovernment.org  
>>> left intact
>>> * Closing connection 0
>>> 
>>> Although perhaps that uploadid might help you diagnose?
>>> 
>>> Please advise.
>>> 
>>> -Joshua
>>> 
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengine+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googleg

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

The problem persists.

It occurs to me that last week was the first time I had to make a small update 
to the app in a while, and I did that using “gcloud app deploy”  instead of 
appcfg.py for the first time on that project.

Is it possible that pushing the app using gcloud introduced a problem uploading 
to blobstore? I wouldn’t think they’d be related, but the timing is suspicious.

Downloading from blobstore still works. It’s only uploads that are failing.

My users are getting upset, and your lack of feedback or an ETA is not helping.

-Joshua

> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> Hi, Joshua
> 
> Thank you for reporting this issue. It seems you may be affected by an 
> internal issue reported about App Engine standard applications returning 
> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
> instances in us-central1 region. This issue is externalized on the GCP Status 
> Page as you can find more information on the Status board[1].
> 
> I apologize for the delay or the impact this issue may be causing. For 
> swifter responses or help on issues that are urgent to you, I think it is 
> best to reach out directly to the GCP Support Engineers[2]. 
> 
> Let us know if your application is not running in the us-central1 region.  
> 
> Thank you. 
> 
> [1]https://status.cloud.google.com/incident/appengine/20007 
> 
> [2]https://cloud.google.com/support-hub 
> 
> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
> The silence here is deafening, but my users are howling.
> 
> Google: Please address this issue immediately! It’s been down at least 24 
> hours!
> 
> I created an issue in public issue tracker with the same info you see here: 
> 170195261 
> 
> I highly doubt I’m the only one experiencing this.
> 
> -Joshua
> 
> 
>> On Oct 5, 2020, at 11:50 AM, Joshua Smith > > wrote:
>> 
>> There is a production issue in my app. (I’m happy to create a private 
>> production issue ticket, but I couldn’t figure out where to do that.)
>> 
>> My app is still on Python 2.7 and uses the legacy blobstore.
>> 
>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>> been working fine for several years.
>> 
>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>> 
>> As of this morning, attempts to upload to the blobstore are resulting in 
>> error 500 and I don’t see the request in my logs.
>> 
>> I simulated one of these uploads using curl, but that produced no insight:
>> 
>> * Using HTTP2, server supports multi-use
>> * Connection state changed (HTTP/2 confirmed)
>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>> len=0
>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>> > Host: —redacted— 
>> > User-Agent: curl/7.64.1
>> > Accept: */*
>> > Content-Length: 12881
>> > Content-Type: multipart/form-data; 
>> > boundary=c1bce33de7c355c9
>> > 
>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>> * We are completely uploaded and fine
>> < HTTP/2 500 
>> < x-guploader-uploadid: 
>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>> < content-length: 0
>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>> < server: UploadServer
>> < content-type: text/html; charset=UTF-8
>> < 
>> * Connection #0 to host mytowngovernment.org  
>> left intact
>> * Closing connection 0
>> 
>> Although perhaps that uploadid might help you diagnose?
>> 
>> Please advise.
>> 
>> -Joshua
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9EC69CDD-FDB4-4CEA-AFA2-BC03919C300A%40gmail.com.

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> I apologize for the delay or the impact this issue may be causing. For 
> swifter responses or help on issues that are urgent to you, I think it is 
> best to reach out directly to the GCP Support Engineers[2]. 

My reading of that the cases page is that I’d have to pay you $100/month for 
the privilege of reporting outages that way. (The free tier says it’s 
read-only.)

Am I missing something?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/87208FE3-1BB7-400A-8F76-BDB9946B9EBE%40gmail.com.

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

It pre-dated that issue, and it is continuing despite the larger outage being 
resolved.

Please address this ASAP.

Thanks.

-Joshua

> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> Hi, Joshua
> 
> Thank you for reporting this issue. It seems you may be affected by an 
> internal issue reported about App Engine standard applications returning 
> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
> instances in us-central1 region. This issue is externalized on the GCP Status 
> Page as you can find more information on the Status board[1].
> 
> I apologize for the delay or the impact this issue may be causing. For 
> swifter responses or help on issues that are urgent to you, I think it is 
> best to reach out directly to the GCP Support Engineers[2]. 
> 
> Let us know if your application is not running in the us-central1 region.  
> 
> Thank you. 
> 
> [1]https://status.cloud.google.com/incident/appengine/20007 
> 
> [2]https://cloud.google.com/support-hub 
> 
> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
> The silence here is deafening, but my users are howling.
> 
> Google: Please address this issue immediately! It’s been down at least 24 
> hours!
> 
> I created an issue in public issue tracker with the same info you see here: 
> 170195261 
> 
> I highly doubt I’m the only one experiencing this.
> 
> -Joshua
> 
> 
>> On Oct 5, 2020, at 11:50 AM, Joshua Smith > > wrote:
>> 
>> There is a production issue in my app. (I’m happy to create a private 
>> production issue ticket, but I couldn’t figure out where to do that.)
>> 
>> My app is still on Python 2.7 and uses the legacy blobstore.
>> 
>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>> been working fine for several years.
>> 
>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>> 
>> As of this morning, attempts to upload to the blobstore are resulting in 
>> error 500 and I don’t see the request in my logs.
>> 
>> I simulated one of these uploads using curl, but that produced no insight:
>> 
>> * Using HTTP2, server supports multi-use
>> * Connection state changed (HTTP/2 confirmed)
>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>> len=0
>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>> > Host: —redacted— 
>> > User-Agent: curl/7.64.1
>> > Accept: */*
>> > Content-Length: 12881
>> > Content-Type: multipart/form-data; 
>> > boundary=c1bce33de7c355c9
>> > 
>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>> * We are completely uploaded and fine
>> < HTTP/2 500 
>> < x-guploader-uploadid: 
>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>> < content-length: 0
>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>> < server: UploadServer
>> < content-type: text/html; charset=UTF-8
>> < 
>> * Connection #0 to host mytowngovernment.org  
>> left intact
>> * Closing connection 0
>> 
>> Although perhaps that uploadid might help you diagnose?
>> 
>> Please advise.
>> 
>> -Joshua
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/17DE773C-4DAF-491C-890B-58DA53512FB3%40gmail.com.

[google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

There is a production issue in my app. (I’m happy to create a private 
production issue ticket, but I couldn’t figure out where to do that.)

My app is still on Python 2.7 and uses the legacy blobstore.

I use blobstore.create_upload_url('/upload’) to populate a form. This has been 
working fine for several years.

The last successful one was: 02/Oct/2020:11:37:23 -0400

As of this morning, attempts to upload to the blobstore are resulting in error 
500 and I don’t see the request in my logs.

I simulated one of these uploads using curl, but that produced no insight:

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa064808200)
> POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
> Host: —redacted— 
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 12881
> Content-Type: multipart/form-data; 
> boundary=c1bce33de7c355c9
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
< HTTP/2 500 
< x-guploader-uploadid: 
ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
< content-length: 0
< date: Mon, 05 Oct 2020 15:38:41 GMT
< server: UploadServer
< content-type: text/html; charset=UTF-8
< 
* Connection #0 to host mytowngovernment.org left intact
* Closing connection 0

Although perhaps that uploadid might help you diagnose?

Please advise.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9D0D7E6F-5B1E-4A36-8BE8-60BFA030BF61%40gmail.com.