[google-appengine] Re: Error 401 when attempting to bulk download with appcfg.py

[Bemmu]

I had the same problem and just got it to work.

As the OP mentioned, I could not manage to get it to work following the 
steps suggested by the appcfg.py error message 
(*oauth2client.client.ApplicationDefaultCredentialsError: 
The Application Default Credentials are not available. They are available 
if running in Google Compute Engine. Otherwise, the environment variable 
GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining 
the credentials. 
See https://developers.google.com/accounts/docs/application-default-credentials 
 
for 
more information.*)

Getting the key by those instructions did not work (I tried the app engine 
key when prompted to select whether I wanted compute engine or app engine 
key, maybe I chose wrong).

I wrote the solution up as a blogspot with the relevant screenshot here: 
Backing 
up data from App Engine with appcfg.py 


Anyway, steps I followed to get it to work:

   1. Go to IAM 
   2. Click the three balls on the right side of "Compute Engine default 
   service account"
   3. "Create key"
   4. JSON
   5. Save the key compute-key.json

Then create a script called "backup":
GOOGLE_APPLICATION_CREDENTIALS=compute-key.json appcfg.py download_data . 
--filename=data_store_backup

Instead of . you can also put the full path to the directory containing 
app.yaml if running from somewhere else.

If you run into "HTTPError: HTTP Error 404: Not Found" when doing 
download_data, your appcfg.py is so old that is hitting a deprecated 
endpoint. Update your appcfg.py and try again.

On Monday, April 18, 2016 at 2:34:43 AM UTC+9, solsTiCe d'Hiver wrote:
>
> hi.
> Like I used to do it several years ago, I am attempting to *bulk download* 
> the *entities* of the datastore of my app with *appcfg.py* (from latest 
> python SDK)
>
> I am using this little script:
> #!/bin/bash
> date=`date +%Y%m%d`
> appid=myappid
> url="https://$appid.appspot.com/_ah/remote_api;
> GOOGLE_APPLICATION_CREDENTIALS=myapp.json appcfg.py download_data -v -e 
> myem...@gmail.com --application=s~$appid --url=$url --filename=allentities
> -$date.sql3
> rm -f bulkloader-*
>
>
> because this command does not ask for an oauth token, I have previously 
> run the command
>
> appcfg.py list_versions
>
> just to get the *oauth2* token
> And as said in 
> https://developers.google.com/accounts/docs/application-default-credentials 
> I have generated a *json* file with the google credentials and use it 
> with GOOGLE_APPLICATION_CREDENTIALS variable as shown above.
>
> However when I run the script I get a long list of
> 2016-04-17 19:21:29,316 INFO client.py:546 Attempting refresh to obtain 
> initial access_token 
> 2016-04-17 19:21:29,378 INFO client.py:804 Refreshing access_token 
> 2016-04-17 19:21:30,901 INFO client.py:578 Refreshing due to a 401 
> (attempt 1/2) 
> 2016-04-17 19:21:30,988 INFO client.py:804 Refreshing access_token
> 2016-04-17 19:21:31,570 INFO client.py:578 Refreshing due to a 401 
> (attempt 2/2) 
> 2016-04-17 19:21:31,659 INFO client.py:804 Refreshing access_token 
> 2016-04-17 19:21:32,436 INFO client.py:578 Refreshing due to a 401 
> (attempt 1/2) 
> 2016-04-17 19:21:32,526 INFO client.py:804 Refreshing access_token 
> 2016-04-17 19:21:33,162 INFO client.py:578 Refreshing due to a 401 
> (attempt 2/2) 
> [...]
>
>
> and this ends with 
> Error 401: --- begin server output ---
> You must be logged in as an administrator to access this.
> --- end server output ---
>
> I have the same 401 error in the log of my app.
>
> What could be wrong ?
> I am the owner of the app and my email is listed as such in the logins
>
> I don't want to enable billing and use the Google Cloud Platform to export 
> entities from the datastore with "Admin Datastore" (by the way clicking the 
> button "Open Admin Datastore" does nothing, one need to use a right click 
> and open in new tab to get something)
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/781611b0-7085-42d6-9146-180bae943a27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Error 401 when attempting to bulk download with appcfg.py

['Nick (Cloud Platform Support)' via Google App Engine]

Hey solsTiCe,

Could you perhaps try adding the relevant service account as an App Engine 
Administrator and Project Owner in your Developers' Console "IAM" section?

On Friday, July 1, 2016 at 12:25:51 PM UTC-4, solsTiCe d'Hiver wrote:
>
> hi. sorry
>
> So to make thing a little clearer I hope:
>
> I follow the standard procedure using appcfg.py using an oauth 2 token. 
> that's when I got that error message
>
> 2016-07-01 17:55:25,147 INFO client.py:981 Timeout attempting to reach GCE 
>> metadata service. 
>> Traceback (most recent call last):
>>   File "/home/solstice/Applications/google_appengine/appcfg.py", line 
>> 133, in 
>> run_file(__file__, globals())
>>   File "/home/solstice/Applications/google_appengine/appcfg.py", line 
>> 129, in run_file
>> execfile(_PATHS.script_file(script_name), globals_)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
>>  
>> line 5495, in 
>> main(sys.argv)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
>>  
>> line 5486, in main
>> result = AppCfgApp(argv).Run()
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
>>  
>> line 2964, in Run
>> self.action(self)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
>>  
>> line 5142, in __call__
>> return method()
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
>>  
>> line 4924, in PerformDownload
>> run_fn(args)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
>>  
>> line 4829, in RunBulkloader
>> sys.exit(bulkloader.Run(arg_dict, self._GetOAuth2Parameters()))
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
>>  
>> line 4349, in Run
>> return _PerformBulkload(arg_dict, oauth2_parameters)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
>>  
>> line 4242, in _PerformBulkload
>> return_code = app.Run()
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
>>  
>> line 3408, in Run
>> oauth2_parameters=self.oauth2_parameters)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
>>  
>> line 1280, in __init__
>> rpc_server_factory=throttled_rpc_server_factory)
>>   File 
>> "/home/solstice/Applications/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
>>  
>> line 747, in ConfigureRemoteApiForOAuth
>> credentials = client.GoogleCredentials.get_application_default()
>>   File 
>> "/home/solstice/Applications/google_appengine/lib/oauth2client/oauth2client/client.py",
>>  
>> line 1211, in get_application_default
>> return GoogleCredentials._get_implicit_credentials()
>>   File 
>> "/home/solstice/Applications/google_appengine/lib/oauth2client/oauth2client/client.py",
>>  
>> line 1201, in _get_implicit_credentials
>> raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
>> oauth2client.client.ApplicationDefaultCredentialsError: The Application 
>> Default Credentials are not available. They are available if running in 
>> Google Compute Engine. Otherwise, the environment variable 
>> GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining 
>> the credentials. See 
>> https://developers.google.com/accounts/docs/application-default-credentials 
>> for more information.
>
>
> So I don't understand why the oauth token does not work and why I am asked 
> to use another method aka the GOOGLE_APPLICATION_CREDENTIALS file. Anyway...
>
> As prompted, I go to 
> https://developers.google.com/accounts/docs/application-default-credentials 
> and follow the procedure from the "How the Application Default Credentials 
> work" paragraph.
>
> Then, I use the .json as second auth method as asked as the first oauth 
> does not work. Therefore, I don't get the exception but get a 401 auth 
> error from the site.
>
> and finally, I have tried without the s~ and it's still the same 401 error.
>
> 2016-04-19 16:40 GMT+02:00 'Nick (Cloud Platform Support)' via Google App 
> Engine :
>
>> Hey solsTiCe,
>>
>> Unfortunately it's not quite clear to me which instructions you're 
>> following. Also, it seems strange that you'd need both credentials and a 
>> token. Could you be a little more specific about exactly what steps you're 
>> following to build "myapp.json"? 
>>
>> Additionally, I can see in your bash script that you use "application=s~
>> $appid", could you attempt a test without the '~s'?
>>
>> Thanks for clarifying,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Tuesday, April 19, 2016 at 6:31:54 AM UTC-4, solsTiCe d'Hiver wrote:
>>>
>>> hi nick,
>>>
>>> - the oauth2 token is associated with an account which is owner of the 
>>> app.
>>> - 

Re: [google-appengine] Re: Error 401 when attempting to bulk download with appcfg.py

[solsTiCe d'Hiver]

hi. sorry

So to make thing a little clearer I hope:

I follow the standard procedure using appcfg.py using an oauth 2 token.
that's when I got that error message

2016-07-01 17:55:25,147 INFO client.py:981 Timeout attempting to reach GCE
> metadata service.
> Traceback (most recent call last):
>   File "/home/solstice/Applications/google_appengine/appcfg.py", line 133,
> in 
> run_file(__file__, globals())
>   File "/home/solstice/Applications/google_appengine/appcfg.py", line 129,
> in run_file
> execfile(_PATHS.script_file(script_name), globals_)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
> line 5495, in 
> main(sys.argv)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
> line 5486, in main
> result = AppCfgApp(argv).Run()
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
> line 2964, in Run
> self.action(self)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
> line 5142, in __call__
> return method()
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
> line 4924, in PerformDownload
> run_fn(args)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py",
> line 4829, in RunBulkloader
> sys.exit(bulkloader.Run(arg_dict, self._GetOAuth2Parameters()))
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
> line 4349, in Run
> return _PerformBulkload(arg_dict, oauth2_parameters)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
> line 4242, in _PerformBulkload
> return_code = app.Run()
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
> line 3408, in Run
> oauth2_parameters=self.oauth2_parameters)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py",
> line 1280, in __init__
> rpc_server_factory=throttled_rpc_server_factory)
>   File
> "/home/solstice/Applications/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py",
> line 747, in ConfigureRemoteApiForOAuth
> credentials = client.GoogleCredentials.get_application_default()
>   File
> "/home/solstice/Applications/google_appengine/lib/oauth2client/oauth2client/client.py",
> line 1211, in get_application_default
> return GoogleCredentials._get_implicit_credentials()
>   File
> "/home/solstice/Applications/google_appengine/lib/oauth2client/oauth2client/client.py",
> line 1201, in _get_implicit_credentials
> raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
> oauth2client.client.ApplicationDefaultCredentialsError: The Application
> Default Credentials are not available. They are available if running in
> Google Compute Engine. Otherwise, the environment variable
> GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining
> the credentials. See
> https://developers.google.com/accounts/docs/application-default-credentials
> for more information.


So I don't understand why the oauth token does not work and why I am asked
to use another method aka the GOOGLE_APPLICATION_CREDENTIALS file. Anyway...

As prompted, I go to
https://developers.google.com/accounts/docs/application-default-credentials
and follow the procedure from the "How the Application Default Credentials
work" paragraph.

Then, I use the .json as second auth method as asked as the first oauth
does not work. Therefore, I don't get the exception but get a 401 auth
error from the site.

and finally, I have tried without the s~ and it's still the same 401 error.

2016-04-19 16:40 GMT+02:00 'Nick (Cloud Platform Support)' via Google App
Engine :

> Hey solsTiCe,
>
> Unfortunately it's not quite clear to me which instructions you're
> following. Also, it seems strange that you'd need both credentials and a
> token. Could you be a little more specific about exactly what steps you're
> following to build "myapp.json"?
>
> Additionally, I can see in your bash script that you use "application=s~
> $appid", could you attempt a test without the '~s'?
>
> Thanks for clarifying,
>
> Nick
> Cloud Platform Community Support
>
> On Tuesday, April 19, 2016 at 6:31:54 AM UTC-4, solsTiCe d'Hiver wrote:
>>
>> hi nick,
>>
>> - the oauth2 token is associated with an account which is owner of the
>> app.
>> - to generate the credentials file I followed the instructions in the url
>> I gave. It seems both the oath2 token and the credentials is needed ? or
>> the oath2 is ignored ?
>> -I have not s~ in my bash script.
>>
>> I am trying to do that on the real server not the local dev server.
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> 

[google-appengine] Re: Error 401 when attempting to bulk download with appcfg.py

['Nick (Cloud Platform Support)' via Google App Engine]

Hey solsTiCe,

Unfortunately it's not quite clear to me which instructions you're 
following. Also, it seems strange that you'd need both credentials and a 
token. Could you be a little more specific about exactly what steps you're 
following to build "myapp.json"? 

Additionally, I can see in your bash script that you use "application=s~
$appid", could you attempt a test without the '~s'?

Thanks for clarifying,

Nick
Cloud Platform Community Support

On Tuesday, April 19, 2016 at 6:31:54 AM UTC-4, solsTiCe d'Hiver wrote:
>
> hi nick,
>
> - the oauth2 token is associated with an account which is owner of the app.
> - to generate the credentials file I followed the instructions in the url 
> I gave. It seems both the oath2 token and the credentials is needed ? or 
> the oath2 is ignored ?
> -I have not s~ in my bash script.
>
> I am trying to do that on the real server not the local dev server.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/a17043c9-1219-4274-9db4-b77938e6e9a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Error 401 when attempting to bulk download with appcfg.py

[solsTiCe d'Hiver]

hi nick,

- the oauth2 token is associated with an account which is owner of the app.
- to generate the credentials file I followed the instructions in the url I 
gave. It seems both the oath2 token and the credentials is needed ? or the 
oath2 is ignored ?
-I have not s~ in my bash script.

I am trying to do that on the real server not the local dev server.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0a8b687e-36da-4b12-a3ca-0b5e7f194551%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Error 401 when attempting to bulk download with appcfg.py

['Nick (Cloud Platform Support)' via Google App Engine]

Hey solsTiCe d'Hiver,

This appears to follow the symptoms of an issue some other users have 
observed . 


Other than this, I can think of some potentially fruitful questions to ask: 

* which oauth2 token are you using now, is it based on the default 
credentials of a service account? Does that service account have OWNER 
permission on your app?

* if not, how are you exactly generating the credentials json file?

* does deployment work if you remove 's~' from the app id in the bash 
script? 

With this information, we'll be much closer to diagnosing the exact nature 
of this issue, and determining what solutions to provide. 

Best wishes,

Nick
Cloud Platform Community Support

On Sunday, April 17, 2016 at 1:34:43 PM UTC-4, solsTiCe d'Hiver wrote:
>
> hi.
> Like I used to do it several years ago, I am attempting to *bulk download* 
> the *entities* of the datastore of my app with *appcfg.py* (from latest 
> python SDK)
>
> I am using this little script:
> #!/bin/bash
> date=`date +%Y%m%d`
> appid=myappid
> url="https://$appid.appspot.com/_ah/remote_api;
> GOOGLE_APPLICATION_CREDENTIALS=myapp.json appcfg.py download_data -v -e 
> myem...@gmail.com --application=s~$appid --url=$url --filename=allentities
> -$date.sql3
> rm -f bulkloader-*
>
>
> because this command does not ask for an oauth token, I have previously 
> run the command
>
> appcfg.py list_versions
>
> just to get the *oauth2* token
> And as said in 
> https://developers.google.com/accounts/docs/application-default-credentials 
> I have generated a *json* file with the google credentials and use it 
> with GOOGLE_APPLICATION_CREDENTIALS variable as shown above.
>
> However when I run the script I get a long list of
> 2016-04-17 19:21:29,316 INFO client.py:546 Attempting refresh to obtain 
> initial access_token 
> 2016-04-17 19:21:29,378 INFO client.py:804 Refreshing access_token 
> 2016-04-17 19:21:30,901 INFO client.py:578 Refreshing due to a 401 
> (attempt 1/2) 
> 2016-04-17 19:21:30,988 INFO client.py:804 Refreshing access_token
> 2016-04-17 19:21:31,570 INFO client.py:578 Refreshing due to a 401 
> (attempt 2/2) 
> 2016-04-17 19:21:31,659 INFO client.py:804 Refreshing access_token 
> 2016-04-17 19:21:32,436 INFO client.py:578 Refreshing due to a 401 
> (attempt 1/2) 
> 2016-04-17 19:21:32,526 INFO client.py:804 Refreshing access_token 
> 2016-04-17 19:21:33,162 INFO client.py:578 Refreshing due to a 401 
> (attempt 2/2) 
> [...]
>
>
> and this ends with 
> Error 401: --- begin server output ---
> You must be logged in as an administrator to access this.
> --- end server output ---
>
> I have the same 401 error in the log of my app.
>
> What could be wrong ?
> I am the owner of the app and my email is listed as such in the logins
>
> I don't want to enable billing and use the Google Cloud Platform to export 
> entities from the datastore with "Admin Datastore" (by the way clicking the 
> button "Open Admin Datastore" does nothing, one need to use a right click 
> and open in new tab to get something)
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9717111c-fa5d-430f-ba50-26b395464ade%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.