[google-appengine] Re: Unable to delete a key from memcache - App engine shows 'Deleting' for a very time

[A. Kong]

Forgot to include the url in my last post

https://console.cloud.google.com/m/gae/memcache/entry?namespace&key=&pid= 
Failed to load resource: the server responded with a status of 405 ()




On Monday, 31 October 2016 15:11:47 UTC+11, A. Kong wrote:
>
> I found that there is an error in chrome dev console when I attempt to 
> drop a key
>
>   Failed to load resource: the server responded with a status of 405 ()
>
> I checked the billing and there is no problem.
>
> It seems like I can flush the cache. 
>
>
>
> On Monday, 31 October 2016 10:35:49 UTC+11, A. Kong wrote:
>>
>> Hi all,
>>
>> Normally it is very easy to delete a key in memcache via console.cloud. 
>> It is usually just a matter of locating memcache result by a key, then 
>> select and delete the item. 
>>
>> However today (31 Oct, 10am Sydney, Australia time) I am unable to carry 
>> out this operation. I have tried a couple of time but the console just 
>> shows 'Deleting...' at the lower left corner but the item is not flushed 
>> from memcache. Nothing in the log to see either.
>>
>> Can someone from GAE team take a look at it?
>>
>>
>> Cheers
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/649a0997-eae6-4125-8f39-e00844a360cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Unable to delete a key from memcache - App engine shows 'Deleting' for a very time

[A. Kong]

I found that there is an error in chrome dev console when I attempt to drop 
a key

  Failed to load resource: the server responded with a status of 405 ()

I checked the billing and there is no problem.

It seems like I can flush the cache. 



On Monday, 31 October 2016 10:35:49 UTC+11, A. Kong wrote:
>
> Hi all,
>
> Normally it is very easy to delete a key in memcache via console.cloud. It 
> is usually just a matter of locating memcache result by a key, then select 
> and delete the item. 
>
> However today (31 Oct, 10am Sydney, Australia time) I am unable to carry 
> out this operation. I have tried a couple of time but the console just 
> shows 'Deleting...' at the lower left corner but the item is not flushed 
> from memcache. Nothing in the log to see either.
>
> Can someone from GAE team take a look at it?
>
>
> Cheers
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0e215ecf-0303-4a2e-97db-461900a1164c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Unable to delete a key from memcache - App engine shows 'Deleting' for a very time

[A. Kong]

Hi all,

Normally it is very easy to delete a key in memcache via console.cloud. It 
is usually just a matter of locating memcache result by a key, then select 
and delete the item. 

However today (31 Oct, 10am Sydney, Australia time) I am unable to carry 
out this operation. I have tried a couple of time but the console just 
shows 'Deleting...' at the lower left corner but the item is not flushed 
from memcache. Nothing in the log to see either.

Can someone from GAE team take a look at it?


Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/41d9c715-553b-4372-a4ff-b567bf7fe356%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: java.lang.IllegalArgumentException: operating on too many entity groups in a single transaction.

[V.B.]

Hi Nick,
   Is that value (the 25 limit) available as an environment variable, i.e. 
to avoid hard coding it in the rare instance when the value is needed?

On Monday, February 15, 2016 at 2:47:41 PM UTC-5, Nick (Cloud Platform 
Support) wrote:
>
> According to the documentation on XG-Transactions 
> ,
>  
> you should be able to operate on up to 25 entity groups in a cross-group 
> transaction. It does mention that it will fail if another transaction 
> running at the same time modifies any of those entity groups. This might be 
> happening, although only you would know your system to be able to say 
> definitively yes or no.
>
> If that isn't happening, I think it would be worth posting this, along 
> with simple code snippets allowing reproduction of the behaviour, in the 
> Public 
> Issue Tracker . We 
> do regular monitoring of the Public Issue Trackers and the issue would get 
> looked-at relatively quickly.
>
> Best wishes,
>
> Nick
>
> On Monday, February 15, 2016 at 11:23:50 AM UTC-5, Louise Elmose Hedegaard 
> wrote:
>>
>> I have a datastore with a kind "shop", a kind "order" and a kind 
>> "transaction".
>> One shop has many orders, and one order has many transactions.
>> The parent of a transaction is an order, the parent of an order is a shop.
>>
>> I want to delete all entries in the transactions table:
>>
>> public void cleanUp() {
>> Query query = new 
>> Query(TransactionDBFields.TRANSACTION_TABLE_NAME);
>> query.setKeysOnly();
>> query.setFilter(
>> new Query.FilterPredicate(
>> TransactionDBFields.CREATED_AT,
>> Query.FilterOperator.LESS_THAN_OR_EQUAL,
>> new Date())
>> );
>>
>> PreparedQuery preparedQuery = datastore.prepare(query);
>> Iterable entities = 
>> preparedQuery.asIterable(FetchOptions.Builder.withLimit(5));
>>
>> TransactionOptions options = 
>> TransactionOptions.Builder.withXG(true);
>> Transaction txn = datastore.beginTransaction(options);
>> for (Entity en : entities) {
>> datastore.delete(txn, en.getKey());
>> }
>> txn.commit();
>> }
>>
>> If I change the limit to anything larger than 5 I get the error:
>>
>> java.lang.IllegalArgumentException: operating on too many entity groups 
>> in a single transaction.
>>
>> - apparently as GAE does not allow operating on more than 5 entity groups 
>> in a single transaction. 
>>
>> Is there a way to delete more than 5 entities at a time? executing the 
>> above code continuously to delete all transaction entities does not seem 
>> reasonable, nor when considering performance.
>>
>> -Louise
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/fddb8a15-ace7-41af-b108-6512398cf389%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Backing up AppEngine

[Evan Jones]

On Sun, Oct 30, 2016 at 9:04 AM, Joshua Fox יהושע פוקס  wrote:
>
>  Worst case: We've discussed writing something ourselves multiple times,
>> that would scan our Datastore and writes entities out in a more useful way,
>> but we haven't prioritized it.
>>
>
> Evan, can you explain why this is not a high priority? Do you see
> accidental deletion as unlikely?
>


So far we are trusting that the existing datastore backup tool does
something useful. :) The reason we've considered doing our own backup thing
is to be able to do more selecting restores. For example, per-entity,
per-namespace, or per date (we have a bunch of entities that include a
timestamp attribute), depending on the reason. We haven't had the right
disaster yet to prioritize us doing that work.

I agree with you: I think protecting against programmer error is valuable!
We are just hoping that we do enough of it already, and haven't been burned
enough to cause us to put that ahead of new product work. Sadly, human
nature being what it is, its hard to invest resources into a possible
future disaster that has not yet happened.

Still: this discussion has reminded me that we still have a bit more work
to do on our Datastore and BigQuery backup stuff, so thanks!

Evan

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CACRPLjgvaL4smG34MLHHB%2BJF1QAW25vQm6dZuaOYp-CqdGHeJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] File referenced by handler not found: $PYTHON_LIB/default_start_handler.py

[Ted Henderson]

Hello! I found this warning level log firing every few minutes on /_ah/start

This request caused a new process to be started for your application, and 
thus caused your application code to be loaded for the first time. This 
request may thus take longer and use more CPU than a typical request for 
your application.

Everything in my app is working fine, and I have no idea what is causing 
this. I am worried it's going to jack up the cost of my billing. Here is my 
app.yaml, which I haven't edited in 
ages: https://gist.github.com/thedore17/3009d1b5cc0e2a2373923d36f1988623

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/f05ee39d-0c24-49ca-903f-8206c26166c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Google Drive API - Get the id of a file when Insert - Python

[Rameez Raja]

I wrote a piece of code in python to upload a file to google drive, which 
was successful. But I cant able to fetch the file id after the upload. 
Could you guys please help me to get that.


.
.
.
DRIVE = build('drive', 'v2', http=http)

FILES = (
('sample.csv', True),
)

for filename,convert in FILES:
metadata = {'title': filename}
res = DRIVE.files().insert(convert=convert, body=metadata, 
media_body=filename,fields='mimeType').execute()
print(res)
 print(a)
 if res:
 print('Uploaded "%s" (%s)' % (filename, res['mimeType']))




Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/198c01cd-1384-48f7-b3a3-fdf9645873b8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: AppEngine (AppInventor TinyWebDb application) access from a python local application

[Christian Feniou]

Hello Anastasios,

I thank you very much for this detailed answer. I think the N° 1 solution 
is the one I will use. I do not want any database or service modification 
it's why I'd like build a local Python App taking exactly the same 
mechanism than the Androïd app to access the datas... 

Thank you again. Have good Sunday.

Best regards, 

Christian.




Le dimanche 30 octobre 2016 01:01:39 UTC+2, Anastasios Hatzis a écrit :
>
> Christian,
>
> maybe you want your local Python client to directly interact with the 
> datastore used by the server app?
> In this case, you would need the Cloud Datastore client library for Python 
> 
>  (and 
> Tutorial  
> here). However, I would like to take a few aspects into consideration 
> (since I don't know what you actually want to achieve).
>
> I haven't looked into App Inventor, but from your server source code 
> sample I assume, that the server is running in Google App Engine (Python). 
> It is able to handle requests from either a client app on a smartphone (by 
> using JSON) or from a Web-browser (rendering and returning an HTML page). 
> The sample also shows that data is stored in Google Cloud Datastore. For 
> doing this inside a server app on GAE (Python), you have two libraries 
> available to access the Datastore API easily and directly from the GAE app: 
> the old `db` library (as in your sample), or the new `ndb` library. I don't 
> think you can use these libs outside of GAE.
>
> If you want to implement a local Python client in this setup you can pick 
> between two approaches:
> (1) Access the server app like your mobile app through some kind of HTTP 
> REST API. Advantage: business logic built into your server app is always 
> applied and the server can safeguard your data in many ways - whatever 
> client and user is interacting with the server app. Consider the potential 
> risk that a client is malicious or its code has been modified. 
> Disadvantage: this approach might require some more efforts in designing 
> the different layers (i.e. separation of concerns).
>
> (2) Directly access the data, with no server that would impose safeguard 
> measures. In this case, your local Python client does not interact with the 
> web-app on GAE, but directly with Google Cloud Datastore API. Advantage: 
> you can build a query directly into the local Python client, no need to 
> modify the server app. Disadvantage: if you allow the Python client to 
> write directly to the datastore, you can mess up the data in many ways 
> (especially if the client is not an internal tool of yours, but also used 
> by many other users).
>
> So, if you want a local Python client as internal admin tool, preferably 
> read-only datastore access, you should be fine with the second approach. In 
> that case, the *Cloud Datastore client library for Python* is your 
> choice. For everything else, I recommend the first approach, so your Python 
> client interacts with the server, just as your mobile app does. In this 
> case, you need to learn how your Python client can send HTTP requests to 
> your server app and how to process the response. And of course you would 
> need to learn how to implement any missing API / feature on your server app.
>
> I hope this was helpful for you.
>
> Anastasios
>
>
> Am Samstag, 29. Oktober 2016 18:51:33 UTC+2 schrieb Christian Feniou:
>>
>> Bonsoir George,
>>
>> Unfortunately, there is not the answer I'm looking for in these links ! 
>> There are many documents and examples to build and deploy a server 
>> application but the sample I search is a "client" python application. 
>>
>> In the server source code it's a difference between access from Web or 
>> access from application :
>>
>> def WriteToPhone(handler,tag,value):
>>  
>> handler.response.headers['Content-Type'] = 'application/jsonrequest'
>> json.dump(["VALUE", tag, value], handler.response.out)
>>
>> def WriteToWeb(handler, tag,value):
>> entries = db.GqlQuery("SELECT * FROM StoredData ORDER BY date desc")
>> template_values={"result":  value,"entryList":entries}  
>> path = os.path.join(os.path.dirname(__file__),'index.html')
>> handler.response.out.write(template.render(path,template_values))
>>
>> My main problem is how to pass the "questions" with the right parameters 
>> from the local python application to the server...
>>
>> I seen a module named "google.appengine.api.datastore"  looking as the 
>> module I need... But I don't know how to implement it.
>>
>> Thanks a lot...
>>
>>
>>
>> Le samedi 29 octobre 2016 09:55:34 UTC+2, Christian Feniou a écrit :
>>>
>>> Hello George,
>>>
>>> Thank you so much for your answer. I'll have a look on the addresses you 
>>> gave me. I hope I'll found the call back example I'm looking for !!
>>> I'll inform you soon for the result of these "investigations"...
>>>
>>>

[google-appengine] Re: Backing up AppEngine

[Joshua Fox יהושע פוקס]

George, thank you for that reply. I don't want to clutter the list, so feel
free to not respond, but there seems to be a real difference of approaches
here.  I now understand your approach better, based on responses from you
and other Googlers, but to me it is so counter-intuitive that I want to
understand where you're coming from.

I seem to be exceptionally concerned about the risk of accidental
deletion/overwriting. Doesn't it make sense to handle that as a high
priority?

Maybe I'll write up up an article if I get my head around this.

On Fri, Oct 28, 2016 at 5:35 PM, George (Cloud Platform Support) <
gsuceve...@google.com> wrote:

> Hello Joshua,
>
> Leaving data in the Blobstore and hoping for the best, as you say, is not
> such an unreasonable policy: as mentioned, there is a standards-validated data
> protection policy  in
> place, covering backups and all kind of risks to data in the cloud,
> including what you mention about hacker interference.
>

How does the policy cover that? If a hacker deletes our  Blobstore or
Datastore --  or indeed, if we delete them with bugs in our  code  or by
accidentally hitting the Delete button at https://console.cloud.
google.com/appengine/blobstore?project=myproject  or
https://ah-builtin-python-bundle-dot-myproject.appspot.
com/_ah/datastore_admin  -- does your data protection policy allow us to
 recover?



[image: Inline image 2][image: Inline image 3]


Your view on versioning and team member error seems to be influenced by a
> code development environment mindset. GAE is not specifically built for
> developing code on-line, you may refer to specialized third-party tools for
> that, for instance git .
>

Sure, we use Git for code.

These tools take care of versioning, and are able to revert data to a
> healthy previous state in case of team-member/human error.
>

Of course we test new code on our dev laptops, then in  special GAE
test-projects. We often delete the data in these test projects. We pay *very
*close attention to make sure we don't delete the data in the production
project -- but we are all human, and mistakes can happen.



> There are in fact versioning features offered for the instances running in
> the cloud; please refer to the “Versioning and instances” paragraph in the 
> overview
> of the app engine
> .
>
>

But from what I can see Git and AppEngine Versions do  not allow reverting
data, just code.


One can find a Backing up or restoring data chapter on the Managing
> Datastore from the Console
> 
> page.
>
> You are perfectly right about the Blobstore, the backup process does not
> include values stored in Blobstore or Cloud Storage
> .
> Applications do not create or modify blob data directly; instead, blobs are
> created indirectly, by a submitted web form or other HTTP POST request
> . This means
> data is not modified within Blobstore workings, so it maintains identity
> with the original data uploaded by the user. There is no immediate need to
> backup cloud data that is already stored in identical state in your system.
>
> Some tools may not yet work 100% as expected. You already submitted bugs
> for that, and we hope these issues are going to be address within
> reasonable timeframes.
>
> In short,  the difference in attitude towards accidental deletion or
> overwrite originates in our confidence in the highest standards implemented
> by our data protection policy
> .
>
>>

On Fri, Oct 28, 2016 at 8:31 PM, Evan Jones 
 wrote:

>  Worst case: We've discussed writing something ourselves multiple times,
> that would scan our Datastore and writes entities out in a more useful way,
> but we haven't prioritized it.
>

Evan, can you explain why this is not a high priority? Do you see
accidental deletion as unlikely?

Thank you,

Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CABfTu2AHnjvQRfQs2smh%2B-6ZiFTWvJYNtP_rV%2BHVhBFh9nmpZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Is there something like a "back_off" for ndb.transaction's?

[timh]

Consider using a roll forward model with tasks ensuring the transactions 
complete.

There is an old article by Nick Johnson covering the approach and I have 
used it to ensure reliable completion of a set of transactions (and that 
was in the day of 1min task deadline)

http://blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine

T

On Saturday, October 29, 2016 at 11:43:46 PM UTC+8, Kaan Soral wrote:
>
> My initial problem with push queues was that they were using their retries 
> completely internally, so once you get them to do "infinite retries", the 
> losses became 0%
>
> The problem with pull queues was throughput, sadly, pull queues are much 
> better aimed to battle contention, especially when used together with
>
> *Ps. I assume, for my original problem, manual retrying is the simplest 
> solution*
>
> What kind of a method with push queues do you use to solve the contention 
> issue tho?
> I forgot my original method that used only push queues, my current method 
> uses pull queues for tasks, and push queues for processing, there are 
> delays, there are losses (it's always an ongoing battle), that's why with 
> my new project, I just kept it strictly to simple transactions
>
> On Saturday, October 29, 2016 at 6:19:34 PM UTC+3, PK wrote:
>>
>> I have found  push task queues very reliable and use them all the time. 
>>
>> PK
>> http://www.gae123.com
>>
>> On Oct 29, 2016, at 7:29 AM, Kaan Soral  wrote:
>>
>> Thanks for sharing retrying, indeed it's pretty interesting
>>
>> My situation is pretty unique, that's why there are usually 3 consecutive 
>> requests in the worst case and at most 5-6, that's why an architecture 
>> change is an overkill
>>
>> I had a lot of experience with both push queue and pull queue based 
>> solutions in the past, and I can safely say that, "better let them content"
>>
>> In my opinion, both pull queues and push queues are extremely unreliable, 
>> so on this project, I made it so that the occasional contention is safe to 
>> occur
>>
>> However, I hoped with retries=6 the transactions would untangle them on 
>> their own, but they don't
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com.
>> To post to this group, send email to google-a...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/64683751-e390-4f16-ad0c-ec4a94246256%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/3d245479-a476-4aba-8fef-d7fc8ebedaf6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.