[google-appengine] Rejecting requests over a set size
PostBin.org is a very popular community tool used for debugging webhooks. Unfortunately, every now and then, somebody posts too much ... not in post count, but in frequency * request size, throwing us over our reasonable 4gb/day incoming bandwidth quota. I've considered running a proxy that checks for content-length before passing it to app engine, but obviously this sort of defeats the point of App engine. It would be nice if you had a way to, ideally, programmatically reject requests before it counts against the quota. I understand resources are already consumed by then, so maybe if you had some kind of header filter in the admin so I can reject requests with, say, over 100kb content-length? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Announcing the Matcher API for Trusted Testers
Yes, I would love to apply since I'm building an app that would be
perfect for this. However, the link is broken as Nick mentioned.
On Oct 19, 5:29 pm, Nickolas Daskalou n...@daskalou.com wrote:
Hi Ikai,
I've tried accessing the trusted tester list but I get this permission error
from Google Docs:
We're sorry, my email address does not have permission to access this
spreadsheet.
You are signed in as my email address, but that email address doesn't have
permission to access this spreadsheet. (Sign in as a different user or
request access to this document)
where my email address is this email address I'm sending from now (it's my
Google Apps + Google Account email address).
Nick
On 20 October 2010 11:10, Ikai Lan (Google)
ikai.l+gro...@google.comikai.l%2bgro...@google.com
wrote:
Hey everyone,
I wanted to announce that we are accepting signups for trusted testers for
the Python Matcher API, which is available for local testing in the 1.3.8
SDK. The Matcher API allows developers to take advantage of Google's high
performance matching infrastructure. Developers will be able to register a
large number of queries for incoming documents to be matched against. The
API will then match these queries against numerical and text properties of
incoming data at a very high rate.
To better illustrate what the Matcher API can do, let’s pretend you are
building a site that notifies users on stock ticker price changes. That is -
a user of the site might sign up and register to receive alerts anytime
BRK.A is greater than $500 but lower than $525 (by the way, if Berkshire
Hathaway is ever in this price range, sell everything you have and buy.
Disclosure: I am not a registered financial advisor). Here’s how this might
be implemented on App Engine before:
1. When a user wants to create a new alert, a new AlertCondition entity is
created. This entity records the ticker_symbol, min_price, max_price, and
email to notify.
2. On an incoming notification of a stock price change of BRK.A between
$500 and $525, we filter AlertCondition entities. Entities that match are
returned, and from these entities, we create offline tasks to email each of
the users about the price change.
This works decently, given that we don’t have many stock price changes or
many alerts in the system. As the number of AlertConditions go up, we will
need to change our application to break the queries into multiple pages, or
even move them into task queues. Unfortunately for us, stock prices change
very frequently, and (we hope) we will have many users. Fetching tens of
thousands of Alert Conditions from the datastore can take on the order of
seconds, causing the implementation detailed above to be difficult to scale
for our expected usage.
How does the Matcher API help us solve this problem?
---
The Matcher API allows us to register a set of queries, then filter
incoming documents against these queries in a scalable, high-performance
fashion. The type of problem being solved in the stock price notification
example is a good example of how the Matcher API can be used. Here’s what
we’d have to do in our application using the Matcher API:
1. When a user wants to create a new alert, we acquire an instance of a
Python matcher object and register queries.
2. On an incoming stock price change, we run the alert against the matcher
and try to find all the queries that matched. As queries are found, the
matcher API enqueues tasks to process the results offline. Unlike the
implementation using the datastore in the earlier example, the Matcher API
performs checks in parallel and runs using a completely different service
optimized for this use case. A single price change notification could match
against hundreds of thousands of queries in the time span of a few seconds.
Let’s show this example in code (also posted here:
http://pastie.org/1234174):
# We’re going to call subscribe. Here’s what we’re passing:
# dict - this means we are going to match against a Python dictionary. We
can also
# pass a db.Model type to match against. For instance, StockPrice
# “symbol: GOOG AND price 500 AND price 525” - this is our query
# “ikai:GOOG” - this is the name of our subscription. We’ll use this to map
back to our
# User. This must be unique, so we are using the User key and
ticket combination
matcher.subscribe(dict, “symbol: GOOG AND price 500 AND price 525”,
“ikai:GOOG”)
# When a new stock price update comes in, we create a Python dictionary
representing
# all the parts we care about
change = { symbol : GOOG, price : 515 }
matcher.match(change)
# The code above doesn’t execute right away. It makes an API call to
Google’s
# matcher service, which, upon completion, begins dispatching matches to a
# task queue at the URI path /_ah/matcher. You’ll need to define the task
[google-appengine] Re: Keymaster to store your keys
I've since built a helper module: http://github.com/hackerdojo/hd-domain/blob/master/keymaster.py On Jan 23, 6:01 am, Barry Hunter barrybhun...@googlemail.com wrote: Do you have an example client implementation? I'm not totally sure how to use it. (but am looking though the source) 2010/1/22 Jeff Lindsay progr...@gmail.com: For a while I've been complaining App Engine needs Admin defined environment variables that can be used to store api keys, passwords, and other sensitive information you wouldn't want in your source code repository. For a while, I was making a keys.py file that I'd import that I'd keep out of the repository. Unfortunately, in rare cases I would switch machines and deploy, forgetting I didn't have keys.py and the app would break. So it's a pretty fragile solution. I had an idea to keep my keys in a hardcoded Scriptlet (scriptlets.org), but it wouldn't be very secure. So I expanded on the idea and built an app just for this purpose: http://thekeymaster.org Maybe some of you will find it useful. You'd first have to trust its encryption, which is done with the PyCrypt library. The source is also open so what's going on is all transparent. No doubt many of you will be afraid to use it. That's fine. I'm not. But I figure some of you might find it useful. Cheers! Jeff Lindsay -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Keymaster to store your keys
For a while I've been complaining App Engine needs Admin defined environment variables that can be used to store api keys, passwords, and other sensitive information you wouldn't want in your source code repository. For a while, I was making a keys.py file that I'd import that I'd keep out of the repository. Unfortunately, in rare cases I would switch machines and deploy, forgetting I didn't have keys.py and the app would break. So it's a pretty fragile solution. I had an idea to keep my keys in a hardcoded Scriptlet (scriptlets.org), but it wouldn't be very secure. So I expanded on the idea and built an app just for this purpose: http://thekeymaster.org Maybe some of you will find it useful. You'd first have to trust its encryption, which is done with the PyCrypt library. The source is also open so what's going on is all transparent. No doubt many of you will be afraid to use it. That's fine. I'm not. But I figure some of you might find it useful. Cheers! Jeff Lindsay -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Task API
Are you testing this locally? I know that at least the local dev server is not actually letting tasks run automatically. If you go to the Admin Console and look at the task queues it actually says Tasks will not run automatically. I find this VERY strange ... and annoying. But perhaps this is the cause of the problem you're having. -jeff On Jul 1, 2:09 pm, Paul Kinlan paul.kin...@gmail.com wrote: Just to add a quick note, there might have been an issue with the url that I was calling with a task (part of the path was not being url encoded), but I still think it is not doing exactly what it should (that and I can't tell what it is in the queue at the moment) Paul On Jul 1, 9:51 pm, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, Are there any issues with Task Queue API at the moment, I currently have about 2000+ items in two separate queues. That seem not to be getting executed even if I am queuing new items I don't believe there is an error in the code because if I run the equivalent code by calling the handler it works ok. The tasks each are using several URL fetches, but there are no errors being logged that I can see. Application name: twitterautofollow Paul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
