Sounds great. Any plans for Java support for this api? Please say YES :) Jeff
On Tue, Oct 19, 2010 at 8:10 PM, Ikai Lan (Google) <ikai.l+gro...@google.com<ikai.l%2bgro...@google.com> > wrote: > Hey everyone, > > I wanted to announce that we are accepting signups for trusted testers for > the Python Matcher API, which is available for local testing in the 1.3.8 > SDK. The Matcher API allows developers to take advantage of Google's high > performance matching infrastructure. Developers will be able to register a > large number of queries for incoming documents to be matched against. The > API will then match these queries against numerical and text properties of > incoming data at a very high rate. > > To better illustrate what the Matcher API can do, let’s pretend you are > building a site that notifies users on stock ticker price changes. That is - > a user of the site might sign up and register to receive alerts anytime > BRK.A is greater than $500 but lower than $525 (by the way, if Berkshire > Hathaway is ever in this price range, sell everything you have and buy. > Disclosure: I am not a registered financial advisor). Here’s how this might > be implemented on App Engine before: > > 1. When a user wants to create a new alert, a new AlertCondition entity is > created. This entity records the ticker_symbol, min_price, max_price, and > email to notify. > > 2. On an incoming notification of a stock price change of BRK.A between > $500 and $525, we filter AlertCondition entities. Entities that match are > returned, and from these entities, we create offline tasks to email each of > the users about the price change. > > This works decently, given that we don’t have many stock price changes or > many alerts in the system. As the number of AlertConditions go up, we will > need to change our application to break the queries into multiple pages, or > even move them into task queues. Unfortunately for us, stock prices change > very frequently, and (we hope) we will have many users. Fetching tens of > thousands of Alert Conditions from the datastore can take on the order of > seconds, causing the implementation detailed above to be difficult to scale > for our expected usage. > > > How does the Matcher API help us solve this problem? > ------------------- > > The Matcher API allows us to register a set of queries, then filter > incoming documents against these queries in a scalable, high-performance > fashion. The type of problem being solved in the stock price notification > example is a good example of how the Matcher API can be used. Here’s what > we’d have to do in our application using the Matcher API: > > 1. When a user wants to create a new alert, we acquire an instance of a > Python matcher object and register queries. > > 2. On an incoming stock price change, we run the alert against the matcher > and try to find all the queries that matched. As queries are found, the > matcher API enqueues tasks to process the results offline. Unlike the > implementation using the datastore in the earlier example, the Matcher API > performs checks in parallel and runs using a completely different service > optimized for this use case. A single price change notification could match > against hundreds of thousands of queries in the time span of a few seconds. > > Let’s show this example in code (also posted here: > http://pastie.org/1234174): > > # We’re going to call subscribe. Here’s what we’re passing: > # dict - this means we are going to match against a Python dictionary. We > can also > # pass a db.Model type to match against. For instance, StockPrice > # “symbol: GOOG AND price > 500 AND price < 525” - this is our query > # “ikai:GOOG” - this is the name of our subscription. We’ll use this to map > back to our > # User. This must be unique, so we are using the User key and > ticket combination > matcher.subscribe(dict, “symbol: GOOG AND price > 500 AND price < 525”, > “ikai:GOOG”) > > > # When a new stock price update comes in, we create a Python dictionary > representing > # all the parts we care about > change = { "symbol" : "GOOG", "price" : 515 } > > matcher.match(change) > > # The code above doesn’t execute right away. It makes an API call to > Google’s > # matcher service, which, upon completion, begins dispatching matches to a > # task queue at the URI path /_ah/matcher. You’ll need to define the task > queue handler: > > application = webapp.WSGIApplication( > [('/_ah/matcher', ChangeNotificationHandler)]) > > # You'd define the handler, a web handler for the results: > > class ChangeNotificationHandler(webapp.RequestHandler): > def post(self): > user_ids = self.request.get_all('id') # Returns > ['ikai:GOOG'] > results_count = self.request.get('results_count') # Returns the total > number of results > results_offset = self.request.get('results_offset') # Returns 0 > > for id in user_ids: > user_id, symbol = id.split(":") > # now we have user_id and symbol > # we’ll use the user_id to find the User and send them an email! > > # Note that subscriptions last, by default, 24 hours, so we'll need to > create a > # cron job that re-registers them. > > What makes Matcher API really powerful are the performance characteristics. > We can easily return hundreds of thousands of matches in seconds. > > > Tip of the iceberg > ------------------- > > It’s possible to filter on many other types data. Here are a few examples > of what this API could be used for: > > - matching incoming status updates for specific words or phrases (think > Google Alerts or Twitter real-time search updates) > - creating a real time notification system for location based services like > Google Latitude, allowing users to subscribe to their favorite locations for > users matching certain criteria > - any kind of notification service with a large number of notifications and > incoming data > > The full API is much more robust than the stock prices example. You can > find more documentation here: > > > http://code.google.com/p/google-app-engine-samples/wiki/AppEngineMatcherService > > You'll also want to see the sample application here: > > > http://code.google.com/p/google-app-engine-samples/source/browse/#svn/trunk/matcher-sample > > > Sounds cool, what do I have to do? > ------------------- > > 1. Start playing around with the Matcher API in your local SDK! > > 2. Add yourself to the trusted tester list here: > > > https://spreadsheets4.google.com/a/google.com/viewform?formkey=dEc5eFp4NmRqdHI5Rk40M0FWdHBCbUE6MQ > > Check it out and sign up if this is something you can make use of! If you > have any questions about what the API can be used for, let us know and we’ll > try to answer any questions to may have. > > - Ikai, posted on behalf of Bob, Bartek and the Matcher API team > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
