[appengine-java] Re: Displaying 1000 tweets eats up datastore quotas in minutes

[jacek.ambroziak]

The only issue is with indexing and access patterns.
When tweets are stored individually they can be retrieved
by queries referring to individual metadata.
This unfortunately is costly.
So the question remains, is it OK to retrieve whole sets
of tweets compressed together into a Blob in Entity?

On Jan 10, 7:30 am, de Witte  wrote:
> 3000 tweets takes about 750 kb, zipped 120Kb, so you can easily store
> 20.000 tweets in a single entity.
>
> Perform any query in memory and you are done. Probably well within the free
> quota limits.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Full-text search engine and its indexes

[jacek.ambroziak]

I have full-text indexed all of O'Reilly eBooks (1,600 of them) using
my own search engine. You can see how the search works if you have an
Android tablet and install (free app) eCarrel (O'Reilly bookstore). To
make searching manageable in the context of GAE I have originally
partitioned all the books into 4 groups, each with its own index. That
way searches can be performed in parallel (merging results when done),
individual (per book group) indexes are smaller, and w/i group search
is faster.

An index for 400 books is about 90 MB in size. To implement the search
engine on GAE
I would dedicate 4 applications to the task (eg. search1.appspot.com,
through search4).
Each application would run exactly the same code, but would have
different "application files"
containing index data.  (I wasn't sure if the index data should be
stored in DataStore entities,
Blobstore blobs, or application files; at the time the SE was first
implemented it seemed
that application files was the only option even if it meant that they
had to be split into
10 MB chunks (1.5.5 supposedly raises the limit to 32 MB but I got an
error attempting that)).

One problem with that approach is that multiple GAE applications are
used to implement
parallel "search servers." Another is that it takes time and resources
to read in the index
from application files into RAM before search results can be computed.
When an instance
is killed all this work goes to waste and will have to be repeated on
next search.
When the number of groups was too small and therefore indexes too big,
I was getting OutOfMemory errors just loading index data to RAM.

Do you guys think it is a good idea to use application files to store
index data?

Since each "search server" runs the same code (and only accesses
different application
files), can it be implemented via a single (versioned?) GAE
application? (I will run out of applications when adding more search
servers, and it will become more costly to run
the search engine).

http://ecarrel.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: How to use XSLT in JAVA

[jacek.ambroziak]

XSLT is not a parser, but a language to express XML transformations.

2 questions:

1) why do you need XSLT in your app?
2) do you need high performance? Here I can help w/ software.

As to StreamSource, you have other constructors for it,
not necessarily using Files!

On Feb 18, 2:56 am, Rick Smith  wrote:
> Hi all
>
> I am new to google app engine and I need to use XSLT parser. I was
> able to create an example by using file.
>
> TransformerFactory tfactory = TransformerFactory.newInstance();
> Transformer transformer = tfactory.newTransformer(new StreamSource(new
> File(xslID)));
> transformer.transform(new StreamSource(new File(sourceID)), new
> StreamResult(System.out));
>
> It works well on local system. But fails on gae as gae does not
> support files.
>
> Any alternate for this will be highly appreciated as I have no other
> way but to use XSLT.
>
> Thanks in advance
>
> Rick

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: how to access all values a property has?

[jacek.ambroziak]

Thank you for answering.
Indeed, I do keep a "running count"
and I use MapRed to compute the extent of a property (of a kind).

Recently I watched again the Google I/O video w/ Brett Slatkin
where he talks about self-join, and the zig-zag algorithm
in about minute 27 into that video.
Indexes need to store all values, and ordered as well.
Depending on index table implementation it could be easy
to retrieve the set of values.

It is not a very big deal -- just wanted to avoid maintaining extents
explicitly.


On Dec 21, 1:38 pm, "Ikai Lan (Google)" 
wrote:
> No, we can't know all possible values without traversing the entire index.
> Your best bet is to keep a "running count" - when someone enters a new
> color, save it. If you need to generate this data based on existing
> properties, I would suggest appengine-mapreduce 
> (http://code.google.com/p/appengine-mapreduce).
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blogger:http://googleappengine.blogspot.com
> Reddit:http://www.reddit.com/r/appengine
> Twitter:http://twitter.com/app_engine
>
> On Tue, Dec 21, 2010 at 9:22 AM, jacek.ambroziak
> wrote:
>
>
>
>
>
>
>
> > Say, I have a set of Person Entities with List favoriteColor
> > property.
> > People can specify their own sets of colors for that property.
>
> > Can I access the union of these color sets, ie. all the values they have
> > entered
> > for this property? The values should sit in the DataStore index, ordered,
> > but can we access such data?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to
> > google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine-java?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] how to access all values a property has?

[jacek.ambroziak]

Say, I have a set of Person Entities with List favoriteColor 
property.
People can specify their own sets of colors for that property.

Can I access the union of these color sets, ie. all the values they have 
entered
for this property? The values should sit in the DataStore index, ordered,
but can we access such data?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: data in xml format from data store

[jacek.ambroziak]

Also minify and/or compress your XML!

On Dec 2, 5:06 am, pac  wrote:
> That’s great then Stephen, appreciated the feedback. Thanks.
>
> On Dec 2, 1:44 am, Stephen Johnson  wrote:
>
> > I believe Ikai's statement "The 30 second limit is imposed on the time it
> > takes the servlet to finish." is confirming what I said.
>
> > On Wed, Dec 1, 2010 at 5:52 PM, pac  wrote:
> > > Hi Ikai,
>
> > > If you could confirm this - from earlier Stephen's post:
>
> > > "then I think you
> > > will be safe even though it might take another minute or two for
> > > Google's
> > > infrastructure to stream it to the client. I have not tested this
> > > though.
> > > Can anyone confirm or deny this assumption on my part."
>
> > > is download outside 30 sec limit i.e. after servlet finished in 30
> > > seconds?
>
> > > Thanks.
>
> > > On Dec 1, 12:00 am, "Ikai Lan (Google)" 
> > > 
>
> > > wrote:
> > > > The 30 second limit is imposed on the time it takes the servlet to
> > > finish.
>
> > > > --
> > > > Ikai Lan
> > > > Developer Programs Engineer, Google App Engine
> > > > Blogger:http://googleappengine.blogspot.com
> > > > Reddit:http://www.reddit.com/r/appengine
> > > > Twitter:http://twitter.com/app_engine
>
> > > > On Mon, Nov 29, 2010 at 10:13 AM, Stephen Johnson <
> > > onepagewo...@gmail.com>wrote:
>
> > > > > I believe the 30 second limit is imposed on the time it takes for your
> > > > > servlet to finish. Thus, if your servlet finished generating and
> > > returning
> > > > > the file to the AppEngine infrastructure in 29 seconds, then I think
> > > you
> > > > > will be safe even though it might take another minute or two for
> > > Google's
> > > > > infrastructure to stream it to the client. I have not tested this
> > > though.
> > > > > Can anyone confirm or deny this assumption on my part.
>
> > > > > On Mon, Nov 29, 2010 at 10:57 AM, pac 
> > > wrote:
>
> > > > >> Sorry, did not see your post earlier Stephen.
>
> > > > >> I was not aware of this 10 minute feature to create a file, I will
> > > > >> look into.
>
> > > > >> File size will be greater then 10MB, it is an existing website which 
> > > > >> I
> > > > >> am thinking to port to gae, file size already gets around 12MB or so.
>
> > > > >> So I guess more than 10MB download will be an issue and I guess
> > > > >> download should also complete in 30 second, is that so?
>
> > > > >> On Nov 29, 5:37 pm, pac  wrote:
> > > > >> > A client will download a xml file instead of uploading.
> > > > >> > Records will be created (or updated, deleted) in a table/kind over
> > > > >> > time and all records from a kind need to be downloaded in form of a
> > > > >> > xml file, I hope I am explaining the issue bit more clearly.
>
> > > > >> > I am not sure if I could make use of blobstore in this case as it
> > > will
> > > > >> > require update of blobstore object (in background may be using 
> > > > >> > task)
> > > > >> > every time a record gets added/updated/deleted i.e. if I create xml
> > > > >> > file programmaticaly and store as blobstore object.
>
> > > > >> > On Nov 29, 5:14 pm, Didier Durand  wrote:
>
> > > > >> > > Hi,
>
> > > > >> > > To emulate files, you have to use the Blob object in the 
> > > > >> > > datastore
> > > to
> > > > >> > > store such xml content: see
> > > com.google.appengine.api.datastore.Blob
> > > > >> inhttp://
> > > code.google.com/appengine/docs/java/datastore/dataclasses.html.
> > > > >> ..
>
> > > > >> > > Blob has a limited max size: 1 Mbyte. So, you can use a 
> > > > >> > > collection
> > > > >> > > (Vector, List, etc..)  of Blobs if you need more than 1 Mbytes.
>
> > > > >> > > If your files get uploaded by a client, you can use the Blobstore
> > > > >> > > where the max size is much bigger(2 Gbytes): seehttp://
> > > > >> code.google.com/appengine/docs/java/blobstore/overview.html
>
> > > > >> > > You have to associate the blob with some other fields (name, 
> > > > >> > > size,
> > > > >> > > last_updated) in you pojo to reproduce what you expect from a
> > > file.
>
> > > > >> > > N.B. if you need to emulate a directory tree, you also have to
> > > write
> > > > >> > > it by yourself.
>
> > > > >> > > Finally, I would recommend alternative Objectify rather than
> > > standard
> > > > >> > > JDO for such services.
>
> > > > >> > > Hope this helps
>
> > > > >> > > regards
> > > > >> > > didier
>
> > > > >> > > On Nov 29, 5:53 pm, pac  wrote:
>
> > > > >> > > > In a website I need to provide a feature to get data in xml
> > > format
> > > > >> > > > i.e. some url
>
> > > > >> > > > e.g.http://www.mysite.com../data.xml
>
> > > > >> > > > I think in gae I can not create a file. Any suggestions to
> > > create
> > > > >> such
> > > > >> > > > feature to get data from data store in this way?
>
> > > > >> > > > Data store will have large number of records, so in general, a
> > > file
> > > > >> > > > created from that could be of good few MBs.
>
> > > > >> --
> > > > >> You received this message because you are subscribed to the Googl

[appengine-java] Re: Why can't i persist a BitSet

[jacek.ambroziak]

You can serialize the BitSet into a ByteArrayOutputStream,
then persist a Blob wrapper around resulting byte array

On Nov 11, 11:21 am, WillSpecht  wrote:
> I am trying to store a BitSet to my data store but when I try to use
> it I get the following error:
>
> java.lang.IllegalArgumentException: thisRuleSet: java.util.BitSet is
> not a supported property type.
>
> Can I not store a BitSet even though it is serializable and if I
> can't, what is the suggested work around?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: GAE or EC2

[jacek.ambroziak]

Ikai,

I do not think you really addressed JY's pain, which I also share.
Some of us have invested a lot of time and energy into building
GAE-based services. We would like to move to "production" stage
but frequent "denials of service" make our applications look bad
to customers.
Also, for what I am trying to do,

1) I need more than 10 applications
2) more space for static files in WARs
3) more memory (quite often I run into OutOfMemory errors)
4) and of course fewer cold starts or 500 errors

I really like GAE's APIs, tools and dashboard
but I find it *difficult to deliver* a service
that I need to host in the Cloud
based on GAE's current reality.

GAE is developer friendly (modulo all the contortions
we need to go through when, say, a data chunk is 1.2 MB),
but as far as production goes ... well ... :-(

Is there a hope for us on the horizon
or do we have to bite the bullet and migrate elsewhere?

PS. For some apps (I do have an Android news app which
preparses Reuters news on GAE and serves binary XML
to Android) I never had any issues; the app does not
need much memory or data to work, and it hardly uses
the datastore) -- so for some small apps things work
just great. More demanding services do run into walls
unfortunately.


On Nov 8, 9:50 pm, "Ikai Lan (Google)" 
wrote:
> I'm biased, naturally, but here's my piece:
>
> They're both great, young platforms, and successful businesses have been
> built on both Google App Engine as well as Amazon EC2. Amazon's lower level
> access appeals to some, whereas Google App Engine's abstractions appeal to
> the folks in this group.
>
> Have you actually used EC2 before? If not, new accounts can sign up for a
> free "micro" instance. You'll want to check this out before you make any
> decisions. Since you're already looking at EC2, why not look at VPS
> solutions? I personally prefer a provider such as Slicehost or Linode to EC2
> for anything I can't currently do on Google App Engine.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blogger:http://googleappengine.blogspot.com
> Reddit:http://www.reddit.com/r/appengine
> Twitter:http://twitter.com/app_engine
>
>
>
>
>
>
>
> On Mon, Nov 8, 2010 at 5:59 PM, JY  wrote:
> > I want to ask fellow GAE users whether you think GAE is good choice to
> > build real business on?
> > I have used GAE for some personal projects, and right now help a
> > friend on a tiny start up (but with big dream). The project is related
> > to social network - I am thinking of two options:
> > 1, GAE
> > The benefits are obvious. However, it is also locking you inyou
> > don't have much control over it. If you are unhappy later, you will
> > have to redo the persistence layer and migrate data etc.
> > 2, EC2
> > You have much more control, and if the start-up gets funding, it is
> > easy to migrate to dedicated hosting or even its own data center. The
> > down side is developer (me, and only me) will have to spend quite some
> > time to take care of the infrastructure (although I like this kind of
> > work...)
>
> > I think my biggest concern of GAE is over the quality of service, and
> > the data-store - if I go EC2, I probably will use a NoSQL solution
> > like MogoDb, or Cassandra. The development efforts might be similar -
> > all products are sort of young, hot, and limited.
>
> > Thanks.
>
> > JY
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to
> > google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine-java?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] mapreduce & cron

[jacek.ambroziak]

How to start a MapReduce job as a cron scheduled task?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Error code 203

[jacek.ambroziak]

And another message I see way too often is
"Request was aborted after waiting too long to attempt to service your
request.
This may happen sporadically when the App Engine serving cluster is
under unexpectedly high or uneven load.
If you see this message frequently, please contact the App Engine
team."

On Oct 22, 10:17 am, "jacek.ambroziak" 
wrote:
> "If you see this message frequently, you should contact the App Engine
> team. (Error code 203)"
>
> Well, I actually *do* see this message frequently :-(
> so who do I contact, and what can we do about it?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Error code 203

[jacek.ambroziak]

"If you see this message frequently, you should contact the App Engine
team. (Error code 203)"

Well, I actually *do* see this message frequently :-(
so who do I contact, and what can we do about it?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Query on Many to Many Relation

[jacek.ambroziak]

Doesn't your 1st table list all the items tagged with a particular
tag?

On Aug 22, 3:44 am, Raghu  wrote:
> Hi All,
>
> I just want to reopen the existing challenge. I'm trying hard to think
> query on 2 tables with m-m relation.
>
> Tags{
> List  items;}
>
> Item{
> List  tags;
>
> }
>
> Please suggest me how to find all the items with given tag. is there
> any trick here.
>
> thanks
>
> raghu

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Reading blobs larger an 1MB

[jacek.ambroziak]

You need to read your Blob's content in several chunks
that are no larger than the limit.
Combining this with XML file compression can also make sense.

On Jul 25, 9:07 pm, korey_sed  wrote:
> I am stumped.  No matter what API call I try to read a large blob, I
> get the following:
>
> Caused by: com.google.apphosting.api.ApiProxy$ApplicationException:
> ApplicationError: 6: Blob fetch size too large.
>
> I have tried using the BlobStoreInputStream,
> BlobStoreService.fetchData, and even BlobStoreService.serve with a
> fetch URL which also fails since URLs also have a 1MB limit.  My
> latest attempt was using fetchData setting the length to
> MAX_BLOB_FETCH_SIZE.
>
> how are you suppose to read large data out of the blob store?  all the
> examples are using serve which does not help me.
>
> I have a 2MB XML file that I am trying to fetch and parse after it is
> uploaded.
>
> any help is much appreciated.  Code examples are even more
> appreciated.
>
> Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: rename a app-id

[jacek.ambroziak]

Not really, but you can delete the app
and after it is gone
create a new one with the (available) name you want

On Jul 18, 12:02 am, aswath satrasala 
wrote:
> Hello,
> Is there any way I can rename my app-id.
>
> -Aswath

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] BlobstoreInputStream

[jacek.ambroziak]

BlobstoreInputStream is documented in the API
but I can't find it in the jars...

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: App Engine SDK 1.3.2 is out!

[jacek.ambroziak]

And in the Java documentation the last index argument is EXCLUSIVE
as is standard in Java (so that end - start = length)(and so that the
previous end can
become start in a loop)

On Mar 31, 12:34 am, Robin B  wrote:
> The docs do mention the index values are inclusive:
>
> http://code.google.com/appengine/docs/python/blobstore/functions.html
>
> "Byte indexes are inclusive, with the first byte having an index of 0.
> For example, a start_index of 9 and an end_index of 19 will return up
> to 11 bytes, starting from the 10th byte (index 9) in the value."
>
> Robin
>
> On Mar 30, 5:21 pm, Ikai Lan  wrote:
>
>
>
> > I've reproduced the off-by-one error. Thanks for bringing this up.
>
> > On Mar 27, 8:32 am, "jacek.ambroziak" 
> > wrote:
>
> > > fetchData is very useful. At the moment its implementation seems to
> > > suffer from an off-by-one bug.
> > > For instance, fetchData(key, 0, 1024) returns an array of length 1025
> > > and
> > > blobstoreService.fetchData(blobKey, 0,
> > > BlobstoreService.MAX_BLOB_FETCH_SIZE)
> > > will, I guess, attempt to read one byte too many and it fails with
>
> > >   com.google.appengine.api.blobstore.BlobstoreFailureException: An
> > > unexpected error occurred.
>
> > > On Mar 25, 8:48 pm, "Ikai L (Google)"  wrote:
>
> > > > Check it out!
>
> > > >http://googleappengine.blogspot.com/2010/03/app-engine-sdk-132-releas...
>
> > > > Blog post:
>
> > > > App Engine SDK 1.3.2
> > > > Released<http://googleappengine.blogspot.com/2010/03/app-engine-sdk-132-releas...>
>
> > > > Today we are excited to announce the release of version 1.3.2 of the App
> > > > Engine SDK for both the Java and Python runtimes. 1.3.2 includes a 
> > > > number of
> > > > changes and bug fixes.
>
> > > > For this release, we have concentrated on removing a number of 
> > > > limitations
> > > > that have been affecting developers:
>
> > > >    - Blobstore API - A new method
> > > > (fetch_data<http://code.google.com/appengine/docs/python/blobstore/functions.html>
> > > > for
> > > >    Python, 
> > > > fetchData<http://code.google.com/appengine/docs/java/javadoc/com/google/appengi...>
> > > > for
> > > >    Java) allows your application to request the contents of a Blob from 
> > > > within
> > > >    your application’s code.
> > > >    - URLFetch API - We’ve expanded the number of ports you can access 
> > > > with
> > > >    the URLFetch API. You can now access ports 80-90, 440-450, and 
> > > > 1024-65535.
> > > >    - Mail API - We’ve expanded the allowed mail attachments to include
> > > >    common document extensions including .doc, .ppt, and .xls.
> > > >    - Task Queue API - We’ve increased the maximum total Task Queue 
> > > > refill
> > > >    rate to 50 per second.
>
> > > > We’re also happy to announce, based on your feedback, a new Denial of
> > > > Service (DoS) blocking system in App Engine. This system allows you to
> > > > blacklist specific IP addresses from accessing your application, and to
> > > > prevent them from costing your application money or eating up your 
> > > > quota.
> > > > You can also view the top IPs that have accessed your application in the
> > > > Admin Console, to help you figure out what IPs you may want to block. 
> > > > More
> > > > information on this feature is available for
> > > > Python<http://code.google.com/appengine/docs/python/config/dos.html>
> > > > and Java <http://code.google.com/appengine/docs/java/config/dos.html>.
>
> > > > There’s a lot of other changes and fixes in this release, including a 
> > > > new
> > > > Java version of the Appstats profiling tool, so read the release notes (
> > > > Python <http://code.google.com/p/googleappengine/wiki/SdkReleaseNotes>,
> > > > Java<http://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes>)
> > > > for a complete list of changes and download the new versions of the
> > > > SDK<http://code.google.com/appengine/downloads.html>
> > > > .
>
> > > > Release Notes:
>
> > > > Version 1.3.2 - March 25, 2010
>
> > > >    - New API to read the contents of uploaded Blobs (fetch_data)
> > > >    -htt

[appengine-java] Re: App Engine SDK 1.3.2 is out!

[jacek.ambroziak]

fetchData is very useful. At the moment its implementation seems to
suffer from an off-by-one bug.
For instance, fetchData(key, 0, 1024) returns an array of length 1025
and
blobstoreService.fetchData(blobKey, 0,
BlobstoreService.MAX_BLOB_FETCH_SIZE)
will, I guess, attempt to read one byte too many and it fails with

  com.google.appengine.api.blobstore.BlobstoreFailureException: An
unexpected error occurred.


On Mar 25, 8:48 pm, "Ikai L (Google)"  wrote:
> Check it out!
>
> http://googleappengine.blogspot.com/2010/03/app-engine-sdk-132-releas...
>
> Blog post:
>
> App Engine SDK 1.3.2
> Released
>
> Today we are excited to announce the release of version 1.3.2 of the App
> Engine SDK for both the Java and Python runtimes. 1.3.2 includes a number of
> changes and bug fixes.
>
> For this release, we have concentrated on removing a number of limitations
> that have been affecting developers:
>
>    - Blobstore API - A new method
> (fetch_data
> for
>    Python, 
> fetchData
> for
>    Java) allows your application to request the contents of a Blob from within
>    your application’s code.
>    - URLFetch API - We’ve expanded the number of ports you can access with
>    the URLFetch API. You can now access ports 80-90, 440-450, and 1024-65535.
>    - Mail API - We’ve expanded the allowed mail attachments to include
>    common document extensions including .doc, .ppt, and .xls.
>    - Task Queue API - We’ve increased the maximum total Task Queue refill
>    rate to 50 per second.
>
> We’re also happy to announce, based on your feedback, a new Denial of
> Service (DoS) blocking system in App Engine. This system allows you to
> blacklist specific IP addresses from accessing your application, and to
> prevent them from costing your application money or eating up your quota.
> You can also view the top IPs that have accessed your application in the
> Admin Console, to help you figure out what IPs you may want to block. More
> information on this feature is available for
> Python
> and Java .
>
> There’s a lot of other changes and fixes in this release, including a new
> Java version of the Appstats profiling tool, so read the release notes (
> Python ,
> Java)
> for a complete list of changes and download the new versions of the
> SDK
> .
>
> Release Notes:
>
> Version 1.3.2 - March 25, 2010
>
>    - New API to read the contents of uploaded Blobs (fetch_data)
>    -http://code.google.com/p/googleappengine/issues/detail?id=2536
>
>    - URLFetch now supports accessing ports 80-90, 440-450, and 1024-65535
>    - Mail API now allows common document formats as attachments
>       -http://code.google.com/p/googleappengine/issues/detail?id=494
>    - The Task Queue API now supports adding multiple tasks in a single call
>    to Queue.add()
>    - Fixed charset handling for inbound emails
>       -http://code.google.com/p/googleappengine/issues/detail?id=2326
>    - Fixed issue with compositing background colors in dev_appserver
>    - New feature in the datastore to specify whether to use strong or
>    eventually consistent reads (the default is strong)
>    - New datastore feature allows setting deadlines for operations
>    - Increased the maximum Task Queue refill rate from 20/s to 50/s
>    - Support for IP blacklisting to prevent denial of service (DoS) attacks
>    - Fix an issue with Mac Launcher in Mac OSX 10.5.5
>       -http://code.google.com/p/googleappengine/issues/detail?id=778
>    - Fix issue with slow updates when there are many skipped files
>       -http://code.google.com/p/googleappengine/issues/detail?id=2492
>    - Fix issue with cursor not updating when using a
> GqlQuery?
>       -http://code.google.com/p/googleappengine/issues/detail?id=2757
>
> You might also want to note that we've updated the public roadmap:
>
> http://code.google.com/appengine/docs/roadmap.html
>
> Features on Deck
>
>    - SSL for third-party domains
>    - Background servers capable of running for longer than 30s
>    - Ability to reserve instances to reduce application loading overhead
>    - Ability to select different availability vs. latency options for
>    Datastore
>    - Support for mapping operations across datasets
>    - Datastore dump and restore facility
>    - Raise request/response size limits for some APIs
>    - Improved monitoring and alerting of application serving
>    - Support for Browser Push (Comet) communication
>    - Built-in

[appengine-java] Re: blobstore & lucene search index

[jacek.ambroziak]

My search engine is a full-text proximity based engine
which you can check out in action in an Android add called NewsDroid.
The indexer part indexes Reuters News semi-continuously
and a user searching on Android will be shown query hits
in matching news where as many keywords as possible
were found together.

This is, btw, an XML search engine -- in addition to keywords
you need to specify in which elements to search for
text passages. I am currently working on another
application of the engine, beyond news.

The original version of the search engine used a number
of files to store indexes. As I "ported" the system to GAE
I had to figure out ways to store and use indexes
stored as blobs in the Datastore.
Basically, any file can be split into chunks of up to 1,000,000 bytes.

Things work OK, but not great on GAE. Loading indexes
into memory is taking time when servlets are restarted.


On Dec 25, 7:51 am, Toby  wrote:
> Hallo Jacek,
>
> Thank you for your mail. Can you give a bit more details about that
> search engine that you implemented?
> I would like to use lucene because I like to use their analyzers and
> things like levenshtein distance result ordering etc.
> Currently I use the compass-lucen framework that comes with a special
> adapter to store the index in the 
> datastore:http://www.kimchy.org/searchable-google-appengine-with-compass/
> I wonder if now the blob-store could be a better way to achieve the
> same goal.
>
> Cheers,
> Tobi
>
> On Dec 23, 3:18 pm, "jacek.ambroziak" 
> wrote:
>
>
>
> > I have been using a search engine of my own design.
> > I had first to adapt it to GAE by generalizing the index
> > storage mechanism so that it can be *implemented*
> > using files (originally the only way) or with Datastore.
> > It was a good couple of days of work but it is not hard.
>
> > On Dec 22, 6:33 pm, Toby  wrote:
>
> > > Hello,
>
> > > I think a sophisticated search api is what a lot of us are missing. I
> > > use currently the compass-lucene project but I am not completely
> > > satisfied with the performance. I think there is also the GAELucene
> > > project that uses a similar aproach.
> > > I guess the main issue was so far that you could not write the index
> > > files to a "regular" file system. I wonder if the new blob-store API
> > > could solve that issue and if anyone has done some experimentation
> > > with it.
>
> > > Cheers,
> > > Tobi

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: blobstore & lucene search index

[jacek.ambroziak]

My search engine is a full-text proximity based engine
which you can check out in action in an Android add called NewsDroid.
The indexer part indexes Reuters News semi-continuously
and a user searching on Android will be shown query hits
in matching news where as many keywords as possible
were found together.

This is, btw, an XML search engine -- in addition to keywords
you need to specify in which elements to search for
text passages. I am currently working on another
application of the engine, beyond news.

The original version of the search engine used a number
of files to store indexes. As I "ported" the system to GAE
I had to figure out ways to store and use indexes
stored as blobs in the Datastore.
Basically, any file can be split into chunks of up to 1,000,000 bytes.

Things work OK, but not great on GAE. Loading indexes
into memory is taking time when servlets are restarted.


On Dec 25, 7:51 am, Toby  wrote:
> Hallo Jacek,
>
> Thank you for your mail. Can you give a bit more details about that
> search engine that you implemented?
> I would like to use lucene because I like to use their analyzers and
> things like levenshtein distance result ordering etc.
> Currently I use the compass-lucen framework that comes with a special
> adapter to store the index in the 
> datastore:http://www.kimchy.org/searchable-google-appengine-with-compass/
> I wonder if now the blob-store could be a better way to achieve the
> same goal.
>
> Cheers,
> Tobi
>
> On Dec 23, 3:18 pm, "jacek.ambroziak" 
> wrote:
>
>
>
> > I have been using a search engine of my own design.
> > I had first to adapt it to GAE by generalizing the index
> > storage mechanism so that it can be *implemented*
> > using files (originally the only way) or with Datastore.
> > It was a good couple of days of work but it is not hard.
>
> > On Dec 22, 6:33 pm, Toby  wrote:
>
> > > Hello,
>
> > > I think a sophisticated search api is what a lot of us are missing. I
> > > use currently the compass-lucene project but I am not completely
> > > satisfied with the performance. I think there is also the GAELucene
> > > project that uses a similar aproach.
> > > I guess the main issue was so far that you could not write the index
> > > files to a "regular" file system. I wonder if the new blob-store API
> > > could solve that issue and if anyone has done some experimentation
> > > with it.
>
> > > Cheers,
> > > Tobi

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: Cache stops returning results under heavy load

[jacek.ambroziak]

I believe I am experiencing a similar issue.
I am using two applications with identical (search engine) code
servicing searches against indexes stored in Datastore.
To speed up access to index blobs I am using memcache.

Experimentally, I enabled billing for one of these apps
to check if I would get better performance from that app.

Unfortunately -- I am getting *worse* performance from the billing
enabled app
as it does not seem to be able to successfully memcache my blobs.


On Dec 15, 3:17 pm, "Ikai L (Google)"  wrote:
> Martin,
>
> Let us know if there's a pattern. I'd like to be able to track this down.
>
> On Mon, Dec 14, 2009 at 2:58 AM, Martin Caslavsky wrote:
>
>
>
> > There are only several objects in the cache (up to 10), each about
> > 10kb. But there is also a bigger one (about 260kb) which I suspect of
> > source of problems.
>
> > I am afraid I am not able to reproduce the problem.
> > Dec 7: 2600 pageviews, cache occasionally does not work (at least one
> > time started to work again after redeploy)
> > Dec 8: 1000 pageviews, cache worked ok
> > Dec 9:  700 pageviews, cache occasionally does not work
>
> > Martin
>
> > On Dec 10, 8:15 pm, "Ikai L (Google)"  wrote:
> > > There are limits to the size of your cache, but Memcache LRU expiration
> > > should be pushing out older items, not newer items that are going into
> > the
> > > cache. Also, a deploy should not be causing this to occur.
>
> > > What steps can be taken to reproduce this? What are the size of the
> > blobs,
> > > and what load can I use to cause this to happen?
>
> > > On Wed, Dec 9, 2009 at 4:04 PM, Martin Caslavsky  > >wrote:
>
> > > > Yes, cache maps Strings (URLs) to byte[] blobs (RSS feed content).
>
> > > > Martin
>
> > > > On Dec 10, 12:13 am, "Ikai L (Google)"  wrote:
> > > > > What kind of objects are you storing the cache? Are you storing large
> > > > binary
> > > > > objects?
>
> > > > > On Tue, Dec 8, 2009 at 12:23 AM, Martin Caslavsky <
> > mcaslav...@gmail.com
> > > > >wrote:
>
> > > > > > Sorry, it is dmczum.
> > > > > > FYI redeploy of the app helps for a few hours.
>
> > > > > > Martin
>
> > > > > > On Dec 8, 2:20 am, "Ikai L (Google)"  wrote:
> > > > > > > Martin,
>
> > > > > > > Can you provide your app ID? I'm not able to look up czumdm.
>
> > > > > > > On Sun, Dec 6, 2009 at 4:40 PM, Martin Caslavsky <
> > > > mcaslav...@gmail.com
> > > > > > >wrote:
>
> > > > > > > > Hi,
>
> > > > > > > > my java app (czumdm) was over quota, so I enabled billing.
> > First
> > > > > > > > request was processed at 12-06 04:10PM 11.559. App worked
> > correctly
> > > > --
> > > > > > > > most request was processed  from JCache (logged as "execute()
> > > > > > > > skipping") and several request was processed directly (logged
> > as
> > > > > > > > "execute()").
>
> > > > > > > > But as of 12-06 04:15PM 49.099 the method call
> > > > cache.containsKey(key)
> > > > > > > > returns false, thus no request is processed from cache!
>
> > > > > > > > Cache statistics shows some cached objects (correct) but hits
> > count
> > > > is
> > > > > > > > stalled and miss count is increasing (weird).
>
> > > > > > > > Greets
> > > > > > > > Martin
>
> > > > > > > > --
>
> > > > > > > > You received this message because you are subscribed to the
> > Google
> > > > > > Groups
> > > > > > > > "Google App Engine for Java" group.
> > > > > > > > To post to this group, send email to
> > > > > > > > google-appengine-j...@googlegroups.com.
> > > > > > > > To unsubscribe from this group, send email to
> > > > > > > > google-appengine-java+unsubscr...@googlegroups.com
> > 
>
> > > > 
> > 
>
> > > > > > 
> > 
>
> > > > 
> > 
>
> > > > > > > > .
> > > > > > > > For more options, visit this group at
> > > > > > > >http://groups.google.com/group/google-appengine-java?hl=en.
>
> > > > > > > --
> > > > > > > Ikai Lan
> > > > > > > Developer Programs Engineer, Google App Engine
>
> > > > > > --
>
> > > > > > You received this message because you are subscribed to the Google
> > > > Groups
> > > > > > "Google App Engine for Java" group.
> > > > > > To post to this group, send email to
> > > > > > google-appengine-j...@googlegroups.com.
> > > > > > To unsubscribe from this group, send email to
> > > > > > google-appengine-java+unsubscr...@googlegroups.com
> > 
>
> > > > 
> > 
>
> > > > > > .
> > > > > > For more options, visit this group at
> > > > > >http://groups.google.com/group/google-appengine-java?hl=en.
>
> > > > > --
> > > > > Ikai Lan
> > > > > Developer Programs Engineer, Google App Engine
>
> > > > --
>
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine for Java" group.
> > > > To post to this group, send email to
> > > > google-appengine-j...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > google-appengine-java+unsubscr...@googlegroups.com
> > 
>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-app

[appengine-java] Re: blobstore & lucene search index

[jacek.ambroziak]

I have been using a search engine of my own design.
I had first to adapt it to GAE by generalizing the index
storage mechanism so that it can be *implemented*
using files (originally the only way) or with Datastore.
It was a good couple of days of work but it is not hard.


On Dec 22, 6:33 pm, Toby  wrote:
> Hello,
>
> I think a sophisticated search api is what a lot of us are missing. I
> use currently the compass-lucene project but I am not completely
> satisfied with the performance. I think there is also the GAELucene
> project that uses a similar aproach.
> I guess the main issue was so far that you could not write the index
> files to a "regular" file system. I wonder if the new blob-store API
> could solve that issue and if anyone has done some experimentation
> with it.
>
> Cheers,
> Tobi

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] unable to deploy new versions after upgrade to 1.2.8

[jacek.ambroziak]

I am going through all the usual motions and yet the deployment of a
new version never finishes successfully;
instead I am getting

java.lang.RuntimeException: Version not ready.

in the /tmp log

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Java heap size

[jacek.ambroziak]

My application, an XML search engine, likes to manipulate its index
data structures in memory during indexing.
Searches are naturally much faster too when index data structures are
in the heap.
Unfortunately, I started running into OutOfMemory errors while
indexing larger numbers of publications,
but not huge number either: ~ 300. Data (typically byte arrays) I
manipulate are on the order of 20 MB,
again a very small mem requirement for this Century (both my dev
computers are now 4 GB).

Yes, I could bend over backwards and start using Datastore for some
form of poor man's virtual memory
but it wouldn't win all that much more space, would slow down and
complicate the code.

So... how much heap space do we have? Can we affect the size?

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: image resize transform

[jacek.ambroziak]

My input was a 100x100 JPEG image, the transform: resize(85, 85),
the result was also JPEG in agreement with my guess that the
original encoding will be preserved, but, surprisingly,
the result data could be up to 5 times longer byte wise.
Explicitly enforcing PNG output did not solve the issue
... BUT ... explicitly asking for JPEG did solve it!
And now I am happy -- images are resized and also smaller.
Still, an unexpected adventure :-)

On Nov 13, 2:22 pm, "Ikai L (Google)"  wrote:
> The list of open issues for the Images API is here:
>
> http://code.google.com/p/googleappengine/issues/list?can=2&q=image
>
> It looks like your issue may be related to this:
>
> http://code.google.com/p/googleappengine/issues/detail?id=2339
>
> Any details you can provide will be a great help. Thanks!
>
> On Thu, Nov 12, 2009 at 7:25 AM, jacek.ambroziak
> wrote:
>
>
>
> > I am trying to resize JPEG images from 100x100 original to smaller
> > 85x85 size.
> > The transformation apparently works, but the smaller image is LARGER
> > bytes-wise?!
> > An important reason for my making the images smaller is to reduce
> > their data size.
>
> > The same transformation using ImageMagick's 'convert' makes images
> > smaller in data size.
> > Additionally 'convert' takes 'quality' option that helps to further
> > reduce the image size;
> > it would make a great addition to the API!
>
> > Why are my transformed images bigger than the originals?
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to
> > google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine-java?hl=.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=.

[appengine-java] image resize transform

[jacek.ambroziak]

I am trying to resize JPEG images from 100x100 original to smaller
85x85 size.
The transformation apparently works, but the smaller image is LARGER
bytes-wise?!
An important reason for my making the images smaller is to reduce
their data size.

The same transformation using ImageMagick's 'convert' makes images
smaller in data size.
Additionally 'convert' takes 'quality' option that helps to further
reduce the image size;
it would make a great addition to the API!

Why are my transformed images bigger than the originals?

--

You received this message because you are subscribed to the Google Groups 
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=.