from:"Mark Wyszomierski"

Re: [google-appengine] Re: Video streaming - possible, or not?

2010-10-20 Thread Mark Wyszomierski

Ah, I grossly over-estimated the size of the videos I'll be working with.
They'll be more in the size range of a few mb. So this seems easier to get
going business-wise.

If anyone has any walkthroughs about video streaming and app engine, that'd
be great. If not, I'll be happy to post a walk-through once I get it done,

Thanks all,
Mark

On Wed, Oct 20, 2010 at 3:41 AM, aah aalvara...@gmail.com wrote:

Hi Mark,

Maybe AppEngine could be combined with something like
http://www.bitsontherun.com/system-apis

-aah

On Oct 19, 4:10 am, Mark mar...@gmail.com wrote:
Hi,

I'm looking to upload a video file into the blobstore, then stream it
to a flash player. I'm not sure if this is possible or not because I
see conflicting messages here about support for it.

I know there's a 30 second http connection limit. I'm not sure if that
counts towards streaming a file.

I would host the videos on youtube, but the video content for this app
is supposed to be 'protected', meaning I don't want to let people view
the video unless authenticated. Using the youtube api would be an easy
win otherwise.

Thanks

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Video streaming - possible, or not?

2010-10-19 Thread Mark Wyszomierski

Hi Ritesh,

Thanks for your response, I really appreciate it. It looks like I have to
see how large my intended video files are, and how much usage I'll get from
users. For example, outgoing bandwidth costs $0.12 per GB. If I have a
single video file that's 1gb, and 100 people watch it that day, then I'll
end up paying $12. Amazon has similar pricing. I don't know much about how
or if browsers will cache any video content, but that would help if users
want to keep replaying the same video. Or replay the same video on different
days (which is likely in my case). The other option is hosting on youtube
for free, but then I believe they take over the rights to the video.

Thanks,
Mark

On Mon, Oct 18, 2010 at 11:54 PM, Ritesh Nadhani rite...@gmail.com wrote:

Hello Mark

AFAIK, the 30sec limit is only for request time. Uploading/storing to
Blobstore does not have this limit. Neither does serving a Blobstore
data has this issue.

http://code.google.com/appengine/docs/python/blobstore/overview.html#Serving_a_Blob

I currently run a music site to which users regularly upload more than
10MB file taking more than 30 seconds. Users are successfuly able to
stream the music using any freely available Flash music player.

On Mon, Oct 18, 2010 at 7:10 PM, Mark mar...@gmail.com wrote:
Hi,

I'm looking to upload a video file into the blobstore, then stream it
to a flash player. I'm not sure if this is possible or not because I
see conflicting messages here about support for it.

I know there's a 30 second http connection limit. I'm not sure if that
counts towards streaming a file.

Thanks

--
Ritesh
http://www.beamto.us

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Pattern for histograms of ratings?

2010-08-09 Thread Mark Wyszomierski

Thanks Jeff. I think the I/O video I saw was from 2008, given by Bret, not
sure if it's the same one you're referencing (maybe I mixed them up) but it
does talk about the fan out problem for microblogging and gives a solution.

The solution in the video is comprehensible but two points stuck out for me
though:

1) You need to write all the recipients into each Message object. If the
author of the message has a large number of friends, writing can take a good
amount of time. This can be handled in the background using the task queue.
2) The Message.recipients list is essentially locked after publication. So
if I write a message today, and you become my follower tomorrow, my message
from yesterday won't show up in your feed (you weren't my friend when I
published the message). That's ok for twitter, but for building a histogram
it won't work because I need knowledge of all your current friends' ratings
for the product.

The first item is a little painful but has to be paid for somewhere, item 2
was a little more problematic. If I treat each product rating instance like
a tweet, and embed the recipients (the author's current friend network) -
then when friends are added or dropped, I need to go through and update
every past product rating instance with the updated friend listing.

I'll see if I can find the 2009 presentation and see if it's different than
the one I saw, thanks for your help,

Mark



On Mon, Aug 9, 2010 at 11:59 AM, Jeff Schwartz jefftschwa...@gmail.comwrote:

 You might want to catch the Google IO 09 vid on YouTube where fan-out is
 discussed. In the vid using listindex entities and key only queries are
 discussed as a way of defining and selecting large groups. If you can wrap
 you hands around the concepts and understand how the mentioned
 implementations works you will have your answer. It is doable but it isn't
 very pretty. The good part is that it provides very quick queries and
 eliminates serializing entities that are only used as indexes.

 Just my $0.02.

 Jeff

 On Mon, Aug 9, 2010 at 2:04 PM, Mark mar...@gmail.com wrote:

 Hi,

 I have a web app where users can add friends, and can rate products.
 The model looks like:

class User {
String username;
}

class Friend {
String username;
String usernameFriend;
}

class ProductRating {
String username;
String productId;
int rating; // 1 - 5
}

 When a user is viewing a product, I want to show them a histogram of
 the ratings all their friends gave the product. Since the histogram is
 not valuable unless all information is known, this becomes difficult
 to do at scale because I need to:

  1) For the given user, load all their friend names (could be
 hundreds or thousands).
  2) For each friend, check if any of them have given a rating for the
 product of interest.
  3) Aggregate all friend ratings into a histogram.

 I'll probably timeout fetching deserializing all those objects on
 steps 1  2. I can precompute histograms for each user for each
 product as everyone submits ratings. This would optimize reads later
 on but would really increase storage requirements and add additional
 cpu use on every rating submission. As friend relationships change, I
 would have to also update all precomputed histograms, which would be a
 pain.



 I'm thinking of doing the following, and wondering how poor an idea it
 is. The basic idea is to keep a flat Text object of a user's friends,
 and a product's ratings to build histograms in application code,
 either on the server or the clients themselves:

  class User {
  String username;
  }

  class UserFriends {
  String username;
  Text friends;
  }

  class ProductRatings {
  String productId;
  Text ratings;
  }

 A user's friends string might look like:

  UserFriends.friends = kim,greg,jen,ed,friendN;

 A product's rating string might look like:

  ProductRatings.ratings = kim:4,tim:5,ed:2,usernameN:ratingN;

 so in order to build the histogram, I need to:

  // get my flat friends string.
  select from UserFriends where username='myusername';

  // get the flat ratings string for the product.
  select from ProductRatings where productId='xyz';

 Once I have both flat strings, I can generate the histogram in
 application code. The idea is that I have a better chance of storing
 all friends and ratings information in the flat Text objects and
 fetching it in a single http connection than if I have if I were to
 fetch all the individual objects.

 I was wondering if anyone else has had to do something similar to
 this, or if there any approaches I'm overlooking. I spent a lot of
 time implementing variations on Bret Slatkin's google i/o 2008 talk
 about building scalable applications on app engine, specifically the
 microblogging example. In the end, the introduction of a changing
 friend network which impacts these histograms made any of my attempts
 too costly to run.

 Any thoughts positive or negative

Re: [appengine-java] Re: Tag clouds on GAE for Java - how to store and how to aggregate

2010-07-21 Thread Mark Wyszomierski

Cool, if you structure your entity class like this though:

class Entity {
 ListString tags;
 String foo;
 }

how are you going to query which Entity instances match a given tag? I think
that all the tags need to be indexed so your query statement can find them -
I'm not 100% sure on this, if you can somehow match on that ListString,
that would be new to me! I was thinking you might have to decide how many
tags an Entity can have, and explicitly declare each like:

class Entity {
 String tag1, tag2, tag3;
 String foo;
 }

only the first tag is required, the other two can be null. Then your query
statement would have to be something like:

select from Entity where tag1 == 'cow' || tag2 == 'cow' || tag3 ==
'cow';

not sure if there's a better way to do it, let us know what you try because
this is probably a pattern that can be reused for others wanting to do the
same thing,

Thanks,
Mark


On Wed, Jul 21, 2010 at 3:15 PM, planetjones jon.r.jo...@gmail.com wrote:

 Mark,

 Thanks very much for this - it's useful.  I'm slowly getting used to
 this non-relational way of thinking.

 The count of the tags doesn't need to be 100% accurate, as it will
 only be used to approximate how many entries exist for a given tag -
 the tag cloud aim is to show tags associated with many entities in a
 larger font and tags associated with fewer entities in a smaller
 font.  If the insert, update or delete of Entity succeeds and gets
 committed, but the persistence action against the TagCounter fails in
 a few instances that would not be a big deal.

 I'll probably allow three tags per Entity, so was planning on:

  class Entity {
  ListString tags;
  String foo;
  }

 for flexibility in of the number of tags.

 So when inserting a new entity I would need to:

 Create Entity
 For each tag in Entity select TagCounter
 If tag counter exists for tag increment count by 1
 If tag counter does not exist for tag create TagCounter with count of
 1

 When deleting an entity I would need to:

 For each tag in Entity select TagCounter
 Decrease count by 1 of TagCounter
 Delete Entity

 This sounds ok to me.  When a user clicks a tag the query is simple
 too.

 I'll give this a try and see how it looks.

 Thanks again.

 On Jul 21, 12:46 pm, Mark mar...@gmail.com wrote:
  I probably have a naive understanding of what you're doing, maybe
  explain a little more, but to me it looks like this:
 
  class TagCounter {
  int count;
  String label; // primary key, unique tag name.
  }
 
  class Entity {
  String tag;
  String foo;
  }
 
  whenever the user creates a new entity, record its tag(s), then
  increment the appropriate TagCounter instance. Keeping counters like
  this introduces a bottleneck and goes against what you want to be
  doing for a highly scalable web service, so you may want to shard them
  (imagine you have thousands of users all trying to increment a single
  tag counter at once).   Since the above two classes won't be in the
  same entity group, you wouldn't be able to perform both operations in
  a single transaction. You can either accept the fact that your tag
  count may be out of sync with the actual number of entities with that
  tag, or you can have a multistep process and clean up dangling tag
  counts with a task (eventual consistency might be the term here). I'm
  referring to Nick Johnson's article on distributed transactions which
  I think you can modify to do what you're looking for:
 
 http://blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine
 
  I'm still learning app engine so take the above with a big grain of
  salt. To me, it seems that if you want to write a highly scalable web
  service, you need to accept that you may lose some data precision, or
  you have to put in additional work to get what you want (compared to
  assuming a single mysql instance etc).
 
  Anyway, after you implement something like the above, your original
  two queries should be easy to do. You probably want to define how many
  tags an Entity can have. If an Entity can have more than one tag, I'd
  explicitly state what that max is. Write out everything you need when
  the user submits an entity, so that your reads will be fast as other
  users browse the site.
 
  Mark
 
  On Jul 19, 9:16 am, planetjones jon.r.jo...@gmail.com wrote:
 
 
 
   Hi,
   I'm looking to store entities using GAE Java which have 1-many tags.
   I
   would like to display a tag cloud, so will need to know how many
   times
   each tag occurs (can't use aggregate functions and group by on GAE to
   do this like I would in SQL). And when I click a tag I would like to
   retrieve all entities with the selected tag.
   Does anyone have any examples of what my Classes should look like to
   do this optimally and efficiently? Sample JPA code would be great
   too.
   Cheers - Jonathan

 --
 You received this message because

Re: [appengine-java] Get objects back in the order they were inserted into the data store?

2010-07-21 Thread Mark Wyszomierski

Hi Ikai,

Sorry to bug you, I didn't quite follow - reworded here:

I *do* need an index on the timestamp field because I want a desc filter.
This ordering may not exactly be the insertion order, but pretty close. If I
wanted to present the objects by real insertion order, I'd need an
additional index.

Did I follow right? Yeah I don't care about insertion order really, just
about the timestamp desc ordering - I figured timestamp desc and insertion
order (in my case) would produce nearly identical results, so was just going
to try springing on the timestamp index. Seems that I should definitely
index on it though,

Thanks,
Mark


On Wed, Jul 21, 2010 at 7:37 PM, Ikai L (Google) ika...@google.com wrote:

 You'll need an index no matter what, as if you did this, you'd need Key
 descending. Note that this may not be exact insert order but it'll be pretty
 close. This seems to save an index, though, as doing it another way requires
 both ascending and descending indexes.

 On Wed, Jul 21, 2010 at 5:52 AM, Mark mar...@gmail.com wrote:

 Hi,

 I'm creating records of a user's actions. When I fetch these records,
 I only ever want them sorted in reverse chronological ordering (the
 order they were inserted into the datastore). Does app engine by
 default return records in this order? I'm trying to avoid having to
 keep an extra index on timestamp to reduce the number of indexes I
 use. Example (in java):

class Action {
@PrimaryKey
@Extension(vendorName=datanucleus, key=gae.encoded-pk,
 value=true)
private String usernameOwner;

@Persistent
long timestamp;
}

public void userActionPerformed(PersistenceManager pm) {
pm.makePersistent(new Foo(myusername,
 System.currentTimeMillis());
}

public void getRecords(String username) {
String stmt = SELECT FROM  + Action.class.getName() + 
 WHERE usernameOwner = ' + username + ';
Query q = pm.newQuery(stmt);
q.setOrdering(timestamp desc); // want to avoid this.

return (ListVRec)q.execute();
}

// desired html: //
You performed a foo action on July 21 at 4:22pm!
You performed a boo action on July 21 at 3:21pm!
You performed a goo action on July 19 at 9:42pm!
...

 so, it would be great if I don't have to index on the timestamp - just
 get the records back in the order I inserted them - possible?

 Thanks

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-j...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




 --
 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog: http://googleappengine.blogspot.com
 Twitter: http://twitter.com/app_engine
 Reddit: http://www.reddit.com/r/appengine

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-j...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] Re: Analytics via log file?

2010-07-16 Thread Mark Wyszomierski

Yes checkout:

http://code.google.com/appengine/docs/java/tools/uploadinganapp.html#Downloading_Logs

the problem I'm facing is that storing all the extra analytics I'm
interested in adds additional complexity and cost to my app. Downloading the
logs and parsing them offline could be a solution, has its own drawbacks
too. Google analytics lets us now log pretty much arbitrary information, so
I've been thinking about using that - only problem is that I'd probably hit
rate limits there too, especially if I want to hook up pulling data out to
my web app for users. For instance, if I log every time a user clicks a
button, and they're interested in knowing that # too, I could create a
handler which queries that from analytics. If I have lots of users though,
that's not going to work,

Thanks,
Mark

On Fri, Jul 16, 2010 at 12:12 PM, Darien Caldwell darien.caldw...@gmail.com
wrote:

This is the first time I've heard anyone suggest downloading App
Engine logs is possible (something I have wished for). How do you do
it?

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

Re: [appengine-java] Re: Performance issue for GAE auto-restart

2010-06-11 Thread Mark Wyszomierski

Hmm ok I see, so I'm just using JDO without any frameworks on top of that,
but as mentioned in the slim3 tests, JDO does add overhead (sometimes a
significant amount). I don't plan to be doing any large queries (the slim3
example fetches 10,000 objects which I don't plan on doing at any point),
but what is important to me is that startup time. One to two seconds is
fine, four seconds starts pushing it, anything above that is pretty bad.

If we start using slim3 though, we lose the flexibility in moving to another
host (not that I know of any that can compare to app engine) since it'll be
interacting directly with big table instead of using jdo or jpa is an
intermediate?

Thanks

On Fri, Jun 11, 2010 at 8:07 AM, Rahul rahul.jun...@gmail.com wrote:

Jake,

Is there any official comments on this from google which says that it
discourage the pinging every 60 seconds or so.

Also if we have some framework which we will always like to have if
doing some production application then what is the way out.

Thanks,
Rahul

On Jun 11, 10:16 am, Jake jbrooko...@cast.org wrote:
Hi Mark,

Yes, you can do better than 4.5 seconds without a framework. Also,
there are layers that can be placed over the low level datastore (e.g
Objectify) that add ease-of-use without noticeable additions to the
startup time.

The 60 second pinging thing is, indeed, done by some users and is
officially discouraged by Google. At one point, I set up a similar
pinging feature and found that it really didn't work. Theoretically
it should, but the pings just caused loading requests every minute or
so, which simply reduced the likelihood that a user would see the
loading request; it didn't solve the problem.

Jake

On Jun 11, 7:32 am, Mark mar...@gmail.com wrote:

Hi, just joining in, trying to sum up:

In the best case (if we only use the low level datastore + no
frameworks on top) we can only hope that a fresh restart of our app
will take 4.5 seconds?

However if at least one user is hitting the site every 60 seconds from
somewhere in the world, then our app should be kept alive and no need
for restarts?

Not that I'm planning on doing it, but what prevents developers from
simply pinging the site every 60 seconds to force the app to stay in
memory? I ask what prevents it because if I play by the rules, I'm
going to get penalized if other developers are 'cheating' and doing
the artificial pinging,

Thanks

On Jun 9, 5:49 am, Jake jbrooko...@cast.org wrote:

Hey Tin,

Several people were using GAE's built in timing mechanism to ping the
server to accomplish the same thing. Google came out and officially
discouraged this as it tends to throw off any attempts they are
currently making to fix the problem. I can't tell people what to do,
but I opted to stop using this hack in the hopes that they will solve
the problem.

Besides, you'll notice that this hack doesn't really work over the
long run. It will essentially restart your server every minute and
won't really prevent the loading request from falling on your users
instead of on your ping.

Jake

On Jun 8, 1:54 pm, Tin tin.c...@gmail.com wrote:

Hi all:

We have found a temporary solution for this issue:
Try with an AJAX timer kicking the server (doing nothing), maybe
one
request per minute (or less) and GAE will keep our site / users in
the
same node.
In our testing it could avoid the GAE web instance reloaded, but if
the request interval is long, it would cause another Datastore
performance issue:http://goo.gl/98zkthatwillbefixed in near
future.

More info please refer to here:http://goo.gl/mzQR

--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.

--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [appengine-java] Re: Hosting gwt project at myapp.com, pulling data from api.myapp.com ?

2010-04-17 Thread Mark Wyszomierski

Hi Stephen,

How does app engine know to let them both access the same data store, is it
just that the last portion of the app identifier is the same? For example,
my project is called orange:

orange.appspot.com

blah.whatever.orange.appspot.com

is it just because orange appears before .appspot, that this linking takes
place? We don't have to set anything up in the xml files to support this?

Thanks

On Sat, Apr 17, 2010 at 4:11 PM, Houston startup coder
stephenh...@gmail.com wrote:

Just create a new version of your app and point api.myproject.com to
that. For example, if your app is at

some-name.appspot.com

you can create a version called

another-version.latest.some-name.appspot.com

and that other version can be a separate application with a completely
different codebase (you can even write it in Python with your original
app being in Java) and both versions of the app will be using the same
datastore.

See this link...

http://code.google.com/appengine/docs/java/config/appconfig.html#About_appengine_web_xml

...and this:

http://googleappengine.blogspot.com/2009/06/10-things-you-probably-didnt-know-about.html

Have fun!

- Stephen Huey
http://www.google.com/profiles/stephenhuey

On Apr 16, 1:43 pm, Mark mar...@gmail.com wrote:
Hi,

I have my gwt project hosted on app engine. This is resolving to:

http://www.myproject.com

how would I go about serving data from:

http://api.myproject.com

from the same datastore? I'd like the gwt project above to be able to
pull its data from api.myproject.com,

Thanks

--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group athttp://
groups.google.com/group/google-appengine-java?hl=en.

--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] Re: Video streaming - possible, or not?

Re: [google-appengine] Video streaming - possible, or not?

Re: [google-appengine] Pattern for histograms of ratings?

Re: [appengine-java] Re: Tag clouds on GAE for Java - how to store and how to aggregate

Re: [appengine-java] Get objects back in the order they were inserted into the data store?

Re: [google-appengine] Re: Analytics via log file?

Re: [appengine-java] Re: Performance issue for GAE auto-restart

Re: [appengine-java] Re: Hosting gwt project at myapp.com, pulling data from api.myapp.com ?

8 matches

Site Navigation

Mail list logo

Footer information