Re: [google-appengine] How to share data files between versions

[website]

On Saturday, April 11, 2015 at 9:29:27 AM UTC+1, web...@macfh.co.uk wrote:
>
>
>
>  :-(GCS is about half the speed of using the local FS.  See the 
> table below.
>

Actually it's even worse than the table implied, because I'd failed to 
notice that the test path that goes over open water between two islands 
(where therefore part of the path had no tiles),  had thrown an error in 
the GCS version.  When I corrected the exception trapped, the time for 
processing this path was anything between about 35 - 55 seconds!  Perhaps 
it would be possible to reduce this significantly by fiddling about with 
the retry parameters, but there seems little point in experimenting further 
when the result is still going to be slower anyway.

Incidentally, I got around the original upload problem by deleting the OSLP 
folder from the previous version, updating it, then adding the OSLP folder 
to the new version, and updating again.  It was a tiresome bore, and you 
couldn't do it with a serious app that had to remain live at all times, but 
as this is a only intended to be a demo, it was an acceptable workaround.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ffef5df9-71a6-4a54-a7cb-18c3d9612177%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How to share data files between versions

[website]

Thanks to both, but ...

On Tuesday, April 7, 2015 at 10:10:58 PM UTC+1, Vinny P wrote:
>
>
> On Mon, Apr 6, 2015 at 8:28 AM, Barry Hunter  > wrote:
>
 
>>
> Copying over the network from that repository, vs copying from dedicated 
>> Google Storage, is probably not significantly different
>>
>
> +1
>

I gave it a try, and ...
 :-(Google's self-contradictory documentation meant that GCS was a 
pain to set up
 :-(GCS is about half the speed of using the local FS.  See the 
table below.

Averaged over five different test paths:
Test Processing TimeTravel TimeTotal Time
OSLP+FS   1.0060.7701.776
OSLP+GCS1.8281.2602.722
OST50+FS  2.2920.8943.186
SRTM+FS   0.1770.9471.124

For comparison, also included are SRTM and the more recent OS Terrain 50 
datasets, both using local FS lookups, averaged over the same five test 
paths.  SRTM is much the fastest, because it is already in short integer 
binary format, and has overlapping tiles, while both Landform Panorama and 
T50 datasets are in ASCII text format, so every point on the path must be 
converted from text to floating-point, and the T50 is much the slowest 
because it doesn't have overlapping tiles.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d09bb374-e6bc-4d6b-9f74-3430fbf92396%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How to share data files between versions

[Vinny P]

On Mon, Apr 6, 2015 at 8:28 AM, Barry Hunter  wrote:

> Copying over the network from that repository, vs copying from dedicated
> Google Storage, is probably not significantly different
>


+1.

As Barry noted, you would want GCS to store your data files. PHP on GAE
does a similar trick: instead of writing to a local filesystem, PHP/GAE
uses gs:// file names to route reads and writes to a GCS bucket.

If you're really focused on speed, one alternative - and this would get
costly - is to (ab)use dedicated memcache
 and store
your dataset in it. You'd have to deal with cache eviction policies and you
might exceed the ops/sec/GB limit, but it's doable given enough programming
expertise.


-
-Vinny P
Technology & Media Consultant
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALSvALDy3q%3DFJRe99oySar-LMujVUg%2Bwvp0Y6dQOiyztNjb_Gw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How to share data files between versions

[Barry Hunter]

>
>
>> (I've found the 'filesystem' itself to often be slow in AppEngine)
>>
>
> But would it really be quicker to transfer the files over the web for the
> Google App to process?
>

Its not really 'over the web'. In general Google Storage is pretty 'close'
to AppEngine, probably within the same data-center.



> Surely it must be quicker to store the necessary data locally to the app?
>

Well AppEngine is itself a distributed system. The 'code' is stored on some
sort of storage subsystem. When an instance is started for your
application, the code has to be copied from that system to the local
instance. In general its copied as needed (rather than transferring the up
to 1Gb on instance startup)

Copying over the network from that repository, vs copying from dedicated
Google Storage, is probably not significantly different. Been a while since
I benchmarked it tho.


... anyway was only a suggestion how to get round your "problem".

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAJCAUuLhu41qhb1RqnUNtXBSX3-_wwfT%2BF_y_ZO8W4B7P7z6%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How to share data files between versions

[website]

Thanks, but see below ...

On Monday, April 6, 2015 at 1:39:59 PM UTC+1, barryhunter wrote:
>
> Where is the "data" actully stored? Just files in the 'filesystem'? 
>

Yes, it is stored in a directory off root, pretty much exactly as 
originally supplied by OS in 812 files over 56 sub-folders.
 

> Might find it worthwhile to use Google Storage instead for the data files
> https://cloud.google.com/appengine/docs/python/googlestorage/
>

As this is intended to be just a demo, I'd rather avoid such complications, 
if possible.
 

> ... Particular if can make 'ranged' requests. ie can seek directly and 
> read part of a file, rather than the whole file. 
> https://cloud.google.com/storage/docs/reference-headers#range
>

Don't think that would apply here.
 

> (I've found the 'filesystem' itself to often be slow in AppEngine)
>

But would it really be quicker to transfer the files over the web for the 
Google App to process?  Surely it must be quicker to store the necessary 
data locally to the app?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/10e1a51a-c2f9-4bc3-bcaa-bbe607e8f595%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How to share data files between versions

[Barry Hunter]

Where is the "data" actully stored? Just files in the 'filesystem'?

Might find it worthwhile to use Google Storage instead for the data files
https://cloud.google.com/appengine/docs/python/googlestorage/

... Particular if can make 'ranged' requests. ie can seek directly and read
part of a file, rather than the whole file.
https://cloud.google.com/storage/docs/reference-headers#range

(I've found the 'filesystem' itself to often be slow in AppEngine)



On 6 April 2015 at 13:01,  wrote:

> I have an experimental Google App ...
> www.macfhoslp.appspot.com
> ... which uses UK Ordnance Survey Landform Panorama terrain data to
> provide either spot heights or a terrain profile between two points, where
> the points are given in UK eastings and northings.  UK OS LP data totals
> 628MB.
>
> Although the current version works fine, it is a little slow, so I want to
> upload a new version which should be a little faster.  However, this fails
> because the storage limit of 1GB is exceeded.  Is there any way to tell the
> system that the UK OS LP data is invariant and should be shared between all
> versions?
>
> Come to that, if needed, how could I share it with other Google Apps?
>
> TIA
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/de9092cc-4953-496c-8b6b-f4e220f1ea6e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAJCAUu%2BGiSf1KJRLYPXnmj52aZGJr8o_cH-coCU9u5ZFL5%2BVNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.