[google-appengine] Re: Morphing PHP Stream Wrappers for App Engine

[WooDzu]

Hi Gary.

I've run the */st* tests on my account and except for the one above they 
all are doing well.


Reading path ggsm://log/mylog.txtWarning: fopen(ggslog://mylog.txt): 
failed to open stream: "garyamort\github_io\stream\syslog::stream_open" call 
failed in 
/base/data/home/apps/s~woodzu123/2.372788013017868220/gae/libraries/garyamort/github_io/stream/morpher.php
 on line 162Warning: file_get_contents(ggsm://log/mylog.txt): failed to open 
stream: "garyamort\github_io\stream\morpher::stream_open" call failed in 
/base/data/home/apps/s~woodzu123/2.372788013017868220/gae/demo/streamtest.php 
on line 156

I've got both the 'log' folder as well as the mylog.txt file in it. Both 
created manually and assigned "Owner" permissions for the GAE application 
in *log/mylog.txt*.

One suggestion I'd have is to use something more obvious for the schema 
names, maybe:
gs+sm:// or gs+joomla:// for the default wrapper
gs+log:// for the log wrapper

According to RFC 1738 the following characters are allowed:

- letters "a"--"z"
- digits
- plus ("+")
- period (".")
- hyphen ("-")


How are you getting on with adding this to the cms? Let me know if I can 
test anything for you.

And what are you thoughts on including media/images folders in the wrapper?
I was thinking of adding an option to the installer (the GAE one) to copy 
both folders over to the Google Storage.

The images folder is used to store all dynamic data for Joomla such as 
images for articles and other components, so it should be fine as is but 
all of the default contents in media folder are core assets. Unfortunatelly 
some 3rd party extensions by default use media to store all dynamic data in 
it
such as forum attachments in Kunena, some K2 files, avatars in 
CommunityBuilder and who knows what else.

One idea is to copy both images and media to the Storage, rewrite the urls 
and use a post-deployment script to update/override them. Less hassle.

The other idea is to keep a list of folders which should be kept on the 
Storage. Example:
/cache
/tmp
/media/kunena
/media/k2

Or even two lists eg. the default one that comes with the joomla-gae repo 
and a custom one, user defined, gitignored file.
This way we can have a file for known folders used dynamically by 
extensions and dev/admins would have an option to add custom folders they 
need write access to.


On Tuesday, December 31, 2013 10:47:10 AM UTC, Gary Mort wrote:
>
> While working on getting Joomla to work without core file hacks on App 
> Engine, I ran into some trouble where it attempts to build paths based on 
> the path the index.php file is in.
>
> Specifically, it tries to send media files[images] to JPATH_BASE.'/media', 
> cache files to JPATH_BASE.'/cache', and some log files to JPATH_BASE.'/log',
>
> Rather then running with all that code imported from google cloud storage, 
> I instead setup a stream morpher/wrapper.  By default it can self register 
> itself to handle all streams beginning with ggsm:// and it can be assigned 
> a set of simple pattern matching rules to access a different stream.
>
> IE for any file that begins with ggsm://log/$pathname it will instead 
> access ggslog://$pathname 
> For anything going to ggsm://cache/$pathname it will instead use 
> gggs://bucketname/$pathname
>
> Where ggslog is the protocol string assigned to a simple syslog wrapper so 
> all data sent to a log file will instead be routed to syslog and use the 
> google logging api.
>
> By the same token, gggs is routed to my own custom copy of the google 
> cloud storage stream handler - the only changes there being that instead of 
> a final class, my stream class is a public class that can be extended for 
> other functionality - and I added append mode support for opening 
> files[which I discovered on App Engine requires using the 
> GoogleStorageClient classes in order to retrieve the data to append - you 
> can't open a read stream to a path if you are in the middle of opening a 
> write stream].
>  
> I've dumped the classes into a github repo if anyone else needs them:
> https://github.com/garyamort/Stream-Morpher
>
> One thing I'd like to do is expand the Google Cloud Storage class to use 
> APC caching[as long as it is available] in addition to memcached.  I am 
> thinking that for APC caching I would limit it to only 1 second - the files 
> I am using don't generally change more quickly then once a second, and if 
> they do it's ok when multiple instances are running for seperate instances 
> to have stale data.  Since a single user will generally be routed to the 
> same instance, anything cached in APC for them will be available there 
> immediately - while memcached can provide up to 15 minutes of caching as a 
> fallback, and if those fail the slower load from GS can be done.
>
> Also it is relatively simple to support file locking and such via GS 
> metadata.  Since each request has a unique request id assigned to it, it's 
> easy to store GS metadata with the request id and time o

[google-appengine] Re: Morphing PHP Stream Wrappers for App Engine

[Gary Mort]

On Friday, January 3, 2014 6:35:40 AM UTC-5, WooDzu wrote:
>
> Hi Gary.
>
> I've run the */st* tests on my account and except for the one above they 
> all are doing well.
>
>
> Reading path ggsm://log/mylog.txtWarning: fopen(ggslog://mylog.txt): 
> failed to open stream: "garyamort\github_io\stream\syslog::stream_open" call 
> failed in 
> /base/data/home/apps/s~woodzu123/2.372788013017868220/gae/libraries/garyamort/github_io/stream/morpher.php
>  on line 162Warning: file_get_contents(ggsm://log/mylog.txt): failed to open 
> stream: "garyamort\github_io\stream\morpher::stream_open" call failed in 
> /base/data/home/apps/s~woodzu123/2.372788013017868220/gae/demo/streamtest.php 
> on line 156
>
> I've got both the 'log' folder as well as the mylog.txt file in it. Both 
> created manually and assigned "Owner" permissions for the GAE application 
> in *log/mylog.txt*.
>

The default stream wrapper I'm using for the log directory is a syslog 
wrapper - so it won't try to write to mylog.txt [I just created the file to 
make sure I wasn't writing to it].  My initial implementation ran into an 
issue where syslog would have problems with the newline charector, I'll 
have to review the code in both places and sync them so the code to 
eliminate newlines is in there.

The way it should work is it will appear in the Google Cloud console logs 
for app engine.  The filename becomes irrelevant - but it will show up in 
the syslog messages in case it's relevant. 

 


> One suggestion I'd have is to use something more obvious for the schema 
> names, maybe:
> gs+sm:// or gs+joomla:// for the default wrapper
> gs+log:// for the log wrapper
>
> According to RFC 1738 the following characters are allowed:
>
> - letters "a"--"z"
> - digits
> - plus ("+")
> - period (".")
> - hyphen ("-")
>
>
>
I'm fine with changing the defaults stream type/string - I'll give yours a 
shot.  Note, the default stream types are completely overridable - it's all 
stored in a public static array
 
 So calling:
$mylogstreams = array('gs+syslog', 'gs+log', 'log');

garyamort\github_io\stream\syslog::types=$mylogstreams;

BEFORE calling:

garyamort\github_io\stream\syslog::registerWrapper();


will change the strings the syslog stream wrapper uses to detect on  Heck, 
I also set it up so you can override it by passing the strings to the method, 
ie:

$mylogstreams = array('gs+syslog', 'gs+log', 'log');

garyamort\github_io\stream\syslog::registerWrapper($mylogstreams);


Does the same thing.   registerWrapper is mainly just a convenience function to 
avoid having to type out 

stream_wrapper_register('gs+log', 'garyamort\github_io\stream\syslog');


Over and over for every one of the silly stream wrappers.  So I made sure that 
everything is completely overrideable/public.  The method prefers to use an 
array of strings given to it, if  none are given it uses the public static 
variable for the class that was called.  And it includes a force option 
defaulted to false that when set, will unregister any existing stream 
wrappers[for the same strings/stream types] and replace it with the one being 
registered.



How are you getting on with adding this to the cms? Let me know if I can 
> test anything for you.
>
>
I had to get some paying work done, I should have some time tomorro to go 
through and make some updates and merge your pull requests.  I need to add 
in file mode checking and stream context modifications and then it should 
be in good shape.

There are a number of git stream wrappers out there, and it's possible to 
deploy to google app engine via git - so for people who want to 
upgrade/install extensions on gae, a morphing rule that is only applied for 
streams opened in write mode and then rewrites the pathname to use git 
might provide most of the solution to upgrading in place...  
 

> And what are you thoughts on including media/images folders in the wrapper?
> I was thinking of adding an option to the installer (the GAE one) to copy 
> both folders over to the Google Storage.
>
> The images folder is used to store all dynamic data for Joomla such as 
> images for articles and other components, so it should be fine as is but 
> all of the default contents in media folder are core assets. Unfortunatelly 
> some 3rd party extensions by default use media to store all dynamic data in 
> it
> such as forum attachments in Kunena, some K2 files, avatars in 
> CommunityBuilder and who knows what else.
>
> One idea is to copy both images and media to the Storage, rewrite the urls 
> and use a post-deployment script to update/override them. Less hassle.
>
> The other idea is to keep a list of folders which should be kept on the 
> Storage. Example:
> /cache
> /tmp
> /media/kunena
> /media/k2
>
> Or even two lists eg. the default one that comes with the joomla-gae repo 
> and a custom one, user defined, gitignored file.
> This way we can have a file for known folders used dynamically by 
> extensions and dev/admins would have an option 

[google-appengine] Re: Morphing PHP Stream Wrappers for App Engine

[WooDzu]

Yes, I want to store the rule data in configuration.php and then on 
> initialization instead of the hardcoded rules I'm testing with, the rules 
> can be dynamically generated and registered with the wrapper.
>

> The only problem I have with that is Joomla won't save arrays of data by 
> default in configuration.php - so it means either a plugin to save the data 
> as an array, or limiting it to X number of rules 
>


Indeed configuration.php might be the right place for it. A custom field 
type could be useful, something that serializes a set rules to json.
I think I even have something in place. Let me know and I'll send it over 
to your repo. 
 


The json format could be like:

{
  "URL_ENCODED_LOCAL": "URL_ENCODED_GS",
  

}

The question mark is whether the gs folders should be only checked or also 
created (api needs to be used) upon saving.

 

> I'm mainly trying to balance my tendency to add in lots and lots of 
> options with keeping things simple so I can get to a release/usable 
> point  I mean, why spend weeks adding a bunch of options I don't need 
> if only 2 people are using it?  There doesn't seem to be a lot of interest 
> in deploying Joomla to GAE - it's fun for me and it's a good brain teaser 
> to force my mind into a coding gear.
>
>  
I completely agree here. I think more people will get involved once it's 
fully functional, officially mentioned on the Joomla site and maybe posted 
on some good blog like http://sitepoint.com
Many developers probably don't consider GAE yet as PHP support in GAE is 
still experimental and there still seem to be many bugs around.
My gut feeling is that devs will be using it for really heavy workloads 
since It really has a big potential as PaaS.

I've been testing Joomla on Azure recently and would be extremely convinced 
to stick on it if not the fact MySQL is provided by a 3rd party company 
with scaling limitations and very harsh limits on free tier for testing (20 
MB database !?), bigger options get very expensive. I'm also not really 
convinced by Windows environment though it's PaaS and I shouldn't care what 
it is running on.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.