Creating tomcat VFS
Hi, I'm working for quite some time on the light-weight VFS layer (with the java.io.* as the only provider at the moment) to be used as the Tomcat's physical file system access. The ultimate goal is to be able to run the Tomcat on top of things like Hadoop or similar distributed file systems (eg. GFS) by just using the provider module. Now, the amount of changes is pretty huge but it involves mostly changing the java.io.* with the o.a.t.vfs.* counterparts. Since this "Pseudo VFS" has exactly the same API (with the Java7 additions), no functional code change is needed. Sure there are some additional config directives used to define which parts are using which vfs provider as default. (or by just using the root="file://web/apps" or hdfs://:/web/apps notion for non-clustered environments) I'd like to create a sandbox project for that (was thinking of /sandbox/tomcat-vfs) Comments? Regards -- ^TM - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
On 15/10/2010 08:02, Mladen Turk wrote: > Hi, > > I'm working for quite some time on the light-weight > VFS layer (with the java.io.* as the only provider at > the moment) to be used as the Tomcat's physical file > system access. The ultimate goal is to be able to run > the Tomcat on top of things like Hadoop or similar distributed > file systems (eg. GFS) by just using the provider module. Sounds interesting. What is the use case? Resilience? How far does just a DirContext implementation get you? > Now, the amount of changes is pretty huge but it involves > mostly changing the java.io.* with the o.a.t.vfs.* > counterparts. "huge" makers me nervous but it if is mostly search and replace that is less of a concern. Any feeling at the moment to the amount of overhead this adds? > Since this "Pseudo VFS" has exactly the same API > (with the Java7 additions), no functional code change is needed. > Sure there are some additional config directives used > to define which parts are using which vfs provider as default. > (or by just using the > root="file://web/apps" or hdfs://:/web/apps > notion for non-clustered environments) > > I'd like to create a sandbox project for that > (was thinking of /sandbox/tomcat-vfs) +1. If there are any re-factorings that would make this easier then we should look at the feasibility of putting them in trunk. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
On 10/15/2010 09:58 AM, Mark Thomas wrote: On 15/10/2010 08:02, Mladen Turk wrote: Hi, I'm working for quite some time on the light-weight VFS layer (with the java.io.* as the only provider at the moment) to be used as the Tomcat's physical file system access. The ultimate goal is to be able to run the Tomcat on top of things like Hadoop or similar distributed file systems (eg. GFS) by just using the provider module. Sounds interesting. What is the use case? Resilience? Current users that use NFS for sharing the common data among cluster of Tomcat's. Real cloud deployments. Some more which I cannot talk about at this moment :) How far does just a DirContext implementation get you? Depending on the provider and the API he offers. Basically anything that touches the file system directly has it's wrapper in o.a.t.vfs Yes, for some providers it can be very complex, but at the end it's plugable module, so outside of the core Tomcat, and as such does not influence the Tomcat code stability. Now, the amount of changes is pretty huge but it involves mostly changing the java.io.* with the o.a.t.vfs.* counterparts. "huge" makers me nervous but it if is mostly search and replace that is less of a concern. Any feeling at the moment to the amount of overhead this adds? For default provider which wraps the java.io at all, almost none. Majority is just ruled out by JIT. For others if you use a custom zip layer the load time can fall dramatically. Sure if running on top of Hadoop things will be noticeably slowly then on physical file system, but we are comparing apples and pears. As final, it'll be in the sandbox, and I really have no plan to touch the core Tomcat until proven stable and performant, and then eventually accepted. Regards -- ^TM - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
2010/10/15 Mladen Turk : > Hi, > > > I'm working for quite some time on the light-weight > VFS layer (with the java.io.* as the only provider at > the moment) to be used as the Tomcat's physical file > system access. The ultimate goal is to be able to run > the Tomcat on top of things like Hadoop or similar distributed > file systems (eg. GFS) by just using the provider module. > > Now, the amount of changes is pretty huge but it involves > mostly changing the java.io.* with the o.a.t.vfs.* > counterparts. > Since this "Pseudo VFS" has exactly the same API > (with the Java7 additions), no functional code change is needed. > Sure there are some additional config directives used > to define which parts are using which vfs provider as default. > (or by just using the > root="file://web/apps" or hdfs://:/web/apps > notion for non-clustered environments) > > I'd like to create a sandbox project for that > (was thinking of /sandbox/tomcat-vfs) > > Comments? Seems to me a good idea, I'm also investigate stuff around VFS and Hadoop. Since the VFS word could be an issue, term should be updated may be (Wrapper FS or whatever), but I like the idea. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
I remembered Mladen and VFS discussion, elsewhere. Google confirmed it to me : http://www.mail-archive.com/commons-...@jakarta.apache.org/msg82888.html Your code is still http://apvfs.sourceforge.net/ ? - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
Are you going to replace DirContext ? If yes - great, but please first send a quick summary comparing your API with the other VFS around in apache. I think commons has few targets, including a hdfs, I remember there are more. If no - not sure what's the point of replacing java.io impl, I assume you just need to add a DirContext for hadoop. I remember many years back I was arguing in favor of DirContext - pro arguments were "it's a standard", bundled in JDK, probably more target implementations will be available, etc. I think Remy was arguing that it's too complex and not a perfect fit. I guess he was right :-) Costin On Fri, Oct 15, 2010 at 12:02 AM, Mladen Turk wrote: > Hi, > > > I'm working for quite some time on the light-weight > VFS layer (with the java.io.* as the only provider at > the moment) to be used as the Tomcat's physical file > system access. The ultimate goal is to be able to run > the Tomcat on top of things like Hadoop or similar distributed > file systems (eg. GFS) by just using the provider module. > > Now, the amount of changes is pretty huge but it involves > mostly changing the java.io.* with the o.a.t.vfs.* > counterparts. > Since this "Pseudo VFS" has exactly the same API > (with the Java7 additions), no functional code change is needed. > Sure there are some additional config directives used > to define which parts are using which vfs provider as default. > (or by just using the > root="file://web/apps" or hdfs://:/web/apps > notion for non-clustered environments) > > I'd like to create a sandbox project for that > (was thinking of /sandbox/tomcat-vfs) > > Comments? > > > Regards > -- > ^TM > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: Creating tomcat VFS
On Fri, 2010-10-15 at 07:57 -0700, Costin Manolache wrote: > I remember many years back I was arguing in favor of DirContext - pro > arguments were > "it's a standard", bundled in JDK, probably more target implementations will > be available, etc. > I think Remy was arguing that it's too complex and not a perfect fit. I > guess he was right :-) I ended up doing it after complaining for a while. I think it was a reasonable option at the time, and it was possible to make it fast enough. Rémy - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
On 10/15/2010 04:57 PM, Costin Manolache wrote: Are you going to replace DirContext ? If yes - great, but please first send a quick summary comparing your API with the other VFS around in apache. I think commons has few targets, including a hdfs, I remember there are more. Well it's not a VFS, and it doesn't have an API. The API is java.io + java.util.zio with o.a.t.vfs namespace instead java.io (Frankly didn't came up to the DirContext in experiment, but it shouldn't be any different then others) So the o.a.t.vfs.File is java.io.File in case the provider in "transparent". In case it's "raw" (Those names are mine) the File is o.a.t.vfs.providers.raw.File which only makes sure that it doesn't cross the "root" mount point. The point here is that provider *can* use commons vfs, and that's the idea, but then again the Hadoop's hdfs has a descent API on it's own. Regards -- ^TM - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Creating tomcat VFS
On Fri, Oct 15, 2010 at 8:24 AM, Mladen Turk wrote: > On 10/15/2010 04:57 PM, Costin Manolache wrote: > >> Are you going to replace DirContext ? >> >> If yes - great, but please first send a quick summary comparing your API >> with the other VFS >> around in apache. I think commons has few targets, including a hdfs, I >> remember there are more. >> >> > Well it's not a VFS, and it doesn't have an API. > The API is java.io + java.util.zio with o.a.t.vfs namespace > instead java.io > (Frankly didn't came up to the DirContext in experiment, > but it shouldn't be any different then others) > > I'm not very comfortable with another layer below DirContext... Big +1 if you get rid of DirContext and replace it with your File. I did an experiment in lite - it is not very hard actually to replace DirContext with raw File. The main problem was that java.io - like DirContext - is an old and not very fit API, lot of cruft, blocking, etc. In any case - having a summary of the other VFS APIs would help a lot. Is there any existing API that support non-blocking access to files / streams ? Costin > So the o.a.t.vfs.File is java.io.File in case the provider > in "transparent". In case it's "raw" (Those names are mine) > the File is o.a.t.vfs.providers.raw.File which only makes > sure that it doesn't cross the "root" mount point. > > The point here is that provider *can* use commons vfs, > and that's the idea, but then again the Hadoop's hdfs has a > descent API on it's own. > > > > Regards > -- > ^TM > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: Creating tomcat VFS
On Fri, Oct 15, 2010 at 8:13 AM, Remy Maucherat wrote: > On Fri, 2010-10-15 at 07:57 -0700, Costin Manolache wrote: > > I remember many years back I was arguing in favor of DirContext - pro > > arguments were > > "it's a standard", bundled in JDK, probably more target implementations > will > > be available, etc. > > I think Remy was arguing that it's too complex and not a perfect fit. I > > guess he was right :-) > > I ended up doing it after complaining for a while. I think it was a > reasonable option at the time, and it was possible to make it fast > enough. > > But I hope you agree you were right, at least in the long term :-) Costin > Rémy > > > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: Creating tomcat VFS
On Fri, 2010-10-15 at 08:43 -0700, Costin Manolache wrote: > But I hope you agree you were right, at least in the long term :-) Who cares about the long term, nobody uses 10 years old code. Oh, wait ;) Rémy - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org