Re: writing a ceph cliente for MS windows
Is here any progress on Cephfs Windows Client ? 2013-11-08 22:15 GMT+08:00 Alphe Salas Michels asa...@kepler.cl: Hello malcom and matt thank you for apporting some more information source. OpenAFS is sure interesting httpfs too. I hope it will help us on deciding the best path to follow in our interface with window. Actually I still trying to isolate the needed client code in the shortest way possible. Regards. Alphe Salas El nov 7, 2013 9:11 p.m., Malcolm Haak malc...@sgi.com mailto:malc...@sgi.com escribió: I'm just going to throw these in there. http://www.acc.umu.se/~bosse/ http://www.acc.umu.se/%7Ebosse/ They are GPLv2 some already use sockets and such from inside the kernel. Heck you might even be able to mod the HTTP one to use rados gateway. I don't know as I havent sat down and pulled them apart enough yet. They might help, but they might be useless. Not sure. On 08/11/13 06:47, Alphe Salas Michels wrote: Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl mailto:asa...@kepler.cl *www.kepler.cl http://www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our core set of mandatory features. Then we try to compile with cygwin a cephfs client library . Then we
Re: writing a ceph cliente for MS windows
Why don't you push your impl? Maybe I can have a BP first. On 27 December 2014 at 01:10, Ketor D d.ke...@gmail.com wrote: Is here any progress on Cephfs Windows Client ? 2013-11-08 22:15 GMT+08:00 Alphe Salas Michels asa...@kepler.cl: Hello malcom and matt thank you for apporting some more information source. OpenAFS is sure interesting httpfs too. I hope it will help us on deciding the best path to follow in our interface with window. Actually I still trying to isolate the needed client code in the shortest way possible. Regards. Alphe Salas El nov 7, 2013 9:11 p.m., Malcolm Haak malc...@sgi.com mailto:malc...@sgi.com escribió: I'm just going to throw these in there. http://www.acc.umu.se/~bosse/ http://www.acc.umu.se/%7Ebosse/ They are GPLv2 some already use sockets and such from inside the kernel. Heck you might even be able to mod the HTTP one to use rados gateway. I don't know as I havent sat down and pulled them apart enough yet. They might help, but they might be useless. Not sure. On 08/11/13 06:47, Alphe Salas Michels wrote: Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl mailto:asa...@kepler.cl *www.kepler.cl http://www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our
Re: writing a ceph cliente for MS windows
Sorry, I mean YOU can have a BP first. On 27 December 2014 at 13:14, Dong Yuan yuandong1...@gmail.com wrote: Why don't you push your impl? Maybe I can have a BP first. On 27 December 2014 at 01:10, Ketor D d.ke...@gmail.com wrote: Is here any progress on Cephfs Windows Client ? 2013-11-08 22:15 GMT+08:00 Alphe Salas Michels asa...@kepler.cl: Hello malcom and matt thank you for apporting some more information source. OpenAFS is sure interesting httpfs too. I hope it will help us on deciding the best path to follow in our interface with window. Actually I still trying to isolate the needed client code in the shortest way possible. Regards. Alphe Salas El nov 7, 2013 9:11 p.m., Malcolm Haak malc...@sgi.com mailto:malc...@sgi.com escribió: I'm just going to throw these in there. http://www.acc.umu.se/~bosse/ http://www.acc.umu.se/%7Ebosse/ They are GPLv2 some already use sockets and such from inside the kernel. Heck you might even be able to mod the HTTP one to use rados gateway. I don't know as I havent sat down and pulled them apart enough yet. They might help, but they might be useless. Not sure. On 08/11/13 06:47, Alphe Salas Michels wrote: Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl mailto:asa...@kepler.cl *www.kepler.cl http://www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from
Re: writing a ceph cliente for MS windows
Hello malcom and matt thank you for apporting some more information source. OpenAFS is sure interesting httpfs too. I hope it will help us on deciding the best path to follow in our interface with window. Actually I still trying to isolate the needed client code in the shortest way possible. Regards. Alphe Salas El nov 7, 2013 9:11 p.m., Malcolm Haak malc...@sgi.com mailto:malc...@sgi.com escribió: I'm just going to throw these in there. http://www.acc.umu.se/~bosse/ http://www.acc.umu.se/%7Ebosse/ They are GPLv2 some already use sockets and such from inside the kernel. Heck you might even be able to mod the HTTP one to use rados gateway. I don't know as I havent sat down and pulled them apart enough yet. They might help, but they might be useless. Not sure. On 08/11/13 06:47, Alphe Salas Michels wrote: Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl mailto:asa...@kepler.cl *www.kepler.cl http://www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our core set of mandatory features. Then we try to compile with cygwin a cephfs client library . Then we will try to interface with a modified windows nfs 4.1 client or pnfs or any other that will accept to be compiled with gcc for win32... the fact that
Re: writing a ceph cliente for MS windows
Commercial libraries are a pain ... If we want the more permossive licence offered by callback file system we have to buy it for 20.000 usd. Then we will have to provide a backbox that we have no control upon and that will kill our product anytime they want anf if they decide to stop their commercial activity we will be in the same situation that with dokanfs but without having the source code of the black box. If i have to spend 20 000 dollars i would prefere paying someone to retake dokanfs or to write from scratch a dokanfs fuselike software make it all shiny and pumpy fantastic and ready to plug to ceph client. I would prefere if people have to pay something to get access to ceph4win that this money goes in ceph main branch pockets... Or as a gift you donante to ceph 10 dollars you get 2 free registration codes for ceph4win... or something like that. If ceph4win as to be comercial then I would prefer delegate the task to a company like south river technologies and their great product webdrive. I would mininaly get involved in that project and simply buy the final product to sell it together with my ceph based product (which could be a calxeda ceph box or something like that). I m open anyway to any proposition. But I doubt that callback filesystem offers us a suitable solution in the way I see ceph4win to be spread and used... I m maybe wrong. And anything that will be done around ceph4win will be public documented etc... And licensed the way that if someone want to build a commercial solution on top of it, that would be a possibility. My idea is to giveback somehow to ceph project and at same time forge a better knowledge in ceph technologies. Because like many in libre world I think the business is in the services around the software more than on the software. That the ones writing code should be financed and benefits from the one selling and giving support of the software at all levels. I m probably too idealistic. And too optimistic after all I m the one saying I will do this stuff I have no idea how but well it is interesting and fun so lets do it. Regards, P.S: using commercial backend libraries appart including their own cost will force you to use commercial IDE like MS VisualStudio because their library has some kind of drm that only that IDE compiler can use. So alot of cost and yet there is nothing done. If I had to open a kickstarter project saying we need 60 000 USD to do ceph4win with that monney we will buy the right to use and share a commercial copyrighted library but abandonned punctually to us in public domaine and that we will eventually produce something out of it. I doubt I will get a dollar. We still can suggest the idea to Edlos the commercial company that has the copyright of Callback FS, Or to buy them their product in a blender way (blender was bought with donation before being put opensource and public domaine), Or to open source their library. But in commercial minds opensourcing = death of their technical advantage and death of their marketing strategy. They will have to invent something more to retrieve monney from it. El nov 6, 2013 11:22 p.m., Ketor D d.ke...@gmail.com mailto:d.ke...@gmail.com escribió: Hi Alphe, I think you could try Callback Filesystem dev framework. It is a commerical dev framework and is maintained by Edlos today. I have communicated with Edlos to get a try code for development. To dokan, Callback Filesystem has vary document and maybe more stabilize. Regards. On Thu, Nov 7, 2013 at 10:00 AM, Alphe Salas asa...@kepler.cl mailto:asa...@kepler.cl wrote: Hello ketor thank you for your interest un ceph4win. Since muy first mail I exposed the lacks of dokanfs and that I m far from being a specialist un filesystems. I exposed what i like un dokanfs bit I not a fanátic of it. Muy goal is to have something working quickly. So I am up to any proposición sure the one with the more docs and support will be the best choice. As for right now what I need is understand what are the files involved what are the interfaces functions and what are the needed library dependencies and if they exist ported to windows with cygwin. And all that is retrieved from source code. Regards. El nov 6, 2013 10:34 p.m., Ketor D d.ke...@gmail.com mailto:d.ke...@gmail.com escribió: Hi Alphe, We are taking an interest in your work on Ceph Client for Windows with Dokan.As we know, the performance of Dokan is not very good, and it's abandoned 3 years ago. I have learned and used OpenDedup(SDFS) for a long time. OpenDedup has a Dokan version. And the author of OpenDedup said The Dokan library is quite flakey and testing should be performed before putting into production So what do
Re: writing a ceph cliente for MS windows
Hi Alphe: Yes Callback Filesystem is very expensive and can't open source. It's not a good choice for ceph4win. Another way for ceph4win maybe develop a kernel-mode fs like pnfs. pnfs has a kernel-mode windows client. I think you can read its src code and maybe migrating from ceph kernel client to windows kernel fs is easier than from userspace ceph fuse client.And a kernel-mode fs client has greater performance than userspace fs like ceph-fuse client and ceph kernel client. Regards. On Thu, Nov 7, 2013 at 8:13 PM, Alphe Salas Michels asa...@kepler.cl wrote: Commercial libraries are a pain ... If we want the more permossive licence offered by callback file system we have to buy it for 20.000 usd. Then we will have to provide a backbox that we have no control upon and that will kill our product anytime they want anf if they decide to stop their commercial activity we will be in the same situation that with dokanfs but without having the source code of the black box. If i have to spend 20 000 dollars i would prefere paying someone to retake dokanfs or to write from scratch a dokanfs fuselike software make it all shiny and pumpy fantastic and ready to plug to ceph client. I would prefere if people have to pay something to get access to ceph4win that this money goes in ceph main branch pockets... Or as a gift you donante to ceph 10 dollars you get 2 free registration codes for ceph4win... or something like that. If ceph4win as to be comercial then I would prefer delegate the task to a company like south river technologies and their great product webdrive. I would mininaly get involved in that project and simply buy the final product to sell it together with my ceph based product (which could be a calxeda ceph box or something like that). I m open anyway to any proposition. But I doubt that callback filesystem offers us a suitable solution in the way I see ceph4win to be spread and used... I m maybe wrong. And anything that will be done around ceph4win will be public documented etc... And licensed the way that if someone want to build a commercial solution on top of it, that would be a possibility. My idea is to giveback somehow to ceph project and at same time forge a better knowledge in ceph technologies. Because like many in libre world I think the business is in the services around the software more than on the software. That the ones writing code should be financed and benefits from the one selling and giving support of the software at all levels. I m probably too idealistic. And too optimistic after all I m the one saying I will do this stuff I have no idea how but well it is interesting and fun so lets do it. Regards, P.S: using commercial backend libraries appart including their own cost will force you to use commercial IDE like MS VisualStudio because their library has some kind of drm that only that IDE compiler can use. So alot of cost and yet there is nothing done. If I had to open a kickstarter project saying we need 60 000 USD to do ceph4win with that monney we will buy the right to use and share a commercial copyrighted library but abandonned punctually to us in public domaine and that we will eventually produce something out of it. I doubt I will get a dollar. We still can suggest the idea to Edlos the commercial company that has the copyright of Callback FS, Or to buy them their product in a blender way (blender was bought with donation before being put opensource and public domaine), Or to open source their library. But in commercial minds opensourcing = death of their technical advantage and death of their marketing strategy. They will have to invent something more to retrieve monney from it. El nov 6, 2013 11:22 p.m., Ketor D d.ke...@gmail.com mailto:d.ke...@gmail.com escribió: Hi Alphe, I think you could try Callback Filesystem dev framework. It is a commerical dev framework and is maintained by Edlos today. I have communicated with Edlos to get a try code for development. To dokan, Callback Filesystem has vary document and maybe more stabilize. Regards. On Thu, Nov 7, 2013 at 10:00 AM, Alphe Salas asa...@kepler.cl mailto:asa...@kepler.cl wrote: Hello ketor thank you for your interest un ceph4win. Since muy first mail I exposed the lacks of dokanfs and that I m far from being a specialist un filesystems. I exposed what i like un dokanfs bit I not a fanátic of it. Muy goal is to have something working quickly. So I am up to any proposición sure the one with the more docs and support will be the best choice. As for right now what I need is understand what are the files involved what are the interfaces functions and what are the needed library dependencies and if they exist ported to windows with cygwin. And all that is retrieved from source code.
Re: writing a ceph cliente for MS windows
Hi, The Window NFS v4.1 client is what we work on, so this may be good for code sharing. The license is lgplv2, like Ceph's. Something important to be aware of is that the client uses rdbss, which is a (partial) fsd abstraction that simplified implementation quite a bit, kind of like a mini driver. However, Microsoft's support for rdbss has been in limbo for a bit. For example, to link with the rdbss symbols you can't use the Windows 8 driver kit--you'll need to use the one for Windows 7. (There's a private rdbss2 used internally by Microsoft's SMB implemenation. A the moment, 3rd party drivers can't use that.) We've been in communication with Microsoft about this issue, and know of a few other fsds using it, but it could be a good thing for that lobbying effort to have another user--or it could be a dead end :(. There are a couple of other choices if you're looking to go this route, that I'm aware of (and we may need to take them too, if RDBSS has no way forward), but the required work could be a lot larger. Matt - Ketor D d.ke...@gmail.com wrote: Hi Alphe: Yes Callback Filesystem is very expensive and can't open source. It's not a good choice for ceph4win. Another way for ceph4win maybe develop a kernel-mode fs like pnfs. pnfs has a kernel-mode windows client. I think you can read its src code and maybe migrating from ceph kernel client to windows kernel fs is easier than from userspace ceph fuse client.And a kernel-mode fs client has greater performance than userspace fs like ceph-fuse client and ceph kernel client. Regards. On Thu, Nov 7, 2013 at 8:13 PM, Alphe Salas Michels asa...@kepler.cl wrote: Commercial libraries are a pain ... If we want the more permossive licence offered by callback file system we have to buy it for 20.000 usd. Then we will have to provide a backbox that we have no control upon and that will kill our product anytime they want anf if they decide to stop their commercial activity we will be in the same situation that with dokanfs but without having the source code of the black box. If i have to spend 20 000 dollars i would prefere paying someone to retake dokanfs or to write from scratch a dokanfs fuselike software make it all shiny and pumpy fantastic and ready to plug to ceph client. I would prefere if people have to pay something to get access to ceph4win that this money goes in ceph main branch pockets... Or as a gift you donante to ceph 10 dollars you get 2 free registration codes for ceph4win... or something like that. If ceph4win as to be comercial then I would prefer delegate the task to a company like south river technologies and their great product webdrive. I would mininaly get involved in that project and simply buy the final product to sell it together with my ceph based product (which could be a calxeda ceph box or something like that). I m open anyway to any proposition. But I doubt that callback filesystem offers us a suitable solution in the way I see ceph4win to be spread and used... I m maybe wrong. And anything that will be done around ceph4win will be public documented etc... And licensed the way that if someone want to build a commercial solution on top of it, that would be a possibility. My idea is to giveback somehow to ceph project and at same time forge a better knowledge in ceph technologies. Because like many in libre world I think the business is in the services around the software more than on the software. That the ones writing code should be financed and benefits from the one selling and giving support of the software at all levels. I m probably too idealistic. And too optimistic after all I m the one saying I will do this stuff I have no idea how but well it is interesting and fun so lets do it. Regards, P.S: using commercial backend libraries appart including their own cost will force you to use commercial IDE like MS VisualStudio because their library has some kind of drm that only that IDE compiler can use. So alot of cost and yet there is nothing done. If I had to open a kickstarter project saying we need 60 000 USD to do ceph4win with that monney we will buy the right to use and share a commercial copyrighted library but abandonned punctually to us in public domaine and that we will eventually produce something out of it. I doubt I will get a dollar. We still can suggest the idea to Edlos the commercial company that has the copyright of Callback FS, Or to buy them their product in a blender way (blender was bought with donation before being put opensource and public domaine), Or to open source their library. But in commercial minds opensourcing = death of their technical advantage and death of their marketing strategy. They will have to invent something more to retrieve monney from it. El nov 6, 2013 11:22 p.m., Ketor D
Re: writing a ceph cliente for MS windows
Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our core set of mandatory features. Then we try to compile with cygwin a cephfs client library . Then we will try to interface with a modified windows nfs 4.1 client or pnfs or any other that will accept to be compiled with gcc for win32... the fact that windows 8.1 is and windows 2012 are out of reach at the moment is not a problem to me. Our first concern is to understand what is ceph protocol. Then adapt it to something that can be used on windows prior windows 8.1. Dokan fs if I remember well use too the WDK (windows driver dev-kit ) for it s compilation so possibly we will see the same limitations. We need to multiply our source of information by example regarding ceph-client (kernel or fuse, radosgw is on a different layer so I will not try anything around it at first.) And we need to multiply our source of information by example regarding virtual file system technologies on windoes OS. Alot of work but all of those available source code everyone point at me will make our best solution. And in the end we will choose technologies knowing what we do and what concequencies they have. regards, Regards signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl On 11/07/13 11:29, Ketor D wrote: Hi Alphe: Yes Callback Filesystem is very expensive and can't open source. It's not a good choice for ceph4win. Another way for ceph4win maybe develop a kernel-mode fs like pnfs. pnfs has a kernel-mode windows client. I think you can read its src code and maybe migrating from ceph kernel client to windows kernel fs is easier than from userspace ceph fuse client.And a kernel-mode fs client has greater performance than userspace fs like ceph-fuse client and ceph kernel client. Regards. On 11/07/13 11:50, Matt W. Benjamin wrote: Hi, The Window NFS v4.1 client is what we work on, so this may be good for code sharing. The license is lgplv2, like Ceph's. Something important to be aware of is that the client uses rdbss, which is a (partial) fsd abstraction that simplified implementation quite a bit, kind of like a mini driver. However, Microsoft's support for rdbss has been in limbo for a bit. For example, to link with the rdbss symbols you can't use the Windows 8 driver kit--you'll need to use the one for Windows 7. (There's a private rdbss2 used internally by Microsoft's SMB implemenation. A the moment, 3rd party drivers can't use that.) We've been in communication with Microsoft about this issue, and know of a few other fsds using it, but it could be a good thing for that lobbying effort to have another user--or it could be a dead end :(. There are a couple of other choices if you're looking to go this route, that I'm aware of (and we may need to take them too, if RDBSS has no way forward), but the required work could be a lot larger. Matt - Ketor Dd.ke...@gmail.com wrote: Hi Alphe: Yes Callback Filesystem is very expensive and can't open source. It's not a good choice for ceph4win. Another way for ceph4win maybe develop a kernel-mode fs like pnfs. pnfs has a kernel-mode windows client. I think you can read its src code and maybe migrating from ceph kernel client to windows kernel fs is easier than from userspace ceph fuse client.And a kernel-mode fs client has greater performance than userspace fs like ceph-fuse client and ceph kernel client. Regards. On Thu, Nov 7, 2013 at 8:13 PM, Alphe Salas Michelsasa...@kepler.cl wrote: Commercial libraries are a pain ... If we want the more permossive licence offered by callback file system we have to buy it for 20.000 usd. Then we will have to provide a backbox that we
Re: writing a ceph cliente for MS windows
Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl *www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our core set of mandatory features. Then we try to compile with cygwin a cephfs client library . Then we will try to interface with a modified windows nfs 4.1 client or pnfs or any other that will accept to be compiled with gcc for win32... the fact that windows 8.1 is and windows 2012 are out of reach at the moment is not a problem to me. Our first concern is to understand what is ceph protocol. Then adapt it to something that can be used on windows prior windows 8.1. Dokan fs if I remember well use too the WDK (windows driver dev-kit ) for it s compilation so possibly we will see the same limitations. We need to multiply our source of information by example regarding ceph-client (kernel or fuse, radosgw is on a different layer so I will not try anything around it at first.) And we need to multiply our source of information by example regarding virtual file system technologies on windoes OS. Alot of work but all of those available source code everyone point at me will make our best solution. And in the end we will choose technologies knowing what we do and what concequencies they have. regards, Regards signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl On 11/07/13 11:29, Ketor D wrote: Hi Alphe: Yes Callback Filesystem is very expensive and can't open source. It's not a good choice for ceph4win. Another way for ceph4win maybe develop a kernel-mode fs like pnfs. pnfs has a kernel-mode windows client. I think you can read its src code and maybe migrating from ceph kernel client to windows kernel fs is easier than from userspace ceph fuse client.And a kernel-mode fs client has greater performance than userspace fs like ceph-fuse client and ceph kernel client. Regards. On 11/07/13 11:50, Matt W. Benjamin wrote: Hi, The Window NFS v4.1 client is what we work on, so this may be good for code sharing. The license is lgplv2, like
Re: writing a ceph cliente for MS windows
I'm just going to throw these in there. http://www.acc.umu.se/~bosse/ They are GPLv2 some already use sockets and such from inside the kernel. Heck you might even be able to mod the HTTP one to use rados gateway. I don't know as I havent sat down and pulled them apart enough yet. They might help, but they might be useless. Not sure. On 08/11/13 06:47, Alphe Salas Michels wrote: Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl *www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our core set of mandatory features. Then we try to compile with cygwin a cephfs client library . Then we will try to interface with a modified windows nfs 4.1 client or pnfs or any other that will accept to be compiled with gcc for win32... the fact that windows 8.1 is and windows 2012 are out of reach at the moment is not a problem to me. Our first concern is to understand what is ceph protocol. Then adapt it to something that can be used on windows prior windows 8.1. Dokan fs if I remember well use too the WDK (windows driver dev-kit ) for it s compilation so possibly we will see the same limitations. We need to multiply our source of information by example regarding ceph-client (kernel or fuse, radosgw is on a different layer so I will not try anything around it at first.) And we need to multiply our source of information by example regarding virtual file system technologies on windoes OS. Alot of work but all of those available source code everyone point at me will make our best solution. And in the end we will choose technologies knowing what we do and what concequencies they have. regards, Regards signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl On 11/07/13 11:29, Ketor D wrote: Hi Alphe: Yes Callback Filesystem is very expensive and can't open source. It's not a good choice for ceph4win. Another way for ceph4win maybe develop a kernel-mode fs like pnfs. pnfs has a kernel-mode windows client. I think you can read its src code and maybe migrating from ceph kernel client to windows kernel fs is easier than from
Re: writing a ceph cliente for MS windows
They certainly could be. The OpenAFS kdfs driver is particularly complete. I've built it and worked on it a bit in earlier versions than those shipping today. It's a hybrid driver. Networking is actually in userspace (that's true of the the NFSv4 client too). The two differ greatly in architecture, however, and the AFS driver has complex licensing. You should be able to use the kernel components by Kernel Drivers/Secure Endpoints (YFS?), but the legacy AFS code is not GPL compatible. (Can't speak for the others.) Matt - Malcolm Haak malc...@sgi.com wrote: I'm just going to throw these in there. http://www.acc.umu.se/~bosse/ They are GPLv2 some already use sockets and such from inside the kernel. Heck you might even be able to mod the HTTP one to use rados gateway. I don't know as I havent sat down and pulled them apart enough yet. They might help, but they might be useless. Not sure. On 08/11/13 06:47, Alphe Salas Michels wrote: Hello all I finally finished my first source code extraction that starts from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the script start from a file extract all the private includes definitions #include something.h and recursively extract private includes too. the best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since most of those directories are the implementation of servers (osd, mon, mds) and even if only a tiny bit of them is needed at client level. you need two structures from ./osd/OSD.h and my script by relation will take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the same results. 131 000 lines of code and 386 files most of the same dirs involved. I think I will spend alot of time doing the manual source code isolation and understand way each #include is set in the files I read (what purpose they have do they allow to integrate a crucial data type or not. The other way around will be to read src/libcephfs.cc. It seems shorter but without understanding what part is used for each included header I can t say anything... I will keep reading the source code and take notes. I think in the case of libcephfs I will gain alot of time. signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl *www.kepler.cl http://www.kepler.cl* On 11/07/13 15:02, Alphe Salas Michels wrote: Hello D.Ketor and Matt Benjamin, You give me alot to think about and this is great! I merged your previous post to make a single reply that anyone can report to easyly Windows NFS 4.1 is available here: http://www.citi.umich.edu/projects/nfsv4/windows/readme.html pnfs is another name for NFS4.X. It is presented as alternative to ceph and we get known terminology as MDS and OSD but without the self healing part if I understand well my rapid look on the topic. (when I say rapid look I mean ... 5 minutes spent in that... which is really small amount of time to get an accurate view on something) starting from mount.ceph ... I know that mount.ceph does little but it is a great hint to know what ceph needs and do things. Basically mount.ceph modprobe the ceph driver in the linux kernel then call mount with the line command passed args and the cephfs type as argument. Then the kernel does the work I don t understand yet what is the start calls that are made to the ceph driver but it seemed to me that is was relatively light. (a first impression compared to ceph-fuse.) I think I will do both isolate source code from ceph-client kernel (cephfs module for linux kernel) and the one pointed by Sage starting from client/fuse_ll.cc in ceph master branch. The common files betwin those 2 extractions will be our core set of mandatory features. Then we try to compile with cygwin a cephfs client library . Then we will try to interface with a modified windows nfs 4.1 client or pnfs or any other that will accept to be compiled with gcc for win32... the fact that windows 8.1 is and windows 2012 are out of reach at the moment is not a problem to me. Our first concern is to understand what is ceph protocol. Then adapt it to something that can be used on windows prior windows 8.1. Dokan fs if I remember well use too the WDK (windows driver dev-kit ) for it s compilation so possibly we will see the same limitations. We need to multiply our source of information by example regarding ceph-client (kernel or fuse, radosgw is on a different layer so I will not try anything around it at first.) And we
Re: writing a ceph cliente for MS windows
Hi Sage, When I initially planned past year to do a client for ceph on windows what I inspected was ceph_fuse.cc from there I tracked all the related files using a script that read the includes precompiler instructions and retreive the related files from the ceph source code git master branch. In the end the work seemed to me gigantic. Now that we have an open discussion on that topic and that you giveme the libcephfs.h and fuseII.cc hints I will start by isolate the source code related. I will modify my previous isolate script to do so. I will create a github branch under ceph if that is ok with you, to put the documentation, the isolated files and the dev tools related. Regards, --- For notes my script working from content and related content ceph_fuse.cc produced this: find . -name '*' | xargs wc -l wc: .: Is a directory 0 . wc: ./common: Is a directory 0 ./common 29 ./common/compiler_extensions.h 39 ./common/simple_spin.h 31 ./common/config_obs.h 481 ./common/admin_socket.cc 33 ./common/pipe.h 119 ./common/LogEntry.h 135 ./common/DoutStreambuf.h 111 ./common/Cond.h 42 ./common/code_environment.h 112 ./common/Formatter.h 760 ./common/config.cc 52 ./common/escape.h 644 ./common/DoutStreambuf.cc 202 ./common/LogClient.cc 573 ./common/ConfUtils.cc 199 ./common/escape.c 84 ./common/Timer.h 25 ./common/hex.h 395 ./common/Formatter.cc 52 ./common/safe_io.h 82 ./common/HeartbeatMap.h 74 ./common/code_environment.cc 17 ./common/errno.cc 42 ./common/signal.h 144 ./common/Mutex.h 96 ./common/admin_socket.h 83 ./common/common_init.h 34 ./common/BackTrace.h 29 ./common/version.h 74 ./common/snap_types.h 238 ./common/ceph_context.cc 84 ./common/Finisher.cc 74 ./common/ceph_argparse.h 187 ./common/utf8.c 39 ./common/Clock.cc 50 ./common/Thread.h 148 ./common/HeartbeatMap.cc 51 ./common/utf8.h 160 ./common/Thread.cc 9 ./common/errno.h 62 ./common/BackTrace.cc 39 ./common/hex.cc 120 ./common/safe_io.c 27 ./common/Clock.h 161 ./common/DecayCounter.h 189 ./common/Timer.cc 119 ./common/Throttle.h 126 ./common/strtol.cc 320 ./common/perf_counters.cc 119 ./common/LogClient.h 33 ./common/debug.h 81 ./common/signal.cc 44 ./common/pipe.c 210 ./common/config.h 24 ./common/static_assert.h 186 ./common/entity_name.cc 124 ./common/armor.c 24 ./common/likely.h 43 ./common/simple_spin.cc 73 ./common/ceph_crypto.cc 79 ./common/entity_name.h 41 ./common/version.cc 106 ./common/dout.h 28 ./common/strtol.h 86 ./common/ConfUtils.h 86 ./common/Finisher.h 108 ./common/LogEntry.cc 179 ./common/perf_counters.h 16 ./common/armor.h 93 ./common/snap_types.cc 409 ./common/ceph_argparse.cc 110 ./common/common_init.cc 114 ./common/ceph_context.h 160 ./common/ceph_crypto.h wc: ./msg: Is a directory 0 ./msg 623 ./msg/SimpleMessenger.h 673 ./msg/Message.cc 263 ./msg/Messenger.h 478 ./msg/Message.h 153 ./msg/msg_types.cc 64 ./msg/Dispatcher.h 2824 ./msg/SimpleMessenger.cc 425 ./msg/msg_types.h wc: ./messages: Is a directory 0 ./messages 81 ./messages/MOSDPGCreate.h 67 ./messages/MRoute.h 72 ./messages/MForward.h 83 ./messages/MMonElection.h 44 ./messages/MPGStatsAck.h 67 ./messages/MClientSession.h 65 ./messages/MOSDBoot.h 90 ./messages/MClientReconnect.h 69 ./messages/MOSDFailure.h 78 ./messages/MOSDPing.h 47 ./messages/MLogAck.h 206 ./messages/MClientRequest.h 51 ./messages/MClientCapRelease.h 57 ./messages/MMonObserve.h 50 ./messages/MMonSubscribeAck.h 57 ./messages/MOSDPGMissing.h 66 ./messages/MClientSnap.h 55 ./messages/MCommandReply.h 106 ./messages/MMonProbe.h 64 ./messages/MOSDPGQuery.h 85 ./messages/MOSDPGNotify.h 95 ./messages/MMonSubscribe.h 223 ./messages/MOSDSubOp.h 55 ./messages/MGetPoolStatsReply.h 62 ./messages/MMonObserveNotify.h 64 ./messages/PaxosServiceMessage.h 59 ./messages/MMonJoin.h 101 ./messages/MPoolOp.h 58 ./messages/MLog.h 57 ./messages/MAuth.h 44 ./messages/MStatfsReply.h 54 ./messages/MMonCommandAck.h 35 ./messages/MMonGetMap.h 78 ./messages/MOSDPGLog.h 52 ./messages/MStatfs.h 54 ./messages/MOSDPGTrim.h 172 ./messages/MClientCaps.h 130 ./messages/MMonPaxos.h 143 ./messages/MOSDMap.h 222 ./messages/MOSDOpReply.h 62 ./messages/MOSDPGRemove.h 81 ./messages/MOSDPGScan.h 39 ./messages/MGenericMessage.h 146 ./messages/MOSDSubOpReply.h 54 ./messages/MOSDPGInfo.h 52 ./messages/MOSDPGTemp.h 70 ./messages/MOSDRepScrub.h 388 ./messages/MOSDOp.h 63 ./messages/MClientRequestForward.h 64
Re: writing a ceph cliente for MS windows
Hello I created the github repository for this project https://github.com/alphe/Ceph4Win Regards, signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl *http://www.kepler.cl* On 11/05/13 21:00, Sage Weil wrote: Hi Alphe, On Tue, 5 Nov 2013, Alphe Salas Michels wrote: signature *Hi, Sage ! thank you for you enthousiast reply. I sure want to make the best use of everything or anything previously done to tend to write ceph cliente for windows. Apart using libre tools for building the future ceph cliente I am open to anything. I would recommand eclipse CDT or Code::BLocks they are based on mingwin open and easyly enhanceable.** more free tools can be found here: http://www.freebyte.com/programming/cpp/#cppcompilers I will read libcephfs source code and take some notes about the protocol. I think you don't need to worry about hte protocol at all, since libcephs implements it for you (and will capture any future changes). I was more going from what I know and trying to track down how mount.ceph work with the parameters passed to it. since it point finally to Kernel/fs/ceph and that I don t really understand how that module work and that it probably points to some other dependencies Reading libcephfs source code could be a big gain of time. (I would also ignore mount.ceph as everything it does it specific to how Linux mounts work.) basically on the protocol what is need are: 1) open and maintain a connection (socket open, auth, etc ) 2) retreive a map of directories and disk Quota (disk sizing Y TB free, Z TB total) 3) procedure to send files / directories in a maner that it will allow our client to fit ceph transmission protocols (limit bandwith for stability?, limit connection amount?, limit cpu use?, Cache for preparing data transfer (a FIFO cache)?) 4)Procedure to retreive files / directory from ceph cluster 5) Management copy/move files /Directories, FS stats, Connection Stats. logging. My idea to progress is to take those main bulletpoint in ceph protocol based on general ideas of what ceph file system does and start identifying parts from libcephfs to match those needs. Instead, I would look at include/cephfs/libcephfs.h, the interface that libcephfs provides, and try to map that to what the fuse layer expects. There is both a path-based that I suspsect lends itself well to the Windows interface and (very soon now) a handle based API that is targetted at the Unix-style VFS layers. I'm mostly guessing, though, since I've never seen any low-level fs code in windows before. In this case, the analogous code for Linux should be client/fuse_ll.cc itself (and not much else), although there will probably be a few tricks necessary to map cleanly onto how the windows interfaces work. Does that make sense? Cheers! sage Any suggestion and contributions are welcome. * On 11/05/13 11:23, Sage Weil wrote: Hi Alphe, On Mon, 4 Nov 2013, Alphe Salas Michels wrote: Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. My company is a ceph enthousiast that use on a dayly basis ceph and that need both transfer speed and big expendable and cheap storage. My company is specialised in data recovery and we want to participate to ceph effort by bringing a ceph cliente for windows. Awesome! Our experience shows us that the best gateway is each clientes being its own gateway, instead of having a bottle neck server or a cluster of bottle neck servers as gateway (FTP, samba, SFTP,webdav, s3, etc..). We already did some research in that domain. Dokan FS is an intent to write an opensource fuse like cliente for MS windows. More information on DOKANFS can be triggered here http://dokan-dev.net/en/download/ Positive points of using DOKANFS. - its opensourced and well licenced mit licence, gpl licence and lgpl licence. Negative point of using DOKAN FS. - unreachable author - Poor documentation . Dev comments in japanese. - Work in progress so it is unstable and needs to be updated, debugged and maintained by a MS Windows file system expert developper. I am not very familiar with windows storage APIs, but somebody told me at once point there were several interfaces against which a new file system could be implemented, everything from a full in-kernel driver to something that is explorer-based. Are any of those suitable? Using a potentially abandoned fuse-like layer makes me a bit nervous. That said, I try past year to do a merge from ceph-fuse to dokanfs here are what I learnt. - Ceph-fuse and related source code is around 60 000 lines of code. - Ceph protocol isn t documented so it is like trying to draw a map of america using only a sextan and a compass. Those led me to those conclusions: - I can t do it alone. - It is easier to draw down the ceph protocol way to work from kernel/fs/ceph sources and mount.ceph - Ceph depending libraries may be unexistant or not up to date in their port
Re: writing a ceph cliente for MS windows
Hi Alphe, On Mon, 4 Nov 2013, Alphe Salas Michels wrote: Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. My company is a ceph enthousiast that use on a dayly basis ceph and that need both transfer speed and big expendable and cheap storage. My company is specialised in data recovery and we want to participate to ceph effort by bringing a ceph cliente for windows. Awesome! Our experience shows us that the best gateway is each clientes being its own gateway, instead of having a bottle neck server or a cluster of bottle neck servers as gateway (FTP, samba, SFTP,webdav, s3, etc..). We already did some research in that domain. Dokan FS is an intent to write an opensource fuse like cliente for MS windows. More information on DOKANFS can be triggered here http://dokan-dev.net/en/download/ Positive points of using DOKANFS. - its opensourced and well licenced mit licence, gpl licence and lgpl licence. Negative point of using DOKAN FS. - unreachable author - Poor documentation . Dev comments in japanese. - Work in progress so it is unstable and needs to be updated, debugged and maintained by a MS Windows file system expert developper. I am not very familiar with windows storage APIs, but somebody told me at once point there were several interfaces against which a new file system could be implemented, everything from a full in-kernel driver to something that is explorer-based. Are any of those suitable? Using a potentially abandoned fuse-like layer makes me a bit nervous. That said, I try past year to do a merge from ceph-fuse to dokanfs here are what I learnt. - Ceph-fuse and related source code is around 60 000 lines of code. - Ceph protocol isn t documented so it is like trying to draw a map of america using only a sextan and a compass. Those led me to those conclusions: - I can t do it alone. - It is easier to draw down the ceph protocol way to work from kernel/fs/ceph sources and mount.ceph - Ceph depending libraries may be unexistant or not up to date in their port on MS Windows (cygwin) I think the most sane path should be to make libcephfs sufficiently portable to build on windows (or cygwin). For the bits used by the client-side coe, I don't think there should be much in the way of dependencies, and the main challenge would be untangling the build for the necessary pieces out from the rest of Ceph. Have you seen the wip-port portability work that is currently underway by Noah and Alan? That may solve many of the cygwin problems you are seeing today. - MS file system specialist are hard do find in the open source libre world so I will try in the commercial world. The commercial world has some problems too. They need ceph protocol draft to implemente it to their own product They will have licencing /commercial politics that infringe lgpl, and hide that most of the work is done by people other than them. They will not participate in a financial way to ceph enhancement and growth. I don't think reimplementing the client code is an efficient way forward. Unless the goal is a pure kernel implementation...but a significant ongoing investment in development resources would be needed for that going forward. I suspect that is a challenge for a platform that does not typically rally that sort of community effort. The easiest thing is of course just to use CIFS and Samba (which works today). A fuse-like approach is probably a reasonably middle ground (both in initial effort and maintainability going forward)... sage -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: writing a ceph cliente for MS windows
signature *Hi, Sage ! thank you for you enthousiast reply. I sure want to make the best use of everything or anything previously done to tend to write ceph cliente for windows. Apart using libre tools for building the future ceph cliente I am open to anything. I would recommand eclipse CDT or Code::BLocks they are based on mingwin open and easyly enhanceable.** more free tools can be found here: http://www.freebyte.com/programming/cpp/#cppcompilers I will read libcephfs source code and take some notes about the protocol. I was more going from what I know and trying to track down how mount.ceph work with the parameters passed to it. since it point finally to Kernel/fs/ceph and that I don t really understand how that module work and that it probably points to some other dependencies Reading libcephfs source code could be a big gain of time. basically on the protocol what is need are: 1) open and maintain a connection (socket open, auth, etc ) 2) retreive a map of directories and disk Quota (disk sizing Y TB free, Z TB total) 3) procedure to send files / directories in a maner that it will allow our client to fit ceph transmission protocols (limit bandwith for stability?, limit connection amount?, limit cpu use?, Cache for preparing data transfer (a FIFO cache)?) 4)Procedure to retreive files / directory from ceph cluster 5) Management copy/move files /Directories, FS stats, Connection Stats. logging. My idea to progress is to take those main bulletpoint in ceph protocol based on general ideas of what ceph file system does and start identifying parts from libcephfs to match those needs. Any suggestion and contributions are welcome. * On 11/05/13 11:23, Sage Weil wrote: Hi Alphe, On Mon, 4 Nov 2013, Alphe Salas Michels wrote: Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. My company is a ceph enthousiast that use on a dayly basis ceph and that need both transfer speed and big expendable and cheap storage. My company is specialised in data recovery and we want to participate to ceph effort by bringing a ceph cliente for windows. Awesome! Our experience shows us that the best gateway is each clientes being its own gateway, instead of having a bottle neck server or a cluster of bottle neck servers as gateway (FTP, samba, SFTP,webdav, s3, etc..). We already did some research in that domain. Dokan FS is an intent to write an opensource fuse like cliente for MS windows. More information on DOKANFS can be triggered here http://dokan-dev.net/en/download/ Positive points of using DOKANFS. - its opensourced and well licenced mit licence, gpl licence and lgpl licence. Negative point of using DOKAN FS. - unreachable author - Poor documentation . Dev comments in japanese. - Work in progress so it is unstable and needs to be updated, debugged and maintained by a MS Windows file system expert developper. I am not very familiar with windows storage APIs, but somebody told me at once point there were several interfaces against which a new file system could be implemented, everything from a full in-kernel driver to something that is explorer-based. Are any of those suitable? Using a potentially abandoned fuse-like layer makes me a bit nervous. That said, I try past year to do a merge from ceph-fuse to dokanfs here are what I learnt. - Ceph-fuse and related source code is around 60 000 lines of code. - Ceph protocol isn t documented so it is like trying to draw a map of america using only a sextan and a compass. Those led me to those conclusions: - I can t do it alone. - It is easier to draw down the ceph protocol way to work from kernel/fs/ceph sources and mount.ceph - Ceph depending libraries may be unexistant or not up to date in their port on MS Windows (cygwin) I think the most sane path should be to make libcephfs sufficiently portable to build on windows (or cygwin). For the bits used by the client-side coe, I don't think there should be much in the way of dependencies, and the main challenge would be untangling the build for the necessary pieces out from the rest of Ceph. Have you seen the wip-port portability work that is currently underway by Noah and Alan? That may solve many of the cygwin problems you are seeing today. - MS file system specialist are hard do find in the open source libre world so I will try in the commercial world. The commercial world has some problems too. They need ceph protocol draft to implemente it to their own product They will have licencing /commercial politics that infringe lgpl, and hide that most of the work is done by people other than them. They will not participate in a financial way to ceph enhancement and growth. I don't think reimplementing the client code is an efficient way forward. Unless the goal is a pure kernel implementation...but a significant ongoing investment in development resources would be needed for
Re: writing a ceph cliente for MS windows
Hello sage, I followed your lead and went a bit further in my source code reading and I notice serveral problems: first most of the id use in ceph osd_clients, mds_clients, mon_client use the data type u64 which will be a problem as most of the windows in use are windows XP or Windows server 2003 in 32 bits. As they are used for id tag for things like snapshot pages (if I understand well) that can be a problem for a port no? What I read so far was mount.ceph files kernel/fs/ceph files which contains the mds_client files and the kernel ioctl interface ... include / linux / ceph files which integrates libcephfs.h. and all the needed files there is three clients, there is auth.h a messenger, a msgpool, buffer, rados that depends on msgr that include in every aspect of the messages formation one or more __le64 message. I m far from understanding the whole code and the interactions betwin all the files and which we can skip and which we can keep. how can we translate those data type into 32bit in order for ceph cluster to understand them and ceph-windows-client to transmit them? Regards signature *Alphé Salas* Ingeniero T.I asa...@kepler.cl On 11/05/13 14:40, Alphe Salas Michels wrote: signature *Hi, Sage ! thank you for you enthousiast reply. I sure want to make the best use of everything or anything previously done to tend to write ceph cliente for windows. Apart using libre tools for building the future ceph cliente I am open to anything. I would recommand eclipse CDT or Code::BLocks they are based on mingwin open and easyly enhanceable.** more free tools can be found here: http://www.freebyte.com/programming/cpp/#cppcompilers I will read libcephfs source code and take some notes about the protocol. I was more going from what I know and trying to track down how mount.ceph work with the parameters passed to it. since it point finally to Kernel/fs/ceph and that I don t really understand how that module work and that it probably points to some other dependencies Reading libcephfs source code could be a big gain of time. basically on the protocol what is need are: 1) open and maintain a connection (socket open, auth, etc ) 2) retreive a map of directories and disk Quota (disk sizing Y TB free, Z TB total) 3) procedure to send files / directories in a maner that it will allow our client to fit ceph transmission protocols (limit bandwith for stability?, limit connection amount?, limit cpu use?, Cache for preparing data transfer (a FIFO cache)?) 4)Procedure to retreive files / directory from ceph cluster 5) Management copy/move files /Directories, FS stats, Connection Stats. logging. My idea to progress is to take those main bulletpoint in ceph protocol based on general ideas of what ceph file system does and start identifying parts from libcephfs to match those needs. Any suggestion and contributions are welcome. * On 11/05/13 11:23, Sage Weil wrote: Hi Alphe, On Mon, 4 Nov 2013, Alphe Salas Michels wrote: Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. My company is a ceph enthousiast that use on a dayly basis ceph and that need both transfer speed and big expendable and cheap storage. My company is specialised in data recovery and we want to participate to ceph effort by bringing a ceph cliente for windows. Awesome! Our experience shows us that the best gateway is each clientes being its own gateway, instead of having a bottle neck server or a cluster of bottle neck servers as gateway (FTP, samba, SFTP,webdav, s3, etc..). We already did some research in that domain. Dokan FS is an intent to write an opensource fuse like cliente for MS windows. More information on DOKANFS can be triggered here http://dokan-dev.net/en/download/ Positive points of using DOKANFS. - its opensourced and well licenced mit licence, gpl licence and lgpl licence. Negative point of using DOKAN FS. - unreachable author - Poor documentation . Dev comments in japanese. - Work in progress so it is unstable and needs to be updated, debugged and maintained by a MS Windows file system expert developper. I am not very familiar with windows storage APIs, but somebody told me at once point there were several interfaces against which a new file system could be implemented, everything from a full in-kernel driver to something that is explorer-based. Are any of those suitable? Using a potentially abandoned fuse-like layer makes me a bit nervous. That said, I try past year to do a merge from ceph-fuse to dokanfs here are what I learnt. - Ceph-fuse and related source code is around 60 000 lines of code. - Ceph protocol isn t documented so it is like trying to draw a map of america using only a sextan and a compass. Those led me to those conclusions: - I can t do it alone. - It is easier to draw down the ceph protocol way to work from kernel/fs/ceph
Re: writing a ceph cliente for MS windows
On Tue, Nov 5, 2013 at 1:49 PM, Alphe Salas Michels asa...@kepler.cl wrote: Hello sage, I followed your lead and went a bit further in my source code reading and I notice serveral problems: first most of the id use in ceph osd_clients, mds_clients, mon_client use the data type u64 which will be a problem as most of the windows in use are windows XP or Windows server 2003 in 32 bits. As they are used for id tag for things like snapshot pages (if I understand well) that can be a problem for a port no? What I read so far was mount.ceph files kernel/fs/ceph files which contains the mds_client files and the kernel ioctl interface ... include / linux / ceph files which integrates libcephfs.h. and all the needed files there is three clients, there is auth.h a messenger, a msgpool, buffer, rados that depends on msgr that include in every aspect of the messages formation one or more __le64 message. I m far from understanding the whole code and the interactions betwin all the files and which we can skip and which we can keep. how can we translate those data type into 32bit in order for ceph cluster to understand them and ceph-windows-client to transmit them? It is possible to define 64 bit variables on 32 bit architectures. The compiler handles it for you. Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: writing a ceph cliente for MS windows
Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. I've looked at porting the rbd client to windows a little while back. That would require a kernel driver and all the rbd stuff is C++ (windows kernel is strictly C) so it was looking like a lot of work. Porting the rbd kernel implementation from Linux would have been easier. An alternative would be to write a fuse-like kernel driver that talks to a userspace implementation of librbd. That would have been slower though, and have some restrictions like no swap space and no booting (although booting would have been a hard problem to solve even with a kernel-only implementation...) Good luck! James -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: writing a ceph cliente for MS windows
Hi Alphe, On Tue, 5 Nov 2013, Alphe Salas Michels wrote: signature *Hi, Sage ! thank you for you enthousiast reply. I sure want to make the best use of everything or anything previously done to tend to write ceph cliente for windows. Apart using libre tools for building the future ceph cliente I am open to anything. I would recommand eclipse CDT or Code::BLocks they are based on mingwin open and easyly enhanceable.** more free tools can be found here: http://www.freebyte.com/programming/cpp/#cppcompilers I will read libcephfs source code and take some notes about the protocol. I think you don't need to worry about hte protocol at all, since libcephs implements it for you (and will capture any future changes). I was more going from what I know and trying to track down how mount.ceph work with the parameters passed to it. since it point finally to Kernel/fs/ceph and that I don t really understand how that module work and that it probably points to some other dependencies Reading libcephfs source code could be a big gain of time. (I would also ignore mount.ceph as everything it does it specific to how Linux mounts work.) basically on the protocol what is need are: 1) open and maintain a connection (socket open, auth, etc ) 2) retreive a map of directories and disk Quota (disk sizing Y TB free, Z TB total) 3) procedure to send files / directories in a maner that it will allow our client to fit ceph transmission protocols (limit bandwith for stability?, limit connection amount?, limit cpu use?, Cache for preparing data transfer (a FIFO cache)?) 4)Procedure to retreive files / directory from ceph cluster 5) Management copy/move files /Directories, FS stats, Connection Stats. logging. My idea to progress is to take those main bulletpoint in ceph protocol based on general ideas of what ceph file system does and start identifying parts from libcephfs to match those needs. Instead, I would look at include/cephfs/libcephfs.h, the interface that libcephfs provides, and try to map that to what the fuse layer expects. There is both a path-based that I suspsect lends itself well to the Windows interface and (very soon now) a handle based API that is targetted at the Unix-style VFS layers. I'm mostly guessing, though, since I've never seen any low-level fs code in windows before. In this case, the analogous code for Linux should be client/fuse_ll.cc itself (and not much else), although there will probably be a few tricks necessary to map cleanly onto how the windows interfaces work. Does that make sense? Cheers! sage Any suggestion and contributions are welcome. * On 11/05/13 11:23, Sage Weil wrote: Hi Alphe, On Mon, 4 Nov 2013, Alphe Salas Michels wrote: Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. My company is a ceph enthousiast that use on a dayly basis ceph and that need both transfer speed and big expendable and cheap storage. My company is specialised in data recovery and we want to participate to ceph effort by bringing a ceph cliente for windows. Awesome! Our experience shows us that the best gateway is each clientes being its own gateway, instead of having a bottle neck server or a cluster of bottle neck servers as gateway (FTP, samba, SFTP,webdav, s3, etc..). We already did some research in that domain. Dokan FS is an intent to write an opensource fuse like cliente for MS windows. More information on DOKANFS can be triggered here http://dokan-dev.net/en/download/ Positive points of using DOKANFS. - its opensourced and well licenced mit licence, gpl licence and lgpl licence. Negative point of using DOKAN FS. - unreachable author - Poor documentation . Dev comments in japanese. - Work in progress so it is unstable and needs to be updated, debugged and maintained by a MS Windows file system expert developper. I am not very familiar with windows storage APIs, but somebody told me at once point there were several interfaces against which a new file system could be implemented, everything from a full in-kernel driver to something that is explorer-based. Are any of those suitable? Using a potentially abandoned fuse-like layer makes me a bit nervous. That said, I try past year to do a merge from ceph-fuse to dokanfs here are what I learnt. - Ceph-fuse and related source code is around 60 000 lines of code. - Ceph protocol isn t documented so it is like trying to draw a map of america using only a sextan and a compass. Those led me to those conclusions: - I can t do it alone. - It is easier to draw down the ceph protocol way to work from kernel/fs/ceph sources and mount.ceph - Ceph depending libraries may be unexistant or not up to date in their
writing a ceph cliente for MS windows
Good day developers! I would like to propose to the one interested work with me to develop a ceph cliente for MS windows world, Basing us on dokanFS. My company is a ceph enthousiast that use on a dayly basis ceph and that need both transfer speed and big expendable and cheap storage. My company is specialised in data recovery and we want to participate to ceph effort by bringing a ceph cliente for windows. Our experience shows us that the best gateway is each clientes being its own gateway, instead of having a bottle neck server or a cluster of bottle neck servers as gateway (FTP, samba, SFTP,webdav, s3, etc..). We already did some research in that domain. Dokan FS is an intent to write an opensource fuse like cliente for MS windows. More information on DOKANFS can be triggered here http://dokan-dev.net/en/download/ Positive points of using DOKANFS. - its opensourced and well licenced mit licence, gpl licence and lgpl licence. Negative point of using DOKAN FS. - unreachable author - Poor documentation . Dev comments in japanese. - Work in progress so it is unstable and needs to be updated, debugged and maintained by a MS Windows file system expert developper. I try past year to do a merge from ceph-fuse to dokanfs here are what I learnt. - Ceph-fuse and related source code is around 60 000 lines of code. - Ceph protocol isn t documented so it is like trying to draw a map of america using only a sextan and a compass. Those led me to those conclusions: - I can t do it alone. - It is easier to draw down the ceph protocol way to work from kernel/fs/ceph sources and mount.ceph - Ceph depending libraries may be unexistant or not up to date in their port on MS Windows (cygwin) - MS file system specialist are hard do find in the open source libre world so I will try in the commercial world. The commercial world has some problems too. They need ceph protocol draft to implemente it to their own product They will have licencing /commercial politics that infringe lgpl, and hide that most of the work is done by people other than them. They will not participate in a financial way to ceph enhancement and growth. As for the modalities of the work if we come to work togather on that topic, it up to you. We can / should use the most possible libre tools for that work (eclipse, mingwin/cygwin etc...) Sharpening the dev tools in libre can already be a big problem. But my way to see this is the more libre are our tools on MS Windows the most people can easyly join us. Best Regards, Alphe Salas asa...@kepler.cl I.T ingeneer. *http://www.kepler.cl* -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
