Re: [sword-devel] Remote Module Repository Wiki
On Mon, Nov 8, 2010 at 5:06 AM, Jonathan Morgan jonmmor...@gmail.com wrote: On Mon, Nov 8, 2010 at 7:48 AM, Greg Hellings greg.helli...@gmail.com wrote: On Sun, Nov 7, 2010 at 8:28 AM, Jonathan Morgan jonmmor...@gmail.com wrote: A lot here depends on evaluation of pros and cons. I personally support HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a list of all the books and where to find them for at least some of the following reasons. I will try and capture why succinctly: For ZIPs: 1. Minimising the number of GETs issued (and potentially file size and download speed): I think this is particularly important for loading the initial list of modules, which should preferably be as fast as it possibly can be. It also makes a difference for image modules like Dore's Woodcuts with lots of loose files (which I remember taking a very long time to install with Install Manager). I'm not sure what your metrics are, but Ubuntu's apt for me hits about 70+ files on a half dozen or more servers to update its cache and this takes less than 2 seconds. It takes installmgr almost no time to process a single refresh. Well, I just tried it with Xiphos on Windows and it took mine 4 or 5 seconds to refresh the main CrossWire repository. My enduring memory of InstallManager is that it feels slow. My guess is that the difference is due to going from Oz to US rather than from US to US (I do have a reasonably fast network connection). Apt on Ubuntu is a lot faster, but that is because it is mirrored to an Oz site and doesn't have the latency. On longer hauls over today's speeds, physical limitations of the speed of light dominates for these small files. Even at only gigabit speeds within the USA this can happen. Transocean it can be very prominent. I'm assuming that we have among our target audience: 1. People who are on the other side of the world from the US. 2. People who do not have a fast network connection. The other thing is that I believe the Install Manager case we are testing is in fact the case I advocate: It is grabbing mods.d.tar.gz rather than doing FTP file traversal and grabbing one conf file at a time. I don't know how to test it without rebuilding SWORD, but I think the InstallMgr would grab one conf file at a time and so have a lot of latency from places like here. This (a repository not requiring mods.d.tar.gz) is the simple FTP server setup that is being promoted. This is basically the model that Ubuntu repositories use. Every repository has a single giant file with meta info on all the available files. However, Debian also ships with a package (debmirror) which makes it almost trivial to setup a local mirror, it will create the tar file, etc. Any brief little Perl script could do this, which is what Debmirror uses, or even a Python script which could be run raw as Python on Linux and even compiled into a binary for our Windows friends. Then the barrier to entry is no bigger than saying Point an FTP or HTTP server at your directory, set this little utility to run daily, and you're good to go. --Greg ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Mon, Nov 8, 2010 at 7:48 AM, Greg Hellings greg.helli...@gmail.comwrote: On Sun, Nov 7, 2010 at 8:28 AM, Jonathan Morgan jonmmor...@gmail.com wrote: A lot here depends on evaluation of pros and cons. I personally support HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a list of all the books and where to find them for at least some of the following reasons. I will try and capture why succinctly: For ZIPs: 1. Minimising the number of GETs issued (and potentially file size and download speed): I think this is particularly important for loading the initial list of modules, which should preferably be as fast as it possibly can be. It also makes a difference for image modules like Dore's Woodcuts with lots of loose files (which I remember taking a very long time to install with Install Manager). I'm not sure what your metrics are, but Ubuntu's apt for me hits about 70+ files on a half dozen or more servers to update its cache and this takes less than 2 seconds. It takes installmgr almost no time to process a single refresh. Well, I just tried it with Xiphos on Windows and it took mine 4 or 5 seconds to refresh the main CrossWire repository. My enduring memory of InstallManager is that it feels slow. My guess is that the difference is due to going from Oz to US rather than from US to US (I do have a reasonably fast network connection). Apt on Ubuntu is a lot faster, but that is because it is mirrored to an Oz site and doesn't have the latency. I'm assuming that we have among our target audience: 1. People who are on the other side of the world from the US. 2. People who do not have a fast network connection. The other thing is that I believe the Install Manager case we are testing * is* in fact the case I advocate: It is grabbing mods.d.tar.gz rather than doing FTP file traversal and grabbing one conf file at a time. I don't know how to test it without rebuilding SWORD, but I think the InstallMgr would grab one conf file at a time and so have a lot of latency from places like here. This (a repository not requiring mods.d.tar.gz) is the simple FTP server setup that is being promoted. The main problem is that we are not setup to handle display from multiple sources and also not to refresh from all the repositories at once, they have to be refreshed one at a time. This isn't so ideal to the user's experience, but that's how it is done. I really doubt we need to worry about the performance of hitting 30 or 40 repositories if that is what the user selects to enable - and if we do, then that indicates we need to rethink how we're hitting the repositories, because the limit certainly isn't the number we'd be hitting at that point. If the vision is having individual publishers publishing their own work, then having 30 or 40 repositories would be a big success - and I don't think we can scale to that point (I don't see any prospect of that any time soon, but if that is the vision it's worth thinking about whether we can support it). Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
A lot here depends on evaluation of pros and cons. I personally support HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a list of all the books and where to find them for at least some of the following reasons. I will try and capture why succinctly: For ZIPs: 1. Minimising the number of GETs issued (and potentially file size and download speed): I think this is particularly important for loading the initial list of modules, which should preferably be as fast as it possibly can be. It also makes a difference for image modules like Dore's Woodcuts with lots of loose files (which I remember taking a very long time to install with Install Manager). 2. Provides one unambiguous URL for a module. 3. Uses the same format for distribution stand-alone and in InstallManager. 4. Removes most/all of the directory traversal problems involved with the installed book structure: This has potential advantages of speed and working with a greater number of servers as well as programmability. For HTTP: 1. HTTP is a more common setup than FTP. Thus it is probably more accessible to a wider group of publishers (and quite possibly users as well). For example, it would certainly be possible to host ZIP files on website / file sharing services such as Google Sites or DropBox, and even a mods.d.tar.gz (though keeping it up to date could be a bit troublesome). FTP support is a whole different kettle of fish. I understand you want to make things easier for publishers (and that is not a bad goal), but for me the holy grail is not using the same format, but providing the best user experience. Speed is an important part of that, and to me that almost guarantees two separate format, since the best format for transferring over a network is almost guaranteed to be different from the best one for reading on disk. Philosophically, I think it is usually better to put more work on publishers rather than on consumers, because there will usually be a lot more consumers than publishers. [I do think supporting filesystem installs from the directory structure as you discuss in your viral section makes sense - I just don't think it makes sense over a network. Also having multiple paths to get modules from makes sense]. I suspect having standard Unix scripts to set up the file structure (for big publishers running their own Apache installation) and other helpers might make it not too much harder than maintaining the raw directory structure. One other thing: Troy, do your plans for sword-modules.org (or whatever it was) have any bearing on this? For example, if it were relatively easy for a repository to be set up there by Joe Random or Small Publisher X and sword-modules.org provided FTP, it might lessen the advantage of HTTP over FTP. Jon On Sat, Nov 6, 2010 at 1:35 AM, Troy A. Griffitts scr...@crosswire.orgwrote: There are a number of finely delineated topics here we should separate. 1) Data format for our module installation source. i.e, is it a list of zips, is it an existing install set, etc. Simply to be explicit, we have obvious use cases which come into play: a) Installing our software -- all implementations will be different because we have a variety of packages running on a variety of platforms. b) Installing modules from a module installation source c) Running our software The final item (c) seems out of place but really is important. We have successfully achieved a common data storage format for an installation set of books (i.e. library/mods.d/..., library/modules/...) across a vast number of platforms. Sure this hierarchy of files moves around to different locations on different platforms, but once the location is pinpointed, everything under that location remains the same for any of our platforms. The 'holy grail' for implementation of the (1) above is for both (b) and (c) use cases to be handled by the SAME dataset, and is what we are working towards. In other words, if an installed module library can, itself, also be an installation source for other instances of any SWORD software, then distribution of Bibles becomes viral. This brings us to topic 2. 2) The viral ability END USERS to share modules. If I can, for example, o use the builtin installer of my SWORD software on my Android device to install my important modules o then plug it into my friends computer and install modules FROM my android device WITHOUT ANY PREPARATION o then from his computer right-click on the top module library folder and Share as... 'SWORD Modules', o then go to any computer on the network and install modules from that SWORD Modules share, o then install FileZilla on one of those computers and let anyone over the Internet install Bibles from that location, o then point IIS or Apache to that location (http transport needs work but is on the way-- more talk below) ... distribution of Bibles then becomes viral. It is
Re: [sword-devel] Remote Module Repository Wiki
On Sun, Nov 7, 2010 at 8:28 AM, Jonathan Morgan jonmmor...@gmail.com wrote: A lot here depends on evaluation of pros and cons. I personally support HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a list of all the books and where to find them for at least some of the following reasons. I will try and capture why succinctly: For ZIPs: 1. Minimising the number of GETs issued (and potentially file size and download speed): I think this is particularly important for loading the initial list of modules, which should preferably be as fast as it possibly can be. It also makes a difference for image modules like Dore's Woodcuts with lots of loose files (which I remember taking a very long time to install with Install Manager). I'm not sure what your metrics are, but Ubuntu's apt for me hits about 70+ files on a half dozen or more servers to update its cache and this takes less than 2 seconds. It takes installmgr almost no time to process a single refresh. The main problem is that we are not setup to handle display from multiple sources and also not to refresh from all the repositories at once, they have to be refreshed one at a time. This isn't so ideal to the user's experience, but that's how it is done. I really doubt we need to worry about the performance of hitting 30 or 40 repositories if that is what the user selects to enable - and if we do, then that indicates we need to rethink how we're hitting the repositories, because the limit certainly isn't the number we'd be hitting at that point. --Greg ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Hi Nic, Thanks for the feedback. I'd like some clarification though. I really am trying to make an honest assessment of where we are and need to go in the future and note the negative feedback from frontend developers, but haven't been able to quantify what exactly that negative feedback is. Help me understand your exact objections and how an alternative scheme solves that issue. Questions below: On 11/06/2010 04:48 AM, Nic Carter wrote: I was going to download the mods.d.tar.gz file (over HTTP) and use that for moduleinformation. Then download the ZIP of the module over HTTP. If a repo doesn't have the zip folder (or the mods.d.tar.gz file), then I would either have a fall-back to use the normal InstallManager, or perhaps I'd just say that PocketSword wouldn't support that repo. But after a year of PocketSword doing things the right way, I've decided to take the JSword approach and hope that doesn't cause too many issues... :( :) OK, This is vital for me to understand your experience. I presume the right way you speak of above is simply using the InstallMgr API. Why are you unhappy with the InstallMgr API? Does it not work? Is it too hard to use? You mention speed, below... my personal tests on an iOS device show that accessing the CrossWire repo via FTP is reasonably slower than accessing it via HTTP. Really? I wouldn't have expected this at all. I want the reasonably large performance gains that I can gain from simply switching protocols! :) Yes, indeed. This is a very valid concern. Let me ask, what FTP plugin were you using, ftplib or curl? I am wondering if it might be the difference in plugin. I recently switched BibleCS's InstallMgr GUI to use the ftplib system instead of CURL and surprisingly seem to have less troubles and more responsiveness. There is still one major shortcoming of ftplib-- it cannot download directly to a buffer for a directory listing, we use a temporary file-- but I hope to resolve that in the next day or two. Speed is a very real and practical concern and if we have a problem, we need to resolve it. On 06/11/2010, at 1:35 AM, Troy A. Griffitts wrote: 2) The viral ability END USERS to share modules. If I can, for example, ouse the builtin installer of my SWORD software on my Android device to install my important modules o then plug it into my friends computer and install modules FROM my android device WITHOUT ANY PREPARATION othen from his computer right-click on the top module library folder and Share as... 'SWORD Modules', o then go to any computer on the network and install modules from that SWORD Modules share, othen install FileZilla on one of those computers and let anyone over the Internet install Bibles from that location, othen point IIS or Apache to that location (http transport needs work but is on the way-- more talk below) ... distribution of Bibles then becomes viral. It is spontaneous. No preparation. I have my library of books I use regularly with me and if someone wants one of my books, then they can install straight from my library. I don't think this should hold back usability in other ways. In what way does usability relate to storage format or network transport? I would rather that we provided some other functionality (or tool) to make this easy for end users to do, rather than make this mean that the final install of a module is a complete install source. I love the concept, but I think that your example is something that 0.0001% of users would be interested in, given the nerdery required to know how to do that! ;) I don't understand the suggestion or the nerdiness of the current method. We (the engine) does provide install functionality and we (frontend developers) do provide a tool that makes this easy for end users. Users already know how to install modules in their favorite software. The user wishing to install follows their familiar procedure to install. When prompted from which Remote Repository they would like to install, the user simply clicks a 'browse to local folder' button. I believe most frontends already support this for installing from a CD. BUT I liked the suggestion made in another post of having a prepare for distribution or publish menu item that would then automatically prepare a module (or selection of modules) for distribution. :) Use that menu item as step one, and then you can proceed through that list of examples that Troy has. As an added extra bonus, you get to specify the location to which to Save the module/s ready for distribution. That way a user doesn't even need to know where her/his modules are installed to, but instead can say desktop (yes, I'm using the LCD of a user who does everything on their desktop!) and then easily find that location share that folder. :) (or it could be specified as a network drive, for easy access to everyone on the network, etc.) The prepare
Re: [sword-devel] Remote Module Repository Wiki
Sure, of course I agree with Karl. If we decide on expanding (or contracting) the format of an installation repository, the InstallMgr class in the engine will operate against that new definition and frontend developers already using InstallMgr shouldn't need to change any code. On 11/05/2010 09:48 PM, Karl Kleinpaste wrote: Matthew Talbert ransom1...@gmail.com writes: Perhaps this isn't any better, but you can see the entire list here: http://ftp.xiphos.org/sword/zip/ Of course, it doesn't use very descriptive names, but it slightly more user friendly than opening the tar.gz. Well, again, I don't want a _human_ cracking open mods.d.tar.gz; I want the app's installer to do it, and display a nice set of available modules with descriptions and sizes and whatnot. I think InstallManager should be taught to do it all, just the way it does for FTP. ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 06/11/2010, at 10:42 PM, Troy A. Griffitts wrote: I was going to download the mods.d.tar.gz file (over HTTP) and use that for moduleinformation. Then download the ZIP of the module over HTTP. If a repo doesn't have the zip folder (or the mods.d.tar.gz file), then I would either have a fall-back to use the normal InstallManager, or perhaps I'd just say that PocketSword wouldn't support that repo. But after a year of PocketSword doing things the right way, I've decided to take the JSword approach and hope that doesn't cause too many issues... :( :) OK, This is vital for me to understand your experience. I presume the right way you speak of above is simply using the InstallMgr API. Why are you unhappy with the InstallMgr API? Does it not work? Is it too hard to use? You mention speed, below... The right way is by directly plugging into the InstallMgr API and getting it to do all the work. :) In order to get it to work, I've made a couple of rather minor changes, but that's not an issue at all, and I'm very happy to maintain my own set of patches required to get the SWORD lib working properly on an iOS device. :) (or, as an aside, I could submit some patches that have #defines for iOS, but I'm not too fussed. Perhaps they would be of interest if someone wanted to use the C++ lib for Android, but I'm not sure.) I want to switch to using HTTP only in PocketSword. Mobile phone carriers are rather horrid to deal with. My old carrier here in Australia had transparent proxies and all sorts of horrid things! Other carriers do other evil things, as well. And I am fairly certain some of them don't have FTP in the common use-set scenario for their users, even in this day and age of wireless internet, and so don't care if it works very well or not... :( I have observed first hand some annoying things with different carriers here -- so perhaps the US is ok where you only have one carrier for the iPhone hence only one set of horridness to deal with? ;) But, HTTP is so commonplace that it's harder to break it. I have yet to add proper proxy support into PocketSword, but that is high up on the list. But given that I'm thinking about rewriting the download code for PocketSword, I've put it off and I'll do both at once? BTW, I think the InstallMgr API works fairly well and is extremely easy to use. :) When I started work on PocketSword (I was helping Ian out at first), I tackled the downloading of modules as my first task and I found it fairly easy to figure out how to get it to work. :) my personal tests on an iOS device show that accessing the CrossWire repo via FTP is reasonably slower than accessing it via HTTP. Really? I wouldn't have expected this at all. This may have to do with phone carriers? altho, I'm fairly certain I tested this over WiFi as well... In my experience, it feels like the internet is moving away from FTP and using HTTP for everything? Just like HTML is now being used for many different things that it was never intended to be used for... ;) But, firewalls and proxies often screw with FTP much more than HTTP and many orgs/companies simply block FTP, hence people moving to HTTP to get around those issues. :( I want the reasonably large performance gains that I can gain from simply switching protocols! :) Yes, indeed. This is a very valid concern. Let me ask, what FTP plugin were you using, ftplib or curl? I am wondering if it might be the difference in plugin. I recently switched BibleCS's InstallMgr GUI to use the ftplib system instead of CURL and surprisingly seem to have less troubles and more responsiveness. There is still one major shortcoming of ftplib-- it cannot download directly to a buffer for a directory listing, we use a temporary file-- but I hope to resolve that in the next day or two. Speed is a very real and practical concern and if we have a problem, we need to resolve it. I see you just committed that change. :) I have only tested with curl -- as that is the only way to get HTTP downloading working. :) I never had the option to use ftplib, as downloading to a temporary file under iOS is NOT a good idea... :( I could test with ftplib, but given the other objections I have to FTP (*sigh* phone carriers *sigh*), I'm hoping to avoid FTP. :( On 06/11/2010, at 1:35 AM, Troy A. Griffitts wrote: 2) The viral ability END USERS to share modules. If I can, for example, o use the builtin installer of my SWORD software on my Android device to install my important modules othen plug it into my friends computer and install modules FROM my android device WITHOUT ANY PREPARATION o then from his computer right-click on the top module library folder and Share as... 'SWORD Modules', o then go to any computer on the network and install modules from that SWORD Modules share, o then install FileZilla on one of those computers and let anyone over the
Re: [sword-devel] Remote Module Repository Wiki
I do understand the very real practical concerns about FTP you point out, but we have practically received almost no support emails from users not being able to install because FTP was blocked. Most hosting services provide FTP access. The issue we ran into before was some didn't provide anonymous ftp access, which we've rectified by adding username / password fields in our repository registry and support for this in our installer, and in reality to install an FTP server on a windows box is trivial compared to the ongoing maintenance of assuring zips are created and placed in the correct folder, an index file for the zips is created and kept updated whenever anything changes, etc. I don't think that installing an FTP server is trivial. I can't agree that 'most hosting services provide FTP access.' Most cheep hosting services provide ftp access only for its maintainer, ftp access is not permitted for public use. HTTP services are name-based , FTP services are IP-based. So if one want to service an FTP server, one must own one IP address for it, contrary to the fact that many HTTP services can share one IP address. Furthermore FTP service needs two connections , one for control and one for data. one is outgoing, another is incoming. this make firewall rules very complexing. Servicing an HTTP server is easy, and servicing an FTP server is annoying. If both methods can serve same jobs, I would rather choose HTTP . It might seem trivial to us, and might actually be trivial for the publishers, in practice (and might eventually be how we end up going if we do find real roadblocks with our current FTP mechanism), but: -- ad...@bible.salterrae.net ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
I think Troy, the concern is correct. For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must. Having a multiplicity of methods of getting modules into the system would certainly be easier. My preference: 1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp 2) add methods for local installation of zips. Look at MK Bible - pull a zip over the programme and it gets installed. This is - emphatically - not how I would want to install a large selection of modules or how I would want to publish the same, but it is probably the best usability i have seen for a frontend using only 2-3 modules (which is what MKBible is laid out for) and a small time publisher or someone who has target audience of little computer literacy Peter Von: ad...@bible.salterrae.net An: SWORD Developers\' Collaboration Forum sword-devel@crosswire.org Betreff: Re: [sword-devel] Remote Module Repository Wiki I do understand the very real practical concerns about FTP you point out, but we have practically received almost no support emails from users not being able to install because FTP was blocked. Most hosting services provide FTP access. The issue we ran into before was some didn't provide anonymous ftp access, which we've rectified by adding username / password fields in our repository registry and support for this in our installer, and in reality to install an FTP server on a windows box is trivial compared to the ongoing maintenance of assuring zips are created and placed in the correct folder, an index file for the zips is created and kept updated whenever anything changes, etc. I don't think that installing an FTP server is trivial. I can't agree that 'most hosting services provide FTP access.' Most cheep hosting services provide ftp access only for its maintainer, ftp access is not permitted for public use. HTTP services are name-based , FTP services are IP-based. So if one want to service an FTP server, one must own one IP address for it, contrary to the fact that many HTTP services can share one IP address. Furthermore FTP service needs two connections , one for control and one for data. one is outgoing, another is incoming. this make firewall rules very complexing. Servicing an HTTP server is easy, and servicing an FTP server is annoying. If both methods can serve same jobs, I would rather choose HTTP . It might seem trivial to us, and might actually be trivial for the publishers, in practice (and might eventually be how we end up going if we do find real roadblocks with our current FTP mechanism), but: -- ad...@bible.salterrae.net ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page -- GMX DSL Doppel-Flat ab 19,99 euro;/mtl.! Jetzt auch mit gratis Notebook-Flat! http://portal.gmx.net/de/go/dsl ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Fri, Nov 5, 2010 at 8:02 AM, Peter von Kaehne ref...@gmx.net wrote: I think Troy, the concern is correct. For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must. Having a multiplicity of methods of getting modules into the system would certainly be easier. My preference: 1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp While I very strongly agree that FTP should not be held up as the best solution for everyone (I have worked with and for several organizations who flat out refused to ever allow FTP of any type to be a part of their systems and who primarily ran IIS for their web content), I'm not sure the argument from excessive cost is accurate. Annual DNS leasing from my provider is less than $40/year (I want to say it's only about $17, but I might be remembering incorrectly) Dedicated VPS hosting with dedicated IP address, about 20 GB of drive space and full root access to my system is $19.95/mo. If $23 per month is excessive for anyone who really intends to distribute modules, they can almost certainly find someone else in the SWORD community who will be willing to host it for them at very little to no cost. Additionally I can host any low-volume traffic I want from home by registering with a place like dyndns.com and setting up a server on any connection that is moderately reliable (my home). The option is not viable for people in developing countries but is certainly more than viable for any first world country I'm aware of. --Greg ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
The option is not viable for people in developing countries but is certainly more than viable for any first world country I'm aware of. And that is to some extent one of our main target audiences - it is nice to have excellent English language bible study material on CW, but we are the only people who provide stuff for about 100 languages or so (and growing). To make this easier and to invite contribution is my real desire. Peter -- Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief! Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Sat, Nov 6, 2010 at 12:02 AM, Peter von Kaehne ref...@gmx.net wrote: I think Troy, the concern is correct. For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must. Having a multiplicity of methods of getting modules into the system would certainly be easier. My preference: 1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp 2) add methods for local installation of zips. Look at MK Bible - pull a zip over the programme and it gets installed. This is - emphatically - not how I would want to install a large selection of modules or how I would want to publish the same, but it is probably the best usability i have seen for a frontend using only 2-3 modules (which is what MKBible is laid out for) and a small time publisher or someone who has target audience of little computer literacy It's capable of far more than just dealing with small numbers of books when you have two tools: 1. DownThemAll!: Download multiple zips at one go. 2. Multi-selection in Explorer / file window: I tried this in MK, and it didn't seem to work correctly, but I probably don't have the latest version. It definitely works in BPBible, and allows you to install multiple books in one go. (though I'm not convinced that a large percentage has these tools at their disposable or is aware of them). While drag and drop installation has a certain coolness factor, I feel having a menu option (like the File Install Books BPBible has, also including multiple book installation) is more discoverable and thus perhaps more useful to the starting off user. Also, people coming from the background of e-Sword or similar tools are probably used to seeing a large collection of books on a web page to download, and when they see a similar list at Crosswire they do the same: download books and look for a way to install them, while some people just like downloading a thing to make sure they have it and could it share it with others if they wanted to (though with the cloud this is probably less common than it used to be). Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Von: Jonathan Morgan jonmmor...@gmail.com (though I'm not convinced that a large percentage has these tools at their disposable or is aware of them). At which point that particular debate probably ends :-) While drag and drop installation has a certain coolness factor, I feel having a menu option (like the File Install Books BPBible has, also including multiple book installation) is more discoverable and thus perhaps more useful to the starting off user. No objection, but also no contradiction. Also, people coming from the background of e-Sword or similar tools are probably used to seeing a large collection of books on a web page to download, and when they see a similar list at Crosswire they do the same: download books and look for a way to install them, while some people just like downloading a thing to make sure they have it and could it share it with others if they wanted to (though with the cloud this is probably less common than it used to be). I think the huge number of support emails I have downloaded x number of modules and now what am I supposed to do suggests the same - though of course us taking away the zips from the webpage would be a helpful step to stop that. Downloading a zip for sharing is of course a very useful way to pass about modules via sneaker net. And that is in turn a way of some importance where the interent is either sparse or controlled. I think we acknowledge this by making the zip's available but we do not exactly facilitate it beyond that point. And that is a shame. Peter Jon -- GRATIS! Movie-FLAT mit über 300 Videos. Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 11/05/2010 09:34 AM, Peter von Kaehne wrote: Von: Jonathan Morganjonmmor...@gmail.com (though I'm not convinced that a large percentage has these tools at their disposable or is aware of them). At which point that particular debate probably ends :-) While drag and drop installation has a certain coolness factor, I feel having a menu option (like the File Install Books BPBible has, also including multiple book installation) is more discoverable and thus perhaps more useful to the starting off user. No objection, but also no contradiction. Also, people coming from the background of e-Sword or similar tools are probably used to seeing a large collection of books on a web page to download, and when they see a similar list at Crosswire they do the same: download books and look for a way to install them, while some people just like downloading a thing to make sure they have it and could it share it with others if they wanted to (though with the cloud this is probably less common than it used to be). I think the huge number of support emails I have downloaded x number of modules and now what am I supposed to do suggests the same - though of course us taking away the zips from the webpage would be a helpful step to stop that. We should at least drop the MacOSX ones. And perhaps the Windows ones. If someone has installed Xiphos or Bible Desktop, but never has installed BibleCS, then these don't work. They also don't work on 64-bit (according to reports). IIRC: Troy said these were there to satisfy a publisher's request that some info from a module's conf be shown for modules installed this way. It only ever worked in BibleCS. Is that still a valid concern? Downloading a zip for sharing is of course a very useful way to pass about modules via sneaker net. And that is in turn a way of some importance where the interent is either sparse or controlled. I think we acknowledge this by making the zip's available but we do not exactly facilitate it beyond that point. And that is a shame. I think every frontend should be able to browse, at least locally, for a raw zip and install it. That way the downloads from the web page become less of a support question. On a different note, I think it might be good to have multiple install locations, one for each repository. Today, we have a KJV, but if this gets going I could see a publisher with their own repository having a KJV. JSword *had* a mechanism to allow for duplicates such as these by disambiguating them by their repository name, e.g. KJV (CrossWire) We pulled it because it didn't happen in reality and because it was not implemented very well. In this situation, zips are anonymous and would go in a private repository. Today, JSword allow for multiple repository locations, but only one for install. I use my own private repository for modules I have created so that I don't accidentally wack them. DM Peter Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Sat, Nov 6, 2010 at 12:34 AM, Peter von Kaehne ref...@gmx.net wrote: Von: Jonathan Morgan jonmmor...@gmail.com (though I'm not convinced that a large percentage has these tools at their disposable or is aware of them). At which point that particular debate probably ends :-) Indeed :). While drag and drop installation has a certain coolness factor, I feel having a menu option (like the File Install Books BPBible has, also including multiple book installation) is more discoverable and thus perhaps more useful to the starting off user. No objection, but also no contradiction. Both things (I suspect) require frontend support as well as SWORD support, if any. I suspect drag and drop needs a little more work to implement than a menu option popping up a file chooser, and then delegating to SWORD, but I'm really not sure. Also, people coming from the background of e-Sword or similar tools are probably used to seeing a large collection of books on a web page to download, and when they see a similar list at Crosswire they do the same: download books and look for a way to install them, while some people just like downloading a thing to make sure they have it and could it share it with others if they wanted to (though with the cloud this is probably less common than it used to be). I think the huge number of support emails I have downloaded x number of modules and now what am I supposed to do suggests the same - though of course us taking away the zips from the webpage would be a helpful step to stop that. Downloading a zip for sharing is of course a very useful way to pass about modules via sneaker net. And that is in turn a way of some importance where the interent is either sparse or controlled. I think we acknowledge this by making the zip's available but we do not exactly facilitate it beyond that point. And that is a shame. Two points here: 1. Part of the trouble comes from having multiple download options (e.g. Raw Zip, Mac and Windows). I have in the past suggested that there only be one (and that that one be the Raw Zip, as the most universal). Apparently BPBible supports all three download formats, but I can't expect everyone else to do that, and it's still potentially quite confusing. 2. Offering a list of downloads at CrossWire tends to suggest that they are the *only* books available. While I can download zip files from Xiphos FTP directly (for example), it's not easy to find out about, and I'm sure some people evaluate the software purely on the basis of the books available without even trying the software. I think the latter point is probably the more important of the two. While it is no doubt a worthy goal to have publishers independently able to publish their own work, I expect to be able to come to a software organisation's website and see what books are available from that website for that software. I don't know if we can facilitate this discovery from the website (and if so how) but people do judge the software on the library and so this is something that bears thinking about. (Once we get to the Logoses of this world people also expect an integrated web store which allows you to purchase books and keeps track of every book you have access to and allows you to sync between software, etc. I don't think we're going there any time soon...) BPBible relies on zip installation at present. I think this is a reasonable decision, though we would like to grow Install Manager support at some point (in part to get easier access to the Xiphos repository). Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Jonathan Morgan jonmmor...@gmail.com writes: 2. Offering a list of downloads at CrossWire tends to suggest that they are the *only* books available. While I can download zip files from Xiphos FTP directly (for example), it's not easy to find out about ... BPBible relies on zip installation at present. I think this is a reasonable decision, though we would like to grow Install Manager support at some point (in part to get easier access to the Xiphos repository). You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open, and see what's available. Just two assumptions: - the zip subdir is at the same level as mods.d - xyz.conf has a corresponding xyz.zip. DM Smith dmsm...@crosswire.org writes: I think every frontend should be able to browse, at least locally, for a raw zip and install it. That way the downloads from the web page become less of a support question. You are re-invoking a question I asked some months ago, about baseline behavior -vs- per-app features. For all apps to support such behavior is to draw a rather hard line in the sand under the baseline category. ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Sat, Nov 6, 2010 at 1:24 AM, Karl Kleinpaste k...@kleinpaste.org wrote: Jonathan Morgan jonmmor...@gmail.com writes: 2. Offering a list of downloads at CrossWire tends to suggest that they are the *only* books available. While I can download zip files from Xiphos FTP directly (for example), it's not easy to find out about ... BPBible relies on zip installation at present. I think this is a reasonable decision, though we would like to grow Install Manager support at some point (in part to get easier access to the Xiphos repository). You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open, and see what's available. Just two assumptions: - the zip subdir is at the same level as mods.d - xyz.conf has a corresponding xyz.zip. Agreed. I can and have. I don't expect my average user to, though. DM Smith dmsm...@crosswire.org writes: I think every frontend should be able to browse, at least locally, for a raw zip and install it. That way the downloads from the web page become less of a support question. You are re-invoking a question I asked some months ago, about baseline behavior -vs- per-app features. For all apps to support such behavior is to draw a rather hard line in the sand under the baseline category. I think it should be baseline for the same reasons as DM. However, since we have very little that is truly baseline this would probably just join the list of would be nice to have, but too much work for right now (and I willingly concede that BPBible has this too). Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
There are a number of finely delineated topics here we should separate. 1) Data format for our module installation source. i.e, is it a list of zips, is it an existing install set, etc. Simply to be explicit, we have obvious use cases which come into play: a) Installing our software -- all implementations will be different because we have a variety of packages running on a variety of platforms. b) Installing modules from a module installation source c) Running our software The final item (c) seems out of place but really is important. We have successfully achieved a common data storage format for an installation set of books (i.e. library/mods.d/..., library/modules/...) across a vast number of platforms. Sure this hierarchy of files moves around to different locations on different platforms, but once the location is pinpointed, everything under that location remains the same for any of our platforms. The 'holy grail' for implementation of the (1) above is for both (b) and (c) use cases to be handled by the SAME dataset, and is what we are working towards. In other words, if an installed module library can, itself, also be an installation source for other instances of any SWORD software, then distribution of Bibles becomes viral. This brings us to topic 2. 2) The viral ability END USERS to share modules. If I can, for example, o use the builtin installer of my SWORD software on my Android device to install my important modules o then plug it into my friends computer and install modules FROM my android device WITHOUT ANY PREPARATION o then from his computer right-click on the top module library folder and Share as... 'SWORD Modules', o then go to any computer on the network and install modules from that SWORD Modules share, o then install FileZilla on one of those computers and let anyone over the Internet install Bibles from that location, o then point IIS or Apache to that location (http transport needs work but is on the way-- more talk below) ... distribution of Bibles then becomes viral. It is spontaneous. No preparation. I have my library of books I use regularly with me and if someone wants one of my books, then they can install straight from my library. 3) The TRANSPORT TECHNOLOGY used to enable this over the Internet. We chose FTP because it was more practical. It is very difficult to implement the programmatic traversal of a directory and file hierarchy over HTTP. Apache and IIS all spew out different html for directory listings. Now having said this, Nic Carter submitted the first pass at this which works for our apache server (and likely many apache servers which spew out the vanilla html index page like here: http://crosswire.org/ftpmirror/pub/ So, HTTP is currently possible in the engine and if we get push back from publishers or find users not able to use FTP due to firewall restrictions, we plan to fully support HTTP moving forward. We simply need to break out our directory parsing code and make it work well with a few of the more mainstream httpd services like Apache and IIS. 4) A single zip file for installation of a module (or modules) is simply a zip of (1c) above. Of the 3 'zip' formats we support for download from our server, only the 'rawzip' format should survive. I believe I can safely say we all agree on that. This is in essence still the same thing. It is a module repository of exactly 1 module in a difference storage medium. Heck it could be 5 installed modules all zipped up and it would still be the same thing. As I see it, it is an end user preference left to the exercise of frontend developers. Can my users operate their file browser and drag and drop to my app easier than they can navigate their Browse to folder dialog box in Install SWORD Modules from path... option in my installer? If so, then I should support drag and drop on my frontend-- after all it is simply an unzip. But the user gets no benefit of our full installer: not description, to versioning upgrade information if they already have an instance of that module installed, to category, no language, etc. 5) If we find we cannot achieve this 'no prep' installation utopia in the real world and it is necessary to have an intermediate format. Matthew Talbert had a great suggestion on IRC last night to simply add a 'prep for publishing' (or in his words simply publish) option to the InstallMgr interface which would create (1) module installation source data format if indeed it technically must be different. Great idea if we need to concede to 2 disparate data formats, but I am still hopeful we can avoid this extra step for would-be module distributors-- end users and publishers alike. Hope this separates out thoughts a little better. Troy On 11/05/2010 01:15 PM, Greg Hellings wrote: On Fri, Nov 5, 2010 at 8:02 AM, Peter von Kaehne ref...@gmx.net wrote: I think Troy, the concern is correct. For the publisher with some decent IT
Re: [sword-devel] Remote Module Repository Wiki
What about using WebDAV for HTTP? It supports directory listings (i.e. collections). On Fri, Nov 5, 2010 at 3:35 PM, Troy A. Griffitts scr...@crosswire.orgwrote: 3) The TRANSPORT TECHNOLOGY used to enable this over the Internet. We chose FTP because it was more practical. It is very difficult to implement the programmatic traversal of a directory and file hierarchy over HTTP. Apache and IIS all spew out different html for directory listings. Now having said this, Nic Carter submitted the first pass at this which works for our apache server (and likely many apache servers which spew out the vanilla html index page like here: http://crosswire.org/ftpmirror/pub/ So, HTTP is currently possible in the engine and if we get push back from publishers or find users not able to use FTP due to firewall restrictions, we plan to fully support HTTP moving forward. We simply need to break out our directory parsing code and make it work well with a few of the more mainstream httpd services like Apache and IIS. -- Weston Ruter http://weston.ruter.net/ @westonruter http://twitter.com/westonruter - Google Profilehttp://www.google.com/profiles/WestonRuter#about ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 11/05/2010 02:00 PM, DM Smith wrote: IIRC: Troy said these were there to satisfy a publisher's request that some info from a module's conf be shown for modules installed this way. It only ever worked in BibleCS. Is that still a valid concern? Well, almost. There is a feature in the engine where you can specify in your module repository: [Globals] AutoInstall=newmods where newmods is a folder which lives at the same place as mods.d/ and modules/ on loading of a module repository, SWMgr does a scan of this folder for .conf files and will copy them to the mods.d folder, but first will call a hook method SWMgr::AddModToConfig which allows a fontend to display a nice dialog box: Hey, you've installed a new module! Here's the About information... or whatever. This mechanism was, as DM mentions, added at the request of a publisher when BibleCS was the only frontend available. Permission to use their module was predicated on us showing their about information upon installation of their module. I'd definitely like to remove the Win32 specific InstallShield module install packages. BibleCS InstallMgr is undergoing some play nice with other frontends updates right now looking toward an eminent update of BibleCS compiled against 1.6.2 and with av11n support. I suggest after this release we remove the windows installer files. Do we have any feedback on the MacOSX installer files? Troy ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Jonathan Morgan jonmmor...@gmail.com writes: You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open, and see what's available. Just two assumptions: - the zip subdir is at the same level as mods.d - xyz.conf has a corresponding xyz.zip. Agreed. I can and have. I don't expect my average user to, though. My point was that a module manager (that is, the InstallManager class) can do this readily. It should be invisible for http+zip retrieval, just as it already is for ftp+components retrieval. It should just happen as part of the request for local refresh of the remote content. The downside is that, for ftp access today, mods.d.tar.gz is technically not required; in its absence, InstallManager will crawl into mods.d and find *.conf the hard way. I don't get warm, fuzzy feelings about the possibility of implementing an adequately general and successful parsing scheme of directory content via http access. ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 11/05/2010 10:35 AM, Troy A. Griffitts wrote: 2) The viral ability END USERS to share modules. Don't we have to be concerned with locked modules? With the unlock key/cipher being held in the conf, isn't that a bad thing? Should we do as BibleTime and have a separate store of keys? If so, where? In Him, DM ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Sat, Nov 6, 2010 at 1:49 AM, Troy A. Griffitts scr...@crosswire.orgwrote: On 11/05/2010 02:00 PM, DM Smith wrote: IIRC: Troy said these were there to satisfy a publisher's request that some info from a module's conf be shown for modules installed this way. It only ever worked in BibleCS. Is that still a valid concern? Well, almost. There is a feature in the engine where you can specify in your module repository: [Globals] AutoInstall=newmods where newmods is a folder which lives at the same place as mods.d/ and modules/ on loading of a module repository, SWMgr does a scan of this folder for .conf files and will copy them to the mods.d folder, but first will call a hook method SWMgr::AddModToConfig which allows a fontend to display a nice dialog box: Hey, you've installed a new module! Here's the About information... or whatever. This mechanism was, as DM mentions, added at the request of a publisher when BibleCS was the only frontend available. Permission to use their module was predicated on us showing their about information upon installation of their module. I'd definitely like to remove the Win32 specific InstallShield module install packages. BibleCS InstallMgr is undergoing some play nice with other frontends updates right now looking toward an eminent update of BibleCS compiled against 1.6.2 and with av11n support. I suggest after this release we remove the windows installer files. Do we have any feedback on the MacOSX installer files? This was discussed extensively over a year ago. See http://www.mail-archive.com/sword-devel@crosswire.org/msg18634.html for Manfred's response to that thread. The relevant quotations would be: Mac OS X zip file is still supported and we will keep support. Actually it only is a zipped folder that has a .swd extension which is registered for MacSword in the OS and is shown as a file in the OS file manager. This makes it possible that you can have MacSword openened automatically when the file is opened. But we also support just placing the unzipped raw folder in the Mac OS X default module location path. Given that response, I'm not sure there is enough advantage in providing two separate options (if the Window zip goes) just to have a separate file extension for Mac (though in some ways I think having a separate file extension would be nice, so that people treat it as a distinct entity rather than just a zip file that gets confused with the hundreds of other zip files that they have in their Download directory - or at least that I do). Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 05/11/10 00:25, Troy A. Griffitts wrote: Is there any chance I can get some of you who have contributed to this valuable resource to clean it up a bit. I personally would like to see: its size reduced as small as possible to show how easy it is to run a remote repository. Initial, simple repository section greatly appreciated, but the rest is still a little cloudy for even me to understand. To come back to the original request 1) I have removed all references to MacOS zip and Windows zips from the wiki. 2) I have removed all references to Jsword needing special treatment from the wiki 3) I have mildly simplified the explanations. Please check it for correctness and do any further improvements as requested by Troy Peter ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Troy A. Griffitts scr...@crosswire.org writes: Matthew Talbert had a great suggestion on IRC last night to simply add a 'prep for publishing' (or in his words simply publish) option to the InstallMgr interface... It's great, but funny, to see all the sudden interest in publishing users' content. I've talked about it a couple times here in sword-devel, but the idea never got any traction before. Why now? Nearly 3 years ago: http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Fri, Nov 5, 2010 at 10:35 AM, Jonathan Morgan jonmmor...@gmail.com wrote: On Sat, Nov 6, 2010 at 1:24 AM, Karl Kleinpaste k...@kleinpaste.org wrote: Jonathan Morgan jonmmor...@gmail.com writes: 2. Offering a list of downloads at CrossWire tends to suggest that they are the *only* books available. While I can download zip files from Xiphos FTP directly (for example), it's not easy to find out about ... BPBible relies on zip installation at present. I think this is a reasonable decision, though we would like to grow Install Manager support at some point (in part to get easier access to the Xiphos repository). You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open, and see what's available. Just two assumptions: - the zip subdir is at the same level as mods.d - xyz.conf has a corresponding xyz.zip. Agreed. I can and have. I don't expect my average user to, though. Perhaps this isn't any better, but you can see the entire list here: http://ftp.xiphos.org/sword/zip/ Of course, it doesn't use very descriptive names, but it slightly more user friendly than opening the tar.gz. Matthew ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
rote: Troy A. Griffitts scr...@crosswire.org writes: Matthew Talbert had a great suggestion on IRC last night to simply add a 'prep for publishing' (or in his words simply publish) option to the InstallMgr interface... It's great, but funny, to see all the sudden interest in publishing users' content. I've talked about it a couple times here in sword-devel, but the idea never got any traction before. Why now? Nearly 3 years ago: http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html Part of this concept of publish was not really pushing to a remote repository, but the idea that we could (say using the Xiphos module manager) pick a subset of installed modules, and prepare them for remote access. If, for example, we were to agree that an HTTP repository could consist of a single file listing the contents (really, no different than mods.tar.gz) + a set of zipped files, then this publish would create this whole directory, ready for simply copying to any http-accessible location. The other part of the publish would be publishing a single module to a remote location where a user has an account. This would be very similar, but it would just be pushing a single zip file, and the remote server would be responsible for putting it in the correct location, and updating the repository listing. Part of why I'd like very much to see a repository able to consist of this directory listing + zipped modules is that it makes the whole idea of remote publishing much easier. With FTP, it's a pain to set up various directories with different permission schemes on the fly, while using HTTP would allow typical server-side software (eg, PHP, django, etc) to handle rather complex schemes easily. It would be relatively simple on the same server to have a system that allowed a user to come sign up for an account, and suddenly that user has publish capability unique to him. Then, if he wanted to publish to a more shared and publicly visible location, a simple flip of the switch would allow him to publish there as well. This would allow everyone to quickly share with their friends, but require interaction with a repository owner before being allowed to publish there. Matthew ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Matthew Talbert ransom1...@gmail.com writes: Perhaps this isn't any better, but you can see the entire list here: http://ftp.xiphos.org/sword/zip/ Of course, it doesn't use very descriptive names, but it slightly more user friendly than opening the tar.gz. Well, again, I don't want a _human_ cracking open mods.d.tar.gz; I want the app's installer to do it, and display a nice set of available modules with descriptions and sizes and whatnot. I think InstallManager should be taught to do it all, just the way it does for FTP. ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
Am 05.11.2010 um 16:41 schrieb DM Smith: On 11/05/2010 10:49 AM, Troy A. Griffitts wrote: Do we have any feedback on the MacOSX installer files? I think the download pages should clearly say, perhaps on every page to use the program's built-in installer. We get lots of support questions from MacSword and BibleDesktop users on how to install the downloads from CrossWire on a Mac. The answer always is the same: Use your program's install manager. The Mac downloads only work for MacSword. Unzip on a Mac creates a folder with the same name as the file, but without the zip extension. By having KJV.swd.zip, this produces KJV.swd on the desktop (or in the download folder or where ever). If MacSword is installed then the *.swd folders don't look like a folder, but a MacOS package and the contents are hidden in Finder. One can double click on it to open it in MacSword. But unless the user moves it to a folder that MacSword looks at (IIRC: ~/Library/Application Support/Sword), it won't be seen by the program. It is not very Mac like to have people drag stuff into ~/Library/Application Support. So we haven't been explaining that to them. (I field MacSword questions to CrossWire's support from time-to-time.) Also,I don't think that PocketSword can use them. I defer to Manfred on whether to keep them. I thought that he said here that they could go away now that MacSword has an installer. But I can't find it now. Well. Actually I would keep them because they are quite comfortable to use on a Mac because of how they are treated by Finder and MacSword. But I understand that it is confusing to have three different module download options on our web page. The user can, if he wishes, rename the unzipped zip download and add the .swd. And since we have the install manager I guess most people do use it. Manfred ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
A year or so ago I wrote the shell of a (PHP) website which would allow a user to browse various selected modules, purchase one, create a unique encryption key for that user and encrypt the module for them. It would then give the user a single download which could be installed with a Python/Qt app I wrote, which would also accept the user's encryption key and enter it into the module conf file. It never went anywhere because of various non-technical reasons, but adding user upload and giving a user the chance to release their own modules, etc, would be easy extra functionality. Note: everything about the site was a temporary hack designed to prove the technical aspects could be done and not to actually be used. E.G. It did not actually require money and there was no user sign up and the like. But those are trivial and off topic from point of view of the SWORD functionality. --Greg On Nov 5, 2010 4:45 PM, Matthew Talbert ransom1...@gmail.com wrote: rote: Troy A. Griffitts scr...@crosswire.org writes: Matthew Talbert had a great suggestion on IRC last night to simply add a 'prep for publishing' (or in his words simply publish) option to the InstallMgr interface... It's great, but funny, to see all the sudden interest in publishing users' content. I've talked about it a couple times here in sword-devel, but the idea never got any traction before. Why now? Nearly 3 years ago: http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html Part of this concept of publish was not really pushing to a remote repository, but the idea that we could (say using the Xiphos module manager) pick a subset of installed modules, and prepare them for remote access. If, for example, we were to agree that an HTTP repository could consist of a single file listing the contents (really, no different than mods.tar.gz) + a set of zipped files, then this publish would create this whole directory, ready for simply copying to any http-accessible location. The other part of the publish would be publishing a single module to a remote location where a user has an account. This would be very similar, but it would just be pushing a single zip file, and the remote server would be responsible for putting it in the correct location, and updating the repository listing. Part of why I'd like very much to see a repository able to consist of this directory listing + zipped modules is that it makes the whole idea of remote publishing much easier. With FTP, it's a pain to set up various directories with different permission schemes on the fly, while using HTTP would allow typical server-side software (eg, PHP, django, etc) to handle rather complex schemes easily. It would be relatively simple on the same server to have a system that allowed a user to come sign up for an account, and suddenly that user has publish capability unique to him. Then, if he wanted to publish to a more shared and publicly visible location, a simple flip of the switch would allow him to publish there as well. This would allow everyone to quickly share with their friends, but require interaction with a repository owner before being allowed to publish there. Matthew ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On Sat, Nov 6, 2010 at 1:53 AM, Karl Kleinpaste k...@kleinpaste.org wrote: Jonathan Morgan jonmmor...@gmail.com writes: You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open, and see what's available. Just two assumptions: - the zip subdir is at the same level as mods.d - xyz.conf has a corresponding xyz.zip. Agreed. I can and have. I don't expect my average user to, though. My point was that a module manager (that is, the InstallManager class) can do this readily. It should be invisible for http+zip retrieval, just as it already is for ftp+components retrieval. It should just happen as part of the request for local refresh of the remote content Indeed. But BPBible does not have InstallManager support. Jon ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 06/11/2010, at 2:41 AM, DM Smith wrote: On 11/05/2010 10:49 AM, Troy A. Griffitts wrote: Do we have any feedback on the MacOSX installer files? I think the download pages should clearly say, perhaps on every page to use the program's built-in installer. We get lots of support questions from MacSword and BibleDesktop users on how to install the downloads from CrossWire on a Mac. The answer always is the same: Use your program's install manager. ... Also,I don't think that PocketSword can use them. No, PocketSword doesn't use them. PocketSword has support for installing a raw zip via a web browser (simple http form that can submit a zip file to the device) (which can, incidentally, contain several modules zipped together) and use of the InstallManager. My suggestion (as a Mac user and MacSword user) would be to simplify things by only having raw zip modules available for download, and push people to use the InstallManager. Perhaps even add an annoying page to the jsp download pages such that if a user clicks on the download link, it initially displays an (annoying!) page describing that the user should be using the internal InstallManager functionality of their frontend, and then providing a link to actually download the module. :) That way we can hopefully get users to install modules the easy way, rather than manually. :) Yes, it would be annoying, but hopefully the annoyingness of it all would mean users would be persuaded to do things the way we'd like? ;) Just my random thoughts... :) ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 06/11/2010, at 1:53 AM, Karl Kleinpaste wrote: Jonathan Morgan jonmmor...@gmail.com writes: You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open, and see what's available. Just two assumptions: - the zip subdir is at the same level as mods.d - xyz.conf has a corresponding xyz.zip. Agreed. I can and have. I don't expect my average user to, though. My point was that a module manager (that is, the InstallManager class) can do this readily. It should be invisible for http+zip retrieval, just as it already is for ftp+components retrieval. It should just happen as part of the request for local refresh of the remote content. The downside is that, for ftp access today, mods.d.tar.gz is technically not required; in its absence, InstallManager will crawl into mods.d and find *.conf the hard way. I don't get warm, fuzzy feelings about the possibility of implementing an adequately general and successful parsing scheme of directory content via http access. Actually, I was planning on implementing this for PocketSword in the near future. The idea was that I was going to move completely away from FTP and only support HTTP. But given that parsing HTTP is too painful (I initially submitted a patch for HTTP parsing, but it only works for CrossWire and not for the Bible.org nor for the Xiphos repos, and I have no intention of modifying the parsing code even more in order to try to support more web servers!), I was going to download the mods.d.tar.gz file (over HTTP) and use that for module information. Then download the ZIP of the module over HTTP. If a repo doesn't have the zip folder (or the mods.d.tar.gz file), then I would either have a fall-back to use the normal InstallManager, or perhaps I'd just say that PocketSword wouldn't support that repo. But after a year of PocketSword doing things the right way, I've decided to take the JSword approach and hope that doesn't cause too many issues... :( FYI, my personal tests on an iOS device show that accessing the CrossWire repo via FTP is reasonably slower than accessing it via HTTP. And that's simply by using the existing InstallManager code (including my HTTP parsing patch) and switching protocols between FTP HTTP. And given that PocketSword only runs on iOS devices, I want the reasonably large performance gains that I can gain from simply switching protocols! :) Given my lack of free programming hours atm, this isn't going to happen this year. But it is in the back of my head to implement at some stage in the future. I also have some other urgent fixes to do to PocketSword's downloads mechanism, such as providing background downloading. I mentioned to David the other day that I'm finishing up my current job at the end of the year in order to resume studying next year, but will have a few weeks of (hopefully!) full-time work on PocketSword in order to try to catch up on some of these urgent tasks. BUT I'm glad this discussion is taking place now, so I can see the resolution of all of this, and that may influence my decision on how I proceed with improving PocketSword's download mechanism. Thanks heaps for all the discussion so far, please keep it coming! ybic nic... :) ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
On 06/11/2010, at 1:35 AM, Troy A. Griffitts wrote: 2) The viral ability END USERS to share modules. If I can, for example, o use the builtin installer of my SWORD software on my Android device to install my important modules o then plug it into my friends computer and install modules FROM my android device WITHOUT ANY PREPARATION o then from his computer right-click on the top module library folder and Share as... 'SWORD Modules', o then go to any computer on the network and install modules from that SWORD Modules share, o then install FileZilla on one of those computers and let anyone over the Internet install Bibles from that location, o then point IIS or Apache to that location (http transport needs work but is on the way-- more talk below) ... distribution of Bibles then becomes viral. It is spontaneous. No preparation. I have my library of books I use regularly with me and if someone wants one of my books, then they can install straight from my library. I don't think this should hold back usability in other ways. I would rather that we provided some other functionality (or tool) to make this easy for end users to do, rather than make this mean that the final install of a module is a complete install source. I love the concept, but I think that your example is something that 0.0001% of users would be interested in, given the nerdery required to know how to do that! ;) BUT I liked the suggestion made in another post of having a prepare for distribution or publish menu item that would then automatically prepare a module (or selection of modules) for distribution. :) Use that menu item as step one, and then you can proceed through that list of examples that Troy has. As an added extra bonus, you get to specify the location to which to Save the module/s ready for distribution. That way a user doesn't even need to know where her/his modules are installed to, but instead can say desktop (yes, I'm using the LCD of a user who does everything on their desktop!) and then easily find that location share that folder. :) (or it could be specified as a network drive, for easy access to everyone on the network, etc.) Anyway, just some more thoughts. Sorry they're coming so much later than when these were posted -- a fun consequence of living in Oz. :) ps: I could easily be persuaded to add an export feature to PocketSword, which would work over WiFi or 3G (similar to the current way of manually installing a module from a ZIP file), so you could share modules on your device with friends. Of course, this would need to be switched on manually, and wouldn't always be running in the background, so as to conserve battery! But would be quite funky. :) But given you need to press a button to switch it on, at that point I could get PocketSword to prep the selected modules for distribution. :) ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
[sword-devel] Remote Module Repository Wiki
Hey guys. We have been in talks with a number of Bible publishers (and SBL is approaching in just over a week where we hope to again pitch our software to even more publishers) and always point there to our wiki for technical details of running their own module repository: http://crosswire.org/wiki/Creating_and_Maintaining_a_Module_Repository This page has a long history of edits by various individuals among whom I have only participated minimally. Is there any chance I can get some of you who have contributed to this valuable resource to clean it up a bit. I personally would like to see: its size reduced as small as possible to show how easy it is to run a remote repository. Initial, simple repository section greatly appreciated, but the rest is still a little cloudy for even me to understand. important technical typos fixed (like mods.d.tar.g[z] fixed). Proper English sentences (sorry non-native speakers. I do value your input). it organized appropriately, with talks about having .zip files handy for people in persecuted countries and for mobile users without builtin installers (do we have any of these?) either moved to another page, as this has nothing to do with running a remote repository, or else moved to the end in a section explicitly delineated as not part of running a remote repository-- again, trying to make this task look very easy. I'm not saying this isn't valuable information, but I want these organizations to buy in for as little work possible. I hope you understand how this benefits the pitch. Thanks tons for your support, Troy On 11/04/2010 10:55 PM, David Haslam wrote: In the KJV module, there are 17 occurences of the alternative method to mark translation changes. seg subType=x-added type=x-transChange.../segThis alternative method is a hack which seems to be required when the transChange text is within a w container. I assume that OSIS does not permit the recommended transChange.../transChange container element in such circumstances. Xiphos 3.1.3 displays these without italics. This is a bug. I have also observed that diatheke with RTF output_format does the same, so perhaps the bug is in the SWORD API. See also http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach I have also reported the bug in the Xiphos tracker. ID=3103244. David ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
I just wanted to add that... It is a huge part of our vision to have a broad number of publishers host their own repositories of material for use with our software. We do not seek out license deals with publishers like other Bible software vendors do. We offer the ability for publishers to make their resources available with minimal work on a vast number of platforms. This is our selling point: no fees for us, minimal effort for them to publish their own materials. We have spent countless hours developing our internal distributed installation mechanism and our frontend developers have done a great job exposing this beautifully to the end user, to work toward this becoming a reality. It's been a long time in the works and now is finally available in most all of our frontend applications. Our primary goal has always been to make the Gospel of Christ as available as possible to a lost world. Bible translations are personally my highest priority, but publishers-- both commercial and non-commercial-- have valuable resources for training the Body of Christ. I would also like to see these resources become available for our users. Just wanted to share my heart on this matter. Troy On 11/05/2010 12:25 AM, Troy A. Griffitts wrote: Hey guys. We have been in talks with a number of Bible publishers (and SBL is approaching in just over a week where we hope to again pitch our software to even more publishers) and always point there to our wiki for technical details of running their own module repository: http://crosswire.org/wiki/Creating_and_Maintaining_a_Module_Repository This page has a long history of edits by various individuals among whom I have only participated minimally. Is there any chance I can get some of you who have contributed to this valuable resource to clean it up a bit. I personally would like to see: its size reduced as small as possible to show how easy it is to run a remote repository. Initial, simple repository section greatly appreciated, but the rest is still a little cloudy for even me to understand. important technical typos fixed (like mods.d.tar.g[z] fixed). Proper English sentences (sorry non-native speakers. I do value your input). it organized appropriately, with talks about having .zip files handy for people in persecuted countries and for mobile users without builtin installers (do we have any of these?) either moved to another page, as this has nothing to do with running a remote repository, or else moved to the end in a section explicitly delineated as not part of running a remote repository-- again, trying to make this task look very easy. I'm not saying this isn't valuable information, but I want these organizations to buy in for as little work possible. I hope you understand how this benefits the pitch. Thanks tons for your support, Troy On 11/04/2010 10:55 PM, David Haslam wrote: In the KJV module, there are 17 occurences of the alternative method to mark translation changes. seg subType=x-added type=x-transChange.../segThis alternative method is a hack which seems to be required when the transChange text is within a w container. I assume that OSIS does not permit the recommended transChange.../transChange container element in such circumstances. Xiphos 3.1.3 displays these without italics. This is a bug. I have also observed that diatheke with RTF output_format does the same, so perhaps the bug is in the SWORD API. See also http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach I have also reported the bug in the Xiphos tracker. ID=3103244. David ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
I know this has been an area of disagreement in the past, and I'm not intending to start any sort of argument, but just share my experience. Among the businesses I work with (mostly small operations), there is almost no familiarity with FTP at all, and in many cases (in the larger ones), FTP is often blocked both incoming and outgoing. Many of these are Windows environments with IIS for a web server. Looking at the skills of these organizations and their IT departments, setting up an FTP server would not be the ideal choice of deployment. Especially when it comes to things like mods.tar.gz which require tools outside their normal tool range. A much more ideal repository format for these types of organizations, would be that used by jsword. It just requires a simple HTTP-accessible directory, in which a bunch of zip files are dropped. Ideally there would be one file with a list of what's available. Again, I'm not arguing what should be done, or even what is the most common setup, but in my experience the current requirements of setting up a repo with FTP would not be the ideal solution for people with little IT experience. And as we have discussed before, it is difficult to find good FTP hosting on the web, and challenging for users to set up. It is much simpler and cheaper to find HTTP hosting. I know we now can access repos over HTTP, but it seems to me quite fragile. It depends on Apache's directory listing, and even running it through Squid (or other proxy servers) completely breaks it. I haven't tried it, but I suspect using IIS would completely break it as well, as it has a different directory listing. So, our repo setup guidelines are not really very simple at all, but pre-suppose a wide variety of things like client and server must not be behind proxies, client and server must have FTP ports open, server must be running apache for HTTP access, and so on. I'm afraid these sort of requirements are going to cause difficulties in getting other organizations set up, if they have IT departments and environments anything like what I see daily. Matthew ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] Remote Module Repository Wiki
I do understand the very real practical concerns about FTP you point out, but we have practically received almost no support emails from users not being able to install because FTP was blocked. Most hosting services provide FTP access. The issue we ran into before was some didn't provide anonymous ftp access, which we've rectified by adding username / password fields in our repository registry and support for this in our installer, and in reality to install an FTP server on a windows box is trivial compared to the ongoing maintenance of assuring zips are created and placed in the correct folder, an index file for the zips is created and kept updated whenever anything changes, etc. It might seem trivial to us, and might actually be trivial for the publishers, in practice (and might eventually be how we end up going if we do find real roadblocks with our current FTP mechanism), but: It sure is easy to sell: just get your library of books installed and working in any of our frontends, and then point an FTP server to that installation and you are now publishing your own remote repository; drop that installation onto a CD or USB stick and you can use that media as an install source with any of our frontends; drop it to a network drive and you can use that as an install source for any of our frontends... (I realize the last 2 don't have anything to do with the FTP issue at hand, but it goes along with the general idea of making it really easy for anyone to share their library with others in most any setting) Or, you currently have 100+ of your Bibles formatted as SWORD modules you are publishing from your server via SWORDWeb. All you have to do is point an FTP server to this same data path and you are now publishing your 100+ Bibles for use in all of our software on any of our supported platforms: Windows, Linux, Mac, iPhone, Android, etc. These are real world cases right now. And the most important thing for us is to have a unified message to the publishers, that all of our software can install from a common repository format. Again, I'm not amiss to requiring zips and an index file if we find it necessary, but if it is not necessary, I would rather us do alot of extra work to make life even the slightest bit simpler for anyone who wishes to make their materials available. And also, as said to the JSword team, I'm not opposed to them having their own mechnism they feel is better, but I only ask that it is optional-- in the sense that, failing to locate their own mechanism on a remote repository, they fall back to the common FTP mechanism. Hope this communicates that I understand the concerns and have considered them and made what I hope we can at least agree is a good solution, if not the best in your mind. Troy .but the fact of the matter is that FTP is the Internet protocol for browsing a tree of files and for transferring those files On 11/05/2010 01:09 AM, Matthew Talbert wrote: I know this has been an area of disagreement in the past, and I'm not intending to start any sort of argument, but just share my experience. Among the businesses I work with (mostly small operations), there is almost no familiarity with FTP at all, and in many cases (in the larger ones), FTP is often blocked both incoming and outgoing. Many of these are Windows environments with IIS for a web server. Looking at the skills of these organizations and their IT departments, setting up an FTP server would not be the ideal choice of deployment. Especially when it comes to things like mods.tar.gz which require tools outside their normal tool range. A much more ideal repository format for these types of organizations, would be that used by jsword. It just requires a simple HTTP-accessible directory, in which a bunch of zip files are dropped. Ideally there would be one file with a list of what's available. Again, I'm not arguing what should be done, or even what is the most common setup, but in my experience the current requirements of setting up a repo with FTP would not be the ideal solution for people with little IT experience. And as we have discussed before, it is difficult to find good FTP hosting on the web, and challenging for users to set up. It is much simpler and cheaper to find HTTP hosting. I know we now can access repos over HTTP, but it seems to me quite fragile. It depends on Apache's directory listing, and even running it through Squid (or other proxy servers) completely breaks it. I haven't tried it, but I suspect using IIS would completely break it as well, as it has a different directory listing. So, our repo setup guidelines are not really very simple at all, but pre-suppose a wide variety of things like client and server must not be behind proxies, client and server must have FTP ports open, server must be running apache for HTTP access, and so on. I'm afraid these sort of requirements are going to cause difficulties in getting other organizations set up, if
