Re: [Xastir] torrents and xastir
On Sat, 2 Aug 2008, Jason KG4WSV wrote: Maybe the relatively low numbers of users actually means those of us at home on low bandwidth connections can still afford to seed these images, as there won't be all that much demand. If more folks realize this and are willing to seed, this torrent thing may work better for our needs than I would have guessed. I was waiting until we had more seeders, or at least until we had a torrent client on wetnet.net, then was going to put postings out on some other lists so the Windows folks could play. That'll bring a few more downloads I expect. No way to know how many until we try it. If/when the maps go to torrents we may get a lot more users, non-Xastir type of folks. Those will be different seeders though in general, so the current seeders don't have anything to worry about. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
although I didn't need it, I fired up the torrent to download the VM image last night. It was finished this morning, and as of now there are no peers connected. Maybe the relatively low numbers of users actually means those of us at home on low bandwidth connections can still afford to seed these images, as there won't be all that much demand. If more folks realize this and are willing to seed, this torrent thing may work better for our needs than I would have guessed. -Jason kg4wsv ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
http://northwest.aprs2.net/rivettracker/index.php This page is pretty cool. One can see if any downloads via Torrent are in progress, how many seeders there are, etc. I took one of my files down as a seeder and refreshed the page, and the number decremented by one, so it definitely sees me. ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, James Ewen wrote: If you have one seeder (has the whole file), and one leecher (wants the whole file), then that download will take a while. However, if you have 2 leechers, they could possibly get the file as fast as a single leecher. Once leecher #1 gets a segment from the seeder, leecher #2 can request that segment from leecher #1. As soon as anyone in the torrent has a segment, they can start sharing it with others in the torrent. You don't need to have the whole file before you can start adding to the transfer capability. So... 1000 people could get a file from one seeder at the same speed as one person could, assuming enough extra links and bandwidth among the 1000 people. That's just freaky. Cool, but freaky. Somebody's breaking some speed laws or some physics laws somewhere, hi hi. So, even if you don't have all the pieces, you are still contributing to the cause, by uploading segments that you do have to the others in the torrent that are in need of them. It's possible to be in a torrent, and download a whole file without ever getting a segment from a seeder. That's very cool. It's really a good way to share content, but only if people leechingmake sure they don't shut down their upload capabilities. Roger. I'm at 1.1 on my sharing of the most current file now. I've never been a seeder before, so that's a big event for me. Your latest image appears to be getting a workout. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, Aug 1, 2008 at 8:10 AM, Curt, WE7U <[EMAIL PROTECTED]> wrote: > I guess once two people have it and are seeding it > changes things, but that initial first download would take forever. Just a minor point, but it is a significant reason why torrents work so well... If you have one seeder (has the whole file), and one leecher (wants the whole file), then that download will take a while. However, if you have 2 leechers, they could possibly get the file as fast as a single leecher. Once leecher #1 gets a segment from the seeder, leecher #2 can request that segment from leecher #1. As soon as anyone in the torrent has a segment, they can start sharing it with others in the torrent. You don't need to have the whole file before you can start adding to the transfer capability. I have been in a torrent with no seeders, sharing a file for a couple weeks, slowly getting more bits and pieces as people came and went. It took a while, but when I finally got all the segments, I was able to seed that file. So, even if you don't have all the pieces, you are still contributing to the cause, by uploading segments that you do have to the others in the torrent that are in need of them. It's possible to be in a torrent, and download a whole file without ever getting a segment from a seeder. It's really a good way to share content, but only if people leechingmake sure they don't shut down their upload capabilities. James VE6SRV ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, Aug 01, 2008 at 11:12:41AM -0700, we recorded a bogon-computron collision of the <[EMAIL PROTECTED]> flavor, containing: > On Fri, 1 Aug 2008, Tom Russo wrote: > > > Not that it matters much, but USGS dropped the 1-degree by 1-degree > > distribution method years ago. Doesn't mean that it's a bad way to go, > > but it *isn't* the way USGS does it. > > > > Their "data partners" tend to do it by state now. > > Rgr. > > Are there additional restrictions now on the map data since it's > being distributed by someone other than the USGS? U.S. Taxpayers > paid for it once, just wondering how many times we'll again pay for > it. I'm not aware of any new restrictions. In fact, I found that buying state-wide "data bundles" from GeoCommunities (one of the USGS "data partners") was significantly cheaper than it used to be for a state's worth of DRGs direct from USGS (where they were $1.00 each plus a $30 set-up fee for the order once they stopped distributing the 1-by-1 degree bundles). -- Tom RussoKM5VY SAR502 DM64ux http://www.swcp.com/~russo/ Tijeras, NM QRPL#1592 K2#398 SOC#236 AHTB#1 http://kevan.org/brain.cgi?DDTNM "Argue for your limitations and sure enough, they're yours." -- R. Bach ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Tom Russo wrote: Not that it matters much, but USGS dropped the 1-degree by 1-degree distribution method years ago. Doesn't mean that it's a bad way to go, but it *isn't* the way USGS does it. Their "data partners" tend to do it by state now. Rgr. Are there additional restrictions now on the map data since it's being distributed by someone other than the USGS? U.S. Taxpayers paid for it once, just wondering how many times we'll again pay for it. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, Aug 01, 2008 at 10:30:17AM -0700, we recorded a bogon-computron collision of the <[EMAIL PROTECTED]> flavor, containing: > > >> If we did it in 1-degree by 1-degree sections the way that the USGS > >> distributes them, this whole scheme would be much more workable. We > >> might be able to get some of the DRG collection groups to contribute > >> data too 'cuz they'd be benefitting from ours. Not that it matters much, but USGS dropped the 1-degree by 1-degree distribution method years ago. Doesn't mean that it's a bad way to go, but it *isn't* the way USGS does it. Their "data partners" tend to do it by state now. -- Tom RussoKM5VY SAR502 DM64ux http://www.swcp.com/~russo/ Tijeras, NM QRPL#1592 K2#398 SOC#236 AHTB#1 http://kevan.org/brain.cgi?DDTNM "Argue for your limitations and sure enough, they're yours." -- R. Bach ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
> You posted that to a public mailing list with archives. You may > need to change that password at some point. I understand. That is why rather than naming the password I gave a hint. However foolishly, I'm banking on the stupidity of pirates. If there is a problem, it will be trivial to delete and change the password. >> There would be people out there with just a few of the CD's that >> would like to contribute. If we do it by state, as we're collecting >> that state and growing the collection on the server the torrent >> would change, making the distributed model less effective, right? We don't want to change a torrent. This ruins the model, because those who have the data can't continue to seed if the torrent changes without redownloading. >> If we did it in 1-degree by 1-degree sections the way that the USGS >> distributes them, this whole scheme would be much more workable. We >> might be able to get some of the DRG collection groups to contribute >> data too 'cuz they'd be benefitting from ours. >> Great, lets distribute by 1-degree by 1-degree sections then. On the Wiki, we can just have a heading for each State and list the multiple torrents that pertain to that state. That should make it simple enough to manage and choose downloads. > Can we do an entire tree of > directories? Yes. Yes, this is very useful. > The state torrents would change as the dataset grew, but the > 1x1-degree torrents would remain the same. Yuck, no, don't change torrents. > What's a good headless client for such purposes, one that can be > used for multiple torrents? Something that wouldn't take hours and > hours to set up. On northwest.aprs2.net I use rtorrent with screen. I'm very happy with this. The keyboard controls are daunting at first, but after reading the user guide (took about 10 min) I realized they are all very intuitive, so I haven't had to reference the guide again. > On Fri, 1 Aug 2008, Curt, WE7U wrote: Quit with the replies or I'll never finished this message! Tom KD7LXL ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Curt, WE7U wrote: Oh yea, I know what else has been nagging at me but I keep forgetting to ask: I saw in the docs that we can name a file or a directory when creating the torrent... Can we do an entire tree of directories? Also: If Gerry or someone were to make scripts to create the torrents automatically, which would be very cool, -and- if torrents can handle trees of files, there's no reason we couldn't have a set of torrents by state and another set of torrents by 1x1-degree slice. The state torrents would change as the dataset grew, but the 1x1-degree torrents would remain the same. And one more thing: We have disk space and bandwidth available on wetnet.net. We should be able to participate in the torrents for the VMWare images and for the map torrents. What's a good headless client for such purposes, one that can be used for multiple torrents? Something that wouldn't take hours and hours to set up. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Curt, WE7U wrote: There would be people out there with just a few of the CD's that would like to contribute. If we do it by state, as we're collecting that state and growing the collection on the server the torrent would change, making the distributed model less effective, right? If we did it in 1-degree by 1-degree sections the way that the USGS distributes them, this whole scheme would be much more workable. We might be able to get some of the DRG collection groups to contribute data too 'cuz they'd be benefitting from ours. My druthers are to collect just the DATA and the METADATA directories from the CD's/DVD's. The rest of the stuff on there is a small amount of docs and several very old software packages for viewing the geoTIFF files. We should distribute only the map data. Oh yea, I know what else has been nagging at me but I keep forgetting to ask: I saw in the docs that we can name a file or a directory when creating the torrent... Can we do an entire tree of directories? Also: If Gerry or someone were to make scripts to create the torrents automatically, which would be very cool, -and- if torrents can handle trees of files, there's no reason we couldn't have a set of torrents by state and another set of torrents by 1x1-degree slice. The state torrents would change as the dataset grew, but the 1x1-degree torrents would remain the same. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Tom Hayward wrote: Standard procedure is to create one .torrent per set. The reasoning behind this is that the downloader can instruct their client which of the files to download (with a multi-file torrent, there is nothing stopping you from only downloading a few of the files). With a data set as huge as "USGS topos", this might not be reasonable. Are the maps segmented geographically? I would suggest one torrent per state, each with geographically labeled directories (by county?) so users could identify what they needed to download. There would be people out there with just a few of the CD's that would like to contribute. If we do it by state, as we're collecting that state and growing the collection on the server the torrent would change, making the distributed model less effective, right? If we did it in 1-degree by 1-degree sections the way that the USGS distributes them, this whole scheme would be much more workable. We might be able to get some of the DRG collection groups to contribute data too 'cuz they'd be benefitting from ours. My druthers are to collect just the DATA and the METADATA directories from the CD's/DVD's. The rest of the stuff on there is a small amount of docs and several very old software packages for viewing the geoTIFF files. We should distribute only the map data. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Tom Hayward wrote: you must first upload the torrent to the tracker. To do this, use the username "blah" and the password "blah-blah". You posted that to a public mailing list with archives. You may need to change that password at some point. Standard procedure is to create one .torrent per set. The reasoning behind this is that the downloader can instruct their client which of the files to download (with a multi-file torrent, there is nothing stopping you from only downloading a few of the files). With a data set as huge as "USGS topos", this might not be reasonable. Are the maps segmented geographically? I would suggest one torrent per state, each with geographically labeled directories (by county?) so users could identify what they needed to download. They are distributed as a one or two-CD set, which is a 1-degree by 1-degree rectangle, containing all of the 1:24k, 1:100k, and 1:250k maps that cover that area, plus sometimes some other maps. There are also smaller files that go with the maps, plus other stuff on the disk that we really don't need to distribute. The stuff of interest is in the DATA and METADATA directories of the CD's. One more thing: If someone were to come up with an auto-torrent kind of thing for this, would we want to back up the scripts and the torrents to another server or two as well? That way one array going bad or site being temporarily unavailable doesn't mean the maps are unavailable. I don't know how that might play in the torrent scheme though: It might not be workable. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
> Imagine a site with a CONUS map (to start, and thinking small; we could > scale up) where you can use a bounding box to identify your region of > interest or cursor to select a particular point (map). After that selection > you see an inventory of different maps and types of maps available, and you > use a check-list to identify the ones you want. The site prepares a > separate page/Torrent stream to provide these, and the page is lightly > persistent (days before it ages out) and indexed on a page of recent > selections. > > You'd have the option of getting the data via download or Torrent at that > point. Simplified data delivery. > > I'm not sure how making delivery of all those maps via Torrent can be done > without some form of simplified selection, but then I'm not a big Torrent > user. I tend to get OS distro's and kernel updates that way, at home, > because it "just happens" and I don't have to worry much about it. This > doesn't make me an expert, though, as I got it working once and locked in > the format. This is a neat idea. I need to do the same thing with a repository of JPEG images (from a time-lapse sequence) at work. I haven't decided how to attack it yet. I need to distribute a collection of files, but I don't want to create an archive (zip/tar/rar) every time a collection is requested (mostly, because the data is in Amazon S3 and I'd have to move it before I could create an archive). Collections are divided by user query such as time-of-day with date bounds (i.e. every 10am shot during the month of July). Anyway, I don't think this takes advantage of the distributed model of BitTorrent, although you would see benefit from BT's simple distribution of multiple files. I think everyone would want different chunks, so every torrent the server created would only have one peer. The index of recent selections would help here, but I'm not sure by how much. Tom KD7LXL ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
> b) People make various maps available for download. For instance I > have three DVD's containing all of the USGS topos for WA that are > freely distributable. I also have a good portion of OR and a few > for ID/MT/HI. I'd love for these to be "out there" for other > people to use. I'm glad you suggested this. With all the BitTorrent buzz, I've been meaning to ask if you could post these. I still haven't got up to the UW Library to copy them myself. > c) These same people create a torrent file out of this data, upload > it to the Wiki, and edit the Wiki to make that torrent link show > up. Yes, just remember to add a step in the middle. Create the torrent, upload the .torrent file to the tracker, so the tracker acknowledges it, then post the link. Some trackers are open and don't require this action, but this attracts pirates who who want to host copyrighted material. To use http://northwest.aprs2.net/rivettracker/, you must first upload the torrent to the tracker. To do this, use the username 'upload' and my callsign for the password. > d) The first download ends up being a server/client relationship as > you described, but if at least a few of the people doing the > downloads leave their torrent client up and running, they become > a shared resource for the same file. More than likely if we put > maps up there we'll have non-Xastir people doing downloads as > well, maybe even becoming additional resources to download from. ...the beauty of a distributed protocol such as BitTorrent. > 1) Would we want to create a torrent for each file? For each CD or > DVD set? Standard procedure is to create one .torrent per set. The reasoning behind this is that the downloader can instruct their client which of the files to download (with a multi-file torrent, there is nothing stopping you from only downloading a few of the files). With a data set as huge as "USGS topos", this might not be reasonable. Are the maps segmented geographically? I would suggest one torrent per state, each with geographically labeled directories (by county?) so users could identify what they needed to download. > 3) Is the idea of distributing maps via torrent workable at all? Yes. Lets get to work. Tom KD7LXL ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Gerry Creager wrote: Not today, but in a week? Or does it have to be today? Nope. Now: Imagine the same kind of a setup as you describe but have it auto-create the torrent files and keep them around, plus post them on a web site. As maps get distributed from BIGSERVER over torrrent, the bandwidth required would go down over time assuming enough people became seeders. I think you just said what I said. or tried to say. Ok, so hit me over the head a few more times and I might get it. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
Curt, WE7U wrote: On Fri, 1 Aug 2008, Gerry Creager wrote: Imagine a site with a CONUS map (to start, and thinking small; we could scale up) where you can use a bounding box to identify your region of interest or cursor to select a particular point (map). After that selection you see an inventory of different maps and types of maps available, and you use a check-list to identify the ones you want. The site prepares a separate page/Torrent stream to provide these, and the page is lightly persistent (days before it ages out) and indexed on a page of recent selections. You'd have the option of getting the data via download or Torrent at that point. Simplified data delivery. In this case you don't gain any advantages of the peer-to-peer distributed transfer. You only gain the advantage of another method of server->client that may be easier to use at the client end. We've seen, in another endeavor I work around, that if one person's interested in a dataset, someone else is likely interested. So, creating that dataset, especially if it's big/bulky and takes some time, should happen as few times as possible. We'd need a server with the space for 100 DVD's worth of data for DRG's and another 100 for DOQQ's plus space for other types of maps, professionaly backed-up. Also need a big pipe 'cuz non-Xastir people will find it too. Not today, but in a week? Or does it have to be today? Now: Imagine the same kind of a setup as you describe but have it auto-create the torrent files and keep them around, plus post them on a web site. As maps get distributed from BIGSERVER over torrrent, the bandwidth required would go down over time assuming enough people became seeders. I think you just said what I said. or tried to say. Of course the reality is the server would become more popular over time, but the torrents might keep the total bandwidth used more under control. Less of an exponential rise anyway. correct -- Gerry Creager -- [EMAIL PROTECTED] Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, Aug 1, 2008 at 9:41 AM, Curt, WE7U <[EMAIL PROTECTED]> wrote: >> You'd have the option of getting the data via download or Torrent at that >> point. Simplified data delivery. > > In this case you don't gain any advantages of the peer-to-peer > distributed transfer. Not exactly - Gerry's selection tool and the download choices are orthogonal issues. The real question is are there enough users (specifically enough users seeding) to make torrents worth the trouble? I'm not enough of a torrent user to guess, but I do know that my client defaulted to leech mode, and I ran that way for a long time not knowing any better. Now I tend to download what I need, seed for a few days or weeks to "give back to the community", then turn it off. As far as the disk space goes, 200 DVDs at 5Gb each are ~= 1Tb, which you can get in a single disk for less than $300. Use a second 1Tb disk as the "backup" mirror, throw in a couple of external disk boxes, and you've got the storage on the cheap for a few hundred dollars. As we don't get a lot of money around here, I use such storage/backup systems for the low intensity and/or low budget projects. There are more failure points than with a proper storage array, but there's an order of magnitude or two less cost getting in the door. Of course, if someone has the budget for doing it right and the will to loan it, we should by all means take them up on it. (: -Jason kg4wsv ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
Excuse the top post, I'd be happy to use Disk space(lots)/network(gigabit) from here to help out, actually if anyone snagged the stuff from the server over IPv6 it would be even better ;) Cheers John On 1 Aug 2008, at 15:41, Curt, WE7U wrote: On Fri, 1 Aug 2008, Gerry Creager wrote: Imagine a site with a CONUS map (to start, and thinking small; we could scale up) where you can use a bounding box to identify your region of interest or cursor to select a particular point (map). After that selection you see an inventory of different maps and types of maps available, and you use a check-list to identify the ones you want. The site prepares a separate page/Torrent stream to provide these, and the page is lightly persistent (days before it ages out) and indexed on a page of recent selections. You'd have the option of getting the data via download or Torrent at that point. Simplified data delivery. In this case you don't gain any advantages of the peer-to-peer distributed transfer. You only gain the advantage of another method of server->client that may be easier to use at the client end. We'd need a server with the space for 100 DVD's worth of data for DRG's and another 100 for DOQQ's plus space for other types of maps, professionaly backed-up. Also need a big pipe 'cuz non-Xastir people will find it too. Now: Imagine the same kind of a setup as you describe but have it auto-create the torrent files and keep them around, plus post them on a web site. As maps get distributed from BIGSERVER over torrrent, the bandwidth required would go down over time assuming enough people became seeders. Of course the reality is the server would become more popular over time, but the torrents might keep the total bandwidth used more under control. Less of an exponential rise anyway. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir -- John Ronan <[EMAIL PROTECTED]>, +353-51-302938 Telecommunications Software & Systems Group, http://www.tssg.org ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Gerry Creager wrote: Imagine a site with a CONUS map (to start, and thinking small; we could scale up) where you can use a bounding box to identify your region of interest or cursor to select a particular point (map). After that selection you see an inventory of different maps and types of maps available, and you use a check-list to identify the ones you want. The site prepares a separate page/Torrent stream to provide these, and the page is lightly persistent (days before it ages out) and indexed on a page of recent selections. You'd have the option of getting the data via download or Torrent at that point. Simplified data delivery. In this case you don't gain any advantages of the peer-to-peer distributed transfer. You only gain the advantage of another method of server->client that may be easier to use at the client end. We'd need a server with the space for 100 DVD's worth of data for DRG's and another 100 for DOQQ's plus space for other types of maps, professionaly backed-up. Also need a big pipe 'cuz non-Xastir people will find it too. Now: Imagine the same kind of a setup as you describe but have it auto-create the torrent files and keep them around, plus post them on a web site. As maps get distributed from BIGSERVER over torrrent, the bandwidth required would go down over time assuming enough people became seeders. Of course the reality is the server would become more popular over time, but the torrents might keep the total bandwidth used more under control. Less of an exponential rise anyway. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
Curt, WE7U wrote: On Fri, 1 Aug 2008, Jason KG4WSV wrote: ...I'm trying to understand if and how we (xastir and APRS users) can benefit from the distribution system in the sharing of our sometimes large chunks of data, like maps and VM images. The basic premise/assumption is that many people will want to download and many of those will make the data once they have downloaded, right? If no one peering, there is nothing to download? I guess it degrades to a traditional master-client download if there's only 1 "peer". I'm trying to figure out if it's worth the effort, given our relatively small numbers, to make things like VM images, TIGER shapefiles, DRGs, DOQQs, etc available via bit torrent. In the recent past I said I didn't know anything about torrent. That was somewhat false, as I've been using it for a few years but only in the "leecher" mode. When it got to "seeding" I always shut it down. Now I see a reason to leave it in seeding mode and have done so. I basically understand how torrent works and am getting used to some of the special lingo for it. I don't yet know if the below scheme would work but I'll put it out there. Someone more familiar with it can shoot it down: a) Someone runs a web page that has multiple ".torrent" files on it, representing all the things we have available to download. This could be a Wiki page, right? We'd have to be careful who has edit access to it (user/password). We have one such page now with three items on it, but it's not a Wiki. This web page ends up with very little usage as the bulk transfers happen client-to-client, not from this web page. b) People make various maps available for download. For instance I have three DVD's containing all of the USGS topos for WA that are freely distributable. I also have a good portion of OR and a few for ID/MT/HI. I'd love for these to be "out there" for other people to use. c) These same people create a torrent file out of this data, upload it to the Wiki, and edit the Wiki to make that torrent link show up. d) The first download ends up being a server/client relationship as you described, but if at least a few of the people doing the downloads leave their torrent client up and running, they become a shared resource for the same file. More than likely if we put maps up there we'll have non-Xastir people doing downloads as well, maybe even becoming additional resources to download from. e) The maps contributed in this manner must be freely distributable to preserve our good project name. Questions I have still: 1) Would we want to create a torrent for each file? For each CD or DVD set? In my case the DVD's were created from a LOT of CD's, and each CD had a LOT of maps on it. Of course each map is megabytes of data as well. I know I could create a torrent for an entire directory, but in the case of the DATA and METADATA directories for these CD's or especially DVD's the download would be huge. I guess once two people have it and are seeding it changes things, but that initial first download would take forever. 2) To make it easier on the map providers, one big download is the way to go. To make it easier on the "leechers" it might be better to have smaller increments. How does one decide this? I'm sure I could write a script that would make a torrent file out of each map file, but am not sure I could do the same to get the multiple thousands of torrents uploaded to a Wiki. 3) Is the idea of distributing maps via torrent workable at all? I've been looking for a method to make USGS DRG's and DOQQ's available for a number of years, and this seems the closest match to date. CAVEAT: I can't even START to think about this 'til I can extricate myself from a particular project, but what I'd like to see would be some variant on this... Imagine a site with a CONUS map (to start, and thinking small; we could scale up) where you can use a bounding box to identify your region of interest or cursor to select a particular point (map). After that selection you see an inventory of different maps and types of maps available, and you use a check-list to identify the ones you want. The site prepares a separate page/Torrent stream to provide these, and the page is lightly persistent (days before it ages out) and indexed on a page of recent selections. You'd have the option of getting the data via download or Torrent at that point. Simplified data delivery. I'm not sure how making delivery of all those maps via Torrent can be done without some form of simplified selection, but then I'm not a big Torrent user. I tend to get OS distro's and kernel updates that way, at home, because it "just happens" and I don't have to worry much about it. This doesn't make me an expert, though, as I got it working once and locked in the format. gc -- Gerry Creager -- [EMAIL PROTECTED] Texas Mesonet -- AAT
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Gerry Creager wrote: We have some folks who don't have a lot of bandwidth, so a Torrent might be useful to them to retrieve stuff, but that's speculation. Overall, repositories in the conventional sense might be better/easier to maintain. What about when one user has a whole lot of data but a limited pipe heading out? I think I have 800kbits up/3Mbits down. A torrent might be a good way to get the data out there, eventually. The sheer volume involved in CD's or DVD's containing images is daunting, i.e. DRG's and DOQQ's. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
On Fri, 1 Aug 2008, Jason KG4WSV wrote: ...I'm trying to understand if and how we (xastir and APRS users) can benefit from the distribution system in the sharing of our sometimes large chunks of data, like maps and VM images. The basic premise/assumption is that many people will want to download and many of those will make the data once they have downloaded, right? If no one peering, there is nothing to download? I guess it degrades to a traditional master-client download if there's only 1 "peer". I'm trying to figure out if it's worth the effort, given our relatively small numbers, to make things like VM images, TIGER shapefiles, DRGs, DOQQs, etc available via bit torrent. In the recent past I said I didn't know anything about torrent. That was somewhat false, as I've been using it for a few years but only in the "leecher" mode. When it got to "seeding" I always shut it down. Now I see a reason to leave it in seeding mode and have done so. I basically understand how torrent works and am getting used to some of the special lingo for it. I don't yet know if the below scheme would work but I'll put it out there. Someone more familiar with it can shoot it down: a) Someone runs a web page that has multiple ".torrent" files on it, representing all the things we have available to download. This could be a Wiki page, right? We'd have to be careful who has edit access to it (user/password). We have one such page now with three items on it, but it's not a Wiki. This web page ends up with very little usage as the bulk transfers happen client-to-client, not from this web page. b) People make various maps available for download. For instance I have three DVD's containing all of the USGS topos for WA that are freely distributable. I also have a good portion of OR and a few for ID/MT/HI. I'd love for these to be "out there" for other people to use. c) These same people create a torrent file out of this data, upload it to the Wiki, and edit the Wiki to make that torrent link show up. d) The first download ends up being a server/client relationship as you described, but if at least a few of the people doing the downloads leave their torrent client up and running, they become a shared resource for the same file. More than likely if we put maps up there we'll have non-Xastir people doing downloads as well, maybe even becoming additional resources to download from. e) The maps contributed in this manner must be freely distributable to preserve our good project name. Questions I have still: 1) Would we want to create a torrent for each file? For each CD or DVD set? In my case the DVD's were created from a LOT of CD's, and each CD had a LOT of maps on it. Of course each map is megabytes of data as well. I know I could create a torrent for an entire directory, but in the case of the DATA and METADATA directories for these CD's or especially DVD's the download would be huge. I guess once two people have it and are seeding it changes things, but that initial first download would take forever. 2) To make it easier on the map providers, one big download is the way to go. To make it easier on the "leechers" it might be better to have smaller increments. How does one decide this? I'm sure I could write a script that would make a torrent file out of each map file, but am not sure I could do the same to get the multiple thousands of torrents uploaded to a Wiki. 3) Is the idea of distributing maps via torrent workable at all? I've been looking for a method to make USGS DRG's and DOQQ's available for a number of years, and this seems the closest match to date. -- Curt, WE7U. archer at eskimo dot com http://www.eskimo.com/~archer Lotto: A tax on people who are bad at math. - unknown Windows: Microsoft's tax on computer illiterates. - WE7U. The world DOES revolve around me: I picked the coordinate system!" ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
Re: [Xastir] torrents and xastir
P2P excels when there's a lot of consumers, and you start with one server. If, however, each subscribers becomes a server (publisher) upon receipt, per-server bandwidth limitations, and system load, are mitigated. For some small value of convenience, too, a user can start a Torrent stream and come back to a whole file. When you're talking movies or songs... or TIGER datasets, this can be useful. We have some folks who don't have a lot of bandwidth, so a Torrent might be useful to them to retrieve stuff, but that's speculation. Overall, repositories in the conventional sense might be better/easier to maintain. Jason KG4WSV wrote: I'm not terribly familiar with bit torrent, so forgive me if these are novice questions, but I'm trying to understand if and how we (xastir and APRS users) can benefit from the distribution system in the sharing of our sometimes large chunks of data, like maps and VM images. The basic premise/assumption is that many people will want to download and many of those will make the data once they have downloaded, right? If no one peering, there is nothing to download? I guess it degrades to a traditional master-client download if there's only 1 "peer". I'm trying to figure out if it's worth the effort, given our relatively small numbers, to make things like VM images, TIGER shapefiles, DRGs, DOQQs, etc available via bit torrent. -- Gerry Creager -- [EMAIL PROTECTED] Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
[Xastir] torrents and xastir
I'm not terribly familiar with bit torrent, so forgive me if these are novice questions, but I'm trying to understand if and how we (xastir and APRS users) can benefit from the distribution system in the sharing of our sometimes large chunks of data, like maps and VM images. The basic premise/assumption is that many people will want to download and many of those will make the data once they have downloaded, right? If no one peering, there is nothing to download? I guess it degrades to a traditional master-client download if there's only 1 "peer". I'm trying to figure out if it's worth the effort, given our relatively small numbers, to make things like VM images, TIGER shapefiles, DRGs, DOQQs, etc available via bit torrent. -- -Jason kg4wsv ___ Xastir mailing list Xastir@xastir.org http://lists.xastir.org/cgi-bin/mailman/listinfo/xastir
