Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/19/2010 08:02 PM, f...@kokkinizita.net wrote: On Sun, Sep 19, 2010 at 03:22:27PM -0400, Matthew Gyurgyik wrote: I have updated the wiki to include some basic information on setting up a local mirror. I believe it provides enough information to help someone set up local mirror while still holding them accountable for a certain amount of knowledge. Please take a look at it and improve it where you see fit. Since I have no need for a local mirror I might be over looking something. http://wiki.archlinux.org/index.php/Local_Mirror Thanks. I know how to use rsync, set up a server (not even needed in my case) etc. The interesting info is what should be excluded. It may be a good idea to provide a full repo / top level directory list for a tier-1 mirror (or a link to it if already available on the wiki) so users can create a complete --exclude-from list from the start instead of incrementally adding to it when they discover they are downloading things they don't need. I know you can use rsync to get such a listing, but still. Ciao, I added this pointer: Take look at the top level directory of the mirror that you choose and *make sure* to exclude anything you do no not need The example for /path/to/exclude.txt includes uncommon/unneeded top level dirs (iso, staging, kde/gnome-unstable, etc...)
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sat, Sep 18, 2010 at 10:46:13PM -0400, Matthew Gyurgyik wrote: There is nothing preventing you from creating a local mirror. If you can't figure out how to create a local mirror using the resource available, you probably shouldn't be using arch. Now, there's a supportive answer if I ever heard one. That's the way to get more people interested in using arch. Most distros like to build up their presence and increase the numbers and usage. Obviously if everyone goes out there and attempts to build local mirrors and all, that would put a big drain on the arch package update process. I don't think many people are doubting that and maybe it should be discouraged however. But the withholding of technical knowledge with such arrogance is in poor taste if you ask me. Like others have been saying all along now, the original information was pulled and no technical explanation was ever offered for why it was wrong. Now because of all this secrecy (in appearance), I've increased my curiosity and may look into building a local mirror just so I know how to do it. Had the thing on the wiki site been corrected, I would have probably just read it and kept it in the back of my mind for a day when I would really need to do it.
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sun, Sep 19, 2010 at 08:45:05AM -0700, Steve Holmes wrote: On Sat, Sep 18, 2010 at 10:46:13PM -0400, Matthew Gyurgyik wrote: There is nothing preventing you from creating a local mirror. If you can't figure out how to create a local mirror using the resource available, you probably shouldn't be using arch. Now, there's a supportive answer if I ever heard one. That's the way to get more people interested in using arch. Most distros like to build up their presence and increase the numbers and usage. Obviously if everyone goes out there and attempts to build local mirrors and all, that would put a big drain on the arch package update process. I don't think many people are doubting that and maybe it should be discouraged however. But the withholding of technical knowledge with such arrogance is in poor taste if you ask me. Like others have been saying all along now, the original information was pulled and no technical explanation was ever offered for why it was wrong. Now because of all this secrecy (in appearance), I've increased my curiosity and may look into building a local mirror just so I know how to do it. Had the thing on the wiki site been corrected, I would have probably just read it and kept it in the back of my mind for a day when I would really need to do it. I couldn't agree more, on all points, and would have written almost the same if you hadn't beaten me at it. Ciao, -- FA There are three of them, and Alleline.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/19/2010 04:45 PM, Steve Holmes wrote: Most distros like to build up their presence and increase the numbers and usage. Obviously if everyone goes out there and attempts to build local mirrors and all, that would put a big drain on the arch package update process. I don't think many people are doubting that and maybe it should be discouraged however. But the withholding of technical knowledge with such arrogance is in poor taste if you ask me. Like others have been saying all along now, the original information was pulled and no technical explanation was ever offered for why it was wrong. Now because of all this secrecy (in appearance), I've increased my curiosity and may look into building a local mirror just so I know how to do it. Had the thing on the wiki site been corrected, I would have probably just read it and kept it in the back of my mind for a day when I would really need to do it. You have a point when you say that all this discussion will increase the interest in creating a local mirror (for the people that read the mailing list). Some people will think about it a bit more but never try it (too much hassle for something they don't need or use), the ones that really need it will either find a way to do it or have already created a local mirror. It seems to me that the major point here is the difference between the wiki sort of endorsing a method to do it (which may put an higher load on a mirror, which the maintainer needs to pay for) or the user to come up with a way to do it. I don't have any doubts that any half decent arch user can do so if he/she wishes/needs to do it (these are the users most likely to need it anyway), but having an example on the wiki that can possibly tax a mirror because the procedure is not the most appropriate and the user trying it just copies the procedures verbatim is just wrong. -- Mauro Santos
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/19/2010 11:45 AM, Steve Holmes wrote: On Sat, Sep 18, 2010 at 10:46:13PM -0400, Matthew Gyurgyik wrote: There is nothing preventing you from creating a local mirror. If you can't figure out how to create a local mirror using the resource available, you probably shouldn't be using arch. Now, there's a supportive answer if I ever heard one. That's the way to get more people interested in using arch. Most distros like to build up their presence and increase the numbers and usage. Obviously if everyone goes out there and attempts to build local mirrors and all, that would put a big drain on the arch package update process. I don't think many people are doubting that and maybe it should be discouraged however. But the withholding of technical knowledge with such arrogance is in poor taste if you ask me. Like others have been saying all along now, the original information was pulled and no technical explanation was ever offered for why it was wrong. Now because of all this secrecy (in appearance), I've increased my curiosity and may look into building a local mirror just so I know how to do it. Had the thing on the wiki site been corrected, I would have probably just read it and kept it in the back of my mind for a day when I would really need to do it. As I posted on the forum... How hard is it to run rsync and look at the man page for rsync? rsync is the *only* command that is needed to create a local mirror. We want to discourage this behavior as much as possible and it is really quite trivial to setup a mirror. Setup a local mirror 1. rsync to local dir (look at the developer's wiki for mirrors and the proper rysnc arguments) 2. Set up webserver to serve local dir (if on a lan) 3. Set local mirror url in mirrorlist 4. Alternatively use Server = file:///mnt/media/mirror/$repo/os/x86_64 in pacman.conf or mirrorlist That is all that has to be done. If one is going to be creating a local mirror, he/she should really have this basic knowledge.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 18/09/10 19:17, Stefan Erik Wilkens wrote: [...] Guus, I checked the current mirrorlist before replying, correct me if I'm wrong but I can't find any entry relating to the domain Nathan included in his mailing, nor is he listed as a tier1 on the developer's wiki. Maybe I've missed something painfully obvious here? -Stefan it's not *official* , when the new mirror scheme went into action I asked about it because from the docs it appeared that I needed to be an official mirror ad thus permission to sync from the Tier-1s, but I was told that that wasn't the case. I didn't and don't want to run an official mirror in the sense that the mirror appears in the mirrorlist but I run the mirror[1] for what I consider to be a useful purpose and until someone pisses me off or it becomes useless I will continue to run it. It doesn't serve any other mirrors and users aren't expected to use it as their daily mirror which is the reason I don't want it in any mirrorlist. [1] the mirror is at arm.konnichi.com, it primary idea is that it can be used to do a blanketed and hopefully seamless downgrade of all packages with `pacman -Suu` or downgrade to any single package. http://arm.konnichi.com/2010/09/19/ provides a mirror that behaves like any other mirror of the official repos except the packages are those synced on September 19th 2010. likewise http://arm.konnichi.com/2009/12/03/ is a mirror with files from December 3rd 2009. ARM syncs once per day so it can't cause much problems and as stated previously the daily average bandwidth is so low that if you're running a mirror then it's almost insignificant unless you have everyone syncing from you which is almost certainly not the case with the Tier-1s. Yesterday or Friday's sync took about 900MiB most of which was from sauerbraten, over 400MiB and some larger packages, today IIRC was less than 10 MiB.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 19/09/10 17:26, Matthew Gyurgyik wrote: [...] As I posted on the forum... How hard is it to run rsync and look at the man page for rsync? rsync is the *only* command that is needed to create a local mirror. We want to discourage this behavior as much as possible and it is really quite trivial to setup a mirror. Setup a local mirror 1. rsync to local dir (look at the developer's wiki for mirrors and the proper rysnc arguments) 2. Set up webserver to serve local dir (if on a lan) 3. Set local mirror url in mirrorlist 4. Alternatively use Server = file:///mnt/media/mirror/$repo/os/x86_64 in pacman.conf or mirrorlist That is all that has to be done. If one is going to be creating a local mirror, he/she should really have this basic knowledge. This elitist attitude is what pisses me off most about the Arch community but I must admit that you sir just took it to the next level. I'll answer your question anyway. It's pretty easy to create a local mirror. But in your haste to show off your holyness you forgot that the issue isn't about creating a mirror, it's about doing it properly without causing issues for upstream. Your idea about throwing an rsync command at is does things like sync the entire mirror(as-is recommended in the NewMirrors wiki) which I'm sure isn't what you actually want if you're going to create a local mirror and this will undoubtedly just waste bandwidth for the mirror, after-all, if it's the packages you want then why would you go and sync the ISOs or even the sources? Now, I've stated my personal use-case and I' sure other have similar and other use-cases for having a local mirror, so I guess you have no argument against it other than it's something else that isn't useful to you so should be available to anyone else... Now, If you think it's a good idea to keep trolling as opposed to go and read what the actual issues are then please continue you are free to do so.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 12:26 Sun 19 Sep , Matthew Gyurgyik wrote: On 09/19/2010 11:45 AM, Steve Holmes wrote: On Sat, Sep 18, 2010 at 10:46:13PM -0400, Matthew Gyurgyik wrote: There is nothing preventing you from creating a local mirror. If you can't figure out how to create a local mirror using the resource available, you probably shouldn't be using arch. Now, there's a supportive answer if I ever heard one. That's the way to get more people interested in using arch. Most distros like to build up their presence and increase the numbers and usage. Obviously if everyone goes out there and attempts to build local mirrors and all, that would put a big drain on the arch package update process. I don't think many people are doubting that and maybe it should be discouraged however. But the withholding of technical knowledge with such arrogance is in poor taste if you ask me. Like others have been saying all along now, the original information was pulled and no technical explanation was ever offered for why it was wrong. Now because of all this secrecy (in appearance), I've increased my curiosity and may look into building a local mirror just so I know how to do it. Had the thing on the wiki site been corrected, I would have probably just read it and kept it in the back of my mind for a day when I would really need to do it. As I posted on the forum... How hard is it to run rsync and look at the man page for rsync? rsync is the *only* command that is needed to create a local mirror. We want to discourage this behavior as much as possible and it is really quite trivial to setup a mirror. Setup a local mirror 1. rsync to local dir (look at the developer's wiki for mirrors and the proper rysnc arguments) 2. Set up webserver to serve local dir (if on a lan) 3. Set local mirror url in mirrorlist 4. Alternatively use Server = file:///mnt/media/mirror/$repo/os/x86_64 in pacman.conf or mirrorlist That is all that has to be done. If one is going to be creating a local mirror, he/she should really have this basic knowledge. One problem... how many servers with ENABLED rsync do you know? Where is list of them? Example? Of course. You cannot sync from archlinux.org. --
Re: [arch-general] 'Local mirror' page was removed from wiki
On 22:31 Sun 19 Sep , Fess wrote: On 12:26 Sun 19 Sep , Matthew Gyurgyik wrote: On 09/19/2010 11:45 AM, Steve Holmes wrote: On Sat, Sep 18, 2010 at 10:46:13PM -0400, Matthew Gyurgyik wrote: There is nothing preventing you from creating a local mirror. If you can't figure out how to create a local mirror using the resource available, you probably shouldn't be using arch. Now, there's a supportive answer if I ever heard one. That's the way to get more people interested in using arch. Most distros like to build up their presence and increase the numbers and usage. Obviously if everyone goes out there and attempts to build local mirrors and all, that would put a big drain on the arch package update process. I don't think many people are doubting that and maybe it should be discouraged however. But the withholding of technical knowledge with such arrogance is in poor taste if you ask me. Like others have been saying all along now, the original information was pulled and no technical explanation was ever offered for why it was wrong. Now because of all this secrecy (in appearance), I've increased my curiosity and may look into building a local mirror just so I know how to do it. Had the thing on the wiki site been corrected, I would have probably just read it and kept it in the back of my mind for a day when I would really need to do it. As I posted on the forum... How hard is it to run rsync and look at the man page for rsync? rsync is the *only* command that is needed to create a local mirror. We want to discourage this behavior as much as possible and it is really quite trivial to setup a mirror. Setup a local mirror 1. rsync to local dir (look at the developer's wiki for mirrors and the proper rysnc arguments) 2. Set up webserver to serve local dir (if on a lan) 3. Set local mirror url in mirrorlist 4. Alternatively use Server = file:///mnt/media/mirror/$repo/os/x86_64 in pacman.conf or mirrorlist That is all that has to be done. If one is going to be creating a local mirror, he/she should really have this basic knowledge. One problem... how many servers with ENABLED rsync do you know? Where is list of them? Example? Of course. You cannot sync from archlinux.org. -- Sorry, replying wrong post =) --
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/19/2010 02:02 PM, Nathan Wayde wrote: On 19/09/10 17:26, Matthew Gyurgyik wrote: [...] As I posted on the forum... How hard is it to run rsync and look at the man page for rsync? rsync is the *only* command that is needed to create a local mirror. We want to discourage this behavior as much as possible and it is really quite trivial to setup a mirror. Setup a local mirror 1. rsync to local dir (look at the developer's wiki for mirrors and the proper rysnc arguments) 2. Set up webserver to serve local dir (if on a lan) 3. Set local mirror url in mirrorlist 4. Alternatively use Server = file:///mnt/media/mirror/$repo/os/x86_64 in pacman.conf or mirrorlist That is all that has to be done. If one is going to be creating a local mirror, he/she should really have this basic knowledge. This elitist attitude is what pisses me off most about the Arch community but I must admit that you sir just took it to the next level. I'll answer your question anyway. It's pretty easy to create a local mirror. But in your haste to show off your holyness you forgot that the issue isn't about creating a mirror, it's about doing it properly without causing issues for upstream. Your idea about throwing an rsync command at is does things like sync the entire mirror(as-is recommended in the NewMirrors wiki) which I'm sure isn't what you actually want if you're going to create a local mirror and this will undoubtedly just waste bandwidth for the mirror, after-all, if it's the packages you want then why would you go and sync the ISOs or even the sources? Now, I've stated my personal use-case and I' sure other have similar and other use-cases for having a local mirror, so I guess you have no argument against it other than it's something else that isn't useful to you so should be available to anyone else... Now, If you think it's a good idea to keep trolling as opposed to go and read what the actual issues are then please continue you are free to do so. You can use the --exclude-from=/path/to/excludefile.txt rsync argument - it exclude directories that you don't need. I have updated the wiki to include some basic information on setting up a local mirror. I believe it provides enough information to help someone set up local mirror while still holding them accountable for a certain amount of knowledge. Please take a look at it and improve it where you see fit. Since I have no need for a local mirror I might be over looking something. http://wiki.archlinux.org/index.php/Local_Mirror If you don't like the attitude don't use arch. Arch isn't here to babysit you and hold your hand. This is truly what sets arch apart. The users who have been here for 4-5+ years know exactly what I'm talking about. However, I digress as this isn't the time or place to discuss this.
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sun, Sep 19, 2010 at 03:22:27PM -0400, Matthew Gyurgyik wrote: If you don't like the attitude don't use arch. Arch isn't here to babysit you and hold your hand. This is truly what sets arch apart. The users who have been here for 4-5+ years know exactly what I'm talking about. Not quite true! What sets Arch appart is the aspect of technical simplicity but yet powerful. I have to say what turned me on to Arch also what I found to be good write-ups on the Arch wiki on how to set up certain items for which I wasn't quite sure how to do. After all, we all started out as beginners at some point. Sure, Arch isn't a turn-key system like some others are but at the same time what stands out a good true community is when people help out and answer questions and head people in the right direction to RTFM most efficiently. The arogance demonstrated with the beginning line of this quoted message is an ice cold way to turn people away from a good powerful distro the first time they run into snags. I don't like the demonstrated attitude but I sure as hell am not gonna leave Arch. The arch way is better than that.
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sun, 2010-09-19 at 15:17 -0700, Steve Holmes wrote: On Sun, Sep 19, 2010 at 03:22:27PM -0400, Matthew Gyurgyik wrote: If you don't like the attitude don't use arch. Arch isn't here to babysit you and hold your hand. This is truly what sets arch apart. The users who have been here for 4-5+ years know exactly what I'm talking about. Not quite true! What sets Arch appart is the aspect of technical simplicity but yet powerful. I have to say what turned me on to Arch also what I found to be good write-ups on the Arch wiki on how to set up certain items for which I wasn't quite sure how to do. After all, we all started out as beginners at some point. Sure, Arch isn't a turn-key system like some others are but at the same time what stands out a good true community is when people help out and answer questions and head people in the right direction to RTFM most efficiently. The arogance demonstrated with the beginning line of this quoted message is an ice cold way to turn people away from a good powerful distro the first time they run into snags. I don't like the demonstrated attitude but I sure as hell am not gonna leave Arch. The arch way is better than that. Let's not turn this into a big debate over what the 'Arch Way' means. In the end its decided by the devs, ie. the ones actually doing the work, and all the rest of the noise is just trolling. Matthew posted up a wiki link, discussing the technical merits or otherwise of that is useful. Another bundle of hot air over arrogance and hand-holding just wastes the time of everyone else reading this list.
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sun, Sep 19, 2010 at 03:22:27PM -0400, Matthew Gyurgyik wrote: I have updated the wiki to include some basic information on setting up a local mirror. I believe it provides enough information to help someone set up local mirror while still holding them accountable for a certain amount of knowledge. Please take a look at it and improve it where you see fit. Since I have no need for a local mirror I might be over looking something. http://wiki.archlinux.org/index.php/Local_Mirror Thanks. I know how to use rsync, set up a server (not even needed in my case) etc. The interesting info is what should be excluded. It may be a good idea to provide a full repo / top level directory list for a tier-1 mirror (or a link to it if already available on the wiki) so users can create a complete --exclude-from list from the start instead of incrementally adding to it when they discover they are downloading things they don't need. I know you can use rsync to get such a listing, but still. Ciao, -- FA There are three of them, and Alleline.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 17-09-10 09:34, Stefan Erik Wilkens wrote: Well then the situation is rather simple, isn't it. If the amount of traffic a private local mirror generates for the official mirror is greater than the amount of traffic that any local clients (excluding any other purposes unrelated to the sync process of course) generate for this local mirror, then obviously something is very wrong. This should be fairly easy to determine and fairly easy to attack (sync less frequently, increase the amount of clients syncing from your local server or perhaps give up the local mirror completely). Stefan, i think you'd better read the post you responded to. It looks like Nathan's local mirror is actually an official mirror... If i understood correctly, he was just being curious what was supposed to be wrong with the mentioned script. ;) mvg, Guus
Re: [arch-general] 'Local mirror' page was removed from wiki
2010/9/18 Guus Snijders gsnijd...@gmail.com: On 17-09-10 09:34, Stefan Erik Wilkens wrote: Well then the situation is rather simple, isn't it. If the amount of traffic a private local mirror generates for the official mirror is greater than the amount of traffic that any local clients (excluding any other purposes unrelated to the sync process of course) generate for this local mirror, then obviously something is very wrong. This should be fairly easy to determine and fairly easy to attack (sync less frequently, increase the amount of clients syncing from your local server or perhaps give up the local mirror completely). Stefan, i think you'd better read the post you responded to. It looks like Nathan's local mirror is actually an official mirror... If i understood correctly, he was just being curious what was supposed to be wrong with the mentioned script. ;) mvg, Guus Guus, I checked the current mirrorlist before replying, correct me if I'm wrong but I can't find any entry relating to the domain Nathan included in his mailing, nor is he listed as a tier1 on the developer's wiki. Maybe I've missed something painfully obvious here? -Stefan -- msn: stefan_wilk...@hotmail.com e-mail: stefanwilk...@gmail.com blog: http://www.stefanwilkens.eu/ adres: Lipperkerkstraat 14 7511 DA Enschede
Re: [arch-general] 'Local mirror' page was removed from wiki
2010/9/18 Stefan Erik Wilkens stefanwilk...@gmail.com: 2010/9/18 Guus Snijders gsnijd...@gmail.com: On 17-09-10 09:34, Stefan Erik Wilkens wrote: Well then the situation is rather simple, isn't it. If the amount of traffic a private local mirror generates for the official mirror is greater than the amount of traffic that any local clients (excluding any other purposes unrelated to the sync process of course) generate for this local mirror, then obviously something is very wrong. This should be fairly easy to determine and fairly easy to attack (sync less frequently, increase the amount of clients syncing from your local server or perhaps give up the local mirror completely). Stefan, i think you'd better read the post you responded to. It looks like Nathan's local mirror is actually an official mirror... If i understood correctly, he was just being curious what was supposed to be wrong with the mentioned script. ;) I checked the current mirrorlist before replying, correct me if I'm wrong but I can't find any entry relating to the domain Nathan included in his mailing, nor is he listed as a tier1 on the developer's wiki. Maybe I've missed something painfully obvious here? yes i think he is specifically asking what was _technically_ wrong with the aforementioned script from the wiki. which seems an appropriate question, given the circumstances of removal; if the only reason was to discourage the creation of local mirrors... well, to me at least, that seems a poor reason. i sympathize with some of the reasons for creating localized mirrors. in particular, when the mirror is located on a removable device it is especially useful... i did this to perform off-site installations, in addition to installing packages i needed when a connection wasn't available (ie. Nathan's train example) could a script be created that simply spreads the load to _all_ the official mirrors? the DB files could be pulled from archlinux.org, but the packages could be retrieved from all mirrors in parallel... or what is the core problem here? C Anthony
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sat, Sep 18, 2010 at 02:50:17PM -0500, C Anthony Risinger wrote: which seems an appropriate question, given the circumstances of removal; if the only reason was to discourage the creation of local mirrors... well, to me at least, that seems a poor reason. i sympathize with some of the reasons for creating localized mirrors. in particular, when the mirror is located on a removable device it is especially useful... i did this to perform off-site installations, in addition to installing packages i needed when a connection wasn't available (ie. Nathan's train example) I'd agree. I have to maintain 7 and soon 11 machines which do not have have any internet access at all. In the past I've been able to get away with taking them home for installation/updates, but that will not be possible anymore. So what is the solution for this if you can't have a local mirror on a portable device ? Ciao, -- FA There are three of them, and Alleline.
Re: [arch-general] 'Local mirror' page was removed from wiki
On Sat, 2010-09-18 at 14:50 -0500, C Anthony Risinger wrote: could a script be created that simply spreads the load to _all_ the official mirrors? the DB files could be pulled from archlinux.org, but the packages could be retrieved from all mirrors in parallel... Archlinux.org is just another mirror, not the 'primary' mirror, I believe.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/18/2010 05:44 PM, f...@kokkinizita.net wrote: On Sat, Sep 18, 2010 at 02:50:17PM -0500, C Anthony Risinger wrote: which seems an appropriate question, given the circumstances of removal; if the only reason was to discourage the creation of local mirrors... well, to me at least, that seems a poor reason. i sympathize with some of the reasons for creating localized mirrors. in particular, when the mirror is located on a removable device it is especially useful... i did this to perform off-site installations, in addition to installing packages i needed when a connection wasn't available (ie. Nathan's train example) I'd agree. I have to maintain 7 and soon 11 machines which do not have have any internet access at all. In the past I've been able to get away with taking them home for installation/updates, but that will not be possible anymore. So what is the solution for this if you can't have a local mirror on a portable device ? Ciao, There is nothing preventing you from creating a local mirror. If you can't figure out how to create a local mirror using the resource available, you probably shouldn't be using arch.
Re: [arch-general] 'Local mirror' page was removed from wiki
Well then the situation is rather simple, isn't it. If the amount of traffic a private local mirror generates for the official mirror is greater than the amount of traffic that any local clients (excluding any other purposes unrelated to the sync process of course) generate for this local mirror, then obviously something is very wrong. This should be fairly easy to determine and fairly easy to attack (sync less frequently, increase the amount of clients syncing from your local server or perhaps give up the local mirror completely). This isn't a matter of politics at all, it's plain and simple math where the outcome should be a situation where we create as little as possible traffic for the official mirrors. It doesn't matter if one is the only one doing it or if traffic is spread around well, it's a matter of plain decency: You're offered a free service which has to be shared with many people, and may cease to be a free service if use of this free service outgrows its capabilities. of course, what you do with your own traffic is yours to determine. And in a way we have little say over what people do with the official server's bandwidth. But a certain amount of good practice is expected, which is what I believe this discussion is about. Thus I hope we're all able to keep this discussion general, not fall in personal attacks. -Stefan 2010/9/17 Nathan Wayde disposa...@konnichi.com: On 17/09/10 00:21, Ng Oon-Ee wrote: On Thu, 2010-09-16 at 16:16 -0700, Steve Holmes wrote: On Thu, Sep 16, 2010 at 09:54:16PM +0200, Stefan Erik Wilkens wrote: On 16/09/10 19:39, Matthew Gyurgyik wrote: you don't get to tell anyone how to use their bandwidth. But can we at least say that grabbing packages without using them is wasting mirror bandwidth, and thus not something we want. In fact, something that should be frowned upon? It sounds like this Nathan fella doesn't grasp the concept that he pays for his own bandwidth and the mirror operator has to pay for the bandwidth used by the mirror server. Sure Nathan can squander his bandwidth however he wants but the mirror operators have to spread their bandwidth around for all of us to get our normal updates and of course, the cost has to be shouldered by someone. While my initial reaction to the thread was to do exactly this (point out that Nathan does not seem to understand that mirrors have to pay for bandwidth as well, and also that the linked article was obviously not read) I think its a bit out-of-line to dismiss him as this 'Nathan fella'. Where I come from such terms would only be used on a brat or delinquent, slightly derogatory in my opinion. Not a comment on the CONTENT but on the STYLE =). like I said, I'd deleted my reply but here I'll state it again... I already run a mirror for other purposes, if the Tier-1 has a problem with that then they should block arm.konnichi.com . since I already run that for other purposes I sync from arm.konnichi.com - in-case you didn't realise I own it. I'm a idiot, that much is no secret, so maybe someone can enlighten me... like I said already, I already run a mirror for other purposes, if I want to waste the bandwidth of my mirror that's my business because I already paid for it, but the important question is the one that's always ignored in favour of petty politics and that is: I want to know specifically what was wrong with that script so I as a mirror operator can take the necessary measures to make sure I'm not abusing the Tier-1 from which I sync. -- msn: stefan_wilk...@hotmail.com e-mail: stefanwilk...@gmail.com blog: http://www.stefanwilkens.eu/ adres: Lipperkerkstraat 14 7511 DA Enschede
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/15/2010 12:20 AM, Fess wrote: On 19:13 Tue 14 Sep , C Anthony Risinger wrote: On Tue, Sep 14, 2010 at 2:27 PM, Nathan Waydedisposa...@konnichi.com wrote: here's what I'd(and I imagine most others who know about sharing the cache) use a local mirror for: to be able to sync all other systems from it. plain and simple. if my systems don't have internet connection or something like that then i simply get the packages from the master, cache sharing doesn't and cannot solve that problem at all, that's a fact. shared cache won't solve that sure... but there are better solutions: ) if you can get it from master, then it sounds like you have a LAN connection; tunnel a connection thru master... ) if you have a LAN, what can't some machines have access anyway? ) if you don't have a LAN, you are manually moving packages? you could do that without a local mirror ) if you have a LAN, but _cannot_ allow some access to the net, then use a different method like a caching proxy local mirror = quick/easy crutch to avoid better utilization of local/peer resources i use a homebrew proxy/cache solution for my home, works fine. one machine pretends to be a repo, others look to it for packages... easy. i'm not using this exact version now, but i implemented this (rather crappily) while first learning python: pacproxy (or something that vaguely resembles an apt-proxy clone) https://bbs.archlinux.org/viewtopic.php?id=87115 now to the bandwidth issue. it's obviously bogus, because: 1) they assume everyone/(lots of people) is going to create a local mirror. 2) they assume that they're all going to sync from the same server. 3) they assume this extra bandwidth waste actually causes a problem for all the mirrors - i.e that there's only 1 mirror. now, if my assumptions are wrong thus leading to false conclusions then please correct me, but so far all I've heard is whining about local mirror causing problems for the mirrors but nothing about what these problems actually are, in the meantime the original wiki was deemed bad with not much of a valid reason and nothing being done to further educate us the users. i don't think it's even about whether or not it _is_ causing a problem, and more a preemptive move to discourage naive implementations. sure, if you have a heterogeneous environment of 200 machines, then a local mirror probably isn't too bad an idea... but it still isn't needed, as faster/better/cheaper methods are available. in my opinion, if you're not publicly seeding your mirror, then you don't need it; else you probably only want it due to an extreme case of laziness. sure maybe mirror XYZ can handle constant sync's from everyone looking at it... but really, do them a favor, and don't; it might piss them off :-). You can probably tell that I'm annoyed by this and the simple fact is that ARM sync script was based off the script on that wiki, it's not the same as I changed a lot of options to cater to my own needs but as have been said the script was bad, no-one is telling us what was bad about it and these alternative methods are wholly inadequate at best. yeah i don't really know the politics here, or have even seen the script. in my own experience back in the day syncing ubuntu repos (for easy install at remote locations from large USB key when client requirements are unknown)... you likely flat out don't need it, and there are _very_ few legitimate use cases for it (the parenthesized use case above is about the best one i know). all i'm suggesting is that just because you can and it's easy doesn't mean you should. but hey, i don't run a mirror, and extreme leeching won't affect me, so ultimately i could care less; if i did though, i would monitor for this kind of crap... i mean, doesn't the official arch mirror impose similar restrictions? just do you part to not be excessive. does one check out the entire library on the possibility of reading 10 books? C Anthony I think, i know(and others, who use this method) better what i'm doing, and why i am doing it. So, i tell you once more - community think, that this is useful. People, who say Hey, man! I have server, and rsync installed, add me please to the list of 3rd party mirrors know what they do. If they offer this service - they think it helps. If they would have 'tiny pipe'(or something else tiny) they wouldn't do it. So, i still don't understand why opinion of community ignored. Ok a few things here 1. There are a *few* instances where having a local mirror is warranted 2. There are many, many, many packages that are in the repos that *you* don't use! Every time you download one of these packages it is wasted bandwidth! 3. Mirror bandwidth is not free! Every time you are downloading unused packages you are wasting the mirrors money! Why waste money? (Keep point 1 in mind) 4. @Fess you and a few other people do not make the community. 5. The majority of the community will agree that hosting a local mirror is silly
Re: [arch-general] 'Local mirror' page was removed from wiki
On 16/09/10 19:39, Matthew Gyurgyik wrote: [..] Ok a few things here 1. There are a *few* instances where having a local mirror is warranted not sure where you were going with that but i feel like you've left a bit off of that sentence. 2. There are many, many, many packages that are in the repos that *you* don't use! Every time you download one of these packages it is wasted bandwidth! you don't get to tell anyone how to use their bandwidth. 3. Mirror bandwidth is not free! Every time you are downloading unused packages you are wasting the mirrors money! Why waste money? (Keep point 1 in mind) since i already payed for that bandwidth and utilize it for other purposes it is in fact free. 4. @Fess you and a few other people do not make the community. not sure what point you're trying to make here 5. The majority of the community will agree that hosting a local mirror is silly considering that there are alternatives! the majority of people at least in the western cultures will agree that paedophilia is sick. the majority of these people don't know what paedophilia is. again not sure where you're going with that so i thought I'd make some wild pointless claims as well. 6. I am quite sure that mirror operators are not and will not be happy with users downloading gigs of data a month so they can have their own local mirror. when you become a mirror operator or bring actual evidence to the table you will have a say in this. 7. Remember, the local mirrors are generously mirroring for us. They are under *no obligation* to do so! Treat them with respect! this doesn't make any sense. 8. If point 1 applies, then those people should be more than capable of producing their own scripts. we are. but you see, the point you decided to side-step is that we're being told that the existing script was bad, now, if it was bad fair enough but no-one can(or will) tell us what was so wrong about it; result: you're now forcing everyone that needs to create their own script to do so and thus risk causing the same problems the old script cause because as I've stated multiple times - no-one is telling us(me) what the problems are with that script. it's all well and good to say this or that is bad but if you're not going to tell anyone what's bad about it then we'd all be better off if you hadn't opened your mouth at all.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/16/2010 02:59 PM, Nathan Wayde wrote: On 16/09/10 19:39, Matthew Gyurgyik wrote: [..] Ok a few things here 1. There are a *few* instances where having a local mirror is warranted not sure where you were going with that but i feel like you've left a bit off of that sentence. 2. There are many, many, many packages that are in the repos that *you* don't use! Every time you download one of these packages it is wasted bandwidth! you don't get to tell anyone how to use their bandwidth. 3. Mirror bandwidth is not free! Every time you are downloading unused packages you are wasting the mirrors money! Why waste money? (Keep point 1 in mind) since i already payed for that bandwidth and utilize it for other purposes it is in fact free. You most certainly do not pay for the Mirror's bandwidth! Just look at this article: http://lwn.net/Articles/178618/ 4. @Fess you and a few other people do not make the community. not sure what point you're trying to make here 5. The majority of the community will agree that hosting a local mirror is silly considering that there are alternatives! the majority of people at least in the western cultures will agree that paedophilia is sick. the majority of these people don't know what paedophilia is. again not sure where you're going with that so i thought I'd make some wild pointless claims as well. 6. I am quite sure that mirror operators are not and will not be happy with users downloading gigs of data a month so they can have their own local mirror. when you become a mirror operator or bring actual evidence to the table you will have a say in this. Again look here: http://lwn.net/Articles/178618/ or ask any admin in charge of bandwidth operations. Aaron if you are reading this, would mind sharing the bandwidth cost for the arch servers? 7. Remember, the local mirrors are generously mirroring for us. They are under *no obligation* to do so! Treat them with respect! this doesn't make any sense. 8. If point 1 applies, then those people should be more than capable of producing their own scripts. we are. but you see, the point you decided to side-step is that we're being told that the existing script was bad, now, if it was bad fair enough but no-one can(or will) tell us what was so wrong about it; result: you're now forcing everyone that needs to create their own script to do so and thus risk causing the same problems the old script cause because as I've stated multiple times - no-one is telling us(me) what the problems are with that script. it's all well and good to say this or that is bad but if you're not going to tell anyone what's bad about it then we'd all be better off if you hadn't opened your mouth at all.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 16/09/10 20:10, Matthew Gyurgyik wrote: On 09/16/2010 02:59 PM, Nathan Wayde wrote: On 16/09/10 19:39, Matthew Gyurgyik wrote: [..] Ok a few things here 1. There are a *few* instances where having a local mirror is warranted not sure where you were going with that but i feel like you've left a bit off of that sentence. 2. There are many, many, many packages that are in the repos that *you* don't use! Every time you download one of these packages it is wasted bandwidth! you don't get to tell anyone how to use their bandwidth. 3. Mirror bandwidth is not free! Every time you are downloading unused packages you are wasting the mirrors money! Why waste money? (Keep point 1 in mind) since i already payed for that bandwidth and utilize it for other purposes it is in fact free. You most certainly do not pay for the Mirror's bandwidth! Just look at this article: http://lwn.net/Articles/178618/ my contract said I paid for it... [...] 6. I am quite sure that mirror operators are not and will not be happy with users downloading gigs of data a month so they can have their own local mirror. when you become a mirror operator or bring actual evidence to the table you will have a say in this. Again look here: http://lwn.net/Articles/178618/ or ask any admin in charge of bandwidth operations. Aaron if you are reading this, would mind sharing the bandwidth cost for the arch servers? [...] At first I typed out a reply but I deleted it because this thread is already dead so I will simply restate my question that no-one has an answer to. What were the issues with that wiki page and the script. I'd like to know so I don't cause these *problems* for Tier-1 mirrors I sync from as I have to implement my own script which is based on the bad script that was in the wiki.
Re: [arch-general] 'Local mirror' page was removed from wiki
You most certainly do not pay for the Mirror's bandwidth! Just look at this article: http://lwn.net/Articles/178618/ my contract said I paid for it... What? Sure, you may pay for n GB of download, but the mirror still has to pay for n GB of _upload_ in order to serve it to you. -- Tavian Barnes
Re: [arch-general] 'Local mirror' page was removed from wiki
On 16/09/10 19:39, Matthew Gyurgyik wrote: you don't get to tell anyone how to use their bandwidth. But can we at least say that grabbing packages without using them is wasting mirror bandwidth, and thus not something we want. In fact, something that should be frowned upon?
Re: [arch-general] 'Local mirror' page was removed from wiki
On Thu, 2010-09-16 at 16:16 -0700, Steve Holmes wrote: On Thu, Sep 16, 2010 at 09:54:16PM +0200, Stefan Erik Wilkens wrote: On 16/09/10 19:39, Matthew Gyurgyik wrote: you don't get to tell anyone how to use their bandwidth. But can we at least say that grabbing packages without using them is wasting mirror bandwidth, and thus not something we want. In fact, something that should be frowned upon? It sounds like this Nathan fella doesn't grasp the concept that he pays for his own bandwidth and the mirror operator has to pay for the bandwidth used by the mirror server. Sure Nathan can squander his bandwidth however he wants but the mirror operators have to spread their bandwidth around for all of us to get our normal updates and of course, the cost has to be shouldered by someone. While my initial reaction to the thread was to do exactly this (point out that Nathan does not seem to understand that mirrors have to pay for bandwidth as well, and also that the linked article was obviously not read) I think its a bit out-of-line to dismiss him as this 'Nathan fella'. Where I come from such terms would only be used on a brat or delinquent, slightly derogatory in my opinion. Not a comment on the CONTENT but on the STYLE =).
Re: [arch-general] 'Local mirror' page was removed from wiki
On 17/09/10 00:21, Ng Oon-Ee wrote: On Thu, 2010-09-16 at 16:16 -0700, Steve Holmes wrote: On Thu, Sep 16, 2010 at 09:54:16PM +0200, Stefan Erik Wilkens wrote: On 16/09/10 19:39, Matthew Gyurgyik wrote: you don't get to tell anyone how to use their bandwidth. But can we at least say that grabbing packages without using them is wasting mirror bandwidth, and thus not something we want. In fact, something that should be frowned upon? It sounds like this Nathan fella doesn't grasp the concept that he pays for his own bandwidth and the mirror operator has to pay for the bandwidth used by the mirror server. Sure Nathan can squander his bandwidth however he wants but the mirror operators have to spread their bandwidth around for all of us to get our normal updates and of course, the cost has to be shouldered by someone. While my initial reaction to the thread was to do exactly this (point out that Nathan does not seem to understand that mirrors have to pay for bandwidth as well, and also that the linked article was obviously not read) I think its a bit out-of-line to dismiss him as this 'Nathan fella'. Where I come from such terms would only be used on a brat or delinquent, slightly derogatory in my opinion. Not a comment on the CONTENT but on the STYLE =). like I said, I'd deleted my reply but here I'll state it again... I already run a mirror for other purposes, if the Tier-1 has a problem with that then they should block arm.konnichi.com . since I already run that for other purposes I sync from arm.konnichi.com - in-case you didn't realise I own it. I'm a idiot, that much is no secret, so maybe someone can enlighten me... like I said already, I already run a mirror for other purposes, if I want to waste the bandwidth of my mirror that's my business because I already paid for it, but the important question is the one that's always ignored in favour of petty politics and that is: I want to know specifically what was wrong with that script so I as a mirror operator can take the necessary measures to make sure I'm not abusing the Tier-1 from which I sync.
Re: [arch-general] 'Local mirror' page was removed from wiki
On 04:00 Sun 12 Sep , David C. Rankin wrote: On 09/09/2010 07:40 AM, Fess wrote: Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. I looked into an Arch local 'mirror/repo' when I started using Arch, similar to what I used to maintain for my SuSE installs. (the local mirror page was a mess and wrong then as well) However, since Arch will check for the presence of packages in /var/cache/pacman/pkg before/(instead of) re-downloading the packages, for local network boxes, it was just easier to use rsync to grab the packages from whatever box did the most recent update and transfer then to the nextbox you need to update (same architecture only). The only caveat is you need a way to manage duplicate (old packages) in /var/cache/pacman/pkg so I came up with a script that does that. So instead of worrying about a 'local mirror', just: (1) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman (2) grab the following 2 scripts: (the wrapper script that calls the main script [twice - see why below]) http://www.3111skyline.com/dl/Archlinux/scripts/fduparch.sh (the main duplicate identification and removal script) http://www.3111skyline.com/dl/Archlinux/scripts/fduppkg Put them both in /usr/local/bin (or link them there), edit fduparch.sh and change the directories you want the duplicates from /var/cache/pacman/pkg moved to (default is /home/backup/pkg-old and for the second pass to /home/backup/pkg-older). Then after updating one box, just call fduparch.sh (the wrapper script) as root and the duplicates in /var/cache/pacman/pkg are moved as follows: Pass 1: /var/cache/pacman/pkg = /home/backup/pkg-old Pass 2: /home/backup/pkg-old = /home/backup/pkg-older Which leaves you with the current set of packages in: /var/cache/pacman/pkg The last used packages before update in: /home/backup/pkg-old And finally all older packages in: /home/backup/pkg-older which can be deleted or archived. (of course you can delete the packages in /home/backup/pkg-old if you like as well) NOTE: the duplicate removal script uses the file ctime and/or mtime to determine which is the newer package, so when copying machine to machine with rsync, make sure you preserve the file attributes. (rsync's -a option works fine). Once you set this up, maintaining a local set of packages is a breeze, just: (1) pacman -Syu as normal on one box (works the same for new installs too) (2) fduparch.sh (as root to remove the older versions of new packages) (3) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman (only for boxes of the same architecture of course) (4) pacman -Syu on the box you rsync'ed the packages to (nextbox) (5) call fduparch.sh on the box you just updated (nextbox), and so on, and so on... As long as your boxes have reasonably similar packages installed, this eliminates 90+% of re-downloading, takes no additional bandwidth because you have to download the updated packages once anyway, and is simple enough for me to manage so you guy will have no problems with it :p And if you wanted to get even more creative, you could simply write a script on your main box that you update to automate the entire process for all your local boxes using nothing more than rsync and ssh calls of run the updates and dup removal scripts. (I haven't been that creative yet) Just remember to separate your boxes into groups/classes by architecture (i686 x86_64) Cheers. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com err.. what? Don't you think, that mirrorsync in crontab is MUCH more easier way to get new packages, huh? I'm using local mirror for 3 years and i have no idea why i shoulnd't use it know. P.S. If you have local mirror, you won't worry about 3-4 another GB's. You just download some less porn. Less porn - more profit. --
Re: [arch-general] 'Local mirror' page was removed from wiki
On Tue, Sep 14, 2010 at 7:13 PM, Fess killall_hum...@lavabit.com wrote: On 04:00 Sun 12 Sep , David C. Rankin wrote: On 09/09/2010 07:40 AM, Fess wrote: Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. I looked into an Arch local 'mirror/repo' when I started using Arch, similar to what I used to maintain for my SuSE installs. (the local mirror page was a mess and wrong then as well) However, since Arch will check for the presence of packages in /var/cache/pacman/pkg before/(instead of) re-downloading the packages, for local network boxes, it was just easier to use rsync to grab the packages from whatever box did the most recent update and transfer then to the nextbox you need to update (same architecture only). The only caveat is you need a way to manage duplicate (old packages) in /var/cache/pacman/pkg so I came up with a script that does that. So instead of worrying about a 'local mirror', just: (1) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman (2) grab the following 2 scripts: (the wrapper script that calls the main script [twice - see why below]) http://www.3111skyline.com/dl/Archlinux/scripts/fduparch.sh (the main duplicate identification and removal script) http://www.3111skyline.com/dl/Archlinux/scripts/fduppkg Put them both in /usr/local/bin (or link them there), edit fduparch.sh and change the directories you want the duplicates from /var/cache/pacman/pkg moved to (default is /home/backup/pkg-old and for the second pass to /home/backup/pkg-older). Then after updating one box, just call fduparch.sh (the wrapper script) as root and the duplicates in /var/cache/pacman/pkg are moved as follows: Pass 1: /var/cache/pacman/pkg = /home/backup/pkg-old Pass 2: /home/backup/pkg-old = /home/backup/pkg-older Which leaves you with the current set of packages in: /var/cache/pacman/pkg The last used packages before update in: /home/backup/pkg-old And finally all older packages in: /home/backup/pkg-older which can be deleted or archived. (of course you can delete the packages in /home/backup/pkg-old if you like as well) NOTE: the duplicate removal script uses the file ctime and/or mtime to determine which is the newer package, so when copying machine to machine with rsync, make sure you preserve the file attributes. (rsync's -a option works fine). Once you set this up, maintaining a local set of packages is a breeze, just: (1) pacman -Syu as normal on one box (works the same for new installs too) (2) fduparch.sh (as root to remove the older versions of new packages) (3) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman (only for boxes of the same architecture of course) (4) pacman -Syu on the box you rsync'ed the packages to (nextbox) (5) call fduparch.sh on the box you just updated (nextbox), and so on, and so on... As long as your boxes have reasonably similar packages installed, this eliminates 90+% of re-downloading, takes no additional bandwidth because you have to download the updated packages once anyway, and is simple enough for me to manage so you guy will have no problems with it :p And if you wanted to get even more creative, you could simply write a script on your main box that you update to automate the entire process for all your local boxes using nothing more than rsync and ssh calls of run the updates and dup removal scripts. (I haven't been that creative yet) Just remember to separate your boxes into groups/classes by architecture (i686 x86_64) Cheers. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com err.. what? Don't you think, that mirrorsync in crontab is MUCH more easier way to get new packages, huh? I'm using local mirror for 3 years and i have no idea why i shoulnd't use it know. P.S. If you have local mirror, you won't worry about 3-4 another GB's. You just download some less porn. Less porn - more profit. -- made my day -- Kirill E. Churin Jabber: reflex...@reflexing.ru, ICQ: 8163230, Skype on demand. In a world *without walls* or fences, *who needs windows and gates?*
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/14/2010 08:13 AM, Fess wrote: err.. what? Don't you think, that mirrorsync in crontab is MUCH more easier way to get new packages, huh? I'm using local mirror for 3 years and i have no idea why i shoulnd't use it know. With commercial bandwidth limited to 1M down, I worry about the extra 3-4G of download. That is 9 hours of the connection pegged at 135K that I don't need to use. Yes mirrorsync is easier, but it also increases the load on the Arch mirrors when multiplied by 100's/1000's of people using it. My approach downloads only the needed packages. A bit more work - yes, but it helps the community be reducing demand on resources :) -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
Re: [arch-general] 'Local mirror' page was removed from wiki
On 11:47 Tue 14 Sep , David C. Rankin wrote: On 09/14/2010 08:13 AM, Fess wrote: err.. what? Don't you think, that mirrorsync in crontab is MUCH more easier way to get new packages, huh? I'm using local mirror for 3 years and i have no idea why i shoulnd't use it know. With commercial bandwidth limited to 1M down, I worry about the extra 3-4G of download. That is 9 hours of the connection pegged at 135K that I don't need to use. Yes mirrorsync is easier, but it also increases the load on the Arch mirrors when multiplied by 100's/1000's of people using it. My approach downloads only the needed packages. A bit more work - yes, but it helps the community be reducing demand on resources :) -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com 1)Have you heard about 3rd party mirrors? 2)People who use gnome must die. No, really - too much traffic on servers. 3)With local mirror you have fast internet, i think. Wiki done by community. Some guys think, that local mirrors are cool - i agree with them. Why should we stop people from being cool? --
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/14/2010 12:08 PM, Fess wrote: 1)Have you heard about 3rd party mirrors? Yep - it's bandwidth for somebody 2)People who use gnome must die. No, really - too much traffic on servers. To each his own - I'm using fluxbox at the moment :p (very small) 3)With local mirror you have fast internet, i think. I'm talking about My slow internet. Go figure, my ISP is the same for home/work. Home I get 5M down + fixed IP for $65, for work (same ISP, but considered 'commercial' I get 1M down + fixed IP for $90) -- screw job for sure. A 1M pipe is a really small pipe for gigabit downloads :( -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
Re: [arch-general] 'Local mirror' page was removed from wiki
On 13:29 Tue 14 Sep , David C. Rankin wrote: On 09/14/2010 12:08 PM, Fess wrote: 1)Have you heard about 3rd party mirrors? Yep - it's bandwidth for somebody 2)People who use gnome must die. No, really - too much traffic on servers. To each his own - I'm using fluxbox at the moment :p (very small) 3)With local mirror you have fast internet, i think. I'm talking about My slow internet. Go figure, my ISP is the same for home/work. Home I get 5M down + fixed IP for $65, for work (same ISP, but considered 'commercial' I get 1M down + fixed IP for $90) -- screw job for sure. A 1M pipe is a really small pipe for gigabit downloads :( -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com I still don't get it.. You have small pipe. But a lot of people have bigger. So - if you can't use - do not use it. Why EVERYONE shouldn't use it? --
Re: [arch-general] 'Local mirror' page was removed from wiki
On Tue, Sep 14, 2010 at 1:36 PM, Fess killall_hum...@lavabit.com wrote: I still don't get it.. You have small pipe. But a lot of people have bigger. So - if you can't use - do not use it. Why EVERYONE shouldn't use it? it really is much more appropriate for everyone involved if you just use a shared cache, and respectably consider the truth that if you're not a seed, you're a leech. you say gnome=too much traffic, then wtf would you want to sync: gnome + kde + insert + insert + insert + insert + insert + insert + insert + insert + ... + ... every single time they are updated, when you're not even going to use it 90% of it? _that_ sounds like too much traffic to me. use what you need; no more, no less. anything else is greedy and wasteful. C Anthony
Re: [arch-general] 'Local mirror' page was removed from wiki
On 19:13 Tue 14 Sep , C Anthony Risinger wrote: On Tue, Sep 14, 2010 at 2:27 PM, Nathan Wayde disposa...@konnichi.com wrote: here's what I'd(and I imagine most others who know about sharing the cache) use a local mirror for: to be able to sync all other systems from it. plain and simple. if my systems don't have internet connection or something like that then i simply get the packages from the master, cache sharing doesn't and cannot solve that problem at all, that's a fact. shared cache won't solve that sure... but there are better solutions: ) if you can get it from master, then it sounds like you have a LAN connection; tunnel a connection thru master... ) if you have a LAN, what can't some machines have access anyway? ) if you don't have a LAN, you are manually moving packages? you could do that without a local mirror ) if you have a LAN, but _cannot_ allow some access to the net, then use a different method like a caching proxy local mirror = quick/easy crutch to avoid better utilization of local/peer resources i use a homebrew proxy/cache solution for my home, works fine. one machine pretends to be a repo, others look to it for packages... easy. i'm not using this exact version now, but i implemented this (rather crappily) while first learning python: pacproxy (or something that vaguely resembles an apt-proxy clone) https://bbs.archlinux.org/viewtopic.php?id=87115 now to the bandwidth issue. it's obviously bogus, because: 1) they assume everyone/(lots of people) is going to create a local mirror. 2) they assume that they're all going to sync from the same server. 3) they assume this extra bandwidth waste actually causes a problem for all the mirrors - i.e that there's only 1 mirror. now, if my assumptions are wrong thus leading to false conclusions then please correct me, but so far all I've heard is whining about local mirror causing problems for the mirrors but nothing about what these problems actually are, in the meantime the original wiki was deemed bad with not much of a valid reason and nothing being done to further educate us the users. i don't think it's even about whether or not it _is_ causing a problem, and more a preemptive move to discourage naive implementations. sure, if you have a heterogeneous environment of 200 machines, then a local mirror probably isn't too bad an idea... but it still isn't needed, as faster/better/cheaper methods are available. in my opinion, if you're not publicly seeding your mirror, then you don't need it; else you probably only want it due to an extreme case of laziness. sure maybe mirror XYZ can handle constant sync's from everyone looking at it... but really, do them a favor, and don't; it might piss them off :-). You can probably tell that I'm annoyed by this and the simple fact is that ARM sync script was based off the script on that wiki, it's not the same as I changed a lot of options to cater to my own needs but as have been said the script was bad, no-one is telling us what was bad about it and these alternative methods are wholly inadequate at best. yeah i don't really know the politics here, or have even seen the script. in my own experience back in the day syncing ubuntu repos (for easy install at remote locations from large USB key when client requirements are unknown)... you likely flat out don't need it, and there are _very_ few legitimate use cases for it (the parenthesized use case above is about the best one i know). all i'm suggesting is that just because you can and it's easy doesn't mean you should. but hey, i don't run a mirror, and extreme leeching won't affect me, so ultimately i could care less; if i did though, i would monitor for this kind of crap... i mean, doesn't the official arch mirror impose similar restrictions? just do you part to not be excessive. does one check out the entire library on the possibility of reading 10 books? C Anthony I think, i know(and others, who use this method) better what i'm doing, and why i am doing it. So, i tell you once more - community think, that this is useful. People, who say Hey, man! I have server, and rsync installed, add me please to the list of 3rd party mirrors know what they do. If they offer this service - they think it helps. If they would have 'tiny pipe'(or something else tiny) they wouldn't do it. So, i still don't understand why opinion of community ignored. --
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/12/2010 09:44 AM, Steve Holmes wrote: Hey David, Thanks for the suggestion; when it comes to learning more about how to do more with and get more out of the Arch environment, I'm all ears. I'm keeping this message here for future reference when I might need to maintain several machines under an Arch repo. This is cool indeed. Thanks, Between home and office, I have 10-12 Arch boxes now. (I think I only have 3 suse boxes left). Finding a way to eliminate 300M downloads was a priority. With Arch, it was easier than I expected. (of course I needed help from the list to figure that out ;-) So I'm glad to help. After 1.5 years with Arch and even going through the pacman tar.gz to tar.xz changes, there are probably only 10-20 packages that I have had to manually move manually. (verses over 10,000+ that have passed through /var/cache/pacman/pkg) I do really full installs of Arch with many desktops (kde3, kde4, iceWM, fluxbox, openbox WindowMaker, Gnome, enlightenment e16 and e17) and full openOffice and compilation tools, etc.. and total packages per box run from ~2200 to ~2800 (for full-server installs). A simple way to manage updates between boxes over the LAN w/o having to re-download over the Internet is worth it's weight in gold. Especially at the office where 'business internet bandwidth' cost is ~ 5x the residential cost. (no wonder the US is losing its competitive edge). Anyway, glad I could help with the idea. Put it in your hip pocket for when you have multiple boxes. This simple little setup works like a champ. Also, as a final tip, if you don't currently use 'basket notepads' to keep your Arch/Linux notes squirreled away in. Give it a look. Even if you don't use kde3 or kde4, it's worth loading the runtime and installing basket notepads. (I'm running fluxbox right now and basket works just fine) It's like a flexible second brain for any type of notes/sounds/colors/images/etc.. (I try to limit it to text notes just to minimize the size) Here is a quick screenshot of some of what I use basket for to give you an idea of what it is and what you can do with it: http://www.3111skyline.com/dl/img/misc/basket-example.jpg -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/09/2010 07:40 AM, Fess wrote: Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. I looked into an Arch local 'mirror/repo' when I started using Arch, similar to what I used to maintain for my SuSE installs. (the local mirror page was a mess and wrong then as well) However, since Arch will check for the presence of packages in /var/cache/pacman/pkg before/(instead of) re-downloading the packages, for local network boxes, it was just easier to use rsync to grab the packages from whatever box did the most recent update and transfer then to the nextbox you need to update (same architecture only). The only caveat is you need a way to manage duplicate (old packages) in /var/cache/pacman/pkg so I came up with a script that does that. So instead of worrying about a 'local mirror', just: (1) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman (2) grab the following 2 scripts: (the wrapper script that calls the main script [twice - see why below]) http://www.3111skyline.com/dl/Archlinux/scripts/fduparch.sh (the main duplicate identification and removal script) http://www.3111skyline.com/dl/Archlinux/scripts/fduppkg Put them both in /usr/local/bin (or link them there), edit fduparch.sh and change the directories you want the duplicates from /var/cache/pacman/pkg moved to (default is /home/backup/pkg-old and for the second pass to /home/backup/pkg-older). Then after updating one box, just call fduparch.sh (the wrapper script) as root and the duplicates in /var/cache/pacman/pkg are moved as follows: Pass 1: /var/cache/pacman/pkg = /home/backup/pkg-old Pass 2: /home/backup/pkg-old = /home/backup/pkg-older Which leaves you with the current set of packages in: /var/cache/pacman/pkg The last used packages before update in: /home/backup/pkg-old And finally all older packages in: /home/backup/pkg-older which can be deleted or archived. (of course you can delete the packages in /home/backup/pkg-old if you like as well) NOTE: the duplicate removal script uses the file ctime and/or mtime to determine which is the newer package, so when copying machine to machine with rsync, make sure you preserve the file attributes. (rsync's -a option works fine). Once you set this up, maintaining a local set of packages is a breeze, just: (1) pacman -Syu as normal on one box (works the same for new installs too) (2) fduparch.sh (as root to remove the older versions of new packages) (3) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman (only for boxes of the same architecture of course) (4) pacman -Syu on the box you rsync'ed the packages to (nextbox) (5) call fduparch.sh on the box you just updated (nextbox), and so on, and so on... As long as your boxes have reasonably similar packages installed, this eliminates 90+% of re-downloading, takes no additional bandwidth because you have to download the updated packages once anyway, and is simple enough for me to manage so you guy will have no problems with it :p And if you wanted to get even more creative, you could simply write a script on your main box that you update to automate the entire process for all your local boxes using nothing more than rsync and ssh calls of run the updates and dup removal scripts. (I haven't been that creative yet) Just remember to separate your boxes into groups/classes by architecture (i686 x86_64) Cheers. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
[arch-general] 'Local mirror' page was removed from wiki
Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. --
Re: [arch-general] 'Local mirror' page was removed from wiki
On Thu, Sep 9, 2010 at 3:40 PM, Fess killall_hum...@lavabit.com wrote: Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. -- I agree with the reasoning, I disagree with the content removal. : D
Re: [arch-general] 'Local mirror' page was removed from wiki
On Thu, 9 Sep 2010 15:55:35 +0300, Evangelos Foutras foutre...@gmail.com wrote: On Thu, Sep 9, 2010 at 3:40 PM, Fess killall_hum...@lavabit.com wrote: Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. -- I agree with the reasoning, I disagree with the content removal. : D See https://bbs.archlinux.org/viewtopic.php?id=103987 The original article was just wrong and causing us problems. Feel free to add an improved one. -- Pierre Schmitz, https://users.archlinux.de/~pierre
Re: [arch-general] 'Local mirror' page was removed from wiki
On 09/09/10 14:11, Pierre Schmitz wrote: On Thu, 9 Sep 2010 15:55:35 +0300, Evangelos Foutras foutre...@gmail.com wrote: On Thu, Sep 9, 2010 at 3:40 PM, Fesskillall_hum...@lavabit.com wrote: Page Local mirros was removed from wiki by this reason: - It is generally frowned upon to create a local mirror due the bandwidth that is required. There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs. - I think it's very useful ability, and why this page was removed... i don't know. It's silly. If anyone else think it must be return - maybe we should do it? P.S. A lot of guys agreed with me this morning, so i think many people want this article back. -- I agree with the reasoning, I disagree with the content removal. : D See https://bbs.archlinux.org/viewtopic.php?id=103987 The original article was just wrong and causing us problems. Feel free to add an improved one. If the script was broken or it caused problems it wouldn't hurt to state what these problems. Correct me if I'm something here but I have idea what these problems were. If the script caused problems then what stops someone creating their own script that works the same way? IMHO the text posted is likely worse than what was there before since it just makes a bunch of claims with no info.
