[SLUG] gunzip 1.3G file over ssh - crawling
Hi guys! I'm pulling a compressed (gzip -9) disk image over the LAN and decompressing prior to writing to disk with dd. The command I am using is ssh [EMAIL PROTECTED] cat client.img.gz | gunzip | dd of=/dev/hda1 This seems to work reasonably well until it gets somewhere around the 1G mark at which point everything seems to have slowed down to a crawl. The result is that it takes nearly as long as when I don't compress it and just pull the whole 8G partition through ssh. I'm guessing it's something to do with buffering of the data in gunzip prior to it getting to dd that's blowing things up. Anyone got any clues on how to do this more on the fly so all data is passed through without any buffering? Why is this sort of question always late on a Friday? Oh well, have a good weekend! -- Simon Wong [EMAIL PROTECTED] -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] ADSL Query
At 02:09 PM 24/02/2006, Rajnish wrote: After looking at the broadbandchoice website for some weeks now, I've figured that I'd ask the sluggers. What are your suggestions for a modest $50/month commitment, minimum 512Kbps plan from an ISP that does not shy away from Linux (slackware, fedora) ? With the market the way it is, the more it seems that you get with your $50, the more restrictions there can be on the service. BBChoice is a good place to start, bearing in mind that most of the comments you will see on whingepool, are complaints :) Linux-supporting ISP's ar few and far between, simply because of the breadth of experience of their helpdesk staff. I'd guess that most users use a router which does the NATing for their LAN, as you would most likely be doing with your setup. And one other question: to avoid having long wires running across the length of the house (phone's in one corner, my computer's in another), I was considering a wireless router (?). What are the draw backs of this ? And also is it likely to interfere with a baby monitor ? Wireless routers are good, use a separate access point myself, simply because I have a cisco 1700 router As for the interference - most likely not baby monitors are usually 433 or in the 900 MHz range, Wireless ethernet is 2.4GHz Cheers, Rob -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Re: slug Digest, Vol 1, Issue 41
Hi Rajnish, Dunno about ISOs. Have you checked TPGs standard ADSL deals? My current one is $50 per month, 1.5Mb down (I get an average of 1.2) and 20G per month. I'm not sure if they've discontinued the deal to new ADSL members...(?) www.Whirlpool.net.au is a good place to check out what is available. I have a friend with iinet and he's very happy. TPG - Linux. I've stopped mentioning it to them :) It just creates a problem where they think it's me and not them. I've had very few problems in the last four/five years or so and all but one was at their end... which they fixed. Regards, Patrick 4. ADSL Query (Rajnish) All, Finally, I am jumping onto the broad-band-wagon. I am currently with TPG dialup and am generally satisfied with their service quality. After looking at the broadbandchoice website for some weeks now, I've figured that I'd ask the sluggers. What are your suggestions for a modest $50/month commitment, minimum 512Kbps plan from an ISP that does not shy away from Linux (slackware, fedora) ? My typical useage will be: remote desktop to work (will use windoze for this), general webbrowsing (news, wikipedia, cricket). I also intend to download linux distros (for experimentation - in the early days at least), some internet radio etc etc ... with bandwidth to burn ... who know what more I might end up doing. (Any ISP that mirrors the distros and don't count them in ones download limits ?) And one other question: to avoid having long wires running across the length of the house (phone's in one corner, my computer's in another), I was considering a wireless router (?). What are the draw backs of this ? And also is it likely to interfere with a baby monitor ? As always, any suggestions/recommendation/thoughts will be most appreciated. Thanks in advance. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] gunzip 1.3G file over ssh - crawling
Simon Wong wrote: ssh [EMAIL PROTECTED] cat client.img.gz | gunzip | dd of=/dev/hda1 This seems to work reasonably well until it gets somewhere around the 1G mark at which point everything seems to have slowed down to a crawl. That first 1GB is the oddity. It's just some buffers filling. The results after that are the real results where buffering no longer gives an advantage and you see the true sustained (lack of) throughput. The result is that it takes nearly as long as when I don't compress it and just pull the whole 8G partition through ssh. That would indicate that you're filling the bus to the hard disk, not the network between the CPUs. dd is writing a lot of empty blocks down to that disk. You might want to consider using tar is the filesystem has a lot of empty space. Anyone got any clues on how to do this more on the fly so all data is passed through without any buffering? That's a very handwaving question since you haven't told us anything about the computer. The bottleneck obviously varies with the hardware. Firstly, work out if what you are doing is possible. Look up the sustained throughput of your drive. Make sure your disk interface can supply that. What's the slowest network link -- 100Mbps or 1000Mbps? What's the length of the network link in 1000Km. For example, 8GB will take at least 160s to be written to a single disk, at least 640s to cross a fast ethernet link. Now set the machine for maximum performance. hdparm should report 32b UDMA, write caching, look ahead, APM disabled, fast AAM goodness. TCP buffers should be larger than the bandwidth-delay product. ssh shouldn't be doing (de)compression, since you've already compressed the file. Test. Are you CPU-bound, I/O-bound or network bound (top, vmstat, etc)? If you are, is that bound reasonable (eg, 90% of the sustained disk write rate)? You might want to try transferring from and to /dev/null to check network+CPU preformance. Note that ssh has a special performance issue -- it uses a 128MB window so the bandwidth- delay product needs to be under that for ssh to run at maximum speed. Now you're found the bottleneck, fix it. Repeat test until one of the hard bounds is reached. If possible, chose a better method to move away from that bound (eg, dd v tar; ssh v ftp of of a gpg-encrypted file). If you're going to be doing this a lot (eg, a disk image server) then you might want to thing about the disk subsystem. For example, using disk striping can double the sustained throughput. If you've only got a LAN and a desktop machine then working though this takes an afternoon. If you've got a supercomputer and a long-haul network it can take several weeks. Remember to record your results at each step. Let us know the interesting numbers (eg, what is the real throughput of your particular drive). There's surprisingly few real benchmarks out there, so you'll be helping a lot by posting your numbers. Cheers, Glen -- Glen Turner Tel: (08) 8303 3936 or +61 8 8303 3936 Australia's Academic Research Network www.aarnet.edu.au -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] How to discover which modules unnecessary
[EMAIL PROTECTED] wrote: Actually this is mostly just a waste of effort. Config swap and let the system swap out all the bits it does not need. Memory used by modules isn't swappable. Inserting and then removing a module isn't a good idea. Rather prevent the module from being loaded in the first place using /etc/modprobe.conf install usb-throb /bin/true I'm not sure why the firewire driver would be loaded without the kernel seeing a firewire PCI ID. Check the init script for an explicit modprobe. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] graduate programmers
James Purser wrote: Economics 101: A Graduate is only worth as much as someone is willing to pay them. Yep, and a quick look at the stats from the Graduate Careers Council of Australia shows that recent median first year salaries for IT graduates under 25 years of age (ie, no work history) in Sydney is $40,000. I can quite easily see a uni grad with only a degree getting $25,000 a year. In that case the person should exercise Economics 101 and find employment elsewhere. Cheers, Glen -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Re: graduate programmers
On Fri, Feb 24, 2006 at 02:48:02PM +0100, Glen Turner wrote: James Purser wrote: Economics 101: A Graduate is only worth as much as someone is willing to pay them. Yep, and a quick look at the stats from the Graduate Careers Council of Australia shows that recent median first year salaries for IT graduates under 25 years of age (ie, no work history) in Sydney is $40,000. under 25 years of age doesn't mean no work history. It means under 25. Also, that $40k is for your 50th percentile employed candidate, so half of the graduates who found work are getting less than $40k. Do the statistics indicate what the unemployment rate is for new IT grads? - Matt -- A polar bear is a rectangular bear after a coordinate transform. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] gunzip 1.3G file over ssh - crawling
Quoting Simon Wong [EMAIL PROTECTED]: Hi guys! I'm pulling a compressed (gzip -9) disk image over the LAN and decompressing prior to writing to disk with dd. The command I am using is ssh [EMAIL PROTECTED] cat client.img.gz | gunzip | dd of=/dev/hda1 This seems to work reasonably well until it gets somewhere around the 1G mark at which point everything seems to have slowed down to a crawl. The result is that it takes nearly as long as when I don't compress it and just pull the whole 8G partition through ssh. I'm guessing it's something to do with buffering of the data in gunzip prior to it getting to dd that's blowing things up. Anyone got any clues on how to do this more on the fly so all data is passed through without any buffering? Why is this sort of question always late on a Friday? Oh well, have a good weekend! -- Simon Wong [EMAIL PROTECTED] What if you use scp to get the compressed file to your end first. I presume this is a regular backup. How much of that 8GB actually changes between backups? We have mirrored servers scattered around the country, and use rsync daily to keep things synchronised. The L in Lan is for local. Can't you access the partition directly with nfs or smbclient; or is this a remote Lan on the end of a vpn tunnel half a world away? Amanda Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Committee Nominations!
Lindsay Holmwood wrote: Robert Collins wrote: On Tue, 2006-02-21 at 19:37 +1100, Chris Deigan wrote: Ken Wilson for Treasurer I accept Ken -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] ADSL Query
On Saturday 25 February 2006 09:00, [EMAIL PROTECTED] wrote: Hi Rajnish, Dunno about ISOs. Have you checked TPGs standard ADSL deals? My current one is $50 per month, 1.5Mb down (I get an average of 1.2) and 20G per month. I'm not sure if they've discontinued the deal to new ADSL members...(?) www.Whirlpool.net.au is a good place to check out what is available. I have a friend with iinet and he's very happy. TPG - Linux. I've stopped mentioning it to them :):) It just creates a problem where they think it's me and not them. I've had very few problems in the last four/five years or so and all but one was at their end... which they fixed. iinet are ultra arrogant, do it our way or Piss Off (literal exact words) They were making a foray 'overeast', don't know ... http://powerdsl.com.au For $59/month: 1.5M Fixed IP No restriction on server-services 20 or 30G linux friendly me 3 others I've pointed at them very happy James -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] ADSL Query
I'm very happy with Internode. Their CEO is a Mac user and is very sympathetic
towards users of alternative OSs. I haven't tested out their Support yet, but
the company claims that they will try to help you no matter what OS you run.
--
Sridhar Dhanapalan [Yama | http://www.pclinuxonline.com/]
{GnuPG/OpenPGP: http://dhanapalan.webhop.net/yama.asc
0x049D38B4 : A7A9 8A02 78CB AB1B FCE4 EEC6 2DD9 249B 049D 38B4}
Let's sell these people a piece of blue sky. -- L. Ron Hubbard
pgpJrJPBLHCE8.pgp
Description: PGP signature
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] rpc.statd clients
I'm after a maintained lightweight rpc.statd client like xmeter or pref. Or least source a version that can be built with a modern compiler. -- Simon Males [EMAIL PROTECTED] -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] ADSL Query
I haven't had any problems with iinet myself. They don't seem to mind Linux. But then I have an Ethernet modem; also I have dual boot XP/FC, so I just get their technical support to fix the XP side, and then translate to FC. On Sat, 2006-02-25 at 09:22 +0800, [EMAIL PROTECTED] wrote: On Saturday 25 February 2006 09:00, [EMAIL PROTECTED] wrote: Hi Rajnish, Dunno about ISOs. Have you checked TPGs standard ADSL deals? My current one is $50 per month, 1.5Mb down (I get an average of 1.2) and 20G per month. I'm not sure if they've discontinued the deal to new ADSL members...(?) www.Whirlpool.net.au is a good place to check out what is available. I have a friend with iinet and he's very happy. TPG - Linux. I've stopped mentioning it to them :):) It just creates a problem where they think it's me and not them. I've had very few problems in the last four/five years or so and all but one was at their end... which they fixed. iinet are ultra arrogant, do it our way or Piss Off (literal exact words) They were making a foray 'overeast', don't know ... http://powerdsl.com.au For $59/month: 1.5M Fixed IP No restriction on server-services 20 or 30G linux friendly me 3 others I've pointed at them very happy James -- Regards, Martin mail: [EMAIL PROTECTED] homepage: http://thereisnoend.org signature.asc Description: This is a digitally signed message part -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] ADSL Query
On 25/02/2006, at 5:40 PM, Martin Ellison wrote: I haven't had any problems with iinet myself. They don't seem to mind Linux. But then I have an Ethernet modem; also I have dual boot XP/FC, so I just get their technical support to fix the XP side, and then translate to FC. I've got nothing but good things to say about iiNet. I've been with the for a few years (both dial up and ADSL). Their tech support is great (once you get through some of the dodgy first line support guys). They offered help with Debian, OSX and Airports - none of which they officially support. Their plans are pretty good too. I've got 1500kbps/80GB/month for a fairly reasonable $90 - if only I was on an iiNet DSLAM! -- Tony Green [EMAIL PROTECTED] -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
