There is this tool available too. http://www.brainless.us/downloads.aspx?cid=1
On Thu, Jan 21, 2010 at 9:48 AM, Philipp Reddigau <phil...@gkserver.de>wrote: > Hi, > I will try to answer all replys in one post. > > First i will describe how our FDL works. > The content server(fdl) connect all 5 minutes to all boxes an check| > someone > uploaded a file? > if there are any changes against the content server the new data will be > transfered. > So in a general time of 10 Minutes all files are up to date. > But with missing *.bz2 files. > The bz2 files are generated each night. After the first compression they > are > going into several checks to be sure that the file is ok and complete. > (Example: Maybe a bsp was not uploaded complete at fdl sync time) > On each night we have about 500 to 1000 new files including. bsp, nav, mp3 > and so on. > why so much? > It is possible that one Custom Map (same Version) differ from a other > Server > cause client upload failures or other crap. > As an example: > We have the map $2000$.bsp in 7 versions. (But it is the same Map and they > are working all) > So we have this map 7 times as bz2 because..the engine.. > Its crap if you upload any of this versions to fdl as normal bsp all > players > can play with it. Now upload any version of the *.bz2 it will not work on > all players, some players can join some not. > I hope you understand what i mean. English is not my best language ;). > > So now to your suggestions/questions: > @Bjorn > --force is needed for some integrity checks we are touch *.bz2 > Something like that. > if [ ! -e ./bz2data_diff/bsp/"$file".bz2 ]; then > touch ./bz2data_diff/bsp/"$file".bz2 > fi > We are packing only new files and delete/blacklist all bz2 of the same file > if we reach a file version limit of 10. > > @Shane > Yes i have checked. Its compatible with bzip2. > The output is a little bit bigger than the bzip2 compressed file, because > pbzip2 breaks the source file in little pieces and each piece is compressed > by a other core. > I think that the developers are using a really old libary to decompress. > Because of that i ask about the method that the developers are using. > concrete witch libary version are in use? > If i know that i will program a multithreaded bzip2 myself. > > Have you tried to upload a compressed pbzip2 file to a fdl? > Maybe someone can test it. > > > Mit freundlichen Grüßen, > Philipp Reddigau (GKServer) > > GKServer > Daniel Zschau > Niederfährer Straße 38 > 01662 Meißen > > > -----Ursprüngliche Nachricht----- > Von: hlds_linux-boun...@list.valvesoftware.com > [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von Shane > Turner > Gesendet: Donnerstag, 21. Januar 2010 17:38 > An: Half-Life dedicated Linux server mailing list > Betreff: Re: [hlds_linux] BZip2 compression methods > > I think we need a slightly better description of the problem you're > trying to solve. > > My interpretation of your original post is that you have a large number > of files to compress, but many cores. You tried pbzip2 (Parallel BZIP2) > and the resulting files couldn't be decompressed. > > Have you confirmed that the pbzip2 compressed files are decompressible > by normal bzip2? The page for pbzip2 indicates that they should be > compatible with bzip2 1.0.2 or newer. > > The problem could be that the game is using a bzip2 algorithm that isn't > compatible with bzip2 1.0.2. > > Shane > > On 3:59 PM, Philipp Reddigau wrote: > > Hi, > > Thank you for the reply. > > But this will compress up to 10 files at once. > > It will not compress one file on more than one core. > > It would be better and much faster if one file compressed by 8 or 16 > cores. > > > > An Example: (we are do it like your skript) > > 5000 files: > > normal method(one file at time one core): > > 7h 34m > > > > like your skript(15 files on 15 cores): !extreme load, hdd is unusable! > > 1h 37m > > > > pbzip2(8 cores): !all is right, hdd usuable! > > 1h 43m > > > > Take care at the number of cores. > > > > Mit freundlichen Grüßen, > > Philipp Reddigau (GKServer) > > > > GKServer > > Daniel Zschau > > Niederfährer Straße 38 > > 01662 Meißen > > > > > > -----Ursprüngliche Nachricht----- > > Von: hlds_linux-boun...@list.valvesoftware.com > > [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von kama > > Gesendet: Donnerstag, 21. Januar 2010 14:17 > > An: Half-Life dedicated Linux server mailing list > > Betreff: Re: [hlds_linux] BZip2 compression methods > > > > > > > > On Thu, 21 Jan 2010, Philipp Reddigau wrote: > > > > > >> Hi, > >> we are using a automatic *.bz2 for our content servers (fastdownload) on > >> nix/lenny. > >> It works fine with bzip2 --compress --keep --force --best "$file" > >> But we have one Problem..we have many files to compress and one bzip2 is > >> > > not > > > >> enough. > >> So we have tried pbzip2 after compress join on server "Could not CRC > >> blabla". > >> So the game cannot decompress the file.. > >> I there a way to get it clean with any other compress program? > >> (multithreaded) > >> Anyone knows witch exact format is required, that the games can > decompress > >> it? > >> I cannot find any information about the decompression. > >> > > maybe do a script that does it for you instead... > > > > This script will keep you at max 10 instances of bzip: > > > > -- snip-- > > #!/usr/local/bin/bash > > > > max > > > > for file in $* > > do > > bzip2 --compress --keep --force --best "$file"& > > while [ `ps aux | grep bzip | awk 'END { print NR }'` -ge $max ] > > do > > sleep 1 > > done > > done > > -- snap -- > > > > This is just an example. There are probably ways to tune it and make it > > better. > > > > /Bjorn > > > > _______________________________________________ > > To unsubscribe, edit your list preferences, or view the list archives, > > please visit: > > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > > > > > > > > > > > _______________________________________________ > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > > > _______________________________________________ > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > _______________________________________________ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
