Yes and I thought that would be another option, it supports linux. On Thu, Jan 21, 2010 at 1:08 PM, Philipp Reddigau <phil...@gkserver.de>wrote:
> Thank you DontWannaName...but do you read anything inside this post? :) > > Mit freundlichen Grüßen, > Philipp Reddigau (GKServer) > > GKServer > Daniel Zschau > Niederfährer Straße 38 > 01662 Meißen > > > -----Ursprüngliche Nachricht----- > Von: hlds_linux-boun...@list.valvesoftware.com > [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von > DontWannaName! > Gesendet: Donnerstag, 21. Januar 2010 21:04 > An: Half-Life dedicated Linux server mailing list > Betreff: Re: [hlds_linux] BZip2 compression methods > > There is this tool available too. > > http://www.brainless.us/downloads.aspx?cid=1 > > On Thu, Jan 21, 2010 at 9:48 AM, Philipp Reddigau > <phil...@gkserver.de>wrote: > > > Hi, > > I will try to answer all replys in one post. > > > > First i will describe how our FDL works. > > The content server(fdl) connect all 5 minutes to all boxes an check| > > someone > > uploaded a file? > > if there are any changes against the content server the new data will be > > transfered. > > So in a general time of 10 Minutes all files are up to date. > > But with missing *.bz2 files. > > The bz2 files are generated each night. After the first compression they > > are > > going into several checks to be sure that the file is ok and complete. > > (Example: Maybe a bsp was not uploaded complete at fdl sync time) > > On each night we have about 500 to 1000 new files including. bsp, nav, > mp3 > > and so on. > > why so much? > > It is possible that one Custom Map (same Version) differ from a other > > Server > > cause client upload failures or other crap. > > As an example: > > We have the map $2000$.bsp in 7 versions. (But it is the same Map and > they > > are working all) > > So we have this map 7 times as bz2 because..the engine.. > > Its crap if you upload any of this versions to fdl as normal bsp all > > players > > can play with it. Now upload any version of the *.bz2 it will not work on > > all players, some players can join some not. > > I hope you understand what i mean. English is not my best language ;). > > > > So now to your suggestions/questions: > > @Bjorn > > --force is needed for some integrity checks we are touch *.bz2 > > Something like that. > > if [ ! -e ./bz2data_diff/bsp/"$file".bz2 ]; then > > touch ./bz2data_diff/bsp/"$file".bz2 > > fi > > We are packing only new files and delete/blacklist all bz2 of the same > file > > if we reach a file version limit of 10. > > > > @Shane > > Yes i have checked. Its compatible with bzip2. > > The output is a little bit bigger than the bzip2 compressed file, because > > pbzip2 breaks the source file in little pieces and each piece is > compressed > > by a other core. > > I think that the developers are using a really old libary to decompress. > > Because of that i ask about the method that the developers are using. > > concrete witch libary version are in use? > > If i know that i will program a multithreaded bzip2 myself. > > > > Have you tried to upload a compressed pbzip2 file to a fdl? > > Maybe someone can test it. > > > > > > Mit freundlichen Grüßen, > > Philipp Reddigau (GKServer) > > > > GKServer > > Daniel Zschau > > Niederfährer Straße 38 > > 01662 Meißen > > > > > > -----Ursprüngliche Nachricht----- > > Von: hlds_linux-boun...@list.valvesoftware.com > > [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von Shane > > Turner > > Gesendet: Donnerstag, 21. Januar 2010 17:38 > > An: Half-Life dedicated Linux server mailing list > > Betreff: Re: [hlds_linux] BZip2 compression methods > > > > I think we need a slightly better description of the problem you're > > trying to solve. > > > > My interpretation of your original post is that you have a large number > > of files to compress, but many cores. You tried pbzip2 (Parallel BZIP2) > > and the resulting files couldn't be decompressed. > > > > Have you confirmed that the pbzip2 compressed files are decompressible > > by normal bzip2? The page for pbzip2 indicates that they should be > > compatible with bzip2 1.0.2 or newer. > > > > The problem could be that the game is using a bzip2 algorithm that isn't > > compatible with bzip2 1.0.2. > > > > Shane > > > > On 3:59 PM, Philipp Reddigau wrote: > > > Hi, > > > Thank you for the reply. > > > But this will compress up to 10 files at once. > > > It will not compress one file on more than one core. > > > It would be better and much faster if one file compressed by 8 or 16 > > cores. > > > > > > An Example: (we are do it like your skript) > > > 5000 files: > > > normal method(one file at time one core): > > > 7h 34m > > > > > > like your skript(15 files on 15 cores): !extreme load, hdd is unusable! > > > 1h 37m > > > > > > pbzip2(8 cores): !all is right, hdd usuable! > > > 1h 43m > > > > > > Take care at the number of cores. > > > > > > Mit freundlichen Grüßen, > > > Philipp Reddigau (GKServer) > > > > > > GKServer > > > Daniel Zschau > > > Niederfährer Straße 38 > > > 01662 Meißen > > > > > > > > > -----Ursprüngliche Nachricht----- > > > Von: hlds_linux-boun...@list.valvesoftware.com > > > [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von kama > > > Gesendet: Donnerstag, 21. Januar 2010 14:17 > > > An: Half-Life dedicated Linux server mailing list > > > Betreff: Re: [hlds_linux] BZip2 compression methods > > > > > > > > > > > > On Thu, 21 Jan 2010, Philipp Reddigau wrote: > > > > > > > > >> Hi, > > >> we are using a automatic *.bz2 for our content servers (fastdownload) > on > > >> nix/lenny. > > >> It works fine with bzip2 --compress --keep --force --best "$file" > > >> But we have one Problem..we have many files to compress and one bzip2 > is > > >> > > > not > > > > > >> enough. > > >> So we have tried pbzip2 after compress join on server "Could not CRC > > >> blabla". > > >> So the game cannot decompress the file.. > > >> I there a way to get it clean with any other compress program? > > >> (multithreaded) > > >> Anyone knows witch exact format is required, that the games can > > decompress > > >> it? > > >> I cannot find any information about the decompression. > > >> > > > maybe do a script that does it for you instead... > > > > > > This script will keep you at max 10 instances of bzip: > > > > > > -- snip-- > > > #!/usr/local/bin/bash > > > > > > max > > > > > > for file in $* > > > do > > > bzip2 --compress --keep --force --best "$file"& > > > while [ `ps aux | grep bzip | awk 'END { print NR }'` -ge $max ] > > > do > > > sleep 1 > > > done > > > done > > > -- snap -- > > > > > > This is just an example. There are probably ways to tune it and make it > > > better. > > > > > > /Bjorn > > > > > > _______________________________________________ > > > To unsubscribe, edit your list preferences, or view the list archives, > > > please visit: > > > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > > > > > > > > > > > > > > > > > > _______________________________________________ > > To unsubscribe, edit your list preferences, or view the list archives, > > please visit: > > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > > > > > > _______________________________________________ > > To unsubscribe, edit your list preferences, or view the list archives, > > please visit: > > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > > > _______________________________________________ > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > > > _______________________________________________ > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > http://list.valvesoftware.com/mailman/listinfo/hlds_linux > _______________________________________________ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
