Re: [wpkg-users] Replace xmlhttp download with curl/wget
Hi Marco, On 18.07.2011 15:24, Marco Gaiarin wrote: Mandi! Rainer Meier In chel di` si favelave... So for me there would be only one compromise we could potentially think about: Ship WPKG with a download-helper script. I ''vote'' for this compromise. ;-) Really: someone insert thedownload / tag, so seems to me a bad idea to remove it. So: a) extend a little with a md5= and/or sha= tag, totally ignored by the internal WSH downloader b) extend a little wpkg.js to support an external downloader, as an option (so, default use the internal) c) build an (example) external download script; i can do some try on the last, but really i've little cmd.exe knowledge... I've just committed a downloader script which allows advanced downloading via existing commands. Maybe it resolves the issue. I do not plan yet to include functionality in WPKG to call downloader from within download / tags as this would clutter package definition with attributes which are not used by WPKG and it would increase complexity in case more checksums (SHA2...) are added. From my point of view calling downloader directly from commands is the most simple and flexible solution. Some discussion can be followed here: http://bugzilla.wpkg.org/show_bug.cgi?id=195 br, Rainer - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Mandi! Rainer Meier In chel di` si favelave... So for me there would be only one compromise we could potentially think about: Ship WPKG with a download-helper script. I ''vote'' for this compromise. ;-) Really: someone insert the download / tag, so seems to me a bad idea to remove it. So: a) extend a little with a md5= and/or sha= tag, totally ignored by the internal WSH downloader b) extend a little wpkg.js to support an external downloader, as an option (so, default use the internal) c) build an (example) external download script; i can do some try on the last, but really i've little cmd.exe knowledge... [ Indeed, this have nothing to do with my ''server side download'' script, using the global /noDownlaod switch on wpkg.js. Clearly, also my script can be easily extended to use md5 hashes... ] Sure, and even when using commands to download files it could be globally disabled by simply supporting an environment variable NO_DOWNLOAD which makes the script to skip its task and exit with code 0 instead (or alternatively still verifying the checksums). But it is a little harder to ''look at'' download rules with and external (server side) script; so, please, keep the download tag... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia''http://www.sv.lnf.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)sv.lnf.it tel +39-0434-842711 fax +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/chi_siamo/5xmille.php (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Mandi! Rainer Meier In chel di` si favelave... Is there a reason this could not be done using existing commands? [...] I agree that using a special download command flag would be possible too. I would then prefer to call a cmd script (included in wplkg distribution) with all required parameters like downloader.cmd URL target MD5 Probably my english are very bad, but i mean exactly that. So this script could handle download and verification as needed using any tool you like and any flags suitable for your environment (e.g. proxy flags). Never minded about proxies. Right. Well, still I am not sure where the advantage over using this script as one of the first commands would be. + WSH, as stated in this list, have poor performance and limited features when downloading + WSH, as stated again here, have no hash function, any hash algorithm have to be implemented in pure WSH or relay on an external script So seems reasonable to, optionally, relay on an external script; seems reasonable, because we can relay on an external script, to add the 'md5' field to download / tag, that the internal WSH downloaded ignore but can be passed, optionally, to the external script. You are right, we can just use existing install / and update / tags to do so, but the download / tag just exist, and i think is more cleaner to use it. [ Indeed, this have nothing to do with my ''server side download'' script, using the global /noDownlaod switch on wpkg.js. Clearly, also my script can be easily extended to use md5 hashes... ] -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia''http://www.sv.lnf.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)sv.lnf.it tel +39-0434-842711 fax +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/chi_siamo/5xmille.php (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Hi Marco, On 15.07.2011 11:32, Marco Gaiarin wrote: + WSH, as stated in this list, have poor performance and limited features when downloading This is absolutely clear. That's also a reason why I do not really support the download / tag in particular. I think it's just a legacy feature for very basic download functionality. From my point of view most people would be better off by using wget and similar tools. + WSH, as stated again here, have no hash function, any hash algorithm have to be implemented in pure WSH or relay on an external script Sure, that's why I don't think using the an md5 attribute in download / makes much sense. As this would be either ignored in default case (pure WPKG deployment without wget/md5sum). Up to now all the features of WPKG are available by the default wpkg.js without the strict requirement for external tools. So seems reasonable to, optionally, relay on an external script; seems reasonable, because we can relay on an external script, to add the 'md5' field todownload / tag, that the internal WSH downloaded ignore but can be passed, optionally, to the external script. As said above some features discussed in this thread would then require additional tools (these tools would then not be optional again). So use of any of these flags would break the package if the tools it relies on are not available. Moreover such a hardcoded implementation would add a dependency to SPECIFIC download/md5 tools like wget and md5sum. Alternative tools cannot be easily used as the interface might be different. For example one might like to use sha1 checksumming or prefers an md5 sum tool which is faster but prints the output in slightly different format etc. You are right, we can just use existinginstall / andupdate / tags to do so, but thedownload / tag just exist, and i think is more cleaner to use it. Yes it might be more clean but using commands is much more flexible and allows you to use any tool which best fits your environment (and not just one which is chosen by WPKG team and enforces you to use this one). So for me there would be only one compromise we could potentially think about: Ship WPKG with a download-helper script. E.g. ./tools/download/wpgk-downloader.cmd This script accepts a clearly defined list of parameters like: - Download source URL - Download target folder - MD5 sum in pre-defined encoding (optional) - SH1 sum in pre-defined encoding (optional) The script could then decide on some internal logic what to download and whether the download was successful or not and then return with pre-defined exit status. E.g. 0 all success, download finished, checksum OK. 1 download failed 2 any checksum failed (md5 or sha1) However I still think it might be easier to provide such an optional script to be used in commands rather than called from download nodes in package.xml. The reason is simply that I see that commands would already support any number of parameters, using environment variables etc. while using a strict download tag structure always limits to certain features. So if later sha2 support or any other special solution is required, then it would requiring to adapt wpkg.js code. Using commands will work no matter which parameters might be added later. [ Indeed, this have nothing to do with my ''server side download'' script, using the global /noDownlaod switch on wpkg.js. Clearly, also my script can be easily extended to use md5 hashes... ] Sure, and even when using commands to download files it could be globally disabled by simply supporting an environment variable NO_DOWNLOAD which makes the script to skip its task and exit with code 0 instead (or alternatively still verifying the checksums). br, Rainer - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Hello, So maybe we could (optionally) have an external tool handle the download tag - curl or wget are excellent specialised (and free/GPL) tools. Pros: * much more robust than the WSH download method * saves memory * faster (probably because of the memory savings) * Support not only for HTTP, but FTP too. Cons: * WPKG would rely on an external binary (so it's foreign code, but at least no license problems) ... What do you think? We could introduce similar option like rebootCmd:special, where psshutdown is used for reboot (from config.xml: Setting rebootCmd to special will use tools\psshutdown.exe from * from www.sysinternals.com - if it exists ). So we can have predefined options for e.g. wget or curl in wpkg.js, while preserving the possibility to use another tool. Best regards, Falko - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Mandi! Falko Trojahn In chel di` si favelave... So we can have predefined options for e.g. wget or curl in wpkg.js, while preserving the possibility to use another tool. ...eg, a script (cmd suffices?!) that also check the MD5 of the file, preventing wpkg tring to install ever an ever that corrupted installer. ;-) This clearly needs an hash= or md5= field in download / tag. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia''http://www.sv.lnf.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)sv.lnf.it tel +39-0434-842711 fax +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/chi_siamo/5xmille.php (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
So we can have predefined options for e.g. wget or curl in wpkg.js, while preserving the possibility to use another tool. ...eg, a script (cmd suffices?!) that also check the MD5 of the file, preventing wpkg tring to install ever an ever that corrupted installer. ;-) This clearly needs an hash= or md5= field in download / tag. Great idea! But this'll require another external tool, because hash calculation via WSH is a world of slow.- wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Am 14.07.2011 14:55, schrieb heiko.hel...@horiba.com: So we can have predefined options for e.g. wget or curl in wpkg.js, while preserving the possibility to use another tool. ...eg, a script (cmd suffices?!) that also check the MD5 of the file, preventing wpkg tring to install ever an ever that corrupted installer. ;-) This clearly needs an hash= or md5= field indownload / tag. Great idea! But this'll require another external tool, because hash calculation via WSH is a world of slow. Yes. BTW there is already a feature request for that: http://bugzilla.wpkg.org/show_bug.cgi?id=195 Falko - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Mandi! heiko.hel...@horiba.com In chel di` si favelave... Great idea! But this'll require another external tool, because hash calculation via WSH is a world of slow. I've speaked about a script exactly for that. It is trivial in a CMD script to execute firstly curl/wget to download files, check exit code and if '0', run 'md5 -c' as in: http://www.fourmilab.ch/md5/ and do and 'exit 0' if and only if both retun zero. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia''http://www.sv.lnf.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)sv.lnf.it tel +39-0434-842711 fax +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/chi_siamo/5xmille.php (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
Re: [wpkg-users] Replace xmlhttp download with curl/wget
Hi Marco, On 14.07.2011 15:13, Marco Gaiarin wrote: Mandi! heiko.hel...@horiba.com In chel di` si favelave... Great idea! But this'll require another external tool, because hash calculation via WSH is a world of slow. I've speaked about a script exactly for that. It is trivial in a CMD script to execute firstly curl/wget to download files, check exit code and if '0', run 'md5 -c' as in: http://www.fourmilab.ch/md5/ and do and 'exit 0' if and only if both retun zero. Is there a reason this could not be done using existing commands? At first look this request somehow looks like someone would use the download tag just to launch an external program or script with some parameters. This is done on install/upgrade/downgrade/remove if the download tag is used. But to execute commands there is already a tag: install /, upgrade /, remove / and downgrade /. There is no requirement that these commands can only launch installers. In fact it's even the case that many packages use these commands to do preparation or post-install work. So simply calling a script with some parameters (e.g. download URL, download target, MD5-sum) the script could take care about downloading and checksumming the file. If anything fails it just returns an exit code different to 0 to indicate an error so WPKG will stop the installation. I know that the download / tag is very limited but the reasons for it have been listed already in this discussion. But I am not sure if there is good reason to abuse the download tag just to invoke some external download tools when such a thing can be done already using existing functionality. Well, probably I am just weary today but somebody might explain in detail why using existing commands is not an option or why the download tag would be much better to achieve this. I agree that using a special download command flag would be possible too. I would then prefer to call a cmd script (included in wplkg distribution) with all required parameters like downloader.cmd URL target MD5 So this script could handle download and verification as needed using any tool you like and any flags suitable for your environment (e.g. proxy flags). Well, still I am not sure where the advantage over using this script as one of the first commands would be. br, Rainer - wpkg-users mailing list archives http://lists.wpkg.org/pipermail/wpkg-users/ ___ wpkg-users mailing list wpkg-users@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/wpkg-users
