Re: Make test drill, next steps, call for help with Winetest
Francois Gouget wrote: > On Sat, 27 Oct 2007, Francois Gouget wrote: > In fact I've been able to verify that it works in real life since > winetest has not been updated since last monday :-( > Compilation problems? Yeah, I guess due to the new d3dx8 tests. Have a look at http://quisquiliae.physics.gla.ac.uk/crossbuild-logs/ for more details. -- Cheers, Paul.
Re: Make test drill, next steps, call for help with Winetest
On Sat, 27 Oct 2007, Francois Gouget wrote:
[...]
> There are other options to specify which version of winetest.exe to
> grab, to set the timeout on the tests execution, etc.
>
> Things still on the todo list:
> * the script also grabs the winetest.exe signature and attempts to
> verify it. But I don't know where to find the corresponding public key
> (651FD487) so I don't know if this really works.
Hmm, as pointed out by Detlef, there a link on Paul's home page pointing
to his public key and the required key is in there. Then it's just a
matter of telling gpg to use it which I have now done and the signature
checking code seems to work as intended.
> * the script gets the 'latest' winetest.exe version by default, but
> when there are compilation problems I guess this may stay stuck at the
> same revision for some time. So it would be nice if the script could
> know which version it's getting so it could skip the tests if that
> version is too old, or has already been tested.
The script can now use a local cache containing the already downloaded
files, and can be told to skip the tests if it finds that a file has not
changed since the last time it has been used in a given vm+snapshot
combination.
In fact I've been able to verify that it works in real life since
winetest has not been updated since last monday :-(
Compilation problems?
I've attached the latest version of the script.
--
Francois Gouget <[EMAIL PROTECTED]> http://fgouget.free.fr/
Hiroshima '45 - Czernobyl '86 - Windows '95#!/bin/sh
# Copyright (C) 2007 Francois Gouget
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA
name0=`basename "$0"`
verbose=""
base_url="http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt";
vmware="/usr/local/opt/vmware/bin"
if [ -z "$DISPLAY" ]
then
DISPLAY=":0.0"
export DISPLAY
fi
verbose()
{
if [ -n "$verbose" ]
then
echo "$@"
fi
}
vm_is_busy()
{
vm="$1"
test -f "$vm.WRITELOCK"
return $?
}
vm_is_running()
{
# For VM matching purposes, only keep the last two elements of the path
# to avoid trouble with symbolic links
vm_match=`dirname "$opt_vm"`
vm_match=`basename "$vm_match"`/`basename "$opt_vm"`
vm_match=`echo "$vm_match" | sed -e 's/\([^a-zA-Z0-9_ /]\)/[\1]/g'`
"$vmware/vmrun" list | egrep "$vm_match" >/dev/null
return $?
}
cached_wget()
{
url="$1"
dst="$2"
cache="$3"
if [ -n "$cache" ]
then
filename=`basename "$url"`
(
cd "$cache" &&
wget -N "$url" &&
cp -pr "$filename" "$dst"
)
else
wget -O "$dst" "$url"
fi
}
### Main
opt_vm=""
opt_tag_prefix=""
opt_tag=""
opt_cache=""
opt_version=""
opt_check_sig=""
opt_submit=""
opt_timeout=""
opt_autorun=""
opt_snapshot=""
opt_cddevice=""
opt_usage=""
while [ $# -gt 0 ]
do
arg="$1"
shift
case "$arg" in
--tag-prefix)
if [ -n "$opt_tag_prefix" ]
then
echo "$name0:error: --tag-prefix can only be used once" >&2
opt_usage="2"
break
elif [ $# -eq 0 ]
then
echo "$name0:error: missing argument for the --tag-prefix option"
>&2
opt_usage="2"
break
else
opt_tag_prefix="$1"
shift
fi
;;
--tag)
if [ -n "$opt_tag" ]
then
echo "$name0:error: --tag can only be used once" >&2
opt_usage="2"
break
elif [ $# -eq 0 ]
then
echo "$name0:error: missing argument for the --tag option" >&2
opt_usage="2"
break
else
opt_tag="$1"
shift
fi
;;
--cache)
if [ -n "$opt_cache" ]
then
echo "$name0:error: --cache can only be used once" >&2
opt_usage="2"
break
elif [ $# -eq 0 ]
then
echo "$name0:error: missing argument for the --cache option" >&2
opt_usage="2"
break
else
opt_cache="$1"
shift
fi
;;
--skip-old)
opt_skip_old="1"
;;
--no-skip-old)
opt_skip_old="0"
;;
--version)
if [ -n "$opt_version" ]
then
echo "$name0:error: --ver
Re: Make test drill, next steps, call for help with Winetest
Francois Gouget wrote:
> While I prefer the autorun approach because it has fewer dependencies on
> the Windows side and thus allows me to test in as clean a Windows as
> desired, your approach could be pretty useful for testing on a real
> Windows machine. Maybe if you post your script with some instructions it
> will inspire some people to set up automated testing on their real
> Windows machines. I'm sure it would make the Direct3D guys happy (if the
> testers have Nvidia ;-) ).
>
>
I forgot... I did a C hack. cc -o runstuff.exe runstuff
Then copied runstuff.exe to c:\win95\Start-Menu\Program\Autostart
#include
#include
int main ()
{
chdir ("/tmp");
unlink ("winetest-latest.exe");
system ("wget
http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/winetest-latest.exe";);
system ("./winetest-latest.exe -q -c -t JakobAuto");
system ("shutdown -h now");
return 0;
}
About virtual machines starting in a clean state, nothing stops anyone
from booting linux
in between and "dd if=windows_part.bin of=/dev/hdb1" and then reboot
into Windows.
This way you could get consistent Windows regression testing on real
hardware.
regards,
Jakob
Re: Make test drill, next steps, call for help with Winetest
On Sat, 27 Oct 2007, Jakob Eriksson wrote: [...] > I achieved much the same thing by putting a wget script in the > Auto-startup folder of Windows. Then I copy in this vmware windows > virtual machine and start it. Then Windows itself downloads winetest > and upon completion, does a shutdown -h now. (Via cygwin.) I often > find cygwin a relief when working with Windows. While I prefer the autorun approach because it has fewer dependencies on the Windows side and thus allows me to test in as clean a Windows as desired, your approach could be pretty useful for testing on a real Windows machine. Maybe if you post your script with some instructions it will inspire some people to set up automated testing on their real Windows machines. I'm sure it would make the Direct3D guys happy (if the testers have Nvidia ;-) ). -- Francois Gouget <[EMAIL PROTECTED]> http://fgouget.free.fr/ Any sufficiently advanced Operating System is indistinguishable from Linux
Re: Make test drill, next steps, call for help with Winetest
On Sun, 28 Oct 2007, Detlef Riekenberg wrote: > On Sa, 2007-10-27 at 14:03 +0200, Francois Gouget wrote: > > > > Things still on the todo list: > > * the script also grabs the winetest.exe signature and attempts to > > verify it. But I don't know where to find the corresponding public > > key > > (651FD487) so I don't know if this really works. > > Paul's homepage: http://www.astro.gla.ac.uk/users/paulm/ > > > $ gpg --list-keys paulm > pub 1024D/BF0F03E9 2000-11-07 > uid Paul Millar (Physics & Astronomy) > sub 1024g/1034BACC 2000-11-07 When I run 'gpg --list-keys paulm' all I get is this first key, but the one I need is the one below... Same thing if I go to the http://wwwkeys.pgp.net/ URL listed on his page. > pub 1024D/651FD487 2002-05-11 > uid quisquiliae (automatic program for testing WINE) > sub 1024g/420F3FE4 2002-05-11 [...] > My script loads the signature and verify the winetest-latest.exe. > When gpg failed to verify the binary, a new winetest is available. > The new binary is downloaded and checked again. > When the check was ok, the winetest-lastest is copied to the destination > and qemu is started for every image. Sounds like a good idea. > To bad, that qemu 0.9.0 crash on "ntdll_text.exe exception" ... Maybe you can try VirtualBox. Apparently it's GPL... Though if it's incompatible with qemu it would force you to make a choice between them. -- Francois Gouget <[EMAIL PROTECTED]> http://fgouget.free.fr/ Avoid the Gates of Hell - use Linux.
Re: Make test drill, next steps, call for help with Winetest
On Sa, 2007-10-27 at 14:03 +0200, Francois Gouget wrote: > > Things still on the todo list: > * the script also grabs the winetest.exe signature and attempts to > verify it. But I don't know where to find the corresponding public > key > (651FD487) so I don't know if this really works. Paul's homepage: http://www.astro.gla.ac.uk/users/paulm/ $ gpg --list-keys paulm pub 1024D/BF0F03E9 2000-11-07 uid Paul Millar (Physics & Astronomy) sub 1024g/1034BACC 2000-11-07 pub 1024D/651FD487 2002-05-11 uid quisquiliae (automatic program for testing WINE) sub 1024g/420F3FE4 2002-05-11 (The email-address was stripped) My script loads the signature and verify the winetest-latest.exe. When gpg failed to verify the binary, a new winetest is available. The new binary is downloaded and checked again. When the check was ok, the winetest-lastest is copied to the destination and qemu is started for every image. To bad, that qemu 0.9.0 crash on "ntdll_text.exe exception" ... -- By by ... Detlef
Re: Make test drill, next steps, call for help with Winetest
Hans Leidekker wrote: > On Saturday 27 October 2007 14:03:02 Francois Gouget wrote: > > >> That's something I've wanted to do for some time and this finally >> spured me to action. So here's a script that will do essentially the >> same thing but with VMware Workstation. >> > > Cool! I think I'll borrow some of your nice extra options. I'm moving > towards VirtualBox now, it's FOSS, as good as VMware for my purposes > and less painful to keep running on a bleeding edge Linux distro. > I achieved much the same thing by putting a wget script in the Auto-startup folder of Windows. Then I copy in this vmware windows virtual machine and start it. Then Windows itself downloads winetest and upon completion, does a shutdown -h now. (Via cygwin.) I often find cygwin a relief when working with Windows. Just FYI. regards, Jakob
Re: Make test drill, next steps, call for help with Winetest
On Saturday 27 October 2007 22:29:00 Jakob Eriksson wrote: > I often find cygwin a relief when working with Windows. Yes, I also run cygwin on my Windows guests. I have openssh running on them and export the Wine tree to the guests via samba so I can do: $ make crosstest $ ssh vm /cygdrive/w/dlls/wininet/tests/wininet_crosstest.exe ftp from my Linux development host. This works for most tests and it gives me a fast edit-compile-test cycle because I don't have to leave the shell I'm working in. -Hans
Re: Make test drill, next steps, call for help with Winetest
On Saturday 27 October 2007 14:03:02 Francois Gouget wrote: > That's something I've wanted to do for some time and this finally > spured me to action. So here's a script that will do essentially the > same thing but with VMware Workstation. Cool! I think I'll borrow some of your nice extra options. I'm moving towards VirtualBox now, it's FOSS, as good as VMware for my purposes and less painful to keep running on a bleeding edge Linux distro. -Hans
Re: Make test drill, next steps, call for help with Winetest
On Mon, 15 Oct 2007, Hans Leidekker wrote:
[...]
> I have written attached script to facilitate automated winetest runs.
> It needs a VirtualBox virtual machine with either Windows or a Wine platform
> configured to run autorun.inf files.
> worry about winetest screwing up your installation.
[...]
That's something I've wanted to do for some time and this finally
spured me to action. So here's a script that will do essentially the
same thing but with VMware Workstation.
Like your script it downloads winetest using wget, then generates an ISO
with an autorun.inf file so the tests will start automatically in the
VM. It also uses snapshots so the state of the virtual machine is
preserved.
Typically you would use it like this:
* I have not found a way to 'insert' the CD if the cdrom device is not
already connected in the VM. So you must first ensure that's the case.
You can do so in a 'winetest' snapshot, the script will automatically
switch to that snapshot.
* Creating a winetest snapshot is probably a good idea anyway, if only
to ensure it's a relatively clean environment.
* Then run the script as follows:
winetest /path/to/virtual/machine.vmx --tag mytests --no-submit
The --no-submit option is so that the tests don't submit their
results to http://test.winehq.org/. The goal here is to avoid sending
lots of duplicate results while you're experimenting. Once you're
confident things work just as expected, remove the option.
There are other options to specify which version of winetest.exe to
grab, to set the timeout on the tests execution, etc.
Things still on the todo list:
* the script also grabs the winetest.exe signature and attempts to
verify it. But I don't know where to find the corresponding public key
(651FD487) so I don't know if this really works.
* the script gets the 'latest' winetest.exe version by default, but
when there are compilation problems I guess this may stay stuck at the
same revision for some time. So it would be nice if the script could
know which version it's getting so it could skip the tests if that
version is too old, or has already been tested.
* I wonder if this works with VMware Server? It would be nice as Server
is free. The script uses snapshots quite a bit though so it may not
work. Let me know how it goes if you try.
--
Francois Gouget <[EMAIL PROTECTED]> http://fgouget.free.fr/
Any sufficiently advanced Operating System is indistinguishable from Linux#!/bin/sh
# Copyright (C) 2007 Francois Gouget
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA
name0=`basename "$0"`
verbose=""
base_url="http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt";
vmware="/usr/local/opt/vmware/bin"
if [ -z "$DISPLAY" ]
then
DISPLAY=":0.0"
export DISPLAY
fi
verbose()
{
if [ -n "$verbose" ]
then
echo "$@"
fi
}
vm_is_busy()
{
vm="$1"
test -f "$vm.WRITELOCK"
return $?
}
vm_is_running()
{
# For VM matching purposes, only keep the last two elements of the path
# to avoid trouble with symbolic links
vm_match=`dirname "$opt_vm"`
vm_match=`basename "$vm_match"`/`basename "$opt_vm"`
vm_match=`echo "$vm_match" | sed -e 's/\([^a-zA-Z0-9_ /]\)/[\1]/g'`
"$vmware/vmrun" list | egrep "$vm_match" >/dev/null
return $?
}
### Main
opt_vm=""
opt_tag=""
opt_version=""
opt_check_sig=""
opt_submit=""
opt_timeout=""
opt_autorun=""
opt_snapshot=""
opt_cddevice=""
opt_usage=""
while [ $# -gt 0 ]
do
arg="$1"
shift
case "$arg" in
--tag)
if [ -n "$opt_tag" ]
then
echo "$name0:error: --tag can only be used once" >&2
opt_usage="2"
break
elif [ $# -eq 0 ]
then
echo "$name0:error: missing argument for the --tag option" >&2
opt_usage="2"
break
else
opt_tag="$1"
shift
fi
;;
--version)
if [ -n "$opt_version" ]
then
echo "$name0:error: --version can only be used once" >&2
opt_usage="2"
break
elif [ $# -eq 0 ]
then
echo "$name0:error: missing argument for the --version option" >&2
opt_usage="2"
break
else
opt_version="$1"
shift
fi
;;
--c
Re: Make test drill, next steps, call for help with Winetest
Jeremy White wrote: >> Why can't you just run the winetest executable under Wine? Or do you mean >> you've >> changed some of the tests and want to see the results immediately? > > Because that's not logically correct. I am trying to run a regression > test on my current source tree. It *may* be identical to the one most > recently used to build winetest.exe, but it is not certain. And shouldn't > runing winetest.exe.so be exactly the same as running winetest.exe, just > skipping a cross compilation step? > I've basically done the same but only to be able to create a new winetest executable for windows. The winetest.exe that's available on http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/ is a bit different from the winetest.exe.so you have in your own tree (mostly resources). I gave it shot and came a bit further: - go to /programs/winetest - create a build.id, build.nfo and tests.url - run 'make dist' - run 'wine winetest-dist.exe.so' When you now run the test it will a create a proper report file with one important lack: the CVS version number of the tests files (have a look in the produced "tests.rc" file). dissect (part of the test.winehq.org 'infrastructure') will choke on that as it expects something like x.yy So no success here either. We can tweak 'dissect' but that you will mean we have all tests in one view and potentially comparing apples and pears/oranges. Any other ideas? -- Cheers, Paul.
Re: Make test drill, next steps, call for help with Winetest
> Why can't you just run the winetest executable under Wine? Or do you mean > you've > changed some of the tests and want to see the results immediately? Because that's not logically correct. I am trying to run a regression test on my current source tree. It *may* be identical to the one most recently used to build winetest.exe, but it is not certain. And shouldn't runing winetest.exe.so be exactly the same as running winetest.exe, just skipping a cross compilation step? Cheers, Jeremy
Re: Make test drill, next steps, call for help with Winetest
Dimi Paun wrote: > > The plan was to have a home page at: > http://test.winehq.org > that would contain the explanation, plus an > index for the tests (hopefully organized nicely). > > That didn't happen... Yeah, I had some thoughts about that that were also discussed in the past but never go the time (and I don't have time in the nearby future unfortunately) to implement any of the thoughts. -- Cheers, Paul.
Re: Make test drill, next steps, call for help with Winetest
Jeremy White wrote: > In my opinion, it needs the following: > > 1. A Wiki page! > > I could not find *any* documentation on this utility. > > For example, I'm still not entirely clear on how to > setup and/or register my build.id so that my results > are available on the web site. (And the rather > non intuitive requirement to 'know' the unpublished > url http://test.winehq.org/data/ is harsh). > Why can't you just run the winetest executable under Wine? Or do you mean you've changed some of the tests and want to see the results immediately? -- Cheers, Paul.
Re: Make test drill, next steps, call for help with Winetest
On Mo, 2007-10-15 at 12:16 -0400, Dimi Paun wrote: > (And the rather > > non intuitive requirement to 'know' the unpublished > > url http://test.winehq.org/data/ is harsh). > > The plan was to have a home page at: > http://test.winehq.org > that would contain the explanation, plus an > index for the tests (hopefully organized nicely). Yes Please: http://bugs.winehq.org/show_bug.cgi?id=3187 -- By by ... Detlef
re: Make test drill, next steps, call for help with Winetest
>> There is a headless operation mode already, it detects >> if you run it as a service, and skips appropriate tests. > > Perhaps on Windows; I'm focused on Linux. How about wine net start winetest? - Dan
Re: Make test drill, next steps, call for help with Winetest
I have written attached script to facilitate automated winetest runs. It needs a VirtualBox virtual machine with either Windows or a Wine platform configured to run autorun.inf files. The script downloads the latest winetest binary and puts it in an iso image, along with an autorun file to trigger its execution when the image is 'inserted' into the virtual drive. A snapshot is used to revert the vm to its original state, so no need to worry about winetest screwing up your installation. -Hans #!/bin/bash if [ $# -lt 2 ]; then echo "Usage: $0 " echo "" echo "vmname: Virtual machine to use (must be in suspended state). echo "tag:Tag built from [-.0-9a-zA-Z] to identify test report. exit 1 fi TMPDIR=/tmp/winetest$$ LOGFILE=$TMPDIR/log mkdir -p $TMPDIR/master echo "downloading winetest binary" wget -P $TMPDIR http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/winetest-latest.exe >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi mv $TMPDIR/winetest-latest.exe $TMPDIR/master/winetest.exe echo "creating autorun.inf" cat > $TMPDIR/master/autorun.inf <<_EOF_ [autorun] open=winetest.exe -q -t $2 _EOF_ echo "creating cd image" mkisofs -o $TMPDIR/winetest.iso $TMPDIR/master >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "checking vm $1 state" VBoxManage showvminfo $1 | grep "^State:" | grep "saved" >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "creating snapshot for vm $1" VBoxManage snapshot $1 take before-winetest >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "starting vm $1" VBoxManage startvm $1 >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "waiting until vm $1 is up and running" RUNNING="false" while [ $RUNNING != "true" ] do VBoxManage showvminfo $1 | grep "^State:" | grep "running" >> $LOGFILE 2>&1 if [ $? -eq 0 ]; then RUNNING="true" fi sleep 1 done echo "attaching cd image to vm $1" VBoxManage controlvm $1 dvdattach $TMPDIR/winetest.iso >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "sleeping while vm $1 runs winetest" sleep 1800 # 30 minutes should be enough for winetest to finish echo "suspending vm $1" VBoxManage controlvm $1 savestate >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi echo "waiting until vm $1 is suspended" SAVED="false" while [ $SAVED != "true" ] do VBoxManage showvminfo $1 | grep "^State:" | grep "saved" >> $LOGFILE 2>&1 if [ $? -eq 0 ]; then SAVED="true" fi sleep 1 done echo "reverting vm $1 to previous state" VBoxManage snapshot $1 discardcurrent -all >> $LOGFILE 2>&1 if [ $? -ne 0 ]; then exit 1 fi rm -rf $TMPDIR exit 0
Re: Make test drill, next steps, call for help with Winetest
On Mon, 2007-10-15 at 11:18 -0500, Jeremy White wrote: > Perhaps on Windows; I'm focused on Linux. Is there > a way to trigger that when doing a 'wine winetest.exe.so'? Maybe "-c" (console mode, no GUI) -- Dimi Paun <[EMAIL PROTECTED]> Lattica, Inc.
Re: Make test drill, next steps, call for help with Winetest
> There is a headless operation mode already, it detects > if you run it as a service, and skips appropriate tests. Perhaps on Windows; I'm focused on Linux. Is there a way to trigger that when doing a 'wine winetest.exe.so'? Cheers, Jeremy
Re: Make test drill, next steps, call for help with Winetest
On Mon, 2007-10-15 at 10:48 -0500, Jeremy White wrote: > In my opinion, it needs the following: > > 1. A Wiki page! > > I could not find *any* documentation on this utility. > > For example, I'm still not entirely clear on how to > setup and/or register my build.id so that my results > are available on the web site. (And the rather > non intuitive requirement to 'know' the unpublished > url http://test.winehq.org/data/ is harsh). The plan was to have a home page at: http://test.winehq.org that would contain the explanation, plus an index for the tests (hopefully organized nicely). That didn't happen... > 2. Headless operation > > From my experiments, there were a number of dialogs > I had to dismiss, and some worrying complaints in > unpacking msvcrtd and msvcrt tests. > > This second point is probably a bit less important > than the first. There is a headless operation mode already, it detects if you run it as a service, and skips appropriate tests. -- Dimi Paun <[EMAIL PROTECTED]> Lattica, Inc.
