On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier <[email protected]> wrote: > On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote: >> w.r.t MSVC builds, it looks like we need entries in $contrib_extraincludes >> in src/tools/msvc/Mkvcbuild.pm at the very least. > > If our goal is to put back to green the Windows nodes as quick as > possible, we could bypass their build this way , and we'll > additionally need to update the install script and > vcregress.pl/contribcheck to bypass those modules accordingly. Now I > don't think that we should make the things produced inconsistent.
OK, attached are two patches to put back the buildfarm nodes using MSVC to green - 0001 adds support for build and installation of the new transform modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that this patch is enough to put back the buildfarm to a green state for MSVC, but it disables the regression tests for those modules, something addressed in the next patch... - 0002 adds support for regression tests for the new modules. The thing is that we need to check the major version version of Python available in configuration and choose what are the extensions to preload before the tests run. It is a little bit hacky... But it has the merit to work, and I am not sure we could have a cleaner solution by looking at the Makefiles of the transform modules that use currently conditions based on $(python_majorversion). Regards, -- Michael
From dedbdb5815b3b0450fcf6991156eb8ae022bd04a Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Tue, 28 Apr 2015 02:13:47 -0700 Subject: [PATCH 1/2] Fix MSVC build for transform modules The following contrib/ modules have their build fixed in the MSVC scripts: - hstore_plperl - hstore_plpython - ltree_plpython With this patch the MSVC build and installation will work correctly with the transforms. However their test is still disabled as some adjustments are needed for plpython transforms. --- src/tools/msvc/Install.pm | 9 +- src/tools/msvc/Mkvcbuild.pm | 314 ++++++++++++++++++++++++++++---------------- src/tools/msvc/vcregress.pl | 10 +- 3 files changed, 212 insertions(+), 121 deletions(-) diff --git a/src/tools/msvc/Install.pm b/src/tools/msvc/Install.pm index bfcdf50..b617835 100644 --- a/src/tools/msvc/Install.pm +++ b/src/tools/msvc/Install.pm @@ -439,9 +439,12 @@ sub CopyContribFiles while (my $d = readdir($D)) { # These configuration-based exclusions must match vcregress.pl - next if ($d eq "uuid-ossp" && !defined($config->{uuid})); - next if ($d eq "sslinfo" && !defined($config->{openssl})); - next if ($d eq "xml2" && !defined($config->{xml})); + next if ($d eq "uuid-ossp" && !defined($config->{uuid})); + next if ($d eq "sslinfo" && !defined($config->{openssl})); + next if ($d eq "xml2" && !defined($config->{xml})); + next if ($d eq "hstore_plperl" && !defined($config->{perl})); + next if ($d eq "hstore_plpython" && !defined($config->{python})); + next if ($d eq "ltree_plpython" && !defined($config->{python})); next if ($d eq "sepgsql"); CopySubdirFiles($subdir, $d, $config, $target); diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm index cfb9b24..9d66368 100644 --- a/src/tools/msvc/Mkvcbuild.pm +++ b/src/tools/msvc/Mkvcbuild.pm @@ -46,7 +46,11 @@ my $contrib_extraincludes = my $contrib_extrasource = { 'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ], 'seg' => [ 'contrib/seg/segscan.l', 'contrib/seg/segparse.y' ], }; -my @contrib_excludes = ('pgcrypto', 'commit_ts', 'intagg', 'sepgsql'); +my @contrib_excludes = ( + 'commit_ts', 'hstore_plperl', + 'hstore_plpython', 'intagg', + 'ltree_plpython', 'pgcrypto', + 'sepgsql'); # Set of variables for frontend modules my $frontend_defines = { 'initdb' => 'FRONTEND' }; @@ -176,119 +180,6 @@ sub mkvcbuild $plpgsql->AddFiles('src/pl/plpgsql/src', 'pl_gram.y'); $plpgsql->AddReference($postgres); - if ($solution->{options}->{perl}) - { - my $plperlsrc = "src/pl/plperl/"; - my $plperl = - $solution->AddProject('plperl', 'dll', 'PLs', 'src/pl/plperl'); - $plperl->AddIncludeDir($solution->{options}->{perl} . '/lib/CORE'); - $plperl->AddDefine('PLPERL_HAVE_UID_GID'); - foreach my $xs ('SPI.xs', 'Util.xs') - { - (my $xsc = $xs) =~ s/\.xs/.c/; - if (Solution::IsNewer("$plperlsrc$xsc", "$plperlsrc$xs")) - { - my $xsubppdir = first { -e "$_/ExtUtils/xsubpp" } @INC; - print "Building $plperlsrc$xsc...\n"; - system( $solution->{options}->{perl} - . '/bin/perl ' - . "$xsubppdir/ExtUtils/xsubpp -typemap " - . $solution->{options}->{perl} - . '/lib/ExtUtils/typemap ' - . "$plperlsrc$xs " - . ">$plperlsrc$xsc"); - if ((!(-f "$plperlsrc$xsc")) || -z "$plperlsrc$xsc") - { - unlink("$plperlsrc$xsc"); # if zero size - die "Failed to create $xsc.\n"; - } - } - } - if (Solution::IsNewer( - 'src/pl/plperl/perlchunks.h', - 'src/pl/plperl/plc_perlboot.pl') - || Solution::IsNewer( - 'src/pl/plperl/perlchunks.h', - 'src/pl/plperl/plc_trusted.pl')) - { - print 'Building src/pl/plperl/perlchunks.h ...' . "\n"; - my $basedir = getcwd; - chdir 'src/pl/plperl'; - system( $solution->{options}->{perl} - . '/bin/perl ' - . 'text2macro.pl ' - . '--strip="^(\#.*|\s*)$$" ' - . 'plc_perlboot.pl plc_trusted.pl ' - . '>perlchunks.h'); - chdir $basedir; - if ((!(-f 'src/pl/plperl/perlchunks.h')) - || -z 'src/pl/plperl/perlchunks.h') - { - unlink('src/pl/plperl/perlchunks.h'); # if zero size - die 'Failed to create perlchunks.h' . "\n"; - } - } - if (Solution::IsNewer( - 'src/pl/plperl/plperl_opmask.h', - 'src/pl/plperl/plperl_opmask.pl')) - { - print 'Building src/pl/plperl/plperl_opmask.h ...' . "\n"; - my $basedir = getcwd; - chdir 'src/pl/plperl'; - system( $solution->{options}->{perl} - . '/bin/perl ' - . 'plperl_opmask.pl ' - . 'plperl_opmask.h'); - chdir $basedir; - if ((!(-f 'src/pl/plperl/plperl_opmask.h')) - || -z 'src/pl/plperl/plperl_opmask.h') - { - unlink('src/pl/plperl/plperl_opmask.h'); # if zero size - die 'Failed to create plperl_opmask.h' . "\n"; - } - } - $plperl->AddReference($postgres); - my @perl_libs = - grep { /perl\d+.lib$/ } - glob($solution->{options}->{perl} . '\lib\CORE\perl*.lib'); - if (@perl_libs == 1) - { - $plperl->AddLibrary($perl_libs[0]); - } - else - { - die "could not identify perl library version"; - } - } - - if ($solution->{options}->{python}) - { - - # Attempt to get python version and location. - # Assume python.exe in specified dir. - open(P, - $solution->{options}->{python} - . "\\python -c \"import sys;print(sys.prefix);print(str(sys.version_info[0])+str(sys.version_info[1]))\" |" - ) || die "Could not query for python version!\n"; - my $pyprefix = <P>; - chomp($pyprefix); - my $pyver = <P>; - chomp($pyver); - close(P); - - # Sometimes (always?) if python is not present, the execution - # appears to work, but gives no data... - die "Failed to query python for version information\n" - if (!(defined($pyprefix) && defined($pyver))); - - my $pymajorver = substr($pyver, 0, 1); - my $plpython = $solution->AddProject('plpython' . $pymajorver, - 'dll', 'PLs', 'src/pl/plpython'); - $plpython->AddIncludeDir($pyprefix . '/include'); - $plpython->AddLibrary($pyprefix . "/Libs/python$pyver.lib"); - $plpython->AddReference($postgres); - } - if ($solution->{options}->{tcl}) { my $pltcl = @@ -571,6 +462,139 @@ sub mkvcbuild closedir($D); } + # Build Perl and Python modules after contrib/ modules to satisfy some + # dependencies with transform contrib modules, like hstore_plpython + # ltree_plpython and hstore_plperl. + if ($solution->{options}->{python}) + { + # Attempt to get python version and location. + # Assume python.exe in specified dir. + open(P, + $solution->{options}->{python} + . "\\python -c \"import sys;print(sys.prefix);print(str(sys.version_info[0])+str(sys.version_info[1]))\" |" + ) || die "Could not query for python version!\n"; + my $pyprefix = <P>; + chomp($pyprefix); + my $pyver = <P>; + chomp($pyver); + close(P); + + # Sometimes (always?) if python is not present, the execution + # appears to work, but gives no data... + die "Failed to query python for version information\n" + if (!(defined($pyprefix) && defined($pyver))); + + my $pymajorver = substr($pyver, 0, 1); + my $pymajorver = substr($pyver, 0, 1); + my $plpython = $solution->AddProject('plpython' . $pymajorver, + 'dll', 'PLs', 'src/pl/plpython'); + $plpython->AddIncludeDir($pyprefix . '/include'); + $plpython->AddLibrary($pyprefix . "/Libs/python$pyver.lib"); + $plpython->AddReference($postgres); + + # Add transform modules dependent on plpython + AddTransformModule('hstore_plpython' . $pymajorver, + 'contrib/hstore_plpython', + 'plpython' . $pymajorver, + 'src/pl/plpython', 'hstore', + 'contrib/hstore'); + AddTransformModule('ltree_plpython' . $pymajorver, + 'contrib/ltree_plpython', + 'plpython' . $pymajorver, + 'src/pl/plpython', 'ltree', + 'contrib/ltree'); + } + + if ($solution->{options}->{perl}) + { + my $plperlsrc = "src/pl/plperl/"; + my $plperl = + $solution->AddProject('plperl', 'dll', 'PLs', 'src/pl/plperl'); + $plperl->AddIncludeDir($solution->{options}->{perl} . '/lib/CORE'); + $plperl->AddDefine('PLPERL_HAVE_UID_GID'); + foreach my $xs ('SPI.xs', 'Util.xs') + { + (my $xsc = $xs) =~ s/\.xs/.c/; + if (Solution::IsNewer("$plperlsrc$xsc", "$plperlsrc$xs")) + { + my $xsubppdir = first { -e "$_/ExtUtils/xsubpp" } @INC; + print "Building $plperlsrc$xsc...\n"; + system( $solution->{options}->{perl} + . '/bin/perl ' + . "$xsubppdir/ExtUtils/xsubpp -typemap " + . $solution->{options}->{perl} + . '/lib/ExtUtils/typemap ' + . "$plperlsrc$xs " + . ">$plperlsrc$xsc"); + if ((!(-f "$plperlsrc$xsc")) || -z "$plperlsrc$xsc") + { + unlink("$plperlsrc$xsc"); # if zero size + die "Failed to create $xsc.\n"; + } + } + } + if (Solution::IsNewer( + 'src/pl/plperl/perlchunks.h', + 'src/pl/plperl/plc_perlboot.pl') + || Solution::IsNewer( + 'src/pl/plperl/perlchunks.h', + 'src/pl/plperl/plc_trusted.pl')) + { + print 'Building src/pl/plperl/perlchunks.h ...' . "\n"; + my $basedir = getcwd; + chdir 'src/pl/plperl'; + system( $solution->{options}->{perl} + . '/bin/perl ' + . 'text2macro.pl ' + . '--strip="^(\#.*|\s*)$$" ' + . 'plc_perlboot.pl plc_trusted.pl ' + . '>perlchunks.h'); + chdir $basedir; + if ((!(-f 'src/pl/plperl/perlchunks.h')) + || -z 'src/pl/plperl/perlchunks.h') + { + unlink('src/pl/plperl/perlchunks.h'); # if zero size + die 'Failed to create perlchunks.h' . "\n"; + } + } + if (Solution::IsNewer( + 'src/pl/plperl/plperl_opmask.h', + 'src/pl/plperl/plperl_opmask.pl')) + { + print 'Building src/pl/plperl/plperl_opmask.h ...' . "\n"; + my $basedir = getcwd; + chdir 'src/pl/plperl'; + system( $solution->{options}->{perl} + . '/bin/perl ' + . 'plperl_opmask.pl ' + . 'plperl_opmask.h'); + chdir $basedir; + if ((!(-f 'src/pl/plperl/plperl_opmask.h')) + || -z 'src/pl/plperl/plperl_opmask.h') + { + unlink('src/pl/plperl/plperl_opmask.h'); # if zero size + die 'Failed to create plperl_opmask.h' . "\n"; + } + } + $plperl->AddReference($postgres); + my @perl_libs = + grep { /perl\d+.lib$/ } + glob($solution->{options}->{perl} . '\lib\CORE\perl*.lib'); + if (@perl_libs == 1) + { + $plperl->AddLibrary($perl_libs[0]); + } + else + { + die "could not identify perl library version"; + } + + # Add transform module dependent on plperl + AddTransformModule('hstore_plperl', 'contrib/hstore_plperl', + 'plperl', 'src/pl/plperl', + 'hstore', 'contrib/hstore'); + } + $mf = Project::read_file('src/backend/utils/mb/conversion_procs/Makefile'); $mf =~ s{\\\r?\n}{}g; @@ -680,6 +704,68 @@ sub AddSimpleFrontend return $p; } +# Add a simple transform module +sub AddTransformModule +{ + my $n = shift; + my $n_src = shift; + my $pl_proj_name = shift; + my $pl_src = shift; + my $transform_name = shift; + my $transform_src = shift; + + my $transform_proj = undef; + foreach my $proj (@{ $solution->{projects}->{'contrib'} }) + { + if ($proj->{name} eq $transform_name) + { + $transform_proj = $proj; + last; + } + } + die "could not find base module $transform_name for transform module $n" + if (!defined($transform_proj)); + + my $pl_proj = undef; + foreach my $proj (@{ $solution->{projects}->{'PLs'} }) + { + if ($proj->{name} eq $pl_proj_name) + { + $pl_proj = $proj; + last; + } + } + die "could not find PL $pl_proj_name for transform module $n" + if (!defined($pl_proj)); + + my $p = $solution->AddProject($n, 'dll', 'contrib', $n_src); + for my $file (glob("$n_src/*.c")) + { + $p->AddFile($file); + } + $p->AddReference($postgres); + + # Add PL dependencies + $p->AddIncludeDir($pl_src); + $p->AddReference($pl_proj); + $p->AddIncludeDir($pl_proj->{includes}); + foreach my $pl_lib (@{$pl_proj->{libraries}}) + { + $p->AddLibrary($pl_lib); + } + + # Add base module dependencies + $p->AddIncludeDir($transform_src); + $p->AddIncludeDir($transform_proj->{includes}); + foreach my $trans_lib (@{$transform_proj->{libraries}}) + { + $p->AddLibrary($trans_lib); + } + $p->AddReference($transform_proj); + + return $p; +} + # Add a simple contrib project sub AddContrib { diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl index 0153f05..8c071a5 100644 --- a/src/tools/msvc/vcregress.pl +++ b/src/tools/msvc/vcregress.pl @@ -262,9 +262,12 @@ sub contribcheck foreach my $module (glob("*")) { # these configuration-based exclusions must match Install.pm - next if ($module eq "uuid-ossp" && !defined($config->{uuid})); - next if ($module eq "sslinfo" && !defined($config->{openssl})); - next if ($module eq "xml2" && !defined($config->{xml})); + next if ($module eq "uuid-ossp" && !defined($config->{uuid})); + next if ($module eq "sslinfo" && !defined($config->{openssl})); + next if ($module eq "xml2" && !defined($config->{xml})); + next if ($module eq "hstore_plperl" && !defined($config->{perl})); + next if ($module eq "hstore_plpython"); + next if ($module eq "ltree_plpython"); next if ($module eq "sepgsql"); subdircheck("$topdir/contrib", $module); @@ -375,7 +378,6 @@ sub fetchRegressOpts $m =~ s{\\\r?\n}{}g; if ($m =~ /^\s*REGRESS_OPTS\s*=(.*)/m) { - # Substitute known Makefile variables, then ignore options that retain # an unhandled variable reference. Ignore anything that isn't an # option starting with "--". -- 2.3.6
From 7aa4d458e9abed2569fe0cc69277d904b94b10fc Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Tue, 28 Apr 2015 05:53:49 -0700 Subject: [PATCH 2/2] Enable version-specific tests of plpython transforms in MSVC Those tests need to check for the version of Python available on system in order to select the extensions that should be loaded for the tests. --- src/tools/msvc/vcregress.pl | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl index 8c071a5..92798dc 100644 --- a/src/tools/msvc/vcregress.pl +++ b/src/tools/msvc/vcregress.pl @@ -244,6 +244,31 @@ sub subdircheck print "Checking $module\n"; my @tests = fetchTests(); my @opts = fetchRegressOpts(); + + # Add some options for transform modules, see their respective + # Makefile for more details regarding Python-version specific + # dependencies. + if ($module eq "hstore_plpython" || + $module eq "ltree_plpython") + { + die "Python not enabled in configuration" + if !defined($config->{python}); + + # Get major version number of Python. + open(P, + $config->{python} + . "\\python -c \"import sys;print(str(sys.version_info[0]))\" |" + ) || die "Could not query for python version!\n"; + my $pyver = <P>; + chomp($pyver); + close(P); + if ($pyver eq "2") + { + push @opts, "--load-extension=plpythonu"; + push @opts, '--load-extension=' . $module . 'u'; + } + } + my @args = ( "${tmp_installdir}/bin/pg_regress", "--bindir=${tmp_installdir}/bin", @@ -266,8 +291,8 @@ sub contribcheck next if ($module eq "sslinfo" && !defined($config->{openssl})); next if ($module eq "xml2" && !defined($config->{xml})); next if ($module eq "hstore_plperl" && !defined($config->{perl})); - next if ($module eq "hstore_plpython"); - next if ($module eq "ltree_plpython"); + next if ($module eq "hstore_plpython" && !defined($config->{python})); + next if ($module eq "ltree_plpython" && !defined($config->{python})); next if ($module eq "sepgsql"); subdircheck("$topdir/contrib", $module); -- 2.3.6
-- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
