Re: Adding CI to our tree (ccache)
On Mon, Mar 07, 2022 at 11:10:54AM -0800, Andres Freund wrote: > On 2022-03-06 10:16:54 -0600, Justin Pryzby wrote: > > On Fri, Mar 04, 2022 at 05:30:03PM -0800, Andres Freund wrote: > > > I tried to use it, but saw that no caching was happening, and debugged > > > it. Which yielded that it can't be used due to the way output files are > > > specified (and due to multiple files, but that can be prevented with an > > > msbuild parameter). > > > > Could you give a hint about to the msbuild param to avoid processing > > multiple > > files with cl.exe? I'm not able to find it... > > /p:UseMultiToolTask=true Wow - thanks ;) > > I don't know about the issue with output filenames .. > > I don't remember the precise details anymore, but it boils down to ccache > requiring the output filename to be specified, but we only specify the output > directory. Or very similar. There's already a problem report and PR for this. I didn't test it, but I hope it'll be fixed in their next minor release. https://github.com/ccache/ccache/issues/1018 -- Justin
Re: Adding CI to our tree (ccache)
Hi, On 2022-03-06 10:16:54 -0600, Justin Pryzby wrote: > On Fri, Mar 04, 2022 at 05:30:03PM -0800, Andres Freund wrote: > > I tried to use it, but saw that no caching was happening, and debugged > > it. Which yielded that it can't be used due to the way output files are > > specified (and due to multiple files, but that can be prevented with an > > msbuild parameter). > > Could you give a hint about to the msbuild param to avoid processing multiple > files with cl.exe? I'm not able to find it... /p:UseMultiToolTask=true > I don't know about the issue with output filenames .. I don't remember the precise details anymore, but it boils down to ccache requiring the output filename to be specified, but we only specify the output directory. Or very similar. Greetings, Andres Freund
Re: Adding CI to our tree (ccache)
On Fri, Mar 04, 2022 at 05:30:03PM -0800, Andres Freund wrote: > I tried to use it, but saw that no caching was happening, and debugged > it. Which yielded that it can't be used due to the way output files are > specified (and due to multiple files, but that can be prevented with an > msbuild parameter). Could you give a hint about to the msbuild param to avoid processing multiple files with cl.exe? I'm not able to find it... I don't know about the issue with output filenames .. -- Justin
Re: Adding CI to our tree (ccache)
Hi, On 2022-03-03 22:56:15 -0600, Justin Pryzby wrote: > On Sun, Feb 20, 2022 at 12:47:31PM -0800, Andres Freund wrote: > > On 2022-02-20 13:36:55 -0600, Justin Pryzby wrote: > > > Have you tried to use the yet-to-be-released ccache with MSVC ? > > > > Yes, it doesn't work, because it requires cl.exe to be used in a specific > > way > > (only a single input file, specific output file naming). Which would > > require a > > decent amount of changes to src/tools/msvc. I think it's more realistic with > > meson etc. > > Did you get to the point that that causes a problem, or did you just realize > that it was a limitation that seems to preclude its use ? I tried to use it, but saw that no caching was happening, and debugged it. Which yielded that it can't be used due to the way output files are specified (and due to multiple files, but that can be prevented with an msbuild parameter). > If so, could you send the branch/commit you had ? This was in a local VM, not cirrus. I ended up making it work with the meson build, after a bit of fiddling. Although bypassing msbuild (by building with ninja, using cl.exe) is a larger win... > The error I'm getting when I try to use ccache involves .rst files, which > don't > exist (and which ccache doesn't know how to find or ignore). > https://cirrus-ci.com/task/5441491957972992 ccache has code to deal with response files. I suspect the problem here is rather that ccache expects the compiler as an argument. > I gather this is the difference between "compiling with MSVC" and compiling > with a visual studio project. I doubt it's related to that. Greetings, Andres Freund
Re: Adding CI to our tree (ccache)
On Sun, Feb 20, 2022 at 12:47:31PM -0800, Andres Freund wrote: > On 2022-02-20 13:36:55 -0600, Justin Pryzby wrote: > > Have you tried to use the yet-to-be-released ccache with MSVC ? > > Yes, it doesn't work, because it requires cl.exe to be used in a specific way > (only a single input file, specific output file naming). Which would require a > decent amount of changes to src/tools/msvc. I think it's more realistic with > meson etc. Did you get to the point that that causes a problem, or did you just realize that it was a limitation that seems to preclude its use ? If so, could you send the branch/commit you had ? The error I'm getting when I try to use ccache involves .rst files, which don't exist (and which ccache doesn't know how to find or ignore). https://cirrus-ci.com/task/5441491957972992 I gather this is the difference between "compiling with MSVC" and compiling with a visual studio project.
Re: Adding CI to our tree (ccache)
On Sun, Feb 20, 2022 at 12:47:31PM -0800, Andres Freund wrote: > > Did you ever try to use clcache (or others) ? > > > > When I tried, it refused to cache because of our debug settings > > (DebugInformationFormat) - which seem to be enabled even in release mode. > > > I wonder if that'll be an issue for ccache, too. I think that line may > > need to > > be conditional on debug mode. > > That's relatively easily solvable by using a different debug format IIRC (/Z7 > or such). Yes. I got that working for CI by overriding with a value from the environment. https://cirrus-ci.com/task/6191974075072512 This is right after rebasing, so it doesn't save anything, but normally cuts build time to 90sec, which isn't impressive, but it's something. BTW, I think it's worth compiling the windows build with optimizations (as I did here). At least with all the tap tests, this pays for itself. I suppose you don't want to use a Release build, but optimizations could be enabled by an(other) environment variable. -- Justin
Re: Adding CI to our tree (ccache)
Hi, On 2022-02-20 13:36:55 -0600, Justin Pryzby wrote: > Have you tried to use the yet-to-be-released ccache with MSVC ? Yes, it doesn't work, because it requires cl.exe to be used in a specific way (only a single input file, specific output file naming). Which would require a decent amount of changes to src/tools/msvc. I think it's more realistic with meson etc. > Also, do you know about msbuild /outputResultsCache ? I don't think it's really usable for what we need. But it's hard to tell. > Did you ever try to use clcache (or others) ? > > When I tried, it refused to cache because of our debug settings > (DebugInformationFormat) - which seem to be enabled even in release mode. > I wonder if that'll be an issue for ccache, too. I think that line may need > to > be conditional on debug mode. That's relatively easily solvable by using a different debug format IIRC (/Z7 or such). Greetings, Andres Freund
Re: Adding CI to our tree (ccache)
Have you tried to use the yet-to-be-released ccache with MSVC ? Also, do you know about msbuild /outputResultsCache ? When I tried that, it gave a bunch of error. https://cirrus-ci.com/task/5697497241747456 |[16:35:13.605] 1>c:\cirrus\pgsql.sln.metaproj : error : MSB4252: Project "c:\cirrus\pgsql.sln" with global properties [c:\cirrus\pgsql.sln] |[16:35:13.615] c:\cirrus\pgsql.sln.metaproj : error : (TrackFileAccess=false; CLToolExe=clcache.exe) [c:\cirrus\pgsql.sln] |[16:35:13.615] c:\cirrus\pgsql.sln.metaproj : error : is building project "c:\cirrus\initdb.vcxproj" with global properties [c:\cirrus\pgsql.sln] |[16:35:13.615] c:\cirrus\pgsql.sln.metaproj : error : (TrackFileAccess=false; CLToolExe=clcache.exe; BuildingSolutionFile=true; CurrentSolutionConfigurationContents= [c:\cirrus\pgsql.sln] |[16:35:13.615] c:\cirrus\pgsql.sln.metaproj : error : Debug|x64 [c:\cirrus\pgsql.sln] |... |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : Debug|x64 [c:\cirrus\pgsql.sln] |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : ; SolutionDir=c:\cirrus\; SolutionExt=.sln; SolutionFileName=pgsql.sln; SolutionName=pgsql; SolutionPath=c:\cirrus\pgsql.sln; Configuration=Debug; Platform=x64) [c:\cirrus\pgsql.sln] |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : with the (default) target(s) but the build result for the built project is not in the engine cache. In isolated builds this could mean one of the following: [c:\cirrus\pgsql.sln] |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : - the reference was called with a target which is not specified in the ProjectReferenceTargets item in project "c:\cirrus\pgsql.sln" [c:\cirrus\pgsql.sln] |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : - the reference was called with global properties that do not match the static graph inferred nodes [c:\cirrus\pgsql.sln] |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : - the reference was not explicitly specified as a ProjectReference item in project "c:\cirrus\pgsql.sln" [c:\cirrus\pgsql.sln] |[16:35:14.518]c:\cirrus\pgsql.sln.metaproj : error : [c:\cirrus\pgsql.sln] |[16:35:14.518] |[16:35:14.518] 0 Warning(s) |[16:35:14.518] 149 Error(s) Did you ever try to use clcache (or others) ? When I tried, it refused to cache because of our debug settings (DebugInformationFormat) - which seem to be enabled even in release mode. I wonder if that'll be an issue for ccache, too. I think that line may need to be conditional on debug mode. https://cirrus-ci.com/task/4808554103177216 |[17:14:28.765] C:\ProgramData\chocolatey\lib\clcache\clcache\clcache.py Expanded commandline '['/c', '/Isrc/include', '/Isrc/include/port/win32', '/Isrc/include/port/win32_msvc', '/Ic:/openssl/1.1/\\include', '/Zi', '/nologo', '/W3', '/WX-', '/diagnostics:column', '/Ox', '/D', 'WIN32', '/D', '_WINDOWS', '/D', '__WINDOWS__', '/D', '__WIN32__', '/D', 'WIN32_STACK_RLIMIT=4194304', '/D', '_CRT_SECURE_NO_DEPRECATE', '/D', '_CRT_NONSTDC_NO_DEPRECATE', '/D', 'FRONTEND', '/D', '_MBCS', '/GF', '/Gm-', '/EHsc', '/MD', '/GS', '/fp:precise', '/Zc:wchar_t', '/Zc:forScope', '/Zc:inline', '/Fo.\\Release\\libpgcommon\\', '/Fd.\\Release\\libpgcommon\\libpgcommon.pdb', '/external:W3', '/Gd', '/TC', '/wd4018', '/wd4244', '/wd4273', '/wd4101', '/wd4102', '/wd4090', '/wd4267', '/FC', '/errorReport:queue', '/MP', 'src/common/archive.c', 'src/common/base64.c', 'src/common/checksum_helper.c', 'src/common/config_info.c', 'src/common/controldata_utils.c', 'src/common/cryptohash_openssl.c', 'src/common/d2s.c', 'src/common/encnames.c', 'src/common/exec.c', 'src/common/f2s.c', 'src/common/fe_memutils.c', 'src/common/file_perm.c', 'src/common/file_utils.c', 'src/common/hashfn.c', 'src/common/hmac_openssl.c', 'src/common/ip.c', 'src/common/jsonapi.c', 'src/common/keywords.c', 'src/common/kwlookup.c', 'src/common/link-canary.c', 'src/common/logging.c', 'src/common/md5_common.c', 'src/common/pg_get_line.c', 'src/common/pg_lzcompress.c', 'src/common/pg_prng.c', 'src/common/pgfnames.c', 'src/common/protocol_openssl.c', 'src/common/psprintf.c', 'src/common/relpath.c', 'src/common/restricted_token.c', 'src/common/rmtree.c', 'src/common/saslprep.c', 'src/common/scram-common.c', 'src/common/sprompt.c', 'src/common/string.c', 'src/common/stringinfo.c', 'src/common/unicode_norm.c', 'src/common/username.c', 'src/common/wait_error.c', 'src/common/wchar.c']' |[17:14:28.765] C:\ProgramData\chocolatey\lib\clcache\clcache\clcache.py Cannot cache invocation as ['/c', '/Isrc/include', '/Isrc/include/port/win32', '/Isrc/include/port/win32_msvc', '/Ic:/openssl/1.1/\\include', '/Zi', '/nologo', '/W3', '/WX-', '/diagnostics:column', '/Ox', '/D', 'WIN32', '/D', '_WINDOWS', '/D', '__WINDOWS__', '/D', '__WIN32__', '/D', 'WIN32_STACK_RLIMIT=4194304', '/D', '_CRT_SECURE_NO_DEPRECATE', '/D', '_CRT_NONSTDC_NO_DEPRECATE', '/D',
