Pavel, I happened to hit this. I'm not sure how you worked around, as I tried to export SDKROOT=macosx but that didn't work for me. Do you have a patch or a series of commands I can run?
Thanks, -- Davide On Thu, Nov 16, 2017 at 4:25 AM, Pavel Labath via lldb-dev <lldb-dev@lists.llvm.org> wrote: > Right, after learning way more than I ever wanted to know about xcrun, > I think I see the issue. When running with empty SDKROOT variable, > xcrun sets SDKROOT to "/" when running clang: > $ SDKROOT= xcrun --no-cache --log --verbose clang > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c > -o > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.so > -framework Python -Xlinker -dylib -v > xcrun: note: PATH = '/usr/bin:/bin' > xcrun: note: SDKROOT = '/' > ^^^^^ WRONG. there are no > python headers there > > ..... > > env SDKROOT=/ > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c > -o > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.so > -framework Python -Xlinker -dylib -v > > ... > > #include "..." search starts here: > #include <...> search starts here: > > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/include > > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include > /usr/include > /System/Library/Frameworks (framework directory) > /Library/Frameworks (framework directory) > ^^^^ clang gets the include path wrong > > End of search list. > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c:15:10: > fatal error: 'Python/Python.h' file not found > #include <Python/Python.h> > ^~~~~~~~~~~~~~~~~ > 1 error generated. > ^^^^^ and errors out > > > > On the other hand, if I invoke xcrun with SDKROOT=macosx, everything > works just fine: > > $ SDKROOT=macosx xcrun --no-cache --log --verbose clang > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c > -o > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.so > -framework Python -Xlinker -dylib -v > xcrun: note: looking up SDK with > '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk > macosx -version Path' > xcrun: note: PATH = '/usr/bin:/bin' > xcrun: note: SDKROOT = 'macosx' > xcrun: note: TOOLCHAINS = '' > xcrun: note: DEVELOPER_DIR = '/Applications/Xcode.app/Contents/Developer' > xcrun: note: XCODE_DEVELOPER_USR_PATH = '' > xcrun: note: xcrun_db = > '/var/folders/bt/tws6gynx0ws1cc4ss53_pvqm0000gq/T/xcrun_db' > xcrun: note: lookup resolved to: > '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk' > xcrun: note: PATH = '/usr/bin:/bin' > xcrun: note: SDKROOT = > '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk' > ^^^^^^ correct. a valid SDK is located there > > ... > > env > SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c > -o > /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.so > -framework Python -Xlinker -dylib -v > > ... > > #include "..." search starts here: > #include <...> search starts here: > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/include > > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include > > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include > > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks > (framework directory) > > ^^^^^ include path is good and compilation succeeds > > > I've checked this on 10.12 and the xcrun behavior is same there (with > the difference that 10.12 does contain headers in > /System/Library/Frameworks. > > It seems we could fix this by changing the invocation in dotest to > explicitly request SDKROOT=macosx. As far, as I can tell, this would > only break if someone tried to run dotest with iOS as host, but I > don't think that's ever going to be supported(?) > > What do you think? > > > On 15 November 2017 at 20:25, Jim Ingham <jing...@apple.com> wrote: >> The build should be finding the version in the SDK within Xcode. I do have >> the CommandLineTools directory on my system, but it doesn’t have an SDK >> directory in it. I wonder if that is causing the problem? >> >> One thing to check, do: >> >> $ xcrun -find clang >> >> Does that find clang in the DeveloperTools directory? If so try: >> >> sudo xcode-select —switch /Applications/Xcode.app >> >> Then everything should point to the toolchain & SDK’s in Xcode. If that >> doesn’t help you might try moving the DeveloperTools aside and see if things >> work then. Xcode should not need that to work. >> >> Jim >> >> >>> On Nov 15, 2017, at 10:33 AM, Pavel Labath <lab...@google.com> wrote: >>> >>> Thanks for the reply, Jim. >>> >>> As far as I can tell, we already have Xcode on that machine (I only >>> have shell access there). >>> $ xcodebuild -version >>> Xcode 9.1 >>> Build version 9B55 >>> $ xcode-select -version >>> xcode-select version 2349. >>> >>> >>> BTW, this is the list of Python.h files on that machine: >>> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h >>> /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h >>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h >>> >>> I guess you were referring to the third one. That one seems to have >>> the "Headers" symlink and everything, but for some reason clang is not >>> picking it up. Do I need to run some fancy xcode-select or set some >>> environment variables? >>> >>> I guess the reason I am able to compile lldb just fine is that cmake >>> is smart enough to find the right python framework. I'll try comparing >>> the cmake command lines with the one we run from dotest. >>> >>> Additional issue which may complicate things is that this was not a >>> clean install, but an upgrade. >>> >>> >>> >>> On 15 November 2017 at 18:16, Jim Ingham <jing...@apple.com> wrote: >>>> Xcode and general development on macOS has moved from using headers in the >>>> base OS (/System/Library/Frameworks…) for building programs to using SDK’s >>>> to contain all the header files. The installed tools will know how to >>>> find the correct SDK. >>>> >>>> On a clean install of 10.13 there are no headers anywhere in the >>>> /System/Library/Frameworks frameworks; I’m a little surprised you can >>>> build anything w/o an Xcode install. There’s a “command line tools” >>>> package in the Developer tools that puts some stuff back in /usr but that >>>> has had some issues, mostly it overwrites the /usr/bin/clang etc. tools >>>> that Xcode uses - which are just shims that find the correct version of >>>> the tools in the xcode-selected Xcode - with fixed versions of the tools >>>> which can lead to subtle errors if you ever try to use Xcode, so I don’t >>>> suggest that. >>>> >>>> I think you need to put an Xcode install on your bot. >>>> >>>> Jim >>>> >>>>> On Nov 15, 2017, at 8:20 AM, Pavel Labath via lldb-dev >>>>> <lldb-dev@lists.llvm.org> wrote: >>>>> >>>>> On a somewhat tangential note: >>>>> >>>>> Is anyone actually using this crashinfo hook? It looks like this could >>>>> be useful in the old dotest days, when we were running all of the >>>>> tests in a single python process, but now with the parallel test >>>>> runner spawning a new process for each test file and with piping test >>>>> results through a socket, it seems much less useful. If noone is using >>>>> that functionality, maybe we could "fix" the problem by deleting it. >>>>> >>>>> >>>>> On 15 November 2017 at 15:17, Pavel Labath <lab...@google.com> wrote: >>>>>> Hello lldb-dev, >>>>>> >>>>>> We've just updated our mac buildbot to 10.13.1 (from 10.10.x), and >>>>>> we're having trouble with the lldb test suite. All of the tests are >>>>>> failing with the following error: >>>>>> >>>>>> /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c:15:10: >>>>>> fatal error: 'Python/Python.h' file not found >>>>>> #include <Python/Python.h> >>>>>> ^~~~~~~~~~~~~~~~~ >>>>>> 1 error generated. >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/Users/lldb_build/lldbSlave/buildDir/scripts/../llvm/tools/lldb/test/dotest.py", >>>>>> line 7, in <module> >>>>>> lldbsuite.test.run_suite() >>>>>> File >>>>>> "/Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/dotest.py", >>>>>> line 1120, in run_suite >>>>>> configuration.setupCrashInfoHook() >>>>>> File >>>>>> "/Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/configuration.py", >>>>>> line 51, in setupCrashInfoHook >>>>>> raise Exception('command failed: "{}"'.format(cmd)) >>>>>> Exception: command failed: "SDKROOT= xcrun clang >>>>>> /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.c >>>>>> -o >>>>>> /Users/lldb_build/lldbSlave/buildDir/llvm/tools/lldb/packages/Python/lldbsuite/test/crashinfo.so >>>>>> -framework Python -Xlinker -dylib" >>>>>> >>>>>> >>>>>> It seems that this is happening because the buildbot is missing the >>>>>> /System/Library/Frameworks/Python.framework/Headers symlink (this link >>>>>> is present on my mac machine, which is still on 10.12). The rest of >>>>>> the framework seems to be there (e.g. the file >>>>>> /System/Library/Frameworks/Python.framework/Versions/Current/include/python2.7/Python.h >>>>>> is present), just this symlink is missing. I cannot even create it >>>>>> manually as System Integrity Protection will not let me do that. >>>>>> >>>>>> Do you have any idea what went wrong? >>>>>> >>>>>> thanks, >>>>>> pl >>>>>> >>>>>> PS: If it helps anything, the version reported by clang is: >>>>>> Apple LLVM version 9.0.0 (clang-900.0.38) >>>>>> Target: x86_64-apple-darwin17.2.0 >>>>>> Thread model: posix >>>>>> InstalledDir: >>>>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin >>>>> _______________________________________________ >>>>> lldb-dev mailing list >>>>> lldb-dev@lists.llvm.org >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>>> >> > _______________________________________________ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev