Re: r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Bruno Cardoso Lopes via cfe-commits]

Looks like r261551's fault. Reverted in r261654.

On Mon, Feb 22, 2016 at 11:12 PM, Bruno Cardoso Lopes
 wrote:
> Reverted r261552 & r261556 in r261613. Waiting for next win bot
> results to see if it's necessary to revert r261551 as well.
>
> On Mon, Feb 22, 2016 at 7:16 PM, Bruno Cardoso Lopes
>  wrote:
>> Yes, but because of a related but different issue. The same as here:
>> http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10026
>>
>> I've been trying to figure it out but it's been hard without a Unix
>> system available. I'll keep investigating and revert later tonight in
>> case I can't guess it.
>>
>> On Mon, Feb 22, 2016 at 7:08 PM, Sean Silva  wrote:
>>> The bot is still red:
>>> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast
>>>
>>> On Mon, Feb 22, 2016 at 6:48 PM, Bruno Cardoso Lopes
>>>  wrote:

 Hi Sean,

 This is hopefully fixed in r261556.

 On Mon, Feb 22, 2016 at 5:45 PM, Sean Silva  wrote:
 > This or r261551 seem to be causing a build failure:
 >
 > http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/527
 >
 > -- Sean SIlva
 >
 > On Mon, Feb 22, 2016 at 10:41 AM, Bruno Cardoso Lopes via cfe-commits
 >  wrote:
 >>
 >> Author: bruno
 >> Date: Mon Feb 22 12:41:09 2016
 >> New Revision: 261552
 >>
 >> URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
 >> Log:
 >> [VFS] Add 'overlay-relative' field to YAML files
 >>
 >> The VFS overlay mapping between virtual paths and real paths is done
 >> through
 >> the 'external-contents' entries in YAML files, which contains hardcoded
 >> paths
 >> to the real files.
 >>
 >> When a module compilation crashes, headers are dumped into
 >> .cache/vfs
 >> directory and are mapped via the .cache/vfs/vfs.yaml. The script
 >> generated for reproduction uses -ivfsoverlay pointing to file to gather
 >> the
 >> mapping between virtual paths and files inside .cache/vfs.
 >> Currently, we
 >> are only capable of reproducing such crashes in the same machine as
 >> they
 >> happen, because of the hardcoded paths in 'external-contents'.
 >>
 >> To be able to reproduce a crash in another machine, this patch
 >> introduces
 >> a new
 >> option in the VFS yaml file called 'overlay-relative'. When it's equal
 >> to
 >> 'true' it means that the provided path to the YAML file through the
 >> -ivfsoverlay option should also be used to prefix the final path for
 >> every
 >> 'external-contents'.
 >>
 >> Example, given the invocation snippet "... -ivfsoverlay
 >> .cache/vfs/vfs.yaml" and the following entry in the yaml file:
 >>
 >> "overlay-relative": "true",
 >> "roots": [
 >> ...
 >>   "type": "directory",
 >>   "name": "/usr/include",
 >>   "contents": [
 >> {
 >>   "type": "file",
 >>   "name": "stdio.h",
 >>   "external-contents": "/usr/include/stdio.h"
 >> },
 >> ...
 >>
 >> Here, a file manager request for virtual "/usr/include/stdio.h", that
 >> will
 >> map
 >> into real path
 >> "//.cache/vfs/usr/include/stdio.h.
 >>
 >> This is a useful feature for debugging module crashes in machines other
 >> than
 >> the one where the error happened.
 >>
 >> Differential Revision: http://reviews.llvm.org/D17457
 >>
 >> rdar://problem/24499339
 >>
 >> Added:
 >> cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
 >>   - copied, changed from r261551,
 >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
 >> Modified:
 >> cfe/trunk/include/clang/Basic/VirtualFileSystem.h
 >> cfe/trunk/lib/Basic/VirtualFileSystem.cpp
 >> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
 >> cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
 >> cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
 >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
 >>
 >> Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
 >> URL:
 >>
 >> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
 >>
 >>
 >> ==
 >> --- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
 >> +++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22
 >> 12:41:09
 >> 2016
 >> @@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
 >>  IntrusiveRefCntPtr
 >>  getVFSFromYAML(std::unique_ptr Buffer,
 >> llvm::SourceMgr::DiagHandlerTy DiagHandler,

Re: r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Bruno Cardoso Lopes via cfe-commits]

Reverted r261552 & r261556 in r261613. Waiting for next win bot
results to see if it's necessary to revert r261551 as well.

On Mon, Feb 22, 2016 at 7:16 PM, Bruno Cardoso Lopes
 wrote:
> Yes, but because of a related but different issue. The same as here:
> http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10026
>
> I've been trying to figure it out but it's been hard without a Unix
> system available. I'll keep investigating and revert later tonight in
> case I can't guess it.
>
> On Mon, Feb 22, 2016 at 7:08 PM, Sean Silva  wrote:
>> The bot is still red:
>> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast
>>
>> On Mon, Feb 22, 2016 at 6:48 PM, Bruno Cardoso Lopes
>>  wrote:
>>>
>>> Hi Sean,
>>>
>>> This is hopefully fixed in r261556.
>>>
>>> On Mon, Feb 22, 2016 at 5:45 PM, Sean Silva  wrote:
>>> > This or r261551 seem to be causing a build failure:
>>> >
>>> > http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/527
>>> >
>>> > -- Sean SIlva
>>> >
>>> > On Mon, Feb 22, 2016 at 10:41 AM, Bruno Cardoso Lopes via cfe-commits
>>> >  wrote:
>>> >>
>>> >> Author: bruno
>>> >> Date: Mon Feb 22 12:41:09 2016
>>> >> New Revision: 261552
>>> >>
>>> >> URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
>>> >> Log:
>>> >> [VFS] Add 'overlay-relative' field to YAML files
>>> >>
>>> >> The VFS overlay mapping between virtual paths and real paths is done
>>> >> through
>>> >> the 'external-contents' entries in YAML files, which contains hardcoded
>>> >> paths
>>> >> to the real files.
>>> >>
>>> >> When a module compilation crashes, headers are dumped into
>>> >> .cache/vfs
>>> >> directory and are mapped via the .cache/vfs/vfs.yaml. The script
>>> >> generated for reproduction uses -ivfsoverlay pointing to file to gather
>>> >> the
>>> >> mapping between virtual paths and files inside .cache/vfs.
>>> >> Currently, we
>>> >> are only capable of reproducing such crashes in the same machine as
>>> >> they
>>> >> happen, because of the hardcoded paths in 'external-contents'.
>>> >>
>>> >> To be able to reproduce a crash in another machine, this patch
>>> >> introduces
>>> >> a new
>>> >> option in the VFS yaml file called 'overlay-relative'. When it's equal
>>> >> to
>>> >> 'true' it means that the provided path to the YAML file through the
>>> >> -ivfsoverlay option should also be used to prefix the final path for
>>> >> every
>>> >> 'external-contents'.
>>> >>
>>> >> Example, given the invocation snippet "... -ivfsoverlay
>>> >> .cache/vfs/vfs.yaml" and the following entry in the yaml file:
>>> >>
>>> >> "overlay-relative": "true",
>>> >> "roots": [
>>> >> ...
>>> >>   "type": "directory",
>>> >>   "name": "/usr/include",
>>> >>   "contents": [
>>> >> {
>>> >>   "type": "file",
>>> >>   "name": "stdio.h",
>>> >>   "external-contents": "/usr/include/stdio.h"
>>> >> },
>>> >> ...
>>> >>
>>> >> Here, a file manager request for virtual "/usr/include/stdio.h", that
>>> >> will
>>> >> map
>>> >> into real path
>>> >> "//.cache/vfs/usr/include/stdio.h.
>>> >>
>>> >> This is a useful feature for debugging module crashes in machines other
>>> >> than
>>> >> the one where the error happened.
>>> >>
>>> >> Differential Revision: http://reviews.llvm.org/D17457
>>> >>
>>> >> rdar://problem/24499339
>>> >>
>>> >> Added:
>>> >> cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
>>> >>   - copied, changed from r261551,
>>> >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>>> >> Modified:
>>> >> cfe/trunk/include/clang/Basic/VirtualFileSystem.h
>>> >> cfe/trunk/lib/Basic/VirtualFileSystem.cpp
>>> >> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
>>> >> cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
>>> >> cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
>>> >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>>> >>
>>> >> Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
>>> >> URL:
>>> >>
>>> >> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
>>> >>
>>> >>
>>> >> ==
>>> >> --- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
>>> >> +++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22
>>> >> 12:41:09
>>> >> 2016
>>> >> @@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
>>> >>  IntrusiveRefCntPtr
>>> >>  getVFSFromYAML(std::unique_ptr Buffer,
>>> >> llvm::SourceMgr::DiagHandlerTy DiagHandler,
>>> >> +   StringRef YAMLFilePath,
>>> >> void *DiagContext = nullptr,
>>> >> IntrusiveRefCntPtr ExternalFS =
>>> >> getRealFileSystem());
>>> >>
>>> >> @@ -323,6 +324,8 @@ struct YAMLVFSEntry {
>>> >>  class YAMLVFSWriter 

Re: r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Bruno Cardoso Lopes via cfe-commits]

Yes, but because of a related but different issue. The same as here:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10026

I've been trying to figure it out but it's been hard without a Unix
system available. I'll keep investigating and revert later tonight in
case I can't guess it.

On Mon, Feb 22, 2016 at 7:08 PM, Sean Silva  wrote:
> The bot is still red:
> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast
>
> On Mon, Feb 22, 2016 at 6:48 PM, Bruno Cardoso Lopes
>  wrote:
>>
>> Hi Sean,
>>
>> This is hopefully fixed in r261556.
>>
>> On Mon, Feb 22, 2016 at 5:45 PM, Sean Silva  wrote:
>> > This or r261551 seem to be causing a build failure:
>> >
>> > http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/527
>> >
>> > -- Sean SIlva
>> >
>> > On Mon, Feb 22, 2016 at 10:41 AM, Bruno Cardoso Lopes via cfe-commits
>> >  wrote:
>> >>
>> >> Author: bruno
>> >> Date: Mon Feb 22 12:41:09 2016
>> >> New Revision: 261552
>> >>
>> >> URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
>> >> Log:
>> >> [VFS] Add 'overlay-relative' field to YAML files
>> >>
>> >> The VFS overlay mapping between virtual paths and real paths is done
>> >> through
>> >> the 'external-contents' entries in YAML files, which contains hardcoded
>> >> paths
>> >> to the real files.
>> >>
>> >> When a module compilation crashes, headers are dumped into
>> >> .cache/vfs
>> >> directory and are mapped via the .cache/vfs/vfs.yaml. The script
>> >> generated for reproduction uses -ivfsoverlay pointing to file to gather
>> >> the
>> >> mapping between virtual paths and files inside .cache/vfs.
>> >> Currently, we
>> >> are only capable of reproducing such crashes in the same machine as
>> >> they
>> >> happen, because of the hardcoded paths in 'external-contents'.
>> >>
>> >> To be able to reproduce a crash in another machine, this patch
>> >> introduces
>> >> a new
>> >> option in the VFS yaml file called 'overlay-relative'. When it's equal
>> >> to
>> >> 'true' it means that the provided path to the YAML file through the
>> >> -ivfsoverlay option should also be used to prefix the final path for
>> >> every
>> >> 'external-contents'.
>> >>
>> >> Example, given the invocation snippet "... -ivfsoverlay
>> >> .cache/vfs/vfs.yaml" and the following entry in the yaml file:
>> >>
>> >> "overlay-relative": "true",
>> >> "roots": [
>> >> ...
>> >>   "type": "directory",
>> >>   "name": "/usr/include",
>> >>   "contents": [
>> >> {
>> >>   "type": "file",
>> >>   "name": "stdio.h",
>> >>   "external-contents": "/usr/include/stdio.h"
>> >> },
>> >> ...
>> >>
>> >> Here, a file manager request for virtual "/usr/include/stdio.h", that
>> >> will
>> >> map
>> >> into real path
>> >> "//.cache/vfs/usr/include/stdio.h.
>> >>
>> >> This is a useful feature for debugging module crashes in machines other
>> >> than
>> >> the one where the error happened.
>> >>
>> >> Differential Revision: http://reviews.llvm.org/D17457
>> >>
>> >> rdar://problem/24499339
>> >>
>> >> Added:
>> >> cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
>> >>   - copied, changed from r261551,
>> >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>> >> Modified:
>> >> cfe/trunk/include/clang/Basic/VirtualFileSystem.h
>> >> cfe/trunk/lib/Basic/VirtualFileSystem.cpp
>> >> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
>> >> cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
>> >> cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
>> >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>> >>
>> >> Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
>> >> URL:
>> >>
>> >> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
>> >>
>> >>
>> >> ==
>> >> --- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
>> >> +++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22
>> >> 12:41:09
>> >> 2016
>> >> @@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
>> >>  IntrusiveRefCntPtr
>> >>  getVFSFromYAML(std::unique_ptr Buffer,
>> >> llvm::SourceMgr::DiagHandlerTy DiagHandler,
>> >> +   StringRef YAMLFilePath,
>> >> void *DiagContext = nullptr,
>> >> IntrusiveRefCntPtr ExternalFS =
>> >> getRealFileSystem());
>> >>
>> >> @@ -323,6 +324,8 @@ struct YAMLVFSEntry {
>> >>  class YAMLVFSWriter {
>> >>std::vector Mappings;
>> >>Optional IsCaseSensitive;
>> >> +  Optional IsOverlayRelative;
>> >> +  std::string OverlayDir;
>> >>
>> >>  public:
>> >>YAMLVFSWriter() {}
>> >> @@ -330,6 +333,11 @@ public:
>> >>void setCaseSensitivity(bool CaseSensitive) {
>> >>  IsCaseSensitive = CaseSensitive;
>> >>}
>> >> +  

Re: r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Sean Silva via cfe-commits]

The bot is still red:
http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast

On Mon, Feb 22, 2016 at 6:48 PM, Bruno Cardoso Lopes <
bruno.card...@gmail.com> wrote:

> Hi Sean,
>
> This is hopefully fixed in r261556.
>
> On Mon, Feb 22, 2016 at 5:45 PM, Sean Silva  wrote:
> > This or r261551 seem to be causing a build failure:
> >
> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/527
> >
> > -- Sean SIlva
> >
> > On Mon, Feb 22, 2016 at 10:41 AM, Bruno Cardoso Lopes via cfe-commits
> >  wrote:
> >>
> >> Author: bruno
> >> Date: Mon Feb 22 12:41:09 2016
> >> New Revision: 261552
> >>
> >> URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
> >> Log:
> >> [VFS] Add 'overlay-relative' field to YAML files
> >>
> >> The VFS overlay mapping between virtual paths and real paths is done
> >> through
> >> the 'external-contents' entries in YAML files, which contains hardcoded
> >> paths
> >> to the real files.
> >>
> >> When a module compilation crashes, headers are dumped into
> >> .cache/vfs
> >> directory and are mapped via the .cache/vfs/vfs.yaml. The script
> >> generated for reproduction uses -ivfsoverlay pointing to file to gather
> >> the
> >> mapping between virtual paths and files inside .cache/vfs.
> >> Currently, we
> >> are only capable of reproducing such crashes in the same machine as they
> >> happen, because of the hardcoded paths in 'external-contents'.
> >>
> >> To be able to reproduce a crash in another machine, this patch
> introduces
> >> a new
> >> option in the VFS yaml file called 'overlay-relative'. When it's equal
> to
> >> 'true' it means that the provided path to the YAML file through the
> >> -ivfsoverlay option should also be used to prefix the final path for
> every
> >> 'external-contents'.
> >>
> >> Example, given the invocation snippet "... -ivfsoverlay
> >> .cache/vfs/vfs.yaml" and the following entry in the yaml file:
> >>
> >> "overlay-relative": "true",
> >> "roots": [
> >> ...
> >>   "type": "directory",
> >>   "name": "/usr/include",
> >>   "contents": [
> >> {
> >>   "type": "file",
> >>   "name": "stdio.h",
> >>   "external-contents": "/usr/include/stdio.h"
> >> },
> >> ...
> >>
> >> Here, a file manager request for virtual "/usr/include/stdio.h", that
> will
> >> map
> >> into real path
> "//.cache/vfs/usr/include/stdio.h.
> >>
> >> This is a useful feature for debugging module crashes in machines other
> >> than
> >> the one where the error happened.
> >>
> >> Differential Revision: http://reviews.llvm.org/D17457
> >>
> >> rdar://problem/24499339
> >>
> >> Added:
> >> cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
> >>   - copied, changed from r261551,
> >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
> >> Modified:
> >> cfe/trunk/include/clang/Basic/VirtualFileSystem.h
> >> cfe/trunk/lib/Basic/VirtualFileSystem.cpp
> >> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
> >> cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
> >> cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
> >> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
> >>
> >> Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
> >> URL:
> >>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
> >>
> >>
> ==
> >> --- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
> >> +++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22
> 12:41:09
> >> 2016
> >> @@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
> >>  IntrusiveRefCntPtr
> >>  getVFSFromYAML(std::unique_ptr Buffer,
> >> llvm::SourceMgr::DiagHandlerTy DiagHandler,
> >> +   StringRef YAMLFilePath,
> >> void *DiagContext = nullptr,
> >> IntrusiveRefCntPtr ExternalFS =
> >> getRealFileSystem());
> >>
> >> @@ -323,6 +324,8 @@ struct YAMLVFSEntry {
> >>  class YAMLVFSWriter {
> >>std::vector Mappings;
> >>Optional IsCaseSensitive;
> >> +  Optional IsOverlayRelative;
> >> +  std::string OverlayDir;
> >>
> >>  public:
> >>YAMLVFSWriter() {}
> >> @@ -330,6 +333,11 @@ public:
> >>void setCaseSensitivity(bool CaseSensitive) {
> >>  IsCaseSensitive = CaseSensitive;
> >>}
> >> +  void setOverlayDir(StringRef OverlayDirectory) {
> >> +IsOverlayRelative = true;
> >> +OverlayDir.assign(OverlayDirectory.str());
> >> +  }
> >> +
> >>void write(llvm::raw_ostream );
> >>  };
> >>
> >>
> >> Modified: cfe/trunk/lib/Basic/VirtualFileSystem.cpp
> >> URL:
> >>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/VirtualFileSystem.cpp?rev=261552=261551=261552=diff
> >>
> >>
> ==
> >> --- cfe/trunk/lib/Basic/VirtualFileSystem.cpp 

Re: r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Bruno Cardoso Lopes via cfe-commits]

Hi Sean,

This is hopefully fixed in r261556.

On Mon, Feb 22, 2016 at 5:45 PM, Sean Silva  wrote:
> This or r261551 seem to be causing a build failure:
> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/527
>
> -- Sean SIlva
>
> On Mon, Feb 22, 2016 at 10:41 AM, Bruno Cardoso Lopes via cfe-commits
>  wrote:
>>
>> Author: bruno
>> Date: Mon Feb 22 12:41:09 2016
>> New Revision: 261552
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
>> Log:
>> [VFS] Add 'overlay-relative' field to YAML files
>>
>> The VFS overlay mapping between virtual paths and real paths is done
>> through
>> the 'external-contents' entries in YAML files, which contains hardcoded
>> paths
>> to the real files.
>>
>> When a module compilation crashes, headers are dumped into
>> .cache/vfs
>> directory and are mapped via the .cache/vfs/vfs.yaml. The script
>> generated for reproduction uses -ivfsoverlay pointing to file to gather
>> the
>> mapping between virtual paths and files inside .cache/vfs.
>> Currently, we
>> are only capable of reproducing such crashes in the same machine as they
>> happen, because of the hardcoded paths in 'external-contents'.
>>
>> To be able to reproduce a crash in another machine, this patch introduces
>> a new
>> option in the VFS yaml file called 'overlay-relative'. When it's equal to
>> 'true' it means that the provided path to the YAML file through the
>> -ivfsoverlay option should also be used to prefix the final path for every
>> 'external-contents'.
>>
>> Example, given the invocation snippet "... -ivfsoverlay
>> .cache/vfs/vfs.yaml" and the following entry in the yaml file:
>>
>> "overlay-relative": "true",
>> "roots": [
>> ...
>>   "type": "directory",
>>   "name": "/usr/include",
>>   "contents": [
>> {
>>   "type": "file",
>>   "name": "stdio.h",
>>   "external-contents": "/usr/include/stdio.h"
>> },
>> ...
>>
>> Here, a file manager request for virtual "/usr/include/stdio.h", that will
>> map
>> into real path "//.cache/vfs/usr/include/stdio.h.
>>
>> This is a useful feature for debugging module crashes in machines other
>> than
>> the one where the error happened.
>>
>> Differential Revision: http://reviews.llvm.org/D17457
>>
>> rdar://problem/24499339
>>
>> Added:
>> cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
>>   - copied, changed from r261551,
>> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>> Modified:
>> cfe/trunk/include/clang/Basic/VirtualFileSystem.h
>> cfe/trunk/lib/Basic/VirtualFileSystem.cpp
>> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
>> cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
>> cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
>> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>>
>> Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
>>
>> ==
>> --- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
>> +++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22 12:41:09
>> 2016
>> @@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
>>  IntrusiveRefCntPtr
>>  getVFSFromYAML(std::unique_ptr Buffer,
>> llvm::SourceMgr::DiagHandlerTy DiagHandler,
>> +   StringRef YAMLFilePath,
>> void *DiagContext = nullptr,
>> IntrusiveRefCntPtr ExternalFS =
>> getRealFileSystem());
>>
>> @@ -323,6 +324,8 @@ struct YAMLVFSEntry {
>>  class YAMLVFSWriter {
>>std::vector Mappings;
>>Optional IsCaseSensitive;
>> +  Optional IsOverlayRelative;
>> +  std::string OverlayDir;
>>
>>  public:
>>YAMLVFSWriter() {}
>> @@ -330,6 +333,11 @@ public:
>>void setCaseSensitivity(bool CaseSensitive) {
>>  IsCaseSensitive = CaseSensitive;
>>}
>> +  void setOverlayDir(StringRef OverlayDirectory) {
>> +IsOverlayRelative = true;
>> +OverlayDir.assign(OverlayDirectory.str());
>> +  }
>> +
>>void write(llvm::raw_ostream );
>>  };
>>
>>
>> Modified: cfe/trunk/lib/Basic/VirtualFileSystem.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/VirtualFileSystem.cpp?rev=261552=261551=261552=diff
>>
>> ==
>> --- cfe/trunk/lib/Basic/VirtualFileSystem.cpp (original)
>> +++ cfe/trunk/lib/Basic/VirtualFileSystem.cpp Mon Feb 22 12:41:09 2016
>> @@ -790,6 +790,7 @@ public:
>>  /// All configuration options are optional.
>>  ///   'case-sensitive': 
>>  ///   'use-external-names': 
>> +///   'overlay-relative': 
>>  ///
>>  /// Virtual directories are represented as
>>  /// \verbatim
>> @@ -832,6 +833,10 @@ class RedirectingFileSystem : public vfs
>>

Re: r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Sean Silva via cfe-commits]

This or r261551 seem to be causing a build failure:
http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/527

-- Sean SIlva

On Mon, Feb 22, 2016 at 10:41 AM, Bruno Cardoso Lopes via cfe-commits <
cfe-commits@lists.llvm.org> wrote:

> Author: bruno
> Date: Mon Feb 22 12:41:09 2016
> New Revision: 261552
>
> URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
> Log:
> [VFS] Add 'overlay-relative' field to YAML files
>
> The VFS overlay mapping between virtual paths and real paths is done
> through
> the 'external-contents' entries in YAML files, which contains hardcoded
> paths
> to the real files.
>
> When a module compilation crashes, headers are dumped into .cache/vfs
> directory and are mapped via the .cache/vfs/vfs.yaml. The script
> generated for reproduction uses -ivfsoverlay pointing to file to gather the
> mapping between virtual paths and files inside .cache/vfs.
> Currently, we
> are only capable of reproducing such crashes in the same machine as they
> happen, because of the hardcoded paths in 'external-contents'.
>
> To be able to reproduce a crash in another machine, this patch introduces
> a new
> option in the VFS yaml file called 'overlay-relative'. When it's equal to
> 'true' it means that the provided path to the YAML file through the
> -ivfsoverlay option should also be used to prefix the final path for every
> 'external-contents'.
>
> Example, given the invocation snippet "... -ivfsoverlay
> .cache/vfs/vfs.yaml" and the following entry in the yaml file:
>
> "overlay-relative": "true",
> "roots": [
> ...
>   "type": "directory",
>   "name": "/usr/include",
>   "contents": [
> {
>   "type": "file",
>   "name": "stdio.h",
>   "external-contents": "/usr/include/stdio.h"
> },
> ...
>
> Here, a file manager request for virtual "/usr/include/stdio.h", that will
> map
> into real path "//.cache/vfs/usr/include/stdio.h.
>
> This is a useful feature for debugging module crashes in machines other
> than
> the one where the error happened.
>
> Differential Revision: http://reviews.llvm.org/D17457
>
> rdar://problem/24499339
>
> Added:
> cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
>   - copied, changed from r261551,
> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
> Modified:
> cfe/trunk/include/clang/Basic/VirtualFileSystem.h
> cfe/trunk/lib/Basic/VirtualFileSystem.cpp
> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
> cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
> cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
> cfe/trunk/test/Modules/crash-vfs-path-traversal.m
>
> Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
> URL:
> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
>
> ==
> --- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
> +++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22 12:41:09
> 2016
> @@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
>  IntrusiveRefCntPtr
>  getVFSFromYAML(std::unique_ptr Buffer,
> llvm::SourceMgr::DiagHandlerTy DiagHandler,
> +   StringRef YAMLFilePath,
> void *DiagContext = nullptr,
> IntrusiveRefCntPtr ExternalFS =
> getRealFileSystem());
>
> @@ -323,6 +324,8 @@ struct YAMLVFSEntry {
>  class YAMLVFSWriter {
>std::vector Mappings;
>Optional IsCaseSensitive;
> +  Optional IsOverlayRelative;
> +  std::string OverlayDir;
>
>  public:
>YAMLVFSWriter() {}
> @@ -330,6 +333,11 @@ public:
>void setCaseSensitivity(bool CaseSensitive) {
>  IsCaseSensitive = CaseSensitive;
>}
> +  void setOverlayDir(StringRef OverlayDirectory) {
> +IsOverlayRelative = true;
> +OverlayDir.assign(OverlayDirectory.str());
> +  }
> +
>void write(llvm::raw_ostream );
>  };
>
>
> Modified: cfe/trunk/lib/Basic/VirtualFileSystem.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/VirtualFileSystem.cpp?rev=261552=261551=261552=diff
>
> ==
> --- cfe/trunk/lib/Basic/VirtualFileSystem.cpp (original)
> +++ cfe/trunk/lib/Basic/VirtualFileSystem.cpp Mon Feb 22 12:41:09 2016
> @@ -790,6 +790,7 @@ public:
>  /// All configuration options are optional.
>  ///   'case-sensitive': 
>  ///   'use-external-names': 
> +///   'overlay-relative': 
>  ///
>  /// Virtual directories are represented as
>  /// \verbatim
> @@ -832,6 +833,10 @@ class RedirectingFileSystem : public vfs
>std::vector Roots;
>/// \brief The file system to use for external references.
>IntrusiveRefCntPtr ExternalFS;
> +  /// If IsRelativeOverlay is set, this represents the directory
> +  /// path that should be prefixed to each 'external-contents' 

r261552 - [VFS] Add 'overlay-relative' field to YAML files

[Bruno Cardoso Lopes via cfe-commits]

Author: bruno
Date: Mon Feb 22 12:41:09 2016
New Revision: 261552

URL: http://llvm.org/viewvc/llvm-project?rev=261552=rev
Log:
[VFS] Add 'overlay-relative' field to YAML files

The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.

When a module compilation crashes, headers are dumped into .cache/vfs
directory and are mapped via the .cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside .cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.

To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.

Example, given the invocation snippet "... -ivfsoverlay
.cache/vfs/vfs.yaml" and the following entry in the yaml file:

"overlay-relative": "true",
"roots": [
...
  "type": "directory",
  "name": "/usr/include",
  "contents": [
{
  "type": "file",
  "name": "stdio.h",
  "external-contents": "/usr/include/stdio.h"
},
...

Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "//.cache/vfs/usr/include/stdio.h.

This is a useful feature for debugging module crashes in machines other than
the one where the error happened.

Differential Revision: http://reviews.llvm.org/D17457

rdar://problem/24499339

Added:
cfe/trunk/test/Modules/crash-vfs-relative-overlay.m
  - copied, changed from r261551, 
cfe/trunk/test/Modules/crash-vfs-path-traversal.m
Modified:
cfe/trunk/include/clang/Basic/VirtualFileSystem.h
cfe/trunk/lib/Basic/VirtualFileSystem.cpp
cfe/trunk/lib/Frontend/CompilerInvocation.cpp
cfe/trunk/lib/Frontend/ModuleDependencyCollector.cpp
cfe/trunk/test/Modules/crash-vfs-path-symlink-component.m
cfe/trunk/test/Modules/crash-vfs-path-traversal.m

Modified: cfe/trunk/include/clang/Basic/VirtualFileSystem.h
URL: 
http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/VirtualFileSystem.h?rev=261552=261551=261552=diff
==
--- cfe/trunk/include/clang/Basic/VirtualFileSystem.h (original)
+++ cfe/trunk/include/clang/Basic/VirtualFileSystem.h Mon Feb 22 12:41:09 2016
@@ -310,6 +310,7 @@ llvm::sys::fs::UniqueID getNextVirtualUn
 IntrusiveRefCntPtr
 getVFSFromYAML(std::unique_ptr Buffer,
llvm::SourceMgr::DiagHandlerTy DiagHandler,
+   StringRef YAMLFilePath,
void *DiagContext = nullptr,
IntrusiveRefCntPtr ExternalFS = 
getRealFileSystem());
 
@@ -323,6 +324,8 @@ struct YAMLVFSEntry {
 class YAMLVFSWriter {
   std::vector Mappings;
   Optional IsCaseSensitive;
+  Optional IsOverlayRelative;
+  std::string OverlayDir;
 
 public:
   YAMLVFSWriter() {}
@@ -330,6 +333,11 @@ public:
   void setCaseSensitivity(bool CaseSensitive) {
 IsCaseSensitive = CaseSensitive;
   }
+  void setOverlayDir(StringRef OverlayDirectory) {
+IsOverlayRelative = true;
+OverlayDir.assign(OverlayDirectory.str());
+  }
+
   void write(llvm::raw_ostream );
 };
 

Modified: cfe/trunk/lib/Basic/VirtualFileSystem.cpp
URL: 
http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/VirtualFileSystem.cpp?rev=261552=261551=261552=diff
==
--- cfe/trunk/lib/Basic/VirtualFileSystem.cpp (original)
+++ cfe/trunk/lib/Basic/VirtualFileSystem.cpp Mon Feb 22 12:41:09 2016
@@ -790,6 +790,7 @@ public:
 /// All configuration options are optional.
 ///   'case-sensitive': 
 ///   'use-external-names': 
+///   'overlay-relative': 
 ///
 /// Virtual directories are represented as
 /// \verbatim
@@ -832,6 +833,10 @@ class RedirectingFileSystem : public vfs
   std::vector Roots;
   /// \brief The file system to use for external references.
   IntrusiveRefCntPtr ExternalFS;
+  /// If IsRelativeOverlay is set, this represents the directory
+  /// path that should be prefixed to each 'external-contents' entry
+  /// when reading from YAML files.
+  std::string ExternalContentsPrefixDir;
 
   /// @name Configuration
   /// @{
@@ -841,6 +846,10 @@ class RedirectingFileSystem : public vfs
   /// Currently, case-insensitive matching only works correctly with ASCII.
   bool CaseSensitive;
 
+  /// IsRelativeOverlay marks whether a IsExternalContentsPrefixDir path must
+  /// be prefixed in every 'external-contents' when reading from YAML files.
+  bool IsRelativeOverlay = false;
+
   /// \brief Whether to