[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/abhina-sree closed https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: @owenca @mydeveloperday Thanks for testing on Windows and taking the time to review the PR. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
mydeveloperday wrote: > > We don't normally commit in clang-format with a unit test > > I assume you mean "without a unit test". In this case the unit test is the > existing test cases. Some fail on z/OS because the files are not read in as > text. It's not really possible to detect this issue on other platforms. I > can't think of a new unit test that wouldn't duplicate what we already have. No its ok.. we kind of have a rule witrh clang-format that all commits need some form of test, but again I can understand this is hard to test. Go ahead. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
mydeveloperday wrote:
Yes I also was able to confirm there was no impact on windows i just wasn't
sure so needed to prove it to myself, I'm ok if you want to commit this into
clang-format, if you break stuff... we'll simply revert it ;-)
https://github.com/llvm/llvm-project/pull/90128
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
owenca wrote:
I tested it on Windows and didn't have a problem. I used LF only, CRLF only,
and mixed line endings for the following:
```
int i;
int j;
```
Running clang-format through all values of the `LineEnding` option, with and
without in-place, seemed ok.
https://github.com/llvm/llvm-project/pull/90128
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: > We don't normally commit in clang-format with a unit test I assume you mean "without a unit test". In this case the unit test is the existing test cases. Some fail on z/OS because the files are not read in as text. It's not really possible to detect this issue on other platforms. I can't think of a new unit test that wouldn't duplicate what we already have. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
perry-ca wrote:
I don't believe this will be a problem for Windows. This is a common problem
that we have fixed in a few places already. I don't have a Windows platform to
confirm. Would you or someone else be able to verify?
https://github.com/llvm/llvm-project/pull/90128
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
mydeveloperday wrote:
Could there be other impacts if the files that aren't utf-8 or contain
internationalize characters in say comments?
https://github.com/llvm/llvm-project/pull/90128
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/mydeveloperday edited https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/mydeveloperday commented: We don't normally commit in clang-format with a unit test https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
mydeveloperday wrote:
On windows won't this have an impact of potentially converting \n to \r\n?
could a clang-format effectively become a unix2dos at the same time (for those
using clang-format on cygwin is that what they want?) can we have a unit test
to understand the impact?
https://github.com/llvm/llvm-project/pull/90128
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/perry-ca updated
https://github.com/llvm/llvm-project/pull/90128
>From bf3ed6819301d9fcc7e001b9e6676d5f4ce66c4d Mon Sep 17 00:00:00 2001
From: Sean Perry
Date: Thu, 25 Apr 2024 15:58:05 -0500
Subject: [PATCH] treat text files as text files so autoconversion is done
---
clang/tools/clang-format/ClangFormat.cpp| 7 ---
llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp | 3 ++-
llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp | 9 ++---
llvm/tools/yaml2obj/yaml2obj.cpp| 2 +-
llvm/utils/lit/lit/builtin_commands/cat.py | 18 --
llvm/utils/lit/lit/llvm/config.py | 7 +++
6 files changed, 36 insertions(+), 10 deletions(-)
diff --git a/clang/tools/clang-format/ClangFormat.cpp
b/clang/tools/clang-format/ClangFormat.cpp
index feb733fe3c9e0b..01f7c6047726e2 100644
--- a/clang/tools/clang-format/ClangFormat.cpp
+++ b/clang/tools/clang-format/ClangFormat.cpp
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
errs() << EC.message() << "\n";
return true;
@@ -558,7 +559,7 @@ static int dumpConfig() {
// Read in the code in case the filename alone isn't enough to detect the
// language.
ErrorOr> CodeOrErr =
-MemoryBuffer::getFileOrSTDIN(FileNames[0]);
+MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
llvm::errs() << EC.message() << "\n";
return 1;
diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
index c3015d895230ea..40ee59c014b09f 100644
--- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
+++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
@@ -95,7 +95,8 @@ static std::vector
getSearchPaths(opt::InputArgList *Args,
// Opens a file. Path has to be resolved already. (used for def file)
std::unique_ptr openFile(const Twine ) {
- ErrorOr> MB =
MemoryBuffer::getFile(Path);
+ ErrorOr> MB =
+ MemoryBuffer::getFile(Path, /*IsText=*/true);
if (std::error_code EC = MB.getError()) {
llvm::errs() << "cannot open file " << Path << ": " << EC.message() <<
"\n";
diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
index 6a5646965df2cf..c5ccd64f116539 100644
--- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
+++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
@@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) {
cl::HideUnrelatedOptions({, ()});
cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
- auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
+ auto OldSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true);
if (!OldSymbolBufOrError)
exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
- auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
+ auto NewSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true);
if (!NewSymbolBufOrError)
exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
- auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
+ auto RemappingBufOrError =
+ MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true);
if (!RemappingBufOrError)
exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp
index b7f5356e22a9e6..4a060e1aad427f 100644
--- a/llvm/tools/yaml2obj/yaml2obj.cpp
+++ b/llvm/tools/yaml2obj/yaml2obj.cpp
@@ -130,7 +130,7 @@ int main(int argc, char **argv) {
}
ErrorOr> Buf =
- MemoryBuffer::getFileOrSTDIN(Input);
+ MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true);
if (!Buf)
return 1;
diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py
b/llvm/utils/lit/lit/builtin_commands/cat.py
index 37f55c0aef210b..6fb2152ef9332d 100644
--- a/llvm/utils/lit/lit/builtin_commands/cat.py
+++ b/llvm/utils/lit/lit/builtin_commands/cat.py
@@ -55,10 +55,24 @@ def main(argv):
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
for filename in filenames:
try:
-fileToCat = open(filename, "rb")
-contents = fileToCat.read()
+contents = None
+is_text = False
+try:
+if sys.platform != "win32":
+fileToCat = open(filename,
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: Failed at `FAIL: BOLT :: RISCV/fake-label-no-entry.c` this time. I'll give it a day or so and try again. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/abhina-sree approved this pull request. LGTM https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: > LGTM, but check why the following test in checks failed: `FAIL: BOLT :: > RISCV/unnamed-sym-no-entry.c` Looks unrelated. I'm kicking off another build. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/perry-ca updated
https://github.com/llvm/llvm-project/pull/90128
>From 2d7034366547b176f89b076eb60bcb44424fae11 Mon Sep 17 00:00:00 2001
From: Sean Perry
Date: Thu, 25 Apr 2024 15:58:05 -0500
Subject: [PATCH] treat text files as text files so autoconversion is done
---
clang/tools/clang-format/ClangFormat.cpp| 7 ---
llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp | 3 ++-
llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp | 9 ++---
llvm/tools/yaml2obj/yaml2obj.cpp| 2 +-
llvm/utils/lit/lit/builtin_commands/cat.py | 18 --
llvm/utils/lit/lit/llvm/config.py | 7 +++
6 files changed, 36 insertions(+), 10 deletions(-)
diff --git a/clang/tools/clang-format/ClangFormat.cpp
b/clang/tools/clang-format/ClangFormat.cpp
index feb733fe3c9e0b..01f7c6047726e2 100644
--- a/clang/tools/clang-format/ClangFormat.cpp
+++ b/clang/tools/clang-format/ClangFormat.cpp
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
errs() << EC.message() << "\n";
return true;
@@ -558,7 +559,7 @@ static int dumpConfig() {
// Read in the code in case the filename alone isn't enough to detect the
// language.
ErrorOr> CodeOrErr =
-MemoryBuffer::getFileOrSTDIN(FileNames[0]);
+MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
llvm::errs() << EC.message() << "\n";
return 1;
diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
index c3015d895230ea..40ee59c014b09f 100644
--- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
+++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
@@ -95,7 +95,8 @@ static std::vector
getSearchPaths(opt::InputArgList *Args,
// Opens a file. Path has to be resolved already. (used for def file)
std::unique_ptr openFile(const Twine ) {
- ErrorOr> MB =
MemoryBuffer::getFile(Path);
+ ErrorOr> MB =
+ MemoryBuffer::getFile(Path, /*IsText=*/true);
if (std::error_code EC = MB.getError()) {
llvm::errs() << "cannot open file " << Path << ": " << EC.message() <<
"\n";
diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
index 6a5646965df2cf..c5ccd64f116539 100644
--- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
+++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
@@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) {
cl::HideUnrelatedOptions({, ()});
cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
- auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
+ auto OldSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true);
if (!OldSymbolBufOrError)
exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
- auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
+ auto NewSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true);
if (!NewSymbolBufOrError)
exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
- auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
+ auto RemappingBufOrError =
+ MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true);
if (!RemappingBufOrError)
exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp
index b7f5356e22a9e6..4a060e1aad427f 100644
--- a/llvm/tools/yaml2obj/yaml2obj.cpp
+++ b/llvm/tools/yaml2obj/yaml2obj.cpp
@@ -130,7 +130,7 @@ int main(int argc, char **argv) {
}
ErrorOr> Buf =
- MemoryBuffer::getFileOrSTDIN(Input);
+ MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true);
if (!Buf)
return 1;
diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py
b/llvm/utils/lit/lit/builtin_commands/cat.py
index 37f55c0aef210b..6fb2152ef9332d 100644
--- a/llvm/utils/lit/lit/builtin_commands/cat.py
+++ b/llvm/utils/lit/lit/builtin_commands/cat.py
@@ -55,10 +55,24 @@ def main(argv):
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
for filename in filenames:
try:
-fileToCat = open(filename, "rb")
-contents = fileToCat.read()
+contents = None
+is_text = False
+try:
+if sys.platform != "win32":
+fileToCat = open(filename,
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/zibi2 approved this pull request. LGTM with clean checks. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
zibi2 wrote: LGTM, but check why the following test in checks failed: `FAIL: BOLT :: RISCV/unnamed-sym-no-entry.c` https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
llvmbot wrote:
@llvm/pr-subscribers-testing-tools
Author: Sean Perry (perry-ca)
Changes
To support auto-conversion on z/OS text files need to be opened as text files.
These changes will fix a number of LIT failures due to text files not being
converted to the internal code page.
- update a number of tools so they open the text files as text files
- add support in the cat.py to open a text file as a text file (Windows will
continue to treat all files as binary so new lines are handled correctly)
- add env var definitions to enable auto-conversion in the lit config file.
---
Full diff: https://github.com/llvm/llvm-project/pull/90128.diff
6 Files Affected:
- (modified) clang/tools/clang-format/ClangFormat.cpp (+4-3)
- (modified) llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp (+2-1)
- (modified) llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp (+6-3)
- (modified) llvm/tools/yaml2obj/yaml2obj.cpp (+1-1)
- (modified) llvm/utils/lit/lit/builtin_commands/cat.py (+16-2)
- (modified) llvm/utils/lit/lit/llvm/config.py (+7)
``diff
diff --git a/clang/tools/clang-format/ClangFormat.cpp
b/clang/tools/clang-format/ClangFormat.cpp
index feb733fe3c9e0b..01f7c6047726e2 100644
--- a/clang/tools/clang-format/ClangFormat.cpp
+++ b/clang/tools/clang-format/ClangFormat.cpp
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
errs() << EC.message() << "\n";
return true;
@@ -558,7 +559,7 @@ static int dumpConfig() {
// Read in the code in case the filename alone isn't enough to detect the
// language.
ErrorOr> CodeOrErr =
-MemoryBuffer::getFileOrSTDIN(FileNames[0]);
+MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
llvm::errs() << EC.message() << "\n";
return 1;
diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
index c3015d895230ea..40ee59c014b09f 100644
--- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
+++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
@@ -95,7 +95,8 @@ static std::vector
getSearchPaths(opt::InputArgList *Args,
// Opens a file. Path has to be resolved already. (used for def file)
std::unique_ptr openFile(const Twine ) {
- ErrorOr> MB =
MemoryBuffer::getFile(Path);
+ ErrorOr> MB =
+ MemoryBuffer::getFile(Path, /*IsText=*/true);
if (std::error_code EC = MB.getError()) {
llvm::errs() << "cannot open file " << Path << ": " << EC.message() <<
"\n";
diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
index 6a5646965df2cf..c5ccd64f116539 100644
--- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
+++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
@@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) {
cl::HideUnrelatedOptions({, ()});
cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
- auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
+ auto OldSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true);
if (!OldSymbolBufOrError)
exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
- auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
+ auto NewSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true);
if (!NewSymbolBufOrError)
exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
- auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
+ auto RemappingBufOrError =
+ MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true);
if (!RemappingBufOrError)
exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp
index b7f5356e22a9e6..4a060e1aad427f 100644
--- a/llvm/tools/yaml2obj/yaml2obj.cpp
+++ b/llvm/tools/yaml2obj/yaml2obj.cpp
@@ -130,7 +130,7 @@ int main(int argc, char **argv) {
}
ErrorOr> Buf =
- MemoryBuffer::getFileOrSTDIN(Input);
+ MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true);
if (!Buf)
return 1;
diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py
b/llvm/utils/lit/lit/builtin_commands/cat.py
index 37f55c0aef210b..6fb2152ef9332d 100644
--- a/llvm/utils/lit/lit/builtin_commands/cat.py
+++ b/llvm/utils/lit/lit/builtin_commands/cat.py
@@ -55,10 +55,24 @@ def main(argv):
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
llvmbot wrote:
@llvm/pr-subscribers-clang-format
Author: Sean Perry (perry-ca)
Changes
To support auto-conversion on z/OS text files need to be opened as text files.
These changes will fix a number of LIT failures due to text files not being
converted to the internal code page.
- update a number of tools so they open the text files as text files
- add support in the cat.py to open a text file as a text file (Windows will
continue to treat all files as binary so new lines are handled correctly)
- add env var definitions to enable auto-conversion in the lit config file.
---
Full diff: https://github.com/llvm/llvm-project/pull/90128.diff
6 Files Affected:
- (modified) clang/tools/clang-format/ClangFormat.cpp (+4-3)
- (modified) llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp (+2-1)
- (modified) llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp (+6-3)
- (modified) llvm/tools/yaml2obj/yaml2obj.cpp (+1-1)
- (modified) llvm/utils/lit/lit/builtin_commands/cat.py (+16-2)
- (modified) llvm/utils/lit/lit/llvm/config.py (+7)
``diff
diff --git a/clang/tools/clang-format/ClangFormat.cpp
b/clang/tools/clang-format/ClangFormat.cpp
index feb733fe3c9e0b..01f7c6047726e2 100644
--- a/clang/tools/clang-format/ClangFormat.cpp
+++ b/clang/tools/clang-format/ClangFormat.cpp
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
errs() << EC.message() << "\n";
return true;
@@ -558,7 +559,7 @@ static int dumpConfig() {
// Read in the code in case the filename alone isn't enough to detect the
// language.
ErrorOr> CodeOrErr =
-MemoryBuffer::getFileOrSTDIN(FileNames[0]);
+MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
llvm::errs() << EC.message() << "\n";
return 1;
diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
index c3015d895230ea..40ee59c014b09f 100644
--- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
+++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
@@ -95,7 +95,8 @@ static std::vector
getSearchPaths(opt::InputArgList *Args,
// Opens a file. Path has to be resolved already. (used for def file)
std::unique_ptr openFile(const Twine ) {
- ErrorOr> MB =
MemoryBuffer::getFile(Path);
+ ErrorOr> MB =
+ MemoryBuffer::getFile(Path, /*IsText=*/true);
if (std::error_code EC = MB.getError()) {
llvm::errs() << "cannot open file " << Path << ": " << EC.message() <<
"\n";
diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
index 6a5646965df2cf..c5ccd64f116539 100644
--- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
+++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
@@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) {
cl::HideUnrelatedOptions({, ()});
cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
- auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
+ auto OldSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true);
if (!OldSymbolBufOrError)
exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
- auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
+ auto NewSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true);
if (!NewSymbolBufOrError)
exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
- auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
+ auto RemappingBufOrError =
+ MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true);
if (!RemappingBufOrError)
exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp
index b7f5356e22a9e6..4a060e1aad427f 100644
--- a/llvm/tools/yaml2obj/yaml2obj.cpp
+++ b/llvm/tools/yaml2obj/yaml2obj.cpp
@@ -130,7 +130,7 @@ int main(int argc, char **argv) {
}
ErrorOr> Buf =
- MemoryBuffer::getFileOrSTDIN(Input);
+ MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true);
if (!Buf)
return 1;
diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py
b/llvm/utils/lit/lit/builtin_commands/cat.py
index 37f55c0aef210b..6fb2152ef9332d 100644
--- a/llvm/utils/lit/lit/builtin_commands/cat.py
+++ b/llvm/utils/lit/lit/builtin_commands/cat.py
@@ -55,10 +55,24 @@ def main(argv):
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/perry-ca created
https://github.com/llvm/llvm-project/pull/90128
To support auto-conversion on z/OS text files need to be opened as text files.
These changes will fix a number of LIT failures due to text files not being
converted to the internal code page.
- update a number of tools so they open the text files as text files
- add support in the cat.py to open a text file as a text file (Windows will
continue to treat all files as binary so new lines are handled correctly)
- add env var definitions to enable auto-conversion in the lit config file.
>From 3c434710f062c011ade4030b0d79254ee5c6a1df Mon Sep 17 00:00:00 2001
From: Sean Perry
Date: Thu, 25 Apr 2024 15:58:05 -0500
Subject: [PATCH] treat text files as text files so autoconversion is done
---
clang/tools/clang-format/ClangFormat.cpp| 7 ---
llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp | 3 ++-
llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp | 9 ++---
llvm/tools/yaml2obj/yaml2obj.cpp| 2 +-
llvm/utils/lit/lit/builtin_commands/cat.py | 18 --
llvm/utils/lit/lit/llvm/config.py | 7 +++
6 files changed, 36 insertions(+), 10 deletions(-)
diff --git a/clang/tools/clang-format/ClangFormat.cpp
b/clang/tools/clang-format/ClangFormat.cpp
index feb733fe3c9e0b..01f7c6047726e2 100644
--- a/clang/tools/clang-format/ClangFormat.cpp
+++ b/clang/tools/clang-format/ClangFormat.cpp
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool
ErrorOnIncompleteFormat = false) {
// On Windows, overwriting a file with an open file mapping doesn't work,
// so read the whole file into memory when formatting in-place.
ErrorOr> CodeOrErr =
- !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName)
-: MemoryBuffer::getFileOrSTDIN(FileName);
+ !OutputXML && Inplace
+ ? MemoryBuffer::getFileAsStream(FileName)
+ : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
errs() << EC.message() << "\n";
return true;
@@ -558,7 +559,7 @@ static int dumpConfig() {
// Read in the code in case the filename alone isn't enough to detect the
// language.
ErrorOr> CodeOrErr =
-MemoryBuffer::getFileOrSTDIN(FileNames[0]);
+MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true);
if (std::error_code EC = CodeOrErr.getError()) {
llvm::errs() << EC.message() << "\n";
return 1;
diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
index c3015d895230ea..40ee59c014b09f 100644
--- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
+++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp
@@ -95,7 +95,8 @@ static std::vector
getSearchPaths(opt::InputArgList *Args,
// Opens a file. Path has to be resolved already. (used for def file)
std::unique_ptr openFile(const Twine ) {
- ErrorOr> MB =
MemoryBuffer::getFile(Path);
+ ErrorOr> MB =
+ MemoryBuffer::getFile(Path, /*IsText=*/true);
if (std::error_code EC = MB.getError()) {
llvm::errs() << "cannot open file " << Path << ": " << EC.message() <<
"\n";
diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
index 6a5646965df2cf..c5ccd64f116539 100644
--- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
+++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp
@@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) {
cl::HideUnrelatedOptions({, ()});
cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
- auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
+ auto OldSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true);
if (!OldSymbolBufOrError)
exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
- auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
+ auto NewSymbolBufOrError =
+ MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true);
if (!NewSymbolBufOrError)
exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
- auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
+ auto RemappingBufOrError =
+ MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true);
if (!RemappingBufOrError)
exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp
index b7f5356e22a9e6..4a060e1aad427f 100644
--- a/llvm/tools/yaml2obj/yaml2obj.cpp
+++ b/llvm/tools/yaml2obj/yaml2obj.cpp
@@ -130,7 +130,7 @@ int main(int argc, char **argv) {
}
ErrorOr> Buf =
- MemoryBuffer::getFileOrSTDIN(Input);
+ MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true);
if (!Buf)
return 1;
diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py
b/llvm/utils/lit/lit/builtin_commands/cat.py
index 37f55c0aef210b..6fb2152ef9332d 100644
---
