Re: [cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding
2016-08-17 16:47 GMT+03:00 Brad King : > I squashed in one warning fix: > > Windows: Encode child process output to internally-used encoding > https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=12924660 > > However, then I had to revert the change from `next` because it > causes the CTest.UpdateGIT test to fail on Windows machines. > I think the problem is that we run Git with a `-z` option to > produce binary output. In such cases we should not do any > encoding conversions. cmProcessTools and RunSingleCommand > will need to gain options for this. While git's `-z` flag caused test to fail it wasn't because of binary output (we're still working with text data like commit messages and file names) but it showed a bug that we would truncate output till first null byte. `-z` option specifies that git will use null bytes as separators between entries and it's independent from used text encoding. We still need to decode git's output (which contains null bytes) to our internal encoding. > Also I noticed that if DecodeText buffers partial characters we > may need a finalize step later to finish them off. Otherwise > invalid byte sequences may be dropped if they appear at the end. > > Please fetch the above version and revise it as needed. > I've fixed it and submitted MR to https://gitlab.kitware.com/cmake/cmake/merge_requests/221 -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding
On 08/17/2016 06:11 AM, Dāvis Mosāns wrote: >> Applied with minor tweaks and merged to `next` for testing: > > Great! Thanks! I squashed in one warning fix: Windows: Encode child process output to internally-used encoding https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=12924660 However, then I had to revert the change from `next` because it causes the CTest.UpdateGIT test to fail on Windows machines. I think the problem is that we run Git with a `-z` option to produce binary output. In such cases we should not do any encoding conversions. cmProcessTools and RunSingleCommand will need to gain options for this. Also I noticed that if DecodeText buffers partial characters we may need a finalize step later to finish them off. Otherwise invalid byte sequences may be dropped if they appear at the end. Please fetch the above version and revise it as needed. Thanks, -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding
2016-08-16 21:09 GMT+03:00 Brad King : > On 08/15/2016 04:34 PM, Dāvis Mosāns wrote: >> Typically Windows applications (eg. MSVC compiler) use current console's >> codepage for output to pipes so we need to encode that to internally used >> encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE). > > Thanks. Applied with minor tweaks and merged to `next` for testing: > > Windows: Encode child process output to internally-used encoding > https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=0a818239 Great! Thanks! -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding
On 08/15/2016 04:34 PM, Dāvis Mosāns wrote: > Typically Windows applications (eg. MSVC compiler) use current console's > codepage for output to pipes so we need to encode that to internally used > encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE). Thanks. Applied with minor tweaks and merged to `next` for testing: Windows: Encode child process output to internally-used encoding https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=0a818239 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding
Typically Windows applications (eg. MSVC compiler) use current console's codepage for output to pipes so we need to encode that to internally used encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE). --- Source/CMakeLists.txt | 6 ++ Source/ProcessOutput.cxx | 141 + Source/ProcessOutput.hxx | 39 ++ Source/cmExecProgramCommand.cxx| 3 + Source/cmExecuteProcessCommand.cxx | 11 ++- Source/cmProcessTools.cxx | 9 ++- Source/cmSystemTools.cxx | 11 ++- bootstrap | 5 +- 8 files changed, 218 insertions(+), 7 deletions(-) create mode 100644 Source/ProcessOutput.cxx create mode 100644 Source/ProcessOutput.hxx diff --git a/Source/CMakeLists.txt b/Source/CMakeLists.txt index cdc8fb1..46dd471 100644 --- a/Source/CMakeLists.txt +++ b/Source/CMakeLists.txt @@ -373,8 +373,14 @@ set(SRCS cm_sha2.c cm_utf8.h cm_utf8.c + + ProcessOutput.cxx + ProcessOutput.hxx ) +SET_PROPERTY(SOURCE ProcessOutput.cxx APPEND PROPERTY COMPILE_DEFINITIONS + KWSYS_ENCODING_DEFAULT_CODEPAGE=${KWSYS_ENCODING_DEFAULT_CODEPAGE}) + set(COMMAND_INCLUDES "#include \"cmTargetPropCommandBase.cxx\"\n") list(APPEND SRCS cmTargetPropCommandBase.cxx) set_property(SOURCE cmTargetPropCommandBase.cxx PROPERTY HEADER_FILE_ONLY ON) diff --git a/Source/ProcessOutput.cxx b/Source/ProcessOutput.cxx new file mode 100644 index 000..6c66087 --- /dev/null +++ b/Source/ProcessOutput.cxx @@ -0,0 +1,141 @@ +/* + CMake - Cross Platform Makefile Generator + Copyright 2000-2016 Kitware, Inc., Insight Software Consortium + + Distributed under the OSI-approved BSD License (the "License"); + see accompanying file Copyright.txt for details. + + This software is distributed WITHOUT ANY WARRANTY; without even the + implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + See the License for more information. +*/ + +#include "ProcessOutput.hxx" + +#if defined(_WIN32) +# include +unsigned int ProcessOutput::defaultCodepage = KWSYS_ENCODING_DEFAULT_CODEPAGE; +#endif + +ProcessOutput::ProcessOutput(unsigned int maxSize) +{ +#if defined(_WIN32) + bufferSize = maxSize; + codepage = GetConsoleCP(); + if (!codepage) { +codepage = GetACP(); + } +#else + static_cast(maxSize); +#endif +} + +ProcessOutput::~ProcessOutput() +{ +} + +bool ProcessOutput::DecodeText(std::string raw, std::string& decoded, size_t id) +{ + bool success = true; + decoded = raw; +#if defined(_WIN32) + if (id > 0) { +if (rawparts.size() < id) { + rawparts.reserve(id); + while (rawparts.size() < id) rawparts.push_back(std::string()); +} +raw = rawparts[id - 1] + raw; +rawparts[id - 1].clear(); +decoded = raw; + } + if (raw.size() > 0 && codepage != defaultCodepage) { +success = false; +CPINFOEXW cpinfo; +if (id > 0 && raw.size() == bufferSize && GetCPInfoExW(codepage, 0, &cpinfo) == 1 && cpinfo.MaxCharSize > 1) { + if (cpinfo.MaxCharSize == 2 && cpinfo.LeadByte[0] != 0) { +LPSTR prevChar = CharPrevExA(codepage, raw.c_str(), raw.c_str() + raw.size(), 0); +bool isLeadByte = (*(prevChar + 1) == 0) && IsDBCSLeadByteEx(codepage, *prevChar); +if (isLeadByte) { + rawparts[id - 1] += *(raw.end() - 1); + raw.resize(raw.size() - 1); +} +success = DoDecodeText(raw, decoded, NULL); + } else { +bool restoreDecoded = false; +std::string firstDecoded = decoded; +wchar_t lastChar = 0; +for (UINT i = 0; i < cpinfo.MaxCharSize; i++) { + success = DoDecodeText(raw, decoded, &lastChar); + if (success && lastChar != 0) { +if (i == 0) { + firstDecoded = decoded; +} +if (lastChar == cpinfo.UnicodeDefaultChar) { + restoreDecoded = true; + rawparts[id - 1] = *(raw.end() - 1) + rawparts[id - 1]; + raw.resize(raw.size() - 1); +} else { + restoreDecoded = false; + break; +} + } else { +break; + } +} +if (restoreDecoded) { + decoded = firstDecoded; + rawparts[id - 1].clear(); +} + } +} else { + success = DoDecodeText(raw, decoded, NULL); +} + } +#else + static_cast(id); +#endif + return success; +} + +bool ProcessOutput::DecodeText(const char* data, size_t length, std::string& decoded, size_t id) +{ + return DecodeText(std::string(data, length), decoded, id); +} + +bool ProcessOutput::DecodeText(std::vector raw, std::vector& decoded, size_t id) +{ + std::string str; + const bool success = DecodeText(std::string(raw.begin(), raw.end()), str, id); + decoded.assign(str.begin(), str.end()); + return success; +} +