Re: r332458 - [AST] Added a helper to extract a user-friendly text of a comment.

2018-05-17 Thread Ilya Biryukov via cfe-commits 
Terribly sorry for the breakage and many thanks for fixing this!

On Thu, May 17, 2018 at 9:04 AM Clement Courbet  wrote:

> I should have fixed it in r332576.
>
> On Wed, May 16, 2018 at 11:49 PM, Galina Kistanova via cfe-commits <
> cfe-commits@lists.llvm.org> wrote:
>
>> Also few other builders are affected:
>>
>> http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test
>> http://lab.llvm.org:8011/builders/clang-lld-x86_64-2stage
>> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu
>>
>>
>> Thanks
>>
>> Galina
>>
>> On Wed, May 16, 2018 at 12:58 PM, Galina Kistanova 
>> wrote:
>>
>>> Hello Ilya,
>>>
>>> This commit broke build step for couple of our builders:
>>>
>>> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8541
>>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu
>>>
>>> . . .
>>> FAILED:
>>> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>>> /usr/bin/c++   -DGTEST_HAS_RTTI=0 -DGTEST_HAS_TR1_TUPLE=0
>>> -DGTEST_LANG_CXX11=1 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
>>> -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/unittests/AST
>>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST
>>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/include
>>> -Itools/clang/include -Iinclude
>>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/include
>>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googletest/include
>>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googlemock/include
>>> -fPIC -fvisibility-inlines-hidden -std=c++11 -Wall -W -Wno-unused-parameter
>>> -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic
>>> -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor
>>> -Wno-comment -ffunction-sections -fdata-sections -fno-common
>>> -Woverloaded-virtual -fno-strict-aliasing -O3 -DNDEBUG
>>> -Wno-variadic-macros -fno-exceptions -fno-rtti -MD -MT
>>> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o -MF
>>> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o.d
>>> -o tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>>> -c
>>> /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST/CommentTextTest.cpp
>>> /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST/CommentTextTest.cpp:62:1:
>>> error: unterminated raw string
>>>  R"cpp(
>>>  ^
>>> . . .
>>>
>>> Please have a look?
>>>
>>> The builder was already red and did not send notifications.
>>>
>>> Thanks
>>>
>>> Galina
>>>
>>>
>>>
>>> On Wed, May 16, 2018 at 5:30 AM, Ilya Biryukov via cfe-commits <
>>> cfe-commits@lists.llvm.org> wrote:
>>>
 Author: ibiryukov
 Date: Wed May 16 05:30:09 2018
 New Revision: 332458

 URL: http://llvm.org/viewvc/llvm-project?rev=332458=rev
 Log:
 [AST] Added a helper to extract a user-friendly text of a comment.

 Summary:
 The helper is used in clangd for documentation shown in code completion
 and storing the docs in the symbols. See D45999.

 This patch reuses the code of the Doxygen comment lexer, disabling the
 bits that do command and html tag parsing.
 The new helper works on all comments, including non-doxygen comments.
 However, it does not understand or transform any doxygen directives,
 i.e. cannot extract brief text, etc.

 Reviewers: sammccall, hokein, ioeric

 Reviewed By: ioeric

 Subscribers: mgorny, cfe-commits

 Differential Revision: https://reviews.llvm.org/D46000

 Added:
 cfe/trunk/unittests/AST/CommentTextTest.cpp
 Modified:
 cfe/trunk/include/clang/AST/CommentLexer.h
 cfe/trunk/include/clang/AST/RawCommentList.h
 cfe/trunk/lib/AST/CommentLexer.cpp
 cfe/trunk/lib/AST/RawCommentList.cpp
 cfe/trunk/unittests/AST/CMakeLists.txt

 Modified: cfe/trunk/include/clang/AST/CommentLexer.h
 URL:
 http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/AST/CommentLexer.h?rev=332458=332457=332458=diff

 ==
 --- cfe/trunk/include/clang/AST/CommentLexer.h (original)
 +++ cfe/trunk/include/clang/AST/CommentLexer.h Wed May 16 05:30:09 2018
 @@ -281,6 +281,11 @@ private:
/// command, including command marker.
SmallString<16> VerbatimBlockEndCommandName;

 +  /// If true, the commands, html tags, etc will be parsed and
 reported as
 +  /// separate tokens inside the comment body. If false, the comment
 text will
 +  /// be parsed into text and newline tokens.
 +  bool ParseCommands;
 +
/// Given a character reference name (e.g., "lt"), return the
 character that
/// it

Re: r332458 - [AST] Added a helper to extract a user-friendly text of a comment.

2018-05-17 Thread Clement Courbet via cfe-commits 
I should have fixed it in r332576.

On Wed, May 16, 2018 at 11:49 PM, Galina Kistanova via cfe-commits <
cfe-commits@lists.llvm.org> wrote:

> Also few other builders are affected:
>
> http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test
> http://lab.llvm.org:8011/builders/clang-lld-x86_64-2stage
> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu
>
>
> Thanks
>
> Galina
>
> On Wed, May 16, 2018 at 12:58 PM, Galina Kistanova 
> wrote:
>
>> Hello Ilya,
>>
>> This commit broke build step for couple of our builders:
>>
>> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8541
>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu
>>
>> . . .
>> FAILED: 
>> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>>
>> /usr/bin/c++   -DGTEST_HAS_RTTI=0 -DGTEST_HAS_TR1_TUPLE=0
>> -DGTEST_LANG_CXX11=1 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
>> -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/unittests/AST
>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST
>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/include
>> -Itools/clang/include -Iinclude -I/home/buildslave/buildslave1
>> a/clang-with-lto-ubuntu/llvm.src/include -I/home/buildslave/buildslave1
>> a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googletest/include
>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.
>> src/utils/unittest/googlemock/include -fPIC -fvisibility-inlines-hidden
>> -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual
>> -Wno-missing-field-initializers -pedantic -Wno-long-long
>> -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment
>> -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual
>> -fno-strict-aliasing -O3 -DNDEBUG-Wno-variadic-macros -fno-exceptions
>> -fno-rtti -MD -MT tools/clang/unittests/AST/CMak
>> eFiles/ASTTests.dir/CommentTextTest.cpp.o -MF
>> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o.d
>> -o tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>> -c /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src
>> /tools/clang/unittests/AST/CommentTextTest.cpp
>> /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src
>> /tools/clang/unittests/AST/CommentTextTest.cpp:62:1: error: unterminated
>> raw string
>>  R"cpp(
>>  ^
>> . . .
>>
>> Please have a look?
>>
>> The builder was already red and did not send notifications.
>>
>> Thanks
>>
>> Galina
>>
>>
>>
>> On Wed, May 16, 2018 at 5:30 AM, Ilya Biryukov via cfe-commits <
>> cfe-commits@lists.llvm.org> wrote:
>>
>>> Author: ibiryukov
>>> Date: Wed May 16 05:30:09 2018
>>> New Revision: 332458
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=332458=rev
>>> Log:
>>> [AST] Added a helper to extract a user-friendly text of a comment.
>>>
>>> Summary:
>>> The helper is used in clangd for documentation shown in code completion
>>> and storing the docs in the symbols. See D45999.
>>>
>>> This patch reuses the code of the Doxygen comment lexer, disabling the
>>> bits that do command and html tag parsing.
>>> The new helper works on all comments, including non-doxygen comments.
>>> However, it does not understand or transform any doxygen directives,
>>> i.e. cannot extract brief text, etc.
>>>
>>> Reviewers: sammccall, hokein, ioeric
>>>
>>> Reviewed By: ioeric
>>>
>>> Subscribers: mgorny, cfe-commits
>>>
>>> Differential Revision: https://reviews.llvm.org/D46000
>>>
>>> Added:
>>> cfe/trunk/unittests/AST/CommentTextTest.cpp
>>> Modified:
>>> cfe/trunk/include/clang/AST/CommentLexer.h
>>> cfe/trunk/include/clang/AST/RawCommentList.h
>>> cfe/trunk/lib/AST/CommentLexer.cpp
>>> cfe/trunk/lib/AST/RawCommentList.cpp
>>> cfe/trunk/unittests/AST/CMakeLists.txt
>>>
>>> Modified: cfe/trunk/include/clang/AST/CommentLexer.h
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/
>>> AST/CommentLexer.h?rev=332458=332457=332458=diff
>>> 
>>> ==
>>> --- cfe/trunk/include/clang/AST/CommentLexer.h (original)
>>> +++ cfe/trunk/include/clang/AST/CommentLexer.h Wed May 16 05:30:09 2018
>>> @@ -281,6 +281,11 @@ private:
>>>/// command, including command marker.
>>>SmallString<16> VerbatimBlockEndCommandName;
>>>
>>> +  /// If true, the commands, html tags, etc will be parsed and reported
>>> as
>>> +  /// separate tokens inside the comment body. If false, the comment
>>> text will
>>> +  /// be parsed into text and newline tokens.
>>> +  bool ParseCommands;
>>> +
>>>/// Given a character reference name (e.g., "lt"), return the
>>> character that
>>>/// it stands for (e.g., "<").
>>>StringRef resolveHTMLNamedCharacterReference(StringRef Name) const;
>>> @@ -315,12 +320,11 @@ private:
>>>/// Eat string matching regexp \code \s*\* \endcode.
>>>void skipLineStartingDecorations();
>>>
>>> -

Re: r332458 - [AST] Added a helper to extract a user-friendly text of a comment.

2018-05-16 Thread Galina Kistanova via cfe-commits 
Also few other builders are affected:

http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test
http://lab.llvm.org:8011/builders/clang-lld-x86_64-2stage
http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu


Thanks

Galina

On Wed, May 16, 2018 at 12:58 PM, Galina Kistanova 
wrote:

> Hello Ilya,
>
> This commit broke build step for couple of our builders:
>
> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8541
> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu
>
> . . .
> FAILED: 
> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>
> /usr/bin/c++   -DGTEST_HAS_RTTI=0 -DGTEST_HAS_TR1_TUPLE=0
> -DGTEST_LANG_CXX11=1 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
> -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/unittests/AST
> -I/home/buildslave/buildslave1a/clang-with-lto-
> ubuntu/llvm.src/tools/clang/unittests/AST -I/home/buildslave/
> buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/include
> -Itools/clang/include -Iinclude -I/home/buildslave/
> buildslave1a/clang-with-lto-ubuntu/llvm.src/include -I/home/buildslave/
> buildslave1a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googletest/include
> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googlemock/include
> -fPIC -fvisibility-inlines-hidden -std=c++11 -Wall -W -Wno-unused-parameter
> -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic
> -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor
> -Wno-comment -ffunction-sections -fdata-sections -fno-common
> -Woverloaded-virtual -fno-strict-aliasing -O3 -DNDEBUG
> -Wno-variadic-macros -fno-exceptions -fno-rtti -MD -MT
> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
> -MF tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o.d
> -o tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
> -c /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.
> src/tools/clang/unittests/AST/CommentTextTest.cpp
> /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.
> src/tools/clang/unittests/AST/CommentTextTest.cpp:62:1: error:
> unterminated raw string
>  R"cpp(
>  ^
> . . .
>
> Please have a look?
>
> The builder was already red and did not send notifications.
>
> Thanks
>
> Galina
>
>
>
> On Wed, May 16, 2018 at 5:30 AM, Ilya Biryukov via cfe-commits <
> cfe-commits@lists.llvm.org> wrote:
>
>> Author: ibiryukov
>> Date: Wed May 16 05:30:09 2018
>> New Revision: 332458
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=332458=rev
>> Log:
>> [AST] Added a helper to extract a user-friendly text of a comment.
>>
>> Summary:
>> The helper is used in clangd for documentation shown in code completion
>> and storing the docs in the symbols. See D45999.
>>
>> This patch reuses the code of the Doxygen comment lexer, disabling the
>> bits that do command and html tag parsing.
>> The new helper works on all comments, including non-doxygen comments.
>> However, it does not understand or transform any doxygen directives,
>> i.e. cannot extract brief text, etc.
>>
>> Reviewers: sammccall, hokein, ioeric
>>
>> Reviewed By: ioeric
>>
>> Subscribers: mgorny, cfe-commits
>>
>> Differential Revision: https://reviews.llvm.org/D46000
>>
>> Added:
>> cfe/trunk/unittests/AST/CommentTextTest.cpp
>> Modified:
>> cfe/trunk/include/clang/AST/CommentLexer.h
>> cfe/trunk/include/clang/AST/RawCommentList.h
>> cfe/trunk/lib/AST/CommentLexer.cpp
>> cfe/trunk/lib/AST/RawCommentList.cpp
>> cfe/trunk/unittests/AST/CMakeLists.txt
>>
>> Modified: cfe/trunk/include/clang/AST/CommentLexer.h
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/
>> AST/CommentLexer.h?rev=332458=332457=332458=diff
>> 
>> ==
>> --- cfe/trunk/include/clang/AST/CommentLexer.h (original)
>> +++ cfe/trunk/include/clang/AST/CommentLexer.h Wed May 16 05:30:09 2018
>> @@ -281,6 +281,11 @@ private:
>>/// command, including command marker.
>>SmallString<16> VerbatimBlockEndCommandName;
>>
>> +  /// If true, the commands, html tags, etc will be parsed and reported
>> as
>> +  /// separate tokens inside the comment body. If false, the comment
>> text will
>> +  /// be parsed into text and newline tokens.
>> +  bool ParseCommands;
>> +
>>/// Given a character reference name (e.g., "lt"), return the
>> character that
>>/// it stands for (e.g., "<").
>>StringRef resolveHTMLNamedCharacterReference(StringRef Name) const;
>> @@ -315,12 +320,11 @@ private:
>>/// Eat string matching regexp \code \s*\* \endcode.
>>void skipLineStartingDecorations();
>>
>> -  /// Lex stuff inside comments.  CommentEnd should be set correctly.
>> +  /// Lex comment text, including commands if ParseCommands is set to
>> true.
>>void lexCommentText(Token );
>>
>> -  void setupAndLexVerbatimBlock(Token ,
>> -const char

Re: r332458 - [AST] Added a helper to extract a user-friendly text of a comment.

2018-05-16 Thread Galina Kistanova via cfe-commits 
Hello Ilya,

This commit broke build step for couple of our builders:

http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8541
http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu

. . .
FAILED:
tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
/usr/bin/c++   -DGTEST_HAS_RTTI=0 -DGTEST_HAS_TR1_TUPLE=0
-DGTEST_LANG_CXX11=1 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
-D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/unittests/AST
-I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST
-I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/include
-Itools/clang/include -Iinclude
-I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/include
-I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googletest/include
-I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googlemock/include
-fPIC -fvisibility-inlines-hidden -std=c++11 -Wall -W -Wno-unused-parameter
-Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic
-Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor
-Wno-comment -ffunction-sections -fdata-sections -fno-common
-Woverloaded-virtual -fno-strict-aliasing -O3 -DNDEBUG
-Wno-variadic-macros -fno-exceptions -fno-rtti -MD -MT
tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o -MF
tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o.d
-o tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
-c
/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST/CommentTextTest.cpp
/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST/CommentTextTest.cpp:62:1:
error: unterminated raw string
 R"cpp(
 ^
. . .

Please have a look?

The builder was already red and did not send notifications.

Thanks

Galina



On Wed, May 16, 2018 at 5:30 AM, Ilya Biryukov via cfe-commits <
cfe-commits@lists.llvm.org> wrote:

> Author: ibiryukov
> Date: Wed May 16 05:30:09 2018
> New Revision: 332458
>
> URL: http://llvm.org/viewvc/llvm-project?rev=332458=rev
> Log:
> [AST] Added a helper to extract a user-friendly text of a comment.
>
> Summary:
> The helper is used in clangd for documentation shown in code completion
> and storing the docs in the symbols. See D45999.
>
> This patch reuses the code of the Doxygen comment lexer, disabling the
> bits that do command and html tag parsing.
> The new helper works on all comments, including non-doxygen comments.
> However, it does not understand or transform any doxygen directives,
> i.e. cannot extract brief text, etc.
>
> Reviewers: sammccall, hokein, ioeric
>
> Reviewed By: ioeric
>
> Subscribers: mgorny, cfe-commits
>
> Differential Revision: https://reviews.llvm.org/D46000
>
> Added:
> cfe/trunk/unittests/AST/CommentTextTest.cpp
> Modified:
> cfe/trunk/include/clang/AST/CommentLexer.h
> cfe/trunk/include/clang/AST/RawCommentList.h
> cfe/trunk/lib/AST/CommentLexer.cpp
> cfe/trunk/lib/AST/RawCommentList.cpp
> cfe/trunk/unittests/AST/CMakeLists.txt
>
> Modified: cfe/trunk/include/clang/AST/CommentLexer.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/
> clang/AST/CommentLexer.h?rev=332458=332457=332458=diff
> 
> ==
> --- cfe/trunk/include/clang/AST/CommentLexer.h (original)
> +++ cfe/trunk/include/clang/AST/CommentLexer.h Wed May 16 05:30:09 2018
> @@ -281,6 +281,11 @@ private:
>/// command, including command marker.
>SmallString<16> VerbatimBlockEndCommandName;
>
> +  /// If true, the commands, html tags, etc will be parsed and reported as
> +  /// separate tokens inside the comment body. If false, the comment text
> will
> +  /// be parsed into text and newline tokens.
> +  bool ParseCommands;
> +
>/// Given a character reference name (e.g., "lt"), return the character
> that
>/// it stands for (e.g., "<").
>StringRef resolveHTMLNamedCharacterReference(StringRef Name) const;
> @@ -315,12 +320,11 @@ private:
>/// Eat string matching regexp \code \s*\* \endcode.
>void skipLineStartingDecorations();
>
> -  /// Lex stuff inside comments.  CommentEnd should be set correctly.
> +  /// Lex comment text, including commands if ParseCommands is set to
> true.
>void lexCommentText(Token );
>
> -  void setupAndLexVerbatimBlock(Token ,
> -const char *TextBegin,
> -char Marker, const CommandInfo *Info);
> +  void setupAndLexVerbatimBlock(Token , const char *TextBegin, char
> Marker,
> +const CommandInfo *Info);
>
>void lexVerbatimBlockFirstLine(Token );
>
> @@ -343,14 +347,13 @@ private:
>
>  public:
>Lexer(llvm::BumpPtrAllocator , DiagnosticsEngine ,
> -const CommandTraits ,
> -SourceLocation FileLoc,
> -const char *BufferStart,

r332458 - [AST] Added a helper to extract a user-friendly text of a comment.

2018-05-16 Thread Ilya Biryukov via cfe-commits 
Author: ibiryukov
Date: Wed May 16 05:30:09 2018
New Revision: 332458

URL: http://llvm.org/viewvc/llvm-project?rev=332458=rev
Log:
[AST] Added a helper to extract a user-friendly text of a comment.

Summary:
The helper is used in clangd for documentation shown in code completion
and storing the docs in the symbols. See D45999.

This patch reuses the code of the Doxygen comment lexer, disabling the
bits that do command and html tag parsing.
The new helper works on all comments, including non-doxygen comments.
However, it does not understand or transform any doxygen directives,
i.e. cannot extract brief text, etc.

Reviewers: sammccall, hokein, ioeric

Reviewed By: ioeric

Subscribers: mgorny, cfe-commits

Differential Revision: https://reviews.llvm.org/D46000

Added:
cfe/trunk/unittests/AST/CommentTextTest.cpp
Modified:
cfe/trunk/include/clang/AST/CommentLexer.h
cfe/trunk/include/clang/AST/RawCommentList.h
cfe/trunk/lib/AST/CommentLexer.cpp
cfe/trunk/lib/AST/RawCommentList.cpp
cfe/trunk/unittests/AST/CMakeLists.txt

Modified: cfe/trunk/include/clang/AST/CommentLexer.h
URL: 
http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/AST/CommentLexer.h?rev=332458=332457=332458=diff
==
--- cfe/trunk/include/clang/AST/CommentLexer.h (original)
+++ cfe/trunk/include/clang/AST/CommentLexer.h Wed May 16 05:30:09 2018
@@ -281,6 +281,11 @@ private:
   /// command, including command marker.
   SmallString<16> VerbatimBlockEndCommandName;
 
+  /// If true, the commands, html tags, etc will be parsed and reported as
+  /// separate tokens inside the comment body. If false, the comment text will
+  /// be parsed into text and newline tokens.
+  bool ParseCommands;
+
   /// Given a character reference name (e.g., "lt"), return the character that
   /// it stands for (e.g., "<").
   StringRef resolveHTMLNamedCharacterReference(StringRef Name) const;
@@ -315,12 +320,11 @@ private:
   /// Eat string matching regexp \code \s*\* \endcode.
   void skipLineStartingDecorations();
 
-  /// Lex stuff inside comments.  CommentEnd should be set correctly.
+  /// Lex comment text, including commands if ParseCommands is set to true.
   void lexCommentText(Token );
 
-  void setupAndLexVerbatimBlock(Token ,
-const char *TextBegin,
-char Marker, const CommandInfo *Info);
+  void setupAndLexVerbatimBlock(Token , const char *TextBegin, char Marker,
+const CommandInfo *Info);
 
   void lexVerbatimBlockFirstLine(Token );
 
@@ -343,14 +347,13 @@ private:
 
 public:
   Lexer(llvm::BumpPtrAllocator , DiagnosticsEngine ,
-const CommandTraits ,
-SourceLocation FileLoc,
-const char *BufferStart, const char *BufferEnd);
+const CommandTraits , SourceLocation FileLoc,
+const char *BufferStart, const char *BufferEnd,
+bool ParseCommands = true);
 
   void lex(Token );
 
-  StringRef getSpelling(const Token ,
-const SourceManager ,
+  StringRef getSpelling(const Token , const SourceManager ,
 bool *Invalid = nullptr) const;
 };
 

Modified: cfe/trunk/include/clang/AST/RawCommentList.h
URL: 
http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/AST/RawCommentList.h?rev=332458=332457=332458=diff
==
--- cfe/trunk/include/clang/AST/RawCommentList.h (original)
+++ cfe/trunk/include/clang/AST/RawCommentList.h Wed May 16 05:30:09 2018
@@ -111,6 +111,30 @@ public:
 return extractBriefText(Context);
   }
 
+  /// Returns sanitized comment text, suitable for presentation in editor UIs.
+  /// E.g. will transform:
+  /// // This is a long multiline comment.
+  /// //   Parts of it  might be indented.
+  /// /* The comments styles might be mixed. */
+  ///  into
+  /// "This is a long multiline comment.\n"
+  /// "  Parts of it  might be indented.\n"
+  /// "The comments styles might be mixed."
+  /// Also removes leading indentation and sanitizes some common cases:
+  /// /* This is a first line.
+  ///  *   This is a second line. It is indented.
+  ///  * This is a third line. */
+  /// and
+  /// /* This is a first line.
+  ///  This is a second line. It is indented.
+  /// This is a third line. */
+  /// will both turn into:
+  /// "This is a first line.\n"
+  /// "  This is a second line. It is indented.\n"
+  /// "This is a third line."
+  std::string getFormattedText(const SourceManager ,
+   DiagnosticsEngine ) const;
+
   /// Parse the comment, assuming it is attached to decl \c D.
   comments::FullComment *parse(const ASTContext ,
const Preprocessor *PP, const Decl *D) const;

Modified: cfe/trunk/lib/AST/CommentLexer.cpp
URL: