[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
llvm-ci wrote:
LLVM Buildbot has detected a new failure on builder `lldb-x86_64-debian`
running on `lldb-x86_64-debian` while building `clang` at step 6 "test".
Full details are available at:
https://lab.llvm.org/buildbot/#/builders/162/builds/16230
Here is the relevant piece of the build log for the reference
```
Step 6 (test) failure: build (failure)
...
PASS: lldb-unit :: Utility/./UtilityTests/77/127 (2731 of 2777)
PASS: lldb-shell :: Subprocess/vfork-follow-parent-wp.test (2732 of 2777)
PASS: lldb-unit :: Utility/./UtilityTests/78/127 (2733 of 2777)
PASS: lldb-unit :: Core/./LLDBCoreTests/48/66 (2734 of 2777)
PASS: lldb-shell :: Unwind/basic-block-sections.test (2735 of 2777)
PASS: lldb-unit :: Core/./LLDBCoreTests/50/66 (2736 of 2777)
PASS: lldb-api :: tools/lldb-server/attach-wait/TestGdbRemoteAttachWait.py
(2737 of 2777)
PASS: lldb-api :: tools/lldb-server/TestGdbRemoteLaunch.py (2738 of 2777)
PASS: lldb-unit :: Host/./HostTests/2/98 (2739 of 2777)
PASS: lldb-unit :: Host/./HostTests/30/98 (2740 of 2777)
FAIL: lldb-api :: tools/lldb-dap/launch/TestDAP_launch.py (2741 of 2777)
TEST 'lldb-api :: tools/lldb-dap/launch/TestDAP_launch.py'
FAILED
Script:
--
/usr/bin/python3
/home/worker/2.0.1/lldb-x86_64-debian/llvm-project/lldb/test/API/dotest.py -u
CXXFLAGS -u CFLAGS --env
LLVM_LIBS_DIR=/home/worker/2.0.1/lldb-x86_64-debian/build/./lib --env
LLVM_INCLUDE_DIR=/home/worker/2.0.1/lldb-x86_64-debian/build/include --env
LLVM_TOOLS_DIR=/home/worker/2.0.1/lldb-x86_64-debian/build/./bin --arch x86_64
--build-dir /home/worker/2.0.1/lldb-x86_64-debian/build/lldb-test-build.noindex
--lldb-module-cache-dir
/home/worker/2.0.1/lldb-x86_64-debian/build/lldb-test-build.noindex/module-cache-lldb/lldb-api
--clang-module-cache-dir
/home/worker/2.0.1/lldb-x86_64-debian/build/lldb-test-build.noindex/module-cache-clang/lldb-api
--executable /home/worker/2.0.1/lldb-x86_64-debian/build/./bin/lldb --compiler
/home/worker/2.0.1/lldb-x86_64-debian/build/./bin/clang --dsymutil
/home/worker/2.0.1/lldb-x86_64-debian/build/./bin/dsymutil --make
/usr/bin/gmake --llvm-tools-dir
/home/worker/2.0.1/lldb-x86_64-debian/build/./bin --lldb-obj-root
/home/worker/2.0.1/lldb-x86_64-debian/build/tools/lldb --lldb-libs-dir
/home/worker/2.0.1/lldb-x86_64-debian/build/./lib -t
/home/worker/2.0.1/lldb-x86_64-debian/llvm-project/lldb/test/API/tools/lldb-dap/launch
-p TestDAP_launch.py
--
Exit Code: 1
Command Output (stdout):
--
lldb version 21.0.0git (https://github.com/llvm/llvm-project.git revision
b41b86a907f653f79bab10d4c80b3a41d146c71b)
clang revision b41b86a907f653f79bab10d4c80b3a41d146c71b
llvm revision b41b86a907f653f79bab10d4c80b3a41d146c71b
Skipping the following test categories: ['libc++', 'dsym', 'gmodules',
'debugserver', 'objc']
= DEBUG ADAPTER PROTOCOL LOGS =
1739554236.560714722 -->
Content-Length: 344
{
"arguments": {
"adapterID": "lldb-native",
"clientID": "vscode",
"columnsStartAt1": true,
"linesStartAt1": true,
"locale": "en-us",
"pathFormat": "path",
"sourceInitFile": false,
"supportsRunInTerminalRequest": true,
"supportsStartDebuggingRequest": true,
"supportsVariablePaging": true,
"supportsVariableType": true
},
"command": "initialize",
"seq": 1,
"type": "request"
}
1739554236.563421011 <--
Content-Length: 1631
```
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic closed https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/llvm-beanz edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/llvm-beanz approved this pull request. One small formatting cleanup in a suggestion. LGTM. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,124 @@ +//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++ -*-===// + +// llvm-beanz wrote: nit: ```suggestion // ``` https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,89 @@ +//===--- ParseHLSLRootSignature.h ---*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===--===// +// +// This file defines the ParseHLSLRootSignature interface. +// +//===--===// + +#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H +#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H + +#include "clang/AST/APValue.h" inbelic wrote: Removed this and the other no longer needed headers https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,89 @@ +//===--- ParseHLSLRootSignature.h ---*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===--===// +// +// This file defines the ParseHLSLRootSignature interface. +// +//===--===// + +#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H +#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H + +#include "clang/AST/APValue.h" llvm-beanz wrote: Is this needed? In general it is also best to prefer forward declarations in headers rather than including headers where possible to reduce compile time overhead. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/llvm-beanz edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/llvm-beanz commented: My last real question here is whether this belongs in clang/Parse or if it should be in clang/Lex. Since it is now strictly Lexing it shouldn't have any dependencies on the parse or AST libraries right? https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 01/18] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b084721..7755c05bc8969 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 0..9625f6a5bd76d
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 0..39069e7cc3998
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes the internal
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,194 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
llvm-beanz wrote:
What did you do to call this resolved? It looks like you're still parsing the
number?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,194 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
inbelic wrote:
Updated to have seperate tokens for the `+` and `-` and to defer this parsing
for the parser.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,166 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
inbelic wrote:
Correct. It is the expected behaviour for this pr, as we are not expecting to
handle floats until https://github.com/llvm/llvm-project/issues/126565. This
was split out because I want to have a pr that will have dedicated
documentation and testing to show that the float parsing is equivalent to DXC.
Which I think would get bogged down and increase the scope when including it in
this pr.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 01/17] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b084721..7755c05bc8969 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 0..9625f6a5bd76d
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 0..39069e7cc3998
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes the internal
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,166 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
llvm-beanz wrote:
If I'm reading this correctly right now if someone writes `1.3` the parser will
break the token at the `.`, call it an integer literal and return `1` for the
value. I assume that's not the expected behavior?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 01/16] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b084721..7755c05bc8969 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 0..9625f6a5bd76d
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 0..39069e7cc3998
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes the internal
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,194 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch when there is a '+' or '-' specified but no literal value after.
+ // This is invalid but the NumericLiteralParser will accept this as valid.
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_expected_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Note: if IsNumberChar allows for hexidecimal we will need to turn this
+ // into a diagnostics for potential fixed-point literals
+ assert(Literal.isIntegerLiteral() && "IsNumberChar will only support
digits");
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ // NOTE: for compabibility with DXC, we will treat any integer with '+' as an
+ // unsigned integer
+ llvm::APSInt X = llvm::APSInt(32, !Negative);
+ if (Literal.GetIntegerValue(X)) {
+// Report that the value has overflowed
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_number_literal_overflow)
+<< (unsigned)!Signed << NumSpelling;
+return true;
+ }
+
+ X = Negative ? -X : X;
+ Result.NumLiteral = APValue(X);
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ // Record where this token is in the text for usage in parser diagnostics
+ Result = RootSignatureToken(SourceLoc);
+
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ default:
+break;
+ }
+
+ // Numeric constant
+ if (isdigit(C) || C == '-' || C == '+')
+return LexNumber(Result);
+
+ // All following tokens require at least one additional character
+ if (Buffer.size() <= 1) {
+PP.getDiagnostics().Report(Result.TokLoc, diag::err_hlsl_invalid_token);
+return true;
+ }
+
+ // Peek at the next character to deteremine token type
+ char NextC = Buffer[1];
+
+ // Registers: [tsub][0-9+]
+ if ((C == 't' || C == 's' || C == 'u' || C == 'b') && isdigit(NextC)) {
+AdvanceBuffer();
+
+if (LexNumber(Result))
+ return true; // Error parsing number which is already reported
+
+// Lex number could also parse a float so ensure it was an unsigned int
+if (Result.Kind != TokenKind::int_literal ||
+Result.NumLiteral.getInt().isSigned()) {
+ // Return invalid number literal for register error
+ PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_register_literal);
+ return true;
+}
+
+// Convert character to the register type.
+// This is done after LexNumber to override the TokenKind
+switch (C) {
+case 'b':
+ Result.Kind = TokenKind::bReg;
+ break;
+case 't':
+ Result.Kind = TokenKind::tReg;
+ break;
+case 'u':
+ Result.Kind = TokenKind::uReg;
+ break;
+case 's':
+ Result.Kind = TokenKind::sReg;
+ break;
+default:
+ llvm_unreachable("Switch for an expected token was not provided");
+}
+return false;
+ }
+
+ // Keywords and Enums:
+ StringRef TokSpelling =
+ Buffer.take_while([](char C) { return isalnum(C) || C == '_'; });
+
+ // Define a large string switch statement for all the keywords and enums
+ auto Switch = llvm::StringSwitch(TokSpelling);
+#define KEYWORD(NAME) Switch.Case(#NAME, TokenKind::kw_##NAME);
+#define ENUM(NAME, LIT) Switch.CaseLower(LIT, To
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/llvm-beanz commented: In general I think this is very much moving in the right direction. I have two high level areas of feedback (see my more specific comments inline). In general, Lexer's don't often have the context to provide actionable error messages. As a result it is _usually_ better for the Lexer to push up an invalid token to the parser and allow the parser to provide a more contextually relevant diagnostic. The other area is around what is the responsibility of the parser vs the lexer. In general lexers just break the input into tokens, the parser then interprets and validates the tokens. In particular around numbers this is doing a lot of interpretation of the token values. I think it is probably better to have a clearer separation. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,194 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
llvm-beanz wrote:
As another note, lexing would usually identify the token, not try to interpret
it, meaning any string of characters that meet the definition of a numeric
constant. For example in the following code:
```
int8_t C = 2048;
float F = -2.8;
```
The token sequence generated by the lexer is [ `int8_t`, `C`, `=`, `2048`, `;`,
`float`, `F`, `=` ,`-`, `2.8`, `;`].
Lexing does not validate that 2048 cannot fit into C.
I think most of what this function does is more accurately "parsing" a number
rather than lexing.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -1017,4 +1017,15 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_expected_number_literal: Error<"expected numberic literal">;
+ def err_hlsl_number_literal_overflow :
+Error<"integer literal is too large to be represented in any
%select{signed |}0 integer type">;
llvm-beanz wrote:
I know I suggested matching the language in the other clang diagnostics, but
for root signatures these are required to be 32-bit types, not _any_ integer
type.
```suggestion
Error<"integer literal is too large to be represented in 32-bit
%select{signed |}0 integer type">;
```
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/llvm-beanz edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Note: if IsNumberChar allows for hexidecimal we will need to turn this
+ // into a diagnostics for potential fixed-point literals
+ assert(Literal.isIntegerLiteral() && "IsNumberChar will only support
digits");
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ // NOTE: for compabibility with DXC, we will treat any integer with '+' as an
+ // unsigned integer
+ llvm::APSInt X = llvm::APSInt(32, !Negative);
+ if (Literal.GetIntegerValue(X)) {
+// Report that the value has overflowed
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_number_literal_overflow)
+<< (unsigned)Signed << NumSpelling;
+return true;
+ }
+
+ X = Negative ? -X : X;
+ Result.NumLiteral = APValue(X);
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+// Record where this token is in the text for usage in parser diagnostics
+RootSignatureToken Result(SourceLoc);
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ default:
+break;
+ }
+
+ // Numeric constant
+ if (isdigit(C) || C == '-' || C == '+')
+return LexNumber(Result);
+
+ // All following tokens require at least one additional character
+ if (Buffer.size() <= 1) {
+PP.getDiagnostics().Report(Result.TokLoc, diag::err_hlsl_invalid_token);
llvm-beanz wrote:
Is it more useful to the user to note a failure in lexing, _or_ is it more
useful at the parsing level to be able to recognize that the token was set to
invalid when something else was expected?
Clang tends to do the later, which is why you get errors like "expected
identifier", because the parser _knows_ the next thing should be an identifier,
but the lexer just knows it didn't produce the right result (or any result).
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,194 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#126565): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
llvm-beanz wrote:
I have a grammar question here. Is the `-` or `+` part of the numeric literal
token, or is it better (as in C++), that the unary operator be its own token?
The reason I ask, is that if you look at the C++ (& HLSL) grammar leading `-`
is a unary operator, _not_ part of the numeric literal token string. This is
why the Clang parser really only supports parsing positive numbers, the
negation is a modifier.
Where this becomes relevant is a few lines below where you emit the
`err_hlsl_expected_number_literal`, that's really an error that should be
coming from _parsing_ not lexing because you're combining adjacent tokens and
recognizing that the grammar requires a number follow the sign token.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Note: if IsNumberChar allows for hexidecimal we will need to turn this
+ // into a diagnostics for potential fixed-point literals
+ assert(Literal.isIntegerLiteral() && "IsNumberChar will only support
digits");
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ // NOTE: for compabibility with DXC, we will treat any integer with '+' as an
+ // unsigned integer
+ llvm::APSInt X = llvm::APSInt(32, !Negative);
+ if (Literal.GetIntegerValue(X)) {
+// Report that the value has overflowed
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_number_literal_overflow)
+<< (unsigned)Signed << NumSpelling;
+return true;
+ }
+
+ X = Negative ? -X : X;
+ Result.NumLiteral = APValue(X);
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+// Record where this token is in the text for usage in parser diagnostics
+RootSignatureToken Result(SourceLoc);
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ default:
+break;
+ }
+
+ // Numeric constant
+ if (isdigit(C) || C == '-' || C == '+')
+return LexNumber(Result);
+
+ // All following tokens require at least one additional character
+ if (Buffer.size() <= 1) {
+PP.getDiagnostics().Report(Result.TokLoc, diag::err_hlsl_invalid_token);
inbelic wrote:
Would you also remove the same error message from
[here](https://github.com/llvm/llvm-project/blob/d648f4ccb8c74060e9e6d75b46f54ac0127e4302/clang/lib/Parse/ParseHLSLRootSignature.cpp#L161)?
They were added just to have _some_ indication that lexing failed.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 01/12] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b0847216c..7755c05bc8969ba 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 000..9625f6a5bd76d9e
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 000..39069e7cc399883
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
inbelic wrote:
Updated the comment to link to
https://github.com/llvm/llvm-project/issues/126565, which better describes this.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
github-actions[bot] wrote:
:warning: C/C++ code formatter, clang-format found issues in your code.
:warning:
You can test this locally with the following command:
``bash
git-clang-format --diff f739aa4004165dc64d3a1f418d5ad3c84886f01a
de364d896b24ea782b42e97fe76f71ebdbeaaa9e --extensions cpp,h --
clang/include/clang/Parse/ParseHLSLRootSignature.h
clang/lib/Parse/ParseHLSLRootSignature.cpp
clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
``
View the diff from clang-format here.
``diff
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
index 31529b0c03..15898f8c24 100644
--- a/clang/include/clang/Parse/ParseHLSLRootSignature.h
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -41,7 +41,8 @@ struct RootSignatureToken {
// Constructors
RootSignatureToken() : TokLoc(SourceLocation()) {}
RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
- RootSignatureToken(enum Kind Kind, clang::SourceLocation TokLoc) :
Kind(Kind), TokLoc(TokLoc) {}
+ RootSignatureToken(enum Kind Kind, clang::SourceLocation TokLoc)
+ : Kind(Kind), TokLoc(TokLoc) {}
};
using TokenKind = enum RootSignatureToken::Kind;
diff --git a/clang/lib/Parse/ParseHLSLRootSignature.cpp
b/clang/lib/Parse/ParseHLSLRootSignature.cpp
index 88d892dd43..84905674df 100644
--- a/clang/lib/Parse/ParseHLSLRootSignature.cpp
+++ b/clang/lib/Parse/ParseHLSLRootSignature.cpp
@@ -165,7 +165,8 @@ bool RootSignatureLexer::ConsumeToken() {
// This will be implicity be true if NextToken->Kind == end_of_stream
if (EndOfBuffer()) {
// Report unexpected end of tokens error
-PP.getDiagnostics().Report(SourceLoc,
diag::err_hlsl_rootsig_unexpected_eos);
+PP.getDiagnostics().Report(SourceLoc,
+ diag::err_hlsl_rootsig_unexpected_eos);
return true;
}
``
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Note: if IsNumberChar allows for hexidecimal we will need to turn this
+ // into a diagnostics for potential fixed-point literals
+ assert(Literal.isIntegerLiteral() && "IsNumberChar will only support
digits");
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ // NOTE: for compabibility with DXC, we will treat any integer with '+' as an
+ // unsigned integer
+ llvm::APSInt X = llvm::APSInt(32, !Negative);
+ if (Literal.GetIntegerValue(X)) {
+// Report that the value has overflowed
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_number_literal_overflow)
+<< (unsigned)Signed << NumSpelling;
+return true;
+ }
+
+ X = Negative ? -X : X;
+ Result.NumLiteral = APValue(X);
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
inbelic wrote:
Understood, I have updated this to only support a
`ConsumeToken`/`PeekNextToken` api.
It could be updated further to define and be used as an iterator. Although from
how it is used in the parser (usually through `PeekNextToken`). I think it
would essentially be just renaming `ConsumeToken` to the `++` operator. Maybe
there are other benefits I don't see?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
inbelic wrote:
That is correct. Updated to the concise message
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 01/11] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b0847216..7755c05bc8969b 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..9625f6a5bd76d9
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 00..39069e7cc39988
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes the in
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Note: if IsNumberChar allows for hexidecimal we will need to turn this
+ // into a diagnostics for potential fixed-point literals
+ assert(Literal.isIntegerLiteral() && "IsNumberChar will only support
digits");
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ // NOTE: for compabibility with DXC, we will treat any integer with '+' as an
+ // unsigned integer
+ llvm::APSInt X = llvm::APSInt(32, !Negative);
+ if (Literal.GetIntegerValue(X)) {
+// Report that the value has overflowed
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_number_literal_overflow)
+<< (unsigned)Signed << NumSpelling;
+return true;
+ }
+
+ X = Negative ? -X : X;
+ Result.NumLiteral = APValue(X);
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+// Record where this token is in the text for usage in parser diagnostics
+RootSignatureToken Result(SourceLoc);
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ default:
+break;
+ }
+
+ // Numeric constant
+ if (isdigit(C) || C == '-' || C == '+')
+return LexNumber(Result);
+
+ // All following tokens require at least one additional character
+ if (Buffer.size() <= 1) {
+PP.getDiagnostics().Report(Result.TokLoc, diag::err_hlsl_invalid_token);
llvm-beanz wrote:
This error doesn't seem particularly useful to end users who make a mistake.
Why not just return false here because it can't construct a token?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
llvm-beanz wrote:
IIUC, this occurs if the parser sees a `-` or `+` but no specified literal
value after that. Is that correct?
A more succinct error message like "expected numeric literal" is probably a
better message. The current one "expected number literal is not a supported
number literal of unsigned integer or integer" is quite confusing.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,151 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+RootSignatureToken Result;
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ // Record where this token is in the text for usage in parser diagnostics
+ Result.TokLoc = SourceLoc;
+
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ default:
+break;
+ }
+
+ // Numeric constant
+ if (isdigit(C) || C == '-')
+return LexNumber(Result);
+
+ // All following tokens require at least one additional character
+ if (Buffer.size() <= 1)
+return true; // TODO: Report invalid token error
+
+ // Peek at the next character to deteremine token type
+ char NextC = Buffer[1];
+
+ // Registers: [tsub][0-9+]
+ if ((C == 't' || C == 's' || C == 'u' || C == 'b') && isdigit(NextC)) {
+AdvanceBuffer();
+
+if (LexNumber(Result))
+ return true;
+
+// Lex number could also parse a float so ensure it was an unsigned int
+if (Result.Kind != TokenKind::int_literal || Result.Signed)
+ return true; // Return invalid number literal for register error
+
+// Convert character to the register type.
+// This is done after LexNumber to override the TokenKind
+switch (C) {
+case 'b':
+ Result.Kind = TokenKind::bReg;
+ break;
+case 't':
+ Result.Kind = TokenKind::tReg;
+ break;
+case 'u':
+ Result.Kind = TokenKind::uReg;
+ break;
+case 's':
+ Result.Kind = TokenKind::sReg;
+ break;
+default:
+ llvm_unreachable("Switch for an expected token was not provided");
+ return true;
+}
+return false;
+ }
+
+ // Keywords and Enums:
llvm-beanz wrote:
I definitely prefer starting with the simple approach and only going
complicated if we need to. DXC had a bunch of hand-rolled string matching
functions that unrolled the strings... and a few of them had some bad bugs. So
this is the perfect way to start.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Note: if IsNumberChar allows for hexidecimal we will need to turn this
+ // into a diagnostics for potential fixed-point literals
+ assert(Literal.isIntegerLiteral() && "IsNumberChar will only support
digits");
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ // NOTE: for compabibility with DXC, we will treat any integer with '+' as an
+ // unsigned integer
+ llvm::APSInt X = llvm::APSInt(32, !Negative);
+ if (Literal.GetIntegerValue(X)) {
+// Report that the value has overflowed
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_number_literal_overflow)
+<< (unsigned)Signed << NumSpelling;
+return true;
+ }
+
+ X = Negative ? -X : X;
+ Result.NumLiteral = APValue(X);
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
llvm-beanz wrote:
I'm not sure this function should exist.
Most modern compilers don't lex entire source files into allocated lists of
tokens then parse, they do the two at the same time never allocating more than
a couple tokens at a time. You can see this in Clang where the `ConsumeToken`
API advances the token to the next token, and is called throughout the parser.
I've seen other compilers where the tokenizer is implemented as effectively an
iterator where the iterator state tracks the current token.
For context, the reason compilers tend to do this is because lexing isn't
always "fast". In particular translation of numeric literals into actual
integer values can be quite slow, so if you lex all the tokens, you do all the
transformations even if an error would be detected during parsing. If you lex
one token at a time, you stop lexing once the error is encountered providing
faster user feedback.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -1017,4 +1017,15 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_number_literal:
+Error<"expected number literal is not a supported number literal of
unsigned integer or integer">;
+ def err_hlsl_number_literal_overflow :
+Error<"provided %select{unsigned integer|signed integer}0 literal '%1'
that overflows the maximum of 32 bits">;
llvm-beanz wrote:
The context of this error is really similar to an existing clang error:
err_integer_literal_too_large. I don't think it makes sense to share the
message, but we should at least use similar language.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/damyanp edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/damyanp commented: LGTM - I'm surprised we don't have all of the tokens we're going to need for root signatures included in this PR. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,171 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
damyanp wrote:
The issue linked here (#120472) is just about implementing the parser. Is there
a reason not to get float tokens recognized by the lexer?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 1/8] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b0847216..7755c05bc8969b 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..9625f6a5bd76d9
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 00..39069e7cc39988
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes the inte
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,173 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ if (!Literal.isIntegerLiteral()) {
+// Note: if IsNumberChar allows for hexidecimal we will need to turn this
+// into a diagnostics for potential fixed-point literals
+llvm_unreachable("IsNumberChar will only support digits");
+return true;
+ }
+
+ // Retrieve the number value to store into the token
+ Result.Kind = TokenKind::int_literal;
+
+ llvm::APSInt X = llvm::APSInt(32, !Signed);
bogner wrote:
What happens if we have a value like `+2147483648`? This would fit in an
unsigned 32-bit int, but not a signed one. However, we're deciding that the
value is signed if the `+` is present above. Will this error and tell us that
there's overflow, treat the value as unsigned because of the range, or give us
a garbage negative number?
If it's one of the first two options, does the one we're implementing match
DXC's behaviour here?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,173 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace clang {
+namespace hlsl {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO(#120472): extend for float support exponents
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ bool Negative = Buffer.front() == '-';
+ bool Signed = Negative || Buffer.front() == '+';
+ if (Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsNumberChar);
+
+ // Catch this now as the Literal Parser will accept it as valid
+ if (NumSpelling.empty()) {
+PP.getDiagnostics().Report(Result.TokLoc,
+ diag::err_hlsl_invalid_number_literal);
+return true;
+ }
+
+ // Parse the numeric value and do semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ if (!Literal.isIntegerLiteral()) {
+// Note: if IsNumberChar allows for hexidecimal we will need to turn this
+// into a diagnostics for potential fixed-point literals
+llvm_unreachable("IsNumberChar will only support digits");
+return true;
+ }
bogner wrote:
`if (x) llvm_unreachable(...)` is a bit of a code smell. Better to simply
`assert(x && ...)`
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,134 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
bogner wrote:
This feels awfully specific for a "don't care" value - it probably deserves a
comment to the effect that "we need a triple in order to create a target info,
but which target doesn't matter for our purposes". Also I'd probably simplify
this to either use a valid dxil triple or just "x86_64" (or "dxil"), just
because the darwin triple in an HLSL test is likely to confuse people.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
inbelic wrote: HLSL Build failure is because there is an assertion that DIAG_SIZE_LEX is not big enough (insufficient). I did add some new diag errors under the Lex category so it seems the threshold is passed on some environments. >From >[here](https://github.com/llvm/llvm-project/blame/617278e7b0c937fccbf7d67d14f053c3409bc33f/clang/include/clang/Basic/DiagnosticIDs.h#L40), > it seems that we could, and there is trend to, bump this limit when needed. >IIUC, I should create a separate pr to bump this. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
inbelic wrote: Updated to include diagnostics output and the relevant testing. I have rebased to split the pr into smaller incremental changes. But I have taken care to address all previous comments. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,96 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Lex/LiteralSupport.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSwitch.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Retain if the uint32_t bits represent a signed integer
+ bool Signed = false;
+ union {
+uint32_t IntLiteral = 0;
+float FloatLiteral;
inbelic wrote:
Yep, good idea
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From 98deff6a407b912852e70b2bdc3618aaec8a1931 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Fri, 24 Jan 2025 22:23:39 +
Subject: [PATCH 1/5] [HLSL][RootSignature] Initial Lexer Definition with
puncuators
- Defines the RootSignatureLexer class
- Defines the test harness required for testing
- Implements the punctuator tokens and tests functionality
---
.../include/clang/Basic/DiagnosticLexKinds.td | 6 +
.../Parse/HLSLRootSignatureTokenKinds.def | 35
.../clang/Parse/ParseHLSLRootSignature.h | 79 +
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 50 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 167 ++
8 files changed, 365 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Basic/DiagnosticLexKinds.td
b/clang/include/clang/Basic/DiagnosticLexKinds.td
index 959376b0847216..7755c05bc8969b 100644
--- a/clang/include/clang/Basic/DiagnosticLexKinds.td
+++ b/clang/include/clang/Basic/DiagnosticLexKinds.td
@@ -1017,4 +1017,10 @@ Error<"'#pragma unsafe_buffer_usage' was not ended">;
def err_pp_pragma_unsafe_buffer_usage_syntax :
Error<"expected 'begin' or 'end'">;
+
+// HLSL Root Signature Lexing Errors
+let CategoryName = "Root Signature Lexical Issue" in {
+ def err_hlsl_invalid_token: Error<"unable to lex a valid Root Signature
token">;
+}
+
}
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..9625f6a5bd76d9
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,35 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+
+// General Tokens:
+TOK(invalid)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+#undef PUNCTUATOR
+#undef TOK
diff --git a/clang/include/clang/Parse/ParseHLSLRootSignature.h
b/clang/include/clang/Parse/ParseHLSLRootSignature.h
new file mode 100644
index 00..39069e7cc39988
--- /dev/null
+++ b/clang/include/clang/Parse/ParseHLSLRootSignature.h
@@ -0,0 +1,79 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Basic/DiagnosticLex.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+namespace clang {
+namespace hlsl {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Constructors
+ RootSignatureToken(clang::SourceLocation TokLoc) : TokLoc(TokLoc) {}
+};
+using TokenKind = enum RootSignatureToken::Kind;
+
+class RootSignatureLexer {
+public:
+ RootSignatureLexer(StringRef Signature, clang::SourceLocation SourceLoc,
+ clang::Preprocessor &PP)
+ : Buffer(Signature), SourceLoc(SourceLoc), PP(PP) {}
+
+ // Consumes the inte
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,134 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
inbelic wrote:
It was taken over from
https://github.com/llvm/llvm-project/blob/main/clang/unittests/Lex/LexerTest.cpp
and it should denote "I don't care"
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,134 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
damyanp wrote:
How was `"x86_64-apple-darwin11.1.0"` chosen as a value to use here? Does it
really mean "I don't care"?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,133 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ std::unique_ptr CreatePP(StringRef Source,
+ TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+std::unique_ptr PP = std::make_unique(
+std::make_shared(), Diags, LangOpts, SourceMgr,
+HeaderInfo, ModLoader,
+/*IILookup =*/nullptr,
+/*OwnsHeaderSearch =*/false);
+PP->Initialize(*Target);
+PP->EnterMainSourceFile();
+return PP;
+ }
+
+ void CheckTokens(SmallVector &Computed,
+ SmallVector &Expected) {
+ASSERT_EQ(Computed.size(), Expected.size());
+for (unsigned I = 0, E = Expected.size(); I != E; ++I) {
+ ASSERT_EQ(Computed[I].Kind, Expected[I]);
+}
+ }
+
+ FileSystemOptions FileMgrOpts;
+ FileManager FileMgr;
+ IntrusiveRefCntPtr DiagID;
+ DiagnosticsEngine Diags;
+ SourceManager SourceMgr;
+ LangOptions LangOpts;
+ std::shared_ptr TargetOpts;
+ IntrusiveRefCntPtr Target;
+};
+
+TEST_F(ParseHLSLRootSignatureTest, LexValidTokensTest) {
+ const llvm::StringLiteral Source = R"cc(
+-42
+
+b0 t43 u987 s234
+
+(),|=
+
+DescriptorTable
+
+CBV SRV UAV Sampler
+space visibility flags
+numDescriptors offset
+
+DESCRIPTOR_RANGE_OFFSET_APPEND
+
+DATA_VOLATILE
+DATA_STATIC_WHILE_SET_AT_EXECUTE
+DATA_STATIC
+DESCRIPTORS_VOLATILE
+DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS
+
+shader_visibility_all
+shader_visibility_vertex
+shader_visibility_hull
+shader_visibility_domain
+shader_visibility_geometry
+shader_visibility_pixel
+shader_visibility_amplification
+shader_visibility_mesh
+
+42 +42
+ )cc";
+
+ TrivialModuleLoader ModLoader;
+ auto PP = CreatePP(Source, ModLoader);
+ auto TokLoc = SourceLocation();
+
+ RootSignatureLexer Lexer(Source, TokLoc, *PP);
+
+ SmallVector Tokens = {
+ RootSignatureToken() // invalid token for completeness
+ };
+ ASSERT_FALSE(Lexer.Lex(Tokens));
llvm-beanz wrote:
I'll reiterate, I think this is the wrong tradeoff, because it makes it
significantly more difficult to review the follow-up change since we'll need to
compare not just the incoming PR but also go back and look at this PR to ensure
all the appropriate error states are captured and tested.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
llvm-beanz wrote: As I said in your other PR, I believe not including error reporting and the appropriate testing makes it significantly more difficult to review the subsequent commit since not all the appropriate code will even be in the PR. https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,96 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Lex/LiteralSupport.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSwitch.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Retain if the uint32_t bits represent a signed integer
+ bool Signed = false;
+ union {
+uint32_t IntLiteral = 0;
+float FloatLiteral;
llvm-beanz wrote:
Should we have a union here or should this be an `llvm::APValue`?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,133 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ std::unique_ptr CreatePP(StringRef Source,
+ TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+std::unique_ptr PP = std::make_unique(
+std::make_shared(), Diags, LangOpts, SourceMgr,
+HeaderInfo, ModLoader,
+/*IILookup =*/nullptr,
+/*OwnsHeaderSearch =*/false);
+PP->Initialize(*Target);
+PP->EnterMainSourceFile();
+return PP;
+ }
+
+ void CheckTokens(SmallVector &Computed,
+ SmallVector &Expected) {
+ASSERT_EQ(Computed.size(), Expected.size());
+for (unsigned I = 0, E = Expected.size(); I != E; ++I) {
+ ASSERT_EQ(Computed[I].Kind, Expected[I]);
+}
+ }
+
+ FileSystemOptions FileMgrOpts;
+ FileManager FileMgr;
+ IntrusiveRefCntPtr DiagID;
+ DiagnosticsEngine Diags;
+ SourceManager SourceMgr;
+ LangOptions LangOpts;
+ std::shared_ptr TargetOpts;
+ IntrusiveRefCntPtr Target;
+};
+
+TEST_F(ParseHLSLRootSignatureTest, LexValidTokensTest) {
+ const llvm::StringLiteral Source = R"cc(
+-42
+
+b0 t43 u987 s234
+
+(),|=
+
+DescriptorTable
+
+CBV SRV UAV Sampler
+space visibility flags
+numDescriptors offset
+
+DESCRIPTOR_RANGE_OFFSET_APPEND
+
+DATA_VOLATILE
+DATA_STATIC_WHILE_SET_AT_EXECUTE
+DATA_STATIC
+DESCRIPTORS_VOLATILE
+DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS
+
+shader_visibility_all
+shader_visibility_vertex
+shader_visibility_hull
+shader_visibility_domain
+shader_visibility_geometry
+shader_visibility_pixel
+shader_visibility_amplification
+shader_visibility_mesh
+
+42 +42
+ )cc";
+
+ TrivialModuleLoader ModLoader;
+ auto PP = CreatePP(Source, ModLoader);
+ auto TokLoc = SourceLocation();
+
+ RootSignatureLexer Lexer(Source, TokLoc, *PP);
+
+ SmallVector Tokens = {
+ RootSignatureToken() // invalid token for completeness
+ };
+ ASSERT_FALSE(Lexer.Lex(Tokens));
V-FEXrt wrote:
No, I'm happy to punt it
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,153 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
V-FEXrt wrote:
Yep that would be great!
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From c1fc823abf07dfb8326b6190672d9881890bbb15 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Tue, 14 Jan 2025 22:22:45 +
Subject: [PATCH 1/7] [HLSL][RootSignature] Implement Lexing of
DescriptorTables
- Define required tokens to parse a Descriptor Table in TokenKinds.def
- Implements a Lexer to handle all of the defined tokens in
ParseHLSLRootSignature
---
.../Parse/HLSLRootSignatureTokenKinds.def | 121 ++
.../clang/Parse/ParseHLSLRootSignature.h | 96 +++
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 151 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 130 +++
7 files changed, 526 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..5e47af8d4b1364
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,121 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+#ifndef KEYWORD
+#define KEYWORD(X) TOK(kw_ ## X)
+#endif
+#ifndef ENUM
+#define ENUM(NAME, LIT) TOK(en_ ## NAME)
+#endif
+
+// Defines the various types of enum
+#ifndef DESCRIPTOR_RANGE_OFFSET_ENUM
+#define DESCRIPTOR_RANGE_OFFSET_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef ROOT_DESCRIPTOR_FLAG_ENUM
+#define ROOT_DESCRIPTOR_FLAG_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+// Note: ON denotes that the flag is unique from the above Root Descriptor
+// Flags. This is required to avoid token kind enum conflicts.
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_OFF
+#define DESCRIPTOR_RANGE_FLAG_ENUM_OFF(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_ON
+#define DESCRIPTOR_RANGE_FLAG_ENUM_ON(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM
+#define DESCRIPTOR_RANGE_FLAG_ENUM(NAME, LIT, ON)
DESCRIPTOR_RANGE_FLAG_ENUM_##ON(NAME, LIT)
+#endif
+#ifndef SHADER_VISIBILITY_ENUM
+#define SHADER_VISIBILITY_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+
+// General Tokens:
+TOK(invalid)
+TOK(int_literal)
+
+// Register Tokens:
+TOK(bReg)
+TOK(tReg)
+TOK(uReg)
+TOK(sReg)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+// RootElement Keywords:
+KEYWORD(DescriptorTable)
+
+// DescriptorTable Keywords:
+KEYWORD(CBV)
+KEYWORD(SRV)
+KEYWORD(UAV)
+KEYWORD(Sampler)
+
+// General Parameter Keywords:
+KEYWORD(space)
+KEYWORD(visibility)
+KEYWORD(flags)
+
+// View Parameter Keywords:
+KEYWORD(numDescriptors)
+KEYWORD(offset)
+
+// Descriptor Range Offset Enum:
+DESCRIPTOR_RANGE_OFFSET_ENUM(DescriptorRangeOffsetAppend,
"DESCRIPTOR_RANGE_OFFSET_APPEND")
+
+// Root Descriptor Flag Enums:
+ROOT_DESCRIPTOR_FLAG_ENUM(DataVolatile, "DATA_VOLATILE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStatic, "DATA_STATIC")
+
+// Descriptor Range Flag Enums:
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsVolatile, "DESCRIPTORS_VOLATILE", ON)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataVolatile, "DATA_VOLATILE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStatic, "DATA_STATIC", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsStaticKeepingBufferBoundsChecks,
"DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS", ON)
+
+// Shader Visibiliy Enums:
+SHADER_VISIBILITY_ENUM(All, "SHADER_VISIBILITY_ALL")
+SHADER_VISIBILITY_ENUM(Vertex, "SHADER_VISIBILITY_VERTEX")
+SHADER_VISIBILITY_ENUM(Hull, "SHADER_VISIBILITY_HULL")
+SHADER_VISIBILITY_E
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,133 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ std::unique_ptr CreatePP(StringRef Source,
+ TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+std::unique_ptr PP = std::make_unique(
+std::make_shared(), Diags, LangOpts, SourceMgr,
+HeaderInfo, ModLoader,
+/*IILookup =*/nullptr,
+/*OwnsHeaderSearch =*/false);
+PP->Initialize(*Target);
+PP->EnterMainSourceFile();
+return PP;
+ }
+
+ void CheckTokens(SmallVector &Computed,
+ SmallVector &Expected) {
+ASSERT_EQ(Computed.size(), Expected.size());
+for (unsigned I = 0, E = Expected.size(); I != E; ++I) {
+ ASSERT_EQ(Computed[I].Kind, Expected[I]);
+}
+ }
+
+ FileSystemOptions FileMgrOpts;
+ FileManager FileMgr;
+ IntrusiveRefCntPtr DiagID;
+ DiagnosticsEngine Diags;
+ SourceManager SourceMgr;
+ LangOptions LangOpts;
+ std::shared_ptr TargetOpts;
+ IntrusiveRefCntPtr Target;
+};
+
+TEST_F(ParseHLSLRootSignatureTest, LexValidTokensTest) {
+ const llvm::StringLiteral Source = R"cc(
+-42
+
+b0 t43 u987 s234
+
+(),|=
+
+DescriptorTable
+
+CBV SRV UAV Sampler
+space visibility flags
+numDescriptors offset
+
+DESCRIPTOR_RANGE_OFFSET_APPEND
+
+DATA_VOLATILE
+DATA_STATIC_WHILE_SET_AT_EXECUTE
+DATA_STATIC
+DESCRIPTORS_VOLATILE
+DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS
+
+shader_visibility_all
+shader_visibility_vertex
+shader_visibility_hull
+shader_visibility_domain
+shader_visibility_geometry
+shader_visibility_pixel
+shader_visibility_amplification
+shader_visibility_mesh
+
+42 +42
+ )cc";
+
+ TrivialModuleLoader ModLoader;
+ auto PP = CreatePP(Source, ModLoader);
+ auto TokLoc = SourceLocation();
+
+ RootSignatureLexer Lexer(Source, TokLoc, *PP);
+
+ SmallVector Tokens = {
+ RootSignatureToken() // invalid token for completeness
+ };
+ ASSERT_FALSE(Lexer.Lex(Tokens));
+
+ SmallVector Expected = {
inbelic wrote:
This test-case illustrates that we can lex all of the defined tokens in the
`TokenKinds.def`, so to save typing all the token kinds out again I construct a
vector of all the tokenkinds using a macro. It does so my creating a big
initialization list for the SmallVector.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,133 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ std::unique_ptr CreatePP(StringRef Source,
+ TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+std::unique_ptr PP = std::make_unique(
+std::make_shared(), Diags, LangOpts, SourceMgr,
+HeaderInfo, ModLoader,
+/*IILookup =*/nullptr,
+/*OwnsHeaderSearch =*/false);
+PP->Initialize(*Target);
+PP->EnterMainSourceFile();
+return PP;
+ }
+
+ void CheckTokens(SmallVector &Computed,
+ SmallVector &Expected) {
+ASSERT_EQ(Computed.size(), Expected.size());
+for (unsigned I = 0, E = Expected.size(); I != E; ++I) {
+ ASSERT_EQ(Computed[I].Kind, Expected[I]);
+}
+ }
+
+ FileSystemOptions FileMgrOpts;
+ FileManager FileMgr;
+ IntrusiveRefCntPtr DiagID;
+ DiagnosticsEngine Diags;
+ SourceManager SourceMgr;
+ LangOptions LangOpts;
+ std::shared_ptr TargetOpts;
+ IntrusiveRefCntPtr Target;
+};
+
+TEST_F(ParseHLSLRootSignatureTest, LexValidTokensTest) {
+ const llvm::StringLiteral Source = R"cc(
+-42
+
+b0 t43 u987 s234
+
+(),|=
+
+DescriptorTable
+
+CBV SRV UAV Sampler
+space visibility flags
+numDescriptors offset
+
+DESCRIPTOR_RANGE_OFFSET_APPEND
+
+DATA_VOLATILE
+DATA_STATIC_WHILE_SET_AT_EXECUTE
+DATA_STATIC
+DESCRIPTORS_VOLATILE
+DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS
+
+shader_visibility_all
+shader_visibility_vertex
+shader_visibility_hull
+shader_visibility_domain
+shader_visibility_geometry
+shader_visibility_pixel
+shader_visibility_amplification
+shader_visibility_mesh
+
+42 +42
+ )cc";
+
+ TrivialModuleLoader ModLoader;
+ auto PP = CreatePP(Source, ModLoader);
+ auto TokLoc = SourceLocation();
+
+ RootSignatureLexer Lexer(Source, TokLoc, *PP);
+
+ SmallVector Tokens = {
+ RootSignatureToken() // invalid token for completeness
+ };
+ ASSERT_FALSE(Lexer.Lex(Tokens));
inbelic wrote:
Yes, I decided to leave out the implementation of generating and testing
diagnostics from this pr, just to help with the size. If you feel that should
they be here to ensure correctness I could add it.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,153 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
inbelic wrote:
Yep, I agree. In this case, it will be added when I get to supporting
`StaticSampler`s so it will be gone by the time the parent issue is resolved. I
could update the todo to reference the parent issue though?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/V-FEXrt edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,153 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
V-FEXrt wrote:
I'm personally not a big fan of untracked TODOs in code. They quickly fall out
of context and never get resolved. Would be good to make sure this is tracked
as part of an issue if it isn't resolved before merge
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,133 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ std::unique_ptr CreatePP(StringRef Source,
+ TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+std::unique_ptr PP = std::make_unique(
+std::make_shared(), Diags, LangOpts, SourceMgr,
+HeaderInfo, ModLoader,
+/*IILookup =*/nullptr,
+/*OwnsHeaderSearch =*/false);
+PP->Initialize(*Target);
+PP->EnterMainSourceFile();
+return PP;
+ }
+
+ void CheckTokens(SmallVector &Computed,
+ SmallVector &Expected) {
+ASSERT_EQ(Computed.size(), Expected.size());
+for (unsigned I = 0, E = Expected.size(); I != E; ++I) {
+ ASSERT_EQ(Computed[I].Kind, Expected[I]);
+}
+ }
+
+ FileSystemOptions FileMgrOpts;
+ FileManager FileMgr;
+ IntrusiveRefCntPtr DiagID;
+ DiagnosticsEngine Diags;
+ SourceManager SourceMgr;
+ LangOptions LangOpts;
+ std::shared_ptr TargetOpts;
+ IntrusiveRefCntPtr Target;
+};
+
+TEST_F(ParseHLSLRootSignatureTest, LexValidTokensTest) {
+ const llvm::StringLiteral Source = R"cc(
+-42
+
+b0 t43 u987 s234
+
+(),|=
+
+DescriptorTable
+
+CBV SRV UAV Sampler
+space visibility flags
+numDescriptors offset
+
+DESCRIPTOR_RANGE_OFFSET_APPEND
+
+DATA_VOLATILE
+DATA_STATIC_WHILE_SET_AT_EXECUTE
+DATA_STATIC
+DESCRIPTORS_VOLATILE
+DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS
+
+shader_visibility_all
+shader_visibility_vertex
+shader_visibility_hull
+shader_visibility_domain
+shader_visibility_geometry
+shader_visibility_pixel
+shader_visibility_amplification
+shader_visibility_mesh
+
+42 +42
+ )cc";
+
+ TrivialModuleLoader ModLoader;
+ auto PP = CreatePP(Source, ModLoader);
+ auto TokLoc = SourceLocation();
+
+ RootSignatureLexer Lexer(Source, TokLoc, *PP);
+
+ SmallVector Tokens = {
+ RootSignatureToken() // invalid token for completeness
+ };
+ ASSERT_FALSE(Lexer.Lex(Tokens));
+
+ SmallVector Expected = {
V-FEXrt wrote:
Its probably my unfamiliarity with the code but I can't see how this is
actually building up the list of expected tokens. Importing the `.def` file
here _seems_ strange
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,133 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ std::unique_ptr CreatePP(StringRef Source,
+ TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+std::unique_ptr PP = std::make_unique(
+std::make_shared(), Diags, LangOpts, SourceMgr,
+HeaderInfo, ModLoader,
+/*IILookup =*/nullptr,
+/*OwnsHeaderSearch =*/false);
+PP->Initialize(*Target);
+PP->EnterMainSourceFile();
+return PP;
+ }
+
+ void CheckTokens(SmallVector &Computed,
+ SmallVector &Expected) {
+ASSERT_EQ(Computed.size(), Expected.size());
+for (unsigned I = 0, E = Expected.size(); I != E; ++I) {
+ ASSERT_EQ(Computed[I].Kind, Expected[I]);
+}
+ }
+
+ FileSystemOptions FileMgrOpts;
+ FileManager FileMgr;
+ IntrusiveRefCntPtr DiagID;
+ DiagnosticsEngine Diags;
+ SourceManager SourceMgr;
+ LangOptions LangOpts;
+ std::shared_ptr TargetOpts;
+ IntrusiveRefCntPtr Target;
+};
+
+TEST_F(ParseHLSLRootSignatureTest, LexValidTokensTest) {
+ const llvm::StringLiteral Source = R"cc(
+-42
+
+b0 t43 u987 s234
+
+(),|=
+
+DescriptorTable
+
+CBV SRV UAV Sampler
+space visibility flags
+numDescriptors offset
+
+DESCRIPTOR_RANGE_OFFSET_APPEND
+
+DATA_VOLATILE
+DATA_STATIC_WHILE_SET_AT_EXECUTE
+DATA_STATIC
+DESCRIPTORS_VOLATILE
+DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS
+
+shader_visibility_all
+shader_visibility_vertex
+shader_visibility_hull
+shader_visibility_domain
+shader_visibility_geometry
+shader_visibility_pixel
+shader_visibility_amplification
+shader_visibility_mesh
+
+42 +42
+ )cc";
+
+ TrivialModuleLoader ModLoader;
+ auto PP = CreatePP(Source, ModLoader);
+ auto TokLoc = SourceLocation();
+
+ RootSignatureLexer Lexer(Source, TokLoc, *PP);
+
+ SmallVector Tokens = {
+ RootSignatureToken() // invalid token for completeness
+ };
+ ASSERT_FALSE(Lexer.Lex(Tokens));
V-FEXrt wrote:
Would be nice to see a more robust set of failure tests
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/V-FEXrt approved this pull request. Personally happy with the code at a high level, though it would be best to have someone with domain knowledge also approve https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
inbelic wrote: Note to self: change namespace from `llvm` to `clang` https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From c1fc823abf07dfb8326b6190672d9881890bbb15 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Tue, 14 Jan 2025 22:22:45 +
Subject: [PATCH 1/5] [HLSL][RootSignature] Implement Lexing of
DescriptorTables
- Define required tokens to parse a Descriptor Table in TokenKinds.def
- Implements a Lexer to handle all of the defined tokens in
ParseHLSLRootSignature
---
.../Parse/HLSLRootSignatureTokenKinds.def | 121 ++
.../clang/Parse/ParseHLSLRootSignature.h | 96 +++
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 151 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 130 +++
7 files changed, 526 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..5e47af8d4b1364
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,121 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+#ifndef KEYWORD
+#define KEYWORD(X) TOK(kw_ ## X)
+#endif
+#ifndef ENUM
+#define ENUM(NAME, LIT) TOK(en_ ## NAME)
+#endif
+
+// Defines the various types of enum
+#ifndef DESCRIPTOR_RANGE_OFFSET_ENUM
+#define DESCRIPTOR_RANGE_OFFSET_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef ROOT_DESCRIPTOR_FLAG_ENUM
+#define ROOT_DESCRIPTOR_FLAG_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+// Note: ON denotes that the flag is unique from the above Root Descriptor
+// Flags. This is required to avoid token kind enum conflicts.
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_OFF
+#define DESCRIPTOR_RANGE_FLAG_ENUM_OFF(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_ON
+#define DESCRIPTOR_RANGE_FLAG_ENUM_ON(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM
+#define DESCRIPTOR_RANGE_FLAG_ENUM(NAME, LIT, ON)
DESCRIPTOR_RANGE_FLAG_ENUM_##ON(NAME, LIT)
+#endif
+#ifndef SHADER_VISIBILITY_ENUM
+#define SHADER_VISIBILITY_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+
+// General Tokens:
+TOK(invalid)
+TOK(int_literal)
+
+// Register Tokens:
+TOK(bReg)
+TOK(tReg)
+TOK(uReg)
+TOK(sReg)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+// RootElement Keywords:
+KEYWORD(DescriptorTable)
+
+// DescriptorTable Keywords:
+KEYWORD(CBV)
+KEYWORD(SRV)
+KEYWORD(UAV)
+KEYWORD(Sampler)
+
+// General Parameter Keywords:
+KEYWORD(space)
+KEYWORD(visibility)
+KEYWORD(flags)
+
+// View Parameter Keywords:
+KEYWORD(numDescriptors)
+KEYWORD(offset)
+
+// Descriptor Range Offset Enum:
+DESCRIPTOR_RANGE_OFFSET_ENUM(DescriptorRangeOffsetAppend,
"DESCRIPTOR_RANGE_OFFSET_APPEND")
+
+// Root Descriptor Flag Enums:
+ROOT_DESCRIPTOR_FLAG_ENUM(DataVolatile, "DATA_VOLATILE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStatic, "DATA_STATIC")
+
+// Descriptor Range Flag Enums:
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsVolatile, "DESCRIPTORS_VOLATILE", ON)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataVolatile, "DATA_VOLATILE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStatic, "DATA_STATIC", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsStaticKeepingBufferBoundsChecks,
"DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS", ON)
+
+// Shader Visibiliy Enums:
+SHADER_VISIBILITY_ENUM(All, "SHADER_VISIBILITY_ALL")
+SHADER_VISIBILITY_ENUM(Vertex, "SHADER_VISIBILITY_VERTEX")
+SHADER_VISIBILITY_ENUM(Hull, "SHADER_VISIBILITY_HULL")
+SHADER_VISIBILITY_E
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,130 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ Preprocessor *CreatePP(StringRef Source, TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+Preprocessor *PP =
+new Preprocessor(std::make_shared(), Diags,
inbelic wrote:
Hm, I have no idea what I was doing before to get it to not work... you can
just pass the unique pointer as is and it works😅
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/V-FEXrt edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/V-FEXrt edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,130 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ Preprocessor *CreatePP(StringRef Source, TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+Preprocessor *PP =
+new Preprocessor(std::make_shared(), Diags,
V-FEXrt wrote:
I think you can just derefence the `unique_ptr`. Something like `*ptr->get()`
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,130 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ Preprocessor *CreatePP(StringRef Source, TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+Preprocessor *PP =
+new Preprocessor(std::make_shared(), Diags,
inbelic wrote:
Ya, I felt bad using these. The reason I did so is that when I tried to use the
`unique_ptr` I wasn't sure how to pass that into the lexer as a `Preprocessor
&`? Will give it another go.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+RootSignatureToken Result;
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ // Record where this token is in the text for usage in parser diagnostics
+ Result.TokLoc = SourceLoc;
+
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
inbelic wrote:
Yes, it is defined this way so that the `PUNCUATOR` that is in the `.def` is
the "default" macro. This is used when we do `#define TOK(X) X,` so then
`PUNCUATOR(X) -> TOK(pu_X) -> X,`. It follows the pattern in some other
`TokenKinds.def`.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
inbelic wrote:
That follows the rest of the code conventions, thanks.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
inbelic wrote:
I guess DXC actually does allow a `+` to be the first character, as well as,
`.`. For the sake of compatibility, we should bring those forward.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/V-FEXrt edited https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+RootSignatureToken Result;
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ // Record where this token is in the text for usage in parser diagnostics
+ Result.TokLoc = SourceLoc;
+
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
V-FEXrt wrote:
if it must always be defined before inclusion you could do
```
#ifndef PUNCTUATOR
#error some message
#endif
```
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+RootSignatureToken Result;
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ // Record where this token is in the text for usage in parser diagnostics
+ Result.TokLoc = SourceLoc;
+
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
V-FEXrt wrote:
Is there a particular reason you define `PUNCTUATOR` both here and in the
`.def` file? It makes the `.def` file harder to follow since the `#define
PUNCTUATOR` there isn't actually used
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,130 @@
+//=== ParseHLSLRootSignatureTest.cpp - Parse Root Signature tests
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/ModuleLoader.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+
+#include "clang/Parse/ParseHLSLRootSignature.h"
+#include "gtest/gtest.h"
+
+using namespace llvm::hlsl::root_signature;
+using namespace clang;
+
+namespace {
+
+// The test fixture.
+class ParseHLSLRootSignatureTest : public ::testing::Test {
+protected:
+ ParseHLSLRootSignatureTest()
+ : FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
+Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
+SourceMgr(Diags, FileMgr), TargetOpts(new TargetOptions) {
+TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
+Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
+ }
+
+ Preprocessor *CreatePP(StringRef Source, TrivialModuleLoader &ModLoader) {
+std::unique_ptr Buf =
+llvm::MemoryBuffer::getMemBuffer(Source);
+SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
+
+HeaderSearch HeaderInfo(std::make_shared(), SourceMgr,
+Diags, LangOpts, Target.get());
+Preprocessor *PP =
+new Preprocessor(std::make_shared(), Diags,
V-FEXrt wrote:
imo unless you have a strong reason for it, you should _really_ avoid
new/delete and instead use C++ smart pointers for allocating/freeing heap
memory. It takes care of the memory management for you and prevents accidental
memory leaks.
In this case, `std::unique_ptr` would work perfectly
https://en.cppreference.com/w/cpp/memory/unique_ptr
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
V-FEXrt wrote:
nit: invert the condition and early exit
```suggestion
if (!Literal.isIntegerLiteral()) {
return true; // TODO: report unsupported number literal specification
}
Result.Kind = TokenKind::int_literal;
APSInt X = APSInt(32, Result.Signed);
if (Literal.GetIntegerValue(X))
return true; // TODO: Report overflow error
X = Result.Signed ? -X : X;
Result.IntLiteral = (uint32_t)X.getZExtValue();
```
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
V-FEXrt wrote:
I'm assuming no, but is unary `+` allowed here?
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
V-FEXrt wrote:
typo?
```suggestion
// Parse the numeric value and do semantic checks on its specification
```
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,121 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+#ifndef KEYWORD
+#define KEYWORD(X) TOK(kw_ ## X)
+#endif
+#ifndef ENUM
+#define ENUM(NAME, LIT) TOK(en_ ## NAME)
+#endif
+
+// Defines the various types of enum
+#ifndef DESCRIPTOR_RANGE_OFFSET_ENUM
+#define DESCRIPTOR_RANGE_OFFSET_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef ROOT_DESCRIPTOR_FLAG_ENUM
+#define ROOT_DESCRIPTOR_FLAG_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+// Note: ON denotes that the flag is unique from the above Root Descriptor
+// Flags. This is required to avoid token kind enum conflicts.
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_OFF
+#define DESCRIPTOR_RANGE_FLAG_ENUM_OFF(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_ON
+#define DESCRIPTOR_RANGE_FLAG_ENUM_ON(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM
+#define DESCRIPTOR_RANGE_FLAG_ENUM(NAME, LIT, ON)
DESCRIPTOR_RANGE_FLAG_ENUM_##ON(NAME, LIT)
+#endif
+#ifndef SHADER_VISIBILITY_ENUM
+#define SHADER_VISIBILITY_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+
+// General Tokens:
+TOK(invalid)
+TOK(int_literal)
+
+// Register Tokens:
+TOK(bReg)
+TOK(tReg)
+TOK(uReg)
+TOK(sReg)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+// RootElement Keywords:
+KEYWORD(DescriptorTable)
+
+// DescriptorTable Keywords:
+KEYWORD(CBV)
+KEYWORD(SRV)
+KEYWORD(UAV)
+KEYWORD(Sampler)
+
+// General Parameter Keywords:
+KEYWORD(space)
+KEYWORD(visibility)
+KEYWORD(flags)
+
+// View Parameter Keywords:
+KEYWORD(numDescriptors)
+KEYWORD(offset)
+
+// Descriptor Range Offset Enum:
+DESCRIPTOR_RANGE_OFFSET_ENUM(DescriptorRangeOffsetAppend,
"DESCRIPTOR_RANGE_OFFSET_APPEND")
inbelic wrote:
I contemplated to have the different enums defined in a seperate `.def` file
with the form of:
`ENUM(NAME, VAL, LIT)` so that we will just define them in a single spot
instead of here and in `llvm/include/llvm/Frontend/HLSL/HLSLRootSignature.h`
(from parser pr).
But I felt that having this `.def` include another `.def` file become more
confusing than helpful, and having them all in a single file would mean having
token logic where it isn't applicable. Would appreciate suggestions here.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
inbelic wrote: I have left the error reporting as a todo just to help with pr size, as it will require adding a good chunk of additional testing https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic ready_for_review https://github.com/llvm/llvm-project/pull/122981 ___ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,152 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+RootSignatureToken Result;
inbelic wrote:
Switch to just using the `SourceLocation` constructor and remove the default
constructor.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,96 @@
+//===--- ParseHLSLRootSignature.h ---*- C++
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the ParseHLSLRootSignature interface.
+//
+//===--===//
+
+#ifndef LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+#define LLVM_CLANG_PARSE_PARSEHLSLROOTSIGNATURE_H
+
+#include "clang/Lex/LiteralSupport.h"
+#include "clang/Lex/Preprocessor.h"
+
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSwitch.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+struct RootSignatureToken {
+ enum Kind {
+#define TOK(X) X,
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ };
+
+ Kind Kind = Kind::invalid;
+
+ // Retain the SouceLocation of the token for diagnostics
+ clang::SourceLocation TokLoc;
+
+ // Retain if the uint32_t bits represent a signed integer
+ bool Signed = false;
+ union {
+uint32_t IntLiteral = 0;
+float FloatLiteral;
inbelic wrote:
This is not currently used, but will be when we consider the `StaticSampler`,
it is included for now just to have the complete token data layout.
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic updated
https://github.com/llvm/llvm-project/pull/122981
>From c1fc823abf07dfb8326b6190672d9881890bbb15 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Tue, 14 Jan 2025 22:22:45 +
Subject: [PATCH 1/2] [HLSL][RootSignature] Implement Lexing of
DescriptorTables
- Define required tokens to parse a Descriptor Table in TokenKinds.def
- Implements a Lexer to handle all of the defined tokens in
ParseHLSLRootSignature
---
.../Parse/HLSLRootSignatureTokenKinds.def | 121 ++
.../clang/Parse/ParseHLSLRootSignature.h | 96 +++
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 151 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 130 +++
7 files changed, 526 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..5e47af8d4b1364
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,121 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+#ifndef KEYWORD
+#define KEYWORD(X) TOK(kw_ ## X)
+#endif
+#ifndef ENUM
+#define ENUM(NAME, LIT) TOK(en_ ## NAME)
+#endif
+
+// Defines the various types of enum
+#ifndef DESCRIPTOR_RANGE_OFFSET_ENUM
+#define DESCRIPTOR_RANGE_OFFSET_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef ROOT_DESCRIPTOR_FLAG_ENUM
+#define ROOT_DESCRIPTOR_FLAG_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+// Note: ON denotes that the flag is unique from the above Root Descriptor
+// Flags. This is required to avoid token kind enum conflicts.
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_OFF
+#define DESCRIPTOR_RANGE_FLAG_ENUM_OFF(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_ON
+#define DESCRIPTOR_RANGE_FLAG_ENUM_ON(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM
+#define DESCRIPTOR_RANGE_FLAG_ENUM(NAME, LIT, ON)
DESCRIPTOR_RANGE_FLAG_ENUM_##ON(NAME, LIT)
+#endif
+#ifndef SHADER_VISIBILITY_ENUM
+#define SHADER_VISIBILITY_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+
+// General Tokens:
+TOK(invalid)
+TOK(int_literal)
+
+// Register Tokens:
+TOK(bReg)
+TOK(tReg)
+TOK(uReg)
+TOK(sReg)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+// RootElement Keywords:
+KEYWORD(DescriptorTable)
+
+// DescriptorTable Keywords:
+KEYWORD(CBV)
+KEYWORD(SRV)
+KEYWORD(UAV)
+KEYWORD(Sampler)
+
+// General Parameter Keywords:
+KEYWORD(space)
+KEYWORD(visibility)
+KEYWORD(flags)
+
+// View Parameter Keywords:
+KEYWORD(numDescriptors)
+KEYWORD(offset)
+
+// Descriptor Range Offset Enum:
+DESCRIPTOR_RANGE_OFFSET_ENUM(DescriptorRangeOffsetAppend,
"DESCRIPTOR_RANGE_OFFSET_APPEND")
+
+// Root Descriptor Flag Enums:
+ROOT_DESCRIPTOR_FLAG_ENUM(DataVolatile, "DATA_VOLATILE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStatic, "DATA_STATIC")
+
+// Descriptor Range Flag Enums:
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsVolatile, "DESCRIPTORS_VOLATILE", ON)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataVolatile, "DATA_VOLATILE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStatic, "DATA_STATIC", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsStaticKeepingBufferBoundsChecks,
"DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS", ON)
+
+// Shader Visibiliy Enums:
+SHADER_VISIBILITY_ENUM(All, "SHADER_VISIBILITY_ALL")
+SHADER_VISIBILITY_ENUM(Vertex, "SHADER_VISIBILITY_VERTEX")
+SHADER_VISIBILITY_ENUM(Hull, "SHADER_VISIBILITY_HULL")
+SHADER_VISIBILITY_E
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
@@ -0,0 +1,151 @@
+#include "clang/Parse/ParseHLSLRootSignature.h"
+
+namespace llvm {
+namespace hlsl {
+namespace root_signature {
+
+// Lexer Definitions
+
+static bool IsPreprocessorNumberChar(char C) {
+ // TODO: extend for float support with or without hexadecimal/exponent
+ return isdigit(C); // integer support
+}
+
+bool RootSignatureLexer::LexNumber(RootSignatureToken &Result) {
+ // NumericLiteralParser does not handle the sign so we will manually apply it
+ Result.Signed = Buffer.front() == '-';
+ if (Result.Signed)
+AdvanceBuffer();
+
+ // Retrieve the possible number
+ StringRef NumSpelling = Buffer.take_while(IsPreprocessorNumberChar);
+
+ // Parse the numeric value and so semantic checks on its specification
+ clang::NumericLiteralParser Literal(NumSpelling, SourceLoc,
+ PP.getSourceManager(), PP.getLangOpts(),
+ PP.getTargetInfo(), PP.getDiagnostics());
+ if (Literal.hadError)
+return true; // Error has already been reported so just return
+
+ // Retrieve the number value to store into the token
+ if (Literal.isIntegerLiteral()) {
+Result.Kind = TokenKind::int_literal;
+
+APSInt X = APSInt(32, Result.Signed);
+if (Literal.GetIntegerValue(X))
+ return true; // TODO: Report overflow error
+
+X = Result.Signed ? -X : X;
+Result.IntLiteral = (uint32_t)X.getZExtValue();
+ } else {
+return true; // TODO: report unsupported number literal specification
+ }
+
+ AdvanceBuffer(NumSpelling.size());
+ return false;
+}
+
+bool RootSignatureLexer::Lex(SmallVector &Tokens) {
+ // Discard any leading whitespace
+ AdvanceBuffer(Buffer.take_while(isspace).size());
+
+ while (!Buffer.empty()) {
+RootSignatureToken Result;
+if (LexToken(Result))
+ return true;
+
+// Successfully Lexed the token so we can store it
+Tokens.push_back(Result);
+
+// Discard any trailing whitespace
+AdvanceBuffer(Buffer.take_while(isspace).size());
+ }
+
+ return false;
+}
+
+bool RootSignatureLexer::LexToken(RootSignatureToken &Result) {
+ // Record where this token is in the text for usage in parser diagnostics
+ Result.TokLoc = SourceLoc;
+
+ char C = Buffer.front();
+
+ // Punctuators
+ switch (C) {
+#define PUNCTUATOR(X, Y)
\
+ case Y: {
\
+Result.Kind = TokenKind::pu_##X;
\
+AdvanceBuffer();
\
+return false;
\
+ }
+#include "clang/Parse/HLSLRootSignatureTokenKinds.def"
+ default:
+break;
+ }
+
+ // Numeric constant
+ if (isdigit(C) || C == '-')
+return LexNumber(Result);
+
+ // All following tokens require at least one additional character
+ if (Buffer.size() <= 1)
+return true; // TODO: Report invalid token error
+
+ // Peek at the next character to deteremine token type
+ char NextC = Buffer[1];
+
+ // Registers: [tsub][0-9+]
+ if ((C == 't' || C == 's' || C == 'u' || C == 'b') && isdigit(NextC)) {
+AdvanceBuffer();
+
+if (LexNumber(Result))
+ return true;
+
+// Lex number could also parse a float so ensure it was an unsigned int
+if (Result.Kind != TokenKind::int_literal || Result.Signed)
+ return true; // Return invalid number literal for register error
+
+// Convert character to the register type.
+// This is done after LexNumber to override the TokenKind
+switch (C) {
+case 'b':
+ Result.Kind = TokenKind::bReg;
+ break;
+case 't':
+ Result.Kind = TokenKind::tReg;
+ break;
+case 'u':
+ Result.Kind = TokenKind::uReg;
+ break;
+case 's':
+ Result.Kind = TokenKind::sReg;
+ break;
+default:
+ llvm_unreachable("Switch for an expected token was not provided");
+ return true;
+}
+return false;
+ }
+
+ // Keywords and Enums:
inbelic wrote:
For the sake of readability all keywords and enums are defined into a single
switch construct. However, it would be possible to match the first character to
reduce the amount of string comparisons if performance is a concern.
For example:
```
if (toupper(C) == 'S') {
Create a StringSwitch with only enums that start with 'S'
}
```
https://github.com/llvm/llvm-project/pull/122981
___
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [HLSL][RootSignature] Implement Lexing of DescriptorTables (PR #122981)
https://github.com/inbelic created
https://github.com/llvm/llvm-project/pull/122981
- Define required tokens to parse a Descriptor Table in TokenKinds.def
- Implements a Lexer to handle all of the defined tokens in
ParseHLSLRootSignature
This is an initial part of #120472
>From c1fc823abf07dfb8326b6190672d9881890bbb15 Mon Sep 17 00:00:00 2001
From: Finn Plummer
Date: Tue, 14 Jan 2025 22:22:45 +
Subject: [PATCH] [HLSL][RootSignature] Implement Lexing of DescriptorTables
- Define required tokens to parse a Descriptor Table in TokenKinds.def
- Implements a Lexer to handle all of the defined tokens in
ParseHLSLRootSignature
---
.../Parse/HLSLRootSignatureTokenKinds.def | 121 ++
.../clang/Parse/ParseHLSLRootSignature.h | 96 +++
clang/lib/Parse/CMakeLists.txt| 1 +
clang/lib/Parse/ParseHLSLRootSignature.cpp| 151 ++
clang/unittests/CMakeLists.txt| 1 +
clang/unittests/Parse/CMakeLists.txt | 26 +++
.../Parse/ParseHLSLRootSignatureTest.cpp | 130 +++
7 files changed, 526 insertions(+)
create mode 100644 clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
create mode 100644 clang/include/clang/Parse/ParseHLSLRootSignature.h
create mode 100644 clang/lib/Parse/ParseHLSLRootSignature.cpp
create mode 100644 clang/unittests/Parse/CMakeLists.txt
create mode 100644 clang/unittests/Parse/ParseHLSLRootSignatureTest.cpp
diff --git a/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
new file mode 100644
index 00..5e47af8d4b1364
--- /dev/null
+++ b/clang/include/clang/Parse/HLSLRootSignatureTokenKinds.def
@@ -0,0 +1,121 @@
+//===--- HLSLRootSignature.def - Tokens and Enum Database ---*- C++
-*-===//
+
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// This file defines the TokenKinds used in the Root Signature DSL. This
+// includes keywords, enums and a small subset of punctuators. Users of this
+// file must optionally #define the TOK, KEYWORD, ENUM or specific ENUM macros
+// to make use of this file.
+//
+//===--===//
+
+#ifndef TOK
+#define TOK(X)
+#endif
+#ifndef PUNCTUATOR
+#define PUNCTUATOR(X,Y) TOK(pu_ ## X)
+#endif
+#ifndef KEYWORD
+#define KEYWORD(X) TOK(kw_ ## X)
+#endif
+#ifndef ENUM
+#define ENUM(NAME, LIT) TOK(en_ ## NAME)
+#endif
+
+// Defines the various types of enum
+#ifndef DESCRIPTOR_RANGE_OFFSET_ENUM
+#define DESCRIPTOR_RANGE_OFFSET_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef ROOT_DESCRIPTOR_FLAG_ENUM
+#define ROOT_DESCRIPTOR_FLAG_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+// Note: ON denotes that the flag is unique from the above Root Descriptor
+// Flags. This is required to avoid token kind enum conflicts.
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_OFF
+#define DESCRIPTOR_RANGE_FLAG_ENUM_OFF(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM_ON
+#define DESCRIPTOR_RANGE_FLAG_ENUM_ON(NAME, LIT) ENUM(NAME, LIT)
+#endif
+#ifndef DESCRIPTOR_RANGE_FLAG_ENUM
+#define DESCRIPTOR_RANGE_FLAG_ENUM(NAME, LIT, ON)
DESCRIPTOR_RANGE_FLAG_ENUM_##ON(NAME, LIT)
+#endif
+#ifndef SHADER_VISIBILITY_ENUM
+#define SHADER_VISIBILITY_ENUM(NAME, LIT) ENUM(NAME, LIT)
+#endif
+
+// General Tokens:
+TOK(invalid)
+TOK(int_literal)
+
+// Register Tokens:
+TOK(bReg)
+TOK(tReg)
+TOK(uReg)
+TOK(sReg)
+
+// Punctuators:
+PUNCTUATOR(l_paren, '(')
+PUNCTUATOR(r_paren, ')')
+PUNCTUATOR(comma, ',')
+PUNCTUATOR(or, '|')
+PUNCTUATOR(equal, '=')
+
+// RootElement Keywords:
+KEYWORD(DescriptorTable)
+
+// DescriptorTable Keywords:
+KEYWORD(CBV)
+KEYWORD(SRV)
+KEYWORD(UAV)
+KEYWORD(Sampler)
+
+// General Parameter Keywords:
+KEYWORD(space)
+KEYWORD(visibility)
+KEYWORD(flags)
+
+// View Parameter Keywords:
+KEYWORD(numDescriptors)
+KEYWORD(offset)
+
+// Descriptor Range Offset Enum:
+DESCRIPTOR_RANGE_OFFSET_ENUM(DescriptorRangeOffsetAppend,
"DESCRIPTOR_RANGE_OFFSET_APPEND")
+
+// Root Descriptor Flag Enums:
+ROOT_DESCRIPTOR_FLAG_ENUM(DataVolatile, "DATA_VOLATILE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE")
+ROOT_DESCRIPTOR_FLAG_ENUM(DataStatic, "DATA_STATIC")
+
+// Descriptor Range Flag Enums:
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsVolatile, "DESCRIPTORS_VOLATILE", ON)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataVolatile, "DATA_VOLATILE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStaticWhileSetAtExecute,
"DATA_STATIC_WHILE_SET_AT_EXECUTE", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DataStatic, "DATA_STATIC", OFF)
+DESCRIPTOR_RANGE_FLAG_ENUM(DescriptorsStaticKeepingBufferBoundsChecks,
"DESCRIPTORS_STATIC_KEEPING_BUFFER_BOUNDS_CHECKS", ON)
+
+// Shader Visibiliy Enums:
+SH
