llvmorg-github-actions[bot] wrote:

<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Markus Böck (zero9178)

<details>
<summary>Changes</summary>

Prior to this PR, ODS had one mechanism for sharding which equally split a 
`&lt;Dialect&gt;Ops.cpp.inc` file into `n` many TUs, as requested on the 
command line (using preprocessor defines for selective enablement). The 
motivation for doing so is that this can speed up compile time by splitting 
class definition compilation to multiple translation units.

However, this approach has one large disadvantage: Since only the 
`&lt;Dialect&gt;Ops.cpp.inc` is split, the correpsonding 
`&lt;Dialect&gt;Ops.h.inc` is still included in each translation unit. For a 
large dialect like NVVM or LLVM this leads to **much higher** concurrent memory 
usage while compiling as every compilation process contains an in-memory copy 
of the header plus can lead to slower compilation time overall if too many 
shards are being used as overall work goes up (`n` header parses instead of 1).

This PR tries to address these issues by introducing a new kind of sharding 
called "natural sharding". The idea is that we split 
`&lt;Dialect&gt;Ops.cpp.inc` and `&lt;Dialect&gt;Ops.h.inc` along so called 
partitions. A new header can be created for every partition and every 
translation unit only contains the code from its partition. This means a TU no 
longer needs to include the entire dialect header but only the header from its 
partition. One can think of these as effectively being sub-dialects from the 
perspective of the required boilerplate.

We call it "Natural Sharding", as the partitions are defined by the dialect 
authors and are meant to be categories that naturally categorizes a dialects 
operations (e.g., Memory ops, control flow ops, arithmetic ops, intrinsic ops, 
etc. in the LLVM Dialect). The new `-op-partition` then filters output 
generation to just a given partition.

This PR introduces the mechanism and for testing, already exercises it on the 
test dialect. Users of `TestOps.h` were further changed to just include a given 
partition if deemed useful. Overall this yielded the following build gains vs 
the existing sharding, measured on a 16 thread machine (Release + assertions, 
16 parallel jobs, median of 3 runs). "Users of `MLIRTestDialect`" are other TUs 
(unittests mainly) that include `MLIRTestDialect` headers. Total memory is the 
sum of the max RSS of every TU:

| TUs                        | Compiler | Wall time (s)       | CPU time (s)    
   | Total memory used during compilation (GiB) |
|----------------------------|----------|---------------------|--------------------|--------------------------------------------|
| `MLIRTestDialect`          | clang    | 49.9 -&gt; 35.2 (-29%) | 710 -&gt; 
365 (-49%)  | 23.3 -&gt; 11.1 (-52%)                        |
| `MLIRTestDialect`          | clang -g | 57.8 -&gt; 43.1 (-25%) | 791 -&gt; 
427 (-46%)  | 25.0 -&gt; 12.5 (-50%)                        |
| `MLIRTestDialect`          | gcc      | 58.6 -&gt; 43.8 (-25%) | 740 -&gt; 
395 (-47%)  | 41.9 -&gt; 19.4 (-54%)                        |
| `MLIRTestDialect`          | gcc -g   | 82.5 -&gt; 61.5 (-25%) | 1059 -&gt; 
570 (-46%) | 52.9 -&gt; 26.7 (-50%)                        |
| Users of `MLIRTestDialect` | clang    | 45.4 -&gt; 35.1 (-23%) | 626 -&gt; 
472 (-25%)  | 27.6 -&gt; 21.5 (-22%)                        |
| Users of `MLIRTestDialect` | clang -g | 46.6 -&gt; 36.1 (-23%) | 644 -&gt; 
493 (-23%)  | 28.4 -&gt; 22.3 (-21%)                        |
| Users of `MLIRTestDialect` | gcc      | 43.6 -&gt; 33.9 (-22%) | 604 -&gt; 
449 (-26%)  | 44.5 -&gt; 32.8 (-26%)                        |
| Users of `MLIRTestDialect` | gcc -g   | 60.9 -&gt; 46.1 (-24%) | 859 -&gt; 
616 (-28%)  | 57.1 -&gt; 42.4 (-26%)                        |

Future plan is to introduce this for types and attributes as well

---

Patch is 145.47 KiB, truncated to 20.00 KiB below, full version: 
https://github.com/llvm/llvm-project/pull/239254.diff


66 Files Affected:

- (modified) mlir/docs/DefiningDialects/Operations.md (+104) 
- (modified) mlir/include/mlir/IR/DialectBase.td (+33) 
- (modified) mlir/include/mlir/IR/OpBase.td (+24) 
- (modified) mlir/include/mlir/TableGen/Dialect.h (+5) 
- (modified) mlir/include/mlir/TableGen/Operator.h (+3) 
- (modified) mlir/lib/TableGen/Dialect.cpp (+5) 
- (modified) mlir/lib/TableGen/Operator.cpp (+5) 
- (modified) 
mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp (+1-1) 
- (modified) 
mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp (+1-2) 
- (modified) mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp (+1-2) 
- (modified) mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp (+1-2) 
- (modified) mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp 
(+1-2) 
- (modified) mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp (+1-1) 
- (modified) mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp (+1-2) 
- (modified) mlir/test/lib/Dialect/Test/CMakeLists.txt (+14-7) 
- (modified) mlir/test/lib/Dialect/Test/TestDialect.cpp (+1-2) 
- (modified) mlir/test/lib/Dialect/Test/TestDialect.td (+19-1) 
- (modified) mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp (+1-2) 
- (modified) mlir/test/lib/Dialect/Test/TestOps.h (+12-238) 
- (modified) mlir/test/lib/Dialect/Test/TestOps.td (+224-174) 
- (renamed) mlir/test/lib/Dialect/Test/TestOpsAttributes.cpp (+4-5) 
- (added) mlir/test/lib/Dialect/Test/TestOpsAttributes.h (+17) 
- (added) mlir/test/lib/Dialect/Test/TestOpsConversion.cpp (+15) 
- (added) mlir/test/lib/Dialect/Test/TestOpsConversion.h (+17) 
- (added) mlir/test/lib/Dialect/Test/TestOpsCore.cpp (+19) 
- (added) mlir/test/lib/Dialect/Test/TestOpsCore.h (+18) 
- (added) mlir/test/lib/Dialect/Test/TestOpsInterfaces.cpp (+16) 
- (added) mlir/test/lib/Dialect/Test/TestOpsInterfaces.h (+90) 
- (added) mlir/test/lib/Dialect/Test/TestOpsPatterns.cpp (+15) 
- (added) mlir/test/lib/Dialect/Test/TestOpsPatterns.h (+17) 
- (added) mlir/test/lib/Dialect/Test/TestOpsProperties.cpp (+16) 
- (added) mlir/test/lib/Dialect/Test/TestOpsProperties.h (+154) 
- (modified) mlir/test/lib/Dialect/Test/TestOpsSyntax.cpp (+2-8) 
- (modified) mlir/test/lib/Dialect/Test/TestOpsSyntax.td (+5-1) 
- (added) mlir/test/lib/Dialect/Test/TestOpsTraits.cpp (+15) 
- (added) mlir/test/lib/Dialect/Test/TestOpsTraits.h (+17) 
- (modified) mlir/test/lib/Dialect/Test/TestToLLVMIRTranslation.cpp (+1-2) 
- (modified) mlir/test/lib/Dialect/Test/TestTraits.cpp (+1-1) 
- (modified) mlir/test/lib/IR/TestBytecodeRoundtrip.cpp (+1-2) 
- (modified) mlir/test/lib/IR/TestClone.cpp (+1-1) 
- (modified) mlir/test/lib/IR/TestLazyLoading.cpp (-1) 
- (modified) mlir/test/lib/IR/TestRegions.cpp (-1) 
- (modified) mlir/test/lib/IR/TestSideEffects.cpp (+1-1) 
- (modified) mlir/test/lib/IR/TestSymbolUses.cpp (+1-1) 
- (modified) mlir/test/lib/IR/TestTypes.cpp (-1) 
- (modified) mlir/test/lib/IR/TestVisitorsGeneric.cpp (+1-1) 
- (modified) mlir/test/lib/Pass/TestPassManager.cpp (+1-2) 
- (modified) mlir/test/lib/Tools/PDLL/TestPDLL.cpp (+1-2) 
- (modified) mlir/test/lib/Transforms/TestInlining.cpp (+1-2) 
- (modified) mlir/test/lib/Transforms/TestInliningCallback.cpp (+2-2) 
- (modified) mlir/test/lib/Transforms/TestMakeIsolatedFromAbove.cpp (+1-2) 
- (added) mlir/test/mlir-tblgen/op-partitions-error.td (+74) 
- (added) mlir/test/mlir-tblgen/op-partitions.td (+110) 
- (modified) mlir/tools/mlir-tblgen/DialectGen.cpp (+25) 
- (modified) mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp (+67-3) 
- (modified) mlir/tools/mlir-tblgen/OpGenHelpers.cpp (+79) 
- (modified) mlir/tools/mlir-tblgen/OpGenHelpers.h (+24-2) 
- (modified) mlir/unittests/IR/AdaptorTest.cpp (+1-2) 
- (modified) mlir/unittests/IR/IRMapping.cpp (+1-2) 
- (modified) mlir/unittests/IR/InterfaceAttachmentTest.cpp (+1-2) 
- (modified) mlir/unittests/IR/InterfaceTest.cpp (+1-2) 
- (modified) mlir/unittests/IR/OperationSupportTest.cpp (+1-2) 
- (modified) mlir/unittests/IR/PatternMatchTest.cpp (+1-2) 
- (modified) mlir/unittests/IR/ValueTest.cpp (-2) 
- (modified) mlir/unittests/Interfaces/MemorySlotUtilsTest.cpp (-1) 
- (modified) mlir/unittests/TableGen/OpBuildGen.cpp (+1-2) 


``````````diff
diff --git a/mlir/docs/DefiningDialects/Operations.md 
b/mlir/docs/DefiningDialects/Operations.md
index f991f5308c5ae4..7c0228772d2488 100644
--- a/mlir/docs/DefiningDialects/Operations.md
+++ b/mlir/docs/DefiningDialects/Operations.md
@@ -1370,6 +1370,110 @@ cc_library(
 )
 ```
 
+#### Natural Sharding with Op Partitions
+
+Sharding by index splits the ops arbitrarily: which shard an op ends up in
+depends on the number of shards and on the names of all other ops. As an
+alternative, a dialect can shard its ops "naturally" by grouping them into
+named *op partitions*, such as arithmetic, memory or control flow ops. Each
+partition is generated into its own declaration and definition files, almost as
+if it were a dialect of its own, but all ops are still registered in one
+dialect.
+
+Partitions are `Partition` records with a name, which must be a C++
+identifier. The dialect lists its partitions, and every op sets its `partition`
+to one of these records (a different record with the same name is rejected),
+typically with a `let` around a group of ops:
+
+```tablegen
+def MyDialect_ArithPartition : Partition<"Arith">;
+def MyDialect_MemoryPartition : Partition<"Memory">;
+
+def MyDialect : Dialect {
+  let name = "my_dialect";
+  let opPartitions = [MyDialect_ArithPartition, MyDialect_MemoryPartition];
+}
+
+let partition = MyDialect_ArithPartition in {
+def AddOp : MyDialect_Op<"add"> { ... }
+def MulOp : MyDialect_Op<"mul"> { ... }
+}
+
+def LoadOp : MyDialect_Op<"load"> {
+  let partition = MyDialect_MemoryPartition;
+  ...
+}
+```
+
+A dialect can also name one of its partitions as `defaultOpPartition`. Ops that
+do not set `partition` are then in this partition. This allows partitioning a
+dialect incrementally: initially all ops are in the default partition, and
+groups of ops can move to their own partitions one at a time:
+
+```tablegen
+def MyDialect : Dialect {
+  let name = "my_dialect";
+  let opPartitions = [MyDialect_CorePartition, MyDialect_MemoryPartition];
+  let defaultOpPartition = MyDialect_CorePartition;
+}
+
+// In partition `Core`.
+def AddOp : MyDialect_Op<"add"> { ... }
+```
+
+Passing `-op-partition=<partition>` with a partition name to `-gen-op-decls` 
and
+`-gen-op-defs` generates only the ops of that partition.
+
+Without `-op-partition`, `-gen-op-defs` generates all ops as before and
+additionally defines the registration hooks of all partitions in
+`GET_OP_CLASSES`. The hooks are private members of the dialect class, and the
+dialect registers the ops of all partitions by calling `registerOperations()`
+in its `initialize()`. A dialect can therefore declare its partitions and
+register its ops with `registerOperations()` first, and split its generated
+files by partition later.
+
+`mlir-tblgen` reports an error if a requested partition does not exist. A
+partition for which no definitions are compiled results in a link error for its
+hook.
+
+In CMake, generate and include the files of each partition:
+
+```cmake
+set(LLVM_TARGET_DEFINITIONS MyDialectOps.td)
+foreach(partition Arith Memory)
+  mlir_tablegen(MyDialect${partition}Ops.h.inc -gen-op-decls 
-op-partition=${partition})
+  mlir_tablegen(MyDialect${partition}Ops.cpp.inc -gen-op-defs 
-op-partition=${partition})
+endforeach()
+```
+
+Since the generated declarations of a partition forward declare the ops of all
+other partitions, each partition can get its own header, so that users only
+include the declarations of the ops they use:
+
+```c++
+// MyDialectArithOps.h
+#include "MyDialectBase.h" // The dialect, its attributes and types.
+
+#define GET_OP_CLASSES
+#include "MyDialectArithOps.h.inc"
+
+// MyDialect.h, for users that want all ops.
+#include "MyDialectArithOps.h"
+#include "MyDialectMemoryOps.h"
+
+// MyDialectArithOps.cpp, a natural home for the C++ implementations of the
+// arithmetic ops as well.
+#include "MyDialectArithOps.h"
+#include "OpUtils.h"
+
+#define GET_OP_CLASSES
+#include "MyDialectArithOps.cpp.inc"
+```
+
+As with index sharding, utilities shared by the generated op definitions must
+be declared in a header shared by the compilation units. A dialect with op
+partitions cannot use `-op-shard-count`.
+
 ## Constraints
 
 Constraint is a core concept in table-driven operation definition: operation
diff --git a/mlir/include/mlir/IR/DialectBase.td 
b/mlir/include/mlir/IR/DialectBase.td
index da669c6ada639f..92b6601b7e2877 100644
--- a/mlir/include/mlir/IR/DialectBase.td
+++ b/mlir/include/mlir/IR/DialectBase.td
@@ -19,6 +19,17 @@ include "mlir/IR/Utils.td"
 // Dialect definitions
 
//===----------------------------------------------------------------------===//
 
+// A partition of the ops of a dialect, for "natural sharding" of the generated
+// op declarations and definitions. See `Dialect::opPartitions`.
+class Partition<string name> {
+  // The name of the partition, a C++ identifier. `-gen-op-decls` and
+  // `-gen-op-defs` select a partition by this name with `-op-partition=...`.
+  string partitionName = name;
+
+  assert !match(partitionName, "^[A-Za-z_][A-Za-z0-9_]*$"),
+         "op partition name '" # partitionName # "' is not a C++ identifier";
+}
+
 class Dialect {
   // The name of the dialect.
   string name = ?;
@@ -99,6 +110,28 @@ class Dialect {
 
   // If this dialect can be extended at runtime with new operations or types.
   bit isExtensible = 0;
+
+  // The op partitions of this dialect, enabling "natural sharding" of its op
+  // declarations and definitions (see `Op::partition`). When non-empty, every
+  // op of the dialect must be in one of these partitions.
+  list<Partition> opPartitions = [];
+
+  // Optional. The partition of the ops of this dialect that do not set their
+  // `partition`. Must be one of `opPartitions`. This allows partitioning the
+  // ops of a dialect incrementally: all ops start in the default partition and
+  // groups of ops can move to other partitions one at a time.
+  Partition defaultOpPartition = ?;
+
+  assert !if(!initialized(defaultOpPartition),
+             !not(!empty(!filter(p, opPartitions,
+                                 !eq(p, defaultOpPartition)))),
+             1),
+         "default op partition '" #
+         !if(!initialized(defaultOpPartition),
+             defaultOpPartition.partitionName # "' (record '" #
+             !cast<string>(defaultOpPartition), "") #
+         "') of dialect '" # name #
+         "' is not one of its `opPartitions`";
 }
 
 #endif // DIALECTBASE_TD
diff --git a/mlir/include/mlir/IR/OpBase.td b/mlir/include/mlir/IR/OpBase.td
index a130d5847a1c72..3357f83e922bba 100644
--- a/mlir/include/mlir/IR/OpBase.td
+++ b/mlir/include/mlir/IR/OpBase.td
@@ -351,6 +351,30 @@ class Op<Dialect dialect, string mnemonic, list<Trait> 
props = []> {
   // Optional. The group of ops this op is part of.
   OpDocGroup opDocGroup = ?;
 
+  // Optional. The partition of this op. A dialect that lists its partitions in
+  // `opPartitions` is "naturally sharded": every op of the dialect must be in
+  // one of the listed partitions, `-gen-op-decls`/`-gen-op-defs` can generate
+  // the ops of one partition at a time with `-op-partition=...` into separate
+  // files.
+  // Defaults to the dialect's `defaultOpPartition`. The partition must be one
+  // of the `opPartitions` records of the dialect.
+  Partition partition = dialect.defaultOpPartition;
+
+  assert !if(!initialized(partition),
+             !not(!empty(!filter(p, opDialect.opPartitions,
+                                 !eq(p, partition)))),
+             1),
+         "op '" # NAME # "' is in partition '" #
+         !if(!initialized(partition),
+             partition.partitionName # "' (record '" #
+             !cast<string>(partition), "") #
+         "'), which is not one of the `opPartitions` of dialect '" #
+         opDialect.name # "'";
+
+  assert !or(!initialized(partition), !empty(opDialect.opPartitions)),
+         "op '" # NAME # "' is in no partition, but dialect '" #
+         opDialect.name # "' has `opPartitions` and no `defaultOpPartition`";
+
   // Dag containing the arguments of the op. Default to 0 arguments.
   dag arguments = (ins);
 
diff --git a/mlir/include/mlir/TableGen/Dialect.h 
b/mlir/include/mlir/TableGen/Dialect.h
index 68230196b17cc0..7913360c294e48 100644
--- a/mlir/include/mlir/TableGen/Dialect.h
+++ b/mlir/include/mlir/TableGen/Dialect.h
@@ -94,6 +94,10 @@ class Dialect {
 
   const llvm::DagInit *getDiscardableAttributes() const;
 
+  /// Returns the names of the op partitions of this dialect, see
+  /// `Op::partition`. A dialect with op partitions is "naturally sharded".
+  ArrayRef<StringRef> getOpPartitions() const;
+
   const llvm::Record *getDef() const { return def; }
 
   // Returns whether two dialects are equal by checking the equality of the
@@ -112,6 +116,7 @@ class Dialect {
 private:
   const llvm::Record *def;
   std::vector<StringRef> dependentDialects;
+  std::vector<StringRef> opPartitions;
 };
 } // namespace tblgen
 } // namespace mlir
diff --git a/mlir/include/mlir/TableGen/Operator.h 
b/mlir/include/mlir/TableGen/Operator.h
index bb36c077708e92..cd321dcd6ca6f3 100644
--- a/mlir/include/mlir/TableGen/Operator.h
+++ b/mlir/include/mlir/TableGen/Operator.h
@@ -330,6 +330,9 @@ class Operator {
   /// Returns the dialect of the op.
   const Dialect &getDialect() const { return dialect; }
 
+  /// Returns the name of the op's partition, or an empty string if it has 
none.
+  StringRef getPartitionName() const;
+
   /// Prints the contents in this operator to the given `os`. This is used for
   /// debugging purposes.
   void print(llvm::raw_ostream &os) const;
diff --git a/mlir/lib/TableGen/Dialect.cpp b/mlir/lib/TableGen/Dialect.cpp
index 84530bfa4f6fd2..a9f9b49b663ead 100644
--- a/mlir/lib/TableGen/Dialect.cpp
+++ b/mlir/lib/TableGen/Dialect.cpp
@@ -21,6 +21,9 @@ Dialect::Dialect(const llvm::Record *def) : def(def) {
     return;
   for (StringRef dialect : def->getValueAsListOfStrings("dependentDialects"))
     dependentDialects.push_back(dialect);
+  for (const llvm::Record *partition :
+       def->getValueAsListOfDefs("opPartitions"))
+    opPartitions.push_back(partition->getValueAsString("partitionName"));
 }
 
 StringRef Dialect::getName() const { return def->getValueAsString("name"); }
@@ -110,6 +113,8 @@ const llvm::DagInit *Dialect::getDiscardableAttributes() 
const {
   return def->getValueAsDag("discardableAttrs");
 }
 
+ArrayRef<StringRef> Dialect::getOpPartitions() const { return opPartitions; }
+
 bool Dialect::operator==(const Dialect &other) const {
   return def == other.def;
 }
diff --git a/mlir/lib/TableGen/Operator.cpp b/mlir/lib/TableGen/Operator.cpp
index f01820bcf411c3..eb9bdb1b2b5e70 100644
--- a/mlir/lib/TableGen/Operator.cpp
+++ b/mlir/lib/TableGen/Operator.cpp
@@ -151,6 +151,11 @@ void Operator::assertInvariants() const {
 
 StringRef Operator::getDialectName() const { return dialect.getName(); }
 
+StringRef Operator::getPartitionName() const {
+  const llvm::Record *partition = def.getValueAsOptionalDef("partition");
+  return partition ? partition->getValueAsString("partitionName") : "";
+}
+
 StringRef Operator::getCppClassName() const { return cppClassName; }
 
 std::string Operator::getQualCppClassName() const {
diff --git 
a/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp 
b/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp
index f969869b924824..5bb7fbaaac6815 100644
--- a/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp
+++ b/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp
@@ -11,7 +11,7 @@
 
//===----------------------------------------------------------------------===//
 
 #include "TestDenseDataFlowAnalysis.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
 #include "mlir/Analysis/DataFlow/DenseAnalysis.h"
 #include "mlir/Analysis/DataFlow/Utils.h"
 #include "mlir/Analysis/DataFlowFramework.h"
diff --git 
a/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp 
b/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp
index c8b67b4ba61c00..725daef9d08e3e 100644
--- a/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp
+++ b/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp
@@ -11,8 +11,7 @@
 
//===----------------------------------------------------------------------===//
 
 #include "TestDenseDataFlowAnalysis.h"
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
 #include "mlir/Analysis/DataFlow/DenseAnalysis.h"
 #include "mlir/Analysis/DataFlow/Utils.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
diff --git a/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp 
b/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp
index 3b29867cc32631..89e02814950fd8 100644
--- a/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp
+++ b/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp
@@ -6,8 +6,7 @@
 //
 
//===----------------------------------------------------------------------===//
 
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsConversion.h"
 #include "TestTypes.h"
 #include "mlir/Conversion/FuncToLLVM/ConvertFuncToLLVM.h"
 #include "mlir/Conversion/LLVMCommon/Pattern.h"
diff --git a/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp 
b/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp
index f11c14bc09fb66..efd8ee7a8ebf3f 100644
--- a/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp
+++ b/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp
@@ -6,8 +6,7 @@
 //
 
//===----------------------------------------------------------------------===//
 
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
 #include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Affine/IR/ValueBoundsOpInterfaceImpl.h"
 #include "mlir/Dialect/Affine/Transforms/Transforms.h"
diff --git a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp 
b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
index d9f50ba0ee1ff4..634469278fa847 100644
--- a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
+++ b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
@@ -15,8 +15,7 @@
 #include "mlir/Pass/Pass.h"
 
 #include "TestAttributes.h" // TestTensorEncodingAttr, TestMemRefLayoutAttr
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
 
 #include "llvm/ADT/TypeSwitch.h"
 
diff --git a/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp 
b/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp
index 9fb89d3b4528b0..d0ca7fa5c2506f 100644
--- a/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp
+++ b/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp
@@ -6,7 +6,7 @@
 //
 
//===----------------------------------------------------------------------===//
 
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
 #include "mlir/Analysis/DataLayoutAnalysis.h"
 #include "mlir/Dialect/DLTI/DLTI.h"
 #include "mlir/IR/BuiltinAttributes.h"
diff --git a/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp 
b/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp
index c67bcd94ea0bd6..0f789a8c822e21 100644
--- a/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp
+++ b/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp
@@ -6,8 +6,7 @@
 //
 
//===----------------------------------------------------------------------===//
 
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Func/Transforms/FuncConversions.h"
 #include "mlir/IR/Builders.h"
diff --git a/mlir/test/lib/Dialect/Test/CMakeLists.txt 
b/mlir/test/lib/Dialect/Test/CMakeLists.txt
index ca0024b8434e16..d2dab9e88304fd 100644
--- a/mlir/test/lib/Dialect/Test/CMakeLists.txt
+++ b/mlir/test/lib/Dialect/Test/CMakeLists.txt
@@ -34,15 +34,18 @@ set(LLVM_TARGET_DEFINITIONS TestOps.td)
 mlir_tablegen(TestOpsDialect.h.inc -gen-dialect-decls -dialect=test)
 mlir_tablegen(TestOpsDialect.cpp.inc -gen-dialect-defs -dialect=test)
 mlir_tablegen(TestPatterns.inc -gen-rewriters)
+# The op partitions of TestOps.td, each with its own header and source file.
+foreach(partition Core Attributes Traits Patterns Conversion Interfaces 
Properties)
+  mlir_tablegen(TestOps${partition}.h.inc -gen-op-decls 
-op-partition=${partition})
+  mlir_tablegen(TestOps${partition}.cpp.inc -gen-op-defs 
-op-partition=${partition})
+endforeach()
 add_public_tablegen_target(MLIRTestOpsIncGen)
 
 set(LLVM_TARGET_DEFINITIONS TestOpsSyntax.td)
-mlir_tablegen(TestOpsSyntax.h.inc -gen-op-decls)
-mlir_tablegen(TestOpsSyntax.cpp.inc -gen-op-defs)
+mlir_tablegen(TestOpsSyntax.h.inc -gen-op-decls -op-partition=Syntax)
+mlir_tablegen(TestOpsSyntax.cpp.inc -gen-op-defs -op-partition=Syntax)
 add_public_tablegen_target(MLIRTestOpsSyntaxIncGen)
 
-add_sharded_ops(TestOps 20)
-
 # Exclude tests from libMLIR.so
 add_mlir_library(MLIRTestDialect
   TestAttributes.cpp
@@ -50,13 +53,18 @@ add_mlir_library(MLIRTestDialect
   TestFormatUtils.cpp
   TestInterfaces.cpp
   TestOpDefs.cpp
-  TestOps.cpp
+  TestOpsAttributes.cpp
+  TestOpsConversion.cpp
+  TestOpsCore.cpp
+  TestOpsInterfaces.cpp
+  TestOpsPatterns.cpp
+  TestOpsProperties.cpp
+  TestOpsTraits.cpp
   TestPatterns.cpp
   TestTraits.cpp
   TestTypes.cpp
   TestOpsSyntax.cpp
   TestDialectInterfaces.cpp
-  ${SHARDED_SRCS}
 
   EXCLUDE_FROM_LIBMLIR
 
@@ -67,7 +75,6 @@ add_mlir_library(MLIRTestDialect
   MLIRTestTypeDefIncGen
   MLIRTestOpsIncGen
   MLIRTestOpsSyntaxIncGen
-  MLIRTestOpsShardGen
   MLIRConvertToEmitCPatternInterfaceIncGen
   )
 mlir_target_link_libraries(MLIRTestDialect PUBLIC
diff --git a/mlir/test/lib/Dialect/Test/TestDialect.cpp 
b/mlir/test/lib/Dialect/Test/TestDialect.cpp
index 57bba2e5749a86..657f25771a45ee 100644
--- a/mlir/test/lib/Dialect/Test/TestDialect.cpp
+++ b/mlir/test/lib/Dialect/Test/TestDialect.cpp
@@ -346,9 +346,8 @@ struct TestOpEffectInterfaceFallback
 void TestDialect::initialize() {
   registerAttributes();
   registerTypes();
-  registerOpsSyntax();
   addOperations<ManualCppOpWithFold>();
-  registerTestDialectOperations(this);
+  registerOperations();
   registerDynamicOp(getDynamicGenericOp(this));
   registerDynamicOp(getDynamicTerminatorOp(this));
   registerDynamicOp(getDynamicNoTerminatorOp(this));
diff --git a/mlir/test/lib/Dialect/Test/TestDialect.td 
b/mlir/test/lib/Dialect/Test/TestDialect.td
index 37a263f1d10b8b..debb239addc056 100644
--- a/mlir/test/lib/Dialect/Test/TestDialect.td
+++ b/mlir/test/lib/Dialect/Test/TestDialect.td
@@ -11,6 +11,19 @@
 
 include "mlir/IR/OpBase.td"
 
+// Op partitions, each generated and compiled separately. Ops from TestOps.td
+// are in `Core` unless a `let` block or their base class picks another
+// partition. All ops from TestOpsSyntax.td are in `Syntax`, which is generated
+// from that file.
+def Test_CorePartition : Partition<"Core">;
+def Test_AttributesPartition : Partition<"Attributes">;
+def Test_TraitsPartition : Partition<"Traits">;
+def Test_PatternsPartition : Partition<"Patterns">;
+def Test_ConversionPartition : Partition<"Conversion">;
+def Test_InterfacesPartition : Partition<"Interfaces">;
+def Test_PropertiesPartition : Partition<"Properties">;
+def Test_SyntaxPartition : Partition<"Syntax">;
+
 def Test_Dialect : Dialect {
   let name = "test";
   let cppNamespace = "::test";
@@ -24,6 +37,12 @@ def Test_Dialect : Dialect {
   let useDefaultTypePrinterParser = 0;
   let useDefaultAttributePrinterParser = 1;
   let isExtensible = 1;
+  let opPartitions = [
+    Test_CorePartition, Test_AttributesPartition, Test_TraitsPartition,
+    Test_PatternsPartition, Test_ConversionPartition, Test_InterfacesPartition,
+    Test_PropertiesPartition, Test_SyntaxPartition
+  ];
+  let defaultOpPartition = Test_CorePartition;
   let dependentDialects = [
     "::mlir::DLTIDialect",
     "::mlir::bufferization::BufferizationDialect"
@@ -37,7 +56,6 @@ def Test_Dialect : Dialect {
     void registerAttributes();
     void registerInterfaces();
     void registerTypes();
-    void registerOpsSyntax();
 
     // Provides a custom printing/parsing for some operations.
     ::std::optional<ParseOpHook>
diff --git a/mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp 
b/mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp
index b98f6ce32cdceb..c1f8629c65df42 100644
--- a/mlir/test/lib/Dialect/Test/TestFromLLVMI...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/239254
_______________________________________________
llvm-branch-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

Reply via email to