meheff created this revision.
meheff added reviewers: tyler.nowicki, hfinkel.
meheff added subscribers: cfe-commits, llvm-commits.
This change adds the new unroll metadata "llvm.loop.unroll.enable" which directs
the optimizer to unroll a loop fully if the trip count is known at compile
time, and
unroll partially if the trip count is not known at compile time. This differs
from
"llvm.loop.unroll.full" which explicitly does not unroll a loop if the trip
count is not
known at compile time
With this change "#pragma unroll" generates "llvm.loop.unroll.enable" rather
than
"llvm.loop.unroll.full" metadata. This changes the semantics of "#pragma
unroll" slightly
to mean "unroll aggressively (fully or partially)" rather than "unroll fully or
not at all".
The motivating example for this change was some internal code with a loop marked
with "#pragma unroll" which only sometimes had a compile-time trip count
depending
on template magic. When the trip count was a compile-time constant, everything
works
as expected and the loop is fully unrolled. However, when the trip count was
not a
compile-time constant the "#pragma unroll" explicitly disabled unrolling of the
loop(!).
Removing "#pragma unroll" caused the loop to be unrolled partially which was
desirable
from a performance perspective.
Optimizer patch will follow shortly.
http://reviews.llvm.org/D11738
Files:
docs/LanguageExtensions.rst
include/clang/Basic/Attr.td
include/clang/Basic/AttrDocs.td
include/clang/Basic/DiagnosticParseKinds.td
lib/CodeGen/CGLoopInfo.cpp
lib/CodeGen/CGLoopInfo.h
lib/Parse/ParsePragma.cpp
lib/Sema/SemaStmtAttr.cpp
test/CodeGenCXX/pragma-unroll.cpp
test/Parser/pragma-loop-safety.cpp
test/Parser/pragma-loop.cpp
test/Parser/pragma-unroll.cpp
Index: test/Parser/pragma-unroll.cpp
===================================================================
--- test/Parser/pragma-unroll.cpp
+++ test/Parser/pragma-unroll.cpp
@@ -67,6 +67,12 @@
List[i] = i;
}
+/* expected-error {{incompatible directives 'unroll(enable)' and '#pragma unroll(4)'}} */ #pragma unroll(4)
+#pragma clang loop unroll(enable)
+ while (i-11 < Length) {
+ List[i] = i;
+ }
+
/* expected-error {{incompatible directives '#pragma unroll' and '#pragma unroll(4)'}} */ #pragma unroll(4)
#pragma unroll
while (i-11 < Length) {
Index: test/Parser/pragma-loop.cpp
===================================================================
--- test/Parser/pragma-loop.cpp
+++ test/Parser/pragma-loop.cpp
@@ -132,7 +132,7 @@
/* expected-error {{missing argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop vectorize()
/* expected-error {{missing argument; expected an integer value}} */ #pragma clang loop interleave_count()
-/* expected-error {{missing argument; expected 'full' or 'disable'}} */ #pragma clang loop unroll()
+/* expected-error {{missing argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll()
/* expected-error {{missing option; expected vectorize, vectorize_width, interleave, interleave_count, unroll, or unroll_count}} */ #pragma clang loop
/* expected-error {{invalid option 'badkeyword'}} */ #pragma clang loop badkeyword
@@ -186,16 +186,16 @@
/* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop vectorize(badidentifier)
/* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop interleave(badidentifier)
-/* expected-error {{invalid argument; expected 'full' or 'disable'}} */ #pragma clang loop unroll(badidentifier)
+/* expected-error {{invalid argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll(badidentifier)
while (i-7 < Length) {
List[i] = i;
}
// PR20069 - Loop pragma arguments that are not identifiers or numeric
// constants crash FE.
/* expected-error {{expected ')'}} */ #pragma clang loop vectorize(()
/* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop interleave(*)
-/* expected-error {{invalid argument; expected 'full' or 'disable'}} */ #pragma clang loop unroll(=)
+/* expected-error {{invalid argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll(=)
/* expected-error {{type name requires a specifier or qualifier}} expected-error {{expected expression}} */ #pragma clang loop vectorize_width(^)
/* expected-error {{expected expression}} expected-error {{expected expression}} */ #pragma clang loop interleave_count(/)
/* expected-error {{expected expression}} expected-error {{expected expression}} */ #pragma clang loop unroll_count(==)
Index: test/Parser/pragma-loop-safety.cpp
===================================================================
--- test/Parser/pragma-loop-safety.cpp
+++ test/Parser/pragma-loop-safety.cpp
@@ -15,11 +15,11 @@
/* expected-error {{expected ')'}} */ #pragma clang loop vectorize(assume_safety
/* expected-error {{expected ')'}} */ #pragma clang loop interleave(assume_safety
-/* expected-error {{invalid argument; expected 'full' or 'disable'}} */ #pragma clang loop unroll(assume_safety)
+/* expected-error {{invalid argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll(assume_safety)
/* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop vectorize(badidentifier)
/* expected-error {{invalid argument; expected 'enable', 'assume_safety' or 'disable'}} */ #pragma clang loop interleave(badidentifier)
-/* expected-error {{invalid argument; expected 'full' or 'disable'}} */ #pragma clang loop unroll(badidentifier)
+/* expected-error {{invalid argument; expected 'enable', 'full' or 'disable'}} */ #pragma clang loop unroll(badidentifier)
while (i-7 < Length) {
List[i] = i;
}
Index: test/CodeGenCXX/pragma-unroll.cpp
===================================================================
--- test/CodeGenCXX/pragma-unroll.cpp
+++ test/CodeGenCXX/pragma-unroll.cpp
@@ -93,8 +93,8 @@
for_template_define_test<double>(List, Length, Value);
}
-// CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], ![[UNROLL_FULL:.*]]}
-// CHECK: ![[UNROLL_FULL]] = !{!"llvm.loop.unroll.full"}
+// CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], ![[UNROLL_ENABLE:.*]]}
+// CHECK: ![[UNROLL_ENABLE]] = !{!"llvm.loop.unroll.enable"}
// CHECK: ![[LOOP_2]] = distinct !{![[LOOP_2:.*]], ![[UNROLL_DISABLE:.*]]}
// CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
// CHECK: ![[LOOP_3]] = distinct !{![[LOOP_3]], ![[UNROLL_8:.*]]}
Index: lib/Sema/SemaStmtAttr.cpp
===================================================================
--- lib/Sema/SemaStmtAttr.cpp
+++ lib/Sema/SemaStmtAttr.cpp
@@ -65,51 +65,62 @@
return nullptr;
}
- LoopHintAttr::OptionType Option;
LoopHintAttr::Spelling Spelling;
- if (PragmaUnroll) {
- Option = ValueExpr ? LoopHintAttr::UnrollCount : LoopHintAttr::Unroll;
- Spelling = LoopHintAttr::Pragma_unroll;
- } else if (PragmaNoUnroll) {
- Option = LoopHintAttr::Unroll;
+ LoopHintAttr::OptionType Option;
+ LoopHintAttr::LoopHintState State;
+ if (PragmaNoUnroll) {
+ // #pragma nounroll
Spelling = LoopHintAttr::Pragma_nounroll;
+ Option = LoopHintAttr::Unroll;
+ State = LoopHintAttr::Disable;
+ } else if (PragmaUnroll) {
+ Spelling = LoopHintAttr::Pragma_unroll;
+ if (ValueExpr) {
+ // #pragma unroll N
+ Option = LoopHintAttr::UnrollCount;
+ State = LoopHintAttr::Numeric;
+ } else {
+ // #pragma unroll
+ Option = LoopHintAttr::Unroll;
+ State = LoopHintAttr::Enable;
+ }
} else {
+ // #pragma clang loop ...
+ Spelling = LoopHintAttr::Pragma_clang_loop;
assert(OptionLoc && OptionLoc->Ident &&
"Attribute must have valid option info.");
- IdentifierInfo *OptionInfo = OptionLoc->Ident;
- Option = llvm::StringSwitch<LoopHintAttr::OptionType>(OptionInfo->getName())
+ Option = llvm::StringSwitch<LoopHintAttr::OptionType>(
+ OptionLoc->Ident->getName())
.Case("vectorize", LoopHintAttr::Vectorize)
.Case("vectorize_width", LoopHintAttr::VectorizeWidth)
.Case("interleave", LoopHintAttr::Interleave)
.Case("interleave_count", LoopHintAttr::InterleaveCount)
.Case("unroll", LoopHintAttr::Unroll)
.Case("unroll_count", LoopHintAttr::UnrollCount)
.Default(LoopHintAttr::Vectorize);
- Spelling = LoopHintAttr::Pragma_clang_loop;
- }
-
- LoopHintAttr::LoopHintState State = LoopHintAttr::Default;
- if (PragmaNoUnroll) {
- State = LoopHintAttr::Disable;
- } else if (Option == LoopHintAttr::VectorizeWidth ||
- Option == LoopHintAttr::InterleaveCount ||
- Option == LoopHintAttr::UnrollCount) {
- assert(ValueExpr && "Attribute must have a valid value expression.");
- if (S.CheckLoopHintExpr(ValueExpr, St->getLocStart()))
- return nullptr;
- } else if (Option == LoopHintAttr::Vectorize ||
- Option == LoopHintAttr::Interleave ||
- Option == LoopHintAttr::Unroll) {
- // Default state is assumed if StateLoc is not specified, such as with
- // '#pragma unroll'.
- if (StateLoc && StateLoc->Ident) {
+ if (Option == LoopHintAttr::VectorizeWidth ||
+ Option == LoopHintAttr::InterleaveCount ||
+ Option == LoopHintAttr::UnrollCount) {
+ assert(ValueExpr && "Attribute must have a valid value expression.");
+ if (S.CheckLoopHintExpr(ValueExpr, St->getLocStart()))
+ return nullptr;
+ State = LoopHintAttr::Numeric;
+ } else if (Option == LoopHintAttr::Vectorize ||
+ Option == LoopHintAttr::Interleave ||
+ Option == LoopHintAttr::Unroll) {
+ assert(StateLoc && StateLoc->Ident && "Loop hint must have an argument");
if (StateLoc->Ident->isStr("disable"))
State = LoopHintAttr::Disable;
else if (StateLoc->Ident->isStr("assume_safety"))
State = LoopHintAttr::AssumeSafety;
- else
+ else if (StateLoc->Ident->isStr("full"))
+ State = LoopHintAttr::Full;
+ else if (StateLoc->Ident->isStr("enable"))
State = LoopHintAttr::Enable;
- }
+ else
+ llvm_unreachable("bad loop hint argument");
+ } else
+ llvm_unreachable("bad loop hint");
}
return LoopHintAttr::CreateImplicit(S.Context, Spelling, Option, State,
@@ -183,7 +194,8 @@
CategoryState.StateAttr->getState() == LoopHintAttr::Disable)) {
// Disable hints are not compatible with numeric hints of the same
// category. As a special case, numeric unroll hints are also not
- // compatible with "enable" form of the unroll pragma, unroll(full).
+ // compatible with enable or full form of the unroll pragma because these
+ // directives indicate full unrolling.
S.Diag(OptionLoc, diag::err_pragma_loop_compatibility)
<< /*Duplicate=*/false
<< CategoryState.StateAttr->getDiagnosticName(Policy)
Index: lib/Parse/ParsePragma.cpp
===================================================================
--- lib/Parse/ParsePragma.cpp
+++ lib/Parse/ParsePragma.cpp
@@ -824,10 +824,9 @@
SourceLocation StateLoc = Toks[0].getLocation();
IdentifierInfo *StateInfo = Toks[0].getIdentifierInfo();
if (!StateInfo ||
- ((OptionUnroll ? !StateInfo->isStr("full")
- : !StateInfo->isStr("enable") &&
- !StateInfo->isStr("assume_safety")) &&
- !StateInfo->isStr("disable"))) {
+ (!StateInfo->isStr("enable") && !StateInfo->isStr("disable") &&
+ (OptionUnroll && !StateInfo->isStr("full") ||
+ !OptionUnroll && !StateInfo->isStr("assume_safety")))) {
Diag(Toks[0].getLocation(), diag::err_pragma_invalid_keyword)
<< /*FullKeyword=*/OptionUnroll;
return false;
@@ -1959,8 +1958,9 @@
/// 'assume_safety'
///
/// unroll-hint-keyword:
-/// 'full'
+/// 'enable'
/// 'disable'
+/// 'full'
///
/// loop-hint-value:
/// constant-expression
@@ -1976,10 +1976,13 @@
/// only works on inner loops.
///
/// The unroll and unroll_count directives control the concatenation
-/// unroller. Specifying unroll(full) instructs llvm to try to
-/// unroll the loop completely, and unroll(disable) disables unrolling
-/// for the loop. Specifying unroll_count(_value_) instructs llvm to
-/// try to unroll the loop the number of times indicated by the value.
+/// unroller. Specifying unroll(enable) instructs llvm to unroll the loop
+/// completely if the trip count is known at compile time and unroll partially
+/// if the trip count is not known. Specifying unroll(full) is similar to
+/// unroll(enable) but will unroll the loop only if the trip count is known at
+/// compile time. Specifying unroll(disable) disables unrolling for the
+/// loop. Specifying unroll_count(_value_) instructs llvm to try to unroll the
+/// loop the number of times indicated by the value.
void PragmaLoopHintHandler::HandlePragma(Preprocessor &PP,
PragmaIntroducerKind Introducer,
Token &Tok) {
Index: lib/CodeGen/CGLoopInfo.h
===================================================================
--- lib/CodeGen/CGLoopInfo.h
+++ lib/CodeGen/CGLoopInfo.h
@@ -41,12 +41,12 @@
bool IsParallel;
/// \brief State of loop vectorization or unrolling.
- enum LVEnableState { Unspecified, Enable, Disable };
+ enum LVEnableState { Unspecified, Enable, Disable, Full };
/// \brief Value for llvm.loop.vectorize.enable metadata.
LVEnableState VectorizeEnable;
- /// \brief Selects no metadata, llvm.unroll.full, or llvm.unroll.disable.
+ /// \brief Value for llvm.loop.unroll.* metadata (enable, disable, or full).
LVEnableState UnrollEnable;
/// \brief Value for llvm.loop.vectorize.width metadata.
@@ -127,9 +127,8 @@
}
/// \brief Set the next pushed loop unroll state.
- void setUnrollEnable(bool Enable = true) {
- StagedAttrs.UnrollEnable =
- Enable ? LoopAttributes::Enable : LoopAttributes::Disable;
+ void setUnrollState(const LoopAttributes::LVEnableState &State) {
+ StagedAttrs.UnrollEnable = State;
}
/// \brief Set the vectorize width for the next loop pushed.
Index: lib/CodeGen/CGLoopInfo.cpp
===================================================================
--- lib/CodeGen/CGLoopInfo.cpp
+++ lib/CodeGen/CGLoopInfo.cpp
@@ -67,10 +67,14 @@
// Setting unroll.full or unroll.disable
if (Attrs.UnrollEnable != LoopAttributes::Unspecified) {
- Metadata *Vals[] = {
- MDString::get(Ctx, (Attrs.UnrollEnable == LoopAttributes::Enable
- ? "llvm.loop.unroll.full"
- : "llvm.loop.unroll.disable"))};
+ std::string Name;
+ if (Attrs.UnrollEnable == LoopAttributes::Enable)
+ Name = "llvm.loop.unroll.enable";
+ else if (Attrs.UnrollEnable == LoopAttributes::Full)
+ Name = "llvm.loop.unroll.full";
+ else
+ Name = "llvm.loop.unroll.disable";
+ Metadata *Vals[] = {MDString::get(Ctx, Name)};
Args.push_back(MDNode::get(Ctx, Vals));
}
@@ -137,7 +141,7 @@
setInterleaveCount(1);
break;
case LoopHintAttr::Unroll:
- setUnrollEnable(false);
+ setUnrollState(LoopAttributes::Disable);
break;
case LoopHintAttr::UnrollCount:
case LoopHintAttr::VectorizeWidth:
@@ -153,7 +157,7 @@
setVectorizeEnable(true);
break;
case LoopHintAttr::Unroll:
- setUnrollEnable(true);
+ setUnrollState(LoopAttributes::Enable);
break;
case LoopHintAttr::UnrollCount:
case LoopHintAttr::VectorizeWidth:
@@ -178,7 +182,21 @@
break;
}
break;
- case LoopHintAttr::Default:
+ case LoopHintAttr::Full:
+ switch (Option) {
+ case LoopHintAttr::Unroll:
+ setUnrollState(LoopAttributes::Full);
+ break;
+ case LoopHintAttr::Vectorize:
+ case LoopHintAttr::Interleave:
+ case LoopHintAttr::UnrollCount:
+ case LoopHintAttr::VectorizeWidth:
+ case LoopHintAttr::InterleaveCount:
+ llvm_unreachable("Options cannot be used with 'full' hint.");
+ break;
+ }
+ break;
+ case LoopHintAttr::Numeric:
switch (Option) {
case LoopHintAttr::VectorizeWidth:
setVectorizeWidth(ValueInt);
@@ -190,13 +208,9 @@
setUnrollCount(ValueInt);
break;
case LoopHintAttr::Unroll:
- // The default option is used when '#pragma unroll' is specified.
- setUnrollEnable(true);
- break;
case LoopHintAttr::Vectorize:
case LoopHintAttr::Interleave:
- llvm_unreachable("Options cannot be assigned a value and do not have a "
- "default value.");
+ llvm_unreachable("Options cannot be assigned a value.");
break;
}
break;
Index: include/clang/Basic/DiagnosticParseKinds.td
===================================================================
--- include/clang/Basic/DiagnosticParseKinds.td
+++ include/clang/Basic/DiagnosticParseKinds.td
@@ -997,12 +997,12 @@
// Pragma loop support.
def err_pragma_loop_missing_argument : Error<
"missing argument; expected %select{an integer value|"
- "%select{'enable', 'assume_safety'|'full'}1 or 'disable'}0">;
+ "'enable', %select{'assume_safety'|'full'}1 or 'disable'}0">;
def err_pragma_loop_invalid_option : Error<
"%select{invalid|missing}0 option%select{ %1|}0; expected vectorize, "
"vectorize_width, interleave, interleave_count, unroll, or unroll_count">;
def err_pragma_invalid_keyword : Error<
- "invalid argument; expected %select{'enable', 'assume_safety'|'full'}0 or 'disable'">;
+ "invalid argument; expected 'enable', %select{'assume_safety'|'full'}0 or 'disable'">;
// Pragma unroll support.
def warn_pragma_unroll_cuda_value_in_parens : Warning<
Index: include/clang/Basic/AttrDocs.td
===================================================================
--- include/clang/Basic/AttrDocs.td
+++ include/clang/Basic/AttrDocs.td
@@ -1371,7 +1371,9 @@
do-while, or c++11 range-based for loop.
Specifying ``#pragma unroll`` without a parameter directs the loop unroller to
-attempt to fully unroll the loop if the trip count is known at compile time:
+attempt to fully unroll the loop if the trip count is known at compile time and
+attempt to partially unroll the loop if the trip count is not known at compile
+time:
.. code-block:: c++
Index: include/clang/Basic/Attr.td
===================================================================
--- include/clang/Basic/Attr.td
+++ include/clang/Basic/Attr.td
@@ -1980,8 +1980,8 @@
["Vectorize", "VectorizeWidth", "Interleave", "InterleaveCount",
"Unroll", "UnrollCount"]>,
EnumArgument<"State", "LoopHintState",
- ["default", "enable", "disable", "assume_safety"],
- ["Default", "Enable", "Disable", "AssumeSafety"]>,
+ ["enable", "disable", "numeric", "assume_safety", "full"],
+ ["Enable", "Disable", "Numeric", "AssumeSafety", "Full"]>,
ExprArgument<"Value">];
let AdditionalMembers = [{
@@ -2020,13 +2020,12 @@
std::string ValueName;
llvm::raw_string_ostream OS(ValueName);
OS << "(";
- if (option == VectorizeWidth || option == InterleaveCount ||
- option == UnrollCount)
+ if (state == Numeric)
value->printPretty(OS, nullptr, Policy);
- else if (state == Default)
- return "";
else if (state == Enable)
- OS << (option == Unroll ? "full" : "enable");
+ OS << "enable";
+ else if (state == Full)
+ OS << "full";
else if (state == AssumeSafety)
OS << "assume_safety";
else
@@ -2041,7 +2040,7 @@
if (SpellingIndex == Pragma_nounroll)
return "#pragma nounroll";
else if (SpellingIndex == Pragma_unroll)
- return "#pragma unroll" + getValueString(Policy);
+ return "#pragma unroll" + (option == UnrollCount ? getValueString(Policy) : "");
assert(SpellingIndex == Pragma_clang_loop && "Unexpected spelling");
return getOptionName(option) + getValueString(Policy);
Index: docs/LanguageExtensions.rst
===================================================================
--- docs/LanguageExtensions.rst
+++ docs/LanguageExtensions.rst
@@ -1990,11 +1990,23 @@
compile time. Partial unrolling replicates the loop body within the loop and
reduces the trip count.
-If ``unroll(full)`` is specified the unroller will attempt to fully unroll the
+If ``unroll(enable)`` is specified the unroller will attempt to fully unroll the
loop if the trip count is known at compile time. If the fully unrolled code size
is greater than an internal limit the loop will be partially unrolled up to this
-limit. If the loop count is not known at compile time the loop will not be
-unrolled.
+limit. If the trip count is not known at compile time the loop will be partially
+unrolled with a heuristically chosen unroll factor.
+
+.. code-block:: c++
+
+ #pragma clang loop unroll(enable)
+ for(...) {
+ ...
+ }
+
+If ``unroll(full)`` is specified the unroller will attempt to fully unroll the
+loop if the trip count is known at compile time identically to
+``unroll(enable)``. However, with ``unroll(full)`` the loop will not be unrolled
+if the loop count is not known at compile time.
.. code-block:: c++
@@ -2006,7 +2018,7 @@
The unroll count can be specified explicitly with ``unroll_count(_value_)`` where
_value_ is a positive integer. If this value is greater than the trip count the
loop will be fully unrolled. Otherwise the loop is partially unrolled subject
-to the same code size limit as with ``unroll(full)``.
+to the same code size limit as with ``unroll(enable)``.
.. code-block:: c++
_______________________________________________
cfe-commits mailing list
cfe-commits@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
