Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer merged PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022202232 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022188411 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022179359 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022176620 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022102606 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022097782 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2022024561 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2021943260 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2021706928 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2019363124 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2019362253 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2019271115 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1538508579
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
Will raise a independent PR to add code doc. The code is under frequent
modification as of now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2018053323 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017901758 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017872999 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017476936 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhanglistar commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1537096832
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
Add commets?
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
Add comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017244651 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017177882 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017108387 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017107151 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017090980 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017090872 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017090065 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017078751 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017062291 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017054009 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017034121 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2017027522 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2016343681 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
Surbhi-Vijay commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2015481845 This is very big feature and the PR. Will go into detail in `gluten-cbo` module later. Appreciate your work @zhztheplayer!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2014177120 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2014174950 Guys, since the change is not affecting current code path unless `spark.gluten.sql.advanced.cbo.enabled=true` I am preparing to merge this after CI passed, to ease the further development iterations. Please feel free to comment if any issues or concerns. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533286761
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboGroup.scala:
##
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoStore
+
+trait CboGroup[T <: AnyRef] {
Review Comment:
A set of nodes that share the same property set in the same cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context (the so-called
"logical properties") in the original input plan. One cluster can derive its
own set of `CboGroup`s. Nodes in one `CboGroup` share the same ("physical")
properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context (the so-called
"logical properties") in the original input plan. One cluster can derive its
own set of `CboGroup`s. Nodes in one `CboGroup` share the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r154672
##
gluten-core/src/main/scala/io/glutenproject/extension/ColumnarOverrides.scala:
##
@@ -223,6 +224,23 @@ case class ColumnarOverrideRules(session: SparkSession)
* the plan will be breakdown and decided to be fallen back or not.
*/
private def transformRules(outputsColumnar: Boolean): List[SparkSession =>
Rule[SparkPlan]] = {
+
+def maybeCbo(outputsColumnar: Boolean): List[SparkSession =>
Rule[SparkPlan]] = {
+ if (GlutenConfig.getConf.enableAdvancedCbo) {
+return List(
+ (_: SparkSession) => FilterTransformRule(),
+ (session: SparkSession) => EnumeratedTransform(session,
outputsColumnar),
+ (_: SparkSession) => RemoveTransitions,
+ (_: SparkSession) => AggregationTransformRule()
+)
+ }
+ List(
+(_: SparkSession) => FilterTransformRule(),
+(_: SparkSession) => TransformPreOverrides(),
+(_: SparkSession) => AggregationTransformRule()
+ )
Review Comment:
So far CBO optimization happens inside Spark rule `EnumeratedTransform`.
##
gluten-core/src/main/scala/io/glutenproject/extension/ColumnarOverrides.scala:
##
@@ -223,6 +224,23 @@ case class ColumnarOverrideRules(session: SparkSession)
* the plan will be breakdown and decided to be fallen back or not.
*/
private def transformRules(outputsColumnar: Boolean): List[SparkSession =>
Rule[SparkPlan]] = {
+
+def maybeCbo(outputsColumnar: Boolean): List[SparkSession =>
Rule[SparkPlan]] = {
+ if (GlutenConfig.getConf.enableAdvancedCbo) {
+return List(
+ (_: SparkSession) => FilterTransformRule(),
+ (session: SparkSession) => EnumeratedTransform(session,
outputsColumnar),
+ (_: SparkSession) => RemoveTransitions,
+ (_: SparkSession) => AggregationTransformRule()
+)
+ }
+ List(
+(_: SparkSession) => FilterTransformRule(),
+(_: SparkSession) => TransformPreOverrides(),
+(_: SparkSession) => AggregationTransformRule()
+ )
Review Comment:
So far ACBO optimization happens inside Spark rule `EnumeratedTransform`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original
input plan. One cluster can derive its own set of `CboGroup`s. Nodes in one
`CboGroup` shares the same properties.
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original
input plan. One cluster can derive its own set of `CboGroup`s. Nodes in one
`CboGroup` share the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533290705
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/Memo.scala:
##
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo.memo
+
+import io.glutenproject.cbo._
+import io.glutenproject.cbo.CboCluster.ImmutableCboCluster
+import io.glutenproject.cbo.util.CanonicalNodeMap
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait Closure[T <: AnyRef] {
+ def defineEquiv(node: CanonicalNode[T], newNode: T): Unit
+}
+
+trait Memo[T <: AnyRef] extends Closure[T] {
Review Comment:
`Memo` is the basic structure that stores the whole search space of planner.
All the nodes stored in it are canonized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533290705
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/Memo.scala:
##
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo.memo
+
+import io.glutenproject.cbo._
+import io.glutenproject.cbo.CboCluster.ImmutableCboCluster
+import io.glutenproject.cbo.util.CanonicalNodeMap
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait Closure[T <: AnyRef] {
+ def defineEquiv(node: CanonicalNode[T], newNode: T): Unit
+}
+
+trait Memo[T <: AnyRef] extends Closure[T] {
Review Comment:
`Memo` is the basic structure that stores the whole search space of planner.
All the codes stored in it are canonized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533289802 ## gluten-cbo/common/src/main/scala/io/glutenproject/cbo/memo/ForwardMemoTable.scala: ## @@ -0,0 +1,217 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.glutenproject.cbo.memo + +import io.glutenproject.cbo._ +import io.glutenproject.cbo.CboCluster.MutableCboCluster +import io.glutenproject.cbo.memo.MemoTable.Probe +import io.glutenproject.cbo.util.IndexDisjointSet + +import scala.collection.mutable + +class ForwardMemoTable[T <: AnyRef] private (override val cbo: Cbo[T]) Review Comment: The memo table that handles cluster merging / forwarding internally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533289122
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboPlanner.scala:
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig.PlannerType
+import io.glutenproject.cbo.dp.DpPlanner
+import io.glutenproject.cbo.exaustive.ExhaustivePlanner
+import io.glutenproject.cbo.memo.MemoState
+import io.glutenproject.cbo.path.{CboPath, PathKeySet}
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait CboPlanner[T <: AnyRef] {
+ def plan(): T
+ def newState(): PlannerState[T]
+}
+
+object CboPlanner {
+ def apply[T <: AnyRef](
+ cbo: Cbo[T],
+ altPropSets: Seq[PropertySet[T]],
+ reqPropSet: PropertySet[T],
+ plan: T): CboPlanner[T] = {
+cbo.config.plannerType match {
+ case PlannerType.Exhaustive =>
+ExhaustivePlanner(cbo, altPropSets, reqPropSet, plan)
+ case PlannerType.Dp =>
+DpPlanner(cbo, altPropSets, reqPropSet, plan)
+}
+ }
+}
+
+trait Best[T <: AnyRef] {
+ import Best._
+ def rootGroupId(): Int
+ def bestNodes(): Set[InGroupNode[T]]
+ def winnerNodes(): Set[InGroupNode[T]]
+ def costs(): InGroupNode[T] => Option[Cost]
+ def path(allGroups: Int => CboGroup[T]): KnownCostPath[T]
+}
+
+object Best {
+ def apply[T <: AnyRef](
+ cbo: Cbo[T],
+ rootGroupId: Int,
+ bestNodes: Seq[InGroupNode[T]],
+ winnerNodes: Seq[InGroupNode[T]],
+ costs: InGroupNode[T] => Option[Cost]): Best[T] = {
+val bestNodeSet = bestNodes.toSet
+val winnerNodeSet = winnerNodes.toSet
+
+BestImpl(cbo, rootGroupId, bestNodeSet, winnerNodeSet, costs)
+ }
+
+ private case class BestImpl[T <: AnyRef](
+ cbo: Cbo[T],
+ override val rootGroupId: Int,
+ override val bestNodes: Set[InGroupNode[T]],
+ override val winnerNodes: Set[InGroupNode[T]],
+ override val costs: InGroupNode[T] => Option[Cost])
+extends Best[T] {
+override def path(allGroups: Int => CboGroup[T]): KnownCostPath[T] = {
+ val groupToBestNode: Map[Int, InGroupNode[T]] =
+bestNodes.map(ign => (ign.groupId, ign)).toMap
+
+ def dfs(groupId: Int): CboPath[T] = {
+val can: CanonicalNode[T] = groupToBestNode(groupId).can
+if (can.isLeaf()) {
+ return CboPath.one(cbo, PathKeySet.trivial, allGroups, can)
+}
+val children = can
+ .getChildrenGroups(allGroups)
+ .map(g => dfs(g.groupId()))
+CboPath(cbo, can, children).get
+ }
+
+ val rootCost = costs(groupToBestNode(rootGroupId))
+
+ KnownCostPath(dfs(rootGroupId), rootCost.get)
+}
+ }
+
+ trait KnownCostPath[T <: AnyRef] {
+def cboPath: CboPath[T]
+def cost: Cost
+ }
+
+ object KnownCostPath {
+def apply[T <: AnyRef](cbo: Cbo[T], cboPath: CboPath[T]): KnownCostPath[T]
= {
+ KnownCostPathImpl(cboPath, cbo.costModel.costOf(cboPath.plan()))
+}
+
+def apply[T <: AnyRef](cboPath: CboPath[T], cost: Cost): KnownCostPath[T]
= {
+ KnownCostPathImpl(cboPath, cost)
+}
+
+private case class KnownCostPathImpl[T <: AnyRef](cboPath: CboPath[T],
cost: Cost)
+ extends KnownCostPath[T]
+ }
+
+ case class BestNotFoundException(message: String, cause: Exception)
+extends RuntimeException(message, cause)
+ object BestNotFoundException {
+def apply(message: String): BestNotFoundException = {
+ BestNotFoundException(message, null)
+}
+def apply(): BestNotFoundException = {
+ BestNotFoundException(null, null)
+}
+ }
+}
+
+trait PlannerState[T <: AnyRef] {
+ def cbo(): Cbo[T]
+ def memoState(): MemoState[T]
+ def rootGroupId(): Int
+ def best(): Best[T]
+}
Review Comment:
The immutable dump of the planner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533288763
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboPlanner.scala:
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig.PlannerType
+import io.glutenproject.cbo.dp.DpPlanner
+import io.glutenproject.cbo.exaustive.ExhaustivePlanner
+import io.glutenproject.cbo.memo.MemoState
+import io.glutenproject.cbo.path.{CboPath, PathKeySet}
+import io.glutenproject.cbo.vis.GraphvizVisualizer
+
+trait CboPlanner[T <: AnyRef] {
+ def plan(): T
+ def newState(): PlannerState[T]
+}
+
+object CboPlanner {
+ def apply[T <: AnyRef](
+ cbo: Cbo[T],
+ altPropSets: Seq[PropertySet[T]],
+ reqPropSet: PropertySet[T],
+ plan: T): CboPlanner[T] = {
+cbo.config.plannerType match {
+ case PlannerType.Exhaustive =>
+ExhaustivePlanner(cbo, altPropSets, reqPropSet, plan)
+ case PlannerType.Dp =>
+DpPlanner(cbo, altPropSets, reqPropSet, plan)
+}
+ }
+}
+
+trait Best[T <: AnyRef] {
+ import Best._
+ def rootGroupId(): Int
+ def bestNodes(): Set[InGroupNode[T]]
+ def winnerNodes(): Set[InGroupNode[T]]
+ def costs(): InGroupNode[T] => Option[Cost]
+ def path(allGroups: Int => CboGroup[T]): KnownCostPath[T]
+}
Review Comment:
`Best` is basically the output of one shot of planning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533288122
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
Review Comment:
A single immutable node wrapper that hides the tree structure from it.
For represent tree structure, use `CboPath`.
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
Review Comment:
A single immutable node wrapper that hides the tree structure from it.
For representing tree structure, use `CboPath`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533287599
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
+ def cbo(): Cbo[T]
+ def self(): T
+ def propSet(): PropertySet[T]
+}
+
+object CboNode {
+ implicit class CboNodeImplicits[T <: AnyRef](node: CboNode[T]) {
+def isCanonical: Boolean = {
+ node.isInstanceOf[CanonicalNode[T]]
+}
+
+def asCanonical(): CanonicalNode[T] = {
+ node.asInstanceOf[CanonicalNode[T]]
+}
+
+def isGroup: Boolean = {
+ node.isInstanceOf[GroupNode[T]]
+}
+
+def asGroup(): GroupNode[T] = {
+ node.asInstanceOf[GroupNode[T]]
+}
+ }
+}
+
+trait CanonicalNode[T <: AnyRef] extends CboNode[T] {
+ def childrenCount: Int
+}
+
+object CanonicalNode {
+ def apply[T <: AnyRef](cbo: Cbo[T], canonical: T): CanonicalNode[T] = {
+assert(cbo.isCanonical(canonical))
+val propSet = cbo.propSetsOf(canonical)
+val children = cbo.planModel.childrenOf(canonical)
+CanonicalNodeImpl[T](cbo, canonical, propSet, children.size)
+ }
+
+ // We put CboNode's API methods that accept mutable input in implicit
definition.
+ // Do not break this rule during further development.
+ implicit class CanonicalNodeImplicits[T <: AnyRef](node: CanonicalNode[T]) {
+def isLeaf(): Boolean = {
+ node.childrenCount == 0
+}
+
+def getChildrenGroups(allGroups: Int => CboGroup[T]): Seq[GroupNode[T]] = {
+ val cbo = node.cbo()
+ cbo.getChildrenGroups(allGroups, node.self()).map(g => GroupNode(cbo, g))
+}
+ }
+
+ private case class CanonicalNodeImpl[T <: AnyRef](
+ cbo: Cbo[T],
+ override val self: T,
+ override val propSet: PropertySet[T],
+ override val childrenCount: Int)
+extends CanonicalNode[T]
+}
+
+trait GroupNode[T <: AnyRef] extends CboNode[T] {
Review Comment:
A node that exactly represents a group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533287265
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboNode.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+trait CboNode[T <: AnyRef] {
+ def cbo(): Cbo[T]
+ def self(): T
+ def propSet(): PropertySet[T]
+}
+
+object CboNode {
+ implicit class CboNodeImplicits[T <: AnyRef](node: CboNode[T]) {
+def isCanonical: Boolean = {
+ node.isInstanceOf[CanonicalNode[T]]
+}
+
+def asCanonical(): CanonicalNode[T] = {
+ node.asInstanceOf[CanonicalNode[T]]
+}
+
+def isGroup: Boolean = {
+ node.isInstanceOf[GroupNode[T]]
+}
+
+def asGroup(): GroupNode[T] = {
+ node.asInstanceOf[GroupNode[T]]
+}
+ }
+}
+
+trait CanonicalNode[T <: AnyRef] extends CboNode[T] {
Review Comment:
Canonical node is a node with all children replaced by resident groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533286761
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboGroup.scala:
##
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoStore
+
+trait CboGroup[T <: AnyRef] {
Review Comment:
A set of nodes that shares the same property set in the same cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533285973
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboConfig.scala:
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig._
+
+case class CboConfig(
+plannerType: PlannerType = PlannerType.Dp
+)
+
+object CboConfig {
+ sealed trait PlannerType
+ object PlannerType {
+case object Exhaustive extends PlannerType
+case object Dp extends PlannerType
+ }
Review Comment:
`Dp` is the default planner implementation while `Exhaustive` is currently
only used for testing. It's expected that we can implement parallelized
optimization on exhaustive planner comparatively easier than on dp planner in
future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533285973
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboConfig.scala:
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.CboConfig._
+
+case class CboConfig(
+plannerType: PlannerType = PlannerType.Dp
+)
+
+object CboConfig {
+ sealed trait PlannerType
+ object PlannerType {
+case object Exhaustive extends PlannerType
+case object Dp extends PlannerType
+ }
Review Comment:
`Dp` is the default planner implementation while `Exhaustive` is currently
only used for testing. It's expected that we can implement parallelized
optimization on exhaustive planner comparatively easier than on dp planner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original
input plan. One cluster can derive a subset of `CboGroup`s. Nodes in one
`CboGroup` shares the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533284131
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/CboCluster.scala:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.memo.MemoTable
+
+import scala.collection.mutable
+
+trait CboClusterKey
+
+object CboClusterKey {
+ implicit class CboClusterKeyImplicits[T <: AnyRef](key: CboClusterKey) {
+def propSets(memoTable: MemoTable[T]): Set[PropertySet[T]] = {
+ memoTable.getClusterPropSets(key)
+}
+ }
+}
+
+trait CboCluster[T <: AnyRef] {
Review Comment:
`CboCluster` is a set of nodes sharing the same context in the original
input plan. One cluster can derives a subset of `CboGroup`s. Nodes in one
`CboGroup` shares the same properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533282556
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala:
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.rule.CboRule
+
+import scala.collection.mutable
+
+/**
+ * Entrypoint of ACBO (Advanced CBO)'s search engine. See basic introduction
of ACBO:
+ * https://github.com/apache/incubator-gluten/issues/5057.
+ */
+trait Optimization[T <: AnyRef] {
+ def newPlanner(
+ plan: T,
+ reqPropSet: PropertySet[T],
+ altPropSets: Seq[PropertySet[T]]): CboPlanner[T]
+
+ def propSetsOf(plan: T): PropertySet[T]
+
+ def withNewConfig(confFunc: CboConfig => CboConfig): Optimization[T]
+}
+
+object Optimization {
+ def apply[T <: AnyRef](
+ costModel: CostModel[T],
+ planModel: PlanModel[T],
+ propertyModel: PropertyModel[T],
+ explain: CboExplain[T],
+ ruleFactory: CboRule.Factory[T]): Optimization[T] = {
+Cbo(costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ implicit class OptimizationImplicits[T <: AnyRef](opt: Optimization[T]) {
+def newPlanner(plan: T): CboPlanner[T] = {
+ opt.newPlanner(plan, opt.propSetsOf(plan), List.empty)
+}
+def newPlanner(plan: T, reqPropSet: PropertySet[T]): CboPlanner[T] = {
+ opt.newPlanner(plan, reqPropSet, List.empty)
+}
+ }
+}
+
+class Cbo[T <: AnyRef] private (
+val config: CboConfig,
+val costModel: CostModel[T],
+val planModel: PlanModel[T],
+val propertyModel: PropertyModel[T],
+val explain: CboExplain[T],
+val ruleFactory: CboRule.Factory[T])
+ extends Optimization[T] {
+ import Cbo._
+
+ override def withNewConfig(confFunc: CboConfig => CboConfig): Cbo[T] = {
+new Cbo(confFunc(config), costModel, planModel, propertyModel, explain,
ruleFactory)
+ }
+
+ // Normal groups start with ID 0, so it's safe to use -1 to do validation.
+ private val dummyGroup: T =
+planModel.newGroupLeaf(-1, PropertySet(Seq.empty), GroupMutator.noop())
+ private val infCost: Cost = costModel.makeInfCost()
+
+ validateModels()
+
+ private def assertThrows(message: String)(u: => Unit): Unit = {
+var notThrew: Boolean = false
+try {
+ u
+ notThrew = true
+} catch {
+ case _: Exception =>
+}
+assert(!notThrew, message)
+ }
+
+ private def validateModels(): Unit = {
Review Comment:
Do validation on user's API implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533281959
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala:
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.rule.CboRule
+
+import scala.collection.mutable
+
+/**
+ * Entrypoint of ACBO (Advanced CBO)'s search engine. See basic introduction
of ACBO:
+ * https://github.com/apache/incubator-gluten/issues/5057.
+ */
+trait Optimization[T <: AnyRef] {
+ def newPlanner(
+ plan: T,
+ reqPropSet: PropertySet[T],
+ altPropSets: Seq[PropertySet[T]]): CboPlanner[T]
+
+ def propSetsOf(plan: T): PropertySet[T]
+
+ def withNewConfig(confFunc: CboConfig => CboConfig): Optimization[T]
+}
+
+object Optimization {
+ def apply[T <: AnyRef](
+ costModel: CostModel[T],
+ planModel: PlanModel[T],
+ propertyModel: PropertyModel[T],
+ explain: CboExplain[T],
+ ruleFactory: CboRule.Factory[T]): Optimization[T] = {
+Cbo(costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ implicit class OptimizationImplicits[T <: AnyRef](opt: Optimization[T]) {
+def newPlanner(plan: T): CboPlanner[T] = {
+ opt.newPlanner(plan, opt.propSetsOf(plan), List.empty)
+}
+def newPlanner(plan: T, reqPropSet: PropertySet[T]): CboPlanner[T] = {
+ opt.newPlanner(plan, reqPropSet, List.empty)
+}
+ }
+}
+
+class Cbo[T <: AnyRef] private (
Review Comment:
`Cbo` is a stateless optimization context consisting of configs and
utilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058:
URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533281959
##
gluten-cbo/common/src/main/scala/io/glutenproject/cbo/Cbo.scala:
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.glutenproject.cbo
+
+import io.glutenproject.cbo.rule.CboRule
+
+import scala.collection.mutable
+
+/**
+ * Entrypoint of ACBO (Advanced CBO)'s search engine. See basic introduction
of ACBO:
+ * https://github.com/apache/incubator-gluten/issues/5057.
+ */
+trait Optimization[T <: AnyRef] {
+ def newPlanner(
+ plan: T,
+ reqPropSet: PropertySet[T],
+ altPropSets: Seq[PropertySet[T]]): CboPlanner[T]
+
+ def propSetsOf(plan: T): PropertySet[T]
+
+ def withNewConfig(confFunc: CboConfig => CboConfig): Optimization[T]
+}
+
+object Optimization {
+ def apply[T <: AnyRef](
+ costModel: CostModel[T],
+ planModel: PlanModel[T],
+ propertyModel: PropertyModel[T],
+ explain: CboExplain[T],
+ ruleFactory: CboRule.Factory[T]): Optimization[T] = {
+Cbo(costModel, planModel, propertyModel, explain, ruleFactory)
+ }
+
+ implicit class OptimizationImplicits[T <: AnyRef](opt: Optimization[T]) {
+def newPlanner(plan: T): CboPlanner[T] = {
+ opt.newPlanner(plan, opt.propSetsOf(plan), List.empty)
+}
+def newPlanner(plan: T, reqPropSet: PropertySet[T]): CboPlanner[T] = {
+ opt.newPlanner(plan, reqPropSet, List.empty)
+}
+ }
+}
+
+class Cbo[T <: AnyRef] private (
Review Comment:
`Cbo` is a stateless optimization context consisting configs and utilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
zhztheplayer commented on code in PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#discussion_r1533281350 ## .github/workflows/velox_be.yml: ## @@ -450,6 +450,16 @@ jobs: --local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \ && GLUTEN_IT_JVM_ARGS=-Xmx20G sbin/gluten-it.sh queries-compare \ --local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=40g -s=10.0 --threads=32 --iterations=1' + - name: TPC-H SF1.0 && TPC-DS SF10.0 Parquet local spark3.2 with advanced CBO +run: | + $PATH_TO_GLUTEN_TE/$OS_IMAGE_NAME/gha/gha-checkout/exec.sh 'cd /opt/gluten/tools/gluten-it && \ + mvn clean install -Pspark-3.2 \ + && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \ +--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \ +--extra-conf=spark.gluten.sql.advanced.cbo.enabled=true \ + && GLUTEN_IT_JVM_ARGS=-Xmx20G sbin/gluten-it.sh queries-compare \ +--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=40g -s=10.0 --threads=32 --iterations=1 \ +--extra-conf=spark.gluten.sql.advanced.cbo.enabled=true' Review Comment: CI Job for ACBO + Velox + TPC-H SF1 + TPC-DS SF10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011266413 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011252200 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011228018 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011222339 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011209005 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011206379 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011169296 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011163149 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011095509 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011021726 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011020568 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011012096 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058: URL: https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011010259 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [PR] [CORE] Basic runnable version of ACBO (Advanced CBO) [incubator-gluten]
github-actions[bot] commented on PR #5058:
URL:
https://github.com/apache/incubator-gluten/pull/5058#issuecomment-2011010109
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title***
in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
