[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/190 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/190#discussion_r80220071 --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala --- @@ -68,9 +70,50 @@ object CarbonOptimizer { */ class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation]) extends Rule[LogicalPlan] with PredicateHelper { - + val LOGGER = LogServiceFactory.getLogService(this.getClass.getName) def apply(plan: LogicalPlan): LogicalPlan = { -transformCarbonPlan(plan, relations) +if (relations.nonEmpty && !isOptimized(plan)) { + LOGGER.info("Starting to optimize plan") + val startTime = System.currentTimeMillis + val result = transformCarbonPlan(plan, relations) + LOGGER.info("Time taken to optimize plan: " + ( System.currentTimeMillis - startTime)) --- End diff -- add this to STATISTICS logger type --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/190#discussion_r80220168 --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala --- @@ -397,20 +452,38 @@ class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation]) case others => others } +} + +val tStartTime = System.currentTimeMillis --- End diff -- this variable not used --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/190#discussion_r80219989 --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala --- @@ -68,9 +70,50 @@ object CarbonOptimizer { */ class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation]) extends Rule[LogicalPlan] with PredicateHelper { - + val LOGGER = LogServiceFactory.getLogService(this.getClass.getName) def apply(plan: LogicalPlan): LogicalPlan = { -transformCarbonPlan(plan, relations) +if (relations.nonEmpty && !isOptimized(plan)) { + LOGGER.info("Starting to optimize plan") + val startTime = System.currentTimeMillis + val result = transformCarbonPlan(plan, relations) + LOGGER.info("Time taken to optimize plan: " + ( System.currentTimeMillis - startTime)) --- End diff -- change the statement to "Time taken for Carbon Optimizer to optimize" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/190#discussion_r80219576 --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/optimizer/CarbonOptimizer.scala --- @@ -87,20 +130,32 @@ class ResolveCarbonFunctions(relations: Seq[CarbonDecoderRelation]) return plan } var decoder = false +val mapOfNodes = new java.util.HashMap[LogicalPlan, ExtraNodeInfo] +fillNodeInfo(plan, mapOfNodes) val aliasMap = CarbonAliasDecoderRelation() // collect alias information before hand. collectInformationOnAttributes(plan, aliasMap) -val transFormedPlan = - plan transformDown { -case cd: CarbonDictionaryTempDecoder if cd.isOuter => - decoder = true - cd + +def hasCarbonRelation(currentPlan: LogicalPlan): Boolean = { + val extraNodeInfo = mapOfNodes.get(currentPlan) --- End diff -- Change the name of mapofNodes to mapOfNonCarbonPlanNodes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #190: [CARBONDATA-268]Improve carbonoptimi...
GitHub user QiangCai opened a pull request: https://github.com/apache/incubator-carbondata/pull/190 [CARBONDATA-268]Improve carbonoptimizer You can merge this pull request into a Git repository by running: $ git pull https://github.com/QiangCai/incubator-carbondata improveoptimizer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/190.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #190 commit 343a3d20fca9bee15a22063b4be8d73d21b81646 Author: c00318382 Date: 2016-09-22T10:57:00Z improve carbonoptimizer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---