[GitHub] incubator-carbondata pull request #810: [CARBONDATA-935] Define PartitionInf...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/810 [CARBONDATA-935] Define PartitionInfo Model modify schema.thrift modify TableSchema.java add new partition related classes You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon_910_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/810.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #810 commit 41a720bbd43f33f1b7a6b0f14143c7805fbf6127 Author: lucao <whuca...@gmail.com> Date: 2017-04-18T06:52:18Z [CARBONDATA-935] Define PartitionInfo model --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #716: [CARBONDATA-840][WIP] improve limit ...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/716 [CARBONDATA-840][WIP] improve limit query performance [Group By] Currently limit query will still scan all data first and limit in the last step. In carbon we can convert limit to filters with dictionary distinct value list... You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon_dev Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/716.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #716 commit e023c36c5f31efa63daf7c2e8f5fec8fa3c19c7e Author: lucao <whuca...@gmail.com> Date: 2017-03-31T06:19:40Z [WIP]improve limit query performance --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #661: remove shutdown dictionary server
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/661 remove shutdown dictionary server remove shutdown dictionary server You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon761 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/661.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #661 commit f4cf54f844758441adecb3a40b16cc2868504a2d Author: lucao <whuca...@gmail.com> Date: 2017-03-16T02:53:08Z remove shutdown dictionary server --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #655: [CARBONDATA-762] Change schemaName t...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/655 [CARBONDATA-762] Change schemaName to databaseName, cubeName to table⦠Change schemaName to databaseName, cubeName to tableName You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon762 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/655.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #655 commit a73d5643d86c4055f3301d1a69616c9b1ff92870 Author: lucao <whuca...@gmail.com> Date: 2017-03-15T07:24:00Z [CARBONDATA-762] Change schemaName to databaseName, cubeName to tableName --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #636: [CARBONDATA-741] Remove useless clas...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/636 [CARBONDATA-741] Remove useless classes Remove useless classes VectorChunkRowIterator CarbonColumnVectorImpl You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon741 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/636.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #636 commit 7533e985d9ff9dd3add7b5e8be6cf075573f3593 Author: lucao <whuca...@gmail.com> Date: 2017-03-09T05:51:39Z remove useless classes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #633: [CARBONDATA-739] Avoid creating mult...
Github user lionelcao commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/633#discussion_r105068561 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java --- @@ -37,6 +34,8 @@ import org.apache.carbondata.core.util.DataTypeUtil; import org.apache.commons.lang3.ArrayUtils; + + --- End diff -- Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #629: [CARBONDATA-740] add logger for total rows ...
Github user lionelcao commented on the issue: https://github.com/apache/incubator-carbondata/pull/629 @ravipesala this PR is closed but not merged yet. I can't reopen it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #633: [CARBONDATA-739] Avoid creating mult...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/633 [CARBONDATA-739] Avoid creating multipul instance of directDictionaryGenerator Avoid creating multipul instance of directDictionaryGenerator You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon739 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/633.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #633 commit 7437c64d0bfdd20928302eb93c204ef31f54e542 Author: lucao <whuca...@gmail.com> Date: 2017-03-08T09:20:58Z [CARBONDATA-739] Avoid creating multipul instance of directDictionaryGenerator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #629: [CARBONDATA-740] add logger for total rows ...
Github user lionelcao commented on the issue: https://github.com/apache/incubator-carbondata/pull/629 @ravipesala Added step name, please check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #628: [CARBONDATA-743] Remove redundant CarbonFil...
Github user lionelcao commented on the issue: https://github.com/apache/incubator-carbondata/pull/628 Hi @ravipesala , I compared those two files before commit and I think the difference has no impact to the functionality. Any ideas? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #629: [CARBONDATA-740] add logger for tota...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/629 [CARBONDATA-740] add logger for total rows processed add logger for total rows processed You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon740 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/629.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #629 commit 11f79f1b4df6826639c8dcb8637e70c773738b5a Author: lucao <whuca...@gmail.com> Date: 2017-03-07T09:56:00Z [CARBONDATA-740] add logger for total rows processed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #628: [CARBONDATA-743] Remove redundant Ca...
GitHub user lionelcao opened a pull request: https://github.com/apache/incubator-carbondata/pull/628 [CARBONDATA-743] Remove redundant CarbonFilters file Remove redundant CarbonFilters file in spark2 and keep the one in spark common You can merge this pull request into a Git repository by running: $ git pull https://github.com/lionelcao/incubator-carbondata carbon743 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/628.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #628 commit 3ac7a8c25650b92cc7819ce10213b3cdfd8b8135 Author: lucao <whuca...@gmail.com> Date: 2017-03-07T09:51:37Z [CARBONDATA-743] Remove redundant CarbonFilters file --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #625: [CARBONDATA-743] Remove redundant Ca...
Github user lionelcao closed the pull request at: https://github.com/apache/incubator-carbondata/pull/625 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #625: [CARBONDATA-743] Remove redundant Ca...
Github user lionelcao commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/625#discussion_r104575573
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/CarbonFilters.scala
---
@@ -1,397 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.carbondata.spark
-
-import scala.collection.mutable.ArrayBuffer
-
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.optimizer.AttributeReferenceWrapper
-import org.apache.spark.sql.sources
-import org.apache.spark.sql.types.StructType
-
-import org.apache.carbondata.core.metadata.datatype.DataType
-import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn
-import org.apache.carbondata.core.scan.expression.{ColumnExpression =>
CarbonColumnExpression, Expression => CarbonExpression, LiteralExpression =>
CarbonLiteralExpression}
-import org.apache.carbondata.core.scan.expression.conditional._
-import org.apache.carbondata.core.scan.expression.logical.{AndExpression,
FalseExpression, OrExpression}
-import org.apache.carbondata.spark.util.CarbonScalaUtil
-
-/**
- * All filter conversions are done here.
- */
-object CarbonFilters {
-
- /**
- * Converts data sources filters to carbon filter predicates.
- */
- def createCarbonFilter(schema: StructType,
- predicate: sources.Filter): Option[CarbonExpression] = {
-val dataTypeOf = schema.map(f => f.name -> f.dataType).toMap
-
-def createFilter(predicate: sources.Filter): Option[CarbonExpression]
= {
- predicate match {
-
-case sources.EqualTo(name, value) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.Not(sources.EqualTo(name, value)) =>
- Some(new NotEqualsExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.EqualNullSafe(name, value) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.Not(sources.EqualNullSafe(name, value)) =>
- Some(new NotEqualsExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.GreaterThan(name, value) =>
- Some(new GreaterThanExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.LessThan(name, value) =>
- Some(new LessThanExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.GreaterThanOrEqual(name, value) =>
- Some(new GreaterThanEqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.LessThanOrEqual(name, value) =>
- Some(new LessThanEqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
-new ListExpression(
- convertToJavaList(values.map(f =>
getCarbonLiteralExpression(name, f)).toList
-case sources.Not(sources.In(name, values)) =>
- Some(new NotInExpression(getCarbonExpression(name),
-new ListExpression(
- convertToJavaList(values.map(f =>
getCarbonLiteralExpression(name, f)).toList
-
-case sources.IsNull(name) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, null), true))
-case sources.IsNotNull(nam
[GitHub] incubator-carbondata pull request #625: [CARBONDATA-743] Remove redundant Ca...
Github user lionelcao commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/625#discussion_r104575582
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/CarbonFilters.scala
---
@@ -1,397 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.carbondata.spark
-
-import scala.collection.mutable.ArrayBuffer
-
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.optimizer.AttributeReferenceWrapper
-import org.apache.spark.sql.sources
-import org.apache.spark.sql.types.StructType
-
-import org.apache.carbondata.core.metadata.datatype.DataType
-import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn
-import org.apache.carbondata.core.scan.expression.{ColumnExpression =>
CarbonColumnExpression, Expression => CarbonExpression, LiteralExpression =>
CarbonLiteralExpression}
-import org.apache.carbondata.core.scan.expression.conditional._
-import org.apache.carbondata.core.scan.expression.logical.{AndExpression,
FalseExpression, OrExpression}
-import org.apache.carbondata.spark.util.CarbonScalaUtil
-
-/**
- * All filter conversions are done here.
- */
-object CarbonFilters {
-
- /**
- * Converts data sources filters to carbon filter predicates.
- */
- def createCarbonFilter(schema: StructType,
- predicate: sources.Filter): Option[CarbonExpression] = {
-val dataTypeOf = schema.map(f => f.name -> f.dataType).toMap
-
-def createFilter(predicate: sources.Filter): Option[CarbonExpression]
= {
- predicate match {
-
-case sources.EqualTo(name, value) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.Not(sources.EqualTo(name, value)) =>
- Some(new NotEqualsExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.EqualNullSafe(name, value) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.Not(sources.EqualNullSafe(name, value)) =>
- Some(new NotEqualsExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.GreaterThan(name, value) =>
- Some(new GreaterThanExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.LessThan(name, value) =>
- Some(new LessThanExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.GreaterThanOrEqual(name, value) =>
- Some(new GreaterThanEqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.LessThanOrEqual(name, value) =>
- Some(new LessThanEqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
-new ListExpression(
- convertToJavaList(values.map(f =>
getCarbonLiteralExpression(name, f)).toList
-case sources.Not(sources.In(name, values)) =>
- Some(new NotInExpression(getCarbonExpression(name),
-new ListExpression(
- convertToJavaList(values.map(f =>
getCarbonLiteralExpression(name, f)).toList
-
-case sources.IsNull(name) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, null), true))
-case sources.IsNotNull(nam
[GitHub] incubator-carbondata pull request #625: [CARBONDATA-743] Remove redundant Ca...
Github user lionelcao commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/625#discussion_r104578046
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/CarbonFilters.scala
---
@@ -1,397 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.carbondata.spark
-
-import scala.collection.mutable.ArrayBuffer
-
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.optimizer.AttributeReferenceWrapper
-import org.apache.spark.sql.sources
-import org.apache.spark.sql.types.StructType
-
-import org.apache.carbondata.core.metadata.datatype.DataType
-import org.apache.carbondata.core.metadata.schema.table.CarbonTable
-import org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn
-import org.apache.carbondata.core.scan.expression.{ColumnExpression =>
CarbonColumnExpression, Expression => CarbonExpression, LiteralExpression =>
CarbonLiteralExpression}
-import org.apache.carbondata.core.scan.expression.conditional._
-import org.apache.carbondata.core.scan.expression.logical.{AndExpression,
FalseExpression, OrExpression}
-import org.apache.carbondata.spark.util.CarbonScalaUtil
-
-/**
- * All filter conversions are done here.
- */
-object CarbonFilters {
-
- /**
- * Converts data sources filters to carbon filter predicates.
- */
- def createCarbonFilter(schema: StructType,
- predicate: sources.Filter): Option[CarbonExpression] = {
-val dataTypeOf = schema.map(f => f.name -> f.dataType).toMap
-
-def createFilter(predicate: sources.Filter): Option[CarbonExpression]
= {
- predicate match {
-
-case sources.EqualTo(name, value) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.Not(sources.EqualTo(name, value)) =>
- Some(new NotEqualsExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.EqualNullSafe(name, value) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.Not(sources.EqualNullSafe(name, value)) =>
- Some(new NotEqualsExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.GreaterThan(name, value) =>
- Some(new GreaterThanExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.LessThan(name, value) =>
- Some(new LessThanExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.GreaterThanOrEqual(name, value) =>
- Some(new GreaterThanEqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-case sources.LessThanOrEqual(name, value) =>
- Some(new LessThanEqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, value)))
-
-case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
-new ListExpression(
- convertToJavaList(values.map(f =>
getCarbonLiteralExpression(name, f)).toList
-case sources.Not(sources.In(name, values)) =>
- Some(new NotInExpression(getCarbonExpression(name),
-new ListExpression(
- convertToJavaList(values.map(f =>
getCarbonLiteralExpression(name, f)).toList
-
-case sources.IsNull(name) =>
- Some(new EqualToExpression(getCarbonExpression(name),
-getCarbonLiteralExpression(name, null), true))
-case sources.IsNotNull(nam
