[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486797641
##
File path:
integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##
@@ -245,16 +242,14 @@ private CarbonTableCacheModel
getValidCacheBySchemaTableName(SchemaTableName sch
*
* @param tableCacheModel cached table
* @param filters carbonData filters
- * @param constraints presto filters
+ * @param filteredPartitions matched partitionSpec for the filter
* @param config hadoop conf
* @return list of multiblock split
* @throws IOException
*/
- public List getInputSplits(
- CarbonTableCacheModel tableCacheModel,
- Expression filters,
- TupleDomain constraints,
- Configuration config) throws IOException {
+ public List getInputSplits(CarbonTableCacheModel
tableCacheModel,
+ Expression filters, List filteredPartitions,
Configuration config)
Review comment:
done
##
File path:
integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.presto.server
+
+import com.facebook.presto.jdbc.PrestoArray
+
Review comment:
done
##
File path:
integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##
@@ -245,16 +242,14 @@ private CarbonTableCacheModel
getValidCacheBySchemaTableName(SchemaTableName sch
*
* @param tableCacheModel cached table
* @param filters carbonData filters
- * @param constraints presto filters
+ * @param filteredPartitions matched partitionSpec for the filter
* @param config hadoop conf
* @return list of multiblock split
* @throws IOException
*/
- public List getInputSplits(
- CarbonTableCacheModel tableCacheModel,
- Expression filters,
- TupleDomain constraints,
- Configuration config) throws IOException {
+ public List getInputSplits(CarbonTableCacheModel
tableCacheModel,
+ Expression filters, List filteredPartitions,
Configuration config)
Review comment:
done
##
File path:
integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.presto.server
+
+import com.facebook.presto.jdbc.PrestoArray
+
Review comment:
done
##
File path:
integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##
@@ -245,16 +242,14 @@ private CarbonTableCacheModel
getValidCacheBySchemaTableName(SchemaTableName sch
*
* @param tableCacheModel cached table
* @param filters carbonData filters
- * @param constraints presto filters
+ * @param filteredPartitions matched partitionSpec for the filter
* @param config hadoop conf
* @return list of multiblock split
* @throws IOException
*/
- public List getInputSplits(
- CarbonTableCacheModel tableCacheModel,
- Expression filters,
- TupleDomain constraints,
- Configuration config) throws IOException {
+ public List getInputSplits(CarbonTableCacheModel
tableCacheModel,
+ Expression filters, List filteredPartitions,
Configuration config)
Review comment:
done
###
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486797641
##
File path:
integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##
@@ -245,16 +242,14 @@ private CarbonTableCacheModel
getValidCacheBySchemaTableName(SchemaTableName sch
*
* @param tableCacheModel cached table
* @param filters carbonData filters
- * @param constraints presto filters
+ * @param filteredPartitions matched partitionSpec for the filter
* @param config hadoop conf
* @return list of multiblock split
* @throws IOException
*/
- public List getInputSplits(
- CarbonTableCacheModel tableCacheModel,
- Expression filters,
- TupleDomain constraints,
- Configuration config) throws IOException {
+ public List getInputSplits(CarbonTableCacheModel
tableCacheModel,
+ Expression filters, List filteredPartitions,
Configuration config)
Review comment:
done
##
File path:
integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.presto.server
+
+import com.facebook.presto.jdbc.PrestoArray
+
Review comment:
done
##
File path:
integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##
@@ -245,16 +242,14 @@ private CarbonTableCacheModel
getValidCacheBySchemaTableName(SchemaTableName sch
*
* @param tableCacheModel cached table
* @param filters carbonData filters
- * @param constraints presto filters
+ * @param filteredPartitions matched partitionSpec for the filter
* @param config hadoop conf
* @return list of multiblock split
* @throws IOException
*/
- public List getInputSplits(
- CarbonTableCacheModel tableCacheModel,
- Expression filters,
- TupleDomain constraints,
- Configuration config) throws IOException {
+ public List getInputSplits(CarbonTableCacheModel
tableCacheModel,
+ Expression filters, List filteredPartitions,
Configuration config)
Review comment:
done
##
File path:
integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.presto.server
+
+import com.facebook.presto.jdbc.PrestoArray
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486797641
##
File path:
integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##
@@ -245,16 +242,14 @@ private CarbonTableCacheModel
getValidCacheBySchemaTableName(SchemaTableName sch
*
* @param tableCacheModel cached table
* @param filters carbonData filters
- * @param constraints presto filters
+ * @param filteredPartitions matched partitionSpec for the filter
* @param config hadoop conf
* @return list of multiblock split
* @throws IOException
*/
- public List getInputSplits(
- CarbonTableCacheModel tableCacheModel,
- Expression filters,
- TupleDomain constraints,
- Configuration config) throws IOException {
+ public List getInputSplits(CarbonTableCacheModel
tableCacheModel,
+ Expression filters, List filteredPartitions,
Configuration config)
Review comment:
done
##
File path:
integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.presto.server
+
+import com.facebook.presto.jdbc.PrestoArray
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486048413
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##
@@ -117,6 +122,16 @@ public ConnectorSplitSource
getSplits(ConnectorTransactionHandle transactionHand
// file metastore case tablePath can be null, so get from location
location = table.getStorage().getLocation();
}
+List filteredPartitions = new ArrayList<>();
+if (layout.getPartitionColumns().size() > 0 &&
layout.getPartitions().isPresent()) {
+ List colNames =
+ layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle)
x).getName())
+ .collect(Collectors.toList());
+ for (HivePartition partition : layout.getPartitions().get()) {
+filteredPartitions.add(new PartitionSpec(colNames,
Review comment:
```
presto:redods> select dtm,hh from dw_log_ubt_partition_carbon_neww;
dtm | hh
-+
part_dtm_01 | part_hh_01
part_dtm_01 | part_hh_01
part_dtm_01 | part_hh_02
part_dtm_20 | part_hh_21
part_dtm_01 | part_hh_03
part_dtm_21 | NULL
(6 rows)
Query 20200910_035416_00017_wv9qh, FINISHED, 3 nodes
Splits: 22 total, 22 done (100.00%)
0:01 [6 rows, 176B] [9 rows/s, 282B/s]
presto:redods> select dtm,hh from dw_log_ubt_partition_carbon_neww where
(dtm = 'part_dtm_01' and hh = 'part_hh_03') or dtm='part_dtm_21';
dtm | hh
-+
part_dtm_01 | part_hh_03
part_dtm_21 | NULL
(2 rows)
Query 20200910_035548_00018_wv9qh, FINISHED, 3 nodes
Splits: 18 total, 18 done (100.00%)
0:01 [2 rows, 0B] [1 rows/s, 0B/s]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486048290
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##
@@ -117,6 +122,16 @@ public ConnectorSplitSource
getSplits(ConnectorTransactionHandle transactionHand
// file metastore case tablePath can be null, so get from location
location = table.getStorage().getLocation();
}
+List filteredPartitions = new ArrayList<>();
+if (layout.getPartitionColumns().size() > 0 &&
layout.getPartitions().isPresent()) {
+ List colNames =
+ layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle)
x).getName())
+ .collect(Collectors.toList());
+ for (HivePartition partition : layout.getPartitions().get()) {
+filteredPartitions.add(new PartitionSpec(colNames,
Review comment:
I have verified. This scenario can work. I guess. No need to modify any
code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r485642687 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java ## @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.presto.readers; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.Optional; + +import com.facebook.presto.spi.block.ArrayBlock; +import com.facebook.presto.spi.block.RowBlock; +import com.facebook.presto.spi.type.*; + +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.metadata.datatype.DataTypes; +import org.apache.carbondata.core.metadata.datatype.StructField; +import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector; +import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl; + +import com.facebook.presto.spi.block.Block; +import com.facebook.presto.spi.block.BlockBuilder; + +import org.apache.carbondata.presto.CarbonVectorBatch; +import org.apache.carbondata.presto.ColumnarVectorWrapperDirect; + +/** + * Class to read the complex type Stream [array/struct/map] + */ + +public class ComplexTypeStreamReader extends CarbonColumnVectorImpl Review comment: Not a bug fix, you need to know how prestodb and prestosql is divided. there are some common files and individual files. This class is individual class There is a complexStreamReader in prestoSql also. same code but import packages are from prestosql. here, import packages are from prestodb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r485642687 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java ## @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.presto.readers; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.Optional; + +import com.facebook.presto.spi.block.ArrayBlock; +import com.facebook.presto.spi.block.RowBlock; +import com.facebook.presto.spi.type.*; + +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.metadata.datatype.DataTypes; +import org.apache.carbondata.core.metadata.datatype.StructField; +import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector; +import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl; + +import com.facebook.presto.spi.block.Block; +import com.facebook.presto.spi.block.BlockBuilder; + +import org.apache.carbondata.presto.CarbonVectorBatch; +import org.apache.carbondata.presto.ColumnarVectorWrapperDirect; + +/** + * Class to read the complex type Stream [array/struct/map] + */ + +public class ComplexTypeStreamReader extends CarbonColumnVectorImpl Review comment: Not a bug fix, you need to know how prestodb and prestosql is divided. there are some common files and individual files. This class is individual class There is a complexStreamReader in prestoSql also. same code but import packages are from prestosql. import packages are from prestodb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485640004
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##
@@ -117,6 +122,16 @@ public ConnectorSplitSource
getSplits(ConnectorTransactionHandle transactionHand
// file metastore case tablePath can be null, so get from location
location = table.getStorage().getLocation();
}
+List filteredPartitions = new ArrayList<>();
+if (layout.getPartitionColumns().size() > 0 &&
layout.getPartitions().isPresent()) {
+ List colNames =
+ layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle)
x).getName())
+ .collect(Collectors.toList());
+ for (HivePartition partition : layout.getPartitions().get()) {
+filteredPartitions.add(new PartitionSpec(colNames,
Review comment:
I think HiveTableHandle will give matching column for the partitions
only , because this is per query derived. Let me retest and get back to you
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485640004
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##
@@ -117,6 +122,16 @@ public ConnectorSplitSource
getSplits(ConnectorTransactionHandle transactionHand
// file metastore case tablePath can be null, so get from location
location = table.getStorage().getLocation();
}
+List filteredPartitions = new ArrayList<>();
+if (layout.getPartitionColumns().size() > 0 &&
layout.getPartitions().isPresent()) {
+ List colNames =
+ layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle)
x).getName())
+ .collect(Collectors.toList());
+ for (HivePartition partition : layout.getPartitions().get()) {
+filteredPartitions.add(new PartitionSpec(colNames,
Review comment:
I think HiveTableHandle will give matching column for the partitions
only , because this is per query derived.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r485636502 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java ## @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.presto.readers; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.Optional; + +import com.facebook.presto.spi.block.ArrayBlock; +import com.facebook.presto.spi.block.RowBlock; +import com.facebook.presto.spi.type.*; + +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.metadata.datatype.DataTypes; +import org.apache.carbondata.core.metadata.datatype.StructField; +import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector; +import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl; + +import com.facebook.presto.spi.block.Block; +import com.facebook.presto.spi.block.BlockBuilder; + +import org.apache.carbondata.presto.CarbonVectorBatch; +import org.apache.carbondata.presto.ColumnarVectorWrapperDirect; + +/** + * Class to read the complex type Stream [array/struct/map] + */ + +public class ComplexTypeStreamReader extends CarbonColumnVectorImpl Review comment: Then I have to raise partition changes only for prestosql, because prestodb profile is not compiling. or I fix the prestodb compile first. you can merge it. I will raise separate PR for partition. what you suggest ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485635431
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##
@@ -117,6 +122,16 @@ public ConnectorSplitSource
getSplits(ConnectorTransactionHandle transactionHand
// file metastore case tablePath can be null, so get from location
location = table.getStorage().getLocation();
}
+List filteredPartitions = new ArrayList<>();
+if (layout.getPartitionColumns().size() > 0 &&
layout.getPartitions().isPresent()) {
+ List colNames =
+ layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle)
x).getName())
+ .collect(Collectors.toList());
+ for (HivePartition partition : layout.getPartitions().get()) {
+filteredPartitions.add(new PartitionSpec(colNames,
Review comment:
you mean all the partition column right ? not the all table columns ?
Ideally they use partitionPath. I have tested it. It didn't impacted. Let me
check again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r485628731 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataModule.java ## @@ -21,6 +21,8 @@ import static java.util.Objects.requireNonNull; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.util.CarbonProperties; Review comment: prestodb compile was broken by #3885 , They have copied code from prestosql but not import statement. so, fixed it. Mentioned in PR description also This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r485628731 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataModule.java ## @@ -21,6 +21,8 @@ import static java.util.Objects.requireNonNull; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.util.CarbonProperties; Review comment: prestodb compile was broken by #3885 , They have copied code but not import statement. so, fixed it. Mentioned in PR description also This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r484845973 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java ## @@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand // file metastore case tablePath can be null, so get from location location = table.getStorage().getLocation(); } +List filteredPartitions = new ArrayList<>(); Review comment: please read the description, I have already mentioned why UT cannot be added now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3913: [CARBONDATA-3974] Improve partition purning performance in presto carbon integration
ajantha-bhat commented on a change in pull request #3913: URL: https://github.com/apache/carbondata/pull/3913#discussion_r484845973 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java ## @@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand // file metastore case tablePath can be null, so get from location location = table.getStorage().getLocation(); } +List filteredPartitions = new ArrayList<>(); Review comment: please read the description, I have mentioned why UT cannot be added now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
