>From Ritik Raj <[email protected]>:
Ritik Raj has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20943?usp=email )
Change subject: [ASTERIXDB-3708][COMP] Fix empty results with columnar
secondary index, LIMIT, and field projection
......................................................................
[ASTERIXDB-3708][COMP] Fix empty results with columnar secondary index, LIMIT,
and field projection
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
When querying a columnar dataset with a secondary index, LIMIT clause,
and field projection (e.g., SELECT VALUE h FROM d1 WHERE
cac_email_domain > 3 LIMIT 2), the query returns empty results with
certain execution sequences.
Root cause: PushLimitIntoPrimarySearchRule moves the SELECT condition
into the UNNEST_MAP operator before field access pushdown.
PushdownOperatorVisitor was calling visitInputs BEFORE registering the
dataset for UnnestMapOperator and LeftOuterUnnestMapOperator. This
timing issue prevented filter fields from being captured in the
def-use chain, so the columnar projection schema only included output
fields but not filter fields.
Fix: Changed PushdownOperatorVisitor to call registerDatasetIfApplicable
BEFORE visitInputs for UnnestMapOperator and LeftOuterUnnestMapOperator,
aligning with the existing pattern for DataSourceScanOperator. This
ensures filter field accesses are properly linked to the scan definition,
causing the projection schema to include both output and filter fields.
Added regression test in column/secondary-index/projection-filter/ to
validate the fix.
Ext-ref: MB-70725
Change-Id: I26e5357141ff77ee210f89b4e93e1e4c33eb6845
---
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/visitor/PushdownOperatorVisitor.java
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.001.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.002.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.003.get.http
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.005.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.006.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.007.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.003.regexadm
A
asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.006.adm
A
asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.007.adm
M
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_single_partition_sqlpp.xml
11 files changed, 147 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/43/20943/1
diff --git
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/visitor/PushdownOperatorVisitor.java
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/visitor/PushdownOperatorVisitor.java
index c425d72..7878ebf 100644
---
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/visitor/PushdownOperatorVisitor.java
+++
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/visitor/PushdownOperatorVisitor.java
@@ -175,9 +175,9 @@
*/
@Override
public Void visitUnnestMapOperator(UnnestMapOperator op, Void arg) throws
AlgebricksException {
- visitInputs(op);
DatasetDataSource datasetDataSource =
getDatasetDataSourceIfApplicable(getDataSourceFromUnnestMapOperator(op));
registerDatasetIfApplicable(datasetDataSource, op);
+ visitInputs(op);
return null;
}
@@ -187,9 +187,9 @@
*/
@Override
public Void visitLeftOuterUnnestMapOperator(LeftOuterUnnestMapOperator op,
Void arg) throws AlgebricksException {
- visitInputs(op);
DatasetDataSource datasetDataSource =
getDatasetDataSourceIfApplicable(getDataSourceFromUnnestMapOperator(op));
registerDatasetIfApplicable(datasetDataSource, op);
+ visitInputs(op);
return null;
}
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.001.ddl.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.001.ddl.sqlpp
new file mode 100644
index 0000000..04c91b6
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.001.ddl.sqlpp
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+DROP DATAVERSE projection_filter IF EXISTS;
+CREATE DATAVERSE projection_filter;
+USE projection_filter;
+
+CREATE TYPE intType AS {
+ id: int
+};
+
+CREATE DATASET d1(intType) PRIMARY KEY id WITH {
+ "storage-format": {"format": "column"}
+};
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.002.update.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.002.update.sqlpp
new file mode 100644
index 0000000..ad04eae
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.002.update.sqlpp
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+USE projection_filter;
+
+INSERT INTO d1 {"id":2,"h":44,"cac_email_domain":66};
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.003.get.http
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.003.get.http
new file mode 100644
index 0000000..eef4e41
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.003.get.http
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/connector?dataverseName=projection_filter&datasetName=d1
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.005.ddl.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.005.ddl.sqlpp
new file mode 100644
index 0000000..00df2b7
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.005.ddl.sqlpp
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF any
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+USE projection_filter;
+
+CREATE INDEX idx_cac_email_domain ON d1(cac_email_domain);
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.006.query.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.006.query.sqlpp
new file mode 100644
index 0000000..b30c14f
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.006.query.sqlpp
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+USE projection_filter;
+
+SELECT VALUE h FROM d1 WHERE cac_email_domain > 3;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.007.query.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.007.query.sqlpp
new file mode 100644
index 0000000..4d65bc3
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/secondary-index/projection-filter/projection-filter.007.query.sqlpp
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+USE projection_filter;
+
+SELECT VALUE h FROM d1 WHERE cac_email_domain > 3 LIMIT 2;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.003.regexadm
b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.003.regexadm
new file mode 100644
index 0000000..78194e6
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.003.regexadm
@@ -0,0 +1 @@
+\Q{"keys":"id","type":{"type":"org.apache.asterix.om.types.ARecordType","name":"intType","open":true,"fields":[{"id":{"type":"AInt64"}}]},"splits":[\E.*\Q]}\E
\ No newline at end of file
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.006.adm
b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.006.adm
new file mode 100644
index 0000000..d2e1cef
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.006.adm
@@ -0,0 +1 @@
+44
\ No newline at end of file
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.007.adm
b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.007.adm
new file mode 100644
index 0000000..d2e1cef
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/secondary-index/projection-filter/projection-filter.007.adm
@@ -0,0 +1 @@
+44
\ No newline at end of file
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_single_partition_sqlpp.xml
b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_single_partition_sqlpp.xml
index bac0fb4..1b41311 100644
---
a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_single_partition_sqlpp.xml
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_single_partition_sqlpp.xml
@@ -411,6 +411,11 @@
</compilation-unit>
</test-case>
<test-case FilePath="column">
+ <compilation-unit name="secondary-index/projection-filter">
+ <output-dir
compare="Text">secondary-index/projection-filter</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="column">
<compilation-unit name="secondary-index/array-index/use-case-1">
<output-dir
compare="Text">secondary-index/array-index/use-case-1</output-dir>
</compilation-unit>
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20943?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I26e5357141ff77ee210f89b4e93e1e4c33eb6845
Gerrit-Change-Number: 20943
Gerrit-PatchSet: 1
Gerrit-Owner: Ritik Raj <[email protected]>
