[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803315#comment-17803315 ] ASF GitHub Bot commented on DRILL-8375: --- paul-rogers opened a new pull request, #2867: URL: https://github.com/apache/drill/pull/2867 # Support for non-projected complex vectors ## Description The EVF mechanism provides scan-time projection for many vector types. The reader code is simple: it deserializes all columns for formats such as JSON, CSV, etc., and writes them to the `ColumnWriter` objects. Internally, EVF simply ignores the data for unprojected columns. This solution simplifies the readers: it is not necessary for each reader to include the complex code to handle projection. This solution is also performant: projection is done at scan time rather than the other approach, which is to read all data into vectors, then allow a PROJECT operator to drop the unprojected columns. Present EVF projection support handles most scalar and "well-structured" columns (repeated types AKA arrays, maps, etc.) However it does not handle the more esoteric types UNION, LIST (AKA repeated UNION), REPEATED LIST (AKA repeated, repeated UNION). This PR provides more support, though holes remain. ## Documentation This is an internal feature: no user-visible documentation is required. ## Testing Extended existing EVF-related unit tests. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803893#comment-17803893 ] ASF GitHub Bot commented on DRILL-8375: --- cgivre commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1879917039 Once we merge this, we should also rebase https://github.com/apache/drill/pull/2515 on the current master and merge that as well. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803992#comment-17803992 ] ASF GitHub Bot commented on DRILL-8375: --- paul-rogers commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1880146814 Thanks @cgivre for the comments and review. @luocooong, I'll commit this. When convenient, please see if this addresses the issue you raised long ago. Otherwise, these capabilities are available for the next person who is seduced into trying out the UNION-based types. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803993#comment-17803993 ] ASF GitHub Bot commented on DRILL-8375: --- paul-rogers merged PR #2867: URL: https://github.com/apache/drill/pull/2867 > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804050#comment-17804050
]
ASF GitHub Bot commented on DRILL-8375:
---
luocooong commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444101786
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
This test case provides a good use guide. In addition, will it be possible
for us to remove Union completely from the data type in the future?
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804074#comment-17804074
]
ASF GitHub Bot commented on DRILL-8375:
---
paul-rogers commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444117648
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
The UNION type has been discussed for as long as I've been involved in the
project: since 2016. The idea is simple: Drill should be able to read any kind
of JSON, and UNION (plus LIST, etc.) have been essential for this. The problem,
as we've also discussed for a long time, is that UNION barely works, and JDBC
and similar clients can't make sense of it. That is, UNION turned out to be the
wrong solution to the problem.
Still, there is always the hope that UNION can be made to work
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804115#comment-17804115
]
ASF GitHub Bot commented on DRILL-8375:
---
jnturton commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444180526
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
Thanks for this Paul! We must remember to squash when merging, we got the
WIP commits from feature branch into master. We've all done something similar
at some point
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804116#comment-17804116
]
ASF GitHub Bot commented on DRILL-8375:
---
jnturton commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444180526
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
Thanks for this Paul! We must remember to squash when merging, we got the
WIP commits from the feature branch into master. We've all done something
similar at some point
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804117#comment-17804117
]
ASF GitHub Bot commented on DRILL-8375:
---
cgivre commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444180941
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
> Thanks for this Paul! We must remember to squash when merging, we got the
WIP commits from the feature branch into master. We've all done something
similar at some point
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804123#comment-17804123
]
ASF GitHub Bot commented on DRILL-8375:
---
jnturton commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444185697
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
P.S. I'm happy to live with the WIP commits in master too, just asking what
folks think.
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804137#comment-17804137 ] ASF GitHub Bot commented on DRILL-8375: --- jnturton commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1880450409 I think we can regard this as a bug fix to framework code already present in 1.21 and therefore backport it. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.21.2 > > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804151#comment-17804151 ] ASF GitHub Bot commented on DRILL-8375: --- paul-rogers commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1880502847 Backporting should be safe: as safe as having the change in master itself. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.21.2 > > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804150#comment-17804150
]
ASF GitHub Bot commented on DRILL-8375:
---
paul-rogers commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444244003
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
My bad. My other project likes to leave these in master; I forgot Drill does
not.
Since there is not much activity, I can squash the commits within the master
branch and do a force push. Normally that is a big NO NO in active projects,
but it should not actually cause problems here. I'll go ahead and do that
tomorrow unless anyone objects.
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.21.2
>
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804270#comment-17804270
]
ASF GitHub Bot commented on DRILL-8375:
---
jnturton commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444618000
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
@paul-rogers, @cgivre [commented that he's in favour of leaving master as
is](https://github.com/apache/drill/pull/2866#issuecomment-1880409413) and I've
since merged [a commit on
top](https://github.com/apache/drill/commit/f5fb7f5a4023651252afb1f907311d71840eb144).
I do think it would still be feasible for us to go back and squash (for
exactly the reason you give) but at this point we could also just leave it
where it is?
P.S. The process is a little laborious. A conventional commit switching off
master branch protection in .asf.yaml, then the force push doing the clean up
and switching master branch protection back on.
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.21.2
>
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804276#comment-17804276
]
ASF GitHub Bot commented on DRILL-8375:
---
jnturton commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444618000
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
@paul-rogers, @cgivre [commented that he's in favour of leaving master as
is](https://github.com/apache/drill/pull/2866#issuecomment-1880409413) and I've
since merged [a commit on
top](https://github.com/apache/drill/commit/f5fb7f5a4023651252afb1f907311d71840eb144).
I do think it would still be feasible for us to go back and squash (for
exactly the reason you give) but at this point we could also just leave it
where it is?
P.S. The process is a little laborious. A conventional commit switching off
master branch protection in .asf.yaml, then the force push doing the clean up
and switching master branch protection back on, the latter being achievable in
the same breath by simply dropping the switch-on commit.
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.21.2
>
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[
https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804317#comment-17804317
]
ASF GitHub Bot commented on DRILL-8375:
---
cgivre commented on code in PR #2867:
URL: https://github.com/apache/drill/pull/2867#discussion_r1444727956
##
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/TestResultSetLoaderUnprojected.java:
##
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.ResultSetOptions;
+import org.apache.drill.exec.physical.resultSet.project.Projections;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetTestUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.test.SubOperatorTest;
+import org.apache.drill.test.rowSet.RowSetUtilities;
+import org.junit.Test;
+
+/**
+ * Verify the correct functioning of the "dummy" columns created
+ * for unprojected columns.
+ */
+public class TestResultSetLoaderUnprojected extends SubOperatorTest {
Review Comment:
Just to be clear... I was just saying that if this is a major headache and
you don't want to deal with it, my vote is to leave it alone. If it isn't a
big headache and you want to, I have no issues there as well.
> Incomplete support for non-projected complex vectors
>
>
> Key: DRILL-8375
> URL: https://issues.apache.org/jira/browse/DRILL-8375
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.21.2
>
>
> The `ResultSetLoader` implementation supports all of Drill's vector types.
> However, DRILL-8188 discovered holes in support for non-projected vectors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804507#comment-17804507 ] ASF GitHub Bot commented on DRILL-8375: --- paul-rogers commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1881965757 I successfully squashed the commits, and provided a proper commit message, while preserving the later commit. Did a force push to master to rewrite history. You should update your own master to pick up the revised history. @cgivre, the `.asf.yaml` file you mentioned has lots of metadata, but does not actually prevent a force push. Perhaps we are missing something? It would generally be a good idea to forbid such things to prevent catastrophic mistakes. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.21.2 > > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804546#comment-17804546 ] ASF GitHub Bot commented on DRILL-8375: --- cgivre commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1882148031 > I successfully squashed the commits, and provided a proper commit message, while preserving the later commit. Did a force push to master to rewrite history. > > You should update your own master to pick up the revised history. You are a braver man than I. > > @cgivre, the `.asf.yaml` file you mentioned has lots of metadata, but does not actually prevent a force push. Perhaps we are missing something? It would generally be a good idea to forbid such things to prevent catastrophic mistakes. Thanks for flagging... I'm not sure how to do that, but I'll investigate. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.21.2 > > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804572#comment-17804572 ] ASF GitHub Bot commented on DRILL-8375: --- jnturton commented on PR #2867: URL: https://github.com/apache/drill/pull/2867#issuecomment-1882408865 > @cgivre, the `.asf.yaml` file you mentioned has lots of metadata, but does not actually prevent a force push. Perhaps we are missing something? It would generally be a good idea to forbid such things to prevent catastrophic mistakes. Oh that's interesting. Something's changed since I last went through this with @vvysotskyi to do something that could only be done on master, perhaps testing of the automatic snapshot artifact publishing which requires access to GitHub Actions secrets. > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.21.2 > > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
