[
https://issues.apache.org/jira/browse/DRILL-7502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007930#comment-17007930
]
ASF GitHub Bot commented on DRILL-7502:
---------------------------------------
paul-rogers commented on issue #1945: DRILL-7502: Invalid codegen for typeof()
with UNION
URL: https://github.com/apache/drill/pull/1945#issuecomment-570761952
@vvysotskyi, @ihuzenko thank you for your reviews. I believe I've addressed
your comments. Please take another look.
This commit contains some significant changes. As it turns out, we have
three different ways we work with Union vectors in code gen. Two worked, the
third did not. Figuring that out was a bit of an involved exercise. I added
quite a few comments and split up some overly-large methods, to help gain
understanding. I added a bit of a hack to work around the missing case. It
works; all tests pass including the new one that initially failed. (And
DRILL-7510 got magically fixed along the way.)
I say the fix is a "hack" because I had to add more state: to access the
Union vector when setting up parameters that use a `FieldReader`. It seems
there should be a simpler way; but I could not find it.
@KazydubB, these changes touched some of the same areas you apparently
touched for your `DICT` work. Can you take a look to 1) make sure I didn't
accidentally break anything, and 2) perhaps see the pattern I may be missing
about how to set up a `FieldReader` for a Union vector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Incorrect/invalid codegen for typeof() with UNION
> -------------------------------------------------
>
> Key: DRILL-7502
> URL: https://issues.apache.org/jira/browse/DRILL-7502
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.17.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Major
> Fix For: 1.18.0
>
>
> The {{typeof()}} function is defined as follows:
> {code:java}
> @FunctionTemplate(names = {"typeOf"},
> scope = FunctionTemplate.FunctionScope.SIMPLE,
> nulls = NullHandling.INTERNAL)
> public static class GetType implements DrillSimpleFunc {
> @Param
> FieldReader input;
> @Output
> VarCharHolder out;
> @Inject
> DrillBuf buf;
> @Override
> public void setup() {}
> @Override
> public void eval() {
> String typeName = input.getTypeString();
> byte[] type = typeName.getBytes();
> buf = buf.reallocIfNeeded(type.length);
> buf.setBytes(0, type);
> out.buffer = buf;
> out.start = 0;
> out.end = type.length;
> }
> }
> {code}
> Note that the {{input}} field is defined as {{FieldReader}} which has a
> method called {{getTypeString()}}. As a result, the code works fine in all
> existing tests in {{TestTypeFns}}.
> I tried to add a function to use {{typeof()}} on a column of type {{UNION}}.
> When I did, the query failed with a compile error in generated code:
> {noformat}
> SYSTEM ERROR: CompileException: Line 42, Column 43:
> A method named "getTypeString" is not declared in any enclosing class nor
> any supertype, nor through a static import
> {noformat}
> The stack trace shows the generated code; Note that the type of {{input}}
> changes from a reader to a holder, causing code to be invalid:
> {code:java}
> public class ProjectorGen0 {
> DrillBuf work0;
> UnionVector vv1;
> VarCharVector vv6;
> DrillBuf work9;
> VarCharVector vv11;
> DrillBuf work14;
> VarCharVector vv16;
> public void doEval(int inIndex, int outIndex)
> throws SchemaChangeException
> {
> {
> UnionHolder out4 = new UnionHolder();
> {
> out4 .isSet = vv1 .getAccessor().isSet((inIndex));
> if (out4 .isSet == 1) {
> vv1 .getAccessor().get((inIndex), out4);
> }
> }
> //---- start of eval portion of typeOf function. ----//
> VarCharHolder out5 = new VarCharHolder();
> {
> final VarCharHolder out = new VarCharHolder();
> UnionHolder input = out4;
> DrillBuf buf = work0;
> UnionFunctions$GetType_eval:
> {
> String typeName = input.getTypeString();
> byte[] type = typeName.getBytes();
> buf = buf.reallocIfNeeded(type.length);
> buf.setBytes(0, type);
> out.buffer = buf;
> out.start = 0;
> out.end = type.length;
> }
> {code}
> By contrast, here is the generated code for one of the existing
> {{TestTypeFns}} tests where things work:
> {code:java}
> public class ProjectorGen0
> extends ProjectorTemplate
> {
> DrillBuf work0;
> NullableBigIntVector vv1;
> VarCharVector vv7;
> public ProjectorGen0() {
> try {
> __DRILL_INIT__();
> } catch (SchemaChangeException e) {
> throw new UnsupportedOperationException(e);
> }
> }
> public void doEval(int inIndex, int outIndex)
> throws SchemaChangeException
> {
> {
> ..
> //---- start of eval portion of typeOf function. ----//
> VarCharHolder out6 = new VarCharHolder();
> {
> final VarCharHolder out = new VarCharHolder();
> FieldReader input = new NullableIntHolderReaderImpl(out5);
> DrillBuf buf = work0;
> UnionFunctions$GetType_eval:
> {
> String typeName = input.getTypeString();
> byte[] type = typeName.getBytes();
> buf = buf.reallocIfNeeded(type.length);
> buf.setBytes(0, type);
> out.buffer = buf;
> out.start = 0;
> out.end = type.length;
> }
> work0 = buf;
> out6 .start = out.start;
> out6 .end = out.end;
> out6 .buffer = out.buffer;
> }
> //---- end of eval portion of typeOf function. ----//
> {code}
> Notice that the {{input}} variable is of type {{FieldReader}} as expected.
> Queries that work:
> {code:java}
> String sql = "SELECT typeof(CAST(a AS " + castType + ")) FROM (VALUES
> (1)) AS T(a)";
> sql = "SELECT typeof(CAST(a AS " + castType + ")) FROM
> cp.`functions/null.json`";
> String sql = "SELECT typeof(" + expr + ") FROM (VALUES (" + value + "))
> AS T(a)";
> {code}
> Query that fails:
> {code:java}
> String sql ="SELECT typeof(a) AS t, modeof(a) as m, drilltypeof(a) AS
> dt\n" +
> "FROM cp.`jsoninput/union/c.json`";
> {code}
> The queries that work all include either a CAST or constant values. The query
> that fails works with data read from a file. Also, the queries that work use
> scalar types, the query that fails uses the UNION type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
