Hello Charles,
Looks like you found another new issue. Maybe I explained unclear, but my
previous suggestion wasn't about EXPLAIN PLAN construct, but rather:
1) Use http client like Postman or simply browser to save response of
requested rest service into json file
2) Try to debug reading the file by Drill in order to compare how
Calcite's conversion from AST SqlNode to RelNode tree differs for existing
dfs storage plugin from same flow in your storage plugin.
>From your last email I can figure out that exists another issue with class
HttpGroupScan, at some point Drill tried to deserialize json into instance
of HttpGroupScan and jackson library didn't find how to do this. Probably
you missed some constructor with jackson metadata, for example see in
HiveScan operator:
@JsonCreator
public HiveScan(@JsonProperty("userName") final String userName,
@JsonProperty("hiveReadEntry") final HiveReadEntry
hiveReadEntry,
@JsonProperty("hiveStoragePluginConfig") final
HiveStoragePluginConfig hiveStoragePluginConfig,
@JsonProperty("columns") final List<SchemaPath> columns,
@JsonProperty("confProperties") final Map<String,
String> confProperties,
@JacksonInject final StoragePluginRegistry
pluginRegistry) throws ExecutionSetupException {
this(userName,
hiveReadEntry,
(HiveStoragePlugin) pluginRegistry.getPlugin(hiveStoragePluginConfig),
columns,
null, confProperties);
}
Kind regards,
Igor
On Fri, Oct 11, 2019 at 10:53 PM Charles Givre <cgi...@gmail.com> wrote:
> Hi Igor,
> Thanks for responding. I'm not sure if this is what you intended, but
> looked at the JSON for the Query plans and found something interesting.
> For the SELECT * query, I'm getting the following when I try to run the
> physical plan that it generates (without modification). Do you think this
> could be a related problem?
>
>
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> InvalidDefinitionException: Cannot construct instance of
> `org.apache.drill.exec.store.http.HttpGroupScan` (no Creators, like default
> construct, exist): cannot deserialize from Object value (no delegate- or
> property-based Creator)
> at [Source: (String)"{
> "head" : {
> "version" : 1,
> "generator" : {
> "type" : "ExplainHandler",
> "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ ],
> "queue" : 0,
> "hasResourcePlan" : false,
> "resultMode" : "EXEC"
> },
> "graph" : [ {
> "pop" : "http-scan",
> "@id" : 2,
> "scanSpec" : {
> "uri" : "/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02"
> },
> "columns" : [ "`**`" ],
> "storageConfig" : {
> "type" : "http",
> "[truncated 766 chars]; line: 16, column: 5] (through reference chain:
> org.apache.drill.exec.physical.PhysicalPlan["graph"]->java.util.ArrayList[0])
>
>
> Please, refer to logs for more information.
>
> [Error Id: 751b6d05-a631-4eca-9d83-162ab4fa839f on localhost:31010]
>
>
> > On Oct 11, 2019, at 12:25 PM, Igor Guzenko <ihor.huzenko....@gmail.com>
> wrote:
> >
> > Hello Charles,
> >
> > You got the error from Apache Calcite at the planning stage while
> > converting SQLIdentifier to RexNode. From your stack trace the conversion
> > starts here DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685)
> and
> > goes to
> SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694). I
> > would suggest to save json returned by rest locally as file and debug
> same
> > trace for query on the json file. So then you can find difference between
> > conversion of sql identifier to rel for standart json reading and for
> your
> > storage plugin.
> >
> > Thanks, Igor
> >
> >
> > On Fri, Oct 11, 2019 at 6:34 PM Charles Givre <cgi...@gmail.com <mailto:
> cgi...@gmail.com>> wrote:
> >
> >> Hello all,
> >> I decided to take the leap and attempt to implement a storage plugin. I
> >> found that a few people had started this, so I thought I'd complete a
> >> simple generic HTTP/REST storage plugin. The use case would be to enrich
> >> data sets with data that's available via public or internal APIs.
> >>
> >> Anyway, I'm a little stuck and need some assistance. i got the plugin
> to
> >> successfully execute a star query and return the results correctly:
> >>
> >> apache drill> SELECT * FROM
> >> http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`;
> >>
> >>
> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+
> >> | sunrise | sunset | solar_noon | day_length |
> >> civil_twilight_begin | civil_twilight_end | nautical_twilight_begin |
> >> nautical_twilight_end | astronomical_twilight_begin |
> >> astronomical_twilight_end |
> >>
> >>
> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+
> >> | 6:13:58 AM | 5:59:55 PM | 12:06:56 PM | 11:45:57 | 5:48:14 AM
> >> | 6:25:38 PM | 5:18:16 AM | 6:55:36 PM |
> >> 4:48:07 AM | 7:25:45 PM |
> >>
> >>
> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+
> >> 1 row selected (0.392 seconds)
> >>
> >> However, when I attempt to select individual fields i get errors. (see
> >> below for full stack trace). I've walked through this with the
> debugger,
> >> but it seems like the code is breaking before it hits my storage plugin
> and
> >> I'm not sure what to do about it. Here's a link to the code:
> >> https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http
> <
> >> https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http
> <https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http>>
> >>
> >> Any assistance would be greatly appreciated. Thanks!!
> >>
> >>
> >>
> >> apache drill> !verbose
> >> verbose: on
> >> apache drill> SELECT sunset FROM
> >> http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`;
> >> Error: SYSTEM ERROR: AssertionError: Field ordinal 1 is invalid for
> type
> >> '(DrillRecordRow[**])'
> >>
> >>
> >> Please, refer to logs for more information.
> >>
> >> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010]
> >> (state=,code=0)
> >> java.sql.SQLException: SYSTEM ERROR: AssertionError: Field ordinal 1 is
> >> invalid for type '(DrillRecordRow[**])'
> >>
> >>
> >> Please, refer to logs for more information.
> >>
> >> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010]
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:538)
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:610)
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278)
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58)
> >> at
> >>
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102)
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113)
> >> at
> >>
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> >> at
> >>
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
> >> at
> >>
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> >> at
> >>
> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
> >> at sqlline.Commands.executeSingleQuery(Commands.java:1008)
> >> at sqlline.Commands.execute(Commands.java:957)
> >> at sqlline.Commands.sql(Commands.java:921)
> >> at sqlline.SqlLine.dispatch(SqlLine.java:717)
> >> at sqlline.SqlLine.begin(SqlLine.java:536)
> >> at sqlline.SqlLine.start(SqlLine.java:266)
> >> at sqlline.SqlLine.main(SqlLine.java:205)
> >> Caused by: org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM
> >> ERROR: AssertionError: Field ordinal 1 is invalid for type
> >> '(DrillRecordRow[**])'
> >>
> >>
> >> Please, refer to logs for more information.
> >>
> >> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010]
> >> at
> >>
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> >> at
> >> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> >> at
> >> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> >> at
> >> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> >> at
> >> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> >> at
> >>
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> >> at
> >>
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> >> at
> >>
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> >> at
> >>
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
> >> at
> >>
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> >> at
> >>
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> >> at
> >>
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> >> at
> >>
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> >> at
> >>
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> >> at
> >>
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> >> at
> >>
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
> >> at
> >>
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> >> at
> >>
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> >> at
> >>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> >> at java.lang.Thread.run(Thread.java:748)
> >> Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> >> Unexpected exception during fragment initialization: Field ordinal 1 is
> >> invalid for type '(DrillRecordRow[**])'
> >> at org.apache.drill.exec.work
> >> .foreman.Foreman.run(Foreman.java:303)
> >> at .......(:0)
> >> Caused by: java.lang.AssertionError: Field ordinal 1 is invalid for
> type
> >> '(DrillRecordRow[**])'
> >> at
> >> org.apache.calcite.rex.RexBuilder.makeFieldAccess(RexBuilder.java:197)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.access$2200(SqlToRelConverter.java:217)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4765)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061)
> >> at
> >> org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4625)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3908)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:670)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3150)
> >> at
> >>
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563)
> >> at
> >>
> org.apache.drill.exec.planner.sql.SqlConverter.toRel(SqlConverter.java:414)
> >> at
> >>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685)
> >> at
> >>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:202)
> >> at
> >>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:172)
> >> at
> >>
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226)
> >> at
> >>
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124)
> >> at
> >>
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90)
> >> at org.apache.drill.exec.work
> >> .foreman.Foreman.runSQL(Foreman.java:591)
> >> at org.apache.drill.exec.work
> >> .foreman.Foreman.run(Foreman.java:276)
> >> ... 1 more
> >> apache drill>
>
>
