Hi Igor, Thanks for responding. I'm not sure if this is what you intended, but looked at the JSON for the Query plans and found something interesting. For the SELECT * query, I'm getting the following when I try to run the physical plan that it generates (without modification). Do you think this could be a related problem?
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: InvalidDefinitionException: Cannot construct instance of `org.apache.drill.exec.store.http.HttpGroupScan` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator) at [Source: (String)"{ "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" : "" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, "hasResourcePlan" : false, "resultMode" : "EXEC" }, "graph" : [ { "pop" : "http-scan", "@id" : 2, "scanSpec" : { "uri" : "/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02" }, "columns" : [ "`**`" ], "storageConfig" : { "type" : "http", "[truncated 766 chars]; line: 16, column: 5] (through reference chain: org.apache.drill.exec.physical.PhysicalPlan["graph"]->java.util.ArrayList[0]) Please, refer to logs for more information. [Error Id: 751b6d05-a631-4eca-9d83-162ab4fa839f on localhost:31010] > On Oct 11, 2019, at 12:25 PM, Igor Guzenko <ihor.huzenko....@gmail.com> wrote: > > Hello Charles, > > You got the error from Apache Calcite at the planning stage while > converting SQLIdentifier to RexNode. From your stack trace the conversion > starts here DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685) and > goes to SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694). I > would suggest to save json returned by rest locally as file and debug same > trace for query on the json file. So then you can find difference between > conversion of sql identifier to rel for standart json reading and for your > storage plugin. > > Thanks, Igor > > > On Fri, Oct 11, 2019 at 6:34 PM Charles Givre <cgi...@gmail.com > <mailto:cgi...@gmail.com>> wrote: > >> Hello all, >> I decided to take the leap and attempt to implement a storage plugin. I >> found that a few people had started this, so I thought I'd complete a >> simple generic HTTP/REST storage plugin. The use case would be to enrich >> data sets with data that's available via public or internal APIs. >> >> Anyway, I'm a little stuck and need some assistance. i got the plugin to >> successfully execute a star query and return the results correctly: >> >> apache drill> SELECT * FROM >> http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`; >> >> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+ >> | sunrise | sunset | solar_noon | day_length | >> civil_twilight_begin | civil_twilight_end | nautical_twilight_begin | >> nautical_twilight_end | astronomical_twilight_begin | >> astronomical_twilight_end | >> >> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+ >> | 6:13:58 AM | 5:59:55 PM | 12:06:56 PM | 11:45:57 | 5:48:14 AM >> | 6:25:38 PM | 5:18:16 AM | 6:55:36 PM | >> 4:48:07 AM | 7:25:45 PM | >> >> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+ >> 1 row selected (0.392 seconds) >> >> However, when I attempt to select individual fields i get errors. (see >> below for full stack trace). I've walked through this with the debugger, >> but it seems like the code is breaking before it hits my storage plugin and >> I'm not sure what to do about it. Here's a link to the code: >> https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http < >> https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http >> <https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http>> >> >> Any assistance would be greatly appreciated. Thanks!! >> >> >> >> apache drill> !verbose >> verbose: on >> apache drill> SELECT sunset FROM >> http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`; >> Error: SYSTEM ERROR: AssertionError: Field ordinal 1 is invalid for type >> '(DrillRecordRow[**])' >> >> >> Please, refer to logs for more information. >> >> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010] >> (state=,code=0) >> java.sql.SQLException: SYSTEM ERROR: AssertionError: Field ordinal 1 is >> invalid for type '(DrillRecordRow[**])' >> >> >> Please, refer to logs for more information. >> >> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010] >> at >> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:538) >> at >> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:610) >> at >> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278) >> at >> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58) >> at >> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667) >> at >> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102) >> at >> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113) >> at >> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) >> at >> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200) >> at >> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) >> at >> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217) >> at sqlline.Commands.executeSingleQuery(Commands.java:1008) >> at sqlline.Commands.execute(Commands.java:957) >> at sqlline.Commands.sql(Commands.java:921) >> at sqlline.SqlLine.dispatch(SqlLine.java:717) >> at sqlline.SqlLine.begin(SqlLine.java:536) >> at sqlline.SqlLine.start(SqlLine.java:266) >> at sqlline.SqlLine.main(SqlLine.java:205) >> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM >> ERROR: AssertionError: Field ordinal 1 is invalid for type >> '(DrillRecordRow[**])' >> >> >> Please, refer to logs for more information. >> >> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010] >> at >> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123) >> at >> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) >> at >> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) >> at >> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) >> at >> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) >> at java.lang.Thread.run(Thread.java:748) >> Caused by: org.apache.drill.exec.work.foreman.ForemanException: >> Unexpected exception during fragment initialization: Field ordinal 1 is >> invalid for type '(DrillRecordRow[**])' >> at org.apache.drill.exec.work >> .foreman.Foreman.run(Foreman.java:303) >> at .......(:0) >> Caused by: java.lang.AssertionError: Field ordinal 1 is invalid for type >> '(DrillRecordRow[**])' >> at >> org.apache.calcite.rex.RexBuilder.makeFieldAccess(RexBuilder.java:197) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.access$2200(SqlToRelConverter.java:217) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4765) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061) >> at >> org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4625) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3908) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:670) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3150) >> at >> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563) >> at >> org.apache.drill.exec.planner.sql.SqlConverter.toRel(SqlConverter.java:414) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:202) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:172) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90) >> at org.apache.drill.exec.work >> .foreman.Foreman.runSQL(Foreman.java:591) >> at org.apache.drill.exec.work >> .foreman.Foreman.run(Foreman.java:276) >> ... 1 more >> apache drill>