Hello Charles, You got the error from Apache Calcite at the planning stage while converting SQLIdentifier to RexNode. From your stack trace the conversion starts here DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685) and goes to SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694). I would suggest to save json returned by rest locally as file and debug same trace for query on the json file. So then you can find difference between conversion of sql identifier to rel for standart json reading and for your storage plugin.
Thanks, Igor On Fri, Oct 11, 2019 at 6:34 PM Charles Givre <[email protected]> wrote: > Hello all, > I decided to take the leap and attempt to implement a storage plugin. I > found that a few people had started this, so I thought I'd complete a > simple generic HTTP/REST storage plugin. The use case would be to enrich > data sets with data that's available via public or internal APIs. > > Anyway, I'm a little stuck and need some assistance. i got the plugin to > successfully execute a star query and return the results correctly: > > apache drill> SELECT * FROM > http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`; > > +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+ > | sunrise | sunset | solar_noon | day_length | > civil_twilight_begin | civil_twilight_end | nautical_twilight_begin | > nautical_twilight_end | astronomical_twilight_begin | > astronomical_twilight_end | > > +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+ > | 6:13:58 AM | 5:59:55 PM | 12:06:56 PM | 11:45:57 | 5:48:14 AM > | 6:25:38 PM | 5:18:16 AM | 6:55:36 PM | > 4:48:07 AM | 7:25:45 PM | > > +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+ > 1 row selected (0.392 seconds) > > However, when I attempt to select individual fields i get errors. (see > below for full stack trace). I've walked through this with the debugger, > but it seems like the code is breaking before it hits my storage plugin and > I'm not sure what to do about it. Here's a link to the code: > https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http < > https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http> > > Any assistance would be greatly appreciated. Thanks!! > > > > apache drill> !verbose > verbose: on > apache drill> SELECT sunset FROM > http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`; > Error: SYSTEM ERROR: AssertionError: Field ordinal 1 is invalid for type > '(DrillRecordRow[**])' > > > Please, refer to logs for more information. > > [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010] > (state=,code=0) > java.sql.SQLException: SYSTEM ERROR: AssertionError: Field ordinal 1 is > invalid for type '(DrillRecordRow[**])' > > > Please, refer to logs for more information. > > [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010] > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:538) > at > org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:610) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58) > at > org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667) > at > org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102) > at > org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113) > at > org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) > at > org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217) > at sqlline.Commands.executeSingleQuery(Commands.java:1008) > at sqlline.Commands.execute(Commands.java:957) > at sqlline.Commands.sql(Commands.java:921) > at sqlline.SqlLine.dispatch(SqlLine.java:717) > at sqlline.SqlLine.begin(SqlLine.java:536) > at sqlline.SqlLine.start(SqlLine.java:266) > at sqlline.SqlLine.main(SqlLine.java:205) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: AssertionError: Field ordinal 1 is invalid for type > '(DrillRecordRow[**])' > > > Please, refer to logs for more information. > > [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123) > at > org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) > at > org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.drill.exec.work.foreman.ForemanException: > Unexpected exception during fragment initialization: Field ordinal 1 is > invalid for type '(DrillRecordRow[**])' > at org.apache.drill.exec.work > .foreman.Foreman.run(Foreman.java:303) > at .......(:0) > Caused by: java.lang.AssertionError: Field ordinal 1 is invalid for type > '(DrillRecordRow[**])' > at > org.apache.calcite.rex.RexBuilder.makeFieldAccess(RexBuilder.java:197) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694) > at > org.apache.calcite.sql2rel.SqlToRelConverter.access$2200(SqlToRelConverter.java:217) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4765) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061) > at > org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4625) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3908) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:670) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3150) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563) > at > org.apache.drill.exec.planner.sql.SqlConverter.toRel(SqlConverter.java:414) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:202) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:172) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90) > at org.apache.drill.exec.work > .foreman.Foreman.runSQL(Foreman.java:591) > at org.apache.drill.exec.work > .foreman.Foreman.run(Foreman.java:276) > ... 1 more > apache drill> > > > >
