Hi Igor,
Thanks for responding. I'm not sure if this is what you intended, but looked
at the JSON for the Query plans and found something interesting. For the
SELECT * query, I'm getting the following when I try to run the physical plan
that it generates (without modification). Do you think this could be a related
problem?
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
InvalidDefinitionException: Cannot construct instance of
`org.apache.drill.exec.store.http.HttpGroupScan` (no Creators, like default
construct, exist): cannot deserialize from Object value (no delegate- or
property-based Creator)
at [Source: (String)"{
"head" : {
"version" : 1,
"generator" : {
"type" : "ExplainHandler",
"info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ ],
"queue" : 0,
"hasResourcePlan" : false,
"resultMode" : "EXEC"
},
"graph" : [ {
"pop" : "http-scan",
"@id" : 2,
"scanSpec" : {
"uri" : "/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02"
},
"columns" : [ "`**`" ],
"storageConfig" : {
"type" : "http",
"[truncated 766 chars]; line: 16, column: 5] (through reference chain:
org.apache.drill.exec.physical.PhysicalPlan["graph"]->java.util.ArrayList[0])
Please, refer to logs for more information.
[Error Id: 751b6d05-a631-4eca-9d83-162ab4fa839f on localhost:31010]
> On Oct 11, 2019, at 12:25 PM, Igor Guzenko <[email protected]> wrote:
>
> Hello Charles,
>
> You got the error from Apache Calcite at the planning stage while
> converting SQLIdentifier to RexNode. From your stack trace the conversion
> starts here DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685) and
> goes to SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694). I
> would suggest to save json returned by rest locally as file and debug same
> trace for query on the json file. So then you can find difference between
> conversion of sql identifier to rel for standart json reading and for your
> storage plugin.
>
> Thanks, Igor
>
>
> On Fri, Oct 11, 2019 at 6:34 PM Charles Givre <[email protected]
> <mailto:[email protected]>> wrote:
>
>> Hello all,
>> I decided to take the leap and attempt to implement a storage plugin. I
>> found that a few people had started this, so I thought I'd complete a
>> simple generic HTTP/REST storage plugin. The use case would be to enrich
>> data sets with data that's available via public or internal APIs.
>>
>> Anyway, I'm a little stuck and need some assistance. i got the plugin to
>> successfully execute a star query and return the results correctly:
>>
>> apache drill> SELECT * FROM
>> http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`;
>>
>> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+
>> | sunrise | sunset | solar_noon | day_length |
>> civil_twilight_begin | civil_twilight_end | nautical_twilight_begin |
>> nautical_twilight_end | astronomical_twilight_begin |
>> astronomical_twilight_end |
>>
>> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+
>> | 6:13:58 AM | 5:59:55 PM | 12:06:56 PM | 11:45:57 | 5:48:14 AM
>> | 6:25:38 PM | 5:18:16 AM | 6:55:36 PM |
>> 4:48:07 AM | 7:25:45 PM |
>>
>> +------------+------------+-------------+------------+----------------------+--------------------+-------------------------+-----------------------+-----------------------------+---------------------------+
>> 1 row selected (0.392 seconds)
>>
>> However, when I attempt to select individual fields i get errors. (see
>> below for full stack trace). I've walked through this with the debugger,
>> but it seems like the code is breaking before it hits my storage plugin and
>> I'm not sure what to do about it. Here's a link to the code:
>> https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http <
>> https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http
>> <https://github.com/cgivre/drill/tree/storage-http/contrib/storage-http>>
>>
>> Any assistance would be greatly appreciated. Thanks!!
>>
>>
>>
>> apache drill> !verbose
>> verbose: on
>> apache drill> SELECT sunset FROM
>> http.`/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02`;
>> Error: SYSTEM ERROR: AssertionError: Field ordinal 1 is invalid for type
>> '(DrillRecordRow[**])'
>>
>>
>> Please, refer to logs for more information.
>>
>> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010]
>> (state=,code=0)
>> java.sql.SQLException: SYSTEM ERROR: AssertionError: Field ordinal 1 is
>> invalid for type '(DrillRecordRow[**])'
>>
>>
>> Please, refer to logs for more information.
>>
>> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010]
>> at
>> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:538)
>> at
>> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:610)
>> at
>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278)
>> at
>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58)
>> at
>> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
>> at
>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102)
>> at
>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113)
>> at
>> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>> at
>> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
>> at
>> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>> at
>> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
>> at sqlline.Commands.executeSingleQuery(Commands.java:1008)
>> at sqlline.Commands.execute(Commands.java:957)
>> at sqlline.Commands.sql(Commands.java:921)
>> at sqlline.SqlLine.dispatch(SqlLine.java:717)
>> at sqlline.SqlLine.begin(SqlLine.java:536)
>> at sqlline.SqlLine.start(SqlLine.java:266)
>> at sqlline.SqlLine.main(SqlLine.java:205)
>> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
>> ERROR: AssertionError: Field ordinal 1 is invalid for type
>> '(DrillRecordRow[**])'
>>
>>
>> Please, refer to logs for more information.
>>
>> [Error Id: d7bccd2f-73e6-40d7-9b8a-73a772f65c02 on 192.168.1.21:31010]
>> at
>> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
>> at
>> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
>> at
>> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
>> at
>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
>> at
>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
>> at
>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>> at
>> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>> at
>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>> at
>> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
>> at
>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>> at
>> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>> at
>> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>> at
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>> at
>> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
>> at
>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.drill.exec.work.foreman.ForemanException:
>> Unexpected exception during fragment initialization: Field ordinal 1 is
>> invalid for type '(DrillRecordRow[**])'
>> at org.apache.drill.exec.work
>> .foreman.Foreman.run(Foreman.java:303)
>> at .......(:0)
>> Caused by: java.lang.AssertionError: Field ordinal 1 is invalid for type
>> '(DrillRecordRow[**])'
>> at
>> org.apache.calcite.rex.RexBuilder.makeFieldAccess(RexBuilder.java:197)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3694)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.access$2200(SqlToRelConverter.java:217)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4765)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061)
>> at
>> org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4625)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3908)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:670)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3150)
>> at
>> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563)
>> at
>> org.apache.drill.exec.planner.sql.SqlConverter.toRel(SqlConverter.java:414)
>> at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685)
>> at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:202)
>> at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:172)
>> at
>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:226)
>> at
>> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:124)
>> at
>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:90)
>> at org.apache.drill.exec.work
>> .foreman.Foreman.runSQL(Foreman.java:591)
>> at org.apache.drill.exec.work
>> .foreman.Foreman.run(Foreman.java:276)
>> ... 1 more
>> apache drill>
