Re: Storage Plugin Assistance

2019-11-15 Thread Paul Rogers
Hi Charles, Looking at the code in your PR, it seems that you are, in fact, using Drill's JSON reader to decode the message JSON. (See [1]). Is that where you are having problems? Looks like this reader handles JSON passed as a string or from a file? In either case, get a local copy of the

Re: Storage Plugin Assistance

2019-11-15 Thread Charles Givre
> On Nov 15, 2019, at 1:39 PM, Paul Rogers wrote: > > Hi Charles, > > A thought on debugging deserialization is to not do it in a query. Capture > the JSON returned from a rest call. Write a simple unit test that > deserializes that by itself from a string or file. Deserialization is a bit

Re: Storage Plugin Assistance

2019-11-15 Thread Paul Rogers
Hi Charles, A thought on debugging deserialization is to not do it in a query. Capture the JSON returned from a rest call. Write a simple unit test that deserializes that by itself from a string or file. Deserialization is a bit of a black art, and is really a problem separate from Drill

Re: Storage Plugin Assistance

2019-11-15 Thread Charles Givre
Hi Igor, Thanks for the advice. I've been doing some digging and am still pretty stuck here. Can you recommend any techniques about how to debug the Jackson serialization/deserialization? I added a unit test that serializes a query and then deserializes it and that test fails. I've tracked

Re: Storage Plugin Assistance

2019-10-12 Thread Igor Guzenko
Hello Charles, Looks like you found another new issue. Maybe I explained unclear, but my previous suggestion wasn't about EXPLAIN PLAN construct, but rather: 1) Use http client like Postman or simply browser to save response of requested rest service into json file 2) Try to debug reading the

Re: Storage Plugin Assistance

2019-10-11 Thread Charles Givre
Hi Igor, Thanks for responding. I'm not sure if this is what you intended, but looked at the JSON for the Query plans and found something interesting. For the SELECT * query, I'm getting the following when I try to run the physical plan that it generates (without modification). Do you think

Re: Storage Plugin Assistance

2019-10-11 Thread Igor Guzenko
Hello Charles, You got the error from Apache Calcite at the planning stage while converting SQLIdentifier to RexNode. From your stack trace the conversion starts here DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685) and goes to

Storage Plugin Assistance

2019-10-11 Thread Charles Givre
Hello all, I decided to take the leap and attempt to implement a storage plugin. I found that a few people had started this, so I thought I'd complete a simple generic HTTP/REST storage plugin. The use case would be to enrich data sets with data that's available via public or internal APIs.