Hey Rafael, I was thinking about this a little more and I actually don't think it would be possible to build on Drill 1.17 without some substantial reworking. Also, the storage-plugins aren't completely "pluggable". They use a protobuf library that is not in the contrib/ folder. Removing this dependency is being discussed. -- C
> On Mar 31, 2020, at 1:51 PM, Jaimes, Rafael - 0993 - MITLL > <[email protected]> wrote: > > Hi Charles, > > (1./2.) > I have not been able to build Drill, from either a full clone of your tagged > http-storage branch or from the standard Drill 1.17 release. > I've narrowed it down to some dependency problems from the POM. In > particular, I run into issues here: > > Downloading: > https://repo.maven.apache.org/maven2/org/apache/apache/21/apache-21.pom > [ERROR] The build could not read 1 project -> [Help 1] > [ERROR] > [ERROR] The project org.apache.drill:drill-root:1.18.0-SNAPSHOT > (/home/ra29435/drill-official/drill/pom.xml) has 1 error > [ERROR] Non-resolvable parent POM: Could not transfer artifact > org.apache:apache:pom:21 from/to conjars (http://conjars.org/repo): > Connection to http://conjars.org refurelativePath' points at no local POM @ > line 24, column 11: Connection timed out (Connection timed out) -> [Help 2] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException > [ERROR] [Help 2] > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException > > I think it has something to do with the fact that I normally resolve > dependencies from our local Maven repo mirrors. We have no problems getting > stuff from Maven Central and common places, but I am unfamiliar with > conjars.org. I wonder if it is related to that? > > (3./4.) > I tried putting the JAR into either jars/ or jars/3rdparty with the same > error. I haven't gone down the dependency tree so I have not made and JARs of > them, that could be a major thing I'm missing. > > Yes this is still in a testing environment. I'm going to use your pre-built > images for testing the REST endpoint, this is extremely helpful. If it works > out I'll go back to trying to build it. Also, hoping that this will make its > way into the next (1.18) release. > > Best, > Rafael > > -----Original Message----- > From: Charles Givre <[email protected]> > Sent: Tuesday, March 31, 2020 1:34 PM > To: user <[email protected]> > Subject: Re: REST data source? > > Hi Rafael, > Glad you're getting some value from Drill. Repackaging that directory as a > truly pluggable jar is tricky. A few questions: > 1. Did you copy the contrib/storage-http into its own folder and then do a > build from that? > 2. Did it build successfully? > 3. Did you copy the JARs into your Drill jars/3rdparty folder? > 4. You'll also have to get JARs of any dependencies as well and copy them to > the jars/3rdparty. Have you done that? > > I actually have a pre-built version of Drill with the storage-http plugin > available here: https://github.com/cgivre/drill/releases > <https://github.com/cgivre/drill/releases>. Please do not use that in any > kind of production setup. If you're just wanting to try this out, it might > be easier to d/l that and use that. > -- C > > > >> On Mar 31, 2020, at 12:57 PM, Jaimes, Rafael - 0993 - MITLL >> <[email protected]> wrote: >> >> Hi Charles, >> >> I am trying to use the http-storage plugin from your branch. I put the >> storage plug-in files in a jar and tried to keep the jar directory structure >> the same as other plug-ins. Upon starting drill-embedded I’m getting the >> error below. I am using your drill-module.conf and >> bootstrap-storage-plugins.json from your branch. Is there another step I >> need to perform to get Drill to recognize the plug-in? I am using 1.17 >> release. >> >> Error: Failure in starting embedded Drillbit: >> java.lang.IllegalStateException: >> com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve >> type id 'http' as a subtype of [simple type, class >> org.apache.drill.common.logical.StoragePluginConfig]: known type ids = >> [InfoSchemaConfig, SystemTablePluginConfig, file, hbase, hive, jdbc, kafka, >> kudu, mock, mongo, named, openTSDB] (for POJO property 'storage') at >> [Source: (String)"{ >> "storage":{ >> "http" : { >> "type":"http", >> "connections": {}, >> "enabled": false >> } >> } >> } >> "; line: 4, column: 14] (through reference chain: >> org.apache.drill.exec.planner.logical.StoragePlugins["storage"]->java. >> util.LinkedHashMap["http"]) (state=,code=0) >> >> Paul, >> >> I don’t know much about this REST service quite yet (it is internal). We >> utilize REST API where all responses are returned as JSON formatted strings >> in many places, I don’t think it is very sophisticated. I am not sure how it >> will handle projection and filter issues. My current pipeline involves using >> python requests.get() and then unpacking the response string. It does have >> an authentication layer, so I am mildly concerned that the >> HTTP-storage-plugin will have a hiccup – although it looks like it can use >> “Basic”. If I can get Drill to query the endpoint I will report back if I >> find anything else that might be useful to you. >> >> Thanks both for your great work with Drill! >> >> - Rafael >
