Ok, I commented in that thread. I think the proxy is the only missing piece. I tried connecting to a different service that is inside the proxy and it worked as expected. This looks like it will work well for our application.
FYI, Although it has basic auth, I am not using the authType field in the
storage config.
Rather, our service authenticates from the header in this format:
{"Authentication": "Basic <token>"}.
The response JSON is nested quite a bit but I think it can be fixed by
modifying the SELECT as you have done in your examples.
Thanks,
Rafael
-----Original Message-----
From: Charles Givre <[email protected]>
Sent: Tuesday, March 31, 2020 3:27 PM
To: [email protected]
Subject: Re: REST data source?
Rafael,
At the moment the plugin does not support proxy servers. However, this is
pretty easy to implement using the current libraries. Could you please add a
comment to the PR for the plugin (https://github.com/apache/drill/pull/1892
<https://github.com/apache/drill/pull/1892>) with some explanation of what you
need?
Thanks,
-- C
> On Mar 31, 2020, at 3:21 PM, Jaimes, Rafael - 0993 - MITLL
> <[email protected]> wrote:
>
> Hi Paul,
>
> I tried that (even tried a vanilla build before on its own) and I run into
> the same dependency problem. There is something in apache-21.pom that I
> cannot resolve. If it works for you I am certain it is a config on our end
> due to the way our proxies and mirrors are setup, we have to go through these
> internal channels when building and it sometimes causes issues.
>
> Charles,
>
> I am almost up and running with your pre-built instance. I have narrowed the
> problem down to possibly being another proxy issue. The GET requests don't
> seem to be honoring my system env variable proxy settings. Do you think
> there's any way to force Drill/plug-in to use a proxy? I'm unable to get the
> examples you have posted working: getting Connection reset error on HTTPS and
> Connect time out with HTTP. The URLs work fine if I test them outside of
> Drill.
>
> Thanks,
> Rafael
>
> -----Original Message-----
> From: Paul Rogers <[email protected]>
> Sent: Tuesday, March 31, 2020 2:36 PM
> To: [email protected]
> Subject: Re: REST data source?
>
> Hi Rafael,
>
> The easiest way to build the plugin will be to build all of Drill 1.18
> Snapshot with the plugin included.
>
> 1. Grab master from GitHub.
>
> 2. Merge in Charle's PR branch.
>
> 3. mvn clean install -DskipTests
>
> The above usually works for me. This process ensures that all the snapshot
> versions come from your own build.
>
> Not sure how we started storing snapshot versions in a Maven repo.
> This causes issues. If you rebuild part of Drill, and have not built
> the other parts in more than a day, Maven helpfully downloads the
> snapshots from the repo, causing all kinds of chaos. (We should fix
> this.)
>
> Once you do the build, you'll have a full Drill distribution, just like you'd
> download. You can use that distribution to run Drill with the plugin included.
>
> There are other ways that also work; the above may be the simplest.
>
>
> Thanks,
> - Paul
>
>
>
> On Tuesday, March 31, 2020, 10:51:18 AM PDT, Jaimes, Rafael - 0993 - MITLL
> <[email protected]> wrote:
>
> Hi Charles,
>
> (1./2.)
> I have not been able to build Drill, from either a full clone of your tagged
> http-storage branch or from the standard Drill 1.17 release.
> I've narrowed it down to some dependency problems from the POM. In
> particular, I run into issues here:
>
> Downloading:
> https://repo.maven.apache.org/maven2/org/apache/apache/21/apache-21.pom
> [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The
> project org.apache.drill:drill-root:1.18.0-SNAPSHOT
> (/home/ra29435/drill-official/drill/pom.xml) has 1 error [ERROR]
> Non-resolvable parent POM: Could not transfer artifact
> org.apache:apache:pom:21 from/to conjars (http://conjars.org/repo):
> Connection to http://conjars.org refurelativePath' points at no local POM @
> line 24, column 11: Connection timed out (Connection timed out) -> [Help 2]
> [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with
> the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingExcept
> ion [ERROR] [Help 2]
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelExce
> ption
>
> I think it has something to do with the fact that I normally resolve
> dependencies from our local Maven repo mirrors. We have no problems getting
> stuff from Maven Central and common places, but I am unfamiliar with
> conjars.org. I wonder if it is related to that?
>
> (3./4.)
> I tried putting the JAR into either jars/ or jars/3rdparty with the same
> error. I haven't gone down the dependency tree so I have not made and JARs of
> them, that could be a major thing I'm missing.
>
> Yes this is still in a testing environment. I'm going to use your pre-built
> images for testing the REST endpoint, this is extremely helpful. If it works
> out I'll go back to trying to build it. Also, hoping that this will make its
> way into the next (1.18) release.
>
> Best,
> Rafael
>
> -----Original Message-----
> From: Charles Givre <[email protected]>
> Sent: Tuesday, March 31, 2020 1:34 PM
> To: user <[email protected]>
> Subject: Re: REST data source?
>
> Hi Rafael,
> Glad you're getting some value from Drill. Repackaging that directory as a
> truly pluggable jar is tricky. A few questions:
> 1. Did you copy the contrib/storage-http into its own folder and then do a
> build from that?
> 2. Did it build successfully?
> 3. Did you copy the JARs into your Drill jars/3rdparty folder?
> 4. You'll also have to get JARs of any dependencies as well and copy them to
> the jars/3rdparty. Have you done that?
>
> I actually have a pre-built version of Drill with the storage-http plugin
> available here: https://github.com/cgivre/drill/releases
> <https://github.com/cgivre/drill/releases>. Please do not use that in any
> kind of production setup. If you're just wanting to try this out, it might
> be easier to d/l that and use that.
> -- C
>
>
>
>> On Mar 31, 2020, at 12:57 PM, Jaimes, Rafael - 0993 - MITLL
>> <[email protected]> wrote:
>>
>> Hi Charles,
>>
>> I am trying to use the http-storage plugin from your branch. I put the
>> storage plug-in files in a jar and tried to keep the jar directory structure
>> the same as other plug-ins. Upon starting drill-embedded I’m getting the
>> error below. I am using your drill-module.conf and
>> bootstrap-storage-plugins.json from your branch. Is there another step I
>> need to perform to get Drill to recognize the plug-in? I am using 1.17
>> release.
>>
>> Error: Failure in starting embedded Drillbit:
>> java.lang.IllegalStateException:
>> com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not
>> resolve type id 'http' as a subtype of [simple type, class
>> org.apache.drill.common.logical.StoragePluginConfig]: known type ids
>> = [InfoSchemaConfig, SystemTablePluginConfig, file, hbase, hive,
>> jdbc, kafka, kudu, mock, mongo, named, openTSDB] (for POJO property
>> 'storage') at [Source: (String)"{
>> "storage":{
>> "http" : {
>> "type":"http",
>> "connections": {},
>> "enabled": false
>> }
>> }
>> }
>> "; line: 4, column: 14] (through reference chain:
>> org.apache.drill.exec.planner.logical.StoragePlugins["storage"]->java.
>> util.LinkedHashMap["http"]) (state=,code=0)
>>
>> Paul,
>>
>> I don’t know much about this REST service quite yet (it is internal). We
>> utilize REST API where all responses are returned as JSON formatted strings
>> in many places, I don’t think it is very sophisticated. I am not sure how it
>> will handle projection and filter issues. My current pipeline involves using
>> python requests.get() and then unpacking the response string. It does have
>> an authentication layer, so I am mildly concerned that the
>> HTTP-storage-plugin will have a hiccup – although it looks like it can use
>> “Basic”. If I can get Drill to query the endpoint I will report back if I
>> find anything else that might be useful to you.
>>
>> Thanks both for your great work with Drill!
>>
>> - Rafael
>
smime.p7s
Description: S/MIME cryptographic signature
