This wiki <https://github.com/paul-rogers/drill/wiki> should be very helpful for understanding plugins development process and the unit testing architecture among many other things.
I'm facing difficulties with running the test cases successfully too using a fresh clone: https://issues.apache.org/jira/browse/DRILL-5606 If you get over this, please share the knowledge. Thanks, Gelbana On Thu, Aug 3, 2017 at 8:57 PM, Paul Rogers <[email protected]> wrote: > Hi Bob, > > Thanks for tackling the new plugin! > > Drill is a huge piece of software build, in part, by combining a wide > variety of libraries and packages. As it turns out, the package authors > have used an even wider variety of loggers, Guava versions and so on. > > Maven provides “dependency management” to control the chaos. Basically, > the pom.xml file first adds library X that, say, uses log4j or commons > logging. Then, elsewhere in the pom file, dependency management removes > these dependencies. Maven provides (very hard to read) documentation. The > “dependency tree” is your friend here. > > From your e-mail, it is unclear if these errors appeared with a “stock” > Drill build, or after you started adding the libraries for WARC. If in > “stock” Drill, then somehow something slipped through the cracks & we need > to fix it. If after adding new jars, then you have to tackle the dependency > management aspect. > > Note that, depending on where you add the code, you may also find yourself > fighting with the “JDBC All” project. That project imports all of Drill > into the JDBC package, then removes a bunch of stuff to keep the file size > under a defined maximum. If your code pushes JDBC over the limit, you’ll > have to find things to throw overboard to get below the limit again. (This > is why storage plugins should, ideally, be in the contrib directory, not in > exec.) > > At present, Drill provides no “SDK”: no way to build plugins except as > part of the Drill source tree. At one point I was able to get this working > for some aspects. Storage plugins, however, do require changes in core > Drill code (for the bootstrap file, for registering the reader, etc.) That > is the bit that must be fixed to allow a true external plugin development. > This is a shame, and should be fixed. If you have time to do so, we’re > always looking for contributions! > > Thanks, > > - Paul > > > > On Aug 3, 2017, at 10:29 AM, Bob Rudis <[email protected]> wrote: > > > > Hey folks, > > > > First: > > > > Inspired by the PCAP support in 1.11.0 I started down the path of > > cloning drill and just doing a test build before I started looking at > > 2 of the issues I posted for the PCAP storage format and also working > > to incorporate a similar WARC plugin (via the jwat jars). > > > > `mvn package -DskipTests` works fine but `mvn install -DskipTests` > generates: > > > > Found Banned Dependency: log4j:log4j:jar:1.2.17 > > Found Banned Dependency: commons-logging:commons-logging:jar:1.1.1 > > > > warnings which causes: > > > > Apache Drill Root POM .............................. FAILURE > > > > and (hence) the overall build to fail. > > > > I'm hoping that I'm just missing something obvious after some failed > googling. > > > > > > Second: > > > > Is there a way to develop the WARC format plugin outside of > > Drill-source-proper vs having to build into it. If so, is there a > > small, example GH repo someone cld point me to (that also has the "how > > to get it into Drill" part). From a scan of the source, it looks like > > the formats are embedded into a few source files in the > > Drill-source-proper. If there's no way to develop one outside of it, > > that's not a problem, I just figured it'd be easier to do it outside > > of modifying Drill source first. > > > > thx, > > > > -Bob > >
