Hi Bob, Thanks for tackling the new plugin!
Drill is a huge piece of software build, in part, by combining a wide variety of libraries and packages. As it turns out, the package authors have used an even wider variety of loggers, Guava versions and so on. Maven provides “dependency management” to control the chaos. Basically, the pom.xml file first adds library X that, say, uses log4j or commons logging. Then, elsewhere in the pom file, dependency management removes these dependencies. Maven provides (very hard to read) documentation. The “dependency tree” is your friend here. From your e-mail, it is unclear if these errors appeared with a “stock” Drill build, or after you started adding the libraries for WARC. If in “stock” Drill, then somehow something slipped through the cracks & we need to fix it. If after adding new jars, then you have to tackle the dependency management aspect. Note that, depending on where you add the code, you may also find yourself fighting with the “JDBC All” project. That project imports all of Drill into the JDBC package, then removes a bunch of stuff to keep the file size under a defined maximum. If your code pushes JDBC over the limit, you’ll have to find things to throw overboard to get below the limit again. (This is why storage plugins should, ideally, be in the contrib directory, not in exec.) At present, Drill provides no “SDK”: no way to build plugins except as part of the Drill source tree. At one point I was able to get this working for some aspects. Storage plugins, however, do require changes in core Drill code (for the bootstrap file, for registering the reader, etc.) That is the bit that must be fixed to allow a true external plugin development. This is a shame, and should be fixed. If you have time to do so, we’re always looking for contributions! Thanks, - Paul > On Aug 3, 2017, at 10:29 AM, Bob Rudis <[email protected]> wrote: > > Hey folks, > > First: > > Inspired by the PCAP support in 1.11.0 I started down the path of > cloning drill and just doing a test build before I started looking at > 2 of the issues I posted for the PCAP storage format and also working > to incorporate a similar WARC plugin (via the jwat jars). > > `mvn package -DskipTests` works fine but `mvn install -DskipTests` generates: > > Found Banned Dependency: log4j:log4j:jar:1.2.17 > Found Banned Dependency: commons-logging:commons-logging:jar:1.1.1 > > warnings which causes: > > Apache Drill Root POM .............................. FAILURE > > and (hence) the overall build to fail. > > I'm hoping that I'm just missing something obvious after some failed googling. > > > Second: > > Is there a way to develop the WARC format plugin outside of > Drill-source-proper vs having to build into it. If so, is there a > small, example GH repo someone cld point me to (that also has the "how > to get it into Drill" part). From a scan of the source, it looks like > the formats are embedded into a few source files in the > Drill-source-proper. If there's no way to develop one outside of it, > that's not a problem, I just figured it'd be easier to do it outside > of modifying Drill source first. > > thx, > > -Bob
