Wasn't there a warning trigger about the NiFi distro size from Apache recently? IMO, before talking alternative languages, solve the modularity and NAR distribution problem. I think the implementation of a module won't matter much then, the point being not everything has to go in the core, base distribution, but can still be easily sourced from a known repo, for example.
I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :) Andrew On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[email protected]> wrote: > This probably necessitates a vote, yeah? > > Frankly, I’m usually happier writing Scala, and I’ve not encountered any > problems using processors written in Scala, but I think it’ll be important > to tread lightly. > > There’s a few things that pop into my head: > > - Maintainability and reviewability. A very very good Java developer need > not, by definition, either know how to write or identify good Scala or spot > problems and bugs. > - Every Scala processor would either end up with a 5MB scala-lang.jar > packaged into the .nar or we’d have to start including it in the core > somewhere, if it’s not. It’s possible it might have already gotten pulled > up from other dependencies. > - Style. There’s a tremendous amount of variation in Scala style because > of its type system, implicits, macros, and functional nature. There are > very good people out there that can write good Scala that isn’t readable by > the 99%. > - Binary compatibility. Scala tends to be a little more brazen about > breaking binary compatibility in major releases and those happen a bit more > often than with Java. That’s not a problem for any potential source code in > the project, but it could present some dependency issues someday. > - Testing. There’s N > 1 test frameworks and testing styles within those, > so there’s a lot of options for introducing more variability into the tests. > - NiFi uses a lot of statics in setting up properties and relationships > and the like, and idiomatic Scala approaches that stuff a bit differently, > so it’ll be necessary to impose some style guidelines so there isn’t too > much variation. > > That said, there are some things that won’t be problematic: > > - As mentioned, processors written in Scala do just work. > - The scala-maven-plugin works just fine allowing mixed Java-Scala > projects (btw, it’d probably be not super great to do mixed Java-Scala and > mixed Maven-SBT though). > - A lot of the above concerns could be addressed by having clear style > guidelines. > > Another thing: most of the projects I see deliver separate jars for Scala > components are delivering idiomatic APIs wrapping Java (or vice versa). I > think publishing a separate set of jars/nars for stuff written in Scala > would be odd since here it’d mostly be processors with new functionality > and not functionality for using Scala. I could imagine a lib of implicits, > traits, classes that could make the Scala development more enjoyable. That > probably would make sense to deliver that way. > > -joey > > On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[email protected]>, wrote: > > I agree more with Andy about sticking with Java. The more varying > languages > > used, the more challenging it is to maintain. Once the code is part of > the > > Apache NiFi git repo, it is now the responsibility of the committers and > > PMC members to maintain it. > > > > I’d even say I am somewhat against the groovy/Spock test code that Andy > > mentioned. I have frequently spent hours trying to fix a Spock test that > > broke from something I was working on. Every committer is familiar with > > JUnit, but only a couple know Spock. Just using this as an example that > > every committer knows Java, but only a couple probably know Scala, > Clojure, > > etc. > > > > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[email protected]> wrote: > > > > > +1 to Joe's response. If you can develop a component in Groovy or Scala > > > (or Clojure!) more quickly/comfortably, or if allowing components > written > > > in other languages would encourage people to contribute more, I'm all > for > > > it. > > > > > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[email protected]> wrote: > > > > > > > i personally would be ok with it for an extension/processor provided > it > > > > integrates well with the build. > > > > > > > > i would agree with andys view for core framework stuff but for > > > extensions i > > > > think we can do it like mikethomsen suggested. > > > > > > > > others? > > > > > > > > thanks > > > > joe > > > > > > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[email protected]> > wrote: > > > > > > > > > I'm just a community contributor, so take that FWIW, but a > compromise > > > > might > > > > > be to publish the Scala code as separate maven modules to maven > central > > > > and > > > > > then submit a thoroughly tested processor written in Java. As long > as > > > you > > > > > have enough unit and integration tests to give strong coverage, I > > > > wouldn't > > > > > imagine anyone here would have issues reviewing it. If the tests > fail > > > > > because of code issues in the external dependencies, the obvious > answer > > > > is > > > > > to just hold the PR until the tests pass. > > > > > > > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam < > > > [email protected] > > > > > wrote: > > > > > > > > > > > Devs, > > > > > > > > > > > > I have some interns starting with my team and we use Scala > internally > > > > for > > > > > > our work. > > > > > > If I wanted to have them work to contribute some new processors, > > > would > > > > > > they have to be Java to be included with the base distribution or > > > could > > > > > > they use Scala as long as it fit within your current build and > test > > > > > process? > > > > > > > > > > > > Thanks, > > > > > > -Adam > > > > > > > > > > > > > > > > > > > > -- > > Sent from Gmail Mobile >
