I think it makes the most sense to me for us to publish a separate repo with a module and nar build for now and post when it's available in the users group.
Thanks for the discussion everyone, hopefully we can start making some helpful contributions soon. -Adam On 2018/02/10 23:43:31, Tony Kurc <[email protected]<mailto:[email protected]>> wrote: > It is like Matt read my mind.> > > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess > <[email protected]<mailto:[email protected]>> wrote:> > > > I'm fine with a vote, but I'll be voting to keep Java as the single> > > language for the (non-test) code. I share the same concerns as many of> > > the other folks as far as accepting other languages, it's mainly the> > > "slippery slope" argument that I don't want to turn into a> > > JVM-language flame war. If Scala, why not Groovy? Certainly the> > > syntax is closer to Java, and the community has accepted it as a valid> > > language for writing unit tests, although we stopped short for> > > allowing it for the deployable NiFi codebase, for the same reasons> > > IIRC. If Scala and/or Groovy, why not Kotlin? The same argument> > > (albeit more tenuous) goes for Clojure and just about every other JVM> > > language (although I don't expect a call for LuaJ processors lol).> > >> > > Whether we decide to support various languages ad-hoc or not, I would> > > strenuously object to multiple/hybrid build systems for the deployed> > > artifacts. If I could switch NiFi completely to Gradle I would, but I> > > realize there are good reasons for not doing so (yet?) in the Apache> > > NiFi community, and I would never want any hybrid Maven/Gradle build> > > for the deployable code, likewise for SBT, Leiningen, etc. With a> > > custom Mojo for Maven NAR builds, and the complexity for hybrid builds> > > in general, I think this would create a maintenance nightmare.> > >> > > The language thing is a tough decision though, it's not awesome that> > > specifying a single language can be a barrier to a more diverse> > > community, certainly Scala-based bundles would be more than welcome in> > > the overall NiFi ecosystem, I just think the cons outweigh the pros> > > for the baseline code. I've written Groovy processors/NARs using> > > Gradle as the build system, and I'm good with keeping them in my own> > > repo, especially when the Extension Registry becomes a thing. I can> > > see the Extension Registry perhaps making this a moot point, but> > > clearly we need to have this discussion in the meantime.> > >> > > Regards,> > > Matt> > >> > >> > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande > > <[email protected]<mailto:[email protected]>> wrote:> > > > Wasn't there a warning trigger about the NiFi distro size from Apache> > > > recently? IMO, before talking alternative languages, solve the modularity> > > > and NAR distribution problem. I think the implementation of a module> > > won't> > > > matter much then, the point being not everything has to go in the core,> > > > base distribution, but can still be easily sourced from a known repo, for> > > > example.> > > >> > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)> > > >> > > > Andrew> > > >> > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee > > > <[email protected]<mailto:[email protected]>>> > > wrote:> > > >> > > >> This probably necessitates a vote, yeah?> > > >>> > > >> Frankly, I’m usually happier writing Scala, and I’ve not encountered any> > > >> problems using processors written in Scala, but I think it’ll be> > > important> > > >> to tread lightly.> > > >>> > > >> There’s a few things that pop into my head:> > > >>> > > >> - Maintainability and reviewability. A very very good Java developer> > > need> > > >> not, by definition, either know how to write or identify good Scala or> > > spot> > > >> problems and bugs.> > > >> - Every Scala processor would either end up with a 5MB scala-lang.jar> > > >> packaged into the .nar or we’d have to start including it in the core> > > >> somewhere, if it’s not. It’s possible it might have already gotten> > > pulled> > > >> up from other dependencies.> > > >> - Style. There’s a tremendous amount of variation in Scala style because> > > >> of its type system, implicits, macros, and functional nature. There are> > > >> very good people out there that can write good Scala that isn’t> > > readable by> > > >> the 99%.> > > >> - Binary compatibility. Scala tends to be a little more brazen about> > > >> breaking binary compatibility in major releases and those happen a bit> > > more> > > >> often than with Java. That’s not a problem for any potential source> > > code in> > > >> the project, but it could present some dependency issues someday.> > > >> - Testing. There’s N > 1 test frameworks and testing styles within> > > those,> > > >> so there’s a lot of options for introducing more variability into the> > > tests.> > > >> - NiFi uses a lot of statics in setting up properties and relationships> > > >> and the like, and idiomatic Scala approaches that stuff a bit> > > differently,> > > >> so it’ll be necessary to impose some style guidelines so there isn’t too> > > >> much variation.> > > >>> > > >> That said, there are some things that won’t be problematic:> > > >>> > > >> - As mentioned, processors written in Scala do just work.> > > >> - The scala-maven-plugin works just fine allowing mixed Java-Scala> > > >> projects (btw, it’d probably be not super great to do mixed Java-Scala> > > and> > > >> mixed Maven-SBT though).> > > >> - A lot of the above concerns could be addressed by having clear style> > > >> guidelines.> > > >>> > > >> Another thing: most of the projects I see deliver separate jars for> > > Scala> > > >> components are delivering idiomatic APIs wrapping Java (or vice versa).> > > I> > > >> think publishing a separate set of jars/nars for stuff written in> > > Scala> > > >> would be odd since here it’d mostly be processors with new functionality> > > >> and not functionality for using Scala. I could imagine a lib of> > > implicits,> > > >> traits, classes that could make the Scala development more enjoyable.> > > That> > > >> probably would make sense to deliver that way.> > > >>> > > >> -joey> > > >>> > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende > > >> <[email protected]<mailto:[email protected]>>, wrote:> > > >> > I agree more with Andy about sticking with Java. The more varying> > > >> languages> > > >> > used, the more challenging it is to maintain. Once the code is part of> > > >> the> > > >> > Apache NiFi git repo, it is now the responsibility of the committers> > > and> > > >> > PMC members to maintain it.> > > >> >> > > >> > I’d even say I am somewhat against the groovy/Spock test code that> > > Andy> > > >> > mentioned. I have frequently spent hours trying to fix a Spock test> > > that> > > >> > broke from something I was working on. Every committer is familiar> > > with> > > >> > JUnit, but only a couple know Spock. Just using this as an example> > > that> > > >> > every committer knows Java, but only a couple probably know Scala,> > > >> Clojure,> > > >> > etc.> > > >> >> > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff > > >> > <[email protected]<mailto:[email protected]>> wrote:> > > >> >> > > >> > > +1 to Joe's response. If you can develop a component in Groovy or> > > Scala> > > >> > > (or Clojure!) more quickly/comfortably, or if allowing components> > > >> written> > > >> > > in other languages would encourage people to contribute more, I'm> > > all> > > >> for> > > >> > > it.> > > >> > >> > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt > > >> > > <[email protected]<mailto:[email protected]>>> > > wrote:> > > >> > >> > > >> > > > i personally would be ok with it for an extension/processor> > > provided> > > >> it> > > >> > > > integrates well with the build.> > > >> > > >> > > >> > > > i would agree with andys view for core framework stuff but for> > > >> > > extensions i> > > >> > > > think we can do it like mikethomsen suggested.> > > >> > > >> > > >> > > > others?> > > >> > > >> > > >> > > > thanks> > > >> > > > joe> > > >> > > >> > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" > > >> > > > <[email protected]<mailto:[email protected]>>> > > >> wrote:> > > >> > > >> > > >> > > > > I'm just a community contributor, so take that FWIW, but a> > > >> compromise> > > >> > > > might> > > >> > > > > be to publish the Scala code as separate maven modules to maven> > > >> central> > > >> > > > and> > > >> > > > > then submit a thoroughly tested processor written in Java. As> > > long> > > >> as> > > >> > > you> > > >> > > > > have enough unit and integration tests to give strong coverage,> > > I> > > >> > > > wouldn't> > > >> > > > > imagine anyone here would have issues reviewing it. If the tests> > > >> fail> > > >> > > > > because of code issues in the external dependencies, the obvious> > > >> answer> > > >> > > > is> > > >> > > > > to just hold the PR until the tests pass.> > > >> > > > >> > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <> > > >> > > [email protected]<mailto:[email protected]>> > > >> > > > > wrote:> > > >> > > > >> > > >> > > > > > Devs,> > > >> > > > > >> > > >> > > > > > I have some interns starting with my team and we use Scala> > > >> internally> > > >> > > > for> > > >> > > > > > our work.> > > >> > > > > > If I wanted to have them work to contribute some new> > > processors,> > > >> > > would> > > >> > > > > > they have to be Java to be included with the base> > > distribution or> > > >> > > could> > > >> > > > > > they use Scala as long as it fit within your current build and> > > >> test> > > >> > > > > process?> > > >> > > > > >> > > >> > > > > > Thanks,> > > >> > > > > > -Adam> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> > --> > > >> > Sent from Gmail Mobile> > > >>> > >> > Adam Weiss | Senior Manager, Digital Solutions R&D PerkinElmer | Innovating for a Healthier World [email protected]<mailto:[email protected]> Mobile: +1 716.429.7324 77 Goodell St. Suite 470, Buffalo, NY, 14203 www.perkinelmer.com
