Updates: I reviewed PR for Stream API and think it would be good start.
I would be really appreciated if some of us also participate reviewing as
well. Thanks in advance.
- Jungtaek Lim (HeartSaVioR)
2017년 1월 4일 (수) 오전 4:00, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 작성:
> Lambdas can also lead to clean code reuse. It is part of evolving to adopt
> new JDK functionalities, and I think that we should do it.
>
> It is expected that contributors starting with a new technology/paradigm
> may not always implement code that follows perfect practices. However, we
> should look at that as an opportunity to have more experienced community
> members give constructive feedback through code reviews, and sharing good
> resources where one can master such best practices.
>
> I don’t think that a few less than ideal implementations should lead us
> not to use this powerful language feature.
>
> Hugo
>
>
> > On Jan 3, 2017, at 10:19 AM, S G <sg.online.em...@gmail.com> wrote:
> >
> > Yes, I agree.
> >
> > +1s on both these lines:
> >
> > *Badly designed APIs can get more complicated when combined with lambdas*
> > *Lambdas and functional style can greatly improve the readability if used
> > properly*.
> >
> >
> >
> > On Mon, Jan 2, 2017 at 9:34 PM, Arun Mahadevan <ar...@apache.org> wrote:
> >
> >> Its more about getting used to and judiciously deciding when to use
> >> functional v/s procedural approach. Badly designed APIs can get more
> >> complicated when combined with lambdas, but overall what I observe is
> >> lambdas and functional style can greatly improve the readability (with
> many
> >> other benefits) if used properly.
> >>
> >> A simple word count pipeline like below can get too verbose when
> expressed
> >> without lambdas.
> >>
> >> Stream<String> stream = …
> >> stream
> >> .flatMap(s -> Arrays.asList(s.split(" ")))
> >> .mapToPair(w -> Pair.of(w, 1))
> >> .countByKey()
> >> .filter(x -> x.getSecond() >= 5)
> >> .print();
> >>
> >> *Without lambda*
> >>
> >> stream
> >> .flatMap(new FlatMapFunction<String, String>() {
> >> @Override
> >> public Iterable<String> apply(String s) {
> >> return Arrays.asList(s.split(" "));
> >> }
> >> })
> >> .mapToPair(new PairFunction<String, String, Integer>() {
> >> @Override
> >> public Pair<String, Integer> apply(String w) {
> >> return Pair.of(w, 1);
> >> }
> >> })
> >> .countByKey()
> >> .filter(new Predicate<Pair<String, Long>>() {
> >> @Override
> >> public boolean test(Pair<String, Long> x) {
> >> return x.getSecond() >= 5;
> >> }
> >> })
> >> .print();
> >>
> >>
> >> Thanks,
> >> Arun
> >>
> >> On 1/3/17, 10:21 AM, "S G" <sg.online.em...@gmail.com> wrote:
> >>
> >>> One bad usage I have found is missing types for lambda arguments.
> >>> When the types for arguments are not provided in the code itself,
> github
> >> or
> >>> vi-editor browsing of such code becomes impossible.
> >>>
> >>> You must have an IDE like Eclipse to infer the types for such cases and
> >>> that becomes a big limitation to browse such code. I have not looked
> into
> >>> the storm code-base to find such occurrences but the above is just a
> >>> general observation from the lambda code usages.
> >>>
> >>>
> >>> Secondly, lambdas tend to generate bad stack-traces on exceptions.
> >>>
> >>>
> >>> And lastly, functional programming is not natural to many Java
> developers.
> >>> https://zeroturnaround.com/rebellabs/how-to-avoid-
> >> ruining-your-world-with-lambdas-in-java-8/
> >>> hints at some of these problems. Java code is usually very easy to
> >>> understand. But once you throw in more than a single level of lambdas,
> >>> things tend to become more and more confusing for several developers.
> >> (This
> >>> last point reflects the thoughts from some of my fellow developers, not
> >>> representative of all the devs)
> >>>
> >>>
> >>> Thanks
> >>> SG
> >>>
> >>>
> >>> On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <kabh...@gmail.com>
> wrote:
> >>>
> >>>> Hi S G,
> >>>>
> >>>> I'd be happy if you could elaborate your opinion. Did you found bad
> >> usages
> >>>> from master branch of Storm code?
> >>>>
> >>>> Regarding comfortable of lambda, IMHO, I don't think many users are
> >>>> unfamiliar with lambda, since they should have been used it with
> various
> >>>> languages. We might not be comfortable with Java 8 lambda (since
> >> transition
> >>>> to Java 8 is going slowly), but it's just a matter of familiarizing.
> >>>>
> >>>> Are there kind of best practices for Java 8 lambda? We can refer these
> >> to
> >>>> construct some guides / restrictions for Storm project.
> >>>>
> >>>> Thanks,
> >>>> Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2017년 1월 3일 (화) 오후 12:26, S G <sg.online.em...@gmail.com>님이 작성:
> >>>>
> >>>>> I have found several bad usages of Java 8 lambdas and many developers
> >> are
> >>>>> not comfortable using them.
> >>>>>
> >>>>> So we should use them only if it really makes the code beautiful and
> >>>> easier
> >>>>> to understand.
> >>>>>
> >>>>> My 2c,
> >>>>> Thanks
> >>>>> SG
> >>>>>
> >>>>>
> >>>>> On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> The streams API implementation has limited usage of 1.8 features and
> >>>> can
> >>>>>> be easily ported to 1.7 if required. The examples are written in
> >> 1.8,
> >>>> the
> >>>>>> thought being users would stick to the Java 8 style usage (lambdas)
> >>>> from
> >>>>>> the beginning. If there is consensus we could also consider moving
> >> the
> >>>>> 1.x
> >>>>>> branch to JDK 8.
> >>>>>>
> >>>>>> Anyways would like interested folks to start reviewing the changes
> >> so
> >>>>> that
> >>>>>> we can take it forward.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Arun
> >>>>>>
> >>>>>>
> >>>>>> On 12/23/16, 10:09 AM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
> >>>>>>
> >>>>>>> FYI, I've realized that internal of Stream API (pull request)
> >> relies
> >>>> on
> >>>>>> JDK
> >>>>>>> 8 (what I've found is 'static method in interface' and maybe more)
> >> so
> >>>>> for
> >>>>>>> now Stream API is expected to be included for at least Storm 2.0.0
> >>>>> unless
> >>>>>>> the PR is modified to fit to JDK 7.
> >>>>>>>
> >>>>>>> - Jungtaek Lim (HeartSaVioR)
> >>>>>>>
> >>>>>>> 2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >>>>>>>
> >>>>>>>> Thanks Manu and Taylor for giving your opinions.
> >>>>>>>>
> >>>>>>>> - Storm SQL improvement
> >>>>>>>>
> >>>>>>>> There're some huge PRs available but there're all about
> >> improvement
> >>>>>> which
> >>>>>>>> shouldn't be blocker for releasing 1.1.0. (I'd like to also
> >> include
> >>>>>> them to
> >>>>>>>> 1.1.0 but not sure it can be happen really soon.)
> >>>>>>>> I'll send a request for reviewing about pending Storm SQL PRs.
> >>>>>>>>
> >>>>>>>> Only one issue (STORM-2200) is linked to release 1.1.0 epic
> >> which is
> >>>>>>>> blocker for me.
> >>>>>>>>
> >>>>>>>> - Java port
> >>>>>>>>
> >>>>>>>> I also had some developers saying 'If core of Storm were written
> >> by
> >>>>>> Java,
> >>>>>>>> I could experiment and even contribute on something'. I was one
> >> of
> >>>>> them,
> >>>>>>>> and to be honest, I'm still a beginner of Clojure. Moving to
> >> Java 8
> >>>>> also
> >>>>>>>> gives great functionalities for us, so Java port is what I think
> >> the
> >>>>>> most
> >>>>>>>> important thing among the huge works now in progress. Ideally,
> >> and
> >>>>>>>> hopefully, I'd like to see us focus on this and make this happen
> >> at
> >>>>> the
> >>>>>>>> very early next year.
> >>>>>>>> (Yes we should do some manual tests and maybe some refactoring
> >> too.)
> >>>>>>>>
> >>>>>>>> - Metrics V2
> >>>>>>>>
> >>>>>>>> I'm not sure when we plan to release Storm 1.2.0, but given that
> >>>>>> there're
> >>>>>>>> only two things left (logviewer / ui) for completing port work
> >>>> (except
> >>>>>>>> tests) I guess Storm 2.0.0 might be happen earlier.
> >>>>>>>> Taylor, when do you expect metrics V2 will be available for
> >>>> reviewing?
> >>>>>>>>
> >>>>>>>> - Stream API
> >>>>>>>>
> >>>>>>>> With labeling as experiment or annotating with evolving, we could
> >>>>>> include
> >>>>>>>> the first version to next minor excluding 1.1.0. (We could even
> >>>>> include
> >>>>>>>> this to 1.1.0 if we start reviewing this very soon.)
> >>>>>>>>
> >>>>>>>> I'd like to hear others' opinions as well.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <ptgo...@gmail.com>님이
> >>>> 작성:
> >>>>>>>>
> >>>>>>>> Hi Jungtaek,
> >>>>>>>>
> >>>>>>>>> - Beam runner
> >>>>>>>>
> >>>>>>>> There’s not been much activity around this, and I haven’t had
> >> much
> >>>>> time
> >>>>>> to
> >>>>>>>> work on it recently, but there’s a decent foundation to build
> >> upon.
> >>>> So
> >>>>>> it
> >>>>>>>> would be fairly easy for others to start contributing to that
> >>>> effort.
> >>>>>>>> There’s also interest from the Beam community in that runner, so
> >> one
> >>>>>>>> possibility is to move that effort to the Apache Beam project.
> >>>>>>>>
> >>>>>>>> This is very preliminary work, so I don’t have a good handle on
> >> what
> >>>>> the
> >>>>>>>> target release would be.
> >>>>>>>>
> >>>>>>>>> - Metrics renewal
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> This is what I’ve been referring to as “metrics_v2”. This is
> >>>>> progressing
> >>>>>>>> fairly well with support for multiple reporters (e.g. Graphite,
> >>>>> Ganglia,
> >>>>>>>> console, etc.), worker metrics, disruptor metrics, etc.
> >>>>>>>>
> >>>>>>>> I would like to target this work for 1.2.0.
> >>>>>>>>
> >>>>>>>>> - Java port
> >>>>>>>>
> >>>>>>>> This effort seems to have picked up (for example Bobby’s
> >> conversion
> >>>> of
> >>>>>>>> Nimbus, etc.) and is progressing steadily. It’s taken a lot
> >> longer
> >>>>> than
> >>>>>>>> initially thought, but a lot of that can be attributed to the ebb
> >>>> and
> >>>>>> flow
> >>>>>>>> of people’s availability to do the work.
> >>>>>>>>
> >>>>>>>>> - Storm SQL improvement (Streaming SQL in future)
> >>>>>>>>
> >>>>>>>> You’ve been spearheading most of the work here, so I’d delegate
> >> to
> >>>> you
> >>>>>> for
> >>>>>>>> your opinion on where it stands. If you need additional reviews,
> >>>> just
> >>>>>> ask
> >>>>>>>> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject
> >> line
> >>>>> might
> >>>>>>>> help get attention).
> >>>>>>>>
> >>>>>>>> My thinking has been that this could be included in the 1.1.0
> >>>> release.
> >>>>>> Is
> >>>>>>>> there a set of JIRA issues you would like to include in order to
> >>>> make
> >>>>>> that
> >>>>>>>> happen?
> >>>>>>>>
> >>>>>>>>> - Stream API
> >>>>>>>>
> >>>>>>>> This seems to have stalled a bit, though there seems to be a lot
> >> of
> >>>>>>>> interest around it. I think we all would agree that when
> >>>> introducing a
> >>>>>> new
> >>>>>>>> API for building topologies, it’s important that we get right
> >> from
> >>>> the
> >>>>>>>> start and have strong buy-in from the development community. I
> >> would
> >>>>>>>> encourage anyone interested in the Streams API to review the
> >>>> proposal
> >>>>>> and
> >>>>>>>> initial code.
> >>>>>>>>
> >>>>>>>> I think it is close, but I’m not sure what release to target.
> >>>> Possibly
> >>>>>> the
> >>>>>>>> 2.0 release?
> >>>>>>>>
> >>>>>>>> Re: 1.1.0 Release
> >>>>>>>>
> >>>>>>>> STORM-2176 is a fairly big concern of mine since the feature it
> >>>>> involves
> >>>>>>>> was introduced in 1.0.0 and did not work then nor in any
> >> subsequent
> >>>> or
> >>>>>>>> future releases (may not be a problem in 2.0). Unfortunately, as
> >>>>> you’ve
> >>>>>>>> seen, finding the root cause is elusive. That issue could
> >> definitely
> >>>>> use
> >>>>>>>> more eyes.
> >>>>>>>>
> >>>>>>>> -Taylor
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <kabh...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi devs,
> >>>>>>>>>
> >>>>>>>>> I'm seeing lots of huge works in parallel, and we individual
> >> are
> >>>>> busy
> >>>>>>>>> regarding each work so common works (review, release,
> >>>> documentation,
> >>>>>>>> etc.)
> >>>>>>>>> have been not made in progress for several months.
> >>>>>>>>>
> >>>>>>>>> - Beam runner
> >>>>>>>>> - Metrics renewal
> >>>>>>>>> - Java port
> >>>>>>>>> - Storm SQL improvement (Streaming SQL in future)
> >>>>>>>>> - Stream API
> >>>>>>>>>
> >>>>>>>>> IMHO, it would be better to set target versions for them, and
> >> set
> >>>> a
> >>>>>>>> roadmap
> >>>>>>>>> (per version), and prioritize based on roadmap.
> >>>>>>>>>
> >>>>>>>>> Stream API (very first version), and Storm SQL improvement are
> >>>>> waiting
> >>>>>>>> for
> >>>>>>>>> review, and personally I would like to publish them soon.
> >>>>>>>>>
> >>>>>>>>> If we're OK to have 2.0.0 without adding much features, I'm in
> >>>> favor
> >>>>>> of
> >>>>>>>>> concentrating Java port work (postponing other things except
> >>>>> releasing
> >>>>>>>> 1.x
> >>>>>>>>> version line) and moving to Apache Storm 2.0.0 really soon.
> >>>>>>>>> (I'm even OK we decide to postpone some clojure files to be
> >>>>> addressed
> >>>>>>>> after
> >>>>>>>>> 2.0.0.)
> >>>>>>>>> Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
> >>>>> (2.x)
> >>>>>>>> which
> >>>>>>>>> is other reason to move to 2.x quickly.
> >>>>>>>>>
> >>>>>>>>> I'd be really happy if we have metrics renewal and beam runner,
> >>>> but
> >>>>>> I'm
> >>>>>>>> not
> >>>>>>>>> sure when they're available to be published. Do we have any
> >>>> updates
> >>>>>> here?
> >>>>>>>>>
> >>>>>>>>> What do you think? It might be ideal, and/or broader discussion
> >>>> but
> >>>>> we
> >>>>>>>>> haven't discussed our plan / vision for a long time so better
> >> to
> >>>>> give
> >>>>>> it
> >>>>>>>> a
> >>>>>>>>> try.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >>
>
>
