Agree, I would target this for Beam 3.0.0.
Regards
JB
On 10/17/2017 06:43 PM, Reuven Lax wrote:
Should this be considered a backwards-incompatible change? If so, do we
need to wait until Beam 3.0.0 is released?
On Tue, Oct 17, 2017 at 9:11 AM, Ismaël Mejía <[email protected]> wrote:
I am bringing the subject to the user mailing list to get some
feedback because it makes sense anyway to discuss this there. But I
also agree with Kenneth about the fact that runner authors must weight
in on this subject.
On Tue, Oct 17, 2017 at 5:24 PM, Kenneth Knowles <[email protected]>
wrote:
+1 to having runner maintainers weigh in as proxies. Added a few in case
they haven't followed this thread.
On Mon, Oct 16, 2017 at 11:38 PM, Eugene Kirpichov <
[email protected]> wrote:
Agreed that polling Dataflow users makes sense, though I think they are
very unlikely to have concerns, because unlike Spark/Flink users they
wouldn't have a "cluster" that they need to migrate to a new JVM: they'd
only need to recompile their pipelines with JDK 8.
On Mon, Oct 16, 2017 at 11:21 PM Reuven Lax <[email protected]>
wrote:
I think the Flink and Spark runner maintainers can weigh in here;
given
that both of those systems are moving to Java 8, I doubt they will
have
concerns. Same is true for the Dataflow runner - we should probably
poll
Dataflow users to make sure this is not a problem for any of them.
On Mon, Oct 16, 2017 at 11:05 PM, Eugene Kirpichov <
[email protected]> wrote:
Reuven - do you mean e.g. a poll on the Flink mailing list asking
whether
there are Flink users who use Beam with Java 7? Or just people who
use
Flink with Java 7? (the latter question I'd assume was settled by
the
poll
about making Flink itself Java8-only?)
On Mon, Oct 16, 2017 at 10:32 PM Reuven Lax
<[email protected]>
wrote:
I don't know if a vote in @user is sufficient, as it's
unfortunately
not
representative of all Beam users. I think the runner communities
should
be
polled as well (though I suspect the answer will be the same,
that we
can
go ahead and deprecate Java 7 support).
On Mon, Oct 16, 2017 at 4:50 PM, Eugene Kirpichov <
[email protected]> wrote:
Yeah, a vote on user@ sounds like a good idea. Ismaël, would
you
be
interested in driving this process, since you're already
working on
Java9
support and hence you have a good understanding of what's
involved
in a
JDK
version migration for a large project?
As due diligence, we can look at how the other data processing
systems
handled dropping Java7.
Flink:
JIRA https://issues.apache.org/jira/browse/FLINK-7242
Discussion
http://apache-flink-user-mailing-list-archive.2336050.
n4.nabble.com/POLL-Who-still-uses-Java-7-with-Flink-
td12216.html
They also did a Twitter poll in addition to the mailing list
poll,
which
seems like a good idea.
Note that Flink had a number of strong reasons for dropping
Java7
that
do
not apply in Beam.
Spark:
JIRA https://issues.apache.org/jira/browse/SPARK-19493
Discussion
http://apache-spark-developers-list.1001551.n3.
nabble.com/discuss-ending-
support-for-Java-7-in-Spark-2-0-td16808.html
(I
couldn't find a formal poll of the user list rather than
developer
list)
Hadoop:
Hadoop 3.0 is Java8-only, but I couldn't quickly find a
discussion
of
where
that decision was made.
https://lists.apache.org/thread.html/e5c8085ada2cca47027b63f
5439839
731a392335770386e10895be06@1444091751@%3Cmapreduce-dev.
hadoop.apache.org%3E
might
be it.
Kafka is considering it:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
118%3A+Drop+Support+for+Java+7+in+Kafka+0.11
and
quotes a number of other open-source projects that have switched
http://markmail.org/message/l7s276y3xkga2eqf
So basically these projects all did a mailing list poll, and one
did
also a
twitter poll.
Beam has the advantage of being a relatively young project with
perhaps a
smaller base of users entrenched in using old versions of Java;
moreover,
Java version would matter only for the smaller subset of users
who
use
Beam
Spark/Flink/Apex/.. runners (as opposed to Cloud Dataflow),
which
is
likely
an even more "early adopter"-ish group of users, as these
runners
generally
receive less support.
It may be a good idea to have at least 1 release pass between
announcing
the intention to drop Java8 and actually dropping it (e.g. if we
decided
it
now, then 2.4 would drop Java7). Also, we could start by
switching
tests
to
compile/run with java8 (Maven allows this). This is, I think,
pretty
much
safe to do immediately.
On Mon, Oct 16, 2017 at 7:35 AM Ismaël Mejía <[email protected]
wrote:
Any progress on this? What is the proposed way to validate if
users
are still interested on Java 7? A vote on user or something
different?
On Wed, Sep 27, 2017 at 7:59 PM, Kenneth Knowles
<[email protected]
wrote:
Agree with polling Beam users as well as each runner's
community
in
aggregate.
On Wed, Sep 27, 2017 at 9:44 AM, Jean-Baptiste Onofré <
[email protected]
wrote:
Definitely agree
On 09/27/2017 06:00 PM, Robert Bradshaw wrote:
I also think that it's time to seriously consider dropping
support
for
Java 7.
On Tue, Sep 26, 2017 at 9:14 PM, Daniel Oliveira
<[email protected]> wrote:
Yes, just as Ismaël said it's a compilation blocker right
now
despite
that
(I believe) we don't use the extension that's breaking.
As for other ways to solve this, if there is a way to
avoid
compiling
the
advanced features of AutoValue that might be worth a
try. We
could
also
try
to get a release of AutoValue with the fix that works in
Java
7.
However
I
feel that slowly moving over to Java 8 is the most
future-proof
solution
if
it's possible.
On Tue, Sep 26, 2017 at 2:47 PM, Ismaël Mejía <
[email protected]>
wrote:
The current issue is that compilation fails on master
because
beam's
parent pom is configured to fail if it finds warnings):
<compiler.error.flag>-Werror</compiler.error.flag>
However if you remove that line from the parent pom the
compilation
passes.
Of course this does not mean that everything is solved
for
Java
9,
there are some tests that break and other issues
because of
other
plugins and dependencies (e.g. bytebuddy), but those are
not
part
of
this discussion.
On Tue, Sep 26, 2017 at 11:38 PM, Eugene Kirpichov
<[email protected]> wrote:
AFAIK we don't use any advanced capabilities of
AutoValue.
Does
that
mean
this issue is moot? I didn't quite understand from your
email
whether
it
is
a compilation blocker for Beam or not.
On Tue, Sep 26, 2017 at 2:32 PM Ismaël Mejía <
[email protected]
wrote:
Great that you are also working on this too Daniel and
thanks
for
bringing this subject to the mailing list, I was
waiting
to
my
return
to office next week, but you did it first :)
Eugene for reference (This is the issue on the
migration
to
Java
9),
notice that here the goal is first that beam passes
mvn
clean
install
with pure Java 9 (and also add this to jenkins), not
to
rewrite
anything to use the new stuff (e.g. modules):
https://issues.apache.org/jira/browse/BEAM-2530
Eugene can you also PTAL at the AutoValue issue, more
details
on
the
issue, this is a warning so I don't know if it is
really
critical
in
particular because we are not using Memoization (do
we?).
https://github.com/google/auto/issues/503
https://github.com/google/auto/commit/71514081f2ca6fb4ead2b7f0a25f5d
02247b8532
Wouldn't the easiest way be that you guys convince the
google.auto
guys to generate that simple fix in a Java 7
compatible
way
and
'voila' ?
However I agree that moving to Java 8 is an excellent
idea
and
as
Eugene mentions there is less friction now since most
projects
are
moving, only pending issue are existing clusters on
java
7
in
the
hadoop world, but those are less frequent now. Anyway
this
discussion
is really important (maybe even worth a vote). Because
moving
to
Java
8 will allow us also to move some of the dependencies
that
we
are
keeping in old versions and in general to move
forward.
What do the others think ?
On Tue, Sep 26, 2017 at 11:09 PM, Eugene Kirpichov
<[email protected]> wrote:
Very excited to hear that there's work on JDK9
support -
is
there
a
public
description of the plans for this work somewhere?
In general, Beam could probably drop Java7 support
altogether
at
some
point
soon: Java7 has reached end-of-life (i.e. it's not
receiving
even
security
patches) 2 years ago, and all major players in the
data
processing
ecosystem have dropped Java7 support (Spark, Flink,
Hadoop),
so
I
presume
the demand for Java7 support in the data processing
industry
is
low.
By
the
way: would a Java8 migration be in the scope of your
work
in
general?
However, until we say that Beam requires Java8, what
would
be
the
implications of using a version of AutoValue that can
only
be
compiled
with
Java8? Are you saying that this is simply a matter
of a
compiler
bug,
and
if we use a Java8 compiler but configured to use source
and
target
versions
of 1.7 and using bootclasspath of rt.jar from 1.7,
then
the
resulting
Beam
artifacts will be usable by people who don't have
Java8?
On Tue, Sep 26, 2017 at 1:53 PM Daniel Oliveira
<[email protected]> wrote:
So I've been working on JDK 9 support for Beam, and I
have a
bug
in
AutoValue that can be fixed by updating our
AutoValue
dependency
to
the
latest. The problem is that AutoValue from 1.5+ seems
to
be
banned
for
Beam
due to requiring Java 8 compilers. However, it should
still
be
possible
to
compile and execute Java 7 code from the Java 8
compiler
by
building
with
the correct arguments. So the fix to this bug would
essentially
require
Java 8 compilers even for compiling Java 7 code.
Does anyone need to use Java 7 compilers? Because if
not
I
would
like to
continue with this fix.
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
