Hi all,
on a performance and functional regression stand point I see no regression:
I looked at nexmark graphs "output pcollection size" and "execution time"
around release cut date on dataflow, spark,
flink and direct runner in batch and streaming modes. There seems to be no
regression.
Etienne
Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen a écrit :
> The SparkRunner validation test (here:
> https://beam.apache.org/contribute/release-guide/#run-validation-tests) passes
> on my machine. It looks like we are likely missing test coverage where
> Romain is hitting issues.
> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay <[email protected]> wrote:
> > Could anyone else help with looking at these issues earlier?
> > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
> > <[email protected]> wrote:
> > > Im running this main [1] through this IT [2]. Was working fine since ~1
> > > year but 2.7.0 broke it. Didnt investigate
> > > more but can have a look later this month if it helps.
> > > [1]
> > > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-clust
> > > er/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
> > > [2]
> > > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-clust
> > > er/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
> > >
> > > Le mar. 11 sept. 2018 20:54, Charles Chen <[email protected]> a écrit :
> > > > Romain: can you give more details on the failure you're encountering,
> > > > i.e. how you are performing this
> > > > validation?
> > > >
> > > > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré
> > > > <[email protected]> wrote:
> > > > > Hi,
> > > > >
> > > > >
> > > > >
> > > > > weird, I didn't have it on Beam samples. Let me try to reproduce and
> > > > > I
> > > > >
> > > > > will create the Jira.
> > > > >
> > > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > JB
> > > > >
> > > > >
> > > > >
> > > > > On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> > > > >
> > > > > > -1, seems spark integration is broken (tested with spark 2.3.1 and
> > > > > > 2.2.1):
> > > > >
> > > > > >
> > > > >
> > > > > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in stage 0.0
> > > > > > (TID 0, RMANNIBUCAU, executor 0):
> > > > > java.lang.ClassCastException: cannot assign instance of
> > > > > scala.collection.immutable.List$SerializationProxy to
> > > > > fieldorg.apache.spark.rdd.RDD.org
> > > > > <http://org.apache.spark.rdd.RDD.org>$apache$spark$rdd$RDD$$dependencies_
> > > > > of
> > > > > type scala.collection.Seq in instance of
> > > > > org.apache.spark.rdd.MapPartitionsRDD
> > > > >
> > > > > > at
> > > > > > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > > Also the issue Lukasz identified is important even if workarounds
> > > > > > can be
> > > > >
> > > > > > put in place so +1 to fix it as well if possible.
> > > > >
> > > > > >
> > > > >
> > > > > > Romain Manni-Bucau
> > > > >
> > > > > > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > > >
> > > > > > <https://rmannibucau.metawerx.net/>t; | Old Blog
> > > > >
> > > > > > <http://rmannibucau.wordpress.com> | Github
> > > > >
> > > > > > <https://github.com/rmannibucau>t; | LinkedIn
> > > > >
> > > > > > <https://www.linkedin.com/in/rmannibucau>t; | Book
> > > > >
> > > > > > <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik <[email protected]
> > > > >
> > > > > > <mailto:[email protected]>> a écrit :
> > > > >
> > > > > >
> > > > >
> > > > > > I found an issue where we are no longer packaging the pom.xml
> > > > > > within
> > > > >
> > > > > > the artifact jars at META-INF/maven/groupId/artifactId. More
> > > > > > details
> > > > >
> > > > > > in https://issues.apache.org/jira/browse/BEAM-5351. I wouldn't
> > > > >
> > > > > > consider this a blocker but it was an easy fix
> > > > >
> > > > > > (https://github.com/apache/beam/pull/6358) and users may rely
> > > > > > on the
> > > > >
> > > > > > pom.xml.
> > > > >
> > > > > >
> > > > >
> > > > > > Should we recut the release candidate to include this?
> > > > >
> > > > > >
> > > > >
> > > > > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré
> > > > >
> > > > > > <[email protected] <mailto:[email protected]>> wrote:
> > > > >
> > > > > >
> > > > >
> > > > > > +1 (binding)
> > > > >
> > > > > >
> > > > >
> > > > > > Tested successfully on Beam Samples.
> > > > >
> > > > > >
> > > > >
> > > > > > Thanks !
> > > > >
> > > > > >
> > > > >
> > > > > > Regards
> > > > >
> > > > > > JB
> > > > >
> > > > > >
> > > > >
> > > > > > On 07/09/2018 23:56, Charles Chen wrote:
> > > > >
> > > > > > > Hi everyone,
> > > > >
> > > > > > >
> > > > >
> > > > > > > Please review and vote on the release candidate #1 for
> > > > > > the
> > > > >
> > > > > > version
> > > > >
> > > > > > > 2.7.0, as follows:
> > > > >
> > > > > > > [ ] +1, Approve the release
> > > > >
> > > > > > > [ ] -1, Do not approve the release (please provide
> > > > > > specific
> > > > >
> > > > > > comments)
> > > > >
> > > > > > >
> > > > >
> > > > > > > The complete staging area is available for your review,
> > > > > > which
> > > > >
> > > > > > includes:
> > > > >
> > > > > > > * JIRA release notes [1],
> > > > >
> > > > > > > * the official Apache source release to be deployed to
> > > > >
> > > > > > dist.apache.org <http://dist.apache.org>
> > > > >
> > > > > > > <http://dist.apache.org> [2], which is signed with the
> > > > > > key with
> > > > >
> > > > > > > fingerprint 45C60AAAD115F560 [3],
> > > > >
> > > > > > > * all artifacts to be deployed to the Maven Central
> > > > >
> > > > > > Repository [4],
> > > > >
> > > > > > > * source code tag "v2.7.0-RC1" [5],
> > > > >
> > > > > > > * website pull request listing the release and
> > > > > > publishing the API
> > > > >
> > > > > > > reference manual [6].
> > > > >
> > > > > > > * Java artifacts were built with Gradle 4.8 and OpenJDK
> > > > >
> > > > > > > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > > > >
> > > > > > > * Python artifacts are deployed along with the source
> > > > > > release
> > > > >
> > > > > > to the
> > > > >
> > > > > > > dist.apache.org <http://dist.apache.org>
> > > > >
> > > > > > <http://dist.apache.org> [2].
> > > > >
> > > > > > >
> > > > >
> > > > > > > The vote will be open for at least 72 hours. It is
> > > > > > adopted by
> > > > >
> > > > > > majority
> > > > >
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > > >
> > > > >
> > > > > > > Thanks,
> > > > >
> > > > > > > Charles
> > > > >
> > > > > > >
> > > > >
> > > > > > > [1]
> > > > >
> > > > > > >
> > > > >
> > > > > >
> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > > > >
> > > > > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > > > >
> > > > > > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > > > >
> > > > > > > [4]
> > > > >
> > > > > >
> > > > > > https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > > > >
> > > > > > > [5] https://github.com/apache/beam/tree/v2.7.0-RC1
> > > > >
> > > > > > > [6] https://github.com/apache/beam-site/pull/549
> > > > >
> > > > > >
> > > > >
> > > > > > --
> > > > >
> > > > > > Jean-Baptiste Onofré
> > > > >
> > > > > > [email protected] <mailto:[email protected]>
> > > > >
> > > > > > http://blog.nanthrax.net
> > > > >
> > > > > > Talend - http://www.talend.com
> > > > >
> > > > > >
> > > > >
