[jira] [Created] (FLINK-11199) Support interactive programming in Flink

[Jiangjie Qin (JIRA)]

Jiangjie Qin created FLINK-11199:


 Summary: Support interactive programming in Flink
 Key: FLINK-11199
 URL: https://issues.apache.org/jira/browse/FLINK-11199
 Project: Flink
  Issue Type: New Feature
Affects Versions: 2.0.0
Reporter: Jiangjie Qin


Generally speaking, Flink applications may consist of one or more jobs, and the 
jobs may want to share the data with others. In Flink, the jobs in the same 
application are independent and share nothing among themselves. If a Flink 
application involves several sequential steps, each step (as an independent 
job) will have to write its intermediate results to an external sink, so that 
its results can be used by the following step (job) as sources. Although this 
works, that means users have to provide external storage and explicitly 
write/read from it. 

 

To improve the user experience, we would like to propose adding support of 
interactive programming in Flink. We would like to start with Table API. And 
the discussion thread is following:

https://docs.google.com/document/d/1d4T2zTyfe7hdncEUAxrlNOYr4e5IMNEZLyqSuuswkA0/edit?ts=5bf639d9#



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11200) Port DataView classes to flink-table-common

[Timo Walther (JIRA)]

Timo Walther created FLINK-11200:


 Summary: Port DataView classes to flink-table-common
 Key: FLINK-11200
 URL: https://issues.apache.org/jira/browse/FLINK-11200
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Reporter: Timo Walther


{{DataView}} are used within aggregate functions for more efficient state 
management. Logically, they should have ported in FLINK-10689.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11202) log split file, every job has one log file

[chauncy (JIRA)]

chauncy created FLINK-11202:
---

 Summary: log split file, every job has one log file 
 Key: FLINK-11202
 URL: https://issues.apache.org/jira/browse/FLINK-11202
 Project: Flink
  Issue Type: Improvement
Reporter: chauncy


find bug is difficult due totask manager  and job manager's  log into one 
big log file  with standalone cluster env  , i think each job has a log file is 
profile 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11201) flink-test-utils dependency issue

[eugen yushin (JIRA)]

eugen yushin created FLINK-11201:


 Summary: flink-test-utils dependency issue
 Key: FLINK-11201
 URL: https://issues.apache.org/jira/browse/FLINK-11201
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.7.0
Reporter: eugen yushin


Starting with Flink 1.7, there's lack of 
`runtime.testutils.MiniClusterResource` class in `flink-test-utils` 
distribution.

Steps to reproduce (Scala code)

build.sbt
{code}
name := "flink-17-test-issue"

organization := "x.y.z"
scalaVersion := "2.11.12"
val flinkVersion = "1.7.0"

libraryDependencies ++= Seq(
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
  "org.scalatest" %% "scalatest" % "3.0.5" % Test,
  "org.apache.flink" %% "flink-test-utils" % flinkVersion % Test
//  ,"org.apache.flink" %% "flink-runtime" % flinkVersion % Test classifier 
Artifact.TestsClassifier
)
{code}

test class:
{code}
class SimpleTest extends AbstractTestBase with FlatSpecLike {
  implicit val env: StreamExecutionEnvironment = 
StreamExecutionEnvironment.getExecutionEnvironment
  env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
  env.setParallelism(1)
  env.setRestartStrategy(RestartStrategies.noRestart())

  "SimpleTest" should "work" in {
val inputDs = env.fromElements(1,2,3)

inputDs.print()

env.execute()
  }
}
{code}

Results in:
{code}
A needed class was not found. This could be due to an error in your runpath. 
Missing class: org/apache/flink/runtime/testutils/MiniClusterResource
java.lang.NoClassDefFoundError: 
org/apache/flink/runtime/testutils/MiniClusterResource
...
Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.runtime.testutils.MiniClusterResource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 31 more
{code}

This can be fixed by flink-runtime distribution with test classifier.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11203) FunctionContext of AggregateFunction will not be initialized for window GroupBy

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-11203:
---

 Summary: FunctionContext of AggregateFunction will not be 
initialized for window GroupBy
 Key: FLINK-11203
 URL: https://issues.apache.org/jira/browse/FLINK-11203
 Project: Flink
  Issue Type: Improvement
  Components: Table API & SQL
Reporter: Hequn Cheng
Assignee: Hequn Cheng


Currently, in tableApi/SQL, the implementable of aggregation of group window is 
base on the WindowStream and 
{{org.apache.flink.api.common.functions.AggregateFunction}}. 
Due to FLINK-11198, metrics cannot be accessed within 
{{org.apache.flink.table.functions.AggregateFunction}} either. It would be nice 
if we support metrics for both of them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11204) flink start-scala-shell.sh do not work in security mode with kerberos authentication.

[kelun wang (JIRA)]

kelun wang created FLINK-11204:
--

 Summary: flink start-scala-shell.sh do not work in security mode 
with kerberos authentication.
 Key: FLINK-11204
 URL: https://issues.apache.org/jira/browse/FLINK-11204
 Project: Flink
  Issue Type: Bug
  Components: Scala Shell
Affects Versions: 1.7.0, 1.6.2, 1.5.5, 1.4.2, 1.3.3
Reporter: kelun wang


Hello,

When using flink start-scala-shell.sh in a cluster with kerberos credential, 
the script do not supports Kerberos authentication, errors like below will 
occur:

1) Fail to deploy Yarn cluster.

 
{code:java}
start-scala-shell.sh yarn -n 3
Exception in thread "main" java.lang.RuntimeException: Error deploying the YARN 
cluster
at 
org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:594)
at 
org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:81)
at 
org.apache.flink.api.scala.FlinkShell$.deployNewYarnCluster(FlinkShell.scala:256)
at 
org.apache.flink.api.scala.FlinkShell$.fetchConnectionInfo(FlinkShell.scala:165)
at org.apache.flink.api.scala.FlinkShell$.liftedTree1$1(FlinkShell.scala:189)
at org.apache.flink.api.scala.FlinkShell$.startShell(FlinkShell.scala:188)
at org.apache.flink.api.scala.FlinkShell$.main(FlinkShell.scala:137)
at org.apache.flink.api.scala.FlinkShell.main(FlinkShell.scala)
Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm
at 
org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
at 
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:318)
at 
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:303)
at 
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:391)
at 
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:385)
at 
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:384)
at 
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:351)
... 8 more
{code}
 

2)Fail to fetch deployed a Flink cluster, when using the following command :

bin/start-scala-shell.sh yarn

 
{code:java}
def fetchDeployedYarnClusterInfo(
configuration: Configuration,
configurationDirectory: String) = {


val args = ArrayBuffer[String](
"-m", "yarn-cluster"
)
{code}
when fething deployed yarn cluster, with param "-m yarn-cluster" it will create 
new one, but has no "-n", still fail.

 
 
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Question about Flink optimizer on Stream API

[Felipe Gutierrez]

Hi,

I was reading some FLIP documents related to the new design of the Flink
Schedule [1] and unification of batch and stream [2]. Then I created two
different programs to learn how Flink optimizes the Query Plan in Batch and
in Stream mode (and how much further it goes). One using batch [3] and one
using Stream [4]. During the code debugging and also as it is depicted on
the document [2], the batch program uses the
org.apache.flink.optimizer.Optimizer class which generates a
"org.apache.flink.optimizer.plan.OptimizedPlan" while stream program uses
the "org.apache.flink.streaming.api.graph.StreamGraph" and every
transformation inside the packet
"org.apache.flink.streaming.api.transformations".

When I am showing the execution plan with "env.getExecutionPlan()" I see
exactly I have written on the Flink program (which it is expected).
However, I was looking for where I can see the optimized plan. I mean
decisions of operators reordering based on cost or statistics. For batch I
could find the "org.apache.flink.optimizer.costs.CostEstimator" and
"org.apache.flink.optimizer.DataStatistics". But for Stream I only found
the creation of the plan. How can I debug that? Or have a better
understanding of what Flink is doing. Do you advise me to read some other
reference about this?

Kind Regards,
Felipe

[1] Group-aware scheduling for Flink -
https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit#heading=h.k15nfgsa5bnk
[2] Unified Core API for Streaming and Batch -
https://docs.google.com/document/d/1G0NUIaaNJvT6CMrNCP6dRXGv88xNhDQqZFrQEuJ0rVU/edit#
[3]
https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/batch/MatrixMultiplication.java
[4]
https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/SensorsReadingMqttJoinQEP.java

*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
*

[jira] [Created] (FLINK-11205) Task Manager Metaspace Memory Leak

[Nawaid Shamim (JIRA)]

Nawaid Shamim created FLINK-11205:
-

 Summary: Task Manager Metaspace Memory Leak 
 Key: FLINK-11205
 URL: https://issues.apache.org/jira/browse/FLINK-11205
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.7.0, 1.6.2, 1.5.5
Reporter: Nawaid Shamim
 Attachments: Screenshot 2018-12-18 at 12.14.11.png

Job Restarts causes task manager to dynamically load duplicate classes. 
Metaspace is unbounded and grows with every restart. YARN aggressively kill 
such containers but this affect is immediately seems on different task manager 
which results in death spiral.

!Screenshot 2018-12-18 at 12.14.11.png!width=480!

Task Manager uses dynamic loader as described in 
[https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html]
{quote}
*YARN*

YARN classloading differs between single job deployments and sessions:
 * When submitting a Flink job/application directly to YARN (via {{bin/flink 
run -m yarn-cluster ...}}), dedicated TaskManagers and JobManagers are started 
for that job. Those JVMs have both Flink framework classes and user code 
classes in the Java classpath. That means that there is _no dynamic 
classloading_ involved in that case.

 * When starting a YARN session, the JobManagers and TaskManagers are started 
with the Flink framework classes in the classpath. The classes from all jobs 
that are submitted against the session are loaded dynamically.
{quote}

The above is not entirely true specially when you set {{-yD 
classloader.resolve-order=parent-first}} . We also above observed the above 
behaviour when submitting a Flink job/application directly to YARN (via 
{{bin/flink run -m yarn-cluster ...}}).




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [SURVEY] Usage of flink-python and flink-streaming-python

[Till Rohrmann]

Thanks a lot for the feedback for this survey. I will close it now since 6
days have passed without new activity.

To me it seems that we currently don't have many users who use flink-python
or flink-streaming-python because of their limitations (mentioned in the
survey by Xianda). This information might be useful when discussing Flink's
future Python strategy and whether to continue supporting flink-python and
flink-streaming-python in the future.

Cheers,
Till

On Thu, Dec 13, 2018 at 10:50 AM Stephan Ewen  wrote:

> You are right. Let's refocus this on the python user survey and spin out
> another thread.
>
> On Thu, Dec 13, 2018 at 9:56 AM Xianda Ke  wrote:
>
> > Hi Folks,
> > To avoid polluting the survey thread with discussions, we started
> separate
> > thread and maybe we can continue the discussion over there.
> >
> > Regards,
> > Xianda
> >
> > On Wed, Dec 12, 2018 at 3:34 AM Stephan Ewen  wrote:
> >
> > > I like that we are having a general discussion about how to use Python
> > and
> > > Flink together in the future.
> > > The current python support has some shortcomings that were mentioned
> > > before, so we clearly need something better.
> > >
> > > Parts of the community have worked together with the Apache Beam
> project,
> > > which is pretty far in adding a portability layer to support Python.
> > > Before we dive deep into a design proposal for a new Python API in
> > Flink, I
> > > think we should figure out in which general direction Python support
> > should
> > > go.
> > >
> > > *Option (1): Language portability via Apache Beam*
> > >
> > > Pro:
> > >   - already exists to a large extend and already has users
> > >   - portability layer offers other languages in addition to python. Go
> is
> > > in the making, NodeJS has been speculated, etc.
> > >   - collaboration with another project / community which means more
> > > manpower and exposure. Beam currently has a strong focus on Flink as a
> > > runner for Python.
> > >   - Python API is used for existing ML libraries from the TensorFlow
> > > ecosystem
> > >
> > > Con:
> > >   - Not Flink's API. Python users need to learn the syntax of another
> API
> > > (Python API is inherently different, but even more different here).
> > >
> > > *Option (2): Implement own Python API*
> > >
> > > Pro:
> > >   - Python API will be closer to Flink Java / Scala APIs
> > >
> > > Con:
> > >   - We will only have Python.
> > >   - Need to to rebuild the Python language bridge (significant work to
> > get
> > > stable)
> > >   - might lose tight collaboration with Beam and the other parties in
> > Beam
> > >   - not benefiting from Beam's ecosystem
> > >
> > > *Option (3): **Implement own portability layer*
> > >
> > > Pro
> > >   - Flexibility to align APIs across languages within Flink ecosystem
> > >
> > > Con
> > >   - A lot of work (for context, to get this feature complete, Beam has
> > > worked on that for a year now)
> > >   - Replicating work that already exists
> > >   - good chance to lose tight collaboration with Beam and parties in
> that
> > > project
> > >   - not benefiting from Beam's ecosystem
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Tue, Dec 11, 2018 at 3:38 PM Thomas Weise  wrote:
> > >
> > > > Did you take a look at Apache Beam? It already provides a
> comprehensive
> > > > Python SDK and can be used with Flink:
> > > > https://beam.apache.org/roadmap/portability/#python-on-flink
> > > >
> > > > We are using it at Lyft for Python streaming pipelines.
> > > >
> > > > Thomas
> > > >
> > > > On Tue, Dec 11, 2018 at 5:54 AM Xianda Ke 
> wrote:
> > > >
> > > > > Hi Till,
> > > > >
> > > > > 1. So far as I know, most of the users at Alibaba are using SQL.
> > Some
> > > of
> > > > > users at Alibaba want integrated python libraries with Flink for
> > > > streaming
> > > > > processing, and Jython is unusable.
> > > > >
> > > > > 2. Python UDFs for SQL:
> > > > > * declaring python UDF based on Alibaba's internal DDL syntax.
> > > > > * start a Python process in open()
> > > > > * communicate with JVM process via Socket.
> > > > > * Yes, it support python libraries, users can upload
> virutalenv/conda
> > > > > Python runtime
> > > > >
> > > > > 3. We've draft a design doc for Python API
> > > > >  [DISCUSS] Flink Python API
> > > > > <
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JNGWdLwbo_btq9RVrc1PjWJV3lYUgPvK0uEWDIfVNJI/edit?usp=drive_web
> > > > > >
> > > > >
> > > > > Python UDF for SQL is not discussed in this documentation, we'll
> > > create a
> > > > > new proposal when the SQL DDL is ready.
> > > > >
> > > > > On Mon, Dec 10, 2018 at 9:52 PM Till Rohrmann <
> trohrm...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Xianda,
> > > > > >
> > > > > > thanks for sharing this detailed feedback. Do I understand you
> > > > correctly
> > > > > > that flink-python and flink-streaming-python are not usable for
> the
> > > use
> > > > > > cases at Alibaba atm?
> > > > > >
> > > > > > Could you sha

Re: [DISCUSS] Proposal of external shuffle service

[Andrey Zagrebin]

Hi Zhijiang,

Thanks for detailed answers! I am glad we are on the same page.

I spent some time on thinking more about our concerns and decided to make more 
suggestions for the discussion.
At the end, I also gathered some points related to possible extensions of 
shuffle API to verify that the final pluggable design can support them later 
with less changes.

It might make sense for shuffle implementation to have component running on 
both JM and TM sides.
JM has a global view of what is happening and can interact with shuffling 
system independently on whether tasks are running or not. The component 
services could internally further communicate to each other outside of existing 
JM/TM APIs, depending on shuffle implementation.
It could help later with partition global life cycle management and cleanup.

Moreover, if we decide to use some ShuffleDeploymentDescriptor instead of 
ResultPartitionLocation or factories to instantiate Readers and Writers, they 
can be created in Task Executor. 
JM is probably not interested in this concern. ShuffleDeploymentDescriptor can 
be specific to shuffle implementation, like factories, and contain specific 
shuffle config for task side.

1. Configuration:

interface ShuffleManager {
  ShuffleMaster createMaster(Configuration flinkConfig);
  ShuffleService createService(Configuration flinkConfig);
}

ShuffleManager is a factory for ShuffleMaster (JM side) and ShuffleService (TM 
side).
Flink config could also contain specific shuffle configuration, like port etc.

Class which implements ShuffleManager in Flink cluster config, default is what 
we have now (can be the first step)

2. Job master side

class PartitionShuffleDescriptor {
  JobID, ExecutionAttemptID, ResultPartitionType, ResultPartitionLocation, 
TaskManagerLocation, etc
  later possibly ShuffleType/Descriptor to choose from available shuffle 
implementations
}

PartitionShuffleDescriptor contains all abstract information which JM can 
provide from the job/execution graph.
ResultPartitionType and ResultPartitionLocation are derived from graph and 
execution mode, 
so I think they are rather general parameters for any shuffle service and do 
not belong to particular shuffle implementation.

interface ShuffleMaster extends AutoClosable {
  ShuffleDeploymentDescriptor registerPartition(PartitionShuffleDescriptor);
  void deregisterPartition(PartionShuffleDescriptor);
}

JM process creates ShuffleMaster from configured per cluster ShuffleManager. JM 
is responsible for its life cycle.
ShuffleMaster is a global manager for partitions.
JM creates PartitionShuffleDescriptor and uses ShuffleMaster to register 
partition, e.g. when producer is deployed. 
ShuffleMaster transforms abstract PartitionShuffleDescriptor into a specific 
ShuffleDeploymentDescriptor.

ShuffleDeploymentDescriptor is put into ResultPartitionDeploymentDescriptor and 
InputGateDeploymentDescriptor.
It can contain specific partition config for ShuffleService on TM side to serve 
record readers and writers. 
If it is channel-based then further break down to channel configs.

Special UnknownShuffleDeploymentDescriptor could be used for eager deployment 
when task input is unknown yet.

Later, we could add an option to release partition globally by deregistering it 
with the ShuffleMaster. e.g. to clean it up.

3. Task executor side

interface ShuffleService extends AutoClosable {
  ResultPartitionWriter 
createResultPartitionWriter(ResultPartitionDeploymentDescriptor);
  InputGate createInputGate(InputGateDeploymentDescriptor);
}

TM process creates ShuffleService from configured per cluster ShuffleManager. 
TM is responsible for its life cycle.
ShuffleService could substitute NetworkEnvironment in TaskManagerServices.

4. Later extensions

4.1 Per job/job edge config

To keep jobs cluster independent, we could introduce abstract predefined 
ShuffleType’s or descriptors
for job developer to set it per job or job edge. The types are 
cluster-independent.

Cluster config could contain provided ShuffleManager implementation class for 
each supported ShuffleType or fallback to default for some types.

Instead of one ShuffleMaster/ShuffleService, JM/TM could have keep a registry 
of ShuffleMaster/ShuffleService’s per ShuffleType.

4.2 Delay TM shutdown until all local partitions have been consumed

JM could keep separately state of partition life cycle (e.g. in job state, HA). 
The task executor is to shutdown (e.g. after timeout in yarn) if all its tasks 
are done and all local partitions are consumed. If there are no local 
partitions then it can shutdown immediately. Whether JM should check that all 
produced by TM partitions are consumed is a feature of ShuffleManager. This 
could be done by calling some ShuffleManager.getFeatures() interface method.

4.3 Speed up failover

If partition is computed JM could reuse it as mention in fine-grained shuffle 
system design. Whether the partition is still available after task or task 
executor crash is also 

Re: [VOTE] Release 1.6.3, release candidate #1

[Aljoscha Krettek]

+1

- signatures/hashes are ok
- verified that the log contains no suspicious output when running a local 
cluster

> On 18. Dec 2018, at 14:31, Chesnay Schepler  wrote:
> 
> +1
> 
> - signatures ok
> - src contains no binaries
> - binary not missing any jars
> - tag exists
> - release notes classification/names seem appropriate
> - maven artifacts not missing any jars
> 
> On 18.12.2018 11:15, Tzu-Li (Gordon) Tai wrote:
>> Hi everyone,
>> 
>> Please review and vote on the release candidate #1 for the version 1.6.3, as 
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be 
>> deployed to dist.apache.org [2], which are signed with the key with 
>> fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag “release-1.6.3-rc1” [5],
>> * website pull request listing the new release and adding announcement blog 
>> post [6].
>> 
>> The vote will be open for at least 72 hours. It is adopted by majority 
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Gordon
>> 
>> [1] 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344314
>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.6.3-rc1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1202
>> [5] 
>> https://gitbox.apache.org/repos/asf?p=flink.git;a=commit;h=54e6cde28493baf35315946fd023ecbe692c95d8
>> [6] https://github.com/apache/flink-web/pull/141
>> 
>> 
> 

Re: [VOTE] Release 1.5.6, release candidate #1

[Aljoscha Krettek]

+1

- signatures/hashes are ok
- manually checked the logs after running an example on a local cluster

There is this exception in the client log when running without Hadoop in the 
classpath:

2018-12-19 18:34:54,876 WARN  org.apache.flink.client.cli.CliFrontend   
- Could not load CLI class 
org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at 
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1218)
at 
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1174)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1100)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more

This is not a problem, though it might seem strange to users.

> On 18. Dec 2018, at 16:52, Chesnay Schepler  wrote:
> 
> +1
> 
> - signing key present in KEYS
> - signatures ok
> - src contains no binaries
> - binary not missing any jars
> - tag exists
> - release notes classification/names seem appropriate
> - maven artifacts not missing any jars
> 
> On 18.12.2018 06:27, Thomas Weise wrote:
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version
>> 1.5.6, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to
>> be deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint D920A98C [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.5.6-rc1" [5].
>> 
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Thomas
>> 
>> [1]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344315
>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.5.6-rc1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1199/
>> [5]
>> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.6-rc1
>> 
> 

Re: [VOTE] Release 1.5.6, release candidate #1

[Timo Walther]

+1

- manually checked the commit diff and could not sport any issues
- run mvn clean verify locally with success
- run a couple of e2e tests locally with success

Thanks,
Timo

Am 19.12.18 um 18:36 schrieb Aljoscha Krettek:

+1

- signatures/hashes are ok
- manually checked the logs after running an example on a local cluster

There is this exception in the client log when running without Hadoop in the 
classpath:

2018-12-19 18:34:54,876 WARN  org.apache.flink.client.cli.CliFrontend   
- Could not load CLI class 
org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at 
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1218)
at 
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1174)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1100)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more

This is not a problem, though it might seem strange to users.


On 18. Dec 2018, at 16:52, Chesnay Schepler  wrote:

+1

- signing key present in KEYS
- signatures ok
- src contains no binaries
- binary not missing any jars
- tag exists
- release notes classification/names seem appropriate
- maven artifacts not missing any jars

On 18.12.2018 06:27, Thomas Weise wrote:

Hi everyone,
Please review and vote on the release candidate #1 for the version
1.5.6, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to
be deployed to dist.apache.org [2], which are signed with the key with
fingerprint D920A98C [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.5.6-rc1" [5].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Thomas

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344315
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.5.6-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1199/
[5]
https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.6-rc1

Re: [VOTE] Release 1.5.6, release candidate #1

[Thomas Weise]

+1

- tested staged maven artifacts and no Hadoop binary release with Beam


On Wed, Dec 19, 2018 at 10:10 AM Timo Walther  wrote:

> +1
>
> - manually checked the commit diff and could not sport any issues
> - run mvn clean verify locally with success
> - run a couple of e2e tests locally with success
>
> Thanks,
> Timo
>
> Am 19.12.18 um 18:36 schrieb Aljoscha Krettek:
> > +1
> >
> > - signatures/hashes are ok
> > - manually checked the logs after running an example on a local cluster
> >
> > There is this exception in the client log when running without Hadoop in
> the classpath:
> >
> > 2018-12-19 18:34:54,876 WARN  org.apache.flink.client.cli.CliFrontend
>- Could not load CLI class
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.
> > java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/exceptions/YarnException
> >   at java.lang.Class.forName0(Native Method)
> >   at java.lang.Class.forName(Class.java:264)
> >   at
> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1218)
> >   at
> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1174)
> >   at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1100)
> > Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.exceptions.YarnException
> >   at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> >   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> >   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >   ... 5 more
> >
> > This is not a problem, though it might seem strange to users.
> >
> >> On 18. Dec 2018, at 16:52, Chesnay Schepler  wrote:
> >>
> >> +1
> >>
> >> - signing key present in KEYS
> >> - signatures ok
> >> - src contains no binaries
> >> - binary not missing any jars
> >> - tag exists
> >> - release notes classification/names seem appropriate
> >> - maven artifacts not missing any jars
> >>
> >> On 18.12.2018 06:27, Thomas Weise wrote:
> >>> Hi everyone,
> >>> Please review and vote on the release candidate #1 for the version
> >>> 1.5.6, as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release and binary convenience releases to
> >>> be deployed to dist.apache.org [2], which are signed with the key with
> >>> fingerprint D920A98C [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag "release-1.5.6-rc1" [5].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Thomas
> >>>
> >>> [1]
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344315
> >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.5.6-rc1/
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1199/
> >>> [5]
> >>>
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.6-rc1
> >>>
>
>

Re: [VOTE] Release 1.6.3, release candidate #1

[Timo Walther]

+1

- manually checked the commit diff and could not spot any issues
- run mvn clean verify locally with success
- run a couple of e2e tests locally with success

Thanks,
Timo

Am 19.12.18 um 18:28 schrieb Aljoscha Krettek:

+1

- signatures/hashes are ok
- verified that the log contains no suspicious output when running a local 
cluster


On 18. Dec 2018, at 14:31, Chesnay Schepler  wrote:

+1

- signatures ok
- src contains no binaries
- binary not missing any jars
- tag exists
- release notes classification/names seem appropriate
- maven artifacts not missing any jars

On 18.12.2018 11:15, Tzu-Li (Gordon) Tai wrote:

Hi everyone,

Please review and vote on the release candidate #1 for the version 1.6.3, as 
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be 
deployed to dist.apache.org [2], which are signed with the key with fingerprint 
1C1E2394D3194E1944613488F320986D35C33D6A [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag “release-1.6.3-rc1” [5],
* website pull request listing the new release and adding announcement blog 
post [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.

Thanks,
Gordon

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344314
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.6.3-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1202
[5] 
https://gitbox.apache.org/repos/asf?p=flink.git;a=commit;h=54e6cde28493baf35315946fd023ecbe692c95d8
[6] https://github.com/apache/flink-web/pull/141