Re: [DISCUSS] Releasing Flink-shaded 7.0

[Tzu-Li (Gordon) Tai]

+1



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[DISCUSS] Features for Apache Flink 1.9.0

[Tzu-Li (Gordon) Tai]

Hi community,

Apache Flink 1.8.0 has been released a few weeks ago, so naturally, it’s
time to start thinking about what we want to aim for 1.9.0.

Kurt and I had collected some features that would be reasonable to consider
including for the next release, based on talking with various people as
well as observations from mailing list discussions and questions.

Note that having specific features listed here does not mean that no other
pull requests or topics will be reviewed. I am sure that there are other
ongoing efforts that we missed here and will likely make it as an
improvement or new feature in the next release. This discussion is merely
for bootstrapping a discussion for 1.9, as well as to give contributors an
idea of what the community is looking to focus on in the next couple of
weeks.

*Proposed features and focus*

In the previous major release, Apache Flink 1.8.0, the community had
prepared for some major Table & SQL additions from the Blink branch. With
this in mind, for the next release, it would be great to wind up those
efforts by merging in the Blink-based Table / SQL planner and runtime for
1.9.

Following Stephan’s previous thread [1] in the mailing list about features
in Blink, we should also start focusing on preparing for Blink’s other
several enhancements for batch execution. This includes resource
optimization, fine-grained failover, pluggable shuffle service, adapting
stream operators for batch execution, as well as better integration with
commonly used systems by batch executions such as Apache Hive.

Moreover, besides efforts related to the Blink merge, we would also like us
to work towards pushing forward some of the most discussed and anticipated
features by the community. Most of these had discussions in the mailing
lists that span multiple releases, and are also frequently brought up in
community events such as Flink Forward. This includes features such as
source event-time alignment and the source interface rework, a savepoint
connector that allows users to manipulate and query state in savepoints,
interactive programming, as well as terminating a job with a final
savepoint.

Last but not least, we have several existing contributions or discussions
for the ecosystem surrounding Flink, which we think is also very valuable
to try to merge in for 1.9. This includes a web UI rework (recently already
merged), active K8s integration, Google PubSub connector, native support
for the Protobuf format, Python support in the Table API, as well as
reworking Flink’s support for machine learning.

To wrap this up as a list of items, some of which already have JIRAs or
mailing list threads to track them:

   - Merge Blink runner for Table & SQL [2]
   -

  Restructure flink-table to separate API from core runtime
  -

  Make table planners pluggable
  -

  Rework Table / SQL type system to integrate better with the SQL
  standard [3]
  -

  Merge Blink planner and runtime for Table / SQL
  - Further preparations for more batch execution optimization from
   Blink
   -

  Dedicated scheduler component [4]
  -

  Fine grained failover for batch [5]
  -

  Selectable input stream operator [6]
  -

  Pluggable Shuffle Service [7]
  -

  FLIP-30: Unified Catalog API & Hive metastore integration [8]
  - Heavily anticipated / discussed features in the community
   -

  FLIP-27: Source interface rework [9]
  -

  Savepoint connector [10]
  -

  FLIP-34: Terminate / Suspend job with savepoint [11]
  -

  FLIP-36: Interactive Programming [12]
  - Ecosystem
   -

  Web UI rework [13]
  -

  Active K8s integration [14]
  -

  Google PubSub connector [15]
  -

  First-class Protobuf support [16]
  -

  FLIP-38: Python support in Table API [17]
  -

  FLIP-39: Flink ML pipeline and libraries on top of Table API [18]

*Suggested release timeline*

Apache Flink 1.8.0 was released earlier this month, so based on our usual
timely release schedule, we should aim for releasing 1.9.0 around mid to
end July.

Since it seems that this is going to be a fairly large release, to give the
community enough testing time, I propose that the feature freeze to be near
the end of June (8-9 weeks from now, probable June 28). This is of course a
ballpark estimation for now; we should follow-up with a separate thread
later in the release cycle to prepare contributors with an official feature
freeze date.

I’d also like to use this opportunity to propose myself and Kurt as the
release managers for 1.9.
AFAIK, we did not used to have 2 RMs for a single release in the past, but
1.9.0 is definitely quite ambitious so it would not hurt to have one more
on board :) Cheers, Gordon [1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-6-features-td22632.html

[2] https://issues.apache.org/jira/browse/FLINK-11439

[3] 

Re: The contradiction between event time and natural time from EventTimeTrigger

[Rong Rong]

Hi Zhipeng,

Please see my explanation below:

>From the default EventTimeTrigger source code, I found that only onElement
> method (will judge the watermark) and onEventTime method only have a chance
> to trigger TriggerResult.FIRE;
> Therefore, the default EventTimeTrigger is assumed and must be "never
> stop! The data stream" will have "correct" Real-time results, so as long as
> the interval between the two eventtimes is too large, greater than the time
> window interval, or the end time of the window has not arrived yet, there
> is no new data (flow interruption, neither element(onElement) nor
> eventtime(onEventtime)), then the latest time the output of the window must
> be untimely or non-real-time (if i use eventtime to do the bounded window
> aggregation of the stream, i must have the near future data support, once
> it is interrupted, it will not be real-time), and i must wait until the new
> data stream is connected.


This is yes and no:
1. If there's no element within a specific window at all, the window will
not be created and will not have anything to fire.
2. Upon the first element arrive at a specific window (assume no
late-arrival), the window will be created as well as an event-time timer.
so, if there's no future element arrival, the existing window (with at
least one element) will still fire promptly.
3. However, since this is an event time trigger, in order for the internal
timer service to "activate" the registered timer, watermark has to advance.

Regarding the #3 point I mentioned: I could've been wrong on this, but if
the source function does not advance watermark at all unless an element is
received from external data source, then yes this will probably be stuck.


1. use processing-time
> 2. "guaranteed" stream data event time interval is small and best
> sequential and never interrupted (if this can be guaranteed, use
> processing-time directly. What is the meaning of using EventTime and
> Watermark in the production environment and how to  test the real-time and
> accuracy of the data results? I am sorry I have confused from some flink
> streaming sql examples about the time window.)
> 3. Add new implementations or improvements:  the end time of window
> determined by assignWindows can trigger TriggerResult as soon as it reaches
> the natural time point or reaches the natural time point plus to watermark
> interval.


Regarding:
#1: The problem you described is not with processing-time because there's
nothing preventing the internal-timer to advance on the processing time
trigger/timer - they use the system time which will always advance.
#2/#3: This is not needed, as long as you guarantee watermark advance
promptly.

I am not exactly sure my explanation is the most accurate one, so if anyone
could share more insight please kindly share your thoughts :-)

Thanks,
Rong




On Sat, Apr 27, 2019 at 10:19 PM 邵志鹏  wrote:

> Dear flinker:
>
>
> Look at the contradiction between event time and natural time from
> EventTimeTrigger.java (the window at the time of the break and the end of
> the window at the end of the end must not be "real time"):
>
>
> From the default EventTimeTrigger source code, I found that only onElement
> method (will judge the watermark) and onEventTime method only have a chance
> to trigger TriggerResult.FIRE;
>
>
> Therefore, the default EventTimeTrigger is assumed and must be "never
> stop! The data stream" will have "correct" Real-time results, so as long as
> the interval between the two eventtimes is too large, greater than the time
> window interval, or the end time of the window has not arrived yet, there
> is no new data (flow interruption, neither element(onElement) nor
> eventtime(onEventtime)), then the latest time the output of the window must
> be untimely or non-real-time (if i use eventtime to do the bounded window
> aggregation of the stream, i must have the near future data support, once
> it is interrupted, it will not be real-time), and i must wait until the new
> data stream is connected.
>
>
> The window result that was not output in time before the new data stream
> is come.
>  (OnProcessingTime will never be called after EventTime is set, so
> modifying onProcessingTime has no effect.
> Called when a processing-time timer that was set using the trigger context
> fires.).
>
>
> Then, to "real time TriggerResult" can only
> 1. use processing-time
> 2. "guaranteed" stream data event time interval is small and best
> sequential and never interrupted (if this can be guaranteed, use
> processing-time directly. What is the meaning of using EventTime and
> Watermark in the production environment and how to  test the real-time and
> accuracy of the data results? I am sorry I have confused from some flink
> streaming sql examples about the time window.)
> 3. Add new implementations or improvements:  the end time of window
> determined by assignWindows can trigger TriggerResult as soon as it reaches
> the natural time point or reaches the natural 

[jira] [Created] (FLINK-12379) Parallelism in job/GCS/Hadoop: Could not finalize the pending checkpoint

[Henrik (JIRA)]

Henrik created FLINK-12379:
--

 Summary: Parallelism in job/GCS/Hadoop: Could not finalize the 
pending checkpoint
 Key: FLINK-12379
 URL: https://issues.apache.org/jira/browse/FLINK-12379
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.8.0
 Environment: GCS +

 
{code:java}
1.8.0
1.8
2.11{code}
{code:java}




  com.google.cloud.bigdataoss
  gcs-connector
  hadoop2-1.9.16


  org.apache.flink
  flink-connector-filesystem_2.11
  ${flink.version}


  org.apache.flink
  flink-hadoop-fs
  ${flink.version}



  org.apache.flink
  flink-shaded-hadoop2
  ${hadoop.version}-${flink.version}

{code}
 

 
Reporter: Henrik


When running a standalone-job w/ parallelism=4 + taskmanager, you will shortly 
get this crash
{code:java}
2019-04-30 22:20:02,928 WARN  org.apache.flink.runtime.jobmaster.JobMaster  
    - Error while processing checkpoint acknowledgement message
org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the 
pending checkpoint 5.
    at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:837)
    at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:756)
    at 
org.apache.flink.runtime.jobmaster.JobMaster.lambda$acknowledgeCheckpoint$9(JobMaster.java:676)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
    at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
'gs://example_bucket/flink/checkpoints//chk-5/_metadata'
 already exists
    at 
com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:85)
    at 
com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.(GoogleHadoopOutputStream.java:74)
    at 
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:797)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:807)
    at 
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
    at 
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
    at 
org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.(FsCheckpointMetadataOutputStream.java:65)
    at 
org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation.createMetadataOutputStream(FsCheckpointStorageLocation.java:104)
    at 
org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:259)
    at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:829)
    ... 8 more
Caused by: java.nio.file.FileAlreadyExistsException: Object 
gs://example_bucket/flink/checkpoints//chk-5/_metadata
 already exists.
    at 
com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getWriteGeneration(GoogleCloudStorageImpl.java:1918)
    at 
com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.create(GoogleCloudStorageImpl.java:410)
    at 
com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.createInternal(GoogleCloudStorageFileSystem.java:286)
    at 
com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.create(GoogleCloudStorageFileSystem.java:264)
    at 
com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:82)
    ... 19 more
2019-04-30 22:20:03,114 INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering 
checkpoint 6 @ 1556662802928 for job .{code}
My guess at why; concurrent checkpoint writers are updating the _metadata 
resource concurrently. They should be using optimistic concurrency control with 
ETag on GCS, and then retry until successful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Releasing Flink-shaded 7.0

[Bowen Li]

+1 for releasing flink-shaded 7.0

On Tue, Apr 30, 2019 at 3:13 AM Chesnay Schepler  wrote:

> I'm in favor creating a flink-shaded release soon. I'm not aware of any
> other changes being planned for flink-shaded, so let's get this out the
> door rather sooner than later.
>
> Note that this release is not just about hadoop, but also eliminates
> multiple security vulnerabilities in jackson.
>
> On 30/04/2019 11:26, jincheng sun wrote:
> > Since flink-1.8.0, Convenience binaries that include hadoop are no longer
> > released. More detail can be found here:
> >
> https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink
> > -1.8.html
> >
> > After flink 1.8 released, @Chesnay Schepler  does a
> lot
> > of efforts to move flink-shaded-hadoop to flink-shaded. more detail can
> be
> > found here: https://issues.apache.org/jira/browse/FLINK-10984
> >
> > Currently, flink-shaded-hadoop2 and flink-shaded-hadoop2-uber already had
> > moved to flink-shaded, then we should remove the flink-shaded-hadoop from
> > flink itself. More detail can be found here:
> > https://github.com/apache/flink/pull/8225
> >
> > So, we need to prepare the release of Flink-shaded 7.0,  I recommend to
> do
> > the release of Flink-shaded 7.0, and interested in becoming the Release
> > Manager of Flink-shaded 7.0 release!
> >
> > Please let me know what you think!
> >
> > Regards,
> > Jincheng
> >
>
>

[jira] [Created] (FLINK-12378) Consolidate FileSystem Documentation

[Seth Wiesman (JIRA)]

Seth Wiesman created FLINK-12378:


 Summary: Consolidate FileSystem Documentation
 Key: FLINK-12378
 URL: https://issues.apache.org/jira/browse/FLINK-12378
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, FileSystems
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Currently flink's filesystem documentation is spread across a number of pages 
without any clear connection. A non-exhaustive list of issues includes: 

* S3 documentation spread across many pages
* OSS filesystem is listed under deployments when it is an object store
* deployments/filesystem.md has a lot of unrelated information

We should create a filesystem subsection under deployments with multiple pages 
containing all relevant information about Flink's filesystem abstraction. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12377) Outdated docs (Flink 0.10) for Google Compute Engine

[Henrik (JIRA)]

Henrik created FLINK-12377:
--

 Summary: Outdated docs (Flink 0.10) for Google Compute Engine
 Key: FLINK-12377
 URL: https://issues.apache.org/jira/browse/FLINK-12377
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.8.0
Reporter: Henrik


[https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/gce_setup.html]
 links to 
[https://github.com/GoogleCloudPlatform/bdutil/blob/master/extensions/flink/flink_env.sh]
 which uses ancient versions of Hadoop and Flink.

Also the barrier to a newcomer is that bdutil itself is deprecated and the 
readme recommends DataFlow instead.

Furthermore, perhaps it would be wise to include GCP in the built-in 
filesystems in 
[https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/filesystems.html?|https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/filesystems.html]

Further, 
[https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/filesystems.html#hdfs-and-hadoop-file-system-support]
 doesn't actually link to any of the other configuration pages for thees other 
hadoop-based filesystems, nor does it explain how what exact library needs to 
be in the flink `lib folder; so it's really hard to go any further from there.

Lastly, it would seem the 1.8.0 release has stopped shipping the Hadoop libs, 
without documenting the change required, e.g. in the Hadoop File System page 
linked above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12376) GCS runtime exn: Request payload size exceeds the limit

[Henrik (JIRA)]

Henrik created FLINK-12376:
--

 Summary: GCS runtime exn: Request payload size exceeds the limit
 Key: FLINK-12376
 URL: https://issues.apache.org/jira/browse/FLINK-12376
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.7.2
 Environment: * k8s latest docker-for-desktop on macOS, and scala 
2.11-compiled Flink
Reporter: Henrik
 Attachments: Screenshot 2019-04-30 at 22.32.34.png

I'm trying to use the google cloud storage file system, but it would seem that 
the FLINK / GCS client libs are creating too-large requests far down in the GCS 
Java client.

 
{code:java}
[analytics-867c867ff6-l622h taskmanager] 2019-04-30 20:23:14,532 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source -> Process -> Timestamps/Watermarks -> app_events (1/1) 
(9a01e96c0271025d5ba73b735847cd4c) switched from RUNNING to FAILED.
[analytics-867c867ff6-l622h taskmanager] java.lang.RuntimeException: Error 
while confirming checkpoint
[analytics-867c867ff6-l622h taskmanager]     at 
org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:1211)
[analytics-867c867ff6-l622h taskmanager]     at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[analytics-867c867ff6-l622h taskmanager]     at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
[analytics-867c867ff6-l622h taskmanager]     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[analytics-867c867ff6-l622h taskmanager]     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[analytics-867c867ff6-l622h taskmanager]     at 
java.lang.Thread.run(Thread.java:748)
[analytics-867c867ff6-l622h taskmanager] Caused by: 
com.google.api.gax.rpc.InvalidArgumentException: 
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Request payload size exceeds 
the limit: 524288 bytes.
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
[analytics-867c867ff6-l622h taskmanager]     at 
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:507)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:694)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
[analytics-867c867ff6-l622h taskmanager]     at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
[analytics-867c867ff6-l622h taskmanager]     at 

[jira] [Created] (FLINK-12375) flink-container job jar does not have read permissions

[Adam Lamar (JIRA)]

Adam Lamar created FLINK-12375:
--

 Summary: flink-container job jar does not have read permissions
 Key: FLINK-12375
 URL: https://issues.apache.org/jira/browse/FLINK-12375
 Project: Flink
  Issue Type: Bug
Reporter: Adam Lamar


When building a custom job jar using flink-container, the job can't be launched 
if the provided job jar does not have world-readable permission.

This is because the job jar in the container is owned by root:root, but the 
docker container executes as the flink user.

In environments with restrictive umasks (e.g. company laptops) that create 
files without group and other read permissions by default, this causes the 
instructions to fail.

To reproduce on master:
{code:java}
cd flink-container/docker
cp ../../flink-examples/flink-examples-streaming/target/WordCount.jar .
chmod go-r WordCount.jar  # still maintain user read permission
./build.sh --job-jar WordCount.jar --from-archive 
flink-1.8.0-bin-scala_2.11.tgz --image-name flink-job:latest
FLINK_DOCKER_IMAGE_NAME=flink-job 
FLINK_JOB=org.apache.flink.streaming.examples.wordcount.WordCount 
docker-compose up{code}
which results in the following error:
{code:java}
job-cluster_1 | 2019-04-30 18:40:57,787 ERROR 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not start cluster 
entrypoint StandaloneJobClusterEntryPoint.
job-cluster_1 | org.apache.flink.runtime.entrypoint.ClusterEntrypointException: 
Failed to initialize the cluster entrypoint StandaloneJobClusterEntryPoint.
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:535)
job-cluster_1 | at 
org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:105)
job-cluster_1 | Caused by: org.apache.flink.util.FlinkException: Could not 
create the DispatcherResourceManagerComponent.
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257)
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
job-cluster_1 | at 
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
job-cluster_1 | ... 2 more
job-cluster_1 | Caused by: org.apache.flink.util.FlinkException: Could not load 
the provided entrypoint class.
job-cluster_1 | at 
org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:119)
job-cluster_1 | at 
org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96)
job-cluster_1 | at 
org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62)
job-cluster_1 | at 
org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41)
job-cluster_1 | at 
org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184)
job-cluster_1 | ... 6 more
job-cluster_1 | Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.streaming.examples.wordcount.WordCount
job-cluster_1 | at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
job-cluster_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
job-cluster_1 | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
job-cluster_1 | at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
job-cluster_1 | at 
org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:116)
job-cluster_1 | ... 10 more{code}
This issue can be fixed by chown'ing the job.jar file to flink:flink in the 
Dockerfile.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12374) Support translation from StreamExecTableSourceScan/BatchExecTableSourceScan to StreamTransformation.

[Jing Zhang (JIRA)]

Jing Zhang created FLINK-12374:
--

 Summary: Support translation from 
StreamExecTableSourceScan/BatchExecTableSourceScan to StreamTransformation.
 Key: FLINK-12374
 URL: https://issues.apache.org/jira/browse/FLINK-12374
 Project: Flink
  Issue Type: Task
  Components: Table SQL / API
Reporter: Jing Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] A more restrictive JIRA workflow

[Robert Metzger]

Hi Fabian,
You are right, I made a mistake. I don't think it makes sense to introduce
a new permission class. This will make the life of JIRA admins
unnecessarily complicated.
I updated the task list:

1. I update the contribution guide
2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
unassigned JIRAs
3. We ask Infra to change the permissions of our JIRA so that:
  a) only committers can assign users to tickets
  b) contributors can't assign users to tickets
  c) Every registered JIRA user is an assignable user in FLINK
4. We remove all existing contributors


On Tue, Apr 30, 2019 at 12:00 PM Fabian Hueske  wrote:

> Hi Robert,
>
> If I understood the decision correctly, we also need to ask Infra to make
> everybody an assignable user, right?
> Or do we want to add a new permission class "Assignable User" such that
> everyone still needs to ask for the right Jira permissions?
>
> Fabian
>
>
> Am Di., 30. Apr. 2019 um 10:46 Uhr schrieb Timo Walther <
> twal...@apache.org
> >:
>
> > Hi Robert,
> >
> > thanks for taking care of this. +1 to your suggested steps.
> >
> > Regards,
> > Timo
> >
> >
> > Am 30.04.19 um 10:42 schrieb Robert Metzger:
> > > @Stephan: I agree. Auto-closing PRs is quite aggressive.
> > > I will only do that for PRs without JIRA ID or "[hotfix]" in the title.
> > > We can always revisit this at a later stage.
> > >
> > >
> > > I'm proposing the following steps:
> > >
> > > 1. I update the contribution guide
> > > 2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
> > > unassigned JIRAs
> > > 3. We ask Infra to change the permissions of our JIRA so that:
> > >a) only committers can assign users to tickets
> > >b) contributors can't assign users to tickets
> > > 4. We remove all existing contributors
> > >
> > >
> > >
> > >
> > > On Wed, Apr 24, 2019 at 11:17 AM vino yang 
> > wrote:
> > >
> > >> also +1 for option 2.
> > >>
> > >> I think auto-close a PR sometimes a bit impertinency.
> > >> The reasons just like Stephan mentioned.
> > >>
> > >> Stephan Ewen  于2019年4月24日周三 下午4:08写道：
> > >>
> > >>> About auto-closing PRs from unassigned issues, consider the following
> > >> case
> > >>> that has happened quite a bit.
> > >>>
> > >>>- a user reports a small bug and immediately wants to provide a
> fix
> > for
> > >>> it
> > >>>- it makes sense to not stall the user for a few days until a
> > committer
> > >>> assigned the issue
> > >>>- not auto-closing the PR would at least allow the user to provide
> > >> their
> > >>> patch.
> > >>>
> > >>> On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen 
> > wrote:
> > >>>
> >  +1 for option #2
> > 
> >  Seems to me that this does not contradict option #1, it only extends
> > >> this
> >  a bit. I think there is a good case for that, to help frequent
> > >>> contributors
> >  on a way to committership.
> > 
> >  @Konstantin: Trivial fixes (typos, docs, javadocs, ...) should still
> > be
> >  possible as "hotfixes".
> > 
> >  On Mon, Apr 15, 2019 at 3:14 PM Timo Walther 
> > >> wrote:
> > > I think this really depends on the contribution.
> > >
> > > Sometimes "triviality" means that people just want to fix a typo in
> > >> some
> > > docs. For this, a hotfix PR is sufficient and does not need a JIRA
> > >>> issue.
> > > However, sometimes "triviality" is only trivial at first glance but
> > > introduces side effects. In any case, any contribution needs to be
> > > reviewed and merged by a committer so follow-up responses and
> > >> follow-up
> > > work might always be required. But you are right, committers need
> to
> > > respond quicker in any case.
> > >
> > > Timo
> > >
> > >
> > > Am 15.04.19 um 14:54 schrieb Konstantin Knauf:
> > >> Hi everyone,
> > >>
> > >> just my two cents:  as a non-committer I appreciate a lightweight,
> > >> frictionless process for trivial changes or small fixes without
> the
> > > need to
> > >> approach a committer beforehand. If it takes 5 days, so that I can
> > >>> start
> > >> with a triviality, I might not bother in the first place. So, in
> > >> order
> > > for
> > >> this not to backfire by making the community more exclusive, we
> need
> > > more
> > >> timely responses & follow ups by committers after the change to
> the
> > >> workflow. Having said that, I am slightly leaning towards Andrey's
> > >> interpretation of option 2.
> > >>
> > >> Cheers,
> > >>
> > >> Konstantin
> > >>
> > >>
> > >>
> > >> On Mon, Apr 15, 2019 at 1:39 PM Andrey Zagrebin <
> > >> and...@ververica.com
> > >> wrote:
> > >>
> > >>> @Robert thanks for pointing out and sorry for confusion. The
> > >> correct
> > > text:
> > >>> +1 for option 1.
> > >>>
> > >>> I also do not mind option 2, after 1-2 contributions, any
> > >> contributor
> > > could
> > >>> just ask the 

Your application for Season of Docs 2019 was unsuccessful

[Andrew Chen]

Thank you for your interest and enthusiasm for Season of Docs.

We’re sorry to say that your organization’s application for Season of Docs
was not successful this year. Because 2019 is the program’s pilot year, we
were only able to accept 50 organizations out of the almost 200
applications submitted. There were many high quality applications, so the
selection of organizations was not easy.

Please do stay in touch with the progress of Season of Docs, and let us
know if you have any questions or feedback by emailing
season-of-docs-supp...@googlegroups.com. We are hoping to grow the
program's capacity in subsequent years, so please apply again next year.

Sincerely,

The Season of Docs team

Re: Why isn't state supported in AsyncFunction?

[Till Rohrmann]

Hi Dominik,

the reason why the AsyncFunction does not support state atm is that we
would to have make sure that the right key is set not only when open and
asyncInvoke is set but also when a potential asynchronous callback is being
executed. It would be super easy to define a callback which keeps a
reference to the state object. If the right key is not set when the
callback is being executed, then you easily can corrupt your state.
Moreover, all state accesses need to run under the checkpoint lock. Due to
these problems, we decided to not support state for AsyncFunctions. Said
differently, if state accesses would set the right key and execute under
the checkpoint lock, then one could support state for AsyncFunctions.

Cheers,
Till

On Tue, Apr 30, 2019 at 12:20 PM Dominik Wosiński  wrote:

> Hey, Currently when trying to get the state inside the *open* method for
> Flink RichAsyncFunction we will get *State is not supported in rich async
> functions *exception.Why doesn't AsyncFunction support state ?
>
> Thanks in advance,
> Best Regards,
> Dom.
>

Why isn't state supported in AsyncFunction?

[Dominik Wosiński]

Hey, Currently when trying to get the state inside the *open* method for
Flink RichAsyncFunction we will get *State is not supported in rich async
functions *exception.Why doesn't AsyncFunction support state ?

Thanks in advance,
Best Regards,
Dom.

Re: [DISCUSS] Releasing Flink-shaded 7.0

[Chesnay Schepler]

I'm in favor creating a flink-shaded release soon. I'm not aware of any 
other changes being planned for flink-shaded, so let's get this out the 
door rather sooner than later.


Note that this release is not just about hadoop, but also eliminates 
multiple security vulnerabilities in jackson.


On 30/04/2019 11:26, jincheng sun wrote:

Since flink-1.8.0, Convenience binaries that include hadoop are no longer
released. More detail can be found here:
https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink
-1.8.html

After flink 1.8 released, @Chesnay Schepler  does a lot
of efforts to move flink-shaded-hadoop to flink-shaded. more detail can be
found here: https://issues.apache.org/jira/browse/FLINK-10984

Currently, flink-shaded-hadoop2 and flink-shaded-hadoop2-uber already had
moved to flink-shaded, then we should remove the flink-shaded-hadoop from
flink itself. More detail can be found here:
https://github.com/apache/flink/pull/8225

So, we need to prepare the release of Flink-shaded 7.0,  I recommend to do
the release of Flink-shaded 7.0, and interested in becoming the Release
Manager of Flink-shaded 7.0 release!

Please let me know what you think!

Regards,
Jincheng

Re: [DISCUSS] A more restrictive JIRA workflow

[Fabian Hueske]

Hi Robert,

If I understood the decision correctly, we also need to ask Infra to make
everybody an assignable user, right?
Or do we want to add a new permission class "Assignable User" such that
everyone still needs to ask for the right Jira permissions?

Fabian


Am Di., 30. Apr. 2019 um 10:46 Uhr schrieb Timo Walther :

> Hi Robert,
>
> thanks for taking care of this. +1 to your suggested steps.
>
> Regards,
> Timo
>
>
> Am 30.04.19 um 10:42 schrieb Robert Metzger:
> > @Stephan: I agree. Auto-closing PRs is quite aggressive.
> > I will only do that for PRs without JIRA ID or "[hotfix]" in the title.
> > We can always revisit this at a later stage.
> >
> >
> > I'm proposing the following steps:
> >
> > 1. I update the contribution guide
> > 2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
> > unassigned JIRAs
> > 3. We ask Infra to change the permissions of our JIRA so that:
> >a) only committers can assign users to tickets
> >b) contributors can't assign users to tickets
> > 4. We remove all existing contributors
> >
> >
> >
> >
> > On Wed, Apr 24, 2019 at 11:17 AM vino yang 
> wrote:
> >
> >> also +1 for option 2.
> >>
> >> I think auto-close a PR sometimes a bit impertinency.
> >> The reasons just like Stephan mentioned.
> >>
> >> Stephan Ewen  于2019年4月24日周三 下午4:08写道：
> >>
> >>> About auto-closing PRs from unassigned issues, consider the following
> >> case
> >>> that has happened quite a bit.
> >>>
> >>>- a user reports a small bug and immediately wants to provide a fix
> for
> >>> it
> >>>- it makes sense to not stall the user for a few days until a
> committer
> >>> assigned the issue
> >>>- not auto-closing the PR would at least allow the user to provide
> >> their
> >>> patch.
> >>>
> >>> On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen 
> wrote:
> >>>
>  +1 for option #2
> 
>  Seems to me that this does not contradict option #1, it only extends
> >> this
>  a bit. I think there is a good case for that, to help frequent
> >>> contributors
>  on a way to committership.
> 
>  @Konstantin: Trivial fixes (typos, docs, javadocs, ...) should still
> be
>  possible as "hotfixes".
> 
>  On Mon, Apr 15, 2019 at 3:14 PM Timo Walther 
> >> wrote:
> > I think this really depends on the contribution.
> >
> > Sometimes "triviality" means that people just want to fix a typo in
> >> some
> > docs. For this, a hotfix PR is sufficient and does not need a JIRA
> >>> issue.
> > However, sometimes "triviality" is only trivial at first glance but
> > introduces side effects. In any case, any contribution needs to be
> > reviewed and merged by a committer so follow-up responses and
> >> follow-up
> > work might always be required. But you are right, committers need to
> > respond quicker in any case.
> >
> > Timo
> >
> >
> > Am 15.04.19 um 14:54 schrieb Konstantin Knauf:
> >> Hi everyone,
> >>
> >> just my two cents:  as a non-committer I appreciate a lightweight,
> >> frictionless process for trivial changes or small fixes without the
> > need to
> >> approach a committer beforehand. If it takes 5 days, so that I can
> >>> start
> >> with a triviality, I might not bother in the first place. So, in
> >> order
> > for
> >> this not to backfire by making the community more exclusive, we need
> > more
> >> timely responses & follow ups by committers after the change to the
> >> workflow. Having said that, I am slightly leaning towards Andrey's
> >> interpretation of option 2.
> >>
> >> Cheers,
> >>
> >> Konstantin
> >>
> >>
> >>
> >> On Mon, Apr 15, 2019 at 1:39 PM Andrey Zagrebin <
> >> and...@ververica.com
> >> wrote:
> >>
> >>> @Robert thanks for pointing out and sorry for confusion. The
> >> correct
> > text:
> >>> +1 for option 1.
> >>>
> >>> I also do not mind option 2, after 1-2 contributions, any
> >> contributor
> > could
> >>> just ask the committer (who merged those contributions) about
> > contributor
> >>> permissions.
> >>>
> >>> Best,
> >>> Andrey
> >>>
> >>> On Mon, Apr 15, 2019 at 10:34 AM Feng LI 
> > wrote:
>  Hello there,
> 
>  New to the community. Just thought you might want some inputs from
> >>> new
>  comers too.
> 
>  I prefer option 2, where you need to prove the ability and
> >>> commitment
> >>> with
>  commits  before contributor permission is assigned.
> 
>  Cheers,
>  Feng
> 
>  Le lun. 15 avr. 2019 à 09:17, Robert Metzger  >>> a
>  écrit :
> 
> > @Andrey: You mention "option 2" two times, I guess one of the two
> > uses
> >>> of
> > "option 2" contains a typo?
> >
> > On Wed, Apr 10, 2019 at 10:33 AM Andrey Zagrebin <
> > and...@ververica.com
> 

Re: Apply for contributor permission

[Fabian Hueske]

Hi,

Welcome to the Flink community.
I've given you contributor permissions for Jira.

Thanks, Fabian

Am Do., 25. Apr. 2019 um 05:58 Uhr schrieb Shi Quan :

> Hi,
>
> I want to contribute to Apache Flink.
> Would you please give me the contributor permission?
>
> My JIRA ID is quan
>
> Best regards
>

[jira] [Created] (FLINK-12373) Improve checkpointing metrics

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-12373:
--

 Summary: Improve checkpointing metrics
 Key: FLINK-12373
 URL: https://issues.apache.org/jira/browse/FLINK-12373
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Checkpointing
Reporter: Gyula Fora


The checkpoint metrics encapsulated in the CheckpointMetrics class currently 
exposes 4 core metrics for each operator: bytesBuffered, alignment time, sync 
duration and async duration

I think it would be a great improvement to break up the tracking of the sync 
duration into the different components as it contains information that is 
critical to improve the SLA of large jobs.

I suggest we break up the sync duration into 4 subcomponents:

 1. prepareSnapshotPreBarrier
 2. Snapshot timers
 3. Snapshot operator states
 4. Sync keyed state checkpoint

Maybe the operator state part could be further broken up into keyed/non-keyed 
part, i dont know.

I think knowing these metrics is crucial for users to minimise the latency 
caused by checkpointing.

Whether we want to show all this info on the web ui is another discussion :)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-23 Model Serving

[Robert Metzger]

Hey all,

I'm wondering if somebody on the list can take a look at the PR from
FLIP-23: https://github.com/apache/flink/pull/7446


On Mon, Oct 1, 2018 at 6:13 PM Rong Rong  wrote:

> Thanks for the contribution Boris!! I've been playing around with the basic
> model for a while back and loved it.
> +1 and really looking forward to having the feature merging back to Flink
> ML.
>
> --
> Rong
>
> On Mon, Oct 1, 2018 at 7:55 AM Fabian Hueske  wrote:
>
> > Hi everybody,
> >
> > The question of how to serve ML models in Flink applications came up in
> > several conversations I had with Flink users in the last months.
> > Recently, Boris approached me and he told me that he'd like to revive the
> > efforts around FLIP-23 [1].
> >
> > In the last days, Boris extended the proposal by a speculative model
> > evaluation which allows for evaluating multiple modes of varying
> complexity
> > to ensure certain SLAs.
> > The code does already exist in a Github repository [2].
> >
> > Due to the frequent user requests and the fact that the code is already
> > present, I think would be a great feature for Flink to have.
> > Since this is a library on top of Flink's existing APIs this should not
> be
> > too hard to review.
> >
> > What do others think?
> >
> > Best, Fabian
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-23+-+Model+Serving
> > [2] https://github.com/FlinkML/flink-speculative-modelServer
> >
> > Am Mo., 5. Feb. 2018 um 13:11 Uhr schrieb Stavros Kontopoulos <
> > st.kontopou...@gmail.com>:
> >
> > > Thanx @Fabian. I will update the document accordingly wrt metrics.
> > > I agree there are pros and cons.
> > >
> > > Best,
> > > Stavros
> > >
> > >
> > > On Wed, Jan 31, 2018 at 1:07 AM, Fabian Hueske 
> > wrote:
> > >
> > > > OK, I think there was plenty of time to comment on this FLIP.
> > > > I'll move it to the ACCEPTED status.
> > > >
> > > > @Stavros, please consider the feedback regarding the metrics.
> > > > I agree with Chesnay that metrics should be primarily exposed via the
> > > > metrics system.
> > > > Storing them in state makes them fault-tolerant and queryable if the
> > > state
> > > > is properly configured.
> > > >
> > > > Thanks,
> > > > Fabian
> > > >
> > > > 2018-01-22 17:19 GMT+01:00 Chesnay Schepler :
> > > >
> > > > > I'm currently looking over it, but one thing that stood out was
> that
> > > the
> > > > > FLIP proposes to use queryable state
> > > > > as a monitoring solution. Given that we have a metric system that
> > > > > integrates with plenty of commonly used
> > > > > metric backends this doesn't really make sense to me.
> > > > >
> > > > > Storing them in state still has value in terms of fault-tolerance
> > > though,
> > > > > since this is something that the metric
> > > > > system doesn't provide by itself.
> > > > >
> > > > >
> > > > > On 18.01.2018 13:57, Fabian Hueske wrote:
> > > > >
> > > > >> Are there any more comments on the FLIP?
> > > > >>
> > > > >> Otherwise, I'd suggest to move the FLIP to the accepted FLIPs [1]
> > and
> > > > >> continue with the implementation.
> > > > >>
> > > > >> Also, is there a committer who'd like to shepherd the FLIP and
> > review
> > > > the
> > > > >> corresponding PRs?
> > > > >> Of course, everybody is welcome to review the code but we need at
> > > least
> > > > >> one
> > > > >> committer who will eventually merge the changes.
> > > > >>
> > > > >> Best,
> > > > >> Fabian
> > > > >>
> > > > >> [1]
> > > > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+
> > > > >> Improvement+Proposals
> > > > >>
> > > > >> 2017-12-04 10:54 GMT+01:00 Fabian Hueske :
> > > > >>
> > > > >> Hi,
> > > > >>>
> > > > >>> Sorry for the late follow up.
> > > > >>>
> > > > >>> I think I understand the motivation for choosing ProtoBuf as the
> > > > >>> representation and serialization format and this makes sense to
> me.
> > > > >>>
> > > > >>> However, it might be a good idea to provide tooling to convert
> > Flink
> > > > >>> types
> > > > >>> (described as TypeInformation) to ProtoBuf.
> > > > >>> Otherwise, users of the model serving library would need to
> > manually
> > > > >>> convert their data types (say Scala tuples, case classes, or Avro
> > > > Pojos)
> > > > >>> to
> > > > >>> ProtoBuf messages.
> > > > >>> I don't think that this needs to be included in the first version
> > but
> > > > it
> > > > >>> might be a good extension to make the library easier to use.
> > > > >>>
> > > > >>> Best,
> > > > >>> Fabian
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> 2017-11-28 17:22 GMT+01:00 Boris Lublinsky <
> > > > >>> boris.lublin...@lightbend.com>
> > > > >>> :
> > > > >>>
> > > > >>> Thanks Fabian,
> > > >  More below
> > > > 
> > > > 
> > > > 
> > > >  Boris Lublinsky
> > > >  FDP Architect
> > > >  boris.lublin...@lightbend.com
> > > >  https://www.lightbend.com/
> > > > 
> > > >  On Nov 28, 2017, at 8:21 AM, Fabian Hueske 
> > > wrote:
> > > > 
> > > >  

RE: [DISCUSS] Bump required Maven version to 3.1.1 (from 3.0.3)

[Visser, M.J.H. (Martijn)]

I wouldn't consider the version bump blocking for us. We do have Redhat 
machines for local development, but we can manually upgrade the Maven version 
there.

Yes, Flink is the only npm project build on those machines. There are other 
projects though that are using npm, so we'll probably can get it working 
internally too. Just requires some more manual work. 

-Original Message-
From: Robert Metzger [mailto:rmetz...@apache.org] 
Sent: maandag 29 april 2019 17:32
To: dev 
Cc: Visser, M.J.H. (Martijn) 
Subject: Re: [DISCUSS] Bump required Maven version to 3.1.1 (from 3.0.3)

Thanks Martijn. RHEL has quite long support cycles :)

Would you still be okay with the version bump?
(The workaround here being to manually download a later maven version for Flink 
builds)

For the npm dependencies: Is Flink the only npm project build on these servers? 
(I'm asking because there might be a similar model with custom repositories for 
npm, or some exceptions in your corporate policies for
NPM?)

On Mon, Apr 29, 2019 at 4:23 PM Chesnay Schepler  wrote:

> The other connection you're referring to is likely the plugin 
> downloading dependencies from npm.
>
> On 29/04/2019 13:50, Visser, M.J.H. (Martijn) wrote:
> > Hi Robert,
> >
> > Just as an FYI, Redhat 7 Enterprise also still ships with 3.0.5 as
> default. I did a manual upgrade to a later version on one of our 
> Redhat servers, but it appears that this plugin doesn't use the 
> repositories as are defined in a settings.xml, but tries to setup an 
> external connection (which fails, because we're behind a proxy).
> >
> > Best regards,
> >
> > Martijn
> >
> > -Original Message-
> > From: Robert Metzger [mailto:rmetz...@apache.org]
> > Sent: vrijdag 26 april 2019 15:43
> > To: dev 
> > Subject: [DISCUSS] Bump required Maven version to 3.1.1 (from 3.0.3)
> >
> > Hey,
> >
> > I recently merged the new Flink UI [1]. After merging it, I was made
> aware that the frontend-maven-plugin requires at least Maven 3.1.0.
> > There was a discussion about bumping the maven version already last 
> > year
> [2]. In that thread, Stephan mentions that some Linux distributions 
> still have Maven 3.0.5 as their default version.
> > Ubuntu 16.04 (the current LTS version) ships Maven 3.3.9.
> > Debian Jessie (8) is indeed still shipping 3.0.5, Stretch (9) is on
> 3.3.9 too.
> >
> > Given this situation, I propose to bump the required Maven version 
> > to
> 3.1.1.
> >
> > Flink releases are build with 3.2.5.
> >
> > If nobody raises any objections till Thursday, May 2nd, I'll open a 
> > PR
> bumping the version.
> >
> > Best,
> > Robert
> >
> >
> > [1] https://github.com/apache/flink/pull/8016
> > [2] https://github.com/apache/flink/pull/6443#issuecomment-410673911
> >
> > -
> > ATTENTION:
> > The information in this e-mail is confidential and only meant for 
> > the
> intended recipient. If you are not the intended recipient, don't use 
> or disclose it in any way. Please let the sender know and delete the 
> message immediately.
> > -
>
>
>

-
ATTENTION:
The information in this e-mail is confidential and only meant for the intended 
recipient. If you are not the intended recipient, don't use or disclose it in 
any way. Please let the sender know and delete the message immediately.
-

[DISCUSS] Releasing Flink-shaded 7.0

[jincheng sun]

Since flink-1.8.0, Convenience binaries that include hadoop are no longer
released. More detail can be found here:
https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink
-1.8.html

After flink 1.8 released, @Chesnay Schepler  does a lot
of efforts to move flink-shaded-hadoop to flink-shaded. more detail can be
found here: https://issues.apache.org/jira/browse/FLINK-10984

Currently, flink-shaded-hadoop2 and flink-shaded-hadoop2-uber already had
moved to flink-shaded, then we should remove the flink-shaded-hadoop from
flink itself. More detail can be found here:
https://github.com/apache/flink/pull/8225

So, we need to prepare the release of Flink-shaded 7.0,  I recommend to do
the release of Flink-shaded 7.0, and interested in becoming the Release
Manager of Flink-shaded 7.0 release!

Please let me know what you think!

Regards,
Jincheng

Re: I want to contribute to Apache Flink

[Fabian Hueske]

Hi,

Welcome to the Flink community.
I gave you contributor permissions for Jira.

Best, Fabian

Am Di., 30. Apr. 2019 um 04:02 Uhr schrieb aihua li :

> Hi,
>
> I want to contribute to Apache Flink.
> Would you please give me the contributor permission?
> My JIRA ID is aiwa.
>
>

Re: [DISCUSS] FLIP-39: Flink ML pipeline and ML libs

[Shaoxuan Wang]

Thanks for all the feedback.

@Jincheng Sun
> I recommend It's better to add a detailed implementation plan to FLIP and
google doc.
Yes, I will add a subsection for implementation plan.

@Chen Qin
>Just share some of insights from operating SparkML side at scale
>- map reduce may not best way to iterative sync partitioned workers.
>- native hardware accelerations is key to adopt rapid changes in ML
improvements in foreseeable future.
Thanks for sharing your experience on SparkML. The purpose of this FLIP is
mainly to provide the interfaces for ML pipeline and ML lib, and the
implementations of most standard algorithms. Besides this FLIP, for AI
computing on Flink, we will continue to contribute the efforts, like the
enhancement of iterative and the integration of deep learning engines (such
as Tensoflow/Pytorch). I have presented part of these work in
https://www.ververica.com/resources/flink-forward-san-francisco-2019/when-table-meets-ai-build-flink-ai-ecosystem-on-table-api
I am not sure if I have fully got your comments. Can you please elaborate
them with more details, and if possible, please provide some suggestions
about what we should work on to address the challenges you have mentioned.

Regards,
Shaoxuan

On Mon, Apr 29, 2019 at 11:28 AM Chen Qin  wrote:

> Just share some of insights from operating SparkML side at scale
> - map reduce may not best way to iterative sync partitioned workers.
> - native hardware accelerations is key to adopt rapid changes in ML
> improvements in foreseeable future.
>
> Chen
>
> On Apr 29, 2019, at 11:02, jincheng sun  wrote:
> >
> > Hi Shaoxuan,
> >
> > Thanks for doing more efforts for the enhances of the scalability and the
> > ease of use of Flink ML and make it one step further. Thank you for
> sharing
> > a lot of context information.
> >
> > big +1 for this proposal!
> >
> > Here only one suggestion, that is, It has been a short time until the
> > release of flink-1.9, so I recommend It's better to add a detailed
> > implementation plan to FLIP and google doc.
> >
> > What do you think?
> >
> > Best,
> > Jincheng
> >
> > Shaoxuan Wang  于2019年4月29日周一 上午10:34写道：
> >
> >> Hi everyone,
> >>
> >> Weihua has proposed to rebuild Flink ML pipeline on top of TableAPI
> several
> >> months ago in this mail thread:
> >>
> >>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Embracing-Table-API-in-Flink-ML-td25368.html
> >>
> >> Luogen, Becket, Xu, Weihua and I have been working on this proposal
> >> offline in
> >> the past a few months. Now we want to share the first phase of the
> entire
> >> proposal with a FLIP. In this FLIP-39, we want to achieve several things
> >> (and hope those can be accomplished and released in Flink-1.9):
> >>
> >>   -
> >>
> >>   Provide a new set of ML core interface (on top of Flink TableAPI)
> >>   -
> >>
> >>   Provide a ML pipeline interface (on top of Flink TableAPI)
> >>   -
> >>
> >>   Provide the interfaces for parameters management and pipeline/mode
> >>   persistence
> >>   -
> >>
> >>   All the above interfaces should facilitate any new ML algorithm. We
> will
> >>   gradually add various standard ML algorithms on top of these new
> >> proposed
> >>   interfaces to ensure their feasibility and scalability.
> >>
> >>
> >> Part of this FLIP has been present in Flink Forward 2019 @ San
> Francisco by
> >> Xu and Me.
> >>
> >>
> >>
> https://sf-2019.flink-forward.org/conference-program#when-table-meets-ai--build-flink-ai-ecosystem-on-table-api
> >>
> >>
> >>
> https://sf-2019.flink-forward.org/conference-program#high-performance-ml-library-based-on-flink
> >>
> >> You can find the videos & slides at
> >> https://www.ververica.com/flink-forward-san-francisco-2019
> >>
> >> The design document for FLIP-39 can be found here:
> >>
> >>
> >>
> https://docs.google.com/document/d/1StObo1DLp8iiy0rbukx8kwAJb0BwDZrQrMWub3DzsEo
> >>
> >>
> >> I am looking forward to your feedback.
> >>
> >> Regards,
> >>
> >> Shaoxuan
> >>
>

Re: [DISCUSS] A more restrictive JIRA workflow

[Timo Walther]

Hi Robert,

thanks for taking care of this. +1 to your suggested steps.

Regards,
Timo


Am 30.04.19 um 10:42 schrieb Robert Metzger:

@Stephan: I agree. Auto-closing PRs is quite aggressive.
I will only do that for PRs without JIRA ID or "[hotfix]" in the title.
We can always revisit this at a later stage.


I'm proposing the following steps:

1. I update the contribution guide
2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
unassigned JIRAs
3. We ask Infra to change the permissions of our JIRA so that:
   a) only committers can assign users to tickets
   b) contributors can't assign users to tickets
4. We remove all existing contributors




On Wed, Apr 24, 2019 at 11:17 AM vino yang  wrote:


also +1 for option 2.

I think auto-close a PR sometimes a bit impertinency.
The reasons just like Stephan mentioned.

Stephan Ewen  于2019年4月24日周三 下午4:08写道：


About auto-closing PRs from unassigned issues, consider the following

case

that has happened quite a bit.

   - a user reports a small bug and immediately wants to provide a fix for
it
   - it makes sense to not stall the user for a few days until a committer
assigned the issue
   - not auto-closing the PR would at least allow the user to provide

their

patch.

On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen  wrote:


+1 for option #2

Seems to me that this does not contradict option #1, it only extends

this

a bit. I think there is a good case for that, to help frequent

contributors

on a way to committership.

@Konstantin: Trivial fixes (typos, docs, javadocs, ...) should still be
possible as "hotfixes".

On Mon, Apr 15, 2019 at 3:14 PM Timo Walther 

wrote:

I think this really depends on the contribution.

Sometimes "triviality" means that people just want to fix a typo in

some

docs. For this, a hotfix PR is sufficient and does not need a JIRA

issue.

However, sometimes "triviality" is only trivial at first glance but
introduces side effects. In any case, any contribution needs to be
reviewed and merged by a committer so follow-up responses and

follow-up

work might always be required. But you are right, committers need to
respond quicker in any case.

Timo


Am 15.04.19 um 14:54 schrieb Konstantin Knauf:

Hi everyone,

just my two cents:  as a non-committer I appreciate a lightweight,
frictionless process for trivial changes or small fixes without the

need to

approach a committer beforehand. If it takes 5 days, so that I can

start

with a triviality, I might not bother in the first place. So, in

order

for

this not to backfire by making the community more exclusive, we need

more

timely responses & follow ups by committers after the change to the
workflow. Having said that, I am slightly leaning towards Andrey's
interpretation of option 2.

Cheers,

Konstantin



On Mon, Apr 15, 2019 at 1:39 PM Andrey Zagrebin <

and...@ververica.com

wrote:


@Robert thanks for pointing out and sorry for confusion. The

correct

text:

+1 for option 1.

I also do not mind option 2, after 1-2 contributions, any

contributor

could

just ask the committer (who merged those contributions) about

contributor

permissions.

Best,
Andrey

On Mon, Apr 15, 2019 at 10:34 AM Feng LI 

wrote:

Hello there,

New to the community. Just thought you might want some inputs from

new

comers too.

I prefer option 2, where you need to prove the ability and

commitment

with

commits  before contributor permission is assigned.

Cheers,
Feng

Le lun. 15 avr. 2019 à 09:17, Robert Metzger 
a

écrit :


@Andrey: You mention "option 2" two times, I guess one of the two

uses

of

"option 2" contains a typo?

On Wed, Apr 10, 2019 at 10:33 AM Andrey Zagrebin <

and...@ververica.com

wrote:


Hi all,

+1 for option 2.

I also do not mind option 2, after 1-2 contributions, any

contributor

could

just ask the committer (who merged those contributions) about

contributor

permissions.

Best,
Andrey

On Wed, Apr 10, 2019 at 3:58 AM Robert Metzger <

rmetz...@apache.org

wrote:


I'm +1 on option 1.

On Tue, Apr 9, 2019 at 1:58 AM Timo Walther <

twal...@apache.org>

wrote:

Hi everyone,

I'd like to bring up this discussion thread again. In

summary, I

think

we all agreed on improving the JIRA workflow to move

design/consensus

discussions from PRs to the issues first, before implementing

them.

Two options have been proposed:
1. Only committers can assign people to issues. PRs of

unassigned

issues

are closed automatically.
2. Committers upgrade assignable users to contributors as an
intermediate step towards committership.

I would prefer option 1 as some people also mentioned that

option 2

requires some standadized processes otherwise it would be

difficult

to

communicate why somebody is a contributor and some somebody is

not.

What do you think?

Regards,
Timo


Am 18.03.19 um 14:25 schrieb Robert Metzger:

@Fabian: I don't think this is a big problem. Moving away

from

"giving

everybody contributor permissions" to "giving it to some

people"

Re: [DISCUSS] A more restrictive JIRA workflow

[Robert Metzger]

@Stephan: I agree. Auto-closing PRs is quite aggressive.
I will only do that for PRs without JIRA ID or "[hotfix]" in the title.
We can always revisit this at a later stage.


I'm proposing the following steps:

1. I update the contribution guide
2. Update Flinkbot to close invalid PRs, and show warnings on PRs with
unassigned JIRAs
3. We ask Infra to change the permissions of our JIRA so that:
  a) only committers can assign users to tickets
  b) contributors can't assign users to tickets
4. We remove all existing contributors




On Wed, Apr 24, 2019 at 11:17 AM vino yang  wrote:

> also +1 for option 2.
>
> I think auto-close a PR sometimes a bit impertinency.
> The reasons just like Stephan mentioned.
>
> Stephan Ewen  于2019年4月24日周三 下午4:08写道：
>
> > About auto-closing PRs from unassigned issues, consider the following
> case
> > that has happened quite a bit.
> >
> >   - a user reports a small bug and immediately wants to provide a fix for
> > it
> >   - it makes sense to not stall the user for a few days until a committer
> > assigned the issue
> >   - not auto-closing the PR would at least allow the user to provide
> their
> > patch.
> >
> > On Wed, Apr 24, 2019 at 10:00 AM Stephan Ewen  wrote:
> >
> > > +1 for option #2
> > >
> > > Seems to me that this does not contradict option #1, it only extends
> this
> > > a bit. I think there is a good case for that, to help frequent
> > contributors
> > > on a way to committership.
> > >
> > > @Konstantin: Trivial fixes (typos, docs, javadocs, ...) should still be
> > > possible as "hotfixes".
> > >
> > > On Mon, Apr 15, 2019 at 3:14 PM Timo Walther 
> wrote:
> > >
> > >> I think this really depends on the contribution.
> > >>
> > >> Sometimes "triviality" means that people just want to fix a typo in
> some
> > >> docs. For this, a hotfix PR is sufficient and does not need a JIRA
> > issue.
> > >>
> > >> However, sometimes "triviality" is only trivial at first glance but
> > >> introduces side effects. In any case, any contribution needs to be
> > >> reviewed and merged by a committer so follow-up responses and
> follow-up
> > >> work might always be required. But you are right, committers need to
> > >> respond quicker in any case.
> > >>
> > >> Timo
> > >>
> > >>
> > >> Am 15.04.19 um 14:54 schrieb Konstantin Knauf:
> > >> > Hi everyone,
> > >> >
> > >> > just my two cents:  as a non-committer I appreciate a lightweight,
> > >> > frictionless process for trivial changes or small fixes without the
> > >> need to
> > >> > approach a committer beforehand. If it takes 5 days, so that I can
> > start
> > >> > with a triviality, I might not bother in the first place. So, in
> order
> > >> for
> > >> > this not to backfire by making the community more exclusive, we need
> > >> more
> > >> > timely responses & follow ups by committers after the change to the
> > >> > workflow. Having said that, I am slightly leaning towards Andrey's
> > >> > interpretation of option 2.
> > >> >
> > >> > Cheers,
> > >> >
> > >> > Konstantin
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Apr 15, 2019 at 1:39 PM Andrey Zagrebin <
> and...@ververica.com
> > >
> > >> > wrote:
> > >> >
> > >> >> @Robert thanks for pointing out and sorry for confusion. The
> correct
> > >> text:
> > >> >>
> > >> >> +1 for option 1.
> > >> >>
> > >> >> I also do not mind option 2, after 1-2 contributions, any
> contributor
> > >> could
> > >> >> just ask the committer (who merged those contributions) about
> > >> contributor
> > >> >> permissions.
> > >> >>
> > >> >> Best,
> > >> >> Andrey
> > >> >>
> > >> >> On Mon, Apr 15, 2019 at 10:34 AM Feng LI 
> > >> wrote:
> > >> >>
> > >> >>> Hello there,
> > >> >>>
> > >> >>> New to the community. Just thought you might want some inputs from
> > new
> > >> >>> comers too.
> > >> >>>
> > >> >>> I prefer option 2, where you need to prove the ability and
> > commitment
> > >> >> with
> > >> >>> commits  before contributor permission is assigned.
> > >> >>>
> > >> >>> Cheers,
> > >> >>> Feng
> > >> >>>
> > >> >>> Le lun. 15 avr. 2019 à 09:17, Robert Metzger  >
> > a
> > >> >>> écrit :
> > >> >>>
> > >>  @Andrey: You mention "option 2" two times, I guess one of the two
> > >> uses
> > >> >> of
> > >>  "option 2" contains a typo?
> > >> 
> > >>  On Wed, Apr 10, 2019 at 10:33 AM Andrey Zagrebin <
> > >> and...@ververica.com
> > >>  wrote:
> > >> 
> > >> > Hi all,
> > >> >
> > >> > +1 for option 2.
> > >> >
> > >> > I also do not mind option 2, after 1-2 contributions, any
> > >> contributor
> > >>  could
> > >> > just ask the committer (who merged those contributions) about
> > >> >>> contributor
> > >> > permissions.
> > >> >
> > >> > Best,
> > >> > Andrey
> > >> >
> > >> > On Wed, Apr 10, 2019 at 3:58 AM Robert Metzger <
> > rmetz...@apache.org
> > >> >
> > >> > wrote:
> > >> >
> > >> >> I'm +1 on option 1.
> > >> >>
> > >> >> On Tue, Apr 9, 2019 at 1:58 AM Timo Walther <
> 

Re: kafka partitions, data locality

[Fabian Hueske]

Hi Sergey,

You are right, keys are managed in key groups. Each key belongs to a key
group and one or more key groups are assigned to each parallel task of an
operator.
Key groups are not exposed to users and the assignments of keys ->
key-groups and key-groups -> tasks cannot be changed without changing Flink
itself (i.e., a custom build).
If you don't want to change Flink, you'd need to change the partitioning in
Kafka (mapping key-groups to partitions) and ensuring that all partitions
are read by the correct task.

I don't think this is possible (with reasonable effort) and if you get it
to work it would be quite fragile with respect to changing parallelism
(Kafka, Flink) etc.
Right now there is no way around partitioning the events with keyBy() if
you want to use keyed state.
After the first keyBy() partitioning, reinterpretAsKeyedStream() can be
used to reuse an existing partitioning.

Best, Fabian

Am Mo., 29. Apr. 2019 um 15:23 Uhr schrieb Smirnov Sergey Vladimirovich
(39833) :

> Hi Stefan,
>
>
>
> Thnx for clarify!
>
> But still it remains an open question for me because we use keyBy method
> and I did not found any public interface of keys reassignment (smth like
> partionCustom for DataStream).
>
> As I heard, there is some internal mechanism with key groups and mapping
> key to groups. Is it supposed to become public?
>
>
>
>
>
> Regards,
>
> Sergey
>
>
>
> *From:* Stefan Richter [mailto:s.rich...@ververica.com]
> *Sent:* Friday, April 26, 2019 11:15 AM
> *To:* Smirnov Sergey Vladimirovich (39833) 
> *Cc:* Dawid Wysakowicz ; Ken Krugler <
> kkrugler_li...@transpac.com>; u...@flink.apache.org; dev@flink.apache.org
> *Subject:* Re: kafka partitions, data locality
>
>
>
> Hi Sergey,
>
>
>
> The point why this I flagged as beta is actually less about stability but
> more about the fact that this is supposed to be more of a "power user"
> feature because bad things can happen if your data is not 100% correctly
> partitioned in the same way as Flink would partition it. This is why
> typically you should only use it if the data was partitioned by Flink and
> you are very sure what your are doing, because the is not really something
> we can to at the API level to protect you from mistakes in using this
> feature. Eventually some runtime exceptions might show you that something
> is going wrong, but that is not exactly a good user experience.
>
>
>
> On a different note, there actually is currently one open issue [1] to be
> aware of in connection with this feature and operator chaining, but at the
> same time this is something that should not hard to fix in for the next
> minor release.
>
>
>
> Best,
>
> Stefan
>
>
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-12296?focusedCommentId=16824945=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16824945
>
>
>
>
> On 26. Apr 2019, at 09:48, Smirnov Sergey Vladimirovich (39833) <
> s.smirn...@tinkoff.ru> wrote:
>
>
>
> Hi,
>
>
>
> Dawid, great, thanks!
>
> Any plans to make it stable? 1.9?
>
>
>
>
>
> Regards,
>
> Sergey
>
>
>
> *From:* Dawid Wysakowicz [mailto:dwysakow...@apache.org
> ]
> *Sent:* Thursday, April 25, 2019 10:54 AM
> *To:* Smirnov Sergey Vladimirovich (39833) ; Ken
> Krugler 
> *Cc:* u...@flink.apache.org; dev@flink.apache.org
> *Subject:* Re: kafka partitions, data locality
>
>
>
> Hi Smirnov,
>
> Actually there is a way to tell Flink that data is already partitioned.
> You can try the reinterpretAsKeyedStream[1] method. I must warn you though
> this is an experimental feature.
>
> Best,
>
> Dawid
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/experimental.html#experimental-features
>
> On 19/04/2019 11:48, Smirnov Sergey Vladimirovich (39833) wrote:
>
> Hi Ken,
>
>
>
> It’s a bad story for us: even for a small window we have a dozens of
> thousands events per job with 10x in peaks or even more. And the number of
> jobs was known to be high. So instead of N operations (our
> producer/consumer mechanism) with shuffle/resorting (current flink
> realization) it will be N*ln(N) - the tenfold loss of execution speed!
>
> 4 all, my next step? Contribute to apache flink? Issues backlog?
>
>
>
>
>
> With best regards,
>
> Sergey
>
> *From:* Ken Krugler [mailto:kkrugler_li...@transpac.com
> ]
> *Sent:* Wednesday, April 17, 2019 9:23 PM
> *To:* Smirnov Sergey Vladimirovich (39833) 
> 
> *Subject:* Re: kafka partitions, data locality
>
>
>
> Hi Sergey,
>
>
>
> As you surmised, once you do a keyBy/max on the Kafka topic, to group by
> clientId and find the max, then the topology will have a partition/shuffle
> to it.
>
>
>
> This is because Flink doesn’t know that client ids don’t span Kafka
> partitions.
>
>
>
> I don’t know of any way to tell Flink that the data doesn’t need to be
> shuffled. There was a discussion
> 
>  about
> adding a 

Re: Apply for confluence Wiki edit permission

[Robert Metzger]

Hey Shuiqiang,
I gave you access to the Wiki.

On Tue, Apr 30, 2019 at 4:29 AM Shuiqiang Chen  wrote:

> Hi,
>
> I am going to create a new FLIP for Proposal of Flink Driver.
> <
> https://mail.google.com/mail/u/0/#sent/KtbxLzFrKvFFhHLDqBlJXNXZrfPzVdFvjq?compose=GTvVlcSGMhgHDDqDFxNxFfWzXMKmLCvtbJNjTHQhNgCRXCwXCGmQGPxLzlVJcjxQQLBFlVpMZJkTk
> >
> Could you please give me the edit permission for this page:
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> My ID is : Shuiqiang Chen
>
> Best,
> Shuiqiang
>

[jira] [Created] (FLINK-12372) Implement ExecutionSlotAllocator

[Gary Yao (JIRA)]

Gary Yao created FLINK-12372:


 Summary: Implement ExecutionSlotAllocator
 Key: FLINK-12372
 URL: https://issues.apache.org/jira/browse/FLINK-12372
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Gary Yao
Assignee: Gary Yao


Add and implement {{ExecutionSlotAllocator}} interface
Design document: 
https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU

*Acceptance criteria*
* {{ExecutionSlotAllocator}} interface is defined and implemented
* interface implementation is unit tested



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Update our Roadmap

[jincheng sun]

Hi Jeff,

I have open the PR about add Python Table API section to the roadmap. I
appreciate if you have time to look at it. :)

https://github.com/apache/flink-web/pull/204

Regards,
Jincheng

jincheng sun  于2019年4月29日周一 下午11:12写道：

> Sure, I will do it！I think the python table api info should in the
>  roadmap! Thank you @Jeff @Fabian
>
> Fabian Hueske 于2019年4月29日 周一23:05写道：
>
>> Great, thanks Jeff and Timo!
>>
>> @Jincheng do you want to write a paragraph about the Python effort and
>> open a PR for it?
>>
>> I'll remove the issue about Hadoop convenience builds (FLINK-11266).
>>
>> Best, Fabian
>>
>> Am Mo., 29. Apr. 2019 um 16:37 Uhr schrieb Jeff Zhang :
>>
>>> jincheng(cc) is driving the python effort, I think he can help to
>>> prepare it.
>>>
>>>
>>>
>>> Fabian Hueske  于2019年4月29日周一 下午10:15写道：
>>>
 Hi everyone,

 Since we had no more comments on this thread, I think we proceed to
 update the roadmap.

 @Jeff Zhang  I agree, we should add the Python
 efforts to the roadmap.
 Do you want to prepare a short paragraph that we can add to the
 document?

 Best, Fabian

 Am Mi., 17. Apr. 2019 um 15:04 Uhr schrieb Jeff Zhang >>> >:

> Hi Fabian,
>
> One thing missing is python api and python udf, we already discussed
> it in
> community, and it is very close to reach consensus.
>
>
> Fabian Hueske  于2019年4月17日周三 下午7:51写道：
>
> > Hi everyone,
> >
> > We recently added a roadmap to our project website [1] and decided to
> > update it after every release. Flink 1.8.0 was released a few days
> ago, so
> > I think it we should check and remove from the roadmap what was
> achieved so
> > far and add features / improvements that we plan for the future.
> >
> > I had a look at the roadmap and found that
> >
> > > We are changing the build setup to not bundle Hadoop by default,
> but
> > rather offer pre-packaged
> > > Hadoop libraries for the use with Yarn, HDFS, etc. as convenience
> > downloads FLINK-11266 <
> https://issues.apache.org/jira/browse/FLINK-11266>.
> >
> > was implemented for 1.8.0 and should be removed from the roadmap.
> > All other issues are still ongoing efforts.
> >
> > Are there any other efforts that we want to put on the roadmap?
> >
> > Best, Fabian
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>