[jira] [Created] (FLINK-33694) GCS filesystem does not respect gs.storage.root.url config option

[Patrick Lucas (Jira)]

Patrick Lucas created FLINK-33694:
-

 Summary: GCS filesystem does not respect gs.storage.root.url 
config option
 Key: FLINK-33694
 URL: https://issues.apache.org/jira/browse/FLINK-33694
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.17.2, 1.18.0
Reporter: Patrick Lucas


The GCS FileSystem's RecoverableWriter implementation uses the GCS SDK directly 
rather than going through Hadoop. While support has been added to configure 
credentials correctly based on the standard Hadoop implementation 
configuration, no other options are passed through to the underlying client.

Because this only affects the RecoverableWriter-related codepaths, it can 
result in very surprising differing behavior whether the FileSystem is being 
used as a source or a sink—while a {{{}gs://{}}}-URI FileSource may work fine, 
a {{{}gs://{}}}-URI FileSink may not work at all.

We use [fake-gcs-server|https://github.com/fsouza/fake-gcs-server] in testing, 
and so we override the Hadoop GCS FileSystem config option 
{{{}gs.storage.root.url{}}}. However, because this option is not considered 
when creating the GCS client for the RecoverableWriter codepath, in a FileSink 
the GCS FileSystem attempts to write to the real GCS service rather than 
fake-gcs-server. At the same time, a FileSource works as expected, reading from 
fake-gcs-server.

The fix should be fairly straightforward, reading the {{gs.storage.root.url}} 
config option from the Hadoop FileSystem config in 
[{{GSFileSystemOptions}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemOptions.java#L30]
 and, if set, passing it to {{storageOptionsBuilder}} in 
[{{GSFileSystemFactory}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemFactory.java].

The only workaround for this is to build a custom flink-gs-fs-hadoop JAR with a 
patch and use it as a plugin.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32583) RestClient can deadlock if request made after Netty event executor terminated

[Patrick Lucas (Jira)]

Patrick Lucas created FLINK-32583:
-

 Summary: RestClient can deadlock if request made after Netty event 
executor terminated
 Key: FLINK-32583
 URL: https://issues.apache.org/jira/browse/FLINK-32583
 Project: Flink
  Issue Type: Bug
  Components: Runtime / REST
Affects Versions: 1.18.0, 1.16.3, 1.17.2
Reporter: Patrick Lucas


The RestClient can deadlock if a request is made after the Netty event executor 
has terminated.

This is due to the listener that would resolve the CompletableFuture that is 
attached to the ChannelFuture returned by the call to Netty to connect not 
being able to run because the executor to run it rejects the execution.

[RestClient.java|https://github.com/apache/flink/blob/release-1.17.1/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java#L471-L482]:
{code:java}
final ChannelFuture connectFuture = bootstrap.connect(targetAddress, 
targetPort);

final CompletableFuture channelFuture = new CompletableFuture<>();

connectFuture.addListener(
(ChannelFuture future) -> {
if (future.isSuccess()) {
channelFuture.complete(future.channel());
} else {
channelFuture.completeExceptionally(future.cause());
}
});
{code}
In this code, the call to {{addListener()}} can fail silently (only logging to 
the console), meaning any code waiting on the CompletableFuture returned by 
this method will deadlock.

There was some work in Netty around this back in 2015, but it's unclear to me 
how this situation is expected to be handled given the discussion and changes 
from these issues:
 * [https://github.com/netty/netty/issues/3449] (open)
 * [https://github.com/netty/netty/pull/3483] (closed)
 * [https://github.com/netty/netty/pull/3566] (closed)
 * [https://github.com/netty/netty/pull/5087] (merged)

I think a reasonable fix for Flink would be to check the state of 
{{connectFuture}} and {{channelFuture}} immediately after the call to 
{{addListener()}}, resolving {{channelFuture}} with {{completeExceptionally()}} 
if {{connectFuture}} is done and failed and {{channelFuture}} has not been 
completed. In the possible race condition where the listener was attached 
successfully and the connection fails instantly, the result is the same, as 
calls to {{CompletableFuture#completeExceptionally()}} are idempotent.

A workaround for users of RestClient is to call {{CompletableFuture#get(long 
timeout, TimeUnit unit)}} rather than {{#get()}} or {{#join()}} on the 
CompletableFutures it returns. However, if the call throws TimeoutException, 
the cause of the failure cannot easily be determined.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Move docker development into versioned branches

[Patrick Lucas]

This also sounds good to me. The master branch will continue to be the
publishing area for the release Dockerfiles and we can support changes
between Flink versions without additional complexity.

My only comment would be that I hope we can still keep eventual convergence
in mind—it should be possible, with some changes in Flink, to make the
differences in the Docker packaging between versions negligible, to the
point that this split might stop being necessary.

On Mon, Apr 27, 2020 at 3:07 PM Till Rohrmann  wrote:

> Thanks for starting this discussion Chesnay. Your proposal sounds good to
> me. I can see how the current setup makes the development of version
> specific features impractical. Hence, +1 for the proposed changes.
>
> Cheers,
> Till
>
> On Mon, Apr 27, 2020 at 12:19 PM David Anderson 
> wrote:
>
> > Makes sense to me. I think this would align well enough with user
> > expectations, and be more straightforward.
> >
> > David
> >
> > On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Currently, all development in docker-flink occurs on the master branch,
> > > for all releases at once. The master branch also serves as a publishing
> > > area for our docker files.
> > >
> > > This means that all versions share the same generators, templates
> (i.e.,
> > > production code that we plugin in some variables like the Flink
> version)
> > > and test setup.
> > >
> > > So far this worked fine because all versions worked the same, but we
> are
> > > now coming to a point where we are adding version-specific features,
> > > like support for Java 11 (FLINK-16260) and per-job-clusters
> > (FLINK-17164).
> > >
> > > By virtue of all files being shared this means that we would have to
> > > introduce version checks in both the generators, templates and test
> > > code, so things continue to work for all versions. This is rather
> > > painful to do, and presents maintenance problems as these conditions
> > > must of course be updated at some point, and diverging behaviors
> require
> > > all versions to be tested on each PR (in contrast to the current state
> > > where we only test 1 version).
> > >
> > > It is also simply different to the process we're using for
> > > flink/flink-shaded, creating an additional entry barrier.
> > >
> > > I propose moving the development of the docker files into dedicated
> > > branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> > > opened against dev-master, possibly ported to other versions, and when
> > > it is time for a release we will update the master branch with the
> > > latest dockerfiles. Quite similar to how things work in the Flink repo.
> > >
> > > The master branch would continue to be our release publishing area to
> > > stick to docker conventions.
> > >
> > > Regards,
> > >
> > > Chesnay
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-111: Docker image unification

[Patrick Lucas]

Thanks Andrey for working on this, and everyone else for your feedback.

This FLIP inspired me to discuss and write down some ideas I've had for a
while about configuring and running Flink (especially in Docker) that go
beyond the scope of this FLIP, but don't contradict what it sets out to do.

The crux of it is that Flink should be maximally configurable using
environment variables, and not require manipulation of the filesystem (i.e.
moving/linking JARs or editing config files) in order to run in a large
majority of cases. And beyond that, particular for running Flink in Docker,
is that as much logic as possible should be a part of Flink itself and not,
for instance, in the docker-entrypoint.sh script. I've resisted adding
additional logic to the Flink Docker images except where necessary since
the beginning, and I believe we can get to the point where the only thing
the entrypoint script does is drop privileges before invoking a script
included in Flink.

Ultimately, my ideal end-goal for running Flink in containers would fulfill
> the following points:
>
>- A user can configure all “start-time” aspects of Flink with
>environment variables, including additions to the classpath
>- Flink automatically adapts to the resources available to the
>container (such as what BashJavaUtils helps with today)
>- A user can include additional JARs using a mounted volume, or at
>image build time with convenient tooling
>- The role/mode (jobmanager, session) is specified as a command line
>argument, with a single entrypoint program sufficing for all uses of the
>image
>
> As a bonus, if we could eliminate some or most of the layers of shell
> scripts that are involved in starting a Flink server, perhaps by
> re-implementing this part of the stack in Java, and exec-ing to actually
> run Flink with the proper java CLI arguments, I think it would be a big win
> for the project.


You can read the rest of my notes here:
https://docs.google.com/document/d/1JCACSeDaqeZiXD9G1XxQBunwi-chwrdnFm38U1JxTDQ/edit

On Wed, Mar 4, 2020 at 10:34 AM Andrey Zagrebin 
wrote:

> Hi All,
>
> If you have ever touched the docker topic in Flink, you
> probably noticed that we have multiple places in docs and repos which
> address its various concerns.
>
> We have prepared a FLIP [1] to simplify the perception of docker topic in
> Flink by users. It mostly advocates for an approach of extending official
> Flink image from the docker hub. For convenience, it can come with a set of
> bash utilities and documented examples of their usage. The utilities allow
> to:
>
>- run the docker image in various modes (single job, session master,
>task manager etc)
>- customise the extending Dockerfile
>- and its entry point
>
> Eventually, the FLIP suggests to remove all other user facing Dockerfiles
> and building scripts from Flink repo, move all docker docs to
> apache/flink-docker and adjust existing docker use cases to refer to this
> new approach (mostly Kubernetes now).
>
> The first contributed version of Flink docker integration also contained
> example and docs for the integration with Bluemix in IBM cloud. We also
> suggest to maintain it outside of Flink repository (cc Markus Müller).
>
> Thanks,
> Andrey
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
>

Re: Adding a new "Docker Images" component to Jira

[Patrick Lucas]

Thanks, Chesnay!

On Fri, Feb 21, 2020 at 11:26 AM Chesnay Schepler 
wrote:

> I've added a "Release System / Docker" component.
>
> On 21/02/2020 11:19, Patrick Lucas wrote:
> > Hi,
> >
> > Could someone with permissions add a new component to the FLINK project
> in
> > Jira for the Docker images <https://github.com/apache/flink-docker>?
> >
> > There is already a "Deployment / Docker" component, but that's not quite
> > the same as maintenance/improvements on the flink-docker images.
> >
> > Either top-level "Docker Images" or perhaps "Release / Docker Images"
> would
> > be fine.
> >
> > Thanks,
> > Patrick
> >
>
>

Adding a new "Docker Images" component to Jira

[Patrick Lucas]

Hi,

Could someone with permissions add a new component to the FLINK project in
Jira for the Docker images ?

There is already a "Deployment / Docker" component, but that's not quite
the same as maintenance/improvements on the flink-docker images.

Either top-level "Docker Images" or perhaps "Release / Docker Images" would
be fine.

Thanks,
Patrick

Re: [VOTE] Release 1.10.0, release candidate #3

[Patrick Lucas]

Now that [FLINK-15828] Integrate docker-flink/docker-flink into Flink
release process  is
complete, the Dockerfiles for 1.10.0 can be published as part of the
release process.

@Gary/@Yu: please let me know if you have any questions regarding the
workflow or its documentation.

--
Patrick

On Mon, Feb 10, 2020 at 1:29 PM Benchao Li  wrote:

> +1 (non-binding)
>
> - build from source
> - start standalone cluster, and run some examples
> - played with sql-client with some simple sql
> - run tests in IDE
> - run some sqls running in 1.9 internal version with 1.10.0-rc3, seems 1.10
> behaves well.
>
> Xintong Song  于2020年2月10日周一 下午8:13写道：
>
> > +1 (non-binding)
> >
> > - build from source (with tests)
> > - run nightly e2e tests
> > - run example jobs in local/standalone/yarn setups
> > - play around with memory configurations on local/standalone/yarn setups
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Feb 10, 2020 at 7:55 PM Jark Wu  wrote:
> >
> > > +1 (binding)
> > >
> > > - build the source release with Scala 2.12 and Scala 2.11 successfully
> > > - checked/verified signatures and hashes
> > > - started cluster for both Scala 2.11 and 2.12, ran examples, verified
> > web
> > > ui and log output, nothing unexpected
> > > - started cluster and run some e2e sql queries, all of them works well
> > and
> > > the results are as expected:
> > >   - read from kafka source, aggregate, write into mysql
> > >   - read from kafka source with watermark defined in ddl, window
> > aggregate,
> > > write into mysql
> > >   - read from kafka with computed column defined in ddl, temporal join
> > with
> > > a mysql table, write into kafka
> > >
> > > Cheers,
> > > Jark
> > >
> > >
> > > On Mon, 10 Feb 2020 at 19:23, Kurt Young  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > - verified signatures and checksums
> > > > - start local cluster, run some examples, randomly play some sql with
> > sql
> > > > client, no suspicious error/warn log found in log files
> > > > - repeat above operation with both scala 2.11 and 2.12 binary
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Mon, Feb 10, 2020 at 6:38 PM Yang Wang 
> > wrote:
> > > >
> > > > >  +1 non-binding
> > > > >
> > > > >
> > > > > - Building from source with all tests skipped
> > > > > - Build a custom image with 1.10-rc3
> > > > > - K8s tests
> > > > > * Deploy a standalone session cluster on K8s and submit
> multiple
> > > jobs
> > > > > * Deploy a standalone per-job cluster
> > > > > * Deploy a native session cluster on K8s with/without HA
> > > configured,
> > > > > kill TM and jobs could recover successfully
> > > > >
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Jingsong Li  于2020年2月10日周一 下午4:29写道：
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > +1 (non-binding) Thanks for driving this, Gary & Yu.
> > > > > >
> > > > > >
> > > > > > There is an unfriendly error here: "OutOfMemoryError: Direct
> buffer
> > > > > memory"
> > > > > > in FileChannelBoundedData$FileBufferReader.
> > > > > >
> > > > > > It forces our batch users to configure
> > > > > > "taskmanager.memory.task.off-heap.size" in production jobs. And
> > users
> > > > are
> > > > > > hard to know how much memory they need configure.
> > > > > >
> > > > > > Even for us developers, it is hard to say how much memory, it
> > depends
> > > > on
> > > > > > tasks left over from the previous stage and the parallelism.
> > > > > >
> > > > > >
> > > > > > It is not a blocker, but hope to resolve it in 1.11.
> > > > > >
> > > > > >
> > > > > > - Verified signatures and checksums
> > > > > >
> > > > > > - Maven build from source skip tests
> > > > > >
> > > > > > - Verified pom files point to the 1.10.0 version
> > > > > >
> > > > > > - Test Hive integration and SQL client: work well
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Jingsong Lee
> > > > > >
> > > > > > On Mon, Feb 10, 2020 at 12:28 PM Zhu Zhu 
> > wrote:
> > > > > >
> > > > > > > My bad. The missing commit info is caused by building from the
> > src
> > > > code
> > > > > > zip
> > > > > > > which does not contain the git info.
> > > > > > > So this is not a problem.
> > > > > > >
> > > > > > > +1 (binding) for rc3
> > > > > > > Here's what's were verified :
> > > > > > >  * built successfully from the source code
> > > > > > >  * run a sample streaming and a batch job with parallelism=1000
> > on
> > > > yarn
> > > > > > > cluster, with the new scheduler and legacy scheduler, the job
> > runs
> > > > well
> > > > > > > (tuned some resource configs to enable the jobs to work well)
> > > > > > >  * killed TMs to trigger failures, the jobs can finally recover
> > > from
> > > > > the
> > > > > > > failures
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Zhu Zhu
> > > > > > >
> > > > > > > Zhu Zhu  于2020年2月10日周一 上午12:31写道：
> > > > > > >
> > > > > > > > The commit info is shown as  on the web UI and in
> > 

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

[Patrick Lucas]

Thanks, Chesnay.

I've updated the release guide
<https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release>
with the new step.

--
Patrick

On Mon, Feb 10, 2020 at 12:34 PM Chesnay Schepler 
wrote:

> @Patrick You should have the required wiki permissions now.
>
> On 10/02/2020 12:12, Patrick Lucas wrote:
> > Thanks for the feedback.
> >
> > Could someone (Ufuk or Till?) grant me access to to the FLINK space
> > Confluence so I can make these changes? My Confluence username is plucas.
> >
> > Thanks,
> > Patrick
> >
> > On Mon, Feb 10, 2020 at 9:54 AM Ufuk Celebi  wrote:
> >
> >> +1 to have the README as source of truth and link to the repository from
> >> the Wiki page.
> >>
> >> – Ufuk
> >>
> >>
> >> On Mon, Feb 10, 2020 at 3:48 AM Yang Wang 
> wrote:
> >>
> >>> +1 to make flink-docker repository self-contained, including the
> document.
> >>> And others refer
> >>> to it.
> >>>
> >>>
> >>> Best,
> >>> Yang
> >>>
> >>> Till Rohrmann  于2020年2月9日周日 下午5:35写道：
> >>>
> >>>> Sounds good to me Patrick. +1 for these changes.
> >>>>
> >>>> Cheers,
> >>>> Till
> >>>>
> >>>> On Fri, Feb 7, 2020 at 3:25 PM Patrick Lucas 
> >>>> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> For FLINK-15831[1], I think the way to start is for the flink-docker
> >>>>> repo[2] itself to sufficiently document the workflow for publishing
> >>> new
> >>>>> Dockerfiles, and then update the Flink release guide in the wiki to
> >>> refer
> >>>>> to this documentation and to include this step in the "Finalize the
> >>>>> release" checklist.
> >>>>>
> >>>>> To the first point, I have opened a PR[3] on flink-docker to improve
> >>> its
> >>>>> documentation.
> >>>>>
> >>>>> And for updating the release guide, I propose the following changes:
> >>>>>
> >>>>> 1. Add a new subsection to "Finalize the release", prior to
> >>> "Checklist to
> >>>>> proceed to the next step" with the following content:
> >>>>>
> >>>>> Publish the Dockerfiles for the new release
> >>>>>> Note: the official Dockerfiles fetch the binary distribution of the
> >>>>> target
> >>>>>> Flink version from an Apache mirror. After publishing the binary
> >>>> release
> >>>>>> artifacts, mirrors can take some hours to start serving the new
> >>>>> artifacts,
> >>>>>> so you may want to wait to do this step until you are ready to
> >>> continue
> >>>>>> with the "Promote the release" steps below.
> >>>>>>
> >>>>>> Follow the instructions in the [flink-docker] repo to build the new
> >>>>>> Dockerfiles and send an updated manifest to Docker Hub so the new
> >>>> images
> >>>>>> are built and published.
> >>>>>>
> >>>>> 2. Add an entry to the "Checklist to proceed to the next step"
> >>> subsection
> >>>>> of "Finalize the release":
> >>>>>
> >>>>>> - Dockerfiles in flink-docker updated for the new Flink release
> >>> and
> >>>>>> pull request opened on the Docker official-images with an
> updated
> >>>>> manifest
> >>>>>> Please let me know if you have any questions or suggestions to
> >>> improve
> >>>>> this proposal.
> >>>>>
> >>>>> Thanks,
> >>>>> Patrick
> >>>>>
> >>>>> [1]https://issues.apache.org/jira/browse/FLINK-15831
> >>>>> [2]https://github.com/apache/flink-docker
> >>>>> [3]https://github.com/apache/flink-docker/pull/5
> >>>>>
>
>

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

[Patrick Lucas]

Thanks for the feedback.

Could someone (Ufuk or Till?) grant me access to to the FLINK space
Confluence so I can make these changes? My Confluence username is plucas.

Thanks,
Patrick

On Mon, Feb 10, 2020 at 9:54 AM Ufuk Celebi  wrote:

> +1 to have the README as source of truth and link to the repository from
> the Wiki page.
>
> – Ufuk
>
>
> On Mon, Feb 10, 2020 at 3:48 AM Yang Wang  wrote:
>
>> +1 to make flink-docker repository self-contained, including the document.
>> And others refer
>> to it.
>>
>>
>> Best,
>> Yang
>>
>> Till Rohrmann  于2020年2月9日周日 下午5:35写道：
>>
>> > Sounds good to me Patrick. +1 for these changes.
>> >
>> > Cheers,
>> > Till
>> >
>> > On Fri, Feb 7, 2020 at 3:25 PM Patrick Lucas 
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > For FLINK-15831[1], I think the way to start is for the flink-docker
>> > > repo[2] itself to sufficiently document the workflow for publishing
>> new
>> > > Dockerfiles, and then update the Flink release guide in the wiki to
>> refer
>> > > to this documentation and to include this step in the "Finalize the
>> > > release" checklist.
>> > >
>> > > To the first point, I have opened a PR[3] on flink-docker to improve
>> its
>> > > documentation.
>> > >
>> > > And for updating the release guide, I propose the following changes:
>> > >
>> > > 1. Add a new subsection to "Finalize the release", prior to
>> "Checklist to
>> > > proceed to the next step" with the following content:
>> > >
>> > > Publish the Dockerfiles for the new release
>> > > >
>> > > > Note: the official Dockerfiles fetch the binary distribution of the
>> > > target
>> > > > Flink version from an Apache mirror. After publishing the binary
>> > release
>> > > > artifacts, mirrors can take some hours to start serving the new
>> > > artifacts,
>> > > > so you may want to wait to do this step until you are ready to
>> continue
>> > > > with the "Promote the release" steps below.
>> > > >
>> > > > Follow the instructions in the [flink-docker] repo to build the new
>> > > > Dockerfiles and send an updated manifest to Docker Hub so the new
>> > images
>> > > > are built and published.
>> > > >
>> > >
>> > > 2. Add an entry to the "Checklist to proceed to the next step"
>> subsection
>> > > of "Finalize the release":
>> > >
>> > > >
>> > > >- Dockerfiles in flink-docker updated for the new Flink release
>> and
>> > > >pull request opened on the Docker official-images with an updated
>> > > manifest
>> > > >
>> > > > Please let me know if you have any questions or suggestions to
>> improve
>> > > this proposal.
>> > >
>> > > Thanks,
>> > > Patrick
>> > >
>> > > [1]https://issues.apache.org/jira/browse/FLINK-15831
>> > > [2]https://github.com/apache/flink-docker
>> > > [3]https://github.com/apache/flink-docker/pull/5
>> > >
>> >
>>
>

[DISCUSS] FLINK-15831: Add Docker image publication to release documentation

[Patrick Lucas]

Hi all,

For FLINK-15831[1], I think the way to start is for the flink-docker
repo[2] itself to sufficiently document the workflow for publishing new
Dockerfiles, and then update the Flink release guide in the wiki to refer
to this documentation and to include this step in the "Finalize the
release" checklist.

To the first point, I have opened a PR[3] on flink-docker to improve its
documentation.

And for updating the release guide, I propose the following changes:

1. Add a new subsection to "Finalize the release", prior to "Checklist to
proceed to the next step" with the following content:

Publish the Dockerfiles for the new release
>
> Note: the official Dockerfiles fetch the binary distribution of the target
> Flink version from an Apache mirror. After publishing the binary release
> artifacts, mirrors can take some hours to start serving the new artifacts,
> so you may want to wait to do this step until you are ready to continue
> with the "Promote the release" steps below.
>
> Follow the instructions in the [flink-docker] repo to build the new
> Dockerfiles and send an updated manifest to Docker Hub so the new images
> are built and published.
>

2. Add an entry to the "Checklist to proceed to the next step" subsection
of "Finalize the release":

>
>- Dockerfiles in flink-docker updated for the new Flink release and
>pull request opened on the Docker official-images with an updated manifest
>
> Please let me know if you have any questions or suggestions to improve
this proposal.

Thanks,
Patrick

[1]https://issues.apache.org/jira/browse/FLINK-15831
[2]https://github.com/apache/flink-docker
[3]https://github.com/apache/flink-docker/pull/5

Re: [RESULT] [VOTE] Release 1.9.2, release candidate #1

[Patrick Lucas]

A pull request to update the Flink Docker images to 1.9.2 is open:
https://github.com/docker-flink/docker-flink/pull/95

(The "pull" build failed because it rejects changes to the generated
Dockerfiles. I would like to improve this after this repo has been
contributed to Apache.)

On Thu, Jan 30, 2020 at 6:35 AM Hequn Cheng  wrote:

> Hi everyone,
>
> I'm happy to announce that we have unanimously approved this release.
>
> There are 4 approving votes, 3 of which are binding:
> * Chesnay (binding)
> * Gordon (binding)
> * Jincheng (binding)
> * Hequn
>
> There are no disapproving votes.
>
> Thanks, everyone!
>

Re: [VOTE] Integrate Flink Docker image publication into Flink release process

[Patrick Lucas]

Thanks for kicking this off, Ufuk.

+1 (non-binding)

--
Patrick

On Mon, Jan 27, 2020 at 5:50 PM Ufuk Celebi  wrote:

> Hey all,
>
> there is a proposal to contribute the Dockerfiles and scripts of
> https://github.com/docker-flink/docker-flink to the Flink project. The
> discussion corresponding to this vote outlines the reasoning for the
> proposal and can be found here: [1].
>
> The proposal is as follows:
> * Request a new repository apache/flink-docker
> * Migrate all files from docker-flink/docker-flink to apache/flink-docker
> * Update the release documentation to describe how to update
> apache/flink-docker for new releases
>
> Please review and vote on this proposal as follows:
> [ ] +1, Approve the proposal
> [ ] -1, Do not approve the proposal (please provide specific comments)
>
> The vote will be open for at least 3 days, ending the earliest on: January
> 30th 2020, 17:00 UTC.
>
> Cheers,
>
> Ufuk
>
> PS: I'm treating this proposal similar to a "Release Plan" as mentioned in
> the project bylaws [2]. Please let me know if you consider this a different
> category.
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Integrate-Flink-Docker-image-publication-into-Flink-release-process-td36139.html
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
>

Re: [DISCUSS] Integrate Flink Docker image publication into Flink release process

[Patrick Lucas]

ter
> > > > > > > to have the Dockerfiles in the (main)Flink repo.
> > > > > > >   - First, I think the Dockerfiles can be treated as part of
> the
> > > > > release.
> > > > > > > And it is also natural to put the corresponding version of the
> > > > > Dockerfile
> > > > > > > in the corresponding Flink release.
> > > > > > >   - Second, we can put the Dockerfiles in the path like
> > > > > > > flink/docker-flink/version/ and the version varies in different
> > > > > releases.
> > > > > > > For example, for release 1.8.3, we have a
> > flink/docker-flink/1.8.3
> > > > > > > folder(or maybe flink/docker-flink/1.8). Even though all
> > > Dockerfiles
> > > > > for
> > > > > > > supported versions are not in one path but they are still in
> one
> > > Git
> > > > > tree
> > > > > > > with different refs.
> > > > > > >   - Third, it seems the Docker Hub also supports specifying
> > > different
> > > > > > refs.
> > > > > > > For the file[1], we can change the GitRepo link from
> > > > > > > https://github.com/docker-flink/docker-flink.git to
> > > > > > > https://github.com/apache/flink.git and add a GitFetch for
> each
> > > tag,
> > > > > > e.g.,
> > > > > > > GitFetch: refs/tags/release-1.8.3. There are some examples in
> the
> > > > file
> > > > > of
> > > > > > > ubuntu[2].
> > > > > > >
> > > > > > > If the above assumptions are right and there are no more
> > obstacles,
> > > > I'm
> > > > > > > intended to have these Dockerfiles in the main Flink repo. In
> > this
> > > > > case,
> > > > > > we
> > > > > > > can reduce the number of repos and reduce the management
> > overhead.
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Best,
> > > > > > > Hequn
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/docker-library/official-images/blob/master/library/flink
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/docker-library/official-images/blob/master/library/ubuntu
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Dec 20, 2019 at 5:29 PM Yang Wang <
> danrtsey...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > >  Big +1 for this effort.
> > > > > > > >
> > > > > > > > It is really exciting we have started this great work. More
> and
> > > > more
> > > > > > > > companies start to
> > > > > > > > use Flink in container environment(docker, Kubernetes, Mesos,
> > > even
> > > > > > > > Yarn-3.x). So it is
> > > > > > > > very important that we could have unified official image
> > building
> > > > and
> > > > > > > > releasing process.
> > > > > > > >
> > > > > > > >
> > > > > > > > The image building process in this proposal is really good
> and
> > i
> > > > just
> > > > > > > have
> > > > > > > > the following thoughts.
> > > > > > > >
> > > > > > > > >> Keep a dedicated repo for Dockerfiles to build official
> > image
> > > > > > > > I think this is a good way and we do not need to make some
> > > > > unnecessary
> > > > > > > > changes to Flink repository.
> > > > > > > >
> > > > > > > > >> Integrate building image into the Flink release process
> > > > > > > > It will bring a better experience for container environment
> > > users.
> > &g

[DISCUSS] Integrate Flink Docker image publication into Flink release process

[Patrick Lucas]

ode.

By standardizing on defining such “production” images in the docker-flink
repository and “development” image(s) in the Flink repository itself, it is
much clearer to developers and users what the right Dockerfile or image
they should use for a given purpose. To that end, we could introduce one or
more documented Maven goals or Make targets for building a Docker image
from the current source tree or a specific release (including unreleased or
unsupported versions).

Additionally, there has been discussion among Flink contributors for some
time about the confusing state of Dockerfiles within the Flink repository,
each meant for a different way of running Flink. I’m not completely up to
speed about these different efforts, but we could possibly solve this by
either building additional “official” images with different entrypoints for
these various purposes, or by developing an improved entrypoint script that
conveniently supports all cases. I defer to Till Rohrmann, Konstantin
Knauf, or Stephan Ewen for further discussion on this point.

I apologize again for the wall of text, but if you made it this far, thank
you! These improvements have been a long time coming, and I hope we can
find a solution that serves the Flink and Docker communities well. Please
don’t hesitate to ask any questions.

--

Patrick Lucas

[1] https://hub.docker.com/_/flink

[2]
https://lists.apache.org/thread.html/c50297f8659aaa59d4f2ae327b69c4d46d1ab8ecc53138e659e4fe91%40%3Cdev.flink.apache.org%3E

[3] On page 2 at the time we went to press:
https://hub.docker.com/search?q==image_filter=official

[4] https://github.com/docker-flink/docker-flink

[5]
https://github.com/docker-library/official-images/pulls?q=is%3Apr+label%3Alibrary%2Fflink

[6] I looked at the 25 most popular “official” images (see [3]) as well as
“official” images of Apache software from the top 125; all use a dedicated
repo
[7] https://hub.docker.com/u/apache

Re: [RESULT] [VOTE] Release 1.8.3, release candidate #3

[Patrick Lucas]

The upstream pull request[1] is open—usually they merge within a day and
the images are available shortly thereafter.

[1] https://github.com/docker-library/official-images/pull/7112

--
Patrick Lucas

On Thu, Dec 12, 2019 at 8:11 AM Hequn Cheng  wrote:

> Hi Patrick,
>
> The release has been announced.
>
> +1 to integrate the publication of Docker images into the Flink release
> process. Thus we can leverage the current release procedure for the docker
> image.
> Looking forward to the proposal.
>
> Best, Hequn
>
> On Thu, Dec 12, 2019 at 1:52 PM Yang Wang  wrote:
>
>> Hi Lucas,
>>
>> That's great if we could integrate the publication of Flink official
>> docker
>> images into
>> the Flink release process. Since many users are using or starting to use
>> Flink in
>> container environments.
>>
>>
>> Best,
>> Yang
>>
>> Patrick Lucas  于2019年12月11日周三 下午11:44写道：
>>
>> > Thanks, Hequn!
>> >
>> > The Dockerfiles for the Flink images on Docker Hub for the 1.8.3 release
>> > are prepared[1] and I'll open a pull request upstream[2] once the
>> release
>> > announcement has gone out.
>> >
>> > And stay tuned: I'm working on a proposal for integrating the
>> publication
>> > of these Docker images into the Flink release process and will send it
>> out
>> > to the dev list before or shortly after the holidays.
>> >
>> > [1]
>> >
>> >
>> https://github.com/docker-flink/docker-flink/commit/4d85b71b7cf9fa4a38f8682ed13aa0f55445e32e
>> > [2] https://github.com/docker-library/official-images/
>> >
>> > On Wed, Dec 11, 2019 at 3:30 AM Hequn Cheng 
>> wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > I'm happy to announce that we have unanimously approved this release.
>> > >
>> > > There are 10 approving votes, 3 of which are binding:
>> > > * Jincheng (binding)
>> > > * Ufuk (binding)
>> > > * Till (binding)
>> > > * Jingsong
>> > > * Fabian
>> > > * Danny
>> > > * Yang Wang
>> > > * Dian
>> > > * Wei
>> > > * Hequn
>> > >
>> > > There are no disapproving votes.
>> > >
>> > > Thanks everyone!
>> > >
>> >
>>
>

Re: [RESULT] [VOTE] Release 1.8.3, release candidate #3

[Patrick Lucas]

Thanks, Hequn!

The Dockerfiles for the Flink images on Docker Hub for the 1.8.3 release
are prepared[1] and I'll open a pull request upstream[2] once the release
announcement has gone out.

And stay tuned: I'm working on a proposal for integrating the publication
of these Docker images into the Flink release process and will send it out
to the dev list before or shortly after the holidays.

[1]
https://github.com/docker-flink/docker-flink/commit/4d85b71b7cf9fa4a38f8682ed13aa0f55445e32e
[2] https://github.com/docker-library/official-images/

On Wed, Dec 11, 2019 at 3:30 AM Hequn Cheng  wrote:

> Hi everyone,
>
> I'm happy to announce that we have unanimously approved this release.
>
> There are 10 approving votes, 3 of which are binding:
> * Jincheng (binding)
> * Ufuk (binding)
> * Till (binding)
> * Jingsong
> * Fabian
> * Danny
> * Yang Wang
> * Dian
> * Wei
> * Hequn
>
> There are no disapproving votes.
>
> Thanks everyone!
>

[jira] [Created] (FLINK-14973) OSS filesystem does not relocate many dependencies

[Patrick Lucas (Jira)]

Patrick Lucas created FLINK-14973:
-

 Summary: OSS filesystem does not relocate many dependencies
 Key: FLINK-14973
 URL: https://issues.apache.org/jira/browse/FLINK-14973
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.9.1
Reporter: Patrick Lucas


Whereas the Azure and S3 Hadoop filesystem jars relocate all of their 
depdendencies:

{noformat}
$ jar tf opt/flink-azure-fs-hadoop-1.9.1.jar | grep -v '^org/apache/fs/shaded/' 
| grep -v '^META-INF' | grep '/' | cut -f -2 -d / | sort | uniq
org/
org/apache

$ jar tf opt/flink-s3-fs-hadoop-1.9.1.jar | grep -v '^org/apache/fs/shaded/' | 
grep -v '^META-INF' | grep '/' | cut -f -2 -d / | sort | uniq
org/
org/apache
{noformat}

The OSS Hadoop filesystem leaves many things un-relocated:

{noformat}
$ jar tf opt/flink-oss-fs-hadoop-1.9.1.jar | grep -v '^org/apache/fs/shaded/' | 
grep -v '^META-INF' | grep '/' | cut -f -2 -d / | sort | uniq
assets/
assets/org
avro/
avro/shaded
com/
com/ctc
com/fasterxml
com/google
com/jcraft
com/nimbusds
com/sun
com/thoughtworks
javax/
javax/activation
javax/el
javax/servlet
javax/ws
javax/xml
jersey/
jersey/repackaged
licenses/
licenses/LICENSE.asm
licenses/LICENSE.cddlv1.0
licenses/LICENSE.cddlv1.1
licenses/LICENSE.jdom
licenses/LICENSE.jzlib
licenses/LICENSE.paranamer
licenses/LICENSE.protobuf
licenses/LICENSE.re2j
licenses/LICENSE.stax2api
net/
net/jcip
net/minidev
org/
org/apache
org/codehaus
org/eclipse
org/jdom
org/objectweb
org/tukaani
org/xerial
{noformat}

The first symptom of this I ran into was that Flink is unable to restore from a 
savepoint if both the OSS and Azure Hadoop filesystems are on the classpath, 
but I assume this has the potential to cause further problems, at least until 
more progress is made on the module/classloading front.

h3. Steps to reproduce

# Copy both the Azure and OSS Hadoop filesystem JARs from opt/ into lib/
# Run a job that restores from a savepoint (the savepoint might need to be 
stored on OSS)
# See a crash and traceback like:

{noformat}
2019-11-26 15:59:25,318 ERROR 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error 
occurred in the cluster entrypoint.
org.apache.flink.runtime.dispatcher.DispatcherException: Failed to take 
leadership with session id ----.
at 
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$30(Dispatcher.java:915)
 ~[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
 ~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
 ~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) 
~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
 ~[?:1.8.0_232]
at 
org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:691)
 ~[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
 ~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
 ~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) 
~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) 
~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:753)
 ~[?:1.8.0_232]
at 
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
 ~[?:1.8.0_232]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
 ~[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
 ~[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
 ~[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
 ~[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) 
[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) 
[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) 
[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) 
[flink-dist_2.12-1.9.1-stream2.jar:1.9.1-stream2

[jira] [Created] (FLINK-12191) Flink SVGs on "Material" page broken, render incorrectly on Firefox

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-12191:
-

 Summary: Flink SVGs on "Material" page broken, render incorrectly 
on Firefox
 Key: FLINK-12191
 URL: https://issues.apache.org/jira/browse/FLINK-12191
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Patrick Lucas
Assignee: Patrick Lucas
 Attachments: Screen Shot 2019-04-15 at 09.48.15.png

Like FLINK-11043, the Flink SVGs on the [Material page of the Flink 
website|https://flink.apache.org/material.html] are invalid and do not render 
correctly on Firefox.

I'm not sure if there is an additional source-of-truth for these images, or if 
these hosted on the website are canonical, but I can fix them nonetheless.

I also noticed that one of the squirrels in both {{color_black.svg}} and 
{{color_white.svg}} is missing the eye gradient, which can also be easily fixed.

 !Screen Shot 2019-04-15 at 09.48.15.png|thumbnail!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Flink 1.8.0 released

[Patrick Lucas]

The Docker images for 1.8.0 are now available.

Note that like Flink itself, starting with Flink 1.8 we are no longer
providing images with bundled Hadoop. Users who need an image with bundled
Hadoop should create their own, starting from the images published on
Docker Hub and fetching the appropriate shaded Hadoop jar as listed on the
Flink website's download page.

If there is demand, we can also look into creating a series of "downstream"
images that include Hadoop.

--
Patrick

On Thu, Apr 11, 2019 at 2:05 PM Richard Deurwaarder  wrote:

> Very nice! Thanks Aljoscha and all contributors!
>
> I have one question, will the docker image for 1.8.0 be released soon as
> well? https://hub.docker.com/_/flink has the versions up to 1.7.2.
>
> Regards,
>
> Richard
>
> On Wed, Apr 10, 2019 at 4:54 PM Rong Rong  wrote:
>
>> Congrats! Thanks Aljoscha for being the release manager and all for
>> making the release possible.
>>
>> --
>> Rong
>>
>>
>> On Wed, Apr 10, 2019 at 4:23 AM Stefan Richter 
>> wrote:
>>
>>> Congrats and thanks to Aljoscha for managing the release!
>>>
>>> Best,
>>> Stefan
>>>
>>> > On 10. Apr 2019, at 13:01, Biao Liu  wrote:
>>> >
>>> > Great news! Thanks Aljoscha and all the contributors.
>>> >
>>> > Till Rohrmann mailto:trohrm...@apache.org>>
>>> 于2019年4月10日周三 下午6:11写道：
>>> > Thanks a lot to Aljoscha for being our release manager and to the
>>> community making this release possible!
>>> >
>>> > Cheers,
>>> > Till
>>> >
>>> > On Wed, Apr 10, 2019 at 12:09 PM Hequn Cheng >> > wrote:
>>> > Thanks a lot for the great release Aljoscha!
>>> > Also thanks for the work by the whole community. :-)
>>> >
>>> > Best, Hequn
>>> >
>>> > On Wed, Apr 10, 2019 at 6:03 PM Fabian Hueske >> > wrote:
>>> > Congrats to everyone!
>>> >
>>> > Thanks Aljoscha and all contributors.
>>> >
>>> > Cheers, Fabian
>>> >
>>> > Am Mi., 10. Apr. 2019 um 11:54 Uhr schrieb Congxian Qiu <
>>> qcx978132...@gmail.com >:
>>> > Cool!
>>> >
>>> > Thanks Aljoscha a lot for being our release manager, and all the
>>> others who make this release possible.
>>> >
>>> > Best, Congxian
>>> > On Apr 10, 2019, 17:47 +0800, Jark Wu >> imj...@gmail.com>>, wrote:
>>> > > Cheers!
>>> > >
>>> > > Thanks Aljoscha and all others who make 1.8.0 possible.
>>> > >
>>> > > On Wed, 10 Apr 2019 at 17:33, vino yang >> > wrote:
>>> > >
>>> > > > Great news!
>>> > > >
>>> > > > Thanks Aljoscha for being the release manager and thanks to all the
>>> > > > contributors!
>>> > > >
>>> > > > Best,
>>> > > > Vino
>>> > > >
>>> > > > Driesprong, Fokko  于2019年4月10日周三 下午4:54写道：
>>> > > >
>>> > > > > Great news! Great effort by the community to make this happen.
>>> Thanks all!
>>> > > > >
>>> > > > > Cheers, Fokko
>>> > > > >
>>> > > > > Op wo 10 apr. 2019 om 10:50 schreef Shaoxuan Wang <
>>> wshaox...@gmail.com >:
>>> > > > >
>>> > > > > > Thanks Aljoscha and all others who made contributions to FLINK
>>> 1.8.0.
>>> > > > > > Looking forward to FLINK 1.9.0.
>>> > > > > >
>>> > > > > > Regards,
>>> > > > > > Shaoxuan
>>> > > > > >
>>> > > > > > On Wed, Apr 10, 2019 at 4:31 PM Aljoscha Krettek <
>>> aljos...@apache.org >
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > The Apache Flink community is very happy to announce the
>>> release of
>>> > > > > > Apache
>>> > > > > > > Flink 1.8.0, which is the next major release.
>>> > > > > > >
>>> > > > > > > Apache Flink® is an open-source stream processing framework
>>> for
>>> > > > > > > distributed, high-performing, always-available, and accurate
>>> data
>>> > > > > > streaming
>>> > > > > > > applications.
>>> > > > > > >
>>> > > > > > > The release is available for download at:
>>> > > > > > > https://flink.apache.org/downloads.html <
>>> https://flink.apache.org/downloads.html>
>>> > > > > > >
>>> > > > > > > Please check out the release blog post for an overview of the
>>> > > > > > improvements
>>> > > > > > > for this bugfix release:
>>> > > > > > > https://flink.apache.org/news/2019/04/09/release-1.8.0.html
>>> 
>>> > > > > > >
>>> > > > > > > The full release notes are available in Jira:
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12344274
>>> <
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12344274
>>> >
>>> > > > > > >
>>> > > > > > > We would like to thank all contributors of the Apache Flink
>>> community
>>> > > > > who
>>> > > > > > > made this release possible!
>>> > > > > > >
>>> > > > > > > Regards,
>>> > > > > > > Aljoscha
>>> > > > > >
>>> > > > >
>>> > > >
>>>
>>>

[jira] [Created] (FLINK-11462) "Powered by Flink" page does not display correctly on Firefox

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-11462:
-

 Summary: "Powered by Flink" page does not display correctly on 
Firefox
 Key: FLINK-11462
 URL: https://issues.apache.org/jira/browse/FLINK-11462
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Patrick Lucas
Assignee: Patrick Lucas
 Attachments: Screen Shot 2019-01-30 at 12.13.03.png

The JavaScript that sets the height of each "tile" does not work as expected in 
Firefox, causing them to overlap (see attached screenshot). [This Stack 
Overflow 
post|https://stackoverflow.com/questions/12184133/jquery-wrong-values-when-trying-to-get-div-height]
 suggests using jQuery's {{outerHeight}} instead of {{height}} method, and 
making this change seems to make the page display correctly in both Firefox and 
Chrome.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11043) Website Flink logo broken on Firefox

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-11043:
-

 Summary: Website Flink logo broken on Firefox
 Key: FLINK-11043
 URL: https://issues.apache.org/jira/browse/FLINK-11043
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Patrick Lucas
Assignee: Patrick Lucas
 Attachments: Screen Shot 2018-11-30 at 14.48.43.png

The gradients the Flink SVG logo do not render on Firefox due to this 
(12-year-old) bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=353575]

Example:

!Screen Shot 2018-11-30 at 14.48.43.png!

This StackOverflow post links to the above issue and explains that Firefox does 
not render gradient elements that are inside a "symbol" element in SVGs, and 
the fix is to extract all the gradient definitions into a top-level "defs" 
section.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9806) Add a canonical link element to documentation HTML

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-9806:


 Summary: Add a canonical link element to documentation HTML
 Key: FLINK-9806
 URL: https://issues.apache.org/jira/browse/FLINK-9806
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.5.0
Reporter: Patrick Lucas


Flink has suffered for a while with non-optimal SEO for its documentation, 
meaning a web search for a topic covered in the documentation often produces 
results for many versions of Flink, even preferring older versions since those 
pages have been around for longer.

Using a canonical link element (see references) may alleviate this by informing 
search engines about where to find the latest documentation (i.e. pages hosted 
under [https://ci.apache.org/projects/flink/flink-docs-master/).]

I think this is at least worth experimenting with, and if it doesn't cause 
problems, even backporting it to the older release branches to eventually clean 
up the Flink docs' SEO and converge on advertising only the latest docs (unless 
a specific version is specified).

References:
 * [https://moz.com/learn/seo/canonicalization]
 * [https://yoast.com/rel-canonical/]
 * [https://support.google.com/webmasters/answer/139066?hl=en]
 * [https://en.wikipedia.org/wiki/Canonical_link_element]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Docker images

[Patrick Lucas]

Hi Felix,

As a brief note, these images are not "official" in that they are owned by
the Apache Flink project or the ASF, but they are maintained by the Flink
community.

The sources currently live in a separate repo to make them much easier to
version and maintain independent of Flink itself. This is a very common
practice and applies to most other popular "official" images hosted on
Docker Hub.

--
Patrick Lucas

On Wed, Dec 20, 2017 at 4:20 AM, Felix Cheung <felixcheun...@hotmail.com>
wrote:

> Hi!
>
> Is there a reason the official docker images
> https://hub.docker.com/r/_/flink/
>
> Has sources in a different repo
> https://github.com/docker-flink/docker-flink
> ?
>

[jira] [Created] (FLINK-7155) Add Influxdb metrics reporter

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-7155:


 Summary: Add Influxdb metrics reporter
 Key: FLINK-7155
 URL: https://issues.apache.org/jira/browse/FLINK-7155
 Project: Flink
  Issue Type: Improvement
Reporter: Patrick Lucas
Assignee: Patrick Lucas
 Fix For: 1.4.0


[~jgrier] has a [simple Influxdb metrics reporter for 
Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] 
that is a thing wrapper around [a lightweight, public-domain Influxdb 
reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.

We can implement this very easily in Java in the same as as 
flink-metrics-graphite.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Apache Flink 1.3.0 released

[Patrick Lucas]

And here's the PR for the new Docker images:
https://github.com/docker-library/official-images/pull/3006

(It was merged 10 min ago; the new images should become available by
tomorrow)

--
Patrick Lucas

On Thu, Jun 1, 2017 at 7:36 PM, Eron Wright <eronwri...@gmail.com> wrote:

> Opened a PR for Homebrew update to `apache-flink` formula:
> https://github.com/Homebrew/homebrew-core/pull/14151
>
> On Thu, Jun 1, 2017 at 7:48 AM, Robert Metzger <rmetz...@apache.org>
> wrote:
>
> > The Apache Flink community is pleased to announce the release of Apache
> > Flink 1.3.0.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> streaming
> > applications.
> >
> >
> > The release is available for download at:
> >
> > https://flink.apache.org/downloads.html
> >
> > There is a blog post available on the Flink site explaining most of the
> > new features:
> >
> >https://flink.apache.org/news/2017/06/01/release-1.3.0.html
> >
> > The full changelog is available at: https://flink.apache.org/
> > blog/release_1.3.0-changelog.html
> >
> >
> >
> > We would like to thank all contributors who made this release possible!
> >
>

Re: Official Flink Docker images

[Patrick Lucas]

I've been informed that images don't make it through the list!

You can see the aforementioned squirrel here
<https://activerain-store.s3.amazonaws.com/image_store/uploads/8/7/6/3/9/ar12988558393678.JPG>
.

--
Patrick Lucas

On Thu, Apr 27, 2017 at 12:21 PM, Patrick Lucas <patr...@data-artisans.com>
wrote:

> As part of an ongoing effort to improve the experience of using Flink on
> Docker, some work has been done over the last two months to publish
> official Flink Docker images to Docker Hub. The goal in the short term is
> to make running a simple Flink cluster (almost) as easy as running docker
> run flink. In the long term, we would like these images to be good enough
> to use in production directly, or as base images for use in an existing
> Docker workflow.
>
> Flink 1.2.1 has some fixes over the last few releases that make running it
> on Docker nicer—and in some cases, possible. Notably, FLINK-2821
> <https://issues.apache.org/jira/browse/FLINK-2821> allowed Dockerized
> Flink to run across multiple hosts, and FLINK-4326
> <https://issues.apache.org/jira/browse/FLINK-4326> added an option to run
> Flink in the foreground, which is greatly preferred when running in Docker.
>
> We (Ismaël Mejía and myself, with some discussion with Stephan Ewen)
> decided it made sense to bring the actual Dockerfiles outside of the Apache
> Flink git repo, primarily to conform with every other Apache project that
> has official images, but also because these scripts logically exist
> decoupled from any particular Flink version. They are still Apache-licensed
> and maintained by the community.
>
> Please reply here or on the relevant JIRA/GitHub issue if you have
> questions or feedback.
>
> Here's a squirrel in a container:
>
>
>
> References:
>
>- FLINK-3026: Publish the flink docker container to the docker registry
><https://issues.apache.org/jira/browse/FLINK-3026>
>- Repo for the Dockerfiles and the scripts that generate them
><https://github.com/docker-flink/docker-flink>
>- GitHub PR to add the official images to Docker Hub
><https://github.com/docker-library/official-images/pull/2895>
>- Improvements to the quality of running Flink on Docker to be made in
>future Flink releases:
>   - FLINK-6300: PID1 of docker images does not behave correctly
>   <https://issues.apache.org/jira/browse/FLINK-6300>
>   - FLINK-6369: Better support for overlay networks
>   <https://issues.apache.org/jira/browse/FLINK-6369>
>
> Thanks to Ismaël Mejía, Jamie Grier, and Stephan Ewen for their
> contributions.
>
> --
> Patrick Lucas
>

Official Flink Docker images

[Patrick Lucas]

As part of an ongoing effort to improve the experience of using Flink on
Docker, some work has been done over the last two months to publish
official Flink Docker images to Docker Hub. The goal in the short term is
to make running a simple Flink cluster (almost) as easy as running docker
run flink. In the long term, we would like these images to be good enough
to use in production directly, or as base images for use in an existing
Docker workflow.

Flink 1.2.1 has some fixes over the last few releases that make running it
on Docker nicer—and in some cases, possible. Notably, FLINK-2821
<https://issues.apache.org/jira/browse/FLINK-2821> allowed Dockerized Flink
to run across multiple hosts, and FLINK-4326
<https://issues.apache.org/jira/browse/FLINK-4326> added an option to run
Flink in the foreground, which is greatly preferred when running in Docker.

We (Ismaël Mejía and myself, with some discussion with Stephan Ewen)
decided it made sense to bring the actual Dockerfiles outside of the Apache
Flink git repo, primarily to conform with every other Apache project that
has official images, but also because these scripts logically exist
decoupled from any particular Flink version. They are still Apache-licensed
and maintained by the community.

Please reply here or on the relevant JIRA/GitHub issue if you have
questions or feedback.

Here's a squirrel in a container:



References:

   - FLINK-3026: Publish the flink docker container to the docker registry
   <https://issues.apache.org/jira/browse/FLINK-3026>
   - Repo for the Dockerfiles and the scripts that generate them
   <https://github.com/docker-flink/docker-flink>
   - GitHub PR to add the official images to Docker Hub
   <https://github.com/docker-library/official-images/pull/2895>
   - Improvements to the quality of running Flink on Docker to be made in
   future Flink releases:
  - FLINK-6300: PID1 of docker images does not behave correctly
  <https://issues.apache.org/jira/browse/FLINK-6300>
  - FLINK-6369: Better support for overlay networks
  <https://issues.apache.org/jira/browse/FLINK-6369>

Thanks to Ismaël Mejía, Jamie Grier, and Stephan Ewen for their
contributions.

--
Patrick Lucas

[jira] [Created] (FLINK-6369) Better support for overlay networks

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-6369:


 Summary: Better support for overlay networks
 Key: FLINK-6369
 URL: https://issues.apache.org/jira/browse/FLINK-6369
 Project: Flink
  Issue Type: Improvement
  Components: Docker, Network
Affects Versions: 1.2.0
Reporter: Patrick Lucas


Running Flink in an environment that utilizes an overlay network (containerized 
environments like Kubernetes or Docker Compose, or cloud platforms like AWS 
VPC) poses various challenges related to networking.

The core problem is that in these environments, applications are frequently 
addressed by a name different from that with which the application sees itself.

For instance, it is plausible that the Flink UI (served by the Jobmanager) is 
accessed via an ELB, which poses a problem in HA mode when the non-leader UI 
returns an HTTP redirect to the leader—but the user may not be able to connect 
directly to the leader.

Or, if a user is using [Docker 
Compose|https://github.com/apache/flink/blob/aa21f853ab0380ec1f68ae1d0b7c8d9268da4533/flink-contrib/docker-flink/docker-compose.yml],
 they cannot submit a job via the CLI since there is a mismatch between the 
name used to address the Jobmanager and what the Jobmanager perceives as its 
hostname. (see \[1] below for more detail)



h3. Problems and proposed solutions

There are four instances of this issue that I've run into so far:

h4. Jobmanagers must be addressed by the same name they are configured with due 
to limitations of Akka

Akka enforces that messages it receives are addressed with the hostname it is 
configured with. Newer versions of Akka (>= 2.4) than what Flink uses 
(2.3-custom) have support for accepting messages with the "wrong" hostname, but 
it limited to a single "external" hostname.

In many environments, it is likely that not all parties that want to connect to 
the Jobmanager have the same way of addressing it (e.g. the ELB example above). 
Other similarly-used protocols like HTTP generally don't have this restriction: 
if you connect on a socket and send a well-formed message, the system assumes 
that it is the desired recipient.

One solution is to not use Akka at all when communicating with the cluster from 
the outside, perhaps using an HTTP API instead. This would be somewhat 
involved, and probabyl best left as a longer-term goal.

A more immediate solution would be to override this behavior within Flakka, the 
custom fork of Akka currently in use by Flink. I'm not sure how much effort 
this would take.

h4. The Jobmanager needs to be able to address the Taskmanagers for e.g. 
metrics collection

Having the Taskmanagers register themselves by IP is probably the best solution 
here. It's a reasonable assumption that IPs can always be used for 
communication between the nodes of a single cluster. Asking that each 
Taskmanager container have a resolvable hostname is unreasonable.

h4. Jobmanagers in HA mode send HTTP redirects to URLs that aren't externally 
resolvable/routable

If multiple Jobmanagers are used in HA mode, HTTP requests to non-leaders (such 
as if you put a Kubernetes Service in front of all Jobmanagers in a cluster) 
get redirected to the (supposed) hostname of the leader, but this is 
potentially unresolvable/unroutable externally.

Enabling non-leader Jobmanagers to proxy API calls to the leader would solve 
this. The non-leaders could even serve static asset requests (e.g. for css or 
js files) directly.

h4. Queryable state requests involve direct communication with Taskmanagers

Currently, queryable state requests involve communication between the client 
and the Jobmanager (for key partitioning lookups) and between the client and 
all Taskmanagers.

If the client is inside the network (as would be common in production use-cases 
where high-volume lookups are required) this is a non-issue, but problems crop 
up if the client is outside the network.

For the communication with the Jobmanager, a similar solution as above can be 
used: if all Jobmanagers can service all key partitioning lookup requests (e.g. 
by proxying) then a simple Service can be used.

The story is a bit different for the Taskmanagers. The partitioning lookup to 
the Jobmanager would return the name of the particular Taskmanager that owned 
the desired data, but that name (likely an IP, as proposed in the second 
section above) is not necessarily resolvable/routable from the client.

In the context of Kubernetes, where individual containers are generally not 
addressible, a very ugly solution would involve creating a Service for each 
Taskmanager, then cleverly configuring things such that the same name could be 
used to address a specific Taskmanager both inside and outside the network. \[2]

A much nicer solution would be, like in the previous section, to enable 
Taskmanagers to proxy any queryable state lo

[jira] [Created] (FLINK-6330) Improve Docker documentation

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-6330:


 Summary: Improve Docker documentation
 Key: FLINK-6330
 URL: https://issues.apache.org/jira/browse/FLINK-6330
 Project: Flink
  Issue Type: Bug
  Components: Docker
Affects Versions: 1.2.0
Reporter: Patrick Lucas
Assignee: Patrick Lucas
 Fix For: 1.2.2


The "Docker" page in the docs exists but is blank.

Add something useful here, including references to the official images that 
should exist once 1.2.1 is released, and add a brief "Kubernetes" page as well, 
referencing the [helm chart|https://github.com/docker-flink/examples]. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6194) More broken links in docs

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-6194:


 Summary: More broken links in docs
 Key: FLINK-6194
 URL: https://issues.apache.org/jira/browse/FLINK-6194
 Project: Flink
  Issue Type: Bug
Reporter: Patrick Lucas
Assignee: Patrick Lucas


My script noticed a few broken links that made it into the docs. I'll fix them 
up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6021) Downloads page references "Hadoop 1 version" which isn't an option

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-6021:


 Summary: Downloads page references "Hadoop 1 version" which isn't 
an option
 Key: FLINK-6021
 URL: https://issues.apache.org/jira/browse/FLINK-6021
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Patrick Lucas


The downloads pages says

{quote}
Apache Flink® 1.2.0 is our latest stable release.

You don’t have to install Hadoop to use Flink, but if you plan to use Flink 
with data stored in Hadoop, pick the version matching your installed Hadoop 
version. If you don’t want to do this, pick the Hadoop 1 version.
{quote}

But Hadoop 1 appears to no longer be an available alternative.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5997) Docker build tools should inline GPG key to verify HTTP Flink release download

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-5997:


 Summary: Docker build tools should inline GPG key to verify HTTP 
Flink release download
 Key: FLINK-5997
 URL: https://issues.apache.org/jira/browse/FLINK-5997
 Project: Flink
  Issue Type: Improvement
  Components: Docker
Reporter: Patrick Lucas
Assignee: Patrick Lucas


See title



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5801) Queryable State from Scala job/client fails with key of type Long

[Patrick Lucas (JIRA)]

Patrick Lucas created FLINK-5801:


 Summary: Queryable State from Scala job/client fails with key of 
type Long
 Key: FLINK-5801
 URL: https://issues.apache.org/jira/browse/FLINK-5801
 Project: Flink
  Issue Type: Bug
  Components: Queryable State
Affects Versions: 1.2.0
 Environment: Flink 1.2.0
Scala 2.10
Reporter: Patrick Lucas
 Attachments: OrderFulfillment.scala, OrderFulfillmentStateQuery.scala

While working on a demo Flink job, to try out Queryable State, I exposed some 
state of type Long -> custom class via the Query server. However, the query 
server returned an exception when I tried to send a query:

{noformat}
Exception in thread "main" java.lang.RuntimeException: Failed to query state 
backend for query 0. Caused by: java.io.IOException: Unable to deserialize key 
and namespace. This indicates a mismatch in the key/namespace serializers used 
by the KvState instance and this access.
at 
org.apache.flink.runtime.query.netty.message.KvStateRequestSerializer.deserializeKeyAndNamespace(KvStateRequestSerializer.java:392)
at 
org.apache.flink.runtime.state.heap.AbstractHeapState.getSerializedValue(AbstractHeapState.java:130)
at 
org.apache.flink.runtime.query.netty.KvStateServerHandler$AsyncKvStateQueryTask.run(KvStateServerHandler.java:220)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at 
org.apache.flink.runtime.util.DataInputDeserializer.readLong(DataInputDeserializer.java:217)
at 
org.apache.flink.api.common.typeutils.base.LongSerializer.deserialize(LongSerializer.java:69)
at 
org.apache.flink.api.common.typeutils.base.LongSerializer.deserialize(LongSerializer.java:27)
at 
org.apache.flink.runtime.query.netty.message.KvStateRequestSerializer.deserializeKeyAndNamespace(KvStateRequestSerializer.java:379)
... 7 more

at 
org.apache.flink.runtime.query.netty.KvStateServerHandler$AsyncKvStateQueryTask.run(KvStateServerHandler.java:257)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}

I banged my head against this for a while, then per [~jgrier]'s suggestion I 
tried simply changing the key from Long to String (modifying the two {{keyBy}} 
calls and the {{keySerializer}} {{TypeHint}} in the attached code) and it 
started working perfectly.

cc [~uce]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)