GitHub user zhgao opened a pull request:
https://github.com/apache/flink/pull/6025
Release 1.4
*Thank you very much for contributing to Apache Flink - we are happy that
you want to help us improve Flink. To help the community review your
contribution in the best possible way, please go through the checklist below,
which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a
hassle. In order to uphold a high standard of quality for code contributions,
while at the same time managing a large number of contributions, we need
contributors to prepare the contributions well, and give reviewers enough
contextual information for the review. Please also understand that
contributions that do not follow this guide will take longer to review and thus
typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA
issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are
made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of
the pull request", where *FLINK-XXXX* should be replaced by the actual issue
number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following
this pattern: `[hotfix] [docs] Fix typo in event time introduction` or
`[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean
verify` passes. You can set up Travis CI to do that following [this
guide](http://flink.apache.org/contribute-code.html#best-practices).
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
(including the JIRA id)
- Once all items of the checklist are addressed, remove the above text
and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
## What is the purpose of the change
*(For example: This pull request makes task deployment go through the blob
server, rather than through RPC. That way we avoid re-transferring them on each
deployment (during recovery).)*
## Brief change log
*(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a
persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache*
## Verifying this change
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(100MB)*
- *Extended integration test for recovery after master (JobManager)
failure*
- *Added test that validates that TaskInfo is transferred only once
across recoveries*
- *Manually verified the change by running a 4 node cluser with 2
JobManagers and 4 TaskManagers, a stateful streaming program, and killing one
JobManager and two TaskManagers during the execution, verifying that recovery
happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no /
don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs /
JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/flink release-1.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6025.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6025
----
commit f0012645986be200f248e9c469fe69e81bbe8d4b
Author: yew1eb <yew1eb@...>
Date: 2017-10-10T18:52:35Z
[hotfix][license] Add missing licenses
This close #4794.
commit 28b3115d23ab6844583578f7f2c7c37316c199be
Author: Joerg Schad <joerg@...>
Date: 2017-11-18T06:23:02Z
[FLINK-8102][docs] Fixed formatting issues in Mesos documentation.
commit 3d4146c10bf3d2f7df6d9992670017247d1952dd
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2017-11-20T15:09:10Z
[FLINK-8115] Fix Kafka download link in end-to-end test
commit 314087e8d31b7965afa5445ad4a8d9c467d0b940
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2017-11-20T15:32:06Z
[hotfix] Fix create_release_branch.sh to use correct branch name
commit ddbc0150d97d5d26de6f9ab9d64d9fb4495c80aa
Author: Bowen Li <bowenli86@...>
Date: 2017-11-04T04:08:24Z
[FLINK-7977][build] Bump version of compatibility check for Flink 1.4
This closes #4945.
commit 5f523e6ab31afeab5b1d9bbf62c6d4ef726ffe1b
Author: Gabor Gevay <ggab90@...>
Date: 2017-11-20T15:51:43Z
[hotfix][docs] Fix some typos in the documentation.
This closes #5039.
commit f26edb88d19623d93e563be417e50df969177c6a
Author: Gabor Gevay <ggab90@...>
Date: 2017-11-20T14:12:17Z
[FLINK-8117] [runtime] Eliminate modulo operation from round-robin
partitioners
This closes #5041
commit 9d28619467c33dec1fe44c6521001298ea1500e2
Author: Nico Kruber <nico@...>
Date: 2017-11-17T13:13:30Z
[FLINK-7841] [docs] Update AWS docs with respect to S3 file system changes
This closes #5029
commit 56c78cbcfd6fddd7af0fee0ee8c179b7a75fb1a7
Author: zentol <chesnay@...>
Date: 2017-11-14T10:55:33Z
[FLINK-8070][yarn][tests] Print errors found in log files
This closes #5012.
commit 7ed2cef83723fadb63f5cccf35ade1c9c78725e2
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2017-11-22T11:00:09Z
[FLINK-8131] Update to Kafka 0.11.0.2
commit a964ef6e218263c664d7b30607b8461c964e6fc8
Author: Cristian <me@...>
Date: 2017-11-22T12:17:39Z
[hotfix][docs] Fix typo in Trigger doc
This closes #5051.
commit 6cb8b5b1286e52f72667f6c4f5794742580c9145
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2017-11-22T15:22:28Z
[hotfix] Make aws docs version agnostic
commit 28157962196cecb94a59720e92cbf3682418e821
Author: twalthr <twalthr@...>
Date: 2017-11-22T16:43:08Z
[FLINK-8136] [table] Fix code generation with JodaTime shading
This closes #5054.
commit 7868ea4a6986ac46917f63a81c00957bee6fb1e6
Author: uybhatti <uybhatti@...>
Date: 2017-03-03T22:55:22Z
[FLINK-2170] [connectors] Add OrcRowInputFormat and OrcTableSource.
This closes #4670.
commit 35517f1291293f73f4466a4fdbed4296b2dd80a5
Author: Fabian Hueske <fhueske@...>
Date: 2017-11-13T13:54:54Z
[FLINK-2170] [connectors] Add OrcRowInputFormat and OrcTableSource.
This closes #5043.
commit 8a052bf0948d92d6fccc4d1c6c4bd2aa459032c9
Author: Bowen Li <bowenli86@...>
Date: 2017-10-10T05:31:17Z
[FLINK-6505] Proactively cleanup local FS for RocksDBKeyedStateBackend on
startup
commit 736b9088dcee64a1d3b19575f29a80c377f94fb8
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-22T10:37:48Z
[hotfix][kafka] Throw FlinkKafkaProducer011Exception with error codes
instead of generic Exception
commit 27564c33955d8e53f0275a5b43d4b2415ba86547
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-22T14:53:08Z
[FLINK-8132][kafka] Re-initialize transactional KafkaProducer on each
checkpoint
Previously faulty scenario with producer pool of 2.
1. started transaction 1 with producerA, written record 42
2. checkpoint 1 triggered, pre committing txn1, started txn2 with
producerB, written record 43
3. checkpoint 1 completed, committing txn1, returning producerA to the pool
4. checkpoint 2 triggered , committing txn2, started txn3 with producerA,
written record 44
5. crash....
6. recover to checkpoint 1, txn1 from producerA found to
"pendingCommitTransactions", attempting to recoverAndCommit(txn1)
7. unfortunately txn1 and txn3 from the same producers are identical from
KafkaBroker perspective and thus txn3 is being committed
result is that both records 42 and 44 are committed.
With this fix, after re-initialization txn3 will have different
producerId/epoch counters compared to txn1.
commit 62bf00189b136dd34c5af5d6181c33c1415f16b9
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-22T14:55:20Z
[hotfix][kafka] Remove unused method in kafka tests
commit e100861f84fe60ec6bb8172bb5a3cc453640fdb3
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-23T12:08:43Z
[hotfix][docs] Improve Kafka exactly-once docs
commit 828ef09b09f872107b412501774b42efaf6caa37
Author: Stefan Richter <s.richter@...>
Date: 2017-11-22T16:52:35Z
[FLINK-5465] [streaming] Wait for pending timer threads to finish or to
exceed a time limit in exceptional stream task shutdown.
This closes #5058.
(cherry picked from commit d86c6b6)
commit 2fb24581a1775084e3be8c2575c129d250f39313
Author: Xingcan Cui <xingcanc@...>
Date: 2017-11-22T16:00:39Z
[FLINK-8118] [table] Allow to specify the offsets of KafkaTableSources
This closes #5056.
commit 13631b9617d32e46eba51c9125019ec5e77c39f3
Author: twalthr <twalthr@...>
Date: 2017-11-23T13:30:15Z
[FLINK-8118] [table] Improve KafkaTableSource documentation
commit 3b58038d65cabdfd9da16150d4328227ca6afba1
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2017-11-20T17:06:11Z
[hotfix] Always explicitly set hadoop.version in create_binary_release
Before, the "hadoop2" profile would create a binary release for whatever
happens to be the default hadoop.version.
commit b5e156f79ae7e9cd2f8d5008f0c350e10ad4a821
Author: Stephan Ewen <sewen@...>
Date: 2017-11-08T19:14:34Z
[hotfix] [core] Fix lots of checkstyle errors in core.fs
commit a11e2cf0b1f37d3ef22e1978e89928fa374960db
Author: Stephan Ewen <sewen@...>
Date: 2017-11-08T22:57:04Z
[FLINK-8125] [core] Introduce limiting of outgoing file system connections
commit f636276dee991160a640442cbfcfdd58bfa57806
Author: Aljoscha Krettek <aljoscha.krettek@...>
Date: 2017-11-24T13:59:36Z
[FLINK-6294] Fix potential NPE in BucketingSink.close()
commit 4a46507e980b880a20bd821ee55d51bc787df124
Author: Piotr Nowojski <piotr.nowojski@...>
Date: 2017-11-23T11:53:37Z
[FLINK-8022][kafka] Bump at-least-once timeout in tests
Increasing timeout for reading the records from 30s to 60s seems to solve
the issue
for failing at-least-one tests.
commit bd1e03374d0f240f5a0b406ace662e4391cce1d5
Author: zentol <chesnay@...>
Date: 2017-11-15T12:14:18Z
[FLINK-8081][metrics] Annotate 'MetricRegistry#getReporters' with
'@VisibleForTesting'
This closes #5049.
commit 728d2ffdd5215223405dceb7469d47466706b043
Author: Matrix42 <934336389@...>
Date: 2017-11-27T09:34:01Z
[hotfix][docs] Fix missing period package declaration
This closes #5077.
----
---
