[jira] [Created] (FLINK-10710) Implement new CompositeSerializerSnapshot

Tzu-Li (Gordon) Tai created FLINK-10710: --- Summary: Implement new CompositeSerializerSnapshot Key: FLINK-10710 URL: https://issues.apache.org/jira/browse/FLINK-10710 Project: Flink

[jira] [Created] (FLINK-10709) Remove NS generic parameter from TypeSerializerSchemaCompatibility

Tzu-Li (Gordon) Tai created FLINK-10709: --- Summary: Remove NS generic parameter from TypeSerializerSchemaCompatibility Key: FLINK-10709 URL: https://issues.apache.org/jira/browse/FLINK-10709

[jira] [Created] (FLINK-10708) Implement a SerializerMigrationTestBase

Tzu-Li (Gordon) Tai created FLINK-10708: --- Summary: Implement a SerializerMigrationTestBase Key: FLINK-10708 URL: https://issues.apache.org/jira/browse/FLINK-10708 Project: Flink Issue

Re: [VOTE] Release 1.5.5, release candidate #1

+1 - Built locally, with Scala 2.11, Hadoop free. Build passes without errors (failed on HBaseConnectorITCase but passed after restarting the build from the Hbase connector). - Verified signature and hashes - Ran e2e tests locally overnight in a loop, no attempts were failing - No missing artifa

Re: [VOTE] Release 1.6.2, release candidate #1

+1 - Verified signature and hashes - Built locally, with Scala 2.11, Hadoop free. Build passes without errors. - Ran e2e tests locally overnight in a loop, no attempts were failing - No missing artifacts in the release staging area. I also agree that the OS-incompatibility in ES shouldn’t be bloc

[jira] [Created] (FLINK-10679) Let TypeSerializerSchemaCompatibility.resolveSchemaCompatibility() be the entry point for compatibility checks in framework code

Tzu-Li (Gordon) Tai created FLINK-10679: --- Summary: Let TypeSerializerSchemaCompatibility.resolveSchemaCompatibility() be the entry point for compatibility checks in framework code Key: FLINK-10679 URL

Re: Question over Incremental Snapshot vs Full Snapshot in rocksDb state backend

Hi, I’m forwarding this question to Stefan (cc’ed). He would most likely be able to answer your question, as he has done substantial work in the RocksDB state backends. Cheers, Gordon On 24 October 2018 at 8:47:24 PM, chandan prakash (chandanbaran...@gmail.com) wrote: Hi, I am new to Flink.

Re: [VOTE] Release flink-shaded 5.0, release candidate #1

+1 - Checked signature and hashes - Built locally with current master branch Couldn't find anything blocking either. Cheers, Gordon On Thu, Oct 11, 2018 at 4:33 PM Timo Walther wrote: > +1 > > - I built locally and checked the JAR files for suspicious things. > - I went throught the change di

[jira] [Created] (FLINK-10492) Document generic support for configuring AwsKinesisClient in the Kinesis Consumer

Tzu-Li (Gordon) Tai created FLINK-10492: --- Summary: Document generic support for configuring AwsKinesisClient in the Kinesis Consumer Key: FLINK-10492 URL: https://issues.apache.org/jira/browse/FLINK-10492

Re: [VOTE] Release 1.5.4, release candidate #1

+1 (binding) - Verified checksums / signatures - Checked announcement PR in flink-web - No new / changed dependencies - Built from source (Hadoop-free, Scala 2.11) - Run end-to-end tests locally, passes On Thu, Sep 20, 2018 at 5:03 AM Fabian Hueske wrote: > +1 binding > > * I checked the diffs

Re: [VOTE] Release 1.6.1, release candidate #1

+1 (binding) - Verified checksums / signatures - Checked announcement PR in flink-web - Built Flink from sources, test + build passes (Hadoop-free, Scala 2.11) - Ran Elasticsearch 6 sink, using quickstart POM. - Ran end-to-end tests locally, passes On Thu, Sep 20, 2018 at 4:31 PM Till Rohrmann w

Re: Could you please assign issue-10002 to me?

Hi, You are now assigned to the JIRA. You can now also assign tickets to yourself in the future. Cheers, Gordon On 11 September 2018 at 10:02:06 AM, Xinyu Zhang (wsz...@gmail.com) wrote: Hi I have finished some works for issue FLINK-10002( https://issues.apache.org/jira/browse/FLINK-10002)

Re: [DISCUSS] Change the Keyed partitioning behavior of the Kafka Producer API

Hi Niels, Your conclusions are accurate, and I also agree with the fact that the combination of the KeyedSerializationSchema / providing partitioners, etc. is all a bit awkward as of the current state. As for the proposed solutions, I personally disagree with 1), since key partitioning, IMO, shou

Re: [VOTE] Release 1.5.3, release candidate #1

+1 (binding) - verified checksum and gpg files - verified source compiles (tests enabled), Scala 2.11 / without Hadoop - e2e tests pass locally - source release contains no binaries - no missing release artifacts in staging area - reviewed announcement PR, is LGTM Cheers, Gordon On Sat, Aug 18,

[jira] [Created] (FLINK-10015) Use Elasticsearch's ESIntegTestCase to rework 5.x and 6.x's ITCases

Tzu-Li (Gordon) Tai created FLINK-10015: --- Summary: Use Elasticsearch's ESIntegTestCase to rework 5.x and 6.x's ITCases Key: FLINK-10015 URL: https://issues.apache.org/jira/browse/F

[jira] [Created] (FLINK-9885) End-to-end test: Elasticsearch 6.x connector

Tzu-Li (Gordon) Tai created FLINK-9885: -- Summary: End-to-end test: Elasticsearch 6.x connector Key: FLINK-9885 URL: https://issues.apache.org/jira/browse/FLINK-9885 Project: Flink Issue

[jira] [Created] (FLINK-9808) Implement state conversion procedure in state backends

Tzu-Li (Gordon) Tai created FLINK-9808: -- Summary: Implement state conversion procedure in state backends Key: FLINK-9808 URL: https://issues.apache.org/jira/browse/FLINK-9808 Project: Flink

Re: Consuming data from dynamoDB streams to flink

t; client when interacting with dynamoDB streams. The new object can be a > > subclass of FlinkKinesisConsumer with a new name e.g, > FlinkDynamoStreamCon > > sumer. > > > > At best this could simply work. But we would like to hear if there are > > other situations to

Re: Consuming data from dynamoDB streams to flink

Hi! I think it would be definitely nice to have this feature. No actual previous work has been made on this issue, but AFAIK, we should be able to build this on top of the FlinkKinesisConsumer. Whether this should live within the Kinesis connector module or an independent module of its own is sti

[jira] [Created] (FLINK-9608) Add test base for TypeSerializer snapshotting / restore / migration to be used by all Flink serializers

Tzu-Li (Gordon) Tai created FLINK-9608: -- Summary: Add test base for TypeSerializer snapshotting / restore / migration to be used by all Flink serializers Key: FLINK-9608 URL: https://issues.apache.org/jira

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.

://issues.apache.org/jira/browse/FLINK-9595. Cheers, Gordon On 11 June 2018 at 1:50:34 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Thanks for bring this up, Bowen. I’m not sure that it’ll be straightforward to just use the Kinesis connector in Flink 1.5 to work with 1.4. So, +1 to have a backport of

[jira] [Created] (FLINK-9595) Add instructions to docs about ceased support of KPL version used in Kinesis connector

Tzu-Li (Gordon) Tai created FLINK-9595: -- Summary: Add instructions to docs about ceased support of KPL version used in Kinesis connector Key: FLINK-9595 URL: https://issues.apache.org/jira/browse/FLINK-9595

[jira] [Created] (FLINK-9574) Add a dedicated documentation page for state evolution

Tzu-Li (Gordon) Tai created FLINK-9574: -- Summary: Add a dedicated documentation page for state evolution Key: FLINK-9574 URL: https://issues.apache.org/jira/browse/FLINK-9574 Project: Flink

Re: WELCOME to dev@flink.apache.org

Hi Sandish, Welcome to the Flink community! Do you mean contributor permissions on JIRA? The community usually only assigns contributor permissions when you find a specific JIRA ticket you would like to start working on. Once you do find one, let us know your JIRA account ID and the ticket, and

[jira] [Created] (FLINK-9569) Confusing construction of AvroSerializers for generic records

Tzu-Li (Gordon) Tai created FLINK-9569: -- Summary: Confusing construction of AvroSerializers for generic records Key: FLINK-9569 URL: https://issues.apache.org/jira/browse/FLINK-9569 Project

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.

rsion, but since it is > going to stop working anyways, there is probably no harm in this instance. > > Thomas > > > > > What do you think? > > > > Thanks, Bowen > > > > > > ------ Forwarded message -- > &g

Re: [VOTE] Release flink-shaded 4.0, release candidate #2

+1 - Checked signatures and hashes - Source builds successfully - Staged source release files are ASF compliant; does not contain md5 files, and have correct sha file name. Cheers, Gordon On Fri, Jun 1, 2018 at 12:45 AM, Yaz Sh wrote: > +1 > > I am new in this but I did following tests: > > -

Re: [VOTE] Release 1.5.0, release candidate #6

+1 - Verified signatures and hashes - Built from source with Hadoop 2.8.1 and Scala 2.11.7, tests pass locally - No binaries in source archives; no missing artifacts - To cover the extra Kafka connector bug fixes that were added in between RCs (FLINK-9349 and FLINK-9295), I executed Kafka end-to-

Re: Merging PR to release branches during RC voting

Hi Piotr, Thanks for bringing the issue up. I agree that this is something we should look into improving. Strictly speaking, I don’t think it is an issue only limited to after a RC vote has started. Taking 1.5.0 as an example, release-1.5 was branched out at the announcement for feature freeze

[jira] [Created] (FLINK-9394) Let externalized checkpoint resume e2e also test rescaling

Tzu-Li (Gordon) Tai created FLINK-9394: -- Summary: Let externalized checkpoint resume e2e also test rescaling Key: FLINK-9394 URL: https://issues.apache.org/jira/browse/FLINK-9394 Project: Flink

Re: Elasticsearch Sink

-- Christophe On Wed, May 16, 2018 at 7:19 AM, Tzu-Li (Gordon) Tai wrote: Hi, What if the user in a ES5.3+ case is calling the deprecated method? You  agree it will fail? I'm not necessarily against that. I just want to make  it clear that we don't have a perfect solution here eith

[jira] [Created] (FLINK-9377) Remove writing serializers as part of the checkpoint meta information

Tzu-Li (Gordon) Tai created FLINK-9377: -- Summary: Remove writing serializers as part of the checkpoint meta information Key: FLINK-9377 URL: https://issues.apache.org/jira/browse/FLINK-9377

[jira] [Created] (FLINK-9376) Allow upgrading to incompatible state serializers (state schema evolution)

Tzu-Li (Gordon) Tai created FLINK-9376: -- Summary: Allow upgrading to incompatible state serializers (state schema evolution) Key: FLINK-9376 URL: https://issues.apache.org/jira/browse/FLINK-9376

Re: [DISCUSS] Drop "canEqual" from TypeInformation, TypeSerializer, etc.

+1. Looking at the implementations of the `canEqual` method in several serializers, it seems like all that is done is a check whether the object is of the same serializer class. We’ll have to be careful and double check all `equals` method on serializers that may have relied on the `canEqual` m

Re: Elasticsearch Sink

/main/java/org/elasticsearch/action/bulk/BulkRequest.java#L128 On 16 May 2018 at 4:12:15 AM, Christophe Jolif (cjo...@gmail.com) wrote: Hi Gordon, On Tue, May 15, 2018 at 6:16 AM, Tzu-Li (Gordon) Tai wrote: > Hi, > > Let me first clarify a few things so that we are on the same page her

Re: Elasticsearch Sink

-- Christophe On Sun, May 13, 2018 at 5:39 AM, Tzu-Li (Gordon) Tai wrote: Hi Christophe, Thanks for bringing this up. Yes, the main issue with the existing PRs and preventing it from moving forward is how it currently breaks initial assumptions of APIs in the `elasticsearch-

Re: Elasticsearch Sink

Hi Christophe, Thanks for bringing this up. Yes, the main issue with the existing PRs and preventing it from moving forward is how it currently breaks initial assumptions of APIs in the `elasticsearch-base` module. Working around that would require introducing a new base module specifically for 5

Re: [VOTE] Release 1.5.0, release candidate #2

Hi Bowen, Thanks for bringing this up! Yes, I think we should definitely always test the Kinesis connector for releases. FYI, I think you can also add modification suggestions to the test plan so that the release manager is aware of that. Some of the more major Kinesis connector changes that I

[jira] [Created] (FLINK-9322) Add exception throwing map function that simulates failures to the general purpose DataStream job

Tzu-Li (Gordon) Tai created FLINK-9322: -- Summary: Add exception throwing map function that simulates failures to the general purpose DataStream job Key: FLINK-9322 URL: https://issues.apache.org/jira/browse

[jira] [Created] (FLINK-9320) Update `test-ha.sh` end-to-end test to use general purpose DataStream job

Tzu-Li (Gordon) Tai created FLINK-9320: -- Summary: Update `test-ha.sh` end-to-end test to use general purpose DataStream job Key: FLINK-9320 URL: https://issues.apache.org/jira/browse/FLINK-9320

Re: Request for contributor permission

Hi, I’ve gave you contributor permissions and assigned you the ticket. Happy contributing! Cheers, Gordon On 8 May 2018 at 11:56:59 AM, blues zheng (kisim...@163.com) wrote: Hi devs, We are working on FLINK-7789, could someone give me the contributor permission? My JIRA username is "kisimple".

[jira] [Created] (FLINK-9303) Unassign partitions from Kafka client if partitions become unavailable

Tzu-Li (Gordon) Tai created FLINK-9303: -- Summary: Unassign partitions from Kafka client if partitions become unavailable Key: FLINK-9303 URL: https://issues.apache.org/jira/browse/FLINK-9303

Re: Assign JIRA issue permission

Hi Sampath, Once you have a target JIRA issue that you would like to pick up and work on, let us know and we’ll have someone assign you Contributor permissions. Just let us know the JIRA ticket ID, and your JIRA account ID. After you have contributor permissions, you can pick up other unassigned

Re: [Discussion] Flink Pulsar Connector

Hi Sijie, Thanks for initiating the discussion. Given that it seems promising that we will have good eyes on this connector after it is merged, +1 to proceed with the PR and reviewing it as a connector contribution to Flink. Cheers, Gordon On 25 April 2018 at 4:07:48 AM, Sijie Guo (guosi...@gm

[jira] [Created] (FLINK-9204) Improve visibility of records being skipped by the `DeserializationSchema` in the Kafka / Kinesis connectors

Tzu-Li (Gordon) Tai created FLINK-9204: -- Summary: Improve visibility of records being skipped by the `DeserializationSchema` in the Kafka / Kinesis connectors Key: FLINK-9204 URL: https://issues.apache.org

[jira] [Created] (FLINK-9202) AvroSerializer should not be serializing the target Avro type class

Tzu-Li (Gordon) Tai created FLINK-9202: -- Summary: AvroSerializer should not be serializing the target Avro type class Key: FLINK-9202 URL: https://issues.apache.org/jira/browse/FLINK-9202

Re: Kinesis getRecords read timeout and retry

re fundamental issues with the > Kinesis connector (and other connectors) that I believe require deeper > design work and rewrite, which go beyond what we discuss here. > > Finally, I'm also curious how much appetite for contributions in the > connector areas there is? I se

Re: Kinesis getRecords read timeout and retry

Hi Thomas, Thanks for your PRs! I understand and fully agree with both points that you pointed out. What I'm still a bit torn with is the current proposed solutions for these issues (and other similar connector issues). This might actually call for a good opportunity to bring some thoughts up a

[jira] [Created] (FLINK-9074) End-to-end test: Resume from retained checkpoints

Tzu-Li (Gordon) Tai created FLINK-9074: -- Summary: End-to-end test: Resume from retained checkpoints Key: FLINK-9074 URL: https://issues.apache.org/jira/browse/FLINK-9074 Project: Flink

[jira] [Created] (FLINK-9073) Resume from savepoint end-to-end tests should be extended for different state backends

Tzu-Li (Gordon) Tai created FLINK-9073: -- Summary: Resume from savepoint end-to-end tests should be extended for different state backends Key: FLINK-9073 URL: https://issues.apache.org/jira/browse/FLINK-9073

[ANNOUNCE] Apache Flink 1.3.3 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.3.3, which is the third bugfix release for the Apache Flink 1.3 series.  Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: [VOTE] Release 1.3.3, release candidate #2

;> Am 15.03.18 um 10:10 schrieb Chesnay Schepler: >> >>> +1 >>> >>> * verified that compilation works >>> * ran a few test examples against a local cluster >>> * src artifact contains no binaries >>> * verified signatures

[VOTE] Release 1.3.3, release candidate #2

Hi everyone, Please review and vote on release candidate #2 for Flink 1.3.3, as follows:   [ ] +1, Approve the release   [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes:   * JIRA release notes [1],   * th

[VOTE] Release 1.3.3, release candidate #1

Hi everyone, Please review and vote on release candidate #1 for Flink 1.3.3, as follows:   [ ] +1, Approve the release   [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes:   * JIRA release notes [1],   * th

Re: [DISCUSS] Releasing Flink 1.3.3 with only critical fixes

+1 to have a minimal 1.3.3 release with the listed issues. I can also help manage this release. It seems like we don't have any objections with Aljoscha's proposal so far. Will wait a little bit more for any other opinions, and try to open a RC by the end of the week. Cheers, Gordon

[ANNOUNCE] Apache Flink 1.4.2 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.4.2, which is the second bugfix release for the Apache Flink 1.4 series.  Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streamin

Re: [VOTE] Release 1.4.2, release candidate #2

/jl/538... Thanks, Thomas On Wed, Feb 28, 2018 at 8:30 PM, Tzu-Li (Gordon) Tai wrote: > Hi everyone, > > Please review and vote on release candidate #2 for Flink 1.4.2, as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (pleas

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

Hi, +1, I think the new ListShards API should provide all the information we need. In the past, we were restricted to update the SDK version because it would break state compatibility (the consumer used to write AWS classes inside state). I think this was fixed since Flink 1.3, so it should be s

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

Btw, could you also open a JIRA to track this improvement? Thanks! On 5 March 2018 at 5:47:09 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Hi, +1, I think the new ListShards API should provide all the information we need. In the past, we were restricted to update the SDK version

[jira] [Created] (FLINK-8836) Duplicating a KryoSerializer does not duplicate registered default serializers

Tzu-Li (Gordon) Tai created FLINK-8836: -- Summary: Duplicating a KryoSerializer does not duplicate registered default serializers Key: FLINK-8836 URL: https://issues.apache.org/jira/browse/FLINK-8836

[VOTE] Release 1.4.2, release candidate #2

Hi everyone,  Please review and vote on release candidate #2 for Flink 1.4.2, as follows:   [ ] +1, Approve the release   [ ] -1, Do not approve the release (please provide specific comments)  The complete staging area is available for your review, which includes:   * JIRA release notes [1],   *

Re: [VOTE] Release 1.4.2, release candidate #2

://issues.apache.org/jira/browse/FLINK-8798 < > https://issues.apache.org/jira/browse/FLINK-8798>. Some users have > already run into this problem and popular Web Containers that use > child-first classloading also all have an exception for commons-logging. > What do you think? And sorry

Re: [VOTE] Release 1.4.2, release candidate #2

Sorry for the incorrect vote thread title. This vote is for release candidate #1, not 2. Cheers, Gordon On 27 February 2018 at 8:30:42 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Hi everyone, Please review and vote on release candidate #1 for Flink 1.4.2, as follows:  [ ] +1, Approve

[VOTE] Release 1.4.2, release candidate #2

Hi everyone, Please review and vote on release candidate #1 for Flink 1.4.2, as follows:  [ ] +1, Approve the release  [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes:  * JIRA release notes [1],  * the of

Re: [DISCUSS] Release Flink 1.4.2

sts did not catch this problem. >>> Maybe we could adapt them such that they guard against it in the future. >>> >>> Cheers, >>> Till >>> >>> On Thu, Feb 22, 2018 at 9:46 AM, Tzu-Li (Gordon) Tai < >>> tzuli...@apache.org

[DISCUSS] Release Flink 1.4.2

Hi all, Unfortunately, we've discovered a bug in 1.4.1, which suggests that we should almost immediately release another bugfix release: https://issues.apache.org/jira/browse/FLINK-8741. Since this issue was introduced only in 1.4.1, it might make sense to release 1.4.2 with only the fix for FLIN

[jira] [Created] (FLINK-8741) KafkaFetcher09/010/011 uses wrong user code classloader

Tzu-Li (Gordon) Tai created FLINK-8741: -- Summary: KafkaFetcher09/010/011 uses wrong user code classloader Key: FLINK-8741 URL: https://issues.apache.org/jira/browse/FLINK-8741 Project: Flink

[ANNOUNCE] Apache Flink 1.4.1 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.4.1, which is the first bugfix release for the Apache Flink 1.4 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming a

Re: [VOTE] Release 1.4.1, release candidate #1

d >>>> dependencies. >>>> I only found a version update of the Snappy dependency. Both, the >>> previous >>>> and the new version, are ASL so this is fine. >>>> >>>> The signatures and hashes are OK.

[VOTE] Release 1.4.1, release candidate #1

Hi everyone,  Please review and vote on release candidate #1 for Flink 1.4.1, as follows:  [ ] +1, Approve the release  [ ] -1, Do not approve the release (please provide specific comments)  The complete staging area is available for your review, which includes:  * JIRA release notes [1],  * the

Re: Timestamp/watermark support in Kinesis consumer

Regarding the two hooks you would like to be available: Provide hook to override discovery (not to hit Kinesis from every subtask) Yes, I think we can easily provide a way, for example setting -1 for SHARD_DISCOVERY_INTERVAL_MILLIS, to disable shard discovery. Though, the user would then have to

Re: Timestamp/watermark support in Kinesis consumer

Hi Thomas, It’s great that you’ve brought out these issues, which IMO are all very valid. They have also been in my head for a while. Here’s a list of things, out of the top of my head, that I would really like to improve as part of a major Kafka / Kinesis connector rework. Some have JIRAs for

Re: [DISCUSS] Releasing Flink 1.4.1

tarts (for fine grained recovery) from guage to counter On 15.01.2018 11:49, Tzu-Li (Gordon) Tai wrote: > Hi all, > > I would like to bring up the discussion of releasing Flink 1.4.1. > We’ve collected a few issues over the past few weeks, some of which is quite > critical. > &g

Re: [DISCUSS] Releasing Flink 1.5.0

Hi Aljoscha, Thanks for starting the discussion. I think there’s a few connector related must-have improvements that we should get in before the feature freeze, since quite a few users have been asking for them: [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start o

[jira] [Created] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer

Tzu-Li (Gordon) Tai created FLINK-8542: -- Summary: Do not indefinitely store closed shard's state in the FlinkKinesisConsumer Key: FLINK-8542 URL: https://issues.apache.org/jira/browse/FLINK

Re: Fwd: Re: Kinesis consumer shard skew - FLINK-8516

quick look at the PR and the comment I added. That will help to polish it up and make it ready for review. Thanks! Thomas -- sent from mobile -- Forwarded message -- From: "Thomas Weise" Date: Jan 30, 2018 5:53 PM Subject: Re: Kinesis consumer shard skew - FLINK-8516 T

Re: [DISCUSS] Releasing Flink 1.4.1

fine grained recovery) from guage to counter On 15.01.2018 11:49, Tzu-Li (Gordon) Tai wrote: > Hi all, > > I would like to bring up the discussion of releasing Flink 1.4.1. > We’ve collected a few issues over the past few weeks, some of which is quite > critical. &g

Re: Kinesis consumer shard skew - FLINK-8516

(Sorry, I accidentally sent out my un-finished reply too early. Here’s the full reply.) > Isn't the issue that some shard assignments may not have been checkpointed  > and so may end up in different subtasks when they are re-discovered?  That is part of the problem, yes. For example, it would al

Re: Kinesis consumer shard skew - FLINK-8516

> Isn't the issue that some shard assignments may not have been checkpointed  > and so may end up in different subtasks when they are re-discovered?  That is part of the problem, yes. For example, it would also be problematic for shard discovery. If the assignment is non-deterministic, a source s

Re: Kinesis consumer shard skew - FLINK-8516

Hi Thomas, Yes, you are right that sorting and then assigning shard-subtask mappings would not have deterministic assignment. Non-deterministic assignments would cause issues when restoring the consumer state. Regarding centralizing shard assignment: there actually has been ongoing discussion

[jira] [Created] (FLINK-8472) Extend migration tests for Flink 1.4

Tzu-Li (Gordon) Tai created FLINK-8472: -- Summary: Extend migration tests for Flink 1.4 Key: FLINK-8472 URL: https://issues.apache.org/jira/browse/FLINK-8472 Project: Flink Issue Type

Re: When is the scheduled release for 1.4.1 ? We have been waiting for a fix https://issues.apache.org/jira/browse/FLINK-8226 which looks a potential fix

Hi Vishal, Thanks for reporting this. There is no scheduled release for 1.4.1 yet, but I’ve just started a thread to track the remaining issues, so hopefully soon. Could you quickly reply on that thread [1] also so we can more easily keep track of this? Cheers, Gordon [1]  http://apache-flink-

[DISCUSS] Releasing Flink 1.4.1

Hi all, I would like to bring up the discussion of releasing Flink 1.4.1. We’ve collected a few issues over the past few weeks, some of which is quite critical. Here’s a list of issues, and their current status, that I’m aware of and IMO should be included: - [FLINK-8275] Flink YARN TMs using wro

[jira] [Created] (FLINK-8421) HeapInternalTimerService should reconfigure compatible key / namespace serializers on restore

Tzu-Li (Gordon) Tai created FLINK-8421: -- Summary: HeapInternalTimerService should reconfigure compatible key / namespace serializers on restore Key: FLINK-8421 URL: https://issues.apache.org/jira/browse

[jira] [Created] (FLINK-8419) Kafka consumer's offset metrics are not registered for dynamically discovered partitions

Tzu-Li (Gordon) Tai created FLINK-8419: -- Summary: Kafka consumer's offset metrics are not registered for dynamically discovered partitions Key: FLINK-8419 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (FLINK-8418) Kafka08ITCase.testStartFromLatest() times out on Travis

Tzu-Li (Gordon) Tai created FLINK-8418: -- Summary: Kafka08ITCase.testStartFromLatest() times out on Travis Key: FLINK-8418 URL: https://issues.apache.org/jira/browse/FLINK-8418 Project: Flink

[jira] [Created] (FLINK-8417) Support STSAssumeRoleSessionCredentialsProvider in FlinkKinesisConsumer

Tzu-Li (Gordon) Tai created FLINK-8417: -- Summary: Support STSAssumeRoleSessionCredentialsProvider in FlinkKinesisConsumer Key: FLINK-8417 URL: https://issues.apache.org/jira/browse/FLINK-8417

[jira] [Created] (FLINK-8416) Kinesis consumer doc examples should demonstrate preferred default credentials provider

Tzu-Li (Gordon) Tai created FLINK-8416: -- Summary: Kinesis consumer doc examples should demonstrate preferred default credentials provider Key: FLINK-8416 URL: https://issues.apache.org/jira/browse/FLINK-8416

[jira] [Created] (FLINK-8410) Kafka consumer's commitedOffsets gauge metric is prematurely set

Tzu-Li (Gordon) Tai created FLINK-8410: -- Summary: Kafka consumer's commitedOffsets gauge metric is prematurely set Key: FLINK-8410 URL: https://issues.apache.org/jira/browse/FLINK-8410 Pr

[jira] [Created] (FLINK-8409) Race condition in KafkaConsumerThread leads to potential NPE

Tzu-Li (Gordon) Tai created FLINK-8409: -- Summary: Race condition in KafkaConsumerThread leads to potential NPE Key: FLINK-8409 URL: https://issues.apache.org/jira/browse/FLINK-8409 Project

[jira] [Created] (FLINK-8398) Stabilize flaky KinesisDataFetcherTests

Tzu-Li (Gordon) Tai created FLINK-8398: -- Summary: Stabilize flaky KinesisDataFetcherTests Key: FLINK-8398 URL: https://issues.apache.org/jira/browse/FLINK-8398 Project: Flink Issue Type

[jira] [Created] (FLINK-8383) flink-mesos build failing: duplicate Jackson relocation in shaded jar

Tzu-Li (Gordon) Tai created FLINK-8383: -- Summary: flink-mesos build failing: duplicate Jackson relocation in shaded jar Key: FLINK-8383 URL: https://issues.apache.org/jira/browse/FLINK-8383

[jira] [Created] (FLINK-8306) FlinkKafkaConsumerBaseTest has invalid mocks on final methods

Tzu-Li (Gordon) Tai created FLINK-8306: -- Summary: FlinkKafkaConsumerBaseTest has invalid mocks on final methods Key: FLINK-8306 URL: https://issues.apache.org/jira/browse/FLINK-8306 Project

[jira] [Created] (FLINK-8296) Rework FlinkKafkaConsumerBestTest to not use Java reflection for dependency injection

Tzu-Li (Gordon) Tai created FLINK-8296: -- Summary: Rework FlinkKafkaConsumerBestTest to not use Java reflection for dependency injection Key: FLINK-8296 URL: https://issues.apache.org/jira/browse/FLINK-8296

[jira] [Created] (FLINK-8287) Flink Kafka Producer docs should clearly state what partitioner is used by default

Tzu-Li (Gordon) Tai created FLINK-8287: -- Summary: Flink Kafka Producer docs should clearly state what partitioner is used by default Key: FLINK-8287 URL: https://issues.apache.org/jira/browse/FLINK-8287

Re: how to assign jira task to myself ?

Hi, Welcome to the Flink community, contributions are always welcome! Could you let us know your JIRA ids, so someone can assign appropriate permissions for you? After that, you can freely assign open and unassigned tasks to yourself to work on. Best, Gordon On 18 December 2017 at 1:02:26 AM,

[jira] [Created] (FLINK-8270) TaskManagers do not use correct local path for shipped Keytab files in Yarn deployment modes

Tzu-Li (Gordon) Tai created FLINK-8270: -- Summary: TaskManagers do not use correct local path for shipped Keytab files in Yarn deployment modes Key: FLINK-8270 URL: https://issues.apache.org/jira/browse/FLINK

Re: [VOTE] Release 1.4.0, release candidate #3

+1 - Staged Apache source & binary convenience releases looks good - Built from source (macOS w/ Scala 2.11, hadoop-free, hadoop-2.8) - Locally tested topic regex subscription for the Kafka consumer - Quickstart projects looks good Other things verified that are carried from previous RC votes - C

[jira] [Created] (FLINK-8192) Properly annotate APIs of all Flink connectors with @Public / @PublicEvolving / @Internal

Tzu-Li (Gordon) Tai created FLINK-8192: -- Summary: Properly annotate APIs of all Flink connectors with @Public / @PublicEvolving / @Internal Key: FLINK-8192 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-8191) Add a RoundRobinPartitioner to be shipped with the Kafka connector

Tzu-Li (Gordon) Tai created FLINK-8191: -- Summary: Add a RoundRobinPartitioner to be shipped with the Kafka connector Key: FLINK-8191 URL: https://issues.apache.org/jira/browse/FLINK-8191 Project

<    1   2   3   4   5   6   7   8   9   >