[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-31872: -- Affects Version/s: (was: aws-connector-4.2.0) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.4.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for {{KinesisStreamsSink}} after configuring > rate limiting strategy. > It appears that optimum values for rate limiting strategy parameters are > dependent on use case (shard number/ parallellism/ record thouroughput) > We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with > one used for TCP congestion control but since parameters are use case > dependent we would like to allow sink users to adjust parameters as suitable. > h2. Requirements > - we *must* allow users to configure increment rate and decrease factor of > AIMDRateLimitingStrategy for {{KinesisStreamsSink}} > - we *must* provide backward compatible default values identical to current > values to introduce no further regressions. > h2. Appendix > h3. Performace Benchmark Results > |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async > Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async > sink/ Maximum Thourouput|% of Improvement| > |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9| > |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57| > |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625| > |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25| > |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6| > |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4| > |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625| > |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2| > |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625| > |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562| > |High/High/Low|256|256|1024|30|127|127|0.49609|0| > |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-31872: -- Fix Version/s: aws-connector-4.4.0 (was: aws-connector-4.3.0) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Affects Versions: aws-connector-4.2.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.4.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for {{KinesisStreamsSink}} after configuring > rate limiting strategy. > It appears that optimum values for rate limiting strategy parameters are > dependent on use case (shard number/ parallellism/ record thouroughput) > We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with > one used for TCP congestion control but since parameters are use case > dependent we would like to allow sink users to adjust parameters as suitable. > h2. Requirements > - we *must* allow users to configure increment rate and decrease factor of > AIMDRateLimitingStrategy for {{KinesisStreamsSink}} > - we *must* provide backward compatible default values identical to current > values to introduce no further regressions. > h2. Appendix > h3. Performace Benchmark Results > |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async > Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async > sink/ Maximum Thourouput|% of Improvement| > |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9| > |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57| > |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625| > |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25| > |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6| > |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4| > |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625| > |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2| > |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625| > |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562| > |High/High/Low|256|256|1024|30|127|127|0.49609|0| > |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-31872: -- Fix Version/s: aws-connector-4.3.0 (was: aws-connector-4.2.0) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.3.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for {{KinesisStreamsSink}} after configuring > rate limiting strategy. > It appears that optimum values for rate limiting strategy parameters are > dependent on use case (shard number/ parallellism/ record thouroughput) > We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with > one used for TCP congestion control but since parameters are use case > dependent we would like to allow sink users to adjust parameters as suitable. > h2. Requirements > - we *must* allow users to configure increment rate and decrease factor of > AIMDRateLimitingStrategy for {{KinesisStreamsSink}} > - we *must* provide backward compatible default values identical to current > values to introduce no further regressions. > h2. Appendix > h3. Performace Benchmark Results > |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async > Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async > sink/ Maximum Thourouput|% of Improvement| > |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9| > |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57| > |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625| > |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25| > |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6| > |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4| > |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625| > |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2| > |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625| > |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562| > |High/High/Low|256|256|1024|30|127|127|0.49609|0| > |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-31872: -- Affects Version/s: aws-connector-4.2.0 > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Affects Versions: aws-connector-4.2.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.3.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for {{KinesisStreamsSink}} after configuring > rate limiting strategy. > It appears that optimum values for rate limiting strategy parameters are > dependent on use case (shard number/ parallellism/ record thouroughput) > We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with > one used for TCP congestion control but since parameters are use case > dependent we would like to allow sink users to adjust parameters as suitable. > h2. Requirements > - we *must* allow users to configure increment rate and decrease factor of > AIMDRateLimitingStrategy for {{KinesisStreamsSink}} > - we *must* provide backward compatible default values identical to current > values to introduce no further regressions. > h2. Appendix > h3. Performace Benchmark Results > |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async > Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async > sink/ Maximum Thourouput|% of Improvement| > |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9| > |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57| > |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625| > |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25| > |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6| > |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4| > |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625| > |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2| > |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625| > |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562| > |High/High/Low|256|256|1024|30|127|127|0.49609|0| > |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Description: h1. Issue As part of FLINK-31772 I performed a complete benchmark for {{KinesisStreamsSink}} after configuring rate limiting strategy. It appears that optimum values for rate limiting strategy parameters are dependent on use case (shard number/ parallellism/ record thouroughput) We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one used for TCP congestion control but since parameters are use case dependent we would like to allow sink users to adjust parameters as suitable. h2. Requirements - we *must* allow users to configure increment rate and decrease factor of AIMDRateLimitingStrategy for {{KinesisStreamsSink}} - we *must* provide backward compatible default values identical to current values to introduce no further regressions. h2. Appendix h3. Performace Benchmark Results |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async sink/ Maximum Thourouput|% of Improvement| |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9| |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57| |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625| |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25| |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6| |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4| |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625| |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2| |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625| |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562| |High/High/Low|256|256|1024|30|127|127|0.49609|0| |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313| was: h1. Issue As part of FLINK-31772 I performed a complete benchmark for {{KinesisStreamsSink}} after configuring rate limiting strategy. It appears that optimum values for rate limiting strategy parameters are dependent on use case (shard number/ parallellism/ record thouroughput) We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one used for TCP congestion control but since parameters are use case dependent we would like to allow sink users to adjust parameters as suitable. h2. Requirements - we *must* allow users to configure increment rate and decrease factor of AIMDRateLimitingStrategy for {{KinesisStreamsSink}} - we *must* provide backward compatible default values identical to current values to introduce no further regressions. h2. Appendix h3. Performace Benchmark Results |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL Thourouput (MB/s)|Async sink/ Maximum Thourouput|Async sink /KPL|% of Improvement| |Low/Low/Low|1|1|1024|1|0.991|1|0.958|1|4.2|0.9| |Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57| |Low/Med/Low|1|8|1024|8|4.12|4.57|6.45|0.57125|-23.5|5.625| |Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25| |Med/Low/Low|8|1|1024|2|0.852|0.846|0.545|0.846|30.1|-0.6| |Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4| |Med/Med/Low|8|8|1024|8|5.37|4.76|5.87|0.595|-13.875|-7.625| |Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2| |Med/High/Low|8|64|1024|8|32.5|37.4|40.7|0.58438|-5.15625|7.65625| |Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562| |High/High/Low|256|256|1024|30|127|127|131|0.49609|-1.5625|0| |High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313| > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.2.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for {{KinesisStreamsSink}} after configuring > rate limiting strategy. > It appears that optimum values for rate limiting strategy parameters are > dependent on use case (shard number/ parallellism/ record thouroughput) > We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with > one used for TCP congestion control but since parameters are use case > dependent we would like to allow sink users to adjust parameters as suitable. > h2. Requirements > - we *must* allow users to configure increment rate and decrease factor of > AIMDRateLimitingStrategy for {{KinesisStreamsSink}} > - we *must* provide bac
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Description: h1. Issue As part of FLINK-31772 I performed a complete benchmark for {{KinesisStreamsSink}} after configuring rate limiting strategy. It appears that optimum values for rate limiting strategy parameters are dependent on use case (shard number/ parallellism/ record thouroughput) We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one used for TCP congestion control but since parameters are use case dependent we would like to allow sink users to adjust parameters as suitable. h2. Requirements - we *must* allow users to configure increment rate and decrease factor of AIMDRateLimitingStrategy for {{KinesisStreamsSink}} - we *must* provide backward compatible default values identical to current values to introduce no further regressions. h2. Appendix h3. Performace Benchmark Results |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL Thourouput (MB/s)|Async sink/ Maximum Thourouput|Async sink /KPL|% of Improvement| |Low/Low/Low|1|1|1024|1|0.991|1|0.958|1|4.2|0.9| |Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57| |Low/Med/Low|1|8|1024|8|4.12|4.57|6.45|0.57125|-23.5|5.625| |Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25| |Med/Low/Low|8|1|1024|2|0.852|0.846|0.545|0.846|30.1|-0.6| |Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4| |Med/Med/Low|8|8|1024|8|5.37|4.76|5.87|0.595|-13.875|-7.625| |Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2| |Med/High/Low|8|64|1024|8|32.5|37.4|40.7|0.58438|-5.15625|7.65625| |Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562| |High/High/Low|256|256|1024|30|127|127|131|0.49609|-1.5625|0| |High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313| was: h1. Issue As part of FLINK-31772 I performed a complete benchmark for > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.2.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for {{KinesisStreamsSink}} after configuring > rate limiting strategy. > It appears that optimum values for rate limiting strategy parameters are > dependent on use case (shard number/ parallellism/ record thouroughput) > We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with > one used for TCP congestion control but since parameters are use case > dependent we would like to allow sink users to adjust parameters as suitable. > h2. Requirements > - we *must* allow users to configure increment rate and decrease factor of > AIMDRateLimitingStrategy for {{KinesisStreamsSink}} > - we *must* provide backward compatible default values identical to current > values to introduce no further regressions. > h2. Appendix > h3. Performace Benchmark Results > |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async > Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL > Thourouput (MB/s)|Async sink/ Maximum Thourouput|Async sink /KPL|% of > Improvement| > |Low/Low/Low|1|1|1024|1|0.991|1|0.958|1|4.2|0.9| > |Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57| > |Low/Med/Low|1|8|1024|8|4.12|4.57|6.45|0.57125|-23.5|5.625| > |Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25| > |Med/Low/Low|8|1|1024|2|0.852|0.846|0.545|0.846|30.1|-0.6| > |Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4| > |Med/Med/Low|8|8|1024|8|5.37|4.76|5.87|0.595|-13.875|-7.625| > |Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2| > |Med/High/Low|8|64|1024|8|32.5|37.4|40.7|0.58438|-5.15625|7.65625| > |Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562| > |High/High/Low|256|256|1024|30|127|127|131|0.49609|-1.5625|0| > |High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Description: h1. Issue As part of FLINK-31772 I performed a complete benchmark for was: h1. Issue While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy {{FlinkKinesisProduced}} , it is observed that the new sink has a performance regression against the deprecated sink for same environment setting. Further investigation identified that the AIMD Ratelimiting strategy is the bottleneck for the regression. Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} h2. Environment Settings - Benchmarking was performed on AWS KDA. - Application logic is just sending records downstream - Application parallelism was tested to be 1. - Kinesis stream number of shards was tested with 8 and 12. - payload size was 1Kb and 100Kb. > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.2.0 > > > h1. Issue > As part of FLINK-31772 > I performed a complete benchmark for -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Affects Version/s: (was: 1.16.0) (was: aws-connector-3.0.0) (was: 1.16.1) (was: aws-connector-4.0.0) (was: aws-connector-4.1.0) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common, Connectors / Firehose, Connectors / > Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.2.0 > > > h1. Issue > While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy > {{FlinkKinesisProduced}} , it is observed that the new sink has a performance > regression against the deprecated sink for same environment setting. > Further investigation identified that the AIMD Ratelimiting strategy is the > bottleneck for the regression. > Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} > and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} > h2. Environment Settings > - Benchmarking was performed on AWS KDA. > - Application logic is just sending records downstream > - Application parallelism was tested to be 1. > - Kinesis stream number of shards was tested with 8 and 12. > - payload size was 1Kb and 100Kb. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Labels: (was: pull-request-available) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Fix For: aws-connector-4.2.0 > > > h1. Issue > While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy > {{FlinkKinesisProduced}} , it is observed that the new sink has a performance > regression against the deprecated sink for same environment setting. > Further investigation identified that the AIMD Ratelimiting strategy is the > bottleneck for the regression. > Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} > and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} > h2. Environment Settings > - Benchmarking was performed on AWS KDA. > - Application logic is just sending records downstream > - Application parallelism was tested to be 1. > - Kinesis stream number of shards was tested with 8 and 12. > - payload size was 1Kb and 100Kb. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Component/s: (was: Connectors / Common) (was: Connectors / Firehose) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.2.0 > > > h1. Issue > While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy > {{FlinkKinesisProduced}} , it is observed that the new sink has a performance > regression against the deprecated sink for same environment setting. > Further investigation identified that the AIMD Ratelimiting strategy is the > bottleneck for the regression. > Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} > and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} > h2. Environment Settings > - Benchmarking was performed on AWS KDA. > - Application logic is just sending records downstream > - Application parallelism was tested to be 1. > - Kinesis stream number of shards was tested with 8 and 12. > - payload size was 1Kb and 100Kb. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Fix Version/s: (was: 1.16.2) (was: 1.18.0) (was: 1.17.1) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common, Connectors / Firehose, Connectors / > Kinesis >Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1, > aws-connector-4.0.0, aws-connector-4.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.2.0 > > > h1. Issue > While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy > {{FlinkKinesisProduced}} , it is observed that the new sink has a performance > regression against the deprecated sink for same environment setting. > Further investigation identified that the AIMD Ratelimiting strategy is the > bottleneck for the regression. > Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} > and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} > h2. Environment Settings > - Benchmarking was performed on AWS KDA. > - Application logic is just sending records downstream > - Application parallelism was tested to be 1. > - Kinesis stream number of shards was tested with 8 and 12. > - payload size was 1Kb and 100Kb. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Issue Type: Improvement (was: Bug) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common, Connectors / Firehose, Connectors / > Kinesis >Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1, > aws-connector-4.0.0, aws-connector-4.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available > Fix For: 1.16.2, 1.18.0, 1.17.1, aws-connector-4.2.0 > > > h1. Issue > While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy > {{FlinkKinesisProduced}} , it is observed that the new sink has a performance > regression against the deprecated sink for same environment setting. > Further investigation identified that the AIMD Ratelimiting strategy is the > bottleneck for the regression. > Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} > and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} > h2. Environment Settings > - Benchmarking was performed on AWS KDA. > - Application logic is just sending records downstream > - Application parallelism was tested to be 1. > - Kinesis stream number of shards was tested with 8 and 12. > - payload size was 1Kb and 100Kb. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink
[ https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hamdy updated FLINK-31872: Summary: Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink (was: CLONE - AsyncSinkWriter Performance regression due to AIMD rate limiting strategy) > Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink > users for KinesisStreamsSink > -- > > Key: FLINK-31872 > URL: https://issues.apache.org/jira/browse/FLINK-31872 > Project: Flink > Issue Type: Bug > Components: Connectors / Common, Connectors / Firehose, Connectors / > Kinesis >Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1, > aws-connector-4.0.0, aws-connector-4.1.0 >Reporter: Ahmed Hamdy >Assignee: Ahmed Hamdy >Priority: Major > Labels: pull-request-available > Fix For: 1.16.2, 1.18.0, 1.17.1, aws-connector-4.2.0 > > > h1. Issue > While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy > {{FlinkKinesisProduced}} , it is observed that the new sink has a performance > regression against the deprecated sink for same environment setting. > Further investigation identified that the AIMD Ratelimiting strategy is the > bottleneck for the regression. > Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer} > and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}} > h2. Environment Settings > - Benchmarking was performed on AWS KDA. > - Application logic is just sending records downstream > - Application parallelism was tested to be 1. > - Kinesis stream number of shards was tested with 8 and 12. > - payload size was 1Kb and 100Kb. > -- This message was sent by Atlassian Jira (v8.20.10#820010)