Hi Thomas,
Good point about counting bytes. It would be difficult to throttle the byte
rate with the existing API. And it seems that for sinks we have to do that
rate limiting in the sink implementation anyways. There are a few ways to
do some abstraction, but maybe adding a RateLimiter is
TisonKun created FLINK-11507:
Summary: Remove invalid test JobClientActorTest
Key: FLINK-11507
URL: https://issues.apache.org/jira/browse/FLINK-11507
Project: Flink
Issue Type: Sub-task
TisonKun created FLINK-11509:
Summary: Remove invalid test
ClientConnectionTest#testJobManagerRetrievalWithHAServices
Key: FLINK-11509
URL: https://issues.apache.org/jira/browse/FLINK-11509
Project:
TisonKun created FLINK-11508:
Summary: Remove invalid test AkkaJobManagerRetrieverTest
Key: FLINK-11508
URL: https://issues.apache.org/jira/browse/FLINK-11508
Project: Flink
Issue Type: Sub-task
Chen Zun created FLINK-11506:
Summary: 通过sql-client.sh登录后，执行sql语句报IO异常，然后客户端中断退出
Key: FLINK-11506
URL: https://issues.apache.org/jira/browse/FLINK-11506
Project: Flink
Issue Type: Bug
TisonKun created FLINK-11505:
Summary: Remove invalid test JobManagerRegistrationTest
Key: FLINK-11505
URL: https://issues.apache.org/jira/browse/FLINK-11505
Project: Flink
Issue Type: Sub-task
TisonKun created FLINK-11504:
Summary: Remove invalid test JobManagerConnectionTest
Key: FLINK-11504
URL: https://issues.apache.org/jira/browse/FLINK-11504
Project: Flink
Issue Type: Sub-task
TisonKun created FLINK-11503:
Summary: Remove invalid test TaskManagerLossFailsTasksTest
Key: FLINK-11503
URL: https://issues.apache.org/jira/browse/FLINK-11503
Project: Flink
Issue Type:
TisonKun created FLINK-11502:
Summary: Remove invalid test FlinkActorTest
Key: FLINK-11502
URL: https://issues.apache.org/jira/browse/FLINK-11502
Project: Flink
Issue Type: Sub-task
I initially thought of an approach similar to the collector idea, by
overriding emitRecord in the fetcher. That makes counting the bytes
difficult, because it's downstream of decoding.
Another idea of solving this in a reusable way was to have a separate rate
limiting operator chained downstream
+1, and something I was planning to comment on in the Jira issue.
Also, if rate limiting could effectively stop the stream, then this could be
used solve a common data enrichment issue.
Logically you want to pause one stream (typically the time series data being
processed) while another stream
Thanks for the connector metric url, Chesnay :)
@Robert, as you can see, the metrics from different connectors are quite
different. And there are different names for similar metrics, which is a
little frustrating when users want to do monitoring / alerting.
Thanks,
Jiangjie (Becket) Qin
On
Thanks Robert.
Sorry I forgot to mention this, but yes, that is my JIRA id :)
And my Apache confluence ID is also becket_ qin (becket@gmail.com).
Could you also help me with the FLIP permissions?
Thanks,
Jiangjie (Becket) Qin
On Thu, Jan 31, 2019 at 5:56 PM Robert Metzger wrote:
> Hey
Hi Jamie,
Thanks for the explanation. That makes sense to me. I am wondering if there
is a more general way to add a rate limiter to all the connecters rather
than doing that for each individual one. For example, maybe we can have the
rate limiting logic in the Collector / Output, thus all the
Lakshmi Rao created FLINK-11501:
---
Summary: Add a ratelimiting feature to the FlinkKafkaConsumer
Key: FLINK-11501
URL: https://issues.apache.org/jira/browse/FLINK-11501
Project: Flink
Issue
Thanks for adding more context @Jamie Grier .
JIRA for this feature: https://issues.apache.org/jira/browse/FLINK-11501.
Thanks
Lakshmi
On Thu, Jan 31, 2019 at 3:20 PM Thomas Weise wrote:
> I think it would be reasonable to have a rate limiter option in the
> consumer, given that others have
I think it would be reasonable to have a rate limiter option in the
consumer, given that others have also looked to solve this.
I think for this and other optional features, it would be good to implement
in a way that overrides are possible. Someone else may want to do the
limiting differently,
I had the same reaction initially as some of the others on this thread --
which is "Use Kafka quotas".. I agree that in general a service should
protect itself with it's own rate limiting rather than building it into
clients like the FlinkKafkaConsumer.
However, there are a few reasons we need
Chesnay Schepler created FLINK-11500:
Summary: 1.8
Key: FLINK-11500
URL: https://issues.apache.org/jira/browse/FLINK-11500
Project: Flink
Issue Type: Bug
Reporter: Chesnay
Seth Wiesman created FLINK-11499:
Summary: Extend StreamingFileSink BulkFormats to support arbitrary
roll policy's
Key: FLINK-11499
URL: https://issues.apache.org/jira/browse/FLINK-11499
Project:
sambi reddy created FLINK-11498:
---
Summary: Getting java.lang.InstantiationError when trying to
install flink 1.7.1 version through ansible
Key: FLINK-11498
URL: https://issues.apache.org/jira/browse/FLINK-11498
TisonKun created FLINK-11497:
Summary: Remove invalid test JobManagerLeaderElectionTest
Key: FLINK-11497
URL: https://issues.apache.org/jira/browse/FLINK-11497
Project: Flink
Issue Type:
Sorry, I haven't been active in the business of merging pull requests
recently. But you are both totally right: There is now a shiny big button
for merging pull requests.
Regarding permissions for the bot: I'm going to ask Infra if the bot can
get permissions to label and merge pull requests.
I
Elango Ganesan created FLINK-11496:
--
Summary: FlinkS3 FileSysten is not handling multiple local temp
directories
Key: FLINK-11496
URL: https://issues.apache.org/jira/browse/FLINK-11496
Project:
Chesnay Schepler created FLINK-11495:
Summary: Remove legacy job archiving paths
Key: FLINK-11495
URL: https://issues.apache.org/jira/browse/FLINK-11495
Project: Flink
Issue Type:
Chesnay Schepler created FLINK-11494:
Summary: Remove legacy WebRuntimeMonitor
Key: FLINK-11494
URL: https://issues.apache.org/jira/browse/FLINK-11494
Project: Flink
Issue Type: Sub-task
Timo Walther created FLINK-11493:
Summary: Finalize the Blink SQL merging efforts
Key: FLINK-11493
URL: https://issues.apache.org/jira/browse/FLINK-11493
Project: Flink
Issue Type: Sub-task
Timo Walther created FLINK-11492:
Summary: Support the full Blink SQL runtime
Key: FLINK-11492
URL: https://issues.apache.org/jira/browse/FLINK-11492
Project: Flink
Issue Type: Sub-task
Timo Walther created FLINK-11491:
Summary: Support all TPC-DS queries
Key: FLINK-11491
URL: https://issues.apache.org/jira/browse/FLINK-11491
Project: Flink
Issue Type: Sub-task
Timo Walther created FLINK-11490:
Summary: Add an initial batch runtime
Key: FLINK-11490
URL: https://issues.apache.org/jira/browse/FLINK-11490
Project: Flink
Issue Type: Sub-task
Timo Walther created FLINK-11489:
Summary: Add an initial Blink streaming runtime
Key: FLINK-11489
URL: https://issues.apache.org/jira/browse/FLINK-11489
Project: Flink
Issue Type: Sub-task
Timo Walther created FLINK-11488:
Summary: Merge a basic Blink planner framework
Key: FLINK-11488
URL: https://issues.apache.org/jira/browse/FLINK-11488
Project: Flink
Issue Type: Sub-task
@Robert:
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors
On 31.01.2019 11:03, Robert Metzger wrote:
Hey Becket,
thanks a lot for your proposal!
Do you have an overview over the current situation of the metrics in the
connectors?
Which connectors
ambition created FLINK-11487:
Summary: Support for writing data to Apache Flume
Key: FLINK-11487
URL: https://issues.apache.org/jira/browse/FLINK-11487
Project: Flink
Issue Type: New Feature
Hey Becket,
thanks a lot for your proposal!
Do you have an overview over the current situation of the metrics in the
connectors?
Which connectors expose metrics at all?
Are they different?
On Thu, Jan 31, 2019 at 8:44 AM Becket Qin wrote:
> Hi folks,
>
> I was trying to add some metrics to
Hey Becket,
I gave you contributor permissions in our JIRA. (Assuming your JIRA id is
"becket_qin").
On Thu, Jan 31, 2019 at 10:37 AM Becket Qin wrote:
> Hi,
>
> Could someone help me with getting the contributor's permission so I can
> create FLIPs and assign JIRA tickets?
>
> Thanks,
>
>
Shimin Yang created FLINK-11486:
---
Summary: Remove RecoveryITCase for jobmanager
Key: FLINK-11486
URL: https://issues.apache.org/jira/browse/FLINK-11486
Project: Flink
Issue Type: Sub-task
Hi,
Could someone help me with getting the contributor's permission so I can
create FLIPs and assign JIRA tickets?
Thanks,
Jiangjie (Becket) Qin
38 matches
Mail list logo
