Recommendation about RocksDB Metrics ?

2020-12-05 Thread Truong Duc Kien
Hi all, We are thinking about enabling RocksDB metrics to better monitor our pipeline. However, since they will have performance impact, we will have to be selective about which metrics we use. Does anyone have experience about which metrics are more important than the others ? And what metrics

Re: Flink on YARN: delegation token expired prevent job restart

2020-11-18 Thread Truong Duc Kien
Hi all, So I've checked the log and it seems that the expired delegation error was triggered during resource localization. Maybe there's something wrong with my Hadoop setup, NMs are supposed to get a good token from RM in order to localize resources automatically. Regards, Kiên 2020-11-17

Idle stream does not advance watermark in connected stream

2020-08-24 Thread Truong Duc Kien
Hi all, We are testing the new Idleness detection feature in Flink 1.11, however, it does not work as we expected: When we connect two data streams, of which one is idle, the output watermark CoProcessOperator does not increase, hence the program cannot progress. I've made a small project to

What is the advantages of ReducingState over ValueState and vice versa ?

2016-12-29 Thread Truong Duc Kien
Hi all, Since both ReducingState and ValueState store a single value, there are many use cases where both can be used. When should we use one instead of the other ? Best regards, Kien

Default parallelism for job submitted using RemoteEnvironment

2015-12-03 Thread Truong Duc Kien
Hi, When I submit a job using RemoteEnvironment without setting parallelism, it always uses only one task slot. Is this a bug or intentional ? I thought it was supposed to be the default configuration of the server (parallelism.default=24 in my cases) I'm using Flink in Standalone cluster mode.

Get an aggregator's value outside of an iteration

2015-11-27 Thread Truong Duc Kien
Hi, I'm looking for a way get the value of aggregators outside of iteration. Specifically, I want the final aggregators' value after the iteration has finished. Is there any API for that ? Thanks, Kien Truong

Re: Compiler Exception

2015-11-19 Thread Truong Duc Kien
with us? Cheers, Till On Nov 18, 2015 9:54 PM, "Truong Duc Kien" <duckientru...@gmail.com <mailto:duckientru...@gmail.com>> wrote: Hi, I'm hitting Compiler Exception with some of my data set, but not all of them. Exception in thread "main"

Compiler Exception

2015-11-18 Thread Truong Duc Kien
Hi, I'm hitting Compiler Exception with some of my data set, but not all of them. Exception in thread "main" org.apache.flink.optimizer.CompilerException: No plan meeting the requirements could be created @ Bulk Iteration (Bulk Iteration) (1:null). Most likely reason: Too restrictive plan hints.

Different CoGroup behavior inside DeltaIteration

2015-11-15 Thread Truong Duc Kien
Hi, When running CoGroup between the solution set and a different dataset inside a DeltaIteration, the CoGroupFunction only get called for items that exist in the other dataset, simillar to an inner join. This is not the documented behavior for CoGroup: If a DataSet has a group with no