Re: Periodic flush sink?

bq. is the mutator thread safe? See HBASE-17361 On Wed, May 3, 2017 at 1:52 AM, Aljoscha Krettek wrote: > Hi Niels, > With any kind of buffering you need to be careful when it comes to fault > tolerance. In your case, you should make sure to flush the buffers when >

Re: Periodic flush sink?

Hi Niels, With any kind of buffering you need to be careful when it comes to fault tolerance. In your case, you should make sure to flush the buffers when checkpointing, otherwise you might lose data because those elements will not be resend after a failure. With the periodic timer my only

Re: Periodic flush sink?

Hi Niels, This sounds to me like a great use case for using window functions. You could partition your data (use keyby) based on website and then hold your window for certain amount of time. After that you could give your sink already batched object and store it directly. On top of that if you

Re: Periodic flush sink?

Thanks. The specific table I have here is used to debugging purposes so at the HBase level I set a TTL of the data of 12 hours. So I'm not worrying about the Hfiles. Doing a lot of 'small' calls has an impact on HBase as a whole (not just this table) so I want buffering. Having a buffer that can

Re: Periodic flush sink?

I expect Flink expert to answer your question. bq. I get a flush of the buffers atleast every few seconds >From hbase point of view, during low traffic period, the above may result in many small hfiles, leading to more work for the compaction. FYI On Sat, Apr 29, 2017 at 7:32 AM, Niels Basjes

Periodic flush sink?

Hi, I have a sink that writes my records into HBase. The data stream is attached to measurements from an internal testing instance of the website. As a consequence there are periods of really high load (someone is doing a load test) and really low load (only a hand full of people are testing