Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

abel = label; } public long getValue() { return value; } public void setValue(long value) { this.value = value; } } } > > > > On Fri, Oct 20, 2023 at 7:26 AM Ken Krugler <mailto:kkrugler_li...@transpac.c

Re: [DISCUSS] FLIP-413: Enable unaligned checkpoints by default

;>> also simultaneously increase the aligned checkpoints timeout from 0ms to >>> 5s. I think this change is the right one to do for the majority of Flink >>> users. >>> >>> For more rationale please take a look into the short FLIP-413 [1]. >>> &

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

on, and we can avoid over-design in > the current FLIP. > > Thanks for your feedback! > > Best, > Fang Yong > > On Fri, Dec 29, 2023 at 12:38 PM Ken Krugler <mailto:kkrugler_li...@transpac.com>> wrote: >> Hi Xintong, >> >> I agree that decoupli

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

er > discussion. At the moment, I'm not aware of any community consensus on > doing so. And even if in the future we decide to do so, the changes needed > should be the same w/ or w/o this FLIP. So I'd suggest not to block this > FLIP on these issues. > > WDYT? > &

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

g/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink > > Best, > Fang Yong -- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot

Using Fury serializer with Flink

ryo. And it supposedly handles schema evolution, though I haven’t tried that, assuming you configure the builder properly. Currently they’re not guaranteeing binary compatibility across 0.x releases until 1.0 is out (currently at 0.4.1), so might make sense to wait for that. — Ken ------

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

egards, > > Dong and Xuannan > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073749 -- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink & Pinot

Re: [DISCUSS] FLIP-317: Upgrade Kryo from 2.24.0 to 5.5.0

riously considered for adoption. > > I'm looking forward to hearing everyone's feedback and suggestions. > > Thank you, > Kurt > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-317%3A+Upgrade+Kryo+from+2.24.0+to+5.5.0 -- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink, Pinot, Solr, Elasticsearch

Re: Failed to commit consumer offsets for checkpoint 108768

mmitting the > latest consumed offsets. > Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: > The coordinator is not available. > > Regards, > Sucheth Shivakumar > website : https://sucheths.com > mobile : +1(650)-576-8050 > San Mateo, United States -

Re: [Consulting] Should a Flink class loader be a singleton?

le jar to flink lib folder, instead, i >> used sql-client.sh embedded -j /huid-flink-bundle-xxx.jar. >> >> So I'm wondering if the root reason is that they are loaded by two >> classloader instances? And it seems that FlinkUserCodeClassLoaders is a >> singleton, but ParentFirstClassLoader and ChildFirstClassLoader aren't from >> flink source code. >> -- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink, Pinot, Solr, Elasticsearch

Re: Stack Overflow Question re compressing broadcast state

wse/FLINK-30112> > [2] https://issues.apache.org/jira/browse/FLINK-30113 > <https://issues.apache.org/jira/browse/FLINK-30113> > > On 18/11/2022 02:13, Ken Krugler wrote: >> Hi Dawid, >> >> Thanks for getting back to me. >> >> And yes,

Re: Stack Overflow Question re compressing broadcast state

n it, as KeyedState should be preferred in > majority of cases. > > Best, > > Dawid > > On 16/11/2022 23:27, Ken Krugler wrote: >> Hi all, >> >> Just posted this question on SO: How to enable compression for Flink >> broadcast state checkpoints &g

Stack Overflow Question re compressing broadcast state

ng the code wrong. Hoping someone (like Dawid Wysakowicz) can chime in here, thanks! — Ken ------ Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink, Pinot, Solr, Elasticsearch

Fwd: Stateful Functions with Flink 1.15 and onwards

a shared Flink >> cluster which will soon be upgrading to Flink 1.15. If I need to re-write >> all our code as a native Flink job (rather than a remote stateful function) >> then I need to get started right away. >> >> Many thanks, >> Fil >> -- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink, Pinot, Solr, Elasticsearch

Re: More issues with top-level build for Tika 1.25 rc1 - Waited more than 5 minutes for a SAXParser

23, 2020, at 4:34 AM, Arvid Heise wrote: > > Hi Ken, > > just to double check, did you intend to send this mail to the tika dev > list? I actually don't know what to do with your email. > > Best, > > Arvid > > On Sat, Nov 21, 2020 at 11:43 PM Ken Krugler

More issues with top-level build for Tika 1.25 rc1 - Waited more than 5 minutes for a SAXParser

a SAXParser. Consider increasing the XMLReaderUtils.POOL_SIZE … and so on… Any suggestions? Thanks! — Ken -- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: Avoiding "too many open files" during leftOuterJoin with Flink 1.11/batch

uppose). > > Are there possibly other sources for files being created? (anything in the > user-code?) > Could it be something annoying like the buffers rapidly switching between > spilling/reading from memory, creating a new file on each spill, overwhelming > the OS? > >

Avoiding "too many open files" during leftOuterJoin with Flink 1.11/batch - now with stack trace

eal source of the problem), but the leftOuterJoin failed first. -- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Avoiding "too many open files" during leftOuterJoin with Flink 1.11/batch

- a CoGroup is happening at the same time, so it’s possible that’s also hurting (or maybe the real source of the problem), but the leftOuterJoin failed first. -- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: HadoopOutputFormat has issues with LocalExecutionEnvironment?

> What do you mean by "algorithm version 2"? Where can you set this? (Sorry > for the question, I'm not an expert with Hadoop's FileOutputCommitter) > > Note to others: There's a related discussion here: > https://issues.apache.org/jira/browse/FLINK-19069 >

HadoopOutputFormat has issues with LocalExecutionEnvironment?

if Flink should always be using version 2 of the algorithm, as that’s more performant when there are a lot of results (which is why it was added). Thanks, — Ken -- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassa

Re: Inconsistent behavior between different states with ListState.get().iterator().remove()

gt; might have come to rely on it. > > The next best thing would be throwing an exception for RocksDB to at least > not silently ignore ineffective #remove() calls. > > Best, > Aljoscha > > On 23.07.20 20:40, Ken Krugler wrote: >> Hi devs, >> If you use the FsState

Inconsistent behavior between different states with ListState.get().iterator().remove()

ing RocksDB (serde cost for entire list), so I’d be in favor of the remove() call throwing an exception, at least with RocksDB. ------ Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: Potential block size issue with S3 binary files

an Ewen wrote: > >> Sounds reasonable. >> >> I am adding Arvid to the thread - IIRC he authored that tool in his >> Stratosphere days. And my a stroke of luck, he is now working on Flink >> again. >> >> @Arvid - what are your thoughts on Ken'

Re: Potential block size issue with S3 binary files

lue, and store the POJO as the data value. Then you could also leverage Hadoop support for compression at either record or block level. — Ken > > On Thu, Aug 29, 2019 at 4:49 AM Ken Krugler <mailto:kkrugler_li...@transpac.com>> wrote: > Hi all, > > Wondering if anyone else

Re: Unit tests consuming a lot of disk

reases. > > Is this a known issue? Or would this be something worth investigating as an > improvement? > > Thanks, > Tim -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

Re: Support for controlling slot assignment based on CPU requirements

ph vertex do not goes into the same > slot/TM. > > Thank you~ > > Xintong Song > > > > On Thu, Jun 13, 2019 at 4:58 AM Ken Krugler > wrote: > >> Hi all, >> >> I’m running a complex (batch) workflow that has a step where it trains >> Fa

Support for controlling slot assignment based on CPU requirements

INK/Flink+Improvement+Proposals>, and didn’t see anything that covered this. I image it would need to be something like YARN’s support for per-node vCore capacity and per-task vCore requirements, but on a per-TM/per-operator basis. ------ Ken Krugler +1 530-210-637

Re: [DISCUSS] Support Local Aggregation in Flink

;>>>> >>>>> Local aggregation is a widely-adopted method to reduce the performance >>>>> degraded by data skew. We can decompose the aggregating operations into >>>> two >>>>> phases. In the first phase, we aggregate the el

Re: [DISCUSS] Apache Flink at Season of Docs

gt; On Thu, 11 Apr 2019 at 02:11, Aizhamal Nurmamat kyzy >>>> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> @Fabian Hueske - SoD setup is a little bit >>> different. >>>>> The ASF determined that each project would be allowed to app

Re: [DISCUSS] Apache Flink at Season of Docs

ge on this, if you, as an organization decide to apply. > > I think it will be great if Flink participates in it too! > > Thanks, > Aizhamal > > [1] https://developers.google.com/season-of-docs/ > [2] https://developers.google.com/season-of-docs/docs/timeline ---

Re: Ratelimiting in the Flink Kafka connector

, taking into account more/other factors. >>> >>> Both, adding the limiter and making the consumer code more adoptable >> could >>> be split into separate work also. >>> >>> BTW is there a JIRA for this? >>> >>> Thomas >

Re: [DISCUSS] Detection Flink Backpressure

r of all buffers) instead. >>>> >>>> >>>>Best, >>>> Yun Gao >>>> >>>> >>>>[1] https://issues.apache.org/jira/browse/FLINK-10981 >>>>[2] https://issues.apache.org/jira/browse/FLINK-11082 >>>> >>>> >>>> -- >>>> From:裴立平 >>>> Send Time:2019 Jan. 3 (Thu.) 13:39 >>>> To:dev >>>> Subject:[DISCUSS] Detection Flink Backpressure >>>> >>>> Recently I want to optimize the way to find the positions where the >>>> backpressures occured . >>>> >>>> I read some blogs about flink-backpressure and have a rough idea of it . >>>> >>>> The method which Flink adopted is thread-stack-sample , it's heavy and >>>> no-lasting . >>>> >>>> The positions where backpressures occured are very important to the >>>> developers . >>>> >>>> They should be treated as monitor-metrics . >>>> >>>> Any other choice that we can take to detection the flink backpressures ? >>>> >>> >> >> -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

Re: Unbalanced processing of KeyedStream

ion. But to me it feels a bit hack-ish. > I would like to know if this is only solution with Flink or do I miss > something? > Can there be more API-ish support for such use-case from Flink? Is there a > reason why there is none? Or is there? > > On Wed, Jan 2, 2019 at 5:29 PM K

Re: Unbalanced processing of KeyedStream

edStream()` more useable for certain > scenarios. > * any other option I have? > > Many thanks in advance. > > Best, > Jozef -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

Re: Enrich testing doc with more unit test examples using AbstractStreamOperator

to > structure the content. > > I would really appreciate any feedback from you. Thanks in advance. > > Best Regards, > Tony Wei > > [1] > https://github.com/apache/flink/compare/master...tony810430:flink-testing-doc -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

State for moving average calculation

serialized accumulator object? Thanks, — Ken -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

Re: Facing issue in RichSinkFunction

ethod, you create those objects using settings from other (serializable) fields that aren’t transient. — Ken -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

Re: [WEBSITE] Proposal to rework the Flink website

Along these lines, it would help to add a sitemap (and the robots.txt required to reference it) for flink.apache.org and ci.apache.org (for /projects/flink) You can see what Tomcat did along these lines - http://tomcat.apache.org/robots.txt references http://tomcat.apache.org/sitemap.xml, which

Re: Failure restarting Flink 1.5.0 job from checkpoint

t; Hi Ken, > > I think you might have independently discovered this issue: > https://issues.apache.org/jira/browse/FLINK-9458 > <https://issues.apache.org/jira/browse/FLINK-9458> > > Best, > Aljoscha > >> On 31. May 2018, at 01:46, Ken Krugler wrote: >> &g

Failure restarting Flink 1.5.0 job from checkpoint

Hi devs, I coded up a simple iteration that uses a KeyedProcessFunction, as a way of showing how to use timers to do state iteration. This worked fine, but then I wanted to try out checkpoints. I modified the KeyedProcessFunction to throw an exception after a fixed number of calls. When this h

Re: Confusing error message in MemCheckpointStreamFactory with large state size?

r parallelism, or using > a non-memory-based state backend". > > Could you please open a JIRA ticket in: > https://issues.apache.org/jira/projects/FLINK/issues/ Sure, see https://issues.apache.org/jira/browse/FLINK-9300 <https://issues.apache.org/jira/browse/FLINK-9300>

[jira] [Created] (FLINK-9300) Improve error message when in-memory state is too large

Ken Krugler created FLINK-9300: -- Summary: Improve error message when in-memory state is too large Key: FLINK-9300 URL: https://issues.apache.org/jira/browse/FLINK-9300 Project: Flink Issue Type

[jira] [Created] (FLINK-9299) ProcessWindowFunction documentation Java examples have errors

Ken Krugler created FLINK-9299: -- Summary: ProcessWindowFunction documentation Java examples have errors Key: FLINK-9299 URL: https://issues.apache.org/jira/browse/FLINK-9299 Project: Flink

Confusing error message in MemCheckpointStreamFactory with large state size?

; So shouldn’t it suggest using the RocksDB state backend instead? — Ken -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra

CoProcessFunction doesn't support timer on keyed stream

Hi devs, I’m using Flink 1.5-SNAPSHOT, and I’ve got a connected stream that I’m using with a CoProcessFunction. One of the streams is keyed, and the other is broadcast. As per the documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.4/api/java/org/apache/flink/streaming/api

Validating markdown doc files before creating a PR

Hi all, I’m curious how devs check that their .md file edits are correct. I've tried a few different plugins and command line utilities, but the mix of HTML and markdown has meant none of them render properly. Thanks, — Ken ------ Ken Krugler http://www.scaleunlimite

Re: Documentation glitch w/AsyncFunction?

resultFuture.complete(Collections.singleton(new > Tuple2<>(str, result))); > > Looking at existing code in unit tests, the complete() call is on the > parameter. > > Cheers > > On Sat, Apr 14, 2018 at 8:34 AM, Ken Krugler > wrote: > >> Hi devs

Documentation glitch w/AsyncFunction?

Hi devs, https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/asyncio.html Has this example of asyncInvoke: > @Override > public void asyncInvoke(final String str, fi

Re: Multi-stream question

checkpointing and other logic accomodate this if I write sufficient custom > code in the operator? > > Michael > >> On Apr 7, 2018, at 10:42 AM, Ken Krugler wrote: >> >> When you say “join” are you talking about a real join (so one or more fields >> can b

Re: [DISCUSS] Inverted (child-first) class loading

e classes > - parent first conflict resolution > - child first conflict resolution > > Having number one as the default and let the error message suggest options > two and three as options would definitely make users aware of the issue... > > On Fri, Mar 9, 2018, 21:09 K

Re: [DISCUSS] Inverted (child-first) class loading

I can’t believe I’m suggesting this, but perhaps the Elasticsearch “Hammer of Thor” (aka “jar hell”) approach would be appropriate here. Basically they prevent a program from running if there are duplicate classes on the classpath. This causes headaches when you really need a different version

[jira] [Created] (FLINK-8849) Wrong link from concepts/runtime to doc on chaining

Ken Krugler created FLINK-8849: -- Summary: Wrong link from concepts/runtime to doc on chaining Key: FLINK-8849 URL: https://issues.apache.org/jira/browse/FLINK-8849 Project: Flink Issue Type

Re: Terminating streaming test

this is a rather common testing need that's already solved?! > > Thanks, > Thomas -- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: Visualizing topologies

gt;> visualizations. >> >> Let's see what others think. >> >> Best, Fabian >> >> >> 2017-02-24 0:15 GMT+01:00 Ken Krugler : >> >>> Hi Ufuk, >>> >>>> On Feb 22, 2017, at 2:18am, Ufuk Celebi wrote: >>>

Re: Visualizing topologies

pen an issue and attach the code. — Ken > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai > wrote: >> Hi Ken, >> >> Maybe you can look into this one : http://flink.apache.org/visualizer/. >> >> - Pat >> >>> On Feb 21, 2017, at 11:59 PM,

Re: Visualizing topologies

ability to edit/annotate as needed. Thoughts on having the iteration sink connect to the iteration source? I found that helpful when working on my topology. Regards, — Ken >> On Feb 21, 2017, at 11:59 PM, Ken Krugler >> wrote: >> >> Hi list, >> >>

Visualizing topologies

#getExecutionPlan and generates a .dot file The results look like this... http://scaleunlimited.com/downloads/CrawlTopology.png If this is useful (and doesn’t already exist) then I can easily drop it in somewhere. — Ken -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com

Re: 1.2-SNAPSHOT documentation

>> >>> the docs on the website need to be built manually at the moment. So they >>> might be out of sync. >>> If you want the most recent documentation you can checkout the git >>> repository and built the docs: >>> >>> git clone https://g

1.2-SNAPSHOT documentation

nction. So should I be using a different link for snapshot Javadocs? Thanks, — Ken ------ Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

[jira] [Created] (FLINK-3880) Use ConcurrentHashMap for Accumulators

Ken Krugler created FLINK-3880: -- Summary: Use ConcurrentHashMap for Accumulators Key: FLINK-3880 URL: https://issues.apache.org/jira/browse/FLINK-3880 Project: Flink Issue Type: Improvement

License check no longer failing (in master)

Hi devs, So it looks like I had some cruft (??) left over from a previous version of Flink? The flink-staging directory no longer exists in master. Looks like apache-rat was configured to search everything from the top down, so it found these Avro-generated files in that directory. Sorry for

License check failing

Hi devs, I was trying to run mvn clean install -DskipTests from the release-1.0.3-c2 branch, but the apache-rat-plugin license check was failing on four files. These were in all Avro-generated files, in flink-staging/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/ I added

[jira] [Created] (FLINK-3878) File cache doesn't support multiple duplicate temp directories

Ken Krugler created FLINK-3878: -- Summary: File cache doesn't support multiple duplicate temp directories Key: FLINK-3878 URL: https://issues.apache.org/jira/browse/FLINK-3878 Project:

[jira] [Created] (FLINK-3839) Support wildcards in classpath parameters

Ken Krugler created FLINK-3839: -- Summary: Support wildcards in classpath parameters Key: FLINK-3839 URL: https://issues.apache.org/jira/browse/FLINK-3839 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-3838) CLI parameter parser is munging application params

Ken Krugler created FLINK-3838: -- Summary: CLI parameter parser is munging application params Key: FLINK-3838 URL: https://issues.apache.org/jira/browse/FLINK-3838 Project: Flink Issue Type: Bug

RE: cascading-flink 1.0 results

ing with Chris the ComparePlatformsTest issue. I'll exclude > that test case. > > Thanks, Fabian > > 2016-03-30 21:46 GMT+02:00 Ken Krugler : > >> Hi Fabian, >> >> I've been trying out the cascading-flink 1.0 branch (updated to >> cascading-3

cascading-flink 1.0 results

d extend > HashJoin support later. > > Best, Fabian > > [1] https://github.com/dataartisans/cascading-flink/tree/flink-1.0 > > 2016-03-30 6:08 GMT+02:00 Ken Krugler : > >> Hi Fabian, >> >>> From: Fabian Hueske >>> Sent: March 29, 2016 3:5

RE: Expected duration for cascading-flink tests?

cascading-wip-56. I also tried updating to Flink 1.0.0 (from 0.10.0), but so far I've run into some compilation errors, e.g. in FlinkFlowStep.java it can't find the JavaPlan class. Thanks again for the help, -- Ken > " > Best, Fabian > > 2016-03-29 20:36 GMT+02:00 K

RE: Expected duration for cascading-flink tests?

a:272) … Caused by: java.net.BindException: Address already in use … 2. FlowStepJob.blockOnJob throws a cascading.flow.FlowException All caused by a 100 second timeout Is the above expected? Thanks, -- Ken > From: Ken Krugler > Sent: March 28, 2016 3:39:12pm PDT > To: dev@fl

Expected duration for cascading-flink tests?

eSetExec's call() method. Maybe this is a sign that it's time for a new Mac :) Thanks, -- Ken ---------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

RE: Guarantees for object reuse modes and documentation

t; > Thanks, Ken! I was wondering how other systems handle these issues. > > Fortunately, the deep copy - shallow copy problem doesn't arise in > Flink: when we copy an object, it is always a deep copy (at least, I > hope so :)). > > Best, > Gábor > > >

RE: Guarantees for object reuse modes and documentation

ot affect performance or be easier to follow for users, IMO. > > > Greg and Gabor, please correct me if I did not get your points right or > missed something. > > What do others think? > > > Fabian > > > > [1] > https://ci.apache.org/projects/flink/

RE: User Feedback

acts many open source projects. And everyone does in fact use Google :) Wouldn't adding a sitemap help here? Regards, -- Ken [snip] -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr