Lookup join or enrichment join against a changelog stream in Apache Flink Table API

I'm interested in doing a "lookup join" or "enrichment join" against a "changelog stream" read by "upsert-kafka". I am wondering if this is possible against the table API. I found

Re: Parallelizing a tumbling group window

clarify things. > > Best, Fabian > > [1] https://issues.apache.org/jira/browse/FLINK-7561 > > 2017-12-30 0:11 GMT+01:00 Colin Williams <colin.williams.seat...@gmail.com > >: > >> Hi Timo and flink-user, >> >> >> It's been a few weeks a

Re: Parallelizing a tumbling group window

task slot, parallelizing each window. But maybe that's not happening? Can you help us to understand why parallelizing the job only has a degraded impact on performance and what I can do to change this? Happy New Year! Colin Williams On Mon, Dec 11, 2017 at 4:15 AM, Timo Walther

Re: Flink upgrade compatibility table

information for consistency. Thanks, Colin Williams On Dec 20, 2017 8:16 PM, "Kien Truong" <duckientru...@gmail.com> wrote: > Hi Colin, > > Did you try to rebuild the application with Flink 1.4 ? You cannot just > take a jar build with 1.3 and run it on 1.4 cluster. Afa

Flink upgrade compatibility table

in the compatibility table: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/upgrading.html#compatibility-table Does 1.4 break compatibility? Maybe the 1.4 docs should be updated to reflect that? Thanks, Colin Williams

Re: flink jobmanager HA zookeeper leadership election - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.

On Tue, Dec 19, 2017 at 7:29 PM, Colin Williams < colin.williams.seat...@gmail.com> wrote: > Hi, > > I've been trying to update my flink-docker jobmanager configuration for > flink 1.4. I think the system is shutting down after a leadership election, > but I'm not sure

flink jobmanager HA zookeeper leadership election - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.

path in case it had some old session information, but that didn't seem to help. Best, Colin Williams out Description: Binary data

docker-flink images and CI

I created the following issue on here: https://github.com/docker-flink/docker-flink/issues/29 where it was suggested I should bring this up on the list. 1.4 has been released. Hurray! But there isn't yet a dockerfile / image for the release. Furthermore, it would be nice to have dockerfile /

Parallelizing a tumbling group window

the documentation. Best, Colin Williams

Re: Docker and AWS taskmanager configuration

n we saw "unique-dns-address" was set for the taskmanagers akka address. Given we are ok with 1 taskmanager container per host, this all worked out. Thanks, Colin Williams On Tue, Nov 21, 2017 at 6:14 AM, Patrick Lucas <patr...@data-artisans.com> wrote: > Hi Colin, > >

Docker and AWS taskmanager configuration

Hi, We noticed that we couldn't parallelize our flink docker containers and this looks like an issue that other have experienced. In our environment we were not setting any hostname in the flink configuration. This worked for the single node, but it looks like the taskmanagers would have the

Re: Streaming User-defined Aggregate Function gives exception when using Table API jars

the code (both for the batch as > well as the streaming queries). > I have a fix that I need to verify. > > It's not necessary to open a new JIRA for that. We can cover all cases > under FLINK-7490. > > Thanks, Fabian > > 2017-11-15 5:32 GMT+01:00 Colin Williams <colin.w

Streaming User-defined Aggregate Function gives exception when using Table API jars

ent (Batch vs Stream). Should another bug report be filed? Also that bug report hasn't really had any activity and it's been a few months. Best Regards, Colin Williams java.io.IOException: Exception while applying AggregateFunction in aggregating st

Re: Writing an Integration test for flink-metrics

ink-docs-release >> -1.4/dev/stream/testing.html >> >> However as we said before, testing metrics would require using custom or >> a imx reporter. >> >> Yes, please report this bug in Jira. >> >> Thanks, Piotrek >> >> On 13 Oct 2017, at 04:31

Re: Writing an Integration test for flink-metrics

as you proposed using JMXReporter (or custom reporter) should >>> work. I think there is no easier way to do this at the moment. >>> >>> Piotrek >>> >>> On 12 Oct 2017, at 04:58, Colin Williams <colin.williams.seattle@gmail. >>&g

Writing an Integration test for flink-metrics

I have a RichMapFunction and I'd like to ensure Meter fields are properly incremented. I've been trying to think of the best way to do this. Currently I think that I'd need to either implement my own reporter (or use JMX) and write to a socket, create a listener and wait for the reporter to send

Re: RichMapFunction parameters in the Streaming API

them in a field. This applies both to the batch and > streaming API. > > Personally i would stay away from the global configuration option as it is > more brittle than the constructor approach, which makes > it explicit that this function requires these parameters. > > > On 11.10.2017

RichMapFunction parameters in the Streaming API

I was looking for withParameters(config) in the Streaming API today. I stumbled across the following thread. http://apache-flink-mailing-list-archive.1008284.n3. nabble.com/withParameters-for-Streaming-API-td9332.html#a9333 It appears that some of the StreamingAPI developers are in favor of

Re: serialization error when using multiple metrics counters

nt for expected behavior. > > > > Seth > > > > *From: *Stephan Ewen <se...@apache.org> > *Date: *Monday, October 9, 2017 at 2:44 PM > *To: *Kostas Kloudas <k.klou...@data-artisans.com> > *Cc: *Colin Williams <colin.williams.seat...@gmail.com>, user < >

serialization error when using multiple metrics counters

I've created a RichMapFunction in scala with multiple counters like: lazy val successCounter = getRuntimeContext. getMetricGroup.counter("successfulParse") lazy val failedCounter = getRuntimeContext. getMetricGroup.counter("failedParse") lazy val errorCounter = getRuntimeContext.