Fabian Hueske created FLINK-1139:
Summary: HadoopOutputFormat fails for DOP > 1
Key: FLINK-1139
URL: https://issues.apache.org/jira/browse/FLINK-1139
Project: Flink
Issue Type: Bug
You actually can check directly in the broadcast set. But since it is a
list, searching in is slower than in a hash set. That is all...
On Mon, Oct 6, 2014 at 1:09 PM, Martin Neumann wrote:
> Yes, thanks.
>
> I was wondering if I can check directly in the broadcast set but since I
> have to get
Yes, thanks.
I was wondering if I can check directly in the broadcast set but since I
have to get it local anyway it should be not to much overhead.
cheers Martin
On Mon, Oct 6, 2014 at 12:34 PM, Stephan Ewen wrote:
> Hej!
>
> Yes, the "getRuntimeEnvironment().getBroadcastVariable() returns a
Hej!
Yes, the "getRuntimeEnvironment().getBroadcastVariable() returns a list,
which you can add to set:
// in the function:
private Set specials;
public void open(Configuration conf) {
List bc =
getRuntimeContect().getBroadcastVariable("the-bc-var-name");
specials = new HashSet(bc);
}
Hej,
I have a Flink job with with a filter step. I now have a list of exceptions
where I need to do some extra work (300k data). I thought I just use a
boradcast set and then for each like compare if its in the exception set.
What is the best way to implement this in Flink? Is there an efficient
