Re: HadoopOutputFormat has issues with LocalExecutionEnvironment?

2020-09-03 Thread Ken Krugler
Hi Robert, I haven’t tried yet with 1.11, on my list. I’ll be spending time on this tomorrow, so hopefully more results. As for setting the algorithm version 2, I do it in code like this: Job job = Job.getInstance(); job.getConfiguration().set("io.serializations", "cascading.tu

Re: HadoopOutputFormat has issues with LocalExecutionEnvironment?

2020-09-02 Thread Robert Metzger
Hi Ken, sorry for the late reply. This could be a bug in Flink. Does the issue also occur on Flink 1.11? Have you set a breakpoint in the HadoopOutputFormat.finalizeGlobal() when running locally to validate that this method doesn't get called? What do you mean by "algorithm version 2"? Where can

HadoopOutputFormat has issues with LocalExecutionEnvironment?

2020-08-25 Thread Ken Krugler
Hi devs, In HadoopOutputFormat.close(), I see code that is trying to rename /tmp-r-1 to be /1 But when I run my Flink 1.9.2 code using a local MiniCluster, the actual location of the tmp-r-1 file is: /_temporary/0/task___r_01/tmp-r-1 I think this is because the default beh