Is there a limit on how many DataStreams can be defined in a streaming program?
Looks like flink has problems handling too many data streams? I simplified my
topology further. For eg, this works (parallelism of 4)
However, when I try to go beyond 51 (found empirically by parametrizing
nParts), it barfs again. Submission fails, it wants me to increase
akka.client.timeout
Here is the reduced code for repro (union at the end itself is not an issue).
It is the parallelism of the first for loop:
int nParts = cfg.getInt("dummyPartitions", 4);
boolean combineAtEnd = cfg.getBoolean("dummyCombineAtEnd", true);
// create lots of streams
List<SingleOutputStreamOperator<String>> streams = new ArrayList<>(nParts);
for (int i = 0; i < nParts; i++) {
streams.add(env
.readFile(
new TextInputFormat(new Path("/tmp/input")),
"/tmp/input",
FileProcessingMode.PROCESS_CONTINUOUSLY,
1000,
FilePathFilter.createDefaultFilter())
.setParallelism(1).name("src"));
}
if (combineAtEnd == true) {
DataStream<String> combined = streams.get(0);
for (int i = 1; i < nParts; i++) {
combined = combined.union(streams.get(i));
}
combined.print().setParallelism(1);
} else { // die parallel
for (int i = 1; i < nParts; i++) {
streams.get(i).print();
}
}
> On Jan 23, 2017, at 6:14 AM, Abhishek R. Singh
> <abhis...@tetrationanalytics.com> wrote:
>
> I even make it 10 minutes:
>
> akka.client.timeout: 600s
>
> But doesn’t feel like it is taking effect. It still comes out at about the
> same time with the same error.
>
> -Abhishek-
>
>> On Jan 23, 2017, at 6:04 AM, Abhishek R. Singh
>> <abhis...@tetrationanalytics.com <mailto:abhis...@tetrationanalytics.com>>
>> wrote:
>>
>> yes, I had increased it to 5 minutes. It just sits there and bails out again.
>>
>>> On Jan 23, 2017, at 1:47 AM, Jonas <jo...@huntun.de
>>> <mailto:jo...@huntun.de>> wrote:
>>>
>>> The exception says that
>>>
>>> Did you already try that?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/weird-client-failure-timeout-tp11201p11204.html
>>>
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/weird-client-failure-timeout-tp11201p11204.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list archive
>>> at Nabble.com <http://nabble.com/>.
>>
>
