Hi Alex,
I’m not very familiar with JsonLinesInputFormat, is that your own
implementation? You may look into the `createInputSplits()` method which should
do the listing work. You may rewrite it with concurrent listing.
> On Dec 13, 2018, at 11:56 PM, Alex Vinnik wrote:
>
> Qi,
>
> Thanks fo
Qi,
Thanks for references! How do enable concurrent s3 file listing? Here is
the code.
// Consume the JSON files
Configuration configuration = new
Configuration(GlobalConfiguration.loadConfiguration());
configuration.setBoolean(JsonLinesInputFormat.ENUMERATE_NESTED_FILES_FLAG,
true);
JsonLinesIn
Hi Alex,
The hard code I’ve found is [1] and [2].
We encountered a similar issue like yours (listing a lot of HDFS files). We end
up with a newer version of HDFSFileInput which lists files concurrently.
Another hack we did is to list the files in client side and pass them to
JobManager via ser
Qi,
Job submission timeout is caused by listing too many files in S3
during env.readFile call to create input DataSet. Is there a way NOT to
list S3 files during a job submission? It seems like it should help to
mitigate that timeout problem.
What hardcoded value you were referring to?
Best,
-Al
Hi Qi,
Thanks for looking into this. Here is ticket
https://issues.apache.org/jira/browse/FLINK-11143
Best,
-Alex
On Tue, Dec 11, 2018 at 8:47 PM qi luo wrote:
> Hi Alex and Lukas,
>
> This error is controlled by another RPC timeout (which is hard coded and
> not affected by “akka.ask.timeout”
Hi Alex and Lukas,
This error is controlled by another RPC timeout (which is hard coded and not
affected by “akka.ask.timeout”). Could you open an JIRA issue so I can propose
a fix on that?
Cheers,
Qi
> On Dec 12, 2018, at 7:07 AM, Alex Vinnik wrote:
>
> Hi there,
>
> Run into the same prob
Hi there,
Run into the same problem running a batch job with Flink 1.6.1/1.6.2 .
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#202546747]] after [1 ms].
Sender[null] sent message of type
"org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
akka
Thanks for your answers.
In my use case I am reading from a large number of individual files. Jobs are
issued directly from the Java API, the results are collected (in memory) and
re-used partially in follow-up jobs.
I feared that using a MiniCluster or local environment I would not be able to
Hi Lukas,
It seems that when using MiniCluster, the config key akka.ask.timeout is not
respected. Instead, a hardcoded timeout of 10s is used [1]. Since all
communication is locally, it would be interesting to see in detail what your
job looks like that it exceeds the timeout.
The key akka.ask.ti
t; specific akka-timout if you don't change this class (if I'm wrong, please
>> correct me), the easiest way is just to change the ASK_TIMEOUT's default
>> value within org/apache/flink/configuration/AkkaOptions.java from "10 s" to
>> "100 s".
o
> "100 s".
>
> Best
> Yun
>
> --
> *From:* Lukas Kircher
> *Sent:* Wednesday, July 18, 2018 14:47
> *To:* user
> *Subject:* Re: Cannot configure akka.ask.timeout
>
> Hello,
>
> does anybody have an idea what is going
e),
the easiest way is just to change the ASK_TIMEOUT's default value within
org/apache/flink/configuration/AkkaOptions.java from "10 s" to "100 s".
Best
Yun
From: Lukas Kircher
Sent: Wednesday, July 18, 2018 14:47
To: user
Subject:
Hello,
does anybody have an idea what is going on? I have not yet found a solution.
Am I doing something wrong? Or is the 'akka.ask.timeout' parameter not related
to the exception stated below?
Could somebody please take a look at this? More details can be found in the
message prior to this.
Hello,
I have problems setting configuration parameters for Akka in Flink 1.5.0. When
I run a job I get the exception listed below which states that Akka timed out
after 1ms. I tried to increase the timeout by following the Flink
configuration documentation. Specifically I did the following
14 matches
Mail list logo
