Hi David,
that's correct. A TM is a single process. A slot is just a virtual concept
in the TM process and runs its program slice in multiple threads.
Besides managed memory (which is split into chunks add assigned to slots)
all other resources (CPU, heap, network, disk) are not isolated and free
I'm trying to create an API to get results from build() but anyway I'm stuck
on the StreamTransformation which is essential to implement
slotSharingGroup(...). I have to provide it from the main class.
Tony Wei wrote
> Hi Andrea,
>
> The `learn` operator is defined in this method [1]. If you nee
Hi Andrea,
I don't think you need to touch `StreamTransformation`. If you can get the
result from build(), you can do the same thing I mentioned above: casting
it as SingleOutputStreamOperator.
Then, you can pass it to select function to add the next operator, and get
the result to add another slo
Hi Ashish,
what you are describing should be a good use case for Flink and it should
be able to run your program.
When you are seeing a GC overhead limit exceeded error, then it means that
Flink or your program are creating too many/too large objects filling up
the memory in a short time. I would
Hi David,
I cannot exactly tell how you ended up seeing an OptionalDataException
without seeing your code.
Flink supports to run multiple jobs on the same cluster. That’s what we
call the session mode.
You should not reuse the ExecutionEnvironment because then, you will create
a single job which
Hi,
there is PR open for fixing the multiple TM metrics registration [1]. We
hope to get it merged in the next days such that it will be part of the
upcoming 1.4 release.
[1] https://github.com/apache/flink/pull/4872
Cheers,
Till
On Thu, Oct 26, 2017 at 5:21 PM, cslotterback wrote:
> Hello 김동
Yes please open the PR against Flink's master branch. You can also ping me
once you've opened the PR. Then we can hopefully quickly merge it :-)
Cheers,
Till
On Thu, Oct 26, 2017 at 12:44 PM, bartektartanus
wrote:
> I think we could try with option number one, as it seems to be easier to
> impl
Hi,
Two Jira issues are mentioned in this thread:
- [FLINK-7069] Catch exceptions for each reporter separately : Fixed
- [FLINK-7100] TaskManager metrics are registered twice : Unresolved
I guess your error message is just an warning message.
You can safely ignore it if you are using 1.3.2
Hi David,
in case of a streaming program with a degree of parallelism of 128, Flink
would create 128 splits. One split for each parallel sub task. The logic is
that one split will have the size of one block unless this would not give
you enough splits such that every task receives at least one spl
Is there a way to determine via an API the required number of task manager
slots for a particular job graph? I’ve looked at
"flink.getStreamGraph().getJobGraph().getNumberOfVertices()” but it doesn’t
seem to correspond with what the job actually ends up using.
--
Jared Stehler
Chief Architect -
I found in the “Flip6LocalStreamEnvironment” the following code, but with the
caveat that it doesn’t include slot reuse?
int slotsCount = 0;
for (org.apache.flink.runtime.jobgraph.JobVertex jobVertex :
flink.getStreamGraph().getJobGraph().getVertices()) {
slotsCount += jobVertex.getParal
11 matches
Mail list logo