Hi David,
Before with the both n and -s it was not the case.
>
What do you mean by before? At least in 1.8 "-s" could be used to specify
the
number of slots per TM.
how can I be sure that my Sink that uses this lib is in one JVM ?
>
Is it enough that no other parallel instance of your sink runs in the same
JVM? If that is the case, it is enough to start your your YARN session with:
./bin/yarn-session.sh -s 1 [...]
This will result in exactly one slot per TM. Note that a single slot may
still
hold several subtasks of the job (Slot Sharing) but never two parallel
instances of your sink [2]. You can also control Slot Sharing manually [3].
So, if I understand I have to keep this Flink release (1.9.2) ?
>
I don't see why 1.10.0 would not work for you.
Best,
Gary
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/yarn_setup.html#start-a-session
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/concepts/runtime.html#task-slots-and-resources
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/#task-chaining-and-resource-groups
On Tue, Feb 25, 2020 at 10:28 AM David Morin <morin.david....@gmail.com>
wrote:
> Hi Xintong,
>
> At the moment I'm using the 1.9.2 with this command:
> yarn-session.sh -d *-s 1* -jm 4096 -tm 4096 -qu "XXX" -nm "MyPipeline"
> So, after a lot of tests, I've noticed that if I increase the parallelism
> of my Custom Sink, each task is embedded into one TS and, the most
> important, each one into one TaskManager (Yarn container in fact).
> So, if I understand I have to keep this Flink release (1.9.2) ?
>
> Thanks
> David
>
>
>
> Le mar. 25 févr. 2020 à 02:02, Xintong Song <tonysong...@gmail.com> a
> écrit :
>
>> Depending on your Flink version, the '-n' option might not take effect.
>> It is removed in the latest release, but before that there were a few
>> versions where this option is neither removed nor taking effect.
>>
>> Anyway, as long as you have multiple containers, I don't think there's a
>> way to make some of the tasks scheduled to the same JVM. Not that I'm aware
>> of.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Mon, Feb 24, 2020 at 8:43 PM David Morin <morin.david....@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks Xintong.
>>> I've noticed than when I use yarn-session.sh with --slots (-s) parameter
>>> but without --container (-n) it creates one task/slot per taskmanager.
>>> Before with the both n and -s it was not the case.
>>> I prefer to use only small container with only one task to scale my
>>> pipeline and of course to prevent from thread-safe issue
>>> Do you think I cannot be confident on that behaviour ?
>>>
>>> Regards,
>>> David
>>>
>>> On 2020/02/22 17:11:25, David Morin <morin.david....@gmail.com> wrote:
>>> > Hi,
>>> > My app is based on a lib that is not thread safe (yet...).
>>> > In waiting of the patch has been pushed, how can I be sure that my
>>> Sink that uses this lib is in one JVM ?
>>> > Context: I use one Yarn session and send my Flink jobs to this session
>>> >
>>> > Regards,
>>> > David
>>> >
>>>
>>
