> wrote:
>>>>
>>>>> Thank you for your quick reply!
>>>>>
>>>>> Is there any plan to improve this?
>>>>>
>>>>> I asked this question due to some investigation on comparing those
>>>>> state of art
ome investigation on comparing those
>>>> state of art streaming systems, among which Flink and DataFlow allow
>>>> changing parallelism number, and by my knowledge of Spark Streaming, it
>>>> seems it is also able to do that: if some “key interval” concept is used,
>>>>
>> also able to do that: if some “key interval” concept is used, then state
>>> can somehow decoupled from partition number by consistent hashing.
>>>
>>>
>>>
>>>
>>>
>>> Regards
>>>
>>> Jialei
>>>
>>&g
;
>>
>>
>> Regards
>>
>> Jialei
>>
>>
>>
>> *From: *Jacek Laskowski
>> *Date: *Wednesday, June 26, 2019 at 11:00 AM
>> *To: *"Rong, Jialei"
>> *Cc: *"user @spark"
>> *Subject: *Re: Change parallelism n
AM
> *To: *"Rong, Jialei"
> *Cc: *"user @spark"
> *Subject: *Re: Change parallelism number in Spark Streaming
>
>
>
> Hi,
>
>
>
> It's not allowed to change the numer of partitions after your streaming
> query is started.
>
>
&g
Fantastic, thanks!
From: Jungtaek Lim
Date: Wednesday, June 26, 2019 at 2:59 PM
To: "Rong, Jialei"
Cc: Jacek Laskowski , "user @spark"
Subject: Re: Change parallelism number in Spark Streaming
Hi,
you could consider state operator's partition numbers as "max para
To: *"Rong, Jialei"
> *Cc: *"user @spark"
> *Subject: *Re: Change parallelism number in Spark Streaming
>
>
>
> Hi,
>
>
>
> It's not allowed to change the numer of partitions after your streaming
> query is started.
>
>
>
> The re
to do
that: if some “key interval” concept is used, then state can somehow decoupled
from partition number by consistent hashing.
Regards
Jialei
From: Jacek Laskowski
Date: Wednesday, June 26, 2019 at 11:00 AM
To: "Rong, Jialei"
Cc: "user @spark"
Subject: Re: Chang
Hi,
It's not allowed to change the numer of partitions after your streaming
query is started.
The reason is exactly the number of state stores which is exactly the
number of partitions (perhaps multiplied by the number of stateful
operators).
I think you'll even get a warning or an exception
Hi Dear Spark Expert
I’m curious about a question regarding Spark Streaming/Structured Streaming:
whether it allows to change parallelism number(the default one or the one
specified in particular operator) in a stream having stateful
transform/operator? Whether this will cause my checkpointed
10 matches
Mail list logo
