Same need here, using Flink runner. We are processing a
pcollection (extracting features per element) then combining
these into groups of features and running the next operator on
those groups.
Each group contains ~50 elements, so the parallelism of the
operator upstream of the groupby should be higher, to be balanced
with the downstream operator.
On Tue, Apr 18, 2023 at 19:17 Jeff Zhang <zjf...@gmail.com> wrote:
Hi Reuven,
It would be better to set parallelism for operators, as I
mentioned before, there may be multiple groupby, join
operators in one pipeline, and their parallelism can be
different due to different input data sizes.
On Wed, Apr 19, 2023 at 3:59 AM Reuven Lax <re...@google.com>
wrote:
Jeff - does setting the global default work for you, or
do you need per-operator control? Seems like it would be
to add this to ResourceHints.
On Tue, Apr 18, 2023 at 12:35 PM Robert Bradshaw
<rober...@google.com> wrote:
Yeah, I don't think we have a good per-operator API
for this. If we were to add it, it probably belongs
in ResourceHints.
On Sun, Apr 16, 2023 at 11:28 PM Reuven Lax
<re...@google.com> wrote:
Looking at FlinkPipelineOptions, there is a
parallelism option you can set. I believe this
sets the default parallelism for all Flink operators.
On Sun, Apr 16, 2023 at 7:20 PM Jeff Zhang
<zjf...@gmail.com> wrote:
Thanks Holden, this would work for Spark, but
Flink doesn't have such kind of mechanism, so
I am looking for a general solution on the
beam side.
On Mon, Apr 17, 2023 at 10:08 AM Holden Karau
<hol...@pigscanfly.ca> wrote:
To a (small) degree Sparks “new” AQE
might be able to help depending on what
kind of operations Beam is compiling it
down to.
Have you tried setting
spark.sql.adaptive.enabled &
spark.sql.adaptive.coalescePartitions.enabled
On Mon, Apr 17, 2023 at 10:34 AM Reuven
Lax via user <user@beam.apache.org> wrote:
I see. Robert - what is the story for
parallelism controls on GBK with the
Spark or Flink runners?
On Sun, Apr 16, 2023 at 6:24 PM Jeff
Zhang <zjf...@gmail.com> wrote:
No, I don't use dataflow, I use
Spark & Flink.
On Mon, Apr 17, 2023 at 8:08 AM
Reuven Lax <re...@google.com> wrote:
Are you running on the
Dataflow runner? If so,
Dataflow - unlike Spark and
Flink - dynamically modifies
the parallelism as the
operator runs, so there is no
need to have such controls.
In fact these specific
controls wouldn't make much
sense for the way Dataflow
implements these operators.
On Sun, Apr 16, 2023 at
12:25 AM Jeff Zhang
<zjf...@gmail.com> wrote:
Just for
performance tuning like
in Spark and Flink.
On Sun, Apr 16, 2023 at
1:10 PM Robert Bradshaw
via user
<user@beam.apache.org> wrote:
What are you trying
to achieve by setting
the parallelism?
On Sat, Apr 15, 2023
at 5:13 PM Jeff Zhang
<zjf...@gmail.com> wrote:
Thanks Reuven,
what I mean is to
set the
parallelism in
operator level.
And the input
size of the
operator is
unknown at
compiling stage
if it is not a
source
operator,
Here's an example
of flink
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/#operator-level
Spark also
support to set
operator level
parallelism (see
groupByKey and
reduceByKey):
https://spark.apache.org/docs/latest/rdd-programming-guide.html
On Sun, Apr 16,
2023 at 1:42 AM
Reuven Lax via
user
<user@beam.apache.org>
wrote:
The maximum
parallelism is
always
determined by
the
parallelism
of your data.
If you do a
GroupByKey
for example,
the number of
keys in your
data
determines
the maximum
parallelism.
Beyond the
limitations
in your data,
it depends on
your
execution
engine. If
you're using
Dataflow,
Dataflow is
designed to
automatically
determine the
parallelism
(e.g. work
will be
dynamically
split and
moved around
between
workers, the
number of
workers will
autoscale,
etc.), so
there's no
need to
explicitly
set the
parallelism
of the execution.
On Sat, Apr
15, 2023 at
1:12 AM Jeff
Zhang
<zjf...@gmail.com>
wrote:
Besides
the
global
parallelism of
beam job,
is there
any way
to set
parallelism for
individual
operators
like
group by
and join?
I
understand the
parallelism setting
depends
on the
underlying
execution
engine,
but it is
very
common to
set
parallelism like
group by
and join
in both
spark &
flink.
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance
Spark, etc.): https://amzn.to/2MaRAG9
<https://amzn.to/2MaRAG9>
YouTube Live Streams:
https://www.youtube.com/user/holdenkarau
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
