If you group by the host that you have computed using the UDF, Spark is
always going to shuffle your dataset, even if the end result is that all
the new partitions look exactly like the old partitions, just placed on
differrent nodes. Remember the hostname will probably hash differrently
than the partition key of the data.
Let's say, you are trying to do is read a file, apply a UDF, and write out
to file. Without your "performance improvement", Spark will read partitions
, apply the UDF to the rows in the partitions, and write the rows out..
With your upgrade, it will read the partitions, apply the hostname udf,
shuffle by host name, apply the UDF on the shuffled rows, and write the
data out
If your intent is to increase efficiency, this will do the opposite of what
you are trying to do
On Mon, Aug 27, 2018 at 1:23 PM Patrick McCarthy
<pmccar...@dstillery.com.invalid> wrote:
> When debugging some behavior on my YARN cluster I wrote the following
> PySpark UDF to figure out what host was operating on what row of data:
>
> @F.udf(T.StringType())
> def add_hostname(x):
>
> import socket
>
> return str(socket.gethostname())
>
> It occurred to me that I could use this to enforce node-locality for other
> operations:
>
> df.withColumn('host', add_hostname(x)).groupBy('host').apply(...)
>
> When working on a big job without obvious partition keys, this seems like
> a very straightforward way to avoid a shuffle, but it seems too easy.
>
> What problems would I introduce by trying to partition on hostname like
> this?
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates and may only be used solely in performance of
work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
