[GitHub] [spark] c21 commented on pull request #29342: [SPARK-32399][SQL] Full outer shuffled hash join

[GitBox]

c21 commented on pull request #29342:
URL: https://github.com/apache/spark/pull/29342#issuecomment-672640787


   @cloud-fan, @agrawaldevesh, @maropu and @viirya - updated the PR with latest 
proposed change (I still need to add unit test for `BytesToBytesMap` and 
`HashedRelation`, but the added unit test in `JoinSuite` should give us enough 
confidence for end-to-end working now. Would like to get feedback first before 
spending more time crafting more unit tests, thanks).
   
   Tested with the same example small benchmark query in PR description, still 
seeing 30% wall clock time improvement compared to sort merge join (I agree 
this is much more a toy benchmark query, but it should give us some confidence 
that we are not doing some very wrong thing here in terms of performance):
   
   ```
   Running benchmark: shuffle hash join
     Running case: shuffle hash join off
     Stopped after 2 iterations, 16602 ms
     Running case: shuffle hash join on
     Stopped after 5 iterations, 31911 ms
   
   Java HotSpot(TM) 64-Bit Server VM 1.8.0_181-b13 on Mac OS X 10.15.4
   Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
   shuffle hash join:                        Best Time(ms)   Avg Time(ms)   
Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   
------------------------------------------------------------------------------------------------------------------------
   shuffle hash join off                              7900           8301       
  567          2.1         470.9       1.0X
   shuffle hash join on                               6250           6382       
   95          2.7         372.5       1.3X
   ```
   
   Also running added unit test in `JoinSuite`. Verified all new added logic 
inside `ShuffledHashJoin` is 100% code covered (the not covered ones are 
related to code-gen, which is irrelevant here):
   
   <img width="1664" alt="Screen Shot 2020-08-11 at 11 10 21 PM" 
src="https://user-images.githubusercontent.com/4629931/89983056-bebd4180-dc2b-11ea-9fe3-cdf06143a002.png";>
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org