Github user carlos-verdes commented on the issue: https://github.com/apache/spark/pull/15229 Hi @rxin, In Hive you have two levels, the partition and the buckets. The partitons are translated to folders on HDFS, for example: ```bash /apps/hive/warehouse/model_table/date=6 ``` Where model_table is the name of the table and date is the partition. Inside a folder you will have n files and Hive let you decide how many files you want to create (buckets) and which data you want to store within. If you create a table like this on Hive: ```sql create table events ( timestamp: long, userId: String, event: String ) partitioned by (event_date int) clustered by (userId) sorted by (userId, timestamp) into 10 buckets; ``` Then when it will be only 10 files per partition and all the events for one user will be only on one partition and sorted by time. If you insert data on this table using the next query on Hive you will see that the clustering policy is respected: ```sql set hive.enforce.bucketing = true; -- (Note: Not needed in Hive 2.x onward) from event_feed_source e insert overwrite table events partition (event_date = 20170307) select e.*, 20170307 where event_day = 20170307; ``` However... if you do the next insert with Spark: ```scala sqlContext.sql("insert overwrite table events partition (event_date = 20170307) select e.*,1 from event_feed_source e") ``` You will see that the data is stored with the same partitioning as it is on the source dataset. What is the benefit of respecting the Hive clustering policy? The main benefit is to avoid shuffle and have a control on the number or partitions. To give an example we have a pipeline that reads thousands of events per user and save them into another table (model), so it means the events table is going to have x times more data than the model table (imagine a factor of 10x). First point is, if the source data are clustered properly we can read all the events per user without shuffle (I mean to do something like `events.groupBy(user).mapValues(_.sortBy(timestamp)` will be done without shuffle). Second point is when we generate the model RDD/Dataser from the event RDD/Dataset. Spark respects the source partitioning (unless you indicate otherwise) which means... is going to save into Hive 10 times the number of files for the model as needed (not respecting the clustering policy on Hive). This implies that we have 10x more partitions than needed and also that the queries over the model table are not "clustered"... which means full scan every time we need to do a query (a full scan over 10 times the optimal number of partitions). I hope I clarify the point on Hive clusters ;)
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org