Re: Why are hash functions seeded with 42?
+1 to doc, seed argument would be great if possible From: Sean Owen Sent: Monday, September 26, 2022 5:26:26 PM To: Nicholas Gustafson Cc: dev Subject: Re: Why are hash functions seeded with 42? Oh yeah I get why we love to pick 42 for random things. I'm guessing it was a bit of an oversight here as the 'seed' is directly initial state and 0 makes much more sense. On Mon, Sep 26, 2022, 7:24 PM Nicholas Gustafson mailto:njgustaf...@gmail.com>> wrote: I don’t know the reason, however would offer a hunch that perhaps it’s a nod to Douglas Adams (author of The Hitchhiker’s Guide to the Galaxy). https://news.mit.edu/2019/answer-life-universe-and-everything-sum-three-cubes-mathematics-0910 On Sep 26, 2022, at 16:59, Sean Owen mailto:sro...@gmail.com>> wrote: OK, it came to my attention today that hash functions in spark, like xxhash64, actually always seed with 42: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L655 This is an issue if you want the hash of some value in Spark to match the hash you compute with xxhash64 somewhere else, and, AFAICT most any other impl will start with seed=0. I'm guessing there wasn't a great reason for this, just seemed like 42 was a nice default seed. And we can't change it now without maybe subtly changing program behaviors. And, I am guessing it's messy to let the function now take a seed argument, esp. in SQL. So I'm left with, I guess we should doc that? I can do it if so. And just a cautionary tale I guess, for hash function users.
Re: Depolying stage-level scheduling for Spark SQL
Thanks for the clarification Tom! A bit more backgrounds for what we want to do: we have proposed a fine-grained (stage-level) resource optimization approach in VLDB22 https://www.vldb.org/pvldb/vol15/p3098-lyu.pdf and would like to try it over Spark. Our approach can recommend the resource configuration for each stage automatically (by using ML and our optimization framework), and we would like to see how to embed it in Spark. Initially, we consider that there is no AQE to make it simpler. Now I see the problem in two folds (In both cases, the stage-level configurations will be automatically configured by our algorithm with the the upper and lower bounds of each tunable resource given by a user): (1) If AQE is disabled in Spark SQL, and hence the RDD DAG will not be changed after the physical plan is selected, do you think it is feasible and worth exposing the RDDs and reusing the existing stage-level scheduling API for optimization? (2) If AQE is enabled in Spark SQL, I would agree and prefer to add the stage-level resource optimization inside the AQE. Since I am not very experienced with the AQE part, would you list more potential challenges it may lead to? Thanks in advance and I would really appreciate it if you could give us more feedback! Cheers, Chenghao On Sep 30, 2022, 4:22 PM +0200, Tom Graves , wrote: > see the original SPIP for as to why we only support RDD: > https://issues.apache.org/jira/browse/SPARK-27495 > > > The main problem is exactly what you are referring to. The RDD level is not > exposed to the user when using SQL or Dataframe API. This is on purpose and > user shouldn't have to know anything about the underlying impelementation > using RDDs. Especially with AQE and other optimizations that could change > things. You may start out with one physical plan and AQE can change it along > the way, so how does user change RDD at that point? It would be very > difficult to expose this to the user and I don't think it should be. I think > we would have to come up with some other way to apply stage level scheduling > to SQL/dataframe, or like mentioned in original issue if AQE gets smart > enough it would just do it for the user, but lots of factors that come into > play that make that difficult as well. > > Tom > On Friday, September 30, 2022, 04:15:36 AM CDT, Chenghao Lyu > wrote: > > > Thanks for the reply! > > To clarify, for issue 2, it could still break apart a query into multiple > jobs without AQE — I have turned off the AQE in my posted example. > > For 1, an end user just needs to turn on/off a knob to use the stage-level > scheduling for Spark SQL — I am considering adding a component between the > Spark SQL module and the Spark Core model to optimize the stage-level > resource. > > Yes, SQL is declarative. It uses a sequence of components (such as a logical > planner, physical planner, and CBO) to get a selected physical plan. The RDDs > (with the transformations) are generated based on the selected physical plan > for execution. For now, we could only get the top-level RDD of the DAG of > RDDs by `spark.sql(q1).queryExecution.toRdd`, but it is not enough to make > stage-level scheduling decisions. The stage-level resources are profiled > based on the RDDs. If we could expose the all RDDs instead of the top-level > RDD, it seems possible to apply the stage-level scheduling here. > > > P.S. let me attach the link for the RDD regeneration explicitly in case it is > not shown on the mail-list website: > https://stackoverflow.com/questions/73895506/how-to-avoid-rdd-regeneration-in-spark-sql > > Cheers, > Chenghao > On Sep 29, 2022, 5:22 PM +0200, Herman van Hovell , > wrote: > > I think issue 2 is caused by adaptive query execution. This will break > > apart queries into multiple jobs, each subsequent job will generate a RDD > > that is based on previous ones. > > > > As for 1. I am not sure how much you want to expose to an end user here. > > SQL is declarative, and it does not specify how a query should be executed. > > I can imagine that you might use different resources for different types of > > stages, e.g. a scan stage and more compute heavy stages. This, IMO, should > > be based on analysis and costing the plan. For this RDD only stage level > > scheduling should be sufficient. > > > > On Thu, Sep 29, 2022 at 8:56 AM Chenghao Lyu wrote: > > > Hi, > > > > > > I plan to deploy the stage-level scheduling for Spark SQL to apply some > > > fine-grained optimizations over the DAG of stages. However, I am blocked > > > by the following issues: > > > > > > 1. The current stage-level scheduling supports RDD APIs only. So is there > > > a way to reuse the stage-level scheduling for Spark SQL? E.g., how to > > > expose the RDD code (the transformations and actions) from a Spark SQL > > > (with SQL syntax)? > > > 2. We do not quite understand why a Spark SQL could trigger multiple > > > jobs, and have some RDDs
Re: Depolying stage-level scheduling for Spark SQL
see the original SPIP for as to why we only support RDD: https://issues.apache.org/jira/browse/SPARK-27495 The main problem is exactly what you are referring to. The RDD level is not exposed to the user when using SQL or Dataframe API. This is on purpose and user shouldn't have to know anything about the underlying impelementation using RDDs. Especially with AQE and other optimizations that could change things. You may start out with one physical plan and AQE can change it along the way, so how does user change RDD at that point? It would be very difficult to expose this to the user and I don't think it should be. I think we would have to come up with some other way to apply stage level scheduling to SQL/dataframe, or like mentioned in original issue if AQE gets smart enough it would just do it for the user, but lots of factors that come into play that make that difficult as well. Tom On Friday, September 30, 2022, 04:15:36 AM CDT, Chenghao Lyu wrote: Thanks for the reply! To clarify, for issue 2, it could still break apart a query into multiple jobs without AQE — I have turned off the AQE in my posted example. For 1, an end user just needs to turn on/off a knob to use the stage-level scheduling for Spark SQL — I am considering adding a component between the Spark SQL module and the Spark Core model to optimize the stage-level resource. Yes, SQL is declarative. It uses a sequence of components (such as a logical planner, physical planner, and CBO) to get a selected physical plan. The RDDs (with the transformations) are generated based on the selected physical plan for execution. For now, we could only get the top-level RDD of the DAG of RDDs by `spark.sql(q1).queryExecution.toRdd`, but it is not enough to make stage-level scheduling decisions. The stage-level resources are profiled based on the RDDs. If we could expose the all RDDs instead of the top-level RDD, it seems possible to apply the stage-level scheduling here. P.S. let me attach the link for the RDD regeneration explicitly in case it is not shown on the mail-list website: https://stackoverflow.com/questions/73895506/how-to-avoid-rdd-regeneration-in-spark-sql Cheers,ChenghaoOn Sep 29, 2022, 5:22 PM +0200, Herman van Hovell , wrote: I think issue 2 is caused by adaptive query execution. This will break apart queries into multiple jobs, each subsequent job will generate a RDD that is based on previous ones. As for 1. I am not sure how much you want to expose to an end user here. SQL is declarative, and it does not specify how a query should be executed. I can imagine that you might use different resources for different types of stages, e.g. a scan stage and more compute heavy stages. This, IMO, should be based on analysis and costing the plan. For this RDD only stage level scheduling should be sufficient. On Thu, Sep 29, 2022 at 8:56 AM Chenghao Lyu wrote: Hi, I plan to deploy the stage-level scheduling for Spark SQL to apply some fine-grained optimizations over the DAG of stages. However, I am blocked by the following issues: - The current stage-level scheduling supports RDD APIs only. So is there a way to reuse the stage-level scheduling for Spark SQL? E.g., how to expose the RDD code (the transformations and actions) from a Spark SQL (with SQL syntax)? - We do not quite understand why a Spark SQL could trigger multiple jobs, and have some RDDs regenerated, as posted in here. Can anyone give us some insight on the reasons and whether we can avoid the RDD regeneration to save execution time? Thanks in advance. Cheers, Chenghao
Re: Depolying stage-level scheduling for Spark SQL
Thanks for the reply! To clarify, for issue 2, it could still break apart a query into multiple jobs without AQE — I have turned off the AQE in my posted example. For 1, an end user just needs to turn on/off a knob to use the stage-level scheduling for Spark SQL — I am considering adding a component between the Spark SQL module and the Spark Core model to optimize the stage-level resource. Yes, SQL is declarative. It uses a sequence of components (such as a logical planner, physical planner, and CBO) to get a selected physical plan. The RDDs (with the transformations) are generated based on the selected physical plan for execution. For now, we could only get the top-level RDD of the DAG of RDDs by `spark.sql(q1).queryExecution.toRdd`, but it is not enough to make stage-level scheduling decisions. The stage-level resources are profiled based on the RDDs. If we could expose the all RDDs instead of the top-level RDD, it seems possible to apply the stage-level scheduling here. P.S. let me attach the link for the RDD regeneration explicitly in case it is not shown on the mail-list website: https://stackoverflow.com/questions/73895506/how-to-avoid-rdd-regeneration-in-spark-sql Cheers, Chenghao On Sep 29, 2022, 5:22 PM +0200, Herman van Hovell , wrote: > I think issue 2 is caused by adaptive query execution. This will break apart > queries into multiple jobs, each subsequent job will generate a RDD that is > based on previous ones. > > As for 1. I am not sure how much you want to expose to an end user here. SQL > is declarative, and it does not specify how a query should be executed. I can > imagine that you might use different resources for different types of stages, > e.g. a scan stage and more compute heavy stages. This, IMO, should be based > on analysis and costing the plan. For this RDD only stage level scheduling > should be sufficient. > > > On Thu, Sep 29, 2022 at 8:56 AM Chenghao Lyu wrote: > > > Hi, > > > > > > I plan to deploy the stage-level scheduling for Spark SQL to apply some > > > fine-grained optimizations over the DAG of stages. However, I am blocked > > > by the following issues: > > > > > > 1. The current stage-level scheduling supports RDD APIs only. So is there > > > a way to reuse the stage-level scheduling for Spark SQL? E.g., how to > > > expose the RDD code (the transformations and actions) from a Spark SQL > > > (with SQL syntax)? > > > 2. We do not quite understand why a Spark SQL could trigger multiple > > > jobs, and have some RDDs regenerated, as posted in here. Can anyone give > > > us some insight on the reasons and whether we can avoid the RDD > > > regeneration to save execution time? > > > > > > Thanks in advance. > > > > > > Cheers, > > > Chenghao
