Terry Kim created SPARK-28217: --------------------------------- Summary: Allow a custom statistics logical plan visitor to be plugged in. Key: SPARK-28217 URL: https://issues.apache.org/jira/browse/SPARK-28217 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.0.0 Reporter: Terry Kim
Spark currently has two built-in statistics plan visitor: SizeInBytesOnlyStatsPlanVisitor and BasicStatsPlanVisitor. However, this is a bit limited since there is no way to plug in a custom plan visitor - from which a custom query optimizer can benefit from. We can provide a Spark conf that the user can specify to override the built-in plan visitor: {code:scala} // First create your custom stat plan visitor. class MyStatsPlanVisitor extends LogicalPlanVisitor[Statistics] { // Implement LogicalPlanVisitor[Statistics] trait } // Set the visitor via Spark conf. spark.conf.set("spark.sql.catalyst.statsPlanVisitorClass", "MyStatsPlanVisitor") // Now, stat() on a LogicalPlan object will use MyStatsPlanVisitor as a stat plan visitor. {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org