[ https://issues.apache.org/jira/browse/SPARK-37055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450191#comment-17450191 ]
dch nguyen commented on SPARK-37055: ------------------------------------ [~hyukjin.kwon] , no, I am not now. I did not find anywhere to apply this conf more :( > Apply 'compute.eager_check' across all the codebase > --------------------------------------------------- > > Key: SPARK-37055 > URL: https://issues.apache.org/jira/browse/SPARK-37055 > Project: Spark > Issue Type: Umbrella > Components: PySpark > Affects Versions: 3.3.0 > Reporter: dch nguyen > Priority: Major > > As [~hyukjin.kwon] guide > 1 Make every input validation like this covered by the new configuration. > For example: > {code:python} > - a == b > + def eager_check(f): # Utility function > + return not config.compute.eager_check and f() > + > + eager_check(lambda: a == b) > {code} > 2 We should check if the output makes sense although the behaviour is not > matched with pandas'. If the output does not make sense, we shouldn't cover > it with this configuration. > 3 Make this configuration enabled by default so we match the behaviour to > pandas' by default. > > We have to make sure listing which API is affected in the description of > 'compute.eager_check' -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org