[ https://issues.apache.org/jira/browse/SPARK-37055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dch nguyen updated SPARK-37055: ------------------------------- Description: As [~hyukjin.kwon] guide 1 Make every input validation like this covered by the new configuration. For example: {code:python} - a == b + def eager_check(f): # Utility function + return not config.compute.eager_check and f() + + eager_check(lambda: a == b) {code} 2 We should check if the output makes sense although the behaviour is not matched with pandas'. If the output does not make sense, we shouldn't cover it with this configuration. 3 Make this configuration enabled by default so we match the behaviour to pandas' by default. We have to make sure listing which API is affected in the description of 'compute.eager_check' > Apply 'compute.eager_check' across all the codebase > --------------------------------------------------- > > Key: SPARK-37055 > URL: https://issues.apache.org/jira/browse/SPARK-37055 > Project: Spark > Issue Type: Umbrella > Components: PySpark > Affects Versions: 3.3.0 > Reporter: dch nguyen > Priority: Major > > As [~hyukjin.kwon] guide > 1 Make every input validation like this covered by the new configuration. > For example: > {code:python} > - a == b > + def eager_check(f): # Utility function > + return not config.compute.eager_check and f() > + > + eager_check(lambda: a == b) > {code} > 2 We should check if the output makes sense although the behaviour is not > matched with pandas'. If the output does not make sense, we shouldn't cover > it with this configuration. > 3 Make this configuration enabled by default so we match the behaviour to > pandas' by default. > > We have to make sure listing which API is affected in the description of > 'compute.eager_check' -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org