gaogaotiantian commented on PR #57911: URL: https://github.com/apache/spark/pull/57911#issuecomment-5290395183
Something does not add up for the benchmark. You did not update the micro benchmark after changing the boolean part right? It does not quite make sense for `array<array<string>>, 100% int (coercion, worst realistic)` to be faster in E2E test if it's slower in micro benchmark. The numbers for other E2E tests are a bit too ideal too. I can believe the number if the `is True` thing improves the micro benchmark so even the worse case is slightly better. Could you try the micro benchmark with the bool change again? Also the benchmark is `spark.createDataFrame`, which means there's no UDF or other encoding involved right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
