[GitHub] [spark] ChenMichael edited a comment on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-23 Thread GitBox
ChenMichael edited a comment on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976849576 In order for this problem to manifest, we have to do join planning in between the time an InMemoryRelation is converted to a RDD and the time where the job executing

[GitHub] [spark] ChenMichael edited a comment on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-23 Thread GitBox
ChenMichael edited a comment on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976849576 In order for this problem to manifest, we have to do join planning in between the time an InMemoryRelation is converted to a rdd and the time where the job executing

[GitHub] [spark] ChenMichael edited a comment on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-23 Thread GitBox
ChenMichael edited a comment on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976849576 In order for this problem to manifest, we have to do join planning in between the time an InMemoryRelation is converted to a rdd and the time where the job executing

[GitHub] [spark] ChenMichael edited a comment on pull request #34684: [SPARK-37442][SQL] InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

2021-11-23 Thread GitBox
ChenMichael edited a comment on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-976849576 In order for this problem to manifest, we have to do join planning between the time a InMemoryRelation is converted to an rdd and the time where the job executing thi