[ https://issues.apache.org/jira/browse/SPARK-37811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tsarangi updated SPARK-37811: ----------------------------- Description: Following query throws HintLoggerError Warnings in v3.1.2. {code:java} // code placeholder val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " + " L1.v1 AS L1V1 " + " L4.* " + " FROM L1 " + " INNER JOIN L2 ON L2.id = L1.id " + " INNER JOIN L3 ON L3.id = L1.id " + " LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time BETWEEN L3.time1 AND L3.time2 " {code} Following is the warning it thorws during runtime: {code:java} WARN HintErrorLogger: Count not find relation 'L1' specified in hint 'BROADCASTJOIN(L1,L2,L3)' WARN HintErrorLogger: Count not find relation 'L2' specified in hint 'BROADCASTJOIN(L1,L2,L3)' WARN HintErrorLogger: Count not find relation 'L3' specified in hint 'BROADCASTJOIN(L1,L2,L3)'{code} The same query didn't have any warnings in v2.4.7. I am not entirely sure if this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}}) when doing a Left Join with a bigger table (L4). I have set \{{autoBroadcastJoinThreshold = 4G }} which is way bigger than L1+L2+L3. Let me know if you need more info. was: Following query throws HintLoggerError Warnings in v3.1.2. {code:java} // code placeholder val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " + " FROM L1 " + " INNER JOIN L2 ON L2.id = L1.id " + " INNER JOIN L3 ON L3.id = L1.id " + " LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time BETWEEN L3.time1 AND L3.time2 " {code} Following is the warning it thorws during runtime: {code:java} WARN HintErrorLogger: Count not find relation 'L1' specified in hint 'BROADCASTJOIN(L1,L2,L3)' WARN HintErrorLogger: Count not find relation 'L2' specified in hint 'BROADCASTJOIN(L1,L2,L3)' WARN HintErrorLogger: Count not find relation 'L3' specified in hint 'BROADCASTJOIN(L1,L2,L3)'{code} The same query didn't have any warnings in v2.4.7. I am not entirely sure if this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}}) when doing a Left Join with a bigger table (L4). I have set {{autoBroadcastJoinThreshold = 4G }} which is way bigger than L1+L2+L3. Let me know if you need more info. > Broadcast Join throws HintErrorLogger for joins with multiple tables > -------------------------------------------------------------------- > > Key: SPARK-37811 > URL: https://issues.apache.org/jira/browse/SPARK-37811 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.2 > Reporter: tsarangi > Priority: Minor > Labels: HINTS, JOIN, SQL, broadcast > > Following query throws HintLoggerError Warnings in v3.1.2. > > {code:java} > // code placeholder > val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " + > " L1.v1 AS L1V1 " + > " L4.* " + > " FROM L1 " + > " INNER JOIN L2 ON L2.id = L1.id " + > " INNER JOIN L3 ON L3.id = L1.id " + > " LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time > BETWEEN L3.time1 AND L3.time2 " > {code} > > Following is the warning it thorws during runtime: > {code:java} > WARN HintErrorLogger: Count not find relation 'L1' specified in hint > 'BROADCASTJOIN(L1,L2,L3)' > WARN HintErrorLogger: Count not find relation 'L2' specified in hint > 'BROADCASTJOIN(L1,L2,L3)' > WARN HintErrorLogger: Count not find relation 'L3' specified in hint > 'BROADCASTJOIN(L1,L2,L3)'{code} > > The same query didn't have any warnings in v2.4.7. I am not entirely sure if > this is inherently not broadcasting the three small tables ({{{}L1, L2, > L3{}}}) when doing a Left Join with a bigger table (L4). > I have set \{{autoBroadcastJoinThreshold = 4G }} which is way bigger than > L1+L2+L3. > > Let me know if you need more info. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org