[ https://issues.apache.org/jira/browse/FLINK-32760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dongwoo.kim updated FLINK-32760: -------------------------------- Description: h2. Summary In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading parquet dependency from *hive-exec* is done. But I think this is not enough and causing errors like below when I try to read parquet file using sql-gateway which requires both *flink-parquet* and *flink-sql-connector-hive* dependencies. !image-2023-08-05-14-50-47-806.png|width=1392,height=909! h2. {color:#172b4d}Cause{color} {color:#172b4d}Parquet dependency not only includes *org.apache.parquet* but also *shaded.parquet* prefix dependencies. ([ref|https://github.com/apache/parquet-mr/blob/515734c373f69b5250e8b63eb3d1c973da893b63/pom.xml#L72]){color} {color:#172b4d}So we need to shade both.{color} {color:#172b4d}- flink-parquet depends on Parquet 1.12.3 with shaded Thrift 0.16.0 (prefix: {{{}shaded.parquet{}}}){color} {color:#172b4d}- flink-sql-connector-hive depends on hive-exec 3.1.3 with Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: {{{}shaded.parquet{}}}){color} {color:#172b4d}- Code compiled against Thrift 0.16.0 attempts to run against 0.9.3, causing the error.{color} h2. {color:#172b4d}Proposed solution{color} Adding new shading rule to flink-sql-connector-hive project. I have confirmed that if we add below rule, the above error is resolved. {code:xml} <relocation> <pattern>shaded.parquet</pattern> <shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern> </relocation>{code} I would be happy to implement it if the proposal is accepted. Thanks was: h2. Summary In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading parquet dependency from *hive-exec* is done. But I think this is not enough and causing errors like below when I try to read parquet file using sql-gateway which requires both *flink-parquet* and *flink-sql-connector-hive* dependencies. !image-2023-08-05-14-50-47-806.png|width=1392,height=909! h2. {color:#172b4d}Cause{color} {color:#172b4d}Parquet dependency not only includes org.apache.parquet but also shaded.parquet prefix dependencies. ([ref|https://github.com/apache/parquet-mr/blob/515734c373f69b5250e8b63eb3d1c973da893b63/pom.xml#L72]){color} {color:#172b4d}- flink-parquet depends on Parquet 1.12.3 with shaded Thrift 0.16.0 (prefix: {{{}shaded.parquet{}}}){color} {color:#172b4d}- flink-sql-connector-hive depends on hive-exec 3.1.3 with Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: {{{}shaded.parquet{}}}){color} {color:#172b4d}- Code compiled against Thrift 0.16.0 attempts to run against 0.9.3, causing the error.{color} h2. {color:#172b4d}Proposed solution{color} Adding new shading rule to flink-sql-connector-hive project. I have confirmed that if we add below rule, the above error is resolved. {code:xml} <relocation> <pattern>shaded.parquet</pattern> <shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern> </relocation>{code} I would be happy to implement it if the proposal is accepted. Thanks > Version Conflict in flink-sql-connector-hive for shaded.parquet prefix > packages > ------------------------------------------------------------------------------- > > Key: FLINK-32760 > URL: https://issues.apache.org/jira/browse/FLINK-32760 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive > Reporter: dongwoo.kim > Priority: Major > Attachments: image-2023-08-05-14-50-47-806.png > > > h2. Summary > In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading > parquet dependency from *hive-exec* is done. > But I think this is not enough and causing errors like below when I try to > read parquet file using sql-gateway which requires both *flink-parquet* and > *flink-sql-connector-hive* dependencies. > !image-2023-08-05-14-50-47-806.png|width=1392,height=909! > > h2. {color:#172b4d}Cause{color} > {color:#172b4d}Parquet dependency not only includes *org.apache.parquet* but > also *shaded.parquet* prefix dependencies. > ([ref|https://github.com/apache/parquet-mr/blob/515734c373f69b5250e8b63eb3d1c973da893b63/pom.xml#L72]){color} > {color:#172b4d}So we need to shade both.{color} > {color:#172b4d}- flink-parquet depends on Parquet 1.12.3 with shaded Thrift > 0.16.0 (prefix: {{{}shaded.parquet{}}}){color} > {color:#172b4d}- flink-sql-connector-hive depends on hive-exec 3.1.3 with > Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: {{{}shaded.parquet{}}}){color} > {color:#172b4d}- Code compiled against Thrift 0.16.0 attempts to run against > 0.9.3, causing the error.{color} > h2. {color:#172b4d}Proposed solution{color} > Adding new shading rule to flink-sql-connector-hive project. > I have confirmed that if we add below rule, the above error is resolved. > {code:xml} > <relocation> > <pattern>shaded.parquet</pattern> > <shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern> > </relocation>{code} > > I would be happy to implement it if the proposal is accepted. Thanks > -- This message was sent by Atlassian Jira (v8.20.10#820010)