[jira] [Created] (FLINK-14398) Codegen produced larger-than-64kb method
Hao Dang created FLINK-14398: Summary: Codegen produced larger-than-64kb method Key: FLINK-14398 URL: https://issues.apache.org/jira/browse/FLINK-14398 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.8.0 Reporter: Hao Dang Attachments: codegen.example.txt In one of our production pipelines, we have a table with 1200+ columns. At runtime, it failed due to a method inside the generated code exceeding 64kb when compiled to bytecode. After we investigated the generated code, it appeared that the *map* method inside a generated *RichMapFunction* was too long. See attached file (codegen.example.txt). In the problematic *map* method, *result setters* were correctly split into individual methods and did not have the largest footprint. However, there were also 1000+ input unboxing expressions inside *reusableInputUnboxingExprs*, which, individually were not trivial and when flattened linearly in the *map* function [here|https://github.com/apache/flink/blob/release-1.8.0/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/codegen/FunctionCodeGenerator.scala#L187], pushed the method size beyond 64kb in bytecode. We think it is worthwhile to split these input unboxing code snippets into individual methods. We were able to verify, in our production environment, that splitting input unboxing code snippets into individual methods resolves the issue. Would love to hear thoughts from the team and find the best path to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14398) Codegen produced larger-than-64kb method
[ https://issues.apache.org/jira/browse/FLINK-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952611#comment-16952611 ] Hao Dang commented on FLINK-14398: -- Indeed, [~twalthr] and [~jark]. Thanks for the quick response. The code in the scenario above was already split based on the existing splitting logic. Yet the sheer number of code snippets produced by *reusableInputUnboxingExprs* took a large footprint in the end, and we probably should split them as well, i.e. putting them into individual methods. > Codegen produced larger-than-64kb method > > > Key: FLINK-14398 > URL: https://issues.apache.org/jira/browse/FLINK-14398 > Project: Flink > Issue Type: Bug > Components: Table SQL / Legacy Planner >Affects Versions: 1.8.0 >Reporter: Hao Dang >Priority: Major > Attachments: codegen.example.txt > > > In one of our production pipelines, we have a table with 1200+ columns. At > runtime, it failed due to a method inside the generated code exceeding 64kb > when compiled to bytecode. > After we investigated the generated code, it appeared that the *map* method > inside a generated *RichMapFunction* was too long. See attached file > (codegen.example.txt). > In the problematic *map* method, *result setters* were correctly split into > individual methods and did not have the largest footprint. > However, there were also 1000+ input unboxing expressions inside > *reusableInputUnboxingExprs*, which, individually were not trivial and when > flattened linearly in the *map* function > [here|https://github.com/apache/flink/blob/release-1.8.0/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/codegen/FunctionCodeGenerator.scala#L187], > pushed the method size beyond 64kb in bytecode. > We think it is worthwhile to split these input unboxing code snippets into > individual methods. We were able to verify, in our production environment, > that splitting input unboxing code snippets into individual methods resolves > the issue. Would love to hear thoughts from the team and find the best path > to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14398) Codegen produced larger-than-64kb method
[ https://issues.apache.org/jira/browse/FLINK-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952676#comment-16952676 ] Hao Dang commented on FLINK-14398: -- Totally agree on a proper design doc for scoping and planning as this is a fundamental piece of the system. We are happy to help in anyway possible to push the fix through, as we will be benefiting from it too. For fixing 1.8.0, would it make sense for us to make a PR to start with? We can go from there for further discussions. I haven't dig into versions newer than 1.8.0, thus cannot say for sure if there will be more work needed for fixing versions post 1.8.0 on top of our existing fix. Also, excuse me for my greenness here: _what's the difference between the blink planner vs. the old planner?_ > Codegen produced larger-than-64kb method > > > Key: FLINK-14398 > URL: https://issues.apache.org/jira/browse/FLINK-14398 > Project: Flink > Issue Type: Bug > Components: Table SQL / Legacy Planner, Table SQL / Planner >Affects Versions: 1.8.0 >Reporter: Hao Dang >Priority: Major > Attachments: codegen.example.txt > > > In one of our production pipelines, we have a table with 1200+ columns. At > runtime, it failed due to a method inside the generated code exceeding 64kb > when compiled to bytecode. > After we investigated the generated code, it appeared that the *map* method > inside a generated *RichMapFunction* was too long. See attached file > (codegen.example.txt). > In the problematic *map* method, *result setters* were correctly split into > individual methods and did not have the largest footprint. > However, there were also 1000+ input unboxing expressions inside > *reusableInputUnboxingExprs*, which, individually were not trivial and when > flattened linearly in the *map* function > [here|https://github.com/apache/flink/blob/release-1.8.0/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/codegen/FunctionCodeGenerator.scala#L187], > pushed the method size beyond 64kb in bytecode. > We think it is worthwhile to split these input unboxing code snippets into > individual methods. We were able to verify, in our production environment, > that splitting input unboxing code snippets into individual methods resolves > the issue. Would love to hear thoughts from the team and find the best path > to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14398) Codegen produced larger-than-64kb method
[ https://issues.apache.org/jira/browse/FLINK-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952747#comment-16952747 ] Hao Dang commented on FLINK-14398: -- Thanks for the explanation! And yea, feel free to assign it to me - happy open a PR for that. > Codegen produced larger-than-64kb method > > > Key: FLINK-14398 > URL: https://issues.apache.org/jira/browse/FLINK-14398 > Project: Flink > Issue Type: Bug > Components: Table SQL / Legacy Planner, Table SQL / Planner >Affects Versions: 1.8.0 >Reporter: Hao Dang >Priority: Major > Attachments: codegen.example.txt > > > In one of our production pipelines, we have a table with 1200+ columns. At > runtime, it failed due to a method inside the generated code exceeding 64kb > when compiled to bytecode. > After we investigated the generated code, it appeared that the *map* method > inside a generated *RichMapFunction* was too long. See attached file > (codegen.example.txt). > In the problematic *map* method, *result setters* were correctly split into > individual methods and did not have the largest footprint. > However, there were also 1000+ input unboxing expressions inside > *reusableInputUnboxingExprs*, which, individually were not trivial and when > flattened linearly in the *map* function > [here|https://github.com/apache/flink/blob/release-1.8.0/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/codegen/FunctionCodeGenerator.scala#L187], > pushed the method size beyond 64kb in bytecode. > We think it is worthwhile to split these input unboxing code snippets into > individual methods. We were able to verify, in our production environment, > that splitting input unboxing code snippets into individual methods resolves > the issue. Would love to hear thoughts from the team and find the best path > to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14398) Codegen produced larger-than-64kb method
[ https://issues.apache.org/jira/browse/FLINK-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952786#comment-16952786 ] Hao Dang commented on FLINK-14398: -- Sounds good, [~twalthr]. I should be able to put up a PR in the next few days. > Codegen produced larger-than-64kb method > > > Key: FLINK-14398 > URL: https://issues.apache.org/jira/browse/FLINK-14398 > Project: Flink > Issue Type: Bug > Components: Table SQL / Legacy Planner, Table SQL / Planner >Affects Versions: 1.8.0 >Reporter: Hao Dang >Assignee: Hao Dang >Priority: Major > Attachments: codegen.example.txt > > > In one of our production pipelines, we have a table with 1200+ columns. At > runtime, it failed due to a method inside the generated code exceeding 64kb > when compiled to bytecode. > After we investigated the generated code, it appeared that the *map* method > inside a generated *RichMapFunction* was too long. See attached file > (codegen.example.txt). > In the problematic *map* method, *result setters* were correctly split into > individual methods and did not have the largest footprint. > However, there were also 1000+ input unboxing expressions inside > *reusableInputUnboxingExprs*, which, individually were not trivial and when > flattened linearly in the *map* function > [here|https://github.com/apache/flink/blob/release-1.8.0/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/codegen/FunctionCodeGenerator.scala#L187], > pushed the method size beyond 64kb in bytecode. > We think it is worthwhile to split these input unboxing code snippets into > individual methods. We were able to verify, in our production environment, > that splitting input unboxing code snippets into individual methods resolves > the issue. Would love to hear thoughts from the team and find the best path > to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-13789) Transactional Id Generation fails due to user code impacting formatting string
Hao Dang created FLINK-13789: Summary: Transactional Id Generation fails due to user code impacting formatting string Key: FLINK-13789 URL: https://issues.apache.org/jira/browse/FLINK-13789 Project: Flink Issue Type: Bug Reporter: Hao Dang In [TransactionalIdsGenerator.java|[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/TransactionalIdsGenerator.java#L94]], prefix contains taskName of the particular task which could ultimately contain user code. In some cases, user code contains conversion specifiers like %, the string formatting could fail. For example, in Flink SQL code, user could have a LIKE statement with a % wildcard, the % wildcard will end up in the prefix and get mistreated during formatting, causing task to fail. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13789) Transactional Id Generation fails due to user code impacting formatting string
[ https://issues.apache.org/jira/browse/FLINK-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Dang updated FLINK-13789: - Fix Version/s: 1.8.0 1.8.1 > Transactional Id Generation fails due to user code impacting formatting string > -- > > Key: FLINK-13789 > URL: https://issues.apache.org/jira/browse/FLINK-13789 > Project: Flink > Issue Type: Bug >Reporter: Hao Dang >Priority: Major > Fix For: 1.8.0, 1.8.1 > > > In [TransactionalIdsGenerator.java|#L94]], prefix contains taskName of the > particular task which could ultimately contain user code. In some cases, > user code contains conversion specifiers like %, the string formatting could > fail. > For example, in Flink SQL code, user could have a LIKE statement with a % > wildcard, the % wildcard will end up in the prefix and get mistreated during > formatting, causing task to fail. > Think we should move prefix out of the string formatting. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13789) Transactional Id Generation fails due to user code impacting formatting string
[ https://issues.apache.org/jira/browse/FLINK-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Dang updated FLINK-13789: - Description: In [TransactionalIdsGenerator.java|#L94]], prefix contains taskName of the particular task which could ultimately contain user code. In some cases, user code contains conversion specifiers like %, the string formatting could fail. For example, in Flink SQL code, user could have a LIKE statement with a % wildcard, the % wildcard will end up in the prefix and get mistreated during formatting, causing task to fail. Think we should move prefix out of the string formatting. was: In [TransactionalIdsGenerator.java|[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/TransactionalIdsGenerator.java#L94]], prefix contains taskName of the particular task which could ultimately contain user code. In some cases, user code contains conversion specifiers like %, the string formatting could fail. For example, in Flink SQL code, user could have a LIKE statement with a % wildcard, the % wildcard will end up in the prefix and get mistreated during formatting, causing task to fail. > Transactional Id Generation fails due to user code impacting formatting string > -- > > Key: FLINK-13789 > URL: https://issues.apache.org/jira/browse/FLINK-13789 > Project: Flink > Issue Type: Bug >Reporter: Hao Dang >Priority: Major > > In [TransactionalIdsGenerator.java|#L94]], prefix contains taskName of the > particular task which could ultimately contain user code. In some cases, > user code contains conversion specifiers like %, the string formatting could > fail. > For example, in Flink SQL code, user could have a LIKE statement with a % > wildcard, the % wildcard will end up in the prefix and get mistreated during > formatting, causing task to fail. > Think we should move prefix out of the string formatting. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13789) Transactional Id Generation fails due to user code impacting formatting string
[ https://issues.apache.org/jira/browse/FLINK-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Dang updated FLINK-13789: - Fix Version/s: 1.7.2 > Transactional Id Generation fails due to user code impacting formatting string > -- > > Key: FLINK-13789 > URL: https://issues.apache.org/jira/browse/FLINK-13789 > Project: Flink > Issue Type: Bug >Reporter: Hao Dang >Priority: Major > Fix For: 1.7.2, 1.8.0, 1.8.1 > > > In [TransactionalIdsGenerator.java|#L94]], prefix contains taskName of the > particular task which could ultimately contain user code. In some cases, > user code contains conversion specifiers like %, the string formatting could > fail. > For example, in Flink SQL code, user could have a LIKE statement with a % > wildcard, the % wildcard will end up in the prefix and get mistreated during > formatting, causing task to fail. > Think we should move prefix out of the string formatting. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-6966) Add maxParallelism and UIDs to all operators generated by the Table API / SQL
[ https://issues.apache.org/jira/browse/FLINK-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767618#comment-16767618 ] Hao Dang commented on FLINK-6966: - Hi [~hequn8128], Understand this work didn't start as of last September, so want to check in and see if there is any news or plan to implement this feature? Thanks! > Add maxParallelism and UIDs to all operators generated by the Table API / SQL > - > > Key: FLINK-6966 > URL: https://issues.apache.org/jira/browse/FLINK-6966 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.4.0 >Reporter: Fabian Hueske >Priority: Major > > At the moment, the Table API does not assign UIDs and the max parallelism to > operators (except for operators with parallelism 1). > We should do that to avoid problems when rescaling or restarting jobs from > savepoints. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
