[
https://issues.apache.org/jira/browse/FLINK-41105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130799#comment-18130799
]
zhanglu153 commented on FLINK-41105:
------------------------------------
h3. Steps to reproduce
# MySQL source table:
CREATE TABLE app_db.orders (
id INT NOT NULL PRIMARY KEY,
price DECIMAL(10,2) NOT NULL
);
INSERT INTO app_db.orders VALUES (1, 4.00), (2, 100.00); # Pipeline YAML (the
standard documented syntax — no tricks):
source:
type: mysql
hostname: ${hostname}
port: 3306
username: ${user}
password: ${pwd}
tables: app_db.orders
server-id: 7003
server-time-zone: Asia/Shanghai
transform:
- source-table: app_db.orders
projection: id, price, price * 1.13 AS price_with_tax
route:
- source-table: app_db.orders
sink-table: app_db.orders_out
sink:
type: doris
fenodes: ${fe}:8030
... # Submit the job and let the snapshot phase emit the first
{{DataChangeEvent}}.
h3. Expected behavior
The official Arithmetic Functions table documents:
||Function||Janino Code||Description||
|{{numeric1 * numeric2}}|{{numeric1 * numeric2}}|Returns NUMERIC1 multiplied by
NUMERIC2|
Operands are only required to be _numeric_ — no type restriction, no CAST
requirement. The 3.1 release announcement example itself multiplies numeric
columns. So {{price * 1.13}} should work out of the box (and it does in ISO SQL
systems such as MySQL).
h3. Actual behavior
*Two-phase contradiction* (this is the most telling part):
# {*}Schema derivation phase succeeds{*}: Calcite derives {{price_with_tax
DECIMAL(13,4)}} (standard numeric widening: DECIMAL(10,2) × DECIMAL(3,2) →
DECIMAL(13,4)) and the sink table is created with that column. The product's
own type system fully accepts the expression.
# *Data path phase fails on the first event* and the job restarts in an
infinite loop (13 failures within 5 minutes in our test; 0 rows ever reach the
sink):
org.apache.flink.cdc.runtime.operators.transform.TransformException:
Failed to post-transform with DataChangeEvent\{tableId=app_db.orders, ...
op=INSERT ...}
at PostTransformOperator.processElement(...)
Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to compile
expression
TransformExpressionKey{
originalExpression='`TB`.`price` * 1.13',
compiledExpression='times($0, 1.13)',
argumentClasses=[class java.math.BigDecimal, ...]}
Caused by: org.apache.flink.api.common.InvalidProgramException:
Expression cannot be compiled. This is a bug. Please file an issue.
Caused by: org.codehaus.commons.compiler.CompileException:
No applicable constructor/method found for actual parameters
"java.math.BigDecimal, double"; candidates are:
times(double,double), times(float,float), times(BigDecimal,BigDecimal),
times(byte,byte), times(short,short), times(int,int), times(long,long)
The same expression is accepted by the schema phase and rejected by the data
path — an internal inconsistency within the same product.
h3. Root cause (source-level)
Two code locations conspire:
# *{{JaninoCompiler.translateSqlSqlLiteral}} (L170-193)* — numeric literal
translation only emits the raw {{toString()}} text. Integers get a dedicated
branch (append {{L}} when exceeding int), but for decimal literals the
BigDecimal identity is silently dropped: Calcite internally holds {{1.13}} as
{{BigDecimal}}, and after translation Janino parses it as {{double}}.
# *{{JaninoCompiler.generateBinaryOperation}} (L332-350) +
{{ArithmeticFunctions}} (L85-111)* — when the derived result type is DECIMAL,
the binary operation is routed to a static function call {{times($0, 1.13)}};
but the {{times/plus/minus/divides}} overloads are all same-type pairs, so
{{(BigDecimal, double)}} has no match.
{*}Scope is wider than it first appears{*}: integer literals are affected too
({{price * 2}} → {{times($0, 2)}} = {{(BigDecimal, int)}}, also no overload),
and so are *INT/BIGINT columns multiplied by decimal literals* ({{cnt * 1.5}}
derives DECIMAL and routes to {{times(Integer, 1.5)}}). Effectively, *any
binary arithmetic whose derived result type is DECIMAL and whose operands do
not strictly match a same-type overload signature fails to compile* — i.e. all
four operators {{+ - * /}}, for both DECIMAL-column×literal and
integer-column×decimal-literal combinations.
h3. Suggested fix (implemented and validated)
Inside the DECIMAL-result branch of {{generateBinaryOperation}}, align each
operand to BigDecimal.
Two details that matter:
* Decimal literals *must be constructed from the string form* ({{new
BigDecimal("1.13")}}), never {{new BigDecimal(1.13)}} — the double constructor
introduces binary floating-point error (forbidden by the Java standard API
docs).
* Non-DECIMAL columns reuse the existing {{castToBigDecimal}} (same semantics
as {{CastingFunctions}}), no new machinery.
This is a one-file change ({{JaninoCompiler.java}}, ~+67 lines), touches no
existing overload sets and no non-DECIMAL code paths.
h3. Validation results (4 rounds, same environment / same source table)
||Round||Jar||Projection||Compile failures||Restarts||Sink data||
|Reproduce|community 3.6.0|{{price * 1.13}} (bare)|13 in 5 min|infinite loop|0
rows|
|Workaround|community 3.6.0|{{price * CAST(1.13 AS DECIMAL(10,2))}}|0|0|4.5200
/ 113.0000 (exact)|
|Fix|patched|{{price * 1.13}} (bare)|0|0|4.5200 / 113.0000 ({*}identical to
CAST path → no precision loss{*})|
|Regression|patched|6 computed columns covering all expression families of the
routing table: {{score*score}} (DOUBLE×DOUBLE, native operator path),
{{score*1.13}} (implicit promotion), {{cnt+1}} (INT+INT), {{cnt*1.5}}
(INT×decimal literal, fixed path), {{price*1.13}}, {{price*2}}|0|0|all columns
exact (6.25 / 2.825 / 11 / 15.0 / 4.5200 / 8.00)|
The regression round confirms the fix does not break any pre-existing normal
path (native-operator and implicit-promotion expressions all still compile and
produce exact values), while covering the whole fixed family including the
INT-column × decimal-literal case.
h3. Workaround (until the fix ships)
# works, data lands exactly:
projection: id, price, price * CAST(1.13 AS DECIMAL(10,2)) AS price_with_tax
# compiles but silently corrupts data — bare DECIMAL defaults to (10,0),
# 1.13 is HALF_UP-rounded to 1:
projection: id, price, price * CAST(1.13 AS DECIMAL) AS price_with_tax
The CAST requirement itself is the proof of the bug: the documentation promises
bare {{numeric1 * numeric2}}, and in fact the schema phase accepts it — only
the runtime compiler rejects it.
> transform projection: arithmetic between DECIMAL column and numeric literal
> fails at Janino compile time — times(BigDecimal, double) has no matching
> overload
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41105
> URL: https://issues.apache.org/jira/browse/FLINK-41105
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.6.0
> Environment: - Flink CDC 3.6.0 (pipeline mode, MySQL → Doris)
> - Source column: MySQL `DECIMAL(10,2)` → Flink CDC `BigDecimal`
> - `ArithmeticFunctions.times/plus/minus/divides` still only have
> same-type-pair overloads
> Reporter: zhanglu153
> Priority: Critical
> Labels: Flink-CDC
>
> When a CDC pipeline transform projection contains an arithmetic computed
> column between a *DECIMAL column and a numeric literal* (e.g. {{price *
> 1.13}}, a pattern explicitly promised by the official Arithmetic Functions
> table and used in the 3.1 release announcement example {{weight / (height *
> height) as bmi}}), the job passes schema derivation and table creation, but
> fails at runtime on the *first DataChangeEvent* with a Janino
> {{CompileException}}: no matching overload for {{(BigDecimal, double)}}. The
> job then restarts forever and can never complete. The product's own error
> message says: _"Expression cannot be compiled. This is a bug. Please file an
> issue."_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
