[
https://issues.apache.org/jira/browse/FLINK-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934109#comment-15934109
]
Luke Hutchison edited comment on FLINK-6114 at 3/21/17 5:18 AM:
----------------------------------------------------------------
Hi Greg, the code that caused the exception was doing the following, if memory
serves correctly:
{code}
public static <K> DataSet<Tuple3<String, K, Float>> sumReduceLastValue(
DataSet<Tuple3<String, K, Float>> input) {
return input.groupBy(0).sum(2);
}
{code}
although I realize now that this leaves field 1 in an indeterminate state: if
not all fields of a tuple are covered in the {{groupBy}} and {{sum}}
parameters, and if different input tuples have different values for the
unspecified fields, then aggregation has to either pick one of the field 1
values, or pick none of them (e.g. set field 1 to null).
I assume that this is what caused the exception, because the typechecker was
not expecting any fields to be dropped like this -- but if so, the correct
exception message would be something along the lines of {{"sum aggregation on
field 2 after grouping by field 0 leaves field 1 undefined"}} (or similar).
was (Author: lukehutch):
Hi Greg, the code that caused the exception was doing the following, if memory
serves correctly:
{code}
public static <K> DataSet<Tuple3<String, K, Float>> doSomething(
DataSet<Tuple3<String, K, Float>> input) {
return input.groupBy(0).sum(2);
}
{code}
although I realize now that this leaves field 1 in an indeterminate state: if
not all fields of a tuple are covered in the {{groupBy}} and {{sum}}
parameters, and if different input tuples have different values for the
unspecified fields, then aggregation has to either pick one of the field 1
values, or pick none of them (e.g. set field 1 to null).
I assume that this is what caused the exception, because the typechecker was
not expecting any fields to be dropped like this -- but if so, the correct
exception message would be something along the lines of {{"sum aggregation on
field 2 after grouping by field 0 leaves field 1 undefined"}} (or similar).
> Type checking fails with generics, even when concrete type of field is not
> needed
> ---------------------------------------------------------------------------------
>
> Key: FLINK-6114
> URL: https://issues.apache.org/jira/browse/FLINK-6114
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0
> Reporter: Luke Hutchison
>
> The Flink type checker does not allow generic types to be used in any field
> of a tuple when a join is being executed, even if the generic is not in a
> field that is involved in the join.
> I have a type Tuple3<String, K, Float>, which contains a generic type
> parameter K. I am joining using .where(0).equalTo(0). The type of field 0 is
> well-defined as String. However, this gives me the following error:
> {noformat}
> Exception in thread "main"
> org.apache.flink.api.common.functions.InvalidTypesException: Type of
> TypeVariable 'K' in 'public static org.apache.flink.api.java.DataSet
> mypkg.MyClass.method(params)' could not be determined. This is most likely a
> type erasure problem. The type extraction currently supports types with
> generic variables only in cases where all variables in the return type can be
> deduced from the input type(s).
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createSubTypesInfo(TypeExtractor.java:989)
> {noformat}
> The code compiles fine, however -- the static type system is able to
> correctly resolve the types in the surrounding code.
> Really only the fields that are affected by joins (or groupBy, aggregation
> etc.) should be checked for concrete types in this way.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
