Hi Martijn, thanks for your reply :) I thought of looking into contributing this too. I am not super deep into the source code of Flink yet and it might be a little out of scope for the bachelor thesis I am doing, so I have to see if I can find the time at the moment. But generally, I would be interested (since obviously it would help my thesis). Maybe you can tell how much effort it would be? I imagine it would need support in the place where the watermarks are registered (the one I sent) and in the place they are actually used (which I have not checked yet at all).
-Theo > On 14. Dec 2022, at 09:23, Theodor Wübker <theo.wueb...@inside-m2m.de> wrote: > > Actually, this behaviour is documented > <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-table> > (See the Watermarks section, where it is stated that the column must be a > “top-level” column). So I suppose, there is a reason. Nevertheless it is > quite a limiting factor, since it makes me unable to use Window queries with > the desired timestamp as watermark … I suppose one workaround could be to > transform the table so the attribute is at top level and then use it as a > watermark. But in my case this would be quite the effort. > > My question remains, what is the reason for this behaviour? Also, are there > any good workarounds for this? > > Thanks, > > -Theo > >> On 14. Dec 2022, at 08:13, Theodor Wübker <theo.wueb...@inside-m2m.de >> <mailto:theo.wueb...@inside-m2m.de>> wrote: >> >> Hey everyone, >> >> I have encountered a problem with my Table API Program. I am trying to use a >> nested attribute as a watermark. The structure of my schema is a row, which >> itself has 3 rows as attributes and they again have some attributes, >> especially the Timestamp that I want to use as a watermark. Flink does not >> let me reference it using the dot operator, sadly. I checked the sourcecode >> and tracked it down to this (part of a) method in the DefaultSchemaResolver: >> >> private Column validateTimeColumn(String columnName, List<Column> columns) { >> final Optional<Column> timeColumn = >> columns.stream().filter(c -> >> c.getName().equals(columnName)).findFirst(); >> if (!timeColumn.isPresent()) { >> throw new ValidationException( >> String.format( >> "Invalid column name '%s' for rowtime attribute in >> watermark declaration. Available columns are: %s", >> columnName, >> >> columns.stream().map(Column::getName).collect(Collectors.toList()))); >> } ... >> The list of available columns is just the 3 rows and none of the nested >> attributes. Is there a reason for nested columns being unavailable for >> watermark declaration? Or am I overseeing something / doing something wrong? >> >> -Theo >
smime.p7s
Description: S/MIME cryptographic signature
