I think I found the root cause of the issue: Although initially the table is created as a TestTable, when setting up the task writer in the Flink test (in TestDeltaTaskWriter) the table is turned into a SerializableTable. This doesn't look correct to me because the metadataLocation will always be null this way and an UnsupportedOperationException is thrown. These tests didn't hit this issue before, because the table is loaded lazily (and it wasn't accessed in a way to hit this initialization step before). Passing in the TestTable directly turns the tests green. Is my assumption correct, that a TestTable shouldn't be converted to a SerializableTable because of the issue mentioned before?
On Fri, Feb 24, 2023 at 5:21 PM Ryan Blue <[email protected]> wrote: > Switching to HadoopTables wouldn't hurt, although if you're having trouble > with TestTables, then it could easily be that the state of the helper is > being cleared. I think you're probably better off trying to figure out why > the table is missing or was deleted, rather than switching. > > On Fri, Feb 24, 2023 at 2:55 AM Tamas Sule <[email protected]> > wrote: > >> Hi Ryan, >> >> Thanks for the reply. I'm working on some new features in the Flink >> connector >> and the changes caused a lot of tests to fail (for example >> in TestDeltaTaskWriter) >> because these test classes use TestTables. As you said that this is an >> internal >> helper class, I assume the Flink connector should not rely on it at all. >> Some other >> tests use HadoopTables in Flink, which work fine. Would it be the correct >> approach >> to refactor the failing Flink test to use tables created using >> HadoopTables? >> >> Thanks, >> Tamas >> >> On 2023/02/23 19:38:17 Ryan Blue wrote: >> > Tamas, >> > >> > What are you trying to do? TestTables is intended to be an internal >> helper >> > class that manages some tables in memory and shouldn't actually be used >> > outside of Iceberg or the tests that currently rely on it. >> > >> > Ryan >> > >> > On Thu, Feb 23, 2023 at 8:55 AM Tamas Sule <[email protected]> >> > wrote: >> > >> > > Hello, I’m having trouble working with the TestTables class: I’m >> writing a >> > > test which hits a code path that uses IcebergGenerics.read, which >> fails >> > > with the following error: Cannot load metadata: metadata file >> location is >> > > null. Looking at the TestTableOperations class, it doesn’t actually >> write >> > > out the metadata json to the filesystem but rather keeps it in >> memory. Is >> > > this a shortcoming of these test classes at the moment, or am I >> supposed to >> > > use something else to test this? Thanks! >> > > >> > >> > >> > -- >> > Ryan Blue >> > Tabular >> > >> > > > -- > Ryan Blue > Tabular >
