+kaikua...@gmail.com explicitly. Please check the response. On Thu, Jan 23, 2025 at 10:45 AM Ahmed Abualsaud <ahmedabuals...@apache.org> wrote:
> Hey Luyao! > > Thanks for reaching out. Beam's Iceberg connector supports dynamic > destinations so long as they have equivalent Schemas. There's no support > for dynamic schemas yet, i.e. there's no `getDataSchema(<destination>)` > method. > > There's a feature request on Github: > https://github.com/apache/beam/issues/33724. > > > I don't think Managed.write support with DynamicDestinations > > ManagedIO does support dynamic destinations, in the format described here: > https://cloud.google.com/dataflow/docs/guides/managed-io#dynamic-destinations. > It also does not support dynamic schemas yet. > > Best, > Ahmed > > On 2025/01/23 04:02:57 Ling Li wrote: > > Hi team, > > > > Happy New Year! I have a question regarding > > the "org.apache.beam:beam-sdks-java-io-iceberg:2.61.0". > > > > Does the org.apache.beam.sdk.schemas.Schema getDataSchema(String > > destination) of class DynamicDestinations already exist? > > > > Our team tried to use DynamicDestinations to decide which iceberg table > to > > write in the runtime but failed. > > > > > > Attached is my code class DynamicIcebergDestinations. Our problem is that > > we don't have a universal schema that can match all events to be > > implemented for getDataSchema(). We need to use a parameter to get the > > correct schema for each event, so we want to use: getDataSchema(String > > destination). But seems getDataSchema(String destination) is not > > implemented for org.apache.beam.sdk.io.iceberg.DynamicDestinations. > Since > > getDataSchema() will be automatically called and it cannot return a > > universal schema, there is no way for us to > > use IcebergIO.writeRows(icebergCatalogConfig).to(dynamicDestinations). I > > also tried to use managed I/O connector, but I don't think > > Managed.write support > > with DynamicDestinations. Could you give some urgent guidance? > > > > Thank you so much. > > > > Sincerely, > > Luyao > > >
