Is there any way to get the ExecutionConfigurations in Dynamic factory class
Hi I have developed a BigQuery Flink connector by implementing DynamicTableSourceFactory. I have a requirement to : get the configured parallelism value of StreamExecutionEnvironment in the Factory class. or set the parallelism at Factory class or Table source class level. Please help me on this. Thanks and Regards, Anitha Thankappan -- _This message contains information that may be privileged or confidential and is the property of the Quantiphi Inc and/or its affiliates_. It is intended only for the person to whom it is addressed. _If you are not the intended recipient, any review, dissemination, distribution, copying, storage or other use of all or any portion of this message is strictly prohibited. If you received this message in error, please immediately notify the sender by reply e-mail and delete this message in its *entirety*___
Re: Is there any way to get the ExecutionConfigurations in Dynamic factory class
Hi Anitha, AFAIK DynamicTableSourceFactory doesn’t expose interface for getting parallelism. Could you elaborate on why you need parallelism in table factory? Maybe we could find other ways to fulfill your requirement. Best regards, Qingsheng > On Apr 7, 2022, at 16:11, Anitha Thankappan > wrote: > > Hi > > I have developed a BigQuery Flink connector by implementing > DynamicTableSourceFactory. > I have a requirement to : >get the configured parallelism value of > StreamExecutionEnvironment in the Factory class. >or >set the parallelism at Factory class or Table source > class level. > Please help me on this. > > > Thanks and Regards, > Anitha Thankappan > > This message contains information that may be privileged or confidential and > is the property of the Quantiphi Inc and/or its affiliates. It is intended > only for the person to whom it is addressed. If you are not the intended > recipient, any review, dissemination, distribution, copying, storage or other > use of all or any portion of this message is strictly prohibited. If you > received this message in error, please immediately notify the sender by reply > e-mail and delete this message in its entirety
Re: Is there any way to get the ExecutionConfigurations in Dynamic factory class
Hi Qingsheng, I am creating BigQuery read sessions in the factory class level. I want to set the number of Streams in the read session using the parallelism set at Execution environment. This is to execute each stream by each executor to achieve parallel execution of streams. Thanks and Regards, Anitha Thankappan On Fri, Apr 8, 2022 at 11:53 AM Qingsheng Ren wrote: > Hi Anitha, > > AFAIK DynamicTableSourceFactory doesn’t expose interface for getting > parallelism. Could you elaborate on why you need parallelism in table > factory? Maybe we could find other ways to fulfill your requirement. > > Best regards, > > Qingsheng > > > On Apr 7, 2022, at 16:11, Anitha Thankappan < > anitha.thankap...@quantiphi.com> wrote: > > > > Hi > > > > I have developed a BigQuery Flink connector by implementing > DynamicTableSourceFactory. > > I have a requirement to : > >get the configured parallelism value of > StreamExecutionEnvironment in the Factory class. > >or > >set the parallelism at Factory class or Table > source class level. > > Please help me on this. > > > > > > Thanks and Regards, > > Anitha Thankappan > > > > This message contains information that may be privileged or confidential > and is the property of the Quantiphi Inc and/or its affiliates. It is > intended only for the person to whom it is addressed. If you are not the > intended recipient, any review, dissemination, distribution, copying, > storage or other use of all or any portion of this message is strictly > prohibited. If you received this message in error, please immediately > notify the sender by reply e-mail and delete this message in its entirety > > -- _This message contains information that may be privileged or confidential and is the property of the Quantiphi Inc and/or its affiliates_. It is intended only for the person to whom it is addressed. _If you are not the intended recipient, any review, dissemination, distribution, copying, storage or other use of all or any portion of this message is strictly prohibited. If you received this message in error, please immediately notify the sender by reply e-mail and delete this message in its *entirety*___
Re: Is there any way to get the ExecutionConfigurations in Dynamic factory class
Hi Anitha, I’m not quite familiar with BigQuery. Is it possible to postpone the operation to SplitEnumerator, in which you could get parallelism by org.apache.flink.api.connector.source.SplitEnumeratorContext#currentParallelism? If you are using SourceFunction you can get access to current parallelism with org.apache.flink.api.common.functions.RuntimeContext#getNumberOfParallelSubtasks, but we strongly suggest to use Source instead of SourceFunction. Cheers, Qingsheng > On Apr 11, 2022, at 13:58, Anitha Thankappan > wrote: > > Hi Qingsheng, > > I am creating BigQuery read sessions in the factory class level. I want to > set the number of Streams in the read session using the parallelism set at > Execution environment. > This is to execute each stream by each executor to achieve parallel execution > of streams. > > > Thanks and Regards, > Anitha Thankappan > > > On Fri, Apr 8, 2022 at 11:53 AM Qingsheng Ren wrote: > Hi Anitha, > > AFAIK DynamicTableSourceFactory doesn’t expose interface for getting > parallelism. Could you elaborate on why you need parallelism in table > factory? Maybe we could find other ways to fulfill your requirement. > > Best regards, > > Qingsheng > > > On Apr 7, 2022, at 16:11, Anitha Thankappan > > wrote: > > > > Hi > > > > I have developed a BigQuery Flink connector by implementing > > DynamicTableSourceFactory. > > I have a requirement to : > >get the configured parallelism value of > > StreamExecutionEnvironment in the Factory class. > >or > >set the parallelism at Factory class or Table source > > class level. > > Please help me on this. > > > > > > Thanks and Regards, > > Anitha Thankappan > > > > This message contains information that may be privileged or confidential > > and is the property of the Quantiphi Inc and/or its affiliates. It is > > intended only for the person to whom it is addressed. If you are not the > > intended recipient, any review, dissemination, distribution, copying, > > storage or other use of all or any portion of this message is strictly > > prohibited. If you received this message in error, please immediately > > notify the sender by reply e-mail and delete this message in its entirety > > > This message contains information that may be privileged or confidential and > is the property of the Quantiphi Inc and/or its affiliates. It is intended > only for the person to whom it is addressed. If you are not the intended > recipient, any review, dissemination, distribution, copying, storage or other > use of all or any portion of this message is strictly prohibited. If you > received this message in error, please immediately notify the sender by reply > e-mail and delete this message in its entirety
