Hi, there:
The "FLIP-108: Add GPU support in Flink"[1] is now working in
progress. However, we met a problem with
"RuntimeContext#getExternalResourceInfos" if we want to leverage the
Plugin[2] mechanism in Flink.
The interface is:
The problem is now:
public interface RuntimeContext {
/**
Thanks for kicking off this discussion, Yangze.
First, let me try to explain a bit more about this problem. Since we
decided to make the `ExternalResourceDriver` a plugin whose implementation
could be provided by user, we think it makes sense to leverage Flink’s
plugin mechanism and load the drive
Thanks for bringing this up Yangze and Xintong. I see the problem. Help me
to understand how the ExternalResourceInfo is intended to be used by the
user. Will she ask for some properties and then pass them to another
component? Where does this component come from?
Cheers,
Till
On Wed, Apr 29, 202
>
> Will she ask for some properties and then pass them to another component?
Yes. Take GPU as an example, the property needed is "GPU index", and the
index will be used to tell the OS which GPU should be used for the
computing workload.
> Where does this component come from?
The component coul
For your convenience, I modified the Tokenizer in "WordCount"[1] case
to show how UDF leverages GPU info and how we found that problem.
[1]
https://github.com/KarmaGYZ/flink/blob/7c5596e43f6d14c65063ab0917f3c0d4bc0211ed/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/stream
I am also in favor of the option3. Since the Flink FileSystem has the very
similar implementation via plugin mechanism. It has a map "FS_FACTORIES"
to store the plugin-loaded specific FileSystem(e.g. S3, AzureFS, OSS, etc.).
And provide some common interfaces.
Best,
Yang
Yangze Guo 于2020年4月29日周
Thanks for the clarification.
I think you are right that the typed approach does not work with the plugin
mechanism because even if we had the specific ExternalResourceInfo subtype
available one could not cast it into this type because the actual instance
has been loaded by a different class loade
I agree with Till and Xintong, if the ExternalResourceInfo is only a
holder of properties that doesn't have any sublasses it can just become
the "properties" itself.
Aljoscha
On 30.04.20 12:49, Till Rohrmann wrote:
Thanks for the clarification.
I think you are right that the typed approach d
Thanks for the feedback, @Aljoscha and @Till!
Glad to see that we reach a consensus on the third proposal.
Regarding the detail of the `ExternalResourceInfo`, I think Till's
proposal might be good enough. It seems these two methods already
fulfill our requirement and using "Properties" itself mig
