Dear Hop thinker, I think we should create a new type of metadata to share OAUTH, VFS, FTP, SSH, IMAP, POP3, AS400, SAP... connection settings that we could reuse in different plugins. This new type of metadata object could be called something like "IConnection" with a name, protocol type, and custom properties depending on protocol type (host/login/password, Secret ID/Key, ....), much like databases.
For VFS, the connection name must be the same as the protocol scheme of the URL used. To configure an amazon connection, you must define the predefined name S3 and select the "Amazon Web Services Authentication" protocol, then set authentication the properties. Some web based protocols may have common properties like proxy settings: We could be inspired by: https://www.visualcron.com/doc/html/connection-visualcron.html Maybe it will give some idea. Regards, Nicolas <#m_-6113234865807150246_m_-2555803484072009614_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le mer. 17 févr. 2021 à 11:29, Matt Casters <[email protected]> a écrit : > Dear Hoppiverse, > > Apache VFS is a very useful tool for supporting alternative ways of storing > files. > We have a plugin type for VFS plugins as well as for allowing us to extend > functionality. > Where we're not doing a great job is in supporting specific driver features > like authentication where this is needed. > For example, Amazon AWS needs an access key and a secret key (among a list > of authentication options). > > Usually the configuration of these access APIs underpinning the VFS drivers > is left as standard as possible. > However, this causes it to be dragged outside of the Hop-isphere in the > sense that you have effects like the following: > - variables can't be used. > - obfuscation/encryption of keys and passwords is not done > - No GUI or configuration elements to configure the security > - No metadata wrappers are available or supported > ... > > To get past these shortcomings I propose to turn HopVfs from a singleton > into a proper class. > Class HopVfs is used about 800 times in the source code. It's a wrapper > around the Apache VFS API. > > In the ideal case we would have a new metadata object type called something > like : > "Amazon Web Services Authentication" which would have fields like access > and secret key but also perhaps a checkbox: [x] configured by the system. > We could then have a global configuration option for the S3 VFS driver > which simply says which "AWS Auth" object to use. > > This would address many if not all concerns. The "only" thing we need to > change is moving from: > > HopVfs.getFileObject(filename) > > to > > new HopVfs(metadataProvider, variables).getFileObject(filename) > > Obviously a case can be made to cache the HopVfs objects at various > locations (one per pipeline, workflow, ...). > You get the idea. > > I have a feeling that if we don't do this we'll continue to carry around > this architectural debt for quite a while more into other territories like > Azure and Google. > > Thoughts? > > Matt > -- > Neo4j Chief Solutions Architect > *✉ *[email protected] > ☎ +32486972937 > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#m_-6113234865807150246_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
