I agree. It can be implemented with usage of plugins. But if I need to enable and configurate the plugin I need to put this information into dt-site.xml. It means The plugin and its parameter must be documented and the list of the added specific jars will be visible and available for updates to the end-user. The implementation via plugins is more dynamic solution that is more convenient for the application developers. But I'm talking about the static configuration of the Apex build or installation that relates more to the platform development.
The current Apex core implementation uses the static unchanged list of jars for long time, because the Apex implementation still contains several basic static assumptions (for instance, the usage of YARN, HDSF, etc.). And the current Apex assumptions are hardcoded in the implementation. But if we are going to improve Apex and use Java interfaces in generic Apex implementation, the current static approach in Apex code to hardcode a list of dependent jars will not work anymore. It will require to include a new solution to add/change jars in specific Apex builds/configurations. And I don't think the usage of the plugins will be good for that. Thanks, Sergey On Thu, Feb 1, 2018 at 1:47 PM, Vlad Rozov <vro...@apache.org> wrote: > There is a way to get the same end result by using plugins. It will be > good to understand why plugin can't be used and can they be extended to > provide the required functionality. > > Thank you, > > Vlad > > > On 1/29/18 15:14, Sergey Golovko wrote: > >> Hello All, >> >> In Apex there are two ways to deploy non-Hadoop jars to the deployed >> cluster. >> >> The first approach is static (hardcoded) and it is used by Apex platform >> developers only. There are several final static arrays of Java classes >> in StramClient.java >> that define which of the available jars should be included into deployment >> for every Apex application. >> >> The second approach is to add paths of all dependent jar-files to the >> value >> of the attribute LIB_JARS. The end-user can set/update the value of the >> attribute LIB_JARS via dt-site.xml files, command line parameters, >> application properties and plugins. The usage of the >> attribute LIB_JARS is the official documented way for all Apex users to >> manage by the deployment jars. >> >> But some of the dependent jars (not from the Apex core) can be common for >> all customer's applications for a specific installation and/or execution >> environment. Unfortunately the Apex implementation does not contain the >> middle solution that would allow the Apex developers and customer support >> to >> define and add new dependent jar-files (jars that should not be >> configurable/managed by the end-user) without the updates/recompilation of >> the Apex Java code during the Apex building process and/or >> installation/configuration. >> >> Also the having of such kind of flexibility would allow the Apex core >> developers to use Java interfaces during the development to define an >> abstraction layer in Apex implementation and configurate Apex core to add >> some specific jars to all Apex applications without recompilation of the >> Apex source code. >> >> For instance, now the usage of HDFS is hardcoded in Apex platform code but >> it can be replaced with any other distributed or cloud base file system. >> The Apex core code can use an interface for all I/O operations but the >> supporting of a real specific file system implementation can be added as >> an >> independent jar-file. Or if the implementation of some of Apex operators >> depend on a specific service, and it is necessary to add some of the >> service jars to every Apex application implicitly. >> >> The proposal: >> >> - add a predefined configuration text file (we can make any choice for the >> file syntax: XML, JSON or Properties) to Apex engine resources with >> predefined values of some of the Apex attributes (now we can include >> LIB_JARS >> attribute only); >> - allow to have a configuration text file with the same functionality in >> the Apex installation folder "conf"; >> - read the content of the predefined configuration text files by the stram >> client in runtime and add the jars to the list of the dependent jars; >> - allow to use paths to jars and Java classes to refer to the dependent >> jars (the references can have the extensions: .class and .jar). >> >> Thanks, >> Sergey >> >> >
