Thanks Masatake for the suggestions.
Yes, Iceberg 1.3.1 supports the component versions which we currently use on bigtop. As per documentation of iceberg, it is recommended to copy the iceberg jars to the respective component directories. If we place jars under /usr/lib/iceberg, we might need to create symlinks in the component rpms to create a version-less symlink in the component directory. If we do not want to place jar/symlink in the component directory, we can use HIVE_AUX_JARS_PATH for hive, We need to check if similar exists for spark and flink. Installing iceberg through ambari can be handled either from ambari component’s metainfo.xml or by adding iceberg in the requires section of spec file of hive, spark, flink. If we plan not to place the iceberg jars in the component lib directories, we need to also check on the stack upgrade scenarios. As iceberg doesnt have any binaries, config files, daemon service, we can try to avoid changes to the bigtop-select. I will start looking on the puppet manifests and smoke tests. Please share your thoughts on the above. Thanks Murali Krishna On Thu, 10 Aug 2023 at 7:16 AM, Masatake Iwasaki <[email protected]> wrote: > > Looking at the packaging perspective the efforts should be minimal. At > > present, Hive and Spark are the only Bigtop components that can support > > iceberg, and only a single jar is required to be packaged into the > > respective component lib directories. > > I guess the versions of Flink, Spark and Hive matters. > https://github.com/apache/iceberg/blob/apache-iceberg-1.3.1/settings.gradle > > If we package the Iceberg, the jars should be installed under > /usr/lib/iceburg. > The jar should be added to the classpaths by settings of depending > products. > We need Puppet manifests for that to deploy testing environment. > Smoke tests should make it certain that the combination works and > the jar in the classpath does not bring conflicting dependencies. > > On 2023/08/09 16:28, Muralikrishna Dmmkr wrote: > > Hi All, > > > > I noticed that there have been various discussions within Bigtop and > Ambari > > communities regarding the addition of Iceberg support, Unfortunately > these > > have not reached a conclusion yet. > > > > So, I am initiating this thread to discuss on the support for iceberg. > > Looking at the packaging perspective the efforts should be minimal. At > > present, Hive and Spark are the only Bigtop components that can support > > iceberg, and only a single jar is required to be packaged into the > > respective component lib directories. > > > > Please share your thoughts on this. > > > > > > Thanks > > Murali Krishna > > >
