#general
@michsre: @michsre has joined the channel
@leon.bam: @leon.bam has joined the channel
@srirams.ganesh: @srirams.ganesh has joined the channel
@madhan.r: @madhan.r has joined the channel
#random
@michsre: @michsre has joined the channel
@leon.bam: @leon.bam has joined the channel
@srirams.ganesh: @srirams.ganesh has joined the channel
@madhan.r: @madhan.r has joined the channel
#feat-text-search
@leon.bam: @leon.bam has joined the channel
#troubleshooting
@michsre: @michsre has joined the channel
@carlos: Hi folks, is there any way of using environment variables in the Pinot Ingestion config. I’ve checked this PR
@ken: Environment variables are set in the OS, e.g. for Linux something like `export MYVARNAME=some_value` , or for Windows maybe `SET MYVARNAME=some_value` (I think).
@leon.bam: @leon.bam has joined the channel
@anu110195: how to check timezone of pinot from dashboard ?
@mayanks: Not sure if I follow. Pinot doesn’t have any time zone of itself. For data time stamp you can query With date time udfs
@srirams.ganesh: @srirams.ganesh has joined the channel
@madhan.r: @madhan.r has joined the channel
#pinot-dev
@andruszd: @andruszd has joined the channel
@richard892: @richard892 has joined the channel
@debabrata: @debabrata has joined the channel
@ken: @elon.azoulay - I’m wondering how thoroughly the classloader change (for Java9+ compatibility) was tested with Hadoop/Spark segment building code, when running with Java 8. Asking because we (and others) have run into class-not-found issues (for plugins) when trying to build segments with 0.8.0 using Hadoop and Spark on Java 8, and I see this change to `PluginClassLoader.java` that I think you made: ```< import java.lang.reflect.Method; 39,47d37 < Method method = null; < try { < method = URLClassLoader.class.getDeclaredMethod("addURL", URL.class); < < } catch (NoSuchMethodException e) { < //this should never happen < ExceptionUtils.rethrow(e); < } < method.setAccessible(true); 50c40,47 < method.invoke(classLoader, url); --- > /** > * ClassLoader in java9+ does not extend URLClassLoader. > * If the class is not found in the parent classloader, > * it will be found in this classloader via findClass(). > * > * @see
@ken: Sorry, edited to add Java 8 being used when running the job
@elon.azoulay: We don't use spark for ingestion here (use direct insert from trino - have an open pr) so I did not test it. Can you create a github issue - I would be happy to help.
@ken: Hi @elon.azoulay - thanks for any help. Issue is at
@g.kishore: yes, this needs to change with all the changes that happened in jvm
@ken: Hi @elon.azoulay @g.kishore I built a version of 0.8.0 that reverted the change to `PluginClassLoader.java`, and the Hadoop job now runs w/o errors. So it seems pretty clear this was the change that broke things for Hadoop/Spark jobs. So…we could add a Java version check (kind of gross), or is there some approach that’s compatible with both Java 8 and Java 9+?
@g.kishore: I remember breaking my head trying to get it to work with various versions without a version check… if you can find something, happy to review
@xiangfu0: hmm, then shell we consider do multi-version releases for this class?
@xiangfu0:
@ken: I’m not clear why Java 8 requires the old code (which uses reflection), as for both Java 8 and Java 9 the URLClassloader.addUrl() method is protected. So I would expect @elon.azoulay’s version to work fine for both, but it clearly doesn’t.
@ken: I guess to confirm that the new code in `PluginClassLoader.java` actually works w/Hadoop, we’d need to set up to run with Java 9 on our Hadoop cluster, which isn’t likely to happen any time soon.
#announcements
@leon.bam: @leon.bam has joined the channel
#getting-started
@andruszd: @andruszd has joined the channel
@richard892: @richard892 has joined the channel
@leon.bam: @leon.bam has joined the channel
@debabrata: @debabrata has joined the channel
#pinot-trino
@leon.bam: @leon.bam has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
