#general
@kavya8428: @kavya8428 has joined the channel
@sulthon.poetra.ahmad: @sulthon.poetra.ahmad has joined the channel
@karinwolok1: Today! with presenters @steotia and @guo.980 :heart_eyes:
@snlee: Hello Community, We are pleased to announce that Apache Pinot 0.8.0 is released! Apache Pinot is a distributed columnar storage engine that can ingest data in real-time and serve analytical queries at low latency. The release can be downloaded at
@karinwolok1: If there's any .NET + Pinot devs out there that want to share their story - .Net conf has it's open CFP and accepting talk submissions! :slightly_smiling_face:
@karinwolok1: For anyone who missed the meetup today
@fritzb: @snlee where can we find the docker image for release-0.8.0 ? I can't seem to pull `apachepinot/pinot:release-0.8.0`
@xiangfu0: I will do it
@jai.patel856: what are the mechanisms to throttle the speed at which messages are ingested via realtime?
#random
@kavya8428: @kavya8428 has joined the channel
@sulthon.poetra.ahmad: @sulthon.poetra.ahmad has joined the channel
#troubleshooting
@kangren.chia: so far from what i’ve seen there are some vague references to default cpu/memory sizing on the docs:
@kavya8428: @kavya8428 has joined the channel
@lovenishgoyal: Hi All, Can anyone help me to connect tableau to pinot ? we are getting below error
@xiangfu0: which version are you running ? have you tried 0.8.0?
@xiangfu0: @kharekartik fyi
@lovenishgoyal: pinot is running on 0.7.0
@lovenishgoyal: for pinot-jdbc we tried 0.7.0 and 0.8.0 both
@xiangfu0: hmm i think it’s missing http client lib
@xiangfu0: can you try to download and put it
@xiangfu0:
@lovenishgoyal: Sure, let me try this
@sulthon.poetra.ahmad: @sulthon.poetra.ahmad has joined the channel
@syedakram93: WARNING: All illegal access operations will be denied in a future release Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.htrace.HTraceConfiguration org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(org.apache.hadoop.conf.Configuration)' at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:635) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:65) at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52) at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72) at org.apache.pinot.controller.BaseControllerStarter.initPinotFSFactory(BaseControllerStarter.java:518) at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:358) at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:308) at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:123) at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:93) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:233) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:285) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:232) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:182) at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:149) at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166) at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186) at org.apache.pinot.tools.admin.PinotController.main(PinotController.java:35)
@syedakram93: iam getting above exception while starting up controller using hdfs as deepstore
@syedakram93: export HADOOP_HOME=/home/sas/zohohadoop/hdfs export HADOOP_VERSION=2.8.5 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 #export GC_LOG_LOCATION=/path/to/gc/log/file export PINOT_VERSION=0.8.0 export PINOT_DISTRIBUTION_DIR=/home/sas/pinot/ export SERVER_CONF_DIR=/home/sas/pinot/conf/ export ZOOKEEPER_ADDRESS=localhost:2191 export CLASSPATH_PREFIX="${HADOOP_HOME}/lib/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/lib/gson-${HADOOP_GSON_VERSION}.jar:${HADOOP_HOME}/lib/htrace-core4-4.0.1-incubating.jar" export JAVA_OPTS="-Xms4G -Xmx4G" ${PINOT_DISTRIBUTION_DIR}/bin/start-controller.sh -configFileName ${SERVER_CONF_DIR}/controller.conf
@syedakram93: using above config
@syedakram93: can someone help?
@syedakram93:
@xiangfu0: try out a lower hadoop version? the doc is using 2.7.1
@xiangfu0: seems there is method signature change in 2.8.x
@syedakram93: @xiangfu0 @npawar @mayanks
@will.gan: Hi, if I want to have a time column that's essentially another column (say epoch_minutes) but bucketed (say every 5 minutes), my understanding was that I can create a column with format `1:MINUTES:EPOCH` and granularity `5:MINUTES` , and that Pinot would handle it for me. I don't think that's the case though, i.e. I have to write an ingestion transform right?
@npawar: you have to write transform function
@will.gan: ok thanks, but out of curiosity if that's the case couldn't I just make my format `5:MINUTES:EPOCH` ? How does granularity differ?
@npawar: `5:MINUTES:EPOCH` would mean, `epoch_millis/(1000*60*5)` whereas, `1:MINUTES:EPOCH` with granularity `5:MINUTES` would mean, `epoch_millis/(1000*60)`, with value rounded to the nearest 5 minutes
@npawar: the former is not a very natural way of representing. The latter is more commonly used
@will.gan: okay
#pinot-dev
@luisfernandez: @luisfernandez has joined the channel
@s.swaminathan: @s.swaminathan has joined the channel
@art.artik: @art.artik has joined the channel
#getting-started
@s.swaminathan: @s.swaminathan has joined the channel
@art.artik: @art.artik has joined the channel
@tiger: Is there a way to enable/disable individual segments?
@kulbir.nijjer: @tiger can you elaborate on what’s the use case ?
@tiger: I want to batch ingest data on a per day basis, but only enable the most recent day once it has finished ingesting. Each day would end up consisting of many segments. Is there a suggested method of doing this?
@kulbir.nijjer: Hmm there is no such API to enable/disable individual segments, only deleting a segment or multiple(JSON array) if you have a list. Just curious why table only needs to have data for most recent day and not others ? If older days data is not needed ever then simply configure the `retention*` settings in the table appropriately:
@mayanks: @tiger the recommended way of doing so is to have time filter in the query to explicitly select the time range. You can additional set retention to auto purge/delete older segments.
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
