#general
@karinwolok1: Hey Pinot community! :wine_glass: Please, help us welcome our newest community members! :pinot: :wave: @syedakram93 @photosban @burgalon @charithforex @dwarvenkat @christil @rkruze @kevdesigned @ntchervenski @digvijaybhk @srinivas96alluri @jonathan1841 @j.wise.hunter @rajathlikeslemons @huytc94 @kartikaybagla @riya.tyagi @mustafa @avasudevan :tada: Please, tell us who you are and what brought you here! :smiley:
@srinivas96alluri: Thank you, Karin. Welcome, all!! I am Srinivas, a Data/ML/Platform Engineer. I am working on an interesting use case to store and serve(via Presto) more than 1 TB of new incoming clickstream/track data per hour; I came here to learn more on operational challenges and best practices to use Pinot in the cloud infrastructure.
@huytc94: Thank you, Karin. I am working in a F&B in Vietnam. I know Pinot through some blogs, website and I want to learn more about the data streaming, track data realtime and how we could make decisions after all.
@burgalon: Thank you Karin and hello everyone! I’m considering a possible project for anomaly detection in multi-dimension structured data, and was interested in ThirdEye. One of the use cases is that my data is not necessarily time recurrent, and I was considering whether ThirdEye fits my use cases, or whether I should build a ML model (NN AutoEncoder/XGboost) to be able to find anomalies.
@avasudevan: Thank you, Karin. I am an Engineer in Expedia Group. I am currently evaluating Presto for a PoC for Real time Querying and Analytics. We currently use Datahub as a Metadata Platform and are looking Presto to help us in providing Data Preview and for Querying. Joined the community to see if anyone is using it in combination with Datahub and to learn Pros/Cons.
@pedro.cls93: Hello, Regarding kafka-based streaming ingestion. When does pinot commit offsets to kafka? Is it after creating a segment? Can Pinot be configured to commit offsets only after a segment has been stored in deep storage to ensure no data is lost, in case segments in the server but not in deep storage are deleted?
@g.kishore: That’s the default behavior. Offsets are checkpointed only after segments are committed to deep store
@pedro.cls93: What happens then when deep store is not configured?
@g.kishore: Note, Pinot does not use Kafka for checkpointing. Pinot has its own checkpointing mechanism
@g.kishore: It will always upload the segment to controller and controller will store it in its local directory
@pedro.cls93: Pinot segment checkpoint means controller storing segment in local directory?
@pedro.cls93: If the controller local directory is corrupted or deleted is the checkpoint invalid?
@pedro.cls93: I.e: Is the controller dir a single point of failure?
@g.kishore: No
@g.kishore:
@pedro.cls93: Is there a way to tell Pinot to discard/delete consuming segments and re-read events from Kafka from the last committed offset?
@pedro.cls93: In the case where the consuming segment was deleted from local storage but the metadata about the segment still exists
@mayanks: Did you try to delete the segment using rest-api / swagger?
@pedro.cls93: Not yet, I will try after configuring deep store
@mayanks: Ok
@sumit.m: @sumit.m has joined the channel
@llesoin: @llesoin has joined the channel
@pedro.cls93: Hello, is there documentation explaining the meaning of each configuration property of Pinot's components? I found
@pedro.cls93: cc @mayanks
@mayanks: Yeah, it might be missing some. The one you mentioned is more for temporary directory to do stuff like download local segment (untar, etc)
@pedro.cls93: Are there javadocs or the like in the codebase I could like at for more info?
@mayanks: Javadocs are there but more for interfaces and classes. I think better to update to official docs with whatever is missing
@pedro.cls93: Sure, I meant to use the javadocs to get this configuration documentation info while the docs don’t have it
@aiyer: Hello -- I am trying to load some 100M records into an offline table. At first attemp, it was a simple table with no additional indexes other than what was in the tutorial doc.... that went fine. Now I am trying to add a star tree index on it and the loading is going on for 30+ mins (last time it tokk 12 min)... This is where it is for the last 20 mins... Is there anyway to monitor progress of this ?? ```Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS Creating an executor service with 1 threads(Job parallelism: 0, available cores: 6.) Submitting one Segment Generation Task for file:/opt/pinot/ai/weather/global_weather100M.csv Using class: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader to read segment, ignoring configured file format: AVRO RecordReaderSegmentCreationDataSource is used Finished building StatsCollector! Collected stats for 100000000 documents Created dictionary for INT column: date with cardinality: 30, range: 0 to 29 Using fixed length dictionary for column: country, size: 110 Created dictionary for STRING column: country with cardinality: 10, max length in bytes: 11, range: Australia to USA Created dictionary for INT column: pincode with cardinality: 10, range: 12324 to 3243678 Created dictionary for INT column: week with cardinality: 53, range: 0 to 52 Using fixed length dictionary for column: city, size: 80 Created dictionary for STRING column: city with cardinality: 10, max length in bytes: 8, range: AMD to SRI Created dictionary for INT column: year with cardinality: 50, range: 1970 to 2019 Created dictionary for INT column: temperature with cardinality: 50, range: 0 to 49 Using fixed length dictionary for column: state, size: 20 Created dictionary for STRING column: state with cardinality: 10, max length in bytes: 2, range: AS to WB Using fixed length dictionary for column: day, size: 63 Created dictionary for STRING column: day with cardinality: 7, max length in bytes: 9, range: Friday to Wednesday Created dictionary for LONG column: ts with cardinality: 530768, range: 1620214278776 to 1620214809690 Start building IndexCreator! Finished records indexing in IndexCreator! Finished segment seal! Converting segment: /tmp/pinot-00edd913-441c-4958-8555-9b380f12991b/output/weather_1_OFFLINE_1620214278776_1620214809690_0 to v3 format v3 segment location for segment: weather_1_OFFLINE_1620214278776_1620214809690_0 is /tmp/pinot-00edd913-441c-4958-8555-9b380f12991b/output/weather_1_OFFLINE_1620214278776_1620214809690_0/v3 Deleting files in v1 segment directory: /tmp/pinot-00edd913-441c-4958-8555-9b380f12991b/output/weather_1_OFFLINE_1620214278776_1620214809690_0 Skip creating default columns for segment: weather_1_OFFLINE_1620214278776_1620214809690_0 without schema Successfully loaded segment weather_1_OFFLINE_1620214278776_1620214809690_0 with readMode: mmap Starting building 1 star-trees with configs: [StarTreeV2BuilderConfig[splitOrder=[country, state, city, pincode, day, date, week],skipStarNodeCreation=[],functionColumnPairs=[max__temperature, minMaxRange__temperature, avg__temperature, min__temperature],maxLeafRecords=1000]] using OFF_HEAP builder Starting building star-tree with config: StarTreeV2BuilderConfig[splitOrder=[country, state, city, pincode, day, date, week],skipStarNodeCreation=[],functionColumnPairs=[max__temperature, minMaxRange__temperature, avg__temperature, min__temperature],maxLeafRecords=1000] Generated 65977917 star-tree records from 100000000 segment records```
@ken: I don’t know about monitoring progress, but I have run into slow segment builds when the heap size being used wasn’t big enough.
@mayanks: This ^^. Probably it is gc'ing. Also, can you share the query you are planning to make (and the data size), so we can propose if Startree or some other indexing is better
@shaileshjha061: @shaileshjha061 has joined the channel
@srini: *Congrats to the StarTree team on the announcement!* :pinot:
@ken: Raising that kind of money is very, very challenging, congratulations! Now comes the hard part :)
@mayanks: Hard and Fun part @ken :wink:
@avasudevan: Hello Everyone….A Quick Question as I am reading thru the docs…..Is the Environment separation (Test/Stage/Prod) as well as achieved thru Tenants in Pinot?
@mayanks: If you want to air-gap, then better to use different Pinot clusters (not different tenants between the same clusters).
@mayanks: As tenants may share the same Pinot cluster.
@avasudevan: One more question….How is the schema change of a Table handled?
@mayanks: We support schema evolution as long as backward compatible. You update schema and either reload segments (populates new columns with default values), or rebootstrap
@avasudevan: For non-backward compatible changes I suppose custom scripts to copy from old to new table with conversion might be the solution right?
@mayanks: It would be like new table
@onderaltinkurt: @onderaltinkurt has joined the channel
#random
@sumit.m: @sumit.m has joined the channel
@llesoin: @llesoin has joined the channel
@shaileshjha061: @shaileshjha061 has joined the channel
@onderaltinkurt: @onderaltinkurt has joined the channel
#troubleshooting
@sumit.m: @sumit.m has joined the channel
@llesoin: @llesoin has joined the channel
@shaileshjha061: @shaileshjha061 has joined the channel
@sleepythread: Looks like there is a huge lack of documentations on HDFS integration in pinot, ```[[email protected] ClickGift]$ /local/spark23/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master yarn --deploy-mode client --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dplugins.include=pinot-s3,pinot-parquet,pinot-hdfs -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-shaded.jar" local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar -jobSpecFile batch-job-spec.yml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/lib/pinot-all-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-0.7.1-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/local/spark23/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See
@sleepythread: I am not sure, if i have to add hadoop jar’s again in the Spark classpath.
@fx19880617: For spark, you need to add the jars into classpath or build a big fat jar
@fx19880617: We will refine the doc for sure
@jaydesai.jd: @fx19880617 Can u approve the workflow trigger for PR:
@fx19880617: sure, let me take a look
@jaydesai.jd: Thanks. I think as I haven’t merged a PR yet it asks me to get workflow approval.
@onderaltinkurt: @onderaltinkurt has joined the channel
#pinot-dev
@avasudevan: @avasudevan has joined the channel
#community
@llesoin: @llesoin has joined the channel
#getting-started
@avasudevan: @avasudevan has joined the channel
@avasudevan: Hello Everyone….A Quick Question as I am reading thru the docs…..Is the Environment separation (Test/Stage/Prod) as well as achieved thru Tenants in Pinot?
#releases
@avasudevan: @avasudevan has joined the channel
#pinot-rack-awareness
@jaydesai.jd: Hey @jackie.jxt @g.kishore @ssubrama @rkanumul Can I get a review for
#minion-improvements
@jackie.jxt: Find the root cause:
@jackie.jxt: Currently pinot ignores the empty string within the MV column
@laxman: Thanks @jackie.jxt. This look right fix to me than using NullTransformer. Never comfortable with that. Will test this today and update you.
@laxman: I have a doubt though. In this whole transformation, the information about fields with null values are totally lost. Isn’t that a problem? `pinot row` -> mapper -> `avro record` -> reducer -> `avro record` -> generator -> `pinot row` `red` - indicates input/output record normal - Indicates a process/phase
@g.kishore: there is a bitmap maintained for that
@g.kishore: in the genericRow
@laxman: yes Kishore. i noticed that. that info is lost in above translation (pinot -> avro -> pinot).
@laxman: so, generator is not aware of null fields. which I feel is a bug
@jackie.jxt: @laxman Yes, this is a very good point. I’ll address this issue when doing the enhancement
#complex-type-support
@amrish.k.lal: > JSON_MATCH filter _expression_ to be JSONPath compatible Nice
@steotia: Thanks @jackie.jxt. Will review tomorrow
@amrish.k.lal: Also, this is the PR for JSON column type support:
#fix_llc_segment_upload
@ssubrama: I will correct myself. The cache should be for the segments that are _not_ saved in deep store. Invalidate the cache whenever the controller loses mastership for the table (and when the controller restarts). If the cache is invalidated and we get a call to get the list of bad segments, then fetch it from zk. Remove bad segments from cache as they get repaired.
@tingchen: I agree in general with what @ssubrama has outlined above. The cache stores segment names (from which you can deduce table names). There is still needs to populate the cache by doing scanning of all realtime tables when (1) a controller starts, (2) leadership changes or (3) the cache is empty. The implies only the master controller performs the fix. In the cache, you can keep track of how many times a fix has been tried and abandon the fix when a threshold has reached.
@ssubrama: Not when cache is empty,. Only if the cache is invalidated by one of the other two events.
@ssubrama: In fact, most of the time the cache will be empty.
@ssubrama: We can have a per table cache, so no need to deduce table names from segment names
@tingchen: I do not get why it is not the case when cache is empty. When a controller periodic job starts and it finds the segment name cache is empty, shouldn't it try to scan the realtime tables to check if there is segment required to be fixed?
@changliu: I think it comes to the case when the table with all segments in deep store. The cache is empty for that table all the time. So we should avoid the scan for this kind of tables just because of the cache is empty?
@changliu: But what about the tables which are created after the controller restart or leadership change?
@changliu: If we only do ZK scan during (1) a controller starts, (2) leadership changes , these tables will be ignored.
@changliu: That’s why I think probably we could do this:
@ssubrama: I am not sure I follow. I proposed that we should keep a list of esgments in mem -- segments for which deep store was not available during commit. In case of restart or contrller mastership change. re-fetch the list.
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
