Apache Pinot Daily Email Digest (2021-07-08)

Pinot Slack Email Digest Thu, 08 Jul 2021 19:00:36 -0700

#general

@arunkumarc2010: @arunkumarc2010 has joined the channel
@sriramdas.sivasai: Hello every one, I have doubt about the Storage and query part of the pinot. Suppose if we hav 6 months of data as pinot segments in deep storage (size of 500gb) and if i want to make the aggregate query on last 6th months data. 1. does my offline data server should have 500gb memory(RAM) to processs the query ?? or Even with 100gb ram and storage of 500gb, the queries will work efficiently ?? 2. Also does my query work, if i didnt have the storage of 500gb ? 3. memory required for loading segment file from disk is same as the size of the file ? i meant, because of loading the compressed file to memory will blow up the ram 3-4x. Also If want to read the single record from previous 6 months, will it do on demand segment loading from deep storage ??
@mayanks: Servers maintain a local copy of segments on their disk. You do need local disk big enough to store the per server data, but segments are memory mapped, so you don’t need a big RAM, you can do away with 64GB ram for example.
@sriramdas.sivasai: understood. if my query is a groupBY query on 6months dataset. does the lesser memory than the size of whole dataset to process the query works ?
@mayanks: Yes. You don't need large ram to match the data size.
@sriramdas.sivasai: Ok. thanks. just to understand the performance of pinot, all the benchmarks of pinot shown in various places are used memory mapped mode segments from disk or HEAP mode ?
@mayanks: MMAP mode only
@liranbri: Hi everyone, we are evaluating Pinot and one of our requirements is to be able to encrypt our client's data on the disk (in memory it can be decrypted). is such a thing possible? and if so, we may also need to encrypt it with a different encryption key per client (each client's data would be encrypted with a unique key dedicated to that client). is there a way to achieve that? thank you so much
@mayanks: Pinot does support encryption of data copy on deepstore. However, the local server copies on disk need to be decrypted to maintain low latency. The per client encryption requirement is an interesting that I came across in the past and opened an issue to track
@liranbri: It would be a great feature! TBH i’m not sure what you mean by “deepstore”. is that that storage consumed by Pinot, or the source of data owned by us and ingested into Pinot?
@mayanks: Pinot uses deep store to maintain a golden copy of the data ingested. It supports deep stores like S3/ADLS/GCP/etc. That copy can be encrypted.
@mayanks: Pinot servers store a copy of the data on local disk for faster serving (today), that copy does not support encryption.
@liranbri: thanks for the explanation. and those are copies of all the data, or just subsets of it ?
@liranbri: because i’m trying to understand whats the actual value of deep-store encryption, if the same data is decrypted on other disks?
@savingoyal: @savingoyal has joined the channel
@orbit2: @orbit2 has joined the channel
@carlos: ```Hi guys! I have a question regarding Kafka integration with Pinot If I'm using a secured Kafka using SASL_SSL. Is there any way of configuring that and use those credentials? Or there is another way of setting security from Pinot to Kafka for data ingestion? Thanks in advance!```
@xiangfu0: I think we have an issue open for SASL_SSL support:
@carlos: Posted in troubleshooting as well
@rkabir: @rkabir has joined the channel

#random

@arunkumarc2010: @arunkumarc2010 has joined the channel
@savingoyal: @savingoyal has joined the channel
@rkabir: @rkabir has joined the channel