Apache Pinot Daily Email Digest (2022-03-28)

[Pinot Slack Email Digest]

#general@ysuo: pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars  @mayanks: Using `/tmp` is not a good idea@ysuo: for server node, the above properties are used to config local attached disk, right?  @mayanks: Yes, set these to a local attached disk mount.  @ysuo: according to the doc , even if I use S3 as deepstore, a local attached disk is still required for a server node, is it correct?  @ysuo: Got it, thanks, @mayanks  @mayanks: Yes, for OSS Pinot, you need both.  @mayanks: Deep store is for storing copy of data for disaster recovery. Local copy is needed for faster processing.  @ysuo: how much storage should i set for my local copy, is there a resource that i can refer to?  @mayanks: Depends on your workload requirements. You can go up to 1 TB per node   @ysuo: I see, thanks, Mayank.@ysuo: and if i want to use EBS,  i should set two properties referred to EBS?@ysuo: and the following properties are used to set deepstore:@ysuo: pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
8
pinot.server.storage.factory.s3.region=us-west-2
9
pinot.server.segment.fetcher.protocols=file,http,s3
10
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher@ysuo: pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher  @mayanks: Deepstore and local attached disk have different purposes for pinot server (you need both, and not either/or). And yes you will need to set the properties you mentioned.  @mayanks: You can refer to:   @ysuo: thanks@ysuo: If i set retentionTimeUnit to 30 DAYS in a table config, will controller periodic task delete data in S3 deep store which is older than 30 days too?@ysuo: By default, no retention is set. In this case, the data in the server local storage will grow increasingly? And if so, what happens if there is not enough space for local storage?  @mayanks: Yes old data is deleted from deep store as well. You don’t want to run out of storage space. There are metrics to monitor and alert on that @kaushalaggarwal349: @kaushalaggarwal349 has joined the channel@robert.gustafsson1: @robert.gustafsson1 has joined the channel@surajkmth29: Hi Folks,
I'm trying to load test Apache pinot with jmeter, From where can I download the PINOT JDBC driver jar ?
I have included the pinot-jdbc-client-0.9.3.jar in the JMETER path however it's resulting in internal dependency related errors ... Is there a PINOT JDBC Driver jar that i could use directly ?  @mayanks: Please use Pinot Java client not JDBC driver @ysuo: Hi, I’m a little confused about the instance name. If i specify a instanceId for a server, take pinot-server for example, the instance name will be Server_pinot-server_port. So what’s the best practice for setting up a pinot cluster with 3 or more servers using Docker or Kubernetes? Should i set a unique instanceId for each one of them or maybe there is a better way?  @mayanks: Yes, you’ll need to have unique ids for them  @ysuo: Useful to know it.:grin:@ysuo: In my test, some properties can be set like this, *pinot-admin.sh StartController -zkAddress localhost:2191 -controllerPort 9000.* @ysuo: My question is, could every single one of the properties in a config file be set in the above way?  @mayanks: Yes for the ones you need @sboggavarapu: @sboggavarapu has joined the channel@sboggavarapu: :wave: Hi everyone! Nice to be here.@sboggavarapu: So, I am trying to get a table configured for upsert. So, I cannot use star tree index as it says in docs? Does that mean I am only limited to use any other index other than star tree?@sboggavarapu: Also, according to docs, it is not guaranteed that order is preserved in case there is any sort index on any column.@sboggavarapu: ```
When two records of the same primary key are ingested, the record with the greater event time (as defined by the time column) is used. When records with the same primary key and event time, then the order is not determined. In most cases, the later ingested record will be used, but may not be so in the cases when the table has a column to sort by.```@sboggavarapu: Does that mean , I cannot use that index too if I want upsert to be configured for a table?  @mayanks: Yes, StarTree Index is not supported with upsert. But note that startree index helps in case of really low selectivity queries (millions of rows selected by the query. And you can still get great perf from other indexes.  @mayanks: You can use sorted index with upsert.@sboggavarapu: Thanks@kavin.kuppusamy2: @kavin.kuppusamy2 has joined the channel#random@kaushalaggarwal349: @kaushalaggarwal349 has joined the channel@robert.gustafsson1: @robert.gustafsson1 has joined the channel@sboggavarapu: @sboggavarapu

[RESULT][VOTE] Apache Pinot 0.10.0 RC0

[Sajjad Moradi]

Thanks to everyone for validating a release candidate. This vote is now
closed.

Apache Pinot 0.10.0 RC0 has passed with three +1 (binding) votes, two +1
(non-binding) votes, and no 0 or -1 votes.

Binding
+1 Xiang Fu
+1 Subbu Subramaniam
+1 Siddharth Teotia

Non-binding
+1 Richard Startin
+1 Atri Sharma

Thanks,
Sajjad

Re: [VOTE] Apache Pinot 0.10.0 RC0

[Siddharth Teotia]

+1

Verified the release on Mac

From: Atri Sharma 
Sent: Friday, March 25, 2022 8:11 PM
To: dev@pinot.apache.org 
Subject: Re: [VOTE] Apache Pinot 0.10.0 RC0

+1(binding)

On Fri, 25 Mar 2022, 01:31 Richard Startin,  wrote:
+1

- verified sha512 hash
- verified signature
- verified git hash
- verified contents based on git commit hash & the downloaded source code
- verified LICENSE, NOTICE are correctly present
- compiled the downloaded source code
- ran quick start scripts

On Mon, Mar 21, 2022 at 7:01 PM Sajjad Moradi 
mailto:saj...@apache.org>> wrote:
Hi Pinot Community,

This is a call for a vote to release Apache Pinot 0.10.0.

The release candidate:
https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-0.10.0-rc0/

Git tag for this release:
https://github.com/apache/pinot/tree/release-0.10.0-rc0

Git hash for this release:
30c4635bfeee88f88aa9c9f63b93bcd4a650607f

The artifact has been signed with key: 9079294B, which can be found in the 
following KEYS file:
https://dist.apache.org/repos/dist/release/pinot/KEYS

Release notes:
https://github.com/apache/pinot/releases/tag/release-0.10.0-rc0

Staging repository:
https://repository.apache.org/content/repositories/orgapachepinot-1035

Documentation on verifying a release candidate:
https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate

The vote will be open for at least 72 hours or until a necessary number of
votes is reached.

Please vote accordingly,

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason

Thanks,
Apache Pinot team