#general
@deemish2: @deemish2 has joined the channel
@karinwolok1: :wave: Welcome to all the new :wine_glass: Pinot members! We're happy to have you! Can you tell us a little about who you are and what brought you here? :smiley: @yhao @b.gilbert @deemish2 @pablomolnar @alvaradojl1986 @madhu.sling @benjamin.djidi @trustokoroego @carlos @radhika.23796 @suresh.swaminathan69 @knowledgeisstrengthfo @d.chang @jaykhatra21 @mark.frenette @yashrsharma44 @azomshahriar05 @ruslanrodriquez @liranbri @nicolas.richard @frederic.vanharen @saravana6m @hemanth.bm12 @leon.liu @sunny.arora
@mayanks: Welcome all :wave:. Yes, as Karin mentioned, would be great to learn a bit about yourselves.
@yashrsharma44: Hi Pinot Community, Thanks for the welcome! I am an Undergraduate student @ IIT Kharagpur, and an open source enthusiast. I recently discovered Pinot, and wanted to contribute to a real-world project, hence I checked in the slack :grin: Looking forward to learn more about the project :hugging_face:
@frederic.vanharen: Hi, I am an HPC/AI consultant and also working on large data project (product) that will require Pinot (or similar). Looking forward to learn more about Pinot.
@b.gilbert: I work at Fetch Rewards and we are looking to leverage a real time OLAP database with mobile event data (impressions, clicks, etc...) for internal and external data products. We see ~6-8 billion records a month and expect that number to continue to grow.
@joshdnv2: @joshdnv2 has joined the channel
@richballa: @richballa has joined the channel
@joshhighley: I want to confirm this: if I create a table 'accounts' for TenentA, I cannot create another table 'accounts' for TenantB. Is that correct? I would have to create unique names 'accounts_tenantA' and 'accounts_tenantB' ?
@mayanks: Yes, that is correct.
@joshhighley: I would argue this is not multi-tenancy
@joshhighley: it's just pinning of data to specific servers
@mayanks: It is multi-tenant in the sense that multiple tenants (clients) can share the same table and set of resources in Pinot. What it does not provide today is client level isolation.
@joshhighley: having to share the same table is the opposite of multi-tenancy
@joshhighley: ...the table would need a tenant/client id of some sort
@joshhighley: and the tenants data would be mixed in the same table
@mayanks: Quick google search for definition MT yields: ```Multi-Tenant – Multi-tenancy means that a single instance of the software and its supporting infrastructure serves multiple customers. Each customer shares the software application and also shares a single database. Each tenant's data is isolated and remains invisible to other tenants.```
@mayanks: Note single database - you won't have multiple tables with same name in the single database.
@mayanks: But I think we have already established that what you are looking for isn't currently available in Pinot. So debating over MT definition isn't going to help your cause :slightly_smiling_face:
@alnourzarroug: @alnourzarroug has joined the channel
#random
@deemish2: @deemish2 has joined the channel
@joshdnv2: @joshdnv2 has joined the channel
@richballa: @richballa has joined the channel
@alnourzarroug: @alnourzarroug has joined the channel
#feat-upsert
@deemish2: @deemish2 has joined the channel
#troubleshooting
@deemish2: @deemish2 has joined the channel
@jainendra1607tarun: Hello Team, when I set alias in a query, it fails : select c.CustomerID from customer c limit 10 . The error is : [ { "errorCode": 190, "message": "TableDoesNotExistError" } ] While one without alias succeeds: select CustomerID from customer limit 10 Can someone please help understand what is wrong in the query with alias?
@kulbir.nijjer: AFAIK Pinot only supports column alias, table alias is not supported. Tagging @mayanks to confirm.
@jainendra1607tarun: I was going through the dimTables design documentation @
@mayanks: Yes correct
@jainendra1607tarun: @mayanks Do you mean table alias is not supported?
@mayanks: Yes
@jainendra1607tarun: It makes sense if there is no join support as there is no possibility of column name conflict.
@joshdnv2: @joshdnv2 has joined the channel
@richballa: @richballa has joined the channel
@alnourzarroug: @alnourzarroug has joined the channel
#docs
@yashrsharma44: @yashrsharma44 has joined the channel
@yashrsharma44: @yashrsharma44 has left the channel
#pinot-dev
@deemish2: @deemish2 has joined the channel
@trustokoroego: @trustokoroego has joined the channel
@b.gilbert: @b.gilbert has joined the channel
@ken: I just updated my Pinot fork from master, and tried a top-level build with `mvn clean install -DskipTests -Pbin-dist -T 4 -Djdk.version=8`. It failed with `[*ERROR*] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce *(enforce-dependency-convergence)* on project pinot-integration-tests: *Some Enforcer rules have failed.*`
@ken: Have others been able to build from master recently? I’m using Java 8. The enforcer rules that failed (even though they are logged as warnings) seem to be numerous issues with version conflicts, e.g. ```[WARNING] Dependency convergence error for software.amazon.awssdk:netty-nio-client:2.14.28 paths to dependency are: +-org.apache.pinot:pinot-integration-tests:0.8.0-SNAPSHOT +-org.apache.pinot:pinot-tools:0.8.0-SNAPSHOT +-software.amazon.awssdk:s3:2.14.28 +-software.amazon.awssdk:netty-nio-client:2.14.28 and +-org.apache.pinot:pinot-integration-tests:0.8.0-SNAPSHOT +-cloud.localstack:localstack-utils:0.2.11 +-software.amazon.awssdk:dynamodb:2.13.39 +-software.amazon.awssdk:netty-nio-client:2.13.39```
@g.kishore: we should not be depending on aws clients, how did the PR build pass
@ken: Maybe because that dependency is for test code?
@ken: Hmm, seems like all of the dependency issues are because the `pinot-integration-tests` project is pulling in `cloud.localstack:localstack-utils`, and that has slightly different versions for a number of jars it depends on.
@ken: For example: ```Dependency convergence error for software.amazon.awssdk:protocol-core:2.14.28 paths to dependency are: +-org.apache.pinot:pinot-integration-tests:0.8.0-SNAPSHOT +-org.apache.pinot:pinot-tools:0.8.0-SNAPSHOT +-software.amazon.awssdk:s3:2.14.28 +-software.amazon.awssdk:aws-xml-protocol:2.14.28 +-software.amazon.awssdk:aws-query-protocol:2.14.28 +-software.amazon.awssdk:protocol-core:2.14.28 and +-org.apache.pinot:pinot-integration-tests:0.8.0-SNAPSHOT +-org.apache.pinot:pinot-tools:0.8.0-SNAPSHOT +-software.amazon.awssdk:s3:2.14.28 +-software.amazon.awssdk:aws-xml-protocol:2.14.28 +-software.amazon.awssdk:protocol-core:2.14.28 and +-org.apache.pinot:pinot-integration-tests:0.8.0-SNAPSHOT +-org.apache.pinot:pinot-tools:0.8.0-SNAPSHOT +-software.amazon.awssdk:s3:2.14.28 +-software.amazon.awssdk:protocol-core:2.14.28 and +-org.apache.pinot:pinot-integration-tests:0.8.0-SNAPSHOT +-cloud.localstack:localstack-utils:0.2.11 +-software.amazon.awssdk:dynamodb:2.13.39 +-software.amazon.awssdk:protocol-core:2.13.39```
@ken: @mayanks sorry to bug you, any input on this (separate) issue? Thanks!
@ken: In Eclipse I’ve pulled in all of the pinot projects from `master`, and `pinot-segment-local` has an error. The ReplicationUtils class isn’t in the correct package (file says `org.apache.pinot.core.util`, but location in project is `org.apache.pinot.segment.local.utils`. See
@ken: This `pinot-segment-local` sub-project is used by other Pinot sub-projects, otherwise I’d assume it’s vestigial. Or is my source tree somehow messed up?
@mayanks: The `pinot-segment-local` module was a big refactor a month or so ago. I am guessing Eclipse wasn't able to handle the refactor incrementally?
@ken: I had to import that sub-project explicitly, so it wasn’t a refactor in the IDE. Also note that the file in GitHub seems to have the wrong package, given where it’s located in the source tree.
@mayanks: You are right. Wow, how wasn't this caught, and how does the build work now?
@mayanks: Let me file a PR
@mayanks: Apparently, a class can be sitting in a wrong package and can still compile as long as it does not have any local dependencies.
@mayanks:
@ken: I see that you can build Pinot using Java 8, via `mvn clean install -DskipTests -Pbin-dist -T 4 -Djdk.version=8`. Is there any way to generate Eclipse projects that also use Java 8? Currently these project files are created with Java 11 as the target (compliance level, and generated .class files).
#pinot-docs
@yashrsharma44: @yashrsharma44 has joined the channel
#getting-started
@yashrsharma44: @yashrsharma44 has joined the channel
@deemish2: @deemish2 has joined the channel
@trustokoroego: @trustokoroego has joined the channel
@b.gilbert: @b.gilbert has joined the channel
@yhao: @yhao has joined the channel
#feat-partial-upsert
@deemish2: @deemish2 has joined the channel
@chinmay.cerebro: @chinmay.cerebro has joined the channel
@yupeng: @jackie.jxt @xiangfu0 @qiaochu @tingchen @chinmay.cerebro fyi, I added some context on the need of segment reader for partial upsert table
@xiangfu0: @xiangfu0 has joined the channel
@jackie.jxt: So basically leveraging partial upsert to merge the incomplete records?
@jackie.jxt: By directly reading the pinot segment, all the records will be returned, instead of the ones with the latest timestamp. @yupeng Do you think it suits your use case?
@yupeng: can the segment reader support upsert?
@jackie.jxt: No, the upsert is not handled in the reader level
@jackie.jxt: In order to use Pinot features, or push down filters, the table dump should be modeled as a query IMO
@jackie.jxt: We can potentially use the gRPC streaming server to reduce the memory footprint
@yupeng: true, you need to read from server to use the upsert metadata
@yupeng: i wonder if it’s easy to write a Hive ETL to do the compaction: read the latest record of the pk
#debug_upsert
@deemish2: @deemish2 has joined the channel
@deemish2: Hello Everyone
@deemish2: Is there any chance to backfill real time data using apache pinot?
@g.kishore: Can you elaborate?
@g.kishore: Also, ask this in <#C011C9JHN7R|troubleshooting>
#flink-pinot-connector
@deemish2: @deemish2 has joined the channel
#pinot-docsrus
@trustokoroego: @trustokoroego has joined the channel
@yashrsharma44: @yashrsharma44 has joined the channel
@yashrsharma44: Hi, I recently started reading the docs, and wanted to setup the codebase on my local machine. I observed in the docs -
@g.kishore: please file a PR
@mayanks: Yes @yashrsharma44 please do file a PR
@yashrsharma44: Thanks! - I made a PR -
@ramabaratam: @ramabaratam has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
