#general
@syedakram93: @syedakram93 has joined the channel
@kautsshukla: Hello, I have 2 nodes each as controller, broker & server under default tenant but under tenants [default tenant] is show only one broker, one server why ?
@kautsshukla: But through api its show 2 ? ```{ "tenants": { "DefaultTenant": [ { "port": 8099, "host": "Broker_1", "instanceName": "Broker_1" }, { "port": 8099, "host": "Broker_1", "instanceName": "Broker_1" } ] }, "tables": {} }```
@fx19880617: hmm, it should be Broker_0, Broker_1?
@fx19880617: why both are Broker_1?
@kautsshukla: I have just masked the IP
@kautsshukla: its two different IP’s
@kautsshukla: Its Broker_0 & Broker_1 only..
@fx19880617: ok, then it’s probably an ui bug, the API will give you the source of truth
@fx19880617:
@fx19880617: I think it’s reported already
@kautsshukla: Ohh, that is I’m using 0.7.1
@kautsshukla: @fx19880617: Thanks..
@photosban: @photosban has joined the channel
@burgalon: @burgalon has joined the channel
@pedro.cls93: Hello, how does Pinot handle processing incomplete messages for realtime tables? If I have the following schema: ```{ "schemaName": "hitexecutionview", "dimensionFieldSpecs": [ { "name": "id", "dataType": "STRING" }, { "name": "jobId", "dataType": "STRING" }, { "name": "crowdMemberId", "dataType": "STRING" }, { "name": "projectId", "dataType": "STRING" }, { "name": "result", "dataTYpe": "STRING" } ], "dateTimeFieldSpecs": [ { "name": "timestamp", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ], "primaryKeyColumns": [ "id" ] }``` And the following kafka message: ```{ "id": "19281-3123n1283-12312-312", "jobId": "245d-2334-fs33-23f4", "crowdMemberId": "xxxxxxxxxx", "projectId": "49mf-f39f-25v2-989m", "timestamp": "1238648237" }``` The field result is not passed, will pinot assume a null value? What happens if there are computed fields based on this non-existing result?
@g.kishore: both result and other fields that are derived from result field will be null
@g.kishore: null --> defaultNullValue
@charithforex: @charithforex has joined the channel
@fx19880617: Want to bring this up here again on the jdk 11 upgrade and drop jdk8 support moving forward. Also want to collect how many users are still on JDK 8/10 and has no plan for JDK 11 upgrade.
@tingchen: Uber is still using Jdk 8 for its Pinot installation. We need to evaluate jdk 11 to check if there is any issue.
@tingchen: thanks for the update.
#random
@syedakram93: @syedakram93 has joined the channel
@photosban: @photosban has joined the channel
@burgalon: @burgalon has joined the channel
@charithforex: @charithforex has joined the channel
#troubleshooting
@phuchdh: what’s difference between `segmentUriPush` and `segmentMetadataPush` ?
@fx19880617:
@phuchdh: oh i see, thank u.
@phuchdh: another question: Example i has a offline table config like this ``` "segmentPartitionConfig": { "columnPartitionMap": { "campaign_id": { "functionName": "Modulo", "numPartitions": 2 } } }``` And the ingest job in spark cluster. So i must build the partition on spark job also, or i dont need pre partition on spark-job ?
@sosyalmedya.oguzhan: Your source files must be partitioned by defined configuration in your table config. For example; raw data (s3) => spark job(analyze data, and partition by campaign_id) => analyzed and partitioned data(s3) => pinot-spark-batch-ingestion-job => segments(s3)
@syedakram93: @syedakram93 has joined the channel
@photosban: @photosban has joined the channel
@burgalon: @burgalon has joined the channel
@burgalon: hi all I’m trying to run pinot thirdeye quickstart and encountering an error `Unrecognized field at: whitelistDatasets` Opened an
@burgalon: moved discussion to thirdeye slack :slightly_smiling_face:
@burgalon: resolved by mounting the config volume with fixes
@mohamed.sultan: I'm just curious to know the issue that after adding the gcs deep store configs in server and controller. The controller external ip (pinot dashboard) says refused to connect. so, I just removed all the gcs deep store config then upgraded the helm chart. It was back to the normal and can able to hit the pinot dashboard. I have pasted the configs gcs configs. Point me if i added something wrong.
@mohamed.sultan:
@mohamed.sultan:
@mohamed.sultan: in addition to that, i have mounted the gcs keys and gcs job spec in broker, controller, server
@mohamed.sultan: i just want to know what is conflicting here
@mayanks: Can you paste the error (or stack)?
@charithforex: @charithforex has joined the channel
@burgalon: i’m encountering a problem with running `docker container exec -it pinot-quickstart bin/generator.sh complexWebsite` pinot just exits, without any error in the log. Trying to follow ThirdEye quickstart
@g.kishore: @slack1 ^^
@burgalon: okay… it seems like it was related to the memory constraints of docker
@burgalon: thank you @g.kishore
@burgalon: Now that I’ve been able to run it, I’m not seeing the store data in thirdeye
@burgalon: seeing these warnings+error: ```Exception in thread "main" org.h2.jdbc.JdbcSQLException: Index "GENERIC_JSON_ENTITY_BEANCLASS_IDX" already exists; SQL statement: create index generic_json_entity_beanclass_idx on generic_json_entity(beanClass) [42111-192] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ... Apr 28, 2021 1:17:20 PM org.glassfish.jersey.internal.Errors logErrors WARNING: The following warnings have been detected: WARNING: Return type, java.util.List<?>, of method, public java.util.List<?> org.apache.pinot.thirdeye.dashboard.resources.v2.DataResource.getWhereNameLike(java.lang.String,java.lang.String), is not resolvable to a concrete type.```
@mayanks: @slack1 Could you take a look ^^
@slack1: Looks like this is blocked on a ThirdEye issue now. @pyne.suvodeep could you check this?
@slack1: My instinct would be to wipe TE’s H2 database files an try re-running
@pyne.suvodeep: @slack1 Yes. I think the initial tables are not coming up properly on h2. How are we running TE? is this on the latest master?
@slack1: @slack1 has joined the channel
#pinot-dev
@fx19880617: Thanks to @elon.azoulay, we have a PR(
@fx19880617: A huge plus :heavy_plus_sign: to the previous statement: Elon and his team has been building and running with Java 11 or a few months - the GC behavior is so much better.
@elon.azoulay: Yep! We never get full gc pauses anymore. There was a marked improvement after we upgraded.
@mayanks: Thanks so much @elon.azoulay
@elon.azoulay: sure:) I'm always happy to contribute, you all help us so much!
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
