#general
@mkorad: @mkorad has joined the channel
#random
@mkorad: @mkorad has joined the channel
#feat-text-search
@amarnath.karthik: @amarnath.karthik has joined the channel
#feat-presto-connector
@amarnath.karthik: @amarnath.karthik has joined the channel
#pql-2-calcite
@amarnath.karthik: @amarnath.karthik has joined the channel
#troubleshooting
@mkorad: @mkorad has joined the channel
@nguyenhoanglam1990: hi team ... Now I see a problem with the realtime table ... every time I count the number of rows of a table, it changes continuously ... do you know what's wrong
@yupeng: coz data flows into pinot from kafka
@nguyenhoanglam1990:
@nguyenhoanglam1990: { "REALTIME": { "tableName": "ERP_ERP_PM_INPUTVOUCHERDETAIL_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "ERP_ERP_PM_INPUTVOUCHERDETAIL", "timeType": "MILLISECONDS", "retentionTimeUnit": "DAYS", "retentionTimeValue": "2", "segmentPushFrequency": "DAILY", "segmentPushType": "APPEND", "timeColumnName": "WARRANTYDATE", "replication": "4", "replicasPerPartition": "4" }, "tenants": { "broker": "DefaultTenant", "server": "inventory" }, "tableIndexConfig": { "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "
@mayanks: Why do you not expect it to change? Do you not have real-time data consumption?
@nguyenhoanglam1990: since the message consuming from kafka doesn't change anymore ... but the table under pinot changes the number of lines continuously
@nguyenhoanglam1990: please help me @yupeng @jackie.jxt @fx19880617
@jackie.jxt: Do you have your primary key partitioned in your Kafka? In order to enable upsert, Kafka topic must be partitioned
@nguyenhoanglam1990: yes
@jackie.jxt: Also, seems the table is not partitioned as well. @yupeng I remember there is a check on the table config to enforce the partitioning?
@nguyenhoanglam1990: I have 1 topic kafka 1 partition ... on pinot I repli it as 4 tables on 4 server nodes
@nguyenhoanglam1990: problem 1 I query on this table the number of rows is missing ... even though I configured replicasPerPartition to be 4 @jackie.jxt
@jackie.jxt: You need to enable partition routing for upsert to work
@jackie.jxt: If there is only 1 Kafka partition, the query have to hit only 1 server
@nguyenhoanglam1990: If there is only 1 Kafka partition, the query have to hit only 1 server ---> yes
@nguyenhoanglam1990: but on 4 server nodes instead of returning the same number of rows ... but here the number of rows is not the same on 4 nodes
@jackie.jxt: Hmm.. this is unexpected
@jackie.jxt: Can you try disabling the upsert with query options and see if the results are the same?
@nguyenhoanglam1990: Can you try disabling the upsert with query options and see if the results are the same? --> yes result same when off upsert
@jackie.jxt: Can you try distinct count on the primary key column?
@jackie.jxt: Also check if there is ERROR logged on the servers
@nguyenhoanglam1990: { "schemaName": "ERP_ERP_PM_INPUTVOUCHERDETAIL", "dimensionFieldSpecs": [ { "name": "INPUTVOUCHERDETAILID", "dataType": "STRING" }, { "name": "INPUTVOUCHERID", "dataType": "STRING" }, { "name": "PRODUCTID", "dataType": "STRING" }, { "name": "QUANTITY", "dataType": "DOUBLE" }, { "name": "PRICE", "dataType": "DOUBLE" }, { "name": "VAT", "dataType": "DOUBLE" }, { "name": "INPUTCUSTOMERID1", "dataType": "DOUBLE" }, { "name": "ISDELETE", "dataType": "DOUBLE" }, { "name": "USERDELETE", "dataType": "STRING" }, { "name": "REPLICATIONSTOREID", "dataType": "DOUBLE" }, { "name": "ISNEW1", "dataType": "DOUBLE" }, { "name": "ISSHOWPRODUCT1", "dataType": "DOUBLE" }, { "name": "FIRSTINPUTTYPEID1", "dataType": "DOUBLE" }, { "name": "BUYINPUTVOUCHERID", "dataType": "STRING" }, { "name": "VATPERCENT", "dataType": "DOUBLE" }, { "name": "BUYPRICE1", "dataType": "DOUBLE" }, { "name": "ISRETURNPRODUCT", "dataType": "DOUBLE" }, { "name": "ISRETURNWITHFEE", "dataType": "DOUBLE" }, { "name": "ISCHECKREALINPUT", "dataType": "DOUBLE" }, { "name": "ADJUSTPRICE", "dataType": "DOUBLE" }, { "name": "ADJUSTPRICECONTENT", "dataType": "STRING" }, { "name": "ADJUSTPRICEUSER", "dataType": "STRING" }, { "name": "ORIGINALPRICE", "dataType": "DOUBLE" }, { "name": "RETURNFEE", "dataType": "DOUBLE" }, { "name": "ISPRODUCTCHANGE", "dataType": "DOUBLE" }, { "name": "ISADDINSTOCK", "dataType": "DOUBLE" }, { "name": "BUYINPUTVOUCHERDETAILID", "dataType": "STRING" }, { "name": "INPUTPRICE", "dataType": "DOUBLE" }, { "name": "ISINPUTCHANGE", "dataType": "DOUBLE" }, { "name": "RETURNPOINTAMOUNT", "dataType": "DOUBLE" }, { "name": "INVENTORYSTATUSID", "dataType": "DOUBLE" }, { "name": "STANDARDPRICE", "dataType": "DOUBLE" }, { "name": "INPUTVOUCHERDETAILSEQ_", "dataType": "DOUBLE" }, { "name": "BUYINPUTVOUCHERSEQ_", "dataType": "DOUBLE" }, { "name": "BUYINPUTVOUCHERDETAILSEQ_", "dataType": "DOUBLE" }, { "name": "INPUTVOUCHERSEQ", "dataType": "DOUBLE" }, { "name": "NOVAT", "dataType": "DOUBLE" }, { "name": "INPUTTYPEID_", "dataType": "DOUBLE" }, { "name": "DISCOUNT", "dataType": "DOUBLE" }, { "name": "SALEPRICE", "dataType": "DOUBLE" }, { "name": "_DELETED", "dataType": "INT" } ], "dateTimeFieldSpecs": [ { "name": "WARRANTYDATE", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "INPUTDATE1", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "DATEDELETE", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "INPUTTIME", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "BUYINVOICEDATE1", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "RETURNDATE", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "PRODUCTIONDATE", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "CHANGETOOLDDATE", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "INPUTCHANGEDATE", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }, { "name": "_TIMESTAMP", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ], "primaryKeyColumns": [ "INPUTVOUCHERDETAILID" ] }
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
