#general
@humengyuk18: Does Pinot support change schema existing column name? I tried change a column name, but got following exceptions on query: ```[ { "errorCode": 500, "message": "MergeResponseError:\nData schema mismatch between merged block: [time_to_hour(LONG),age_decade(STRING),age_level(STRING),city(STRING),company_id(STRING),company_name(STRING),count_impression(LONG),count_in(LONG),count_passby(LONG),create_time(LONG),day(STRING),day_in_week(STRING),district(STRING),gate_id(STRING),gender(STRING),holiday_id(STRING),holiday_name(STRING),hour(STRING),is_holiday(STRING),month(STRING),province(STRING),region(STRING),shop_id(STRING),shop_name(STRING),temperature(STRING),temperature_id(STRING),total_duration(LONG),total_impression_duration(LONG),weather_cate_id(STRING),weather_cate_name(STRING),year(STRING)] and block to merge: [time_to_hour(LONG),age_decade(STRING),age_level(STRING),city(STRING),company_id(STRING),company_name(STRING),count_impression(LONG),count_in(LONG),count_passby(LONG),create_time(LONG),day(STRING),day_in_week(STRING),district(STRING),gate_id(STRING),gender(STRING),holiday_id(STRING),holiday_name(STRING),hour(STRING),is_holiday(STRING),month(STRING),province(STRING),region(STRING),shop_id(STRING),shop_name(STRING),temperature(STRING),temperature_id(STRING),total_duration(LONG),total_impression_duraion(LONG),weather_cate_id(STRING),weather_cate_name(STRING),year(STRING)], drop block to merge" } ]```
@mayanks: Hello, schema evolution is supported as long as it is backward compatible. Changing a column name or type is considered backward incompatible, and is not supported
@humengyuk18: Thanks, so in this case, I should delete all the segment and re-ingest all the data?
@mayanks: Yes, for incompatible schema change, that is the option
@pankaj: If we extend a table schema in Pinot to add new columns (so it does not break backward compatibility); do we have to backfill data or can Pinot use null/default values to handle the older segments?
@mayanks: Pinot can auto fill null/default value in this case
@npawar: Pinot can also fill derived value i.e. if the value of new column is derived from existing columns, Pinot will calculate it using the function you provide
@1705ayush: *How to ingest Data into pinot on kubernetes using native batch ingestion?* Hi, I am trying to ingest csv data into pinot deployed on kubernetes using LaunchDataIngestionJob arg. I have verified that the table has been created on pinot and the job-spec and csv data are present on the node. This is my job-spec file ```apiVersion: batch/v1 kind: Job metadata: name: pinot-case-offline-ingestion namespace: my-pinot-kube spec: template: spec: containers: - name: pinot-load-case-offline image: apachepinot/pinot:0.3.0-SNAPSHOT args: ["LaunchDataIngestionJob", "-jobSpecFile", "/opt/data/table-configs/case_history/job-spec.yml"] volumeMounts: - name: mount-data mountPath: /opt/data restartPolicy: OnFailure volumes: - name: mount-data hostPath: path: /opt/data backoffLimit: 100``` After applying this job to node, nothing happens and this is the log of the pod. ```SegmentGenerationJobSpec: !!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec excludeFileNamePattern: null executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner, segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner, segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner} includeFileNamePattern: glob:**/*.csv inputDirURI: /opt/data/csv_data/case_prod_data jobType: SegmentCreationAndTarPush outputDirURI: /pinot-segments/case_history overwriteOutput: true pinotClusterSpecs: - {controllerURI: '
@fx19880617: I think you are missing pushJobSpec?
@fx19880617: ```pushJobSpec: null```
@1705ayush: Hi @fx19880617, Thank you for helping. I tried adding pushJobSpec to job-spec ```pushJobSpec: pushParallelism: 2 pushAttempts: 2 pushRetryIntervalMillis: 1000``` But the job gets completed with no errors. And the pod log is ```SegmentGenerationJobSpec: !!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec excludeFileNamePattern: null executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner, segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner, segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner} includeFileNamePattern: glob:**/*.csv inputDirURI: /opt/data/csv_data/case_prod_data jobType: SegmentCreationAndTarPush outputDirURI: /pinot-segments/case_history overwriteOutput: true pinotClusterSpecs: - {controllerURI: '
@fx19880617: ok
@fx19880617: what’s the logs for the job?
@1705ayush: Here is the log of the job: ```16:26:48:ayush@:pinot :alien: kubectl -n my-pinot-kube describe jobs.batch pinot-case-offline-ingestion Name: pinot-case-offline-ingestion Namespace: my-pinot-kube Selector: controller-uid=25b4e843-b600-4de2-a2ad-584ac8ce17b5 Labels: controller-uid=25b4e843-b600-4de2-a2ad-584ac8ce17b5 job-name=pinot-case-offline-ingestion Annotations: <none> Parallelism: 1 Completions: 1 Start Time: Fri, 05 Mar 2021 16:26:41 -0500 Completed At: Fri, 05 Mar 2021 16:26:44 -0500 Duration: 3s Pods Statuses: 0 Running / 1 Succeeded / 0 Failed Pod Template: Labels: controller-uid=25b4e843-b600-4de2-a2ad-584ac8ce17b5 job-name=pinot-case-offline-ingestion Containers: pinot-load-case-offline: Image: apachepinot/pinot:0.3.0-SNAPSHOT Port: <none> Host Port: <none> Args: LaunchDataIngestionJob -jobSpecFile /opt/data/table-configs/case_history/job-spec.yml Environment: <none> Mounts: /opt/data from mount-data (rw) Volumes: mount-data: Type: HostPath (bare host directory volume) Path: /opt/data HostPathType: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 27s job-controller Created pod: pinot-case-offline-ingestion-mfvrx Normal Completed 24s job-controller Job completed``` The following is the job spec file to refer. What should be the pinotClusterSpecs.controllerURI value? I tried changing it to anything gibberish and I faced the same logs. I think, my value of pinotClusterSpecs.controllerURI is incorrect. ```executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/opt/data/csv_data/case_prod_data' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '/pinot-segments/case_history' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' configs: delimiter: '|' multiValueDelimiter: '' tableSpec: tableName: 'case_history' pinotClusterSpecs: # - controllerURI: 'pinot-controller:9000' - controllerURI: '
@fx19880617: then are there data on `/opt/data/csv_data/case_prod_data`
@1705ayush: yes. I checked by running a ubuntu container and bashed into it. there is data present on this path
@fx19880617: can you try a newer image as well
@fx19880617: ```apachepinot/pinot:0.6.0```
@fx19880617: 0.3.0 is very old image which I cannot recall the details
@1705ayush: ok. so changed the image. it worked. at the very end of the log it says ```Response for pushing table case_history segment case_history to location
@1705ayush: But, wondering why I cannot query it on the pinot query UI
@1705ayush: there are no records returned from the query select * from case_history limit 10
@fx19880617: hmm
@fx19880617: it should be
@1705ayush: seems, like another issue that I have to look into. But anyways, thank you very much @fx19880617 for you promt responses and help. The new image worked out well.
@fx19880617: can you check pinot server log?
@fx19880617: seems like so
@1705ayush: ok. I do see some errors on pinot-server.
@1705ayush: ```2021/03/05 20:45:00.943 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Starting Pinot server 2021/03/05 20:45:00.944 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Initializing Helix manager with zkAddress: pinot-zookeeper:2181, clusterName: pinot-quickstart, instanceId: Server_pinot-server-0.pinot-server-headless.my-pinot-kube.svc.cluster.local_8098 2021/03/05 20:45:02.560 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Initializing server instance and registering state model factory 2021/03/05 20:45:51.252 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Connecting Helix manager 2021/03/05 20:46:42.537 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Client session timed out, have not heard from server in 31084ms for sessionid 0x0 2021/03/05 20:46:44.353 WARN [ParticipantHealthReportTask] [Start a Pinot [SERVER]] ParticipantHealthReportTimerTask already stopped 2021/03/05 20:47:10.343 WARN [CallbackHandler] [Start a Pinot [SERVER]] Callback handler received event in wrong order. Listener: org.apache.helix.messaging.handling.HelixTaskExecutor@2767bcd8, path: /pinot-quickstart/INSTANCES/Server_pinot-server-0.pinot-server-headless.my-pinot-kube.svc.cluster.local_8098/MESSAGES, expected types: [CALLBACK, FINALIZE] but was INIT 2021/03/05 20:47:11.245 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Instance config for instance: Server_pinot-server-0.pinot-server-headless.my-pinot-kube.svc.cluster.local_8098 has instance tags: [DefaultTenant_OFFLINE, DefaultTenant_REALTIME], host: pinot-server-0.pinot-server-headless.my-pinot-kube.svc.cluster.local, port: 8098, no need to update 2021/03/05 20:47:11.249 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Using class: org.apache.pinot.server.api.access.AllowAllAccessFactory as the AccessControlFactory 2021/03/05 20:47:11.455 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Starting server admin application on:
@1705ayush: I dont know why it is looking for pinot-quickstart configs
@fx19880617: hmm when you start pinot server did you give a clustername?
@1705ayush: I start pinot using helm like ```kubectl create ns my-pinot-kube helm install pinot /home/ayush/spyne/incubator-pinot/kubernetes/helm/pinot -n my-pinot-kube --set replicas=1```
@fx19880617: hmmm
@fx19880617: can you describe the statefulset of pinot-controller and pinot-server and see what's the arguments for that
@1705ayush: ok. All the pinot workers are in running state. I do see these 2 errors on pinot-controller ```WARN [PinotInstanceRestletResource] [grizzly-http-server-1] Admin port is not set for instance: Server_pinot-server-0.pinot-server-headless.my-pinot-kube.svc.cluster.local_8098 ... ...``` ```WARN [PinotInstanceRestletResource] [grizzly-http-server-1] Grpc port is not set for instance: Controller_pinot-controller-0.pinot-controller-headless.my-pinot-kube.svc.cluster.local_9000 ... ...```
@1705ayush: or, I think this could mean something (log on pinot-controller) ```WARN [SegmentStatusChecker] [pool-7-thread-2] Table case_history_OFFLINE has 1 segments with no online replicas WARN [SegmentStatusChecker] [pool-7-thread-2] Table case_history_OFFLINE has 0 replicas, below replication threshold :1```
@fx19880617: this means your controller is up, but no pinot server is connected to the cluster
@fx19880617: i feel something goes wrong with the server setup
@fx19880617: can you try to restart pinot-server pod and see if it's reconnecting?
@1705ayush: yes. restarting the node
@1705ayush: yes. restarting the node worked out! Thank you very much @fx19880617. :pray:
@fx19880617: cool!
@fx19880617: I think the issue is that pinot server pod started before pinot controller which requires setup the zookeeper structure
@fx19880617: so restart should fix it
@1705ayush: yes. whenever I start using helm, zookeeper and controller are the last ones to start and because of that server and broker takes multiple restarts.
#segment-write-api
@npawar: Hey @yupeng , here's the branch i'm working on:
@npawar: i have a basic no-frills file based impl in there. everything is sync and single threaded at the moment.
@npawar: But should be good enough if you want to start trying it out in your flink connector POC
@npawar: if you do try it, lmk if you have any feedback
@yupeng: thanks!
@yupeng: that’s fast
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
