[GitHub] [pulsar] gaoran10 commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema for debezium connector

[GitBox]

gaoran10 commented on issue #6034:
URL: https://github.com/apache/pulsar/pull/6034#issuecomment-618533772


   /pulsarbot run-failure-checks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema for debezium connector

[GitBox]

gaoran10 commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema 
for debezium connector
URL: https://github.com/apache/pulsar/pull/6034#issuecomment-615020718
 
 
   I have made some changes, use the JsonConverter as before, and the 
AvroConverter is used differently from the JsonConverter, the pulsar client has 
two ways to use it.
   
   1. The schema is some as below, we could process the GenericRecord. Tip: use 
the KeyValueEncodingType.SEPARATED encoding type.
   `Schema> schema = 
 Schema.KeyValue(Schema.AUTO_CONSUME(), 
Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED);` 
   
   2. We could get the original byte array through the method `getKeyBytes()` 
and `getData()` of the `Message`, the original byte array is converted by the 
AvroConverter.
   
   @tuteng @sijie @jiazhai @codelipenghui Please have a review. Thanks.
   
   @dennisylyung Please have a try. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] gaoran10 commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema for debezium connector

[GitBox]

gaoran10 commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema 
for debezium connector
URL: https://github.com/apache/pulsar/pull/6034#issuecomment-612804157
 
 
   @dennisylyung hi, could you provide some config info about the error logs, 
such as the pulsar source config and the reproduce steps, thanks.
   
   This is my test steps to follow the demo 
http://pulsar.apache.org/docs/en/io-debezium-source/#example-of-mysql, but I 
didn't reproduce your problem.
   
   ```
   # start debezium
   
   docker run -it --rm \
   --name mysql \
   -p 3306:3306 \
   -e MYSQL_ROOT_PASSWORD=debezium \
   -e MYSQL_USER=mysqluser \
   -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql:0.8
   
   # start pulsar
   bin/pulsar standalone
   
   # start source connector yaml
   bin/pulsar-admin source localrun \
   --source-config-file debezium-mysql-source-config.yaml
   
   # subscribe the topic
   bin/pulsar-client consume -s "sub-products" 
public/default/dbserver1.inventory.products -n 0
   
   # start mysql
   docker run -it --rm \
   --name mysqlterm \
   --link mysql \
   --rm mysql:5.7 sh \
   -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" 
-uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services