jm7647 commented on issue #5454: mysql JDBC Sink - consumer error URL: https://github.com/apache/pulsar/issues/5454#issuecomment-547456603 **Tried to do localrun. Below is result from main broker log.** _(Notice "Closing consumer: 0" - next "Closing consumer: 1" - it will continue in a loop)_ 14:25:12.806 [pulsar-io-50-3] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /135.25.67.53:33764 14:25:12.810 [pulsar-io-50-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/135.25.67.53:33764][persistent://public/default/pulsar-mysql-jdbc-sink-topic] Creating producer. producerId=0 14:25:12.817 [ForkJoinPool.commonPool-worker-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/135.25.67.53:33764] Created new producer: Producer{topic=PersistentTopic{topic=persistent://public/default/pulsar-mysql-jdbc-sink-topic}, client=/135.25.67.53:33764, producerName=standalone-8-51, producerId=0} 14:25:12.818 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [127.0.0.1:3181] is not adhering to Placement Policy. quarantinedBookies: [] 14:25:12.831 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [127.0.0.1:3181] for ledger: 25036 14:25:12.831 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/pulsar-mysql-jdbc-sink-topic] Created new ledger 25036 14:25:12.836 [pulsar-io-50-3] INFO org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://public/default/pulsar-mysql-jdbc-sink-topic}][standalone-8-51] Closing producer on cnx /135.25.67.53:33764 14:25:12.836 [pulsar-io-50-3] INFO org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://public/default/pulsar-mysql-jdbc-sink-topic}][standalone-8-51] Closed producer on cnx /135.25.67.53:33764 14:25:12.844 [pulsar-io-50-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /135.25.67.53:33764 14:25:12.922 [pulsar-io-50-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:42854] Closing consumer: 0 14:25:12.922 [pulsar-io-50-2] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://public/default/pulsar-mysql-jdbc-sink-topic, name=public/default/pulsar-mysql-jdbc-sink}, consumerId=0, consumerName=d4dfc, address=/127.0.0.1:42854} with pending 1 acks 14:25:12.922 [pulsar-io-50-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:42854] Closed consumer Consumer{subscription=PersistentSubscription{topic=persistent://public/default/pulsar-mysql-jdbc-sink-topic, name=public/default/pulsar-mysql-jdbc-sink}, consumerId=0, consumerName=d4dfc, address=/127.0.0.1:42854} 14:25:14.810 [pulsar-io-50-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:42854] Subscribing on topic persistent://public/default/pulsar-mysql-jdbc-sink-topic / public/default/pulsar-mysql-jdbc-sink 14:25:14.813 [ForkJoinPool.commonPool-worker-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/pulsar-mysql-jdbc-sink-topic-public%2Fdefault%2Fpulsar-mysql-jdbc-sink] Rewind from 25036:1 to 25036:0 14:25:14.813 [ForkJoinPool.commonPool-worker-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/pulsar-mysql-jdbc-sink-topic] There are no replicated subscriptions on the topic 14:25:14.813 [ForkJoinPool.commonPool-worker-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/pulsar-mysql-jdbc-sink-topic][public/default/pulsar-mysql-jdbc-sink] Created new subscription for 1 14:25:14.813 [ForkJoinPool.commonPool-worker-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:42854] Created subscription on topic persistent://public/default/pulsar-mysql-jdbc-sink-topic / public/default/pulsar-mysql-jdbc-sink 14:25:14.818 [pulsar-io-50-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:42854] Closing consumer: 1 14:25:14.818 [pulsar-io-50-2] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://public/default/pulsar-mysql-jdbc-sink-topic, name=public/default/pulsar-mysql-jdbc-sink}, consumerId=1, consumerName=03170, address=/127.0.0.1:42854} with pending 1 acks **And here is log from SINK - it is showing same NULL error:** 14:22:15.535 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x1643772d, L:/127.0.0.1:42854 - R:localhost/127.0.0.1:6650]] Connected to server 14:22:15.535 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x1643772d, L:/127.0.0.1:42854 - R:localhost/127.0.0.1:6650] Connected through proxy to target broker at mtn21cvaas04.mt.att.com:6650 14:22:15.537 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [pulsar-mysql-jdbc-sink-topic][public/default/pulsar-mysql-jdbc-sink] Subscribing to topic on cnx [id: 0x1643772d, L:/127.0.0.1:42854 - R:localhost/127.0.0.1:6650] 14:22:15.567 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [pulsar-mysql-jdbc-sink-topic][public/default/pulsar-mysql-jdbc-sink] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0 14:25:12.868 [pulsar-client-io-1-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider 14:25:12.935 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [pulsar-mysql-jdbc-sink-topic] [public/default/pulsar-mysql-jdbc-sink] Closed consumer 14:25:12.943 [main] INFO org.apache.pulsar.functions.LocalRunner - RuntimeSpawner quit because of java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:770) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4153) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147) ~[com.google.guava-guava-21.0.jar:?] at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:94) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:72) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:36) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.client.api.Schema.decode(Schema.java:97) ~[org.apache.pulsar-pulsar-client-api-2.4.1.jar:2.4.1] at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:268) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.source.PulsarRecord.getValue(PulsarRecord.java:74) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.readInput(JavaInstanceRunnable.java:463) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:236) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212] 14:25:14.130 [function-timer-thread-5-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/pulsar-mysql-jdbc-sink-java.lang.NullPointerException Function Container is dead with exception.. restarting 14:25:14.131 [function-timer-thread-5-1] INFO org.apache.pulsar.functions.runtime.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=ebfdf9d9-c659-4ea0-8ec0-4743afdef987, functionVersion=e1c30948-ad78-4aee-a2cd-f245a77256ed, functionDetails=tenant: "public" namespace: "default" name: "pulsar-mysql-jdbc-sink" className: "org.apache.pulsar.functions.api.utils.IdentityFunction" autoAck: true parallelism: 1 source { typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord" inputSpecs { key: "pulsar-mysql-jdbc-sink-topic" value { } } cleanupSubscription: true } sink { className: "org.apache.pulsar.io.jdbc.JdbcAutoSchemaSink" configs: "{\"userName\":\"root\",\"password\":\"jdbc\",\"jdbcUrl\":\"jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink?useSSL\\u003dfalse\",\"tableName\":\"pulsar_mysql_jdbc_sink\"}" typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord" } resources { cpu: 1.0 ram: 1073741824 disk: 10737418240 } componentType: SINK , maxBufferedTuples=1024, functionAuthenticationSpec=null, port=35219, clusterName=local) Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 14:25:14.797 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Configuring schema for topic pulsar-mysql-jdbc-sink-topic : { "name": "public/default/pulsar-mysql-jdbc-sink-topic", "schema": { "type": "record", "name": "Test", "fields": [ { "name": "id", "type": [ "null", "int" ] }, { "name": "name", "type": [ "null", "string" ] } ] }, "type": "AVRO", "properties": {} } 14:25:14.799 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.schema.AutoConsumeSchema - Configure topic schema for topic pulsar-mysql-jdbc-sink-topic : {"type":"record","name":"Test","fields":[{"name":"id","type":["null","int"]},{"name":"name","type":["null","string"]}]} 14:25:14.806 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer perf with config: { "topicNames" : [ "pulsar-mysql-jdbc-sink-topic" ], "topicsPattern" : null, "subscriptionName" : "public/default/pulsar-mysql-jdbc-sink", "subscriptionType" : "Shared", "receiverQueueSize" : 1000, "acknowledgementsGroupTimeMicros" : 100000, "negativeAckRedeliveryDelayMicros" : 60000000, "maxTotalReceiverQueueSizeAcrossPartitions" : 50000, "consumerName" : null, "ackTimeoutMillis" : 0, "tickDurationMillis" : 1000, "priorityLevel" : 0, "cryptoFailureAction" : "CONSUME", "properties" : { "application" : "pulsar-sink", "id" : "public/default/pulsar-mysql-jdbc-sink", "instance_id" : "0" }, "readCompacted" : false, "subscriptionInitialPosition" : "Latest", "patternAutoDiscoveryPeriod" : 1, "regexSubscriptionMode" : "PersistentOnly", "deadLetterPolicy" : null, "autoUpdatePartitions" : true, "replicateSubscriptionState" : false, "resetIncludeHead" : false } 14:25:14.809 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: { "serviceUrl" : "pulsar://localhost:6650", "authPluginClassName" : null, "authParams" : null, "operationTimeoutMs" : 30000, "statsIntervalSeconds" : 60, "numIoThreads" : 1, "numListenerThreads" : 1, "connectionsPerBroker" : 1, "useTcpNoDelay" : true, "useTls" : false, "tlsTrustCertsFilePath" : null, "tlsAllowInsecureConnection" : false, "tlsHostnameVerificationEnable" : false, "concurrentLookupRequest" : 5000, "maxLookupRequest" : 50000, "maxNumberOfRejectedRequestPerConnection" : 50, "keepAliveIntervalSeconds" : 30, "connectionTimeoutMs" : 10000, "requestTimeoutMs" : 60000, "defaultBackoffIntervalNanos" : 100000000, "maxBackoffIntervalNanos" : 30000000000 } 14:25:14.810 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [pulsar-mysql-jdbc-sink-topic][public/default/pulsar-mysql-jdbc-sink] Subscribing to topic on cnx [id: 0x1643772d, L:/127.0.0.1:42854 - R:localhost/127.0.0.1:6650] 14:25:14.814 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [pulsar-mysql-jdbc-sink-topic][public/default/pulsar-mysql-jdbc-sink] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 1 14:25:14.818 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [pulsar-mysql-jdbc-sink-topic] [public/default/pulsar-mysql-jdbc-sink] Closed consumer
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
