Hi, Thanks for your help.
I think checkpoint stream and coordinate stream will be auto-create per job if using kaka, but not metrics. After I manually created metrics stream in Kafka, the metrics is written into the stream. ShuQi > 在 2016年9月7日，23:15，Jagadish Venkatraman <jagadish1...@gmail.com> 写道： > > Can you run your program in DEBUG log-level? Does sending the metric to the > producer fail? Is the metric reporter thread showing an exception? (check > the stderr file too) > > Producing to a kafka topic should usually auto-create it. > > On Wed, Sep 7, 2016 at 2:09 AM, 舒琦 <sh...@eefung.com> wrote: > >> Hi, >> >> My samza job has following metrics configuration: >> >> serializers.registry.metrics.class=org.apache.samza.serializers. >> MetricsSnapshotSerdeFactory >> >> systems.kafka.samza.factory=org.apache.samza.system.kafka. >> KafkaSystemFactory >> systems.kafka.consumer.zookeeper.connect=zk11:3181,zk12:3181,zk13:3181 >> systems.kafka.producer.bootstrap.servers=buka1:9096,buka2:9096,buka3:9096 >> >> systems.kafka.streams.samza-metrics.samza.msg.serde=metrics >> >> metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter. >> MetricsSnapshotReporterFactory >> metrics.reporter.snapshot.stream=kafka.samza-metrics >> metrics.reporters=snapshot >> >> And the job is deployed on yarn, after job started, everything is fine, I >> also can find coordinator stream and checkpoint stream in the same kafka >> cluster, but there is no samza-metrics stream. >> >> One of the container log : >> >> 2016-09-07 16:32:31.947 [main] MetricsSnapshotReporterFactory [WARN] >> Unable to find implementation version in jar's meta info. Defaulting to >> 0.0.1. >> 2016-09-07 16:32:31.948 [main] MetricsSnapshotReporterFactory [INFO] Got >> system stream SystemStream [system=kafka, stream=samza-metrics]. >> 2016-09-07 16:32:31.949 [main] MetricsSnapshotReporterFactory [INFO] Got >> system factory org.apache.samza.system.kafka.KafkaSystemFactory@1eed1f10. >> 2016-09-07 16:32:31.950 [main] MetricsSnapshotReporterFactory [INFO] Got >> producer org.apache.samza.system.kafka.KafkaSystemProducer@16d96b45. >> 2016-09-07 16:32:31.951 [main] MetricsSnapshotReporterFactory [INFO] Got >> serde org.apache.samza.serializers.MetricsSnapshotSerde@569f129d. >> 2016-09-07 16:32:31.952 [main] MetricsSnapshotReporterFactory [INFO] >> Setting polling interval to 60 >> 2016-09-07 16:32:31.954 [main] MetricsSnapshotReporter [INFO] got metrics >> snapshot reporter properties [job name: data-status-persistent-hstore, job >> id: 1, containerName: samza-container-1, version: 0.0.1, samzaVersion: >> 0.10.1, host: store116, pollingInterval 60] >> 2016-09-07 16:32:31.955 [main] MetricsSnapshotReporter [INFO] Registering >> MetricsSnapshotReporterFactory with producer. >> 2016-09-07 16:32:31.955 [main] SamzaContainer$ [INFO] Got metrics >> reporters: Set(snapshot) >> >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering >> TaskName-Partition 7 with producer. >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering >> TaskName-Partition 1 with producer. >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering >> TaskName-Partition 5 with producer. >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering >> TaskName-Partition 3 with producer. >> 2016-09-07 16:32:32.017 [main] SamzaContainer [INFO] Starting JVM metrics. >> 2016-09-07 16:32:32.017 [main] SamzaContainer [INFO] Starting metrics >> reporters. >> 2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Registering >> samza-container-1 with producer. >> 2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Starting >> producer. >> 2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Starting >> reporter timer. >> 2016-09-07 16:32:32.019 [main] SamzaContainer [INFO] Registering task >> instances with offsets. >> 2016-09-07 16:32:32.022 [main] SamzaContainer [INFO] Starting offset >> manager. >> >> 2016-09-07 16:32:32.212 [SAMZA-METRIC-SNAPSHOT-REPORTER] >> KafkaSystemProducer [INFO] Creating a new producer for system kafka. >> 2016-09-07 16:32:32.221 [SAMZA-METRIC-SNAPSHOT-REPORTER] ProducerConfig >> [INFO] ProducerConfig values: >> value.serializer = class org.apache.kafka.common.serialization. >> ByteArraySerializer >> key.serializer = class org.apache.kafka.common.serialization. >> ByteArraySerializer >> block.on.buffer.full = true >> retry.backoff.ms = 100 >> buffer.memory = 33554432 >> batch.size = 16384 >> metrics.sample.window.ms = 30000 >> metadata.max.age.ms = 300000 >> receive.buffer.bytes = 32768 >> timeout.ms = 30000 >> max.in.flight.requests.per.connection = 1 >> bootstrap.servers = [buka1:9096, buka2:9096, buka3:9096] >> metric.reporters = [] >> client.id = samza_producer-data_status_persistent_hstore-1- >> 1473237151949-4 >> compression.type = none >> retries = 2147483647 >> max.request.size = 1048576 >> send.buffer.bytes = 131072 >> acks = 1 >> reconnect.backoff.ms = 10 >> linger.ms = 0 >> metrics.num.samples = 2 >> metadata.fetch.timeout.ms = 60000 >> >> Thanks. >> >> ———————— >> 舒琦 >> 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F >> 网址：http://www.eefung.com >> 微博：http://weibo.com/eefung >> 邮编：410013 >> 电话：400-677-0986 >> 传真：0731-88519609 >> >> > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University
