Messages lost after broker failure
Hi everyone, my team is building real-time system using Samza (version 0.11.0) and we are facing some issues with data loss so we would like to hear your thoughts. Due to using some additional tools for monitoring and alerting we exceeded number of allowed open files so TooManyOpenFiles exception caused our brokers to fail. After fixing this issue failed brokers and all Samza jobs were restarted. Issue was gone but it seems like we are constantly losing almost half or the messages from some of our topics after this incident. To keep things as simple as possible I will focus just on a small part of the pipeline. On the picture below we can see two topics, both with 80 partitions, that are input and output for one of our Samza jobs. Number of messages in those topics should be the same but we see that output topic has almost two times less messages than the input one. There is no some kind of bottleneck so messages are not kept in Kafka for too long and they are not deleted by log retention before processing. http://ibb.co/iJrDbF Another strange thing is that some old messages are appearing after day or two. All this is leading us to conclusion that Samza's consumers are somehow not aware of all of the partitions. Is it possible that consumers are not aware of new partition leaders, since new leader selection occurred after broker failures, and somehow they are trying to get data from the old ones that are not the leaders anymore and have a lower offsets meaning that a new messages are skipped. Is there some kind of topic metadata caching that could lead us to this situation? While debugging we discovered KafkaSystemConsumer exception that says no leader for partition. Looking at the Kafka Manager all partitions have their leaders. Here are some additional details that might be useful. Our Samza jobs are built on top of Samza v 0.11.0. Kafka 0.8.2.1 consumers/producers are used in jobs. Kafka cluster: - 8 brokers - Kafka version 0.10.1 - unclean.leader.election.enable false - replica.lag.time.max.ms 1 - log.flush.interval.ms 720 Input topic: - segment.bytes 2147483647 - retention.ms 17280 Some warnings from logs: Error processing fetch operation on partition [topic_name,35], offset 232841013 (kafka.server.ReplicaManager) java.lang.IllegalStateException: Failed to read complete buffer for targetOffset 241769924 startPosition 2147479938 in BrokerProxy [WARN] Got non-recoverable error codes during multifetch. Throwing an exception to trigger reconnect. Errors: Error([topic_name,47],-1,kafka.common.UnknownException BrokerProxy [WARN] It appears that we received an invalid or empty offset Some(366399914) for [topic_name,60]. Attempting to use Kafka's auto.offset.reset setting. This can result in data loss if processing continues. Any help and suggestion will be appreciated. Thanks, Aleksandar Bircakovic
RE: Messages lost after broker failure
Thank you Jagadish. Regarding leader selection, we exposed some additional metrics that are supposed to tell us which broker is leader for each partition and we confirmed your theory, Samza consumer if fully aware who is the current leader. Entire logs are pretty big so here are some parts instead. File container_logs.txt contains logs from containers. Here we can see kafka.common.UnknownException. Kafka_logs.txt is log from one of the brokers where we have 'Error processing fetch operation on partition' during the whole day. It seems like that issue is fixed in scope of this Kafka ticket (https://issues.apache.org/jira/browse/KAFKA-4576). For those offsets that appear in Kafka log we have Samza system logs with INFO severity saying that Samza is constantly validating those offsets. Samza system logs are in file samza_system_logs.txt. If you need anything else please say so. Container log: https://drive.google.com/open?id=0B9_OEEXuRc2aQVRFTFpGb3BKcXM Kafka log: https://drive.google.com/open?id=0B9_OEEXuRc2aWXB4NTRkUXBxcGs Samza log as JSON: https://drive.google.com/open?id=0B9_OEEXuRc2acEZ4VlVWQkQ1aTA While analyzing issues reported on Kafka board we found out that older versions of Kafka had some edge cases where broker failure recovery didn't work as expected, should we switch to new Kafka consumer since we are using Kafka 0.10.1 but our Samza jobs are using Kafka consumer 0.8.2? Also, since some messages are appearing after even few days, is it possible that producer gets stuck and sends some messages with really big latency after many unsuccessful retries? Many thanks, Aleksandar Bircakovic -Original Message- From: Jagadish Venkatraman [mailto:jagadish1...@gmail.com] Sent: Tuesday, April 4, 2017 6:31 PM To: dev@samza.apache.org Cc: m.mis...@improvedigital.com Subject: Re: Messages lost after broker failure >> All this is leading us to conclusion that Samza's consumers are >> somehow not aware of all of the partitions We have had a number of broker failures at LinkedIn, and have not run into data loss issues due to consumers being unaware of partitions. You can use the metrics emitted at a per-partition level (like messages read, offset lags etc.) to validate this theory. >> BrokerProxy [WARN] It appears that we received an invalid or empty offset Some(366399914) for [topic_name,60]. Attempting to use Kafka's auto.offset.reset setting. Usually, attempting to fetch from an invalid offset will reset the consumer to the upcoming offset. This will cause data-loss since you will only process new messages. It will be interesting to find out what caused the consumer to receive an invalid offset / why the received offset was invalid. Also, the entire log will be helpful (assuming there's no sensitive information that must be redacted). On Tue, Apr 4, 2017 at 1:12 AM, Aleksandar Bircakovic < a.bircako...@levi9.com> wrote: > Hi everyone, > my team is building real-time system using Samza (version 0.11.0) and > we are facing some issues with data loss so we would like to hear your > thoughts. > > Due to using some additional tools for monitoring and alerting we > exceeded number of allowed open files so TooManyOpenFiles exception > caused our brokers to fail. > After fixing this issue failed brokers and all Samza jobs were restarted. > Issue was gone but it seems like we are constantly losing almost half > or the messages from some of our topics after this incident. > To keep things as simple as possible I will focus just on a small part > of the pipeline. On the picture below we can see two topics, both > with 80 partitions, that are input and output for one of our Samza > jobs. Number of messages in those topics should be the same but we see > that output topic has almost two times less messages than the input > one. There is no some kind of bottleneck so messages are not kept in > Kafka for too long and they are not deleted by log retention before > processing. > > http://ibb.co/iJrDbF > > Another strange thing is that some old messages are appearing after > day or two. All this is leading us to conclusion that Samza's > consumers are somehow not aware of all of the partitions. Is it > possible that consumers are not aware of new partition leaders, since > new leader selection occurred after broker failures, and somehow they > are trying to get data from the old ones that are not the leaders > anymore and have a lower offsets meaning that a new messages are > skipped. Is there some kind of topic metadata caching that could lead > us to this situation? While debugging we discovered > KafkaSystemConsumer exception that says no leader for partition. Looking at > the Kafka Manager all partitions have their leaders. > > > Here are some additional details that might be useful.
RE: Messages lost after broker failure
Hello again, just a quick update regarding this message loss issue. Our further investigation proved that Samza job was unable to read 10 out of 80 partitions from input topic. Tasks that were consuming those partitions were stuck. All those tasks were running on the same container. When trying to fetch messages with the offset obtained from checkpoint topic it would try to reset offsets and fail throwing an unknown exception (BrokerProxy [WARN] Got non-recoverable error codes during multifetch. Throwing an exception to trigger reconnect. Errors: Error([topic_name,47],-1,kafka.common.UnknownException) when it would enter an infinite loop. The solution was to temporary disable checkpointing for that topic and after that those tasks were able to consume their partitions. In a meanwhile, console consumers/producer and standalone apps were able to read from those partitions. We are currently trying to reproduce this without luck, do you perhaps have a scenario in mind that could cause such behavior? Thank you, Aleksandar Bircakovic -Original Message- From: Aleksandar Bircakovic [mailto:a.bircako...@levi9.com] Sent: Wednesday, April 5, 2017 5:26 PM To: dev@samza.apache.org Cc: m.mis...@improvedigital.com Subject: RE: Messages lost after broker failure Thank you Jagadish. Regarding leader selection, we exposed some additional metrics that are supposed to tell us which broker is leader for each partition and we confirmed your theory, Samza consumer if fully aware who is the current leader. Entire logs are pretty big so here are some parts instead. File container_logs.txt contains logs from containers. Here we can see kafka.common.UnknownException. Kafka_logs.txt is log from one of the brokers where we have 'Error processing fetch operation on partition' during the whole day. It seems like that issue is fixed in scope of this Kafka ticket (https://issues.apache.org/jira/browse/KAFKA-4576). For those offsets that appear in Kafka log we have Samza system logs with INFO severity saying that Samza is constantly validating those offsets. Samza system logs are in file samza_system_logs.txt. If you need anything else please say so. Container log: https://drive.google.com/open?id=0B9_OEEXuRc2aQVRFTFpGb3BKcXM Kafka log: https://drive.google.com/open?id=0B9_OEEXuRc2aWXB4NTRkUXBxcGs Samza log as JSON: https://drive.google.com/open?id=0B9_OEEXuRc2acEZ4VlVWQkQ1aTA While analyzing issues reported on Kafka board we found out that older versions of Kafka had some edge cases where broker failure recovery didn't work as expected, should we switch to new Kafka consumer since we are using Kafka 0.10.1 but our Samza jobs are using Kafka consumer 0.8.2? Also, since some messages are appearing after even few days, is it possible that producer gets stuck and sends some messages with really big latency after many unsuccessful retries? Many thanks, Aleksandar Bircakovic -Original Message- From: Jagadish Venkatraman [mailto:jagadish1...@gmail.com] Sent: Tuesday, April 4, 2017 6:31 PM To: dev@samza.apache.org Cc: m.mis...@improvedigital.com Subject: Re: Messages lost after broker failure >> All this is leading us to conclusion that Samza's consumers are >> somehow not aware of all of the partitions We have had a number of broker failures at LinkedIn, and have not run into data loss issues due to consumers being unaware of partitions. You can use the metrics emitted at a per-partition level (like messages read, offset lags etc.) to validate this theory. >> BrokerProxy [WARN] It appears that we received an invalid or empty offset Some(366399914) for [topic_name,60]. Attempting to use Kafka's auto.offset.reset setting. Usually, attempting to fetch from an invalid offset will reset the consumer to the upcoming offset. This will cause data-loss since you will only process new messages. It will be interesting to find out what caused the consumer to receive an invalid offset / why the received offset was invalid. Also, the entire log will be helpful (assuming there's no sensitive information that must be redacted). On Tue, Apr 4, 2017 at 1:12 AM, Aleksandar Bircakovic < a.bircako...@levi9.com> wrote: > Hi everyone, > my team is building real-time system using Samza (version 0.11.0) and > we are facing some issues with data loss so we would like to hear your > thoughts. > > Due to using some additional tools for monitoring and alerting we > exceeded number of allowed open files so TooManyOpenFiles exception > caused our brokers to fail. > After fixing this issue failed brokers and all Samza jobs were restarted. > Issue was gone but it seems like we are constantly losing almost half > or the messages from some of our topics after this incident. > To keep things as simple as possible I will focus just on a small part > of the pipeline. On the picture below we can see two topic
Review Request 35348: SAMZA-707 Remove dependency on samza-serializers in the samza.apache.org/startup/download page
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35348/ --- Review request for samza. Repository: samza Description --- For issue described in [SAMZA-707](https://issues.apache.org/jira/browse/SAMZA-707) removed dependency samza-serializers_2.10 in index.md. Diffs - docs/startup/download/index.md 37f948b Diff: https://reviews.apache.org/r/35348/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 35348: SAMZA-707 Remove dependency on samza-serializers in the samza.apache.org/startup/download page
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35348/ --- (Updated June 17, 2015, 8:15 p.m.) Review request for samza. Changes --- Updated the doc again saying that versions less than 0.9 should include additional dependency for serializers. Repository: samza Description --- For issue described in [SAMZA-707](https://issues.apache.org/jira/browse/SAMZA-707) removed dependency samza-serializers_2.10 in index.md. Diffs (updated) - docs/startup/download/index.md 37f948b Diff: https://reviews.apache.org/r/35348/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 35601: SAMZA-418 Additional information on AM web ui
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35601/ --- Review request for samza. Repository: samza Description --- For issue described in [SAMZA-418](https://issues.apache.org/jira/browse/SAMZA-418?jql=project%20%3D%20SAMZA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open) additional information added to the table on AM web ui. Diffs - samza-yarn/src/main/less/main.less a0a6785 samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml a874b0e Diff: https://reviews.apache.org/r/35601/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 35606: SAMZA-716 One Link in Spark Streaming and Samza comparison page is broken
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35606/ --- Review request for samza. Repository: samza Description --- The "post" is now pointing to the correct link. Diffs - docs/learn/documentation/versioned/comparisons/spark-streaming.md e1ccc3e Diff: https://reviews.apache.org/r/35606/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 35601: SAMZA-418 Additional information on AM web ui
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35601/ --- (Updated June 19, 2015, 8:21 a.m.) Review request for samza. Changes --- Added the yarn.package.path to the UI and whitespace removed. Repository: samza Description --- For issue described in [SAMZA-418](https://issues.apache.org/jira/browse/SAMZA-418?jql=project%20%3D%20SAMZA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open) additional information added to the table on AM web ui. Diffs (updated) - samza-yarn/src/main/less/main.less a0a6785 samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml a874b0e Diff: https://reviews.apache.org/r/35601/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 35918: Monitoring page for REST API and the dashboard
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35918/ --- Review request for samza. Repository: samza Description --- Added new monitoring page for REST API and the dashboard and removed dashboard from ApplicationMaster. Also added table that shortly explains REST service. Diffs - docs/learn/documentation/versioned/index.html e1b9f2d docs/learn/documentation/versioned/jobs/reprocessing.md 28d9925 docs/learn/documentation/versioned/jobs/web-ui-rest-api.md PRE-CREATION Diff: https://reviews.apache.org/r/35918/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 35918: SAMZA-709 Monitoring page for REST API and the dashboard
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35918/ --- (Updated June 26, 2015, 11:50 a.m.) Review request for samza. Summary (updated) - SAMZA-709 Monitoring page for REST API and the dashboard Repository: samza Description --- Added new monitoring page for REST API and the dashboard and removed dashboard from ApplicationMaster. Also added table that shortly explains REST service. Diffs - docs/learn/documentation/versioned/index.html e1b9f2d docs/learn/documentation/versioned/jobs/reprocessing.md 28d9925 docs/learn/documentation/versioned/jobs/web-ui-rest-api.md PRE-CREATION Diff: https://reviews.apache.org/r/35918/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 35918: SAMZA-709 Monitoring page for REST API and the dashboard
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35918/ --- (Updated July 2, 2015, 1:53 p.m.) Review request for samza. Changes --- New screenshot of AM and basic html table. Repository: samza Description --- Added new monitoring page for REST API and the dashboard and removed dashboard from ApplicationMaster. Also added table that shortly explains REST service. Diffs (updated) - docs/img/versioned/learn/documentation/yarn/samza-am-dashboard.png 949a2f0 docs/learn/documentation/versioned/index.html e1b9f2d docs/learn/documentation/versioned/jobs/reprocessing.md 28d9925 docs/learn/documentation/versioned/jobs/web-ui-rest-api.md PRE-CREATION Diff: https://reviews.apache.org/r/35918/diff/ Testing --- File Attachments (updated) samza-am-dashboard.png https://reviews.apache.org/media/uploaded/files/2015/07/02/2b557ea4-5718-48a9-a619-78931fdc30d7__samza-am-dashboard.png Thanks, Aleksandar Bircakovic
Review Request 37521: SAMZA-737 property name stores.store-name.changelog.topic-level-property is not effective
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37521/ --- Review request for samza. Repository: samza Description --- For issue described in [SAMZA-737](https://issues.apache.org/jira/browse/SAMZA-737) changed property stores.store-name.changelog.topic-level-property to stores.store-name.changelog.kafka.topic-level-property Diffs - docs/learn/documentation/versioned/jobs/configuration-table.html 8177fe5 Diff: https://reviews.apache.org/r/37521/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 37536: SAMZA-710 Update WebServlet and RestServlet to read coordinatorStream information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37536/ --- Review request for samza. Repository: samza Description --- WebServlet and RestServlet now read information from coordinator stream consumer and get new config. Diffs - samza-core/src/main/java/org/apache/samza/coordinator/stream/AbstractCoordinatorStreamManager.java ca97ce8 samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 7b59274 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala 09f4dc3 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterWebServlet.scala 7fd5122 Diff: https://reviews.apache.org/r/37536/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 37604: SAMZA-760 Samza Container should catch Throwables instead of just catching Exceptions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37604/ --- Review request for samza. Repository: samza Description --- Added a catch for Throwables in Samza container. Catching Throwables can cause problems in specific situations so I also added a partial function 'safely' that should take care of that specific situations. Diffs - samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 85b012b Diff: https://reviews.apache.org/r/37604/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 37536: SAMZA-710 Update WebServlet and RestServlet to read coordinatorStream information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37536/ --- (Updated Aug. 20, 2015, 9:19 a.m.) Review request for samza. Changes --- As the AbstractCoordinatorStreamManager is used to access and modify the coordinatorStreamConsumer I kept method getLatestConfig there but I added a new class ConfigManager that is used to provide the latest config instead of LocalityManager to avoid 'overuse'. Here is a simple unit test that checks if getLatestConfig returns notNull. I also sanitized configMap as it was mentioned. If this is a wrong way of thinking any suggestions will be appreciated. Thanks. Repository: samza Description --- WebServlet and RestServlet now read information from coordinator stream consumer and get new config. Diffs (updated) - checkstyle/import-control.xml aaa235a samza-core/src/main/java/org/apache/samza/container/ConfigManager.java PRE-CREATION samza-core/src/main/java/org/apache/samza/coordinator/stream/AbstractCoordinatorStreamManager.java ca97ce8 samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 7b59274 samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala a926ce6 samza-core/src/test/scala/org/apache/samza/config/TestConfigManager.scala PRE-CREATION samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala 09f4dc3 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterWebServlet.scala 7fd5122 Diff: https://reviews.apache.org/r/37536/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 37604: SAMZA-760 Samza Container should catch Throwables instead of just catching Exceptions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37604/ --- (Updated Aug. 25, 2015, 11:48 a.m.) Review request for samza. Changes --- Added a unit test for verification. Repository: samza Description --- Added a catch for Throwables in Samza container. Catching Throwables can cause problems in specific situations so I also added a partial function 'safely' that should take care of that specific situations. Diffs (updated) - samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 85b012b samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 4db6d5c Diff: https://reviews.apache.org/r/37604/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 37795: SAMZA-549: Rat shouldn't fail unlicensed files on build
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37795/ --- Review request for samza. Repository: samza Description --- Rat will just warn about unlicensed files, build will not fail. Diffs - gradle/rat.gradle 9cbc7c7 Diff: https://reviews.apache.org/r/37795/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 37994: SAMZA 368 AM UI should show failed container information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37994/ --- Review request for samza. Repository: samza Description --- Failed containers (ID, exit code and message) are displayed on the AM UI under Containers section. Diffs - samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml ce2145a samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala f667c83 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala 1743c86 Diff: https://reviews.apache.org/r/37994/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 38043: SAMZA-768 Update the downloading link for the source code
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38043/ --- Review request for samza. Repository: samza Description --- Updated downloading links for the source code. Diffs - docs/startup/download/index.md 2f4bf2c Diff: https://reviews.apache.org/r/38043/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 38049: SAMZA-769 Replace deprecated method call and fix warnings
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38049/ --- Review request for samza. Repository: samza Description --- Replaced deprecated method call and suppressed some warnings. Diffs - samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java bc926c5 samza-autoscaling/src/main/java/org/apache/samza/autoscaling/deployer/ConfigManager.java 7089796 samza-autoscaling/src/main/java/org/apache/samza/autoscaling/utils/YarnUtil.java b2d37a7 samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java c564964 samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamWriter.java 4eaaec2 samza-log4j/src/main/java/org/apache/samza/logging/log4j/serializers/LoggingEventJsonSerde.java a18d8e0 Diff: https://reviews.apache.org/r/38049/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 37536: SAMZA-710 Update WebServlet and RestServlet to read coordinatorStream information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37536/ --- (Updated Sept. 8, 2015, 6:47 a.m.) Review request for samza. Changes --- Added Unit test and fixed things mentioned in previous review. Sorry for that kind of mistakes (wrong names and so on). Thanks for suggestions. Repository: samza Description --- WebServlet and RestServlet now read information from coordinator stream consumer and get new config. Diffs (updated) - checkstyle/import-control.xml bc07ae8 samza-core/src/main/java/org/apache/samza/container/LatestConfigManager.java PRE-CREATION samza-core/src/main/java/org/apache/samza/coordinator/stream/AbstractCoordinatorStreamManager.java ca97ce8 samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 7b59274 samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala a926ce6 samza-core/src/test/scala/org/apache/samza/config/TestLatestConfigManager.scala PRE-CREATION samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala 09f4dc3 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterWebServlet.scala 7fd5122 Diff: https://reviews.apache.org/r/37536/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 38180: SAMZA-436 Update RocksDB KV's putAll to use write batch
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38180/ --- Review request for samza. Repository: samza Description --- WriteBatch is now used inside RocksDB putAll implementation instead of put sequence. Diffs - samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala a423f7b Diff: https://reviews.apache.org/r/38180/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 38049: SAMZA-769 Replace deprecated method call and fix warnings
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38049/ --- (Updated Sept. 16, 2015, 1:06 p.m.) Review request for samza. Changes --- Fixed things mentioned in review. Repository: samza Description --- Replaced deprecated method call and suppressed some warnings. Diffs (updated) - samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java bc926c5 samza-autoscaling/src/main/java/org/apache/samza/autoscaling/deployer/ConfigManager.java 7089796 samza-autoscaling/src/main/java/org/apache/samza/autoscaling/utils/YarnUtil.java b2d37a7 samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java c564964 samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamWriter.java 4eaaec2 samza-log4j/src/main/java/org/apache/samza/logging/log4j/serializers/LoggingEventJsonSerde.java a18d8e0 Diff: https://reviews.apache.org/r/38049/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 38668: SAMZA-776 building instructions have bad URL - http instead of https
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38668/ --- Review request for samza. Repository: samza Description --- Corrected URL in docs Diffs - RELEASE.md 72a56ea docs/contribute/code.md be85c8d docs/learn/tutorials/versioned/remote-debugging-samza.md efef045 docs/learn/tutorials/versioned/run-in-multi-node-yarn.md 7a9fdf4 docs/learn/tutorials/versioned/upgrading-from-0.7.0-to-0.8.0.md 340762e docs/startup/download/index.md ee05714 Diff: https://reviews.apache.org/r/38668/diff/ Testing --- Thanks, Aleksandar Bircakovic
Re: Review Request 37604: SAMZA-760 Samza Container should catch Throwables instead of just catching Exceptions
> On Oct. 1, 2015, 7:44 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, > > line 599 > > <https://reviews.apache.org/r/37604/diff/2/?file=1051095#file1051095line599> > > > > Forgive me on ignorance on this ControlThrowable. Why should we skip > > logging this one? You mentioned in the RB description that there are some > > nuance involved by catching all Throwables. Could you elaborate a bit more > > here? > > > > Thanks! Thank you for review. I found some articles saying that catching Throwables in Scala isn't so wise (like this one https://www.sumologic.com/2014/05/05/why-you-should-never-catch-throwable-in-scala/). They say it can have negative impact on JVM. > On Oct. 1, 2015, 7:44 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, > > line 582 > > <https://reviews.apache.org/r/37604/diff/2/?file=1051095#file1051095line582> > > > > The bug description states that making the Throwable as a cause of the > > SamzaException, I would perfer to use SamzaException(String s, Throwable t) > > s.t. more detailed cause info would be print out. Agree with that. - Aleksandar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37604/#review101272 --- On Aug. 25, 2015, 11:48 a.m., Aleksandar Bircakovic wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37604/ > --- > > (Updated Aug. 25, 2015, 11:48 a.m.) > > > Review request for samza. > > > Repository: samza > > > Description > --- > > Added a catch for Throwables in Samza container. Catching Throwables can > cause problems in specific situations so I also added a partial function > 'safely' that should take care of that specific situations. > > > Diffs > - > > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala > 85b012b > > samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala > 4db6d5c > > Diff: https://reviews.apache.org/r/37604/diff/ > > > Testing > --- > > > Thanks, > > Aleksandar Bircakovic > >
Re: Review Request 37604: SAMZA-760 Samza Container should catch Throwables instead of just catching Exceptions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37604/ --- (Updated Oct. 8, 2015, 9:42 a.m.) Review request for samza. Changes --- Used SamzaException(String s, Throwable t) instead of SamzaException(String s) as Yi Pan suggested. Repository: samza Description --- Added a catch for Throwables in Samza container. Catching Throwables can cause problems in specific situations so I also added a partial function 'safely' that should take care of that specific situations. Diffs (updated) - samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala f351ad6 samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 6de8710 Diff: https://reviews.apache.org/r/37604/diff/ Testing --- Thanks, Aleksandar Bircakovic
Review Request 39130: SAMZA-791 KafkaSystemFactory narrows return types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39130/ --- Review request for samza. Repository: samza Description --- Specified return types for methods in KafkaSystemFactory. Diffs - samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala d84bf06 Diff: https://reviews.apache.org/r/39130/diff/ Testing --- ./gradlew clean build Thanks, Aleksandar Bircakovic
Re: Review Request 37994: SAMZA 368 AM UI should show failed container information
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37994/ --- (Updated Oct. 12, 2015, 11:59 a.m.) Review request for samza. Changes --- Patch refactored after SAMZA-619 Repository: samza Description --- Failed containers (ID, exit code and message) are displayed on the AM UI under Containers section. Diffs (updated) - samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaAppState.java d5be36e samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaTaskManager.java d17ffe0 samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml 2d16fe0 Diff: https://reviews.apache.org/r/37994/diff/ Testing (updated) --- ./gradlew clean build successful and failed containers displayed on AM UI Thanks, Aleksandar Bircakovic
Review Request 40857: SAMZA 826 Fix string format issue with log message in ContainerUtil
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40857/
---
Review request for samza.
Repository: samza
Description
---
The '%s' replaced with '{}'
Diffs
-
samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 1fb6a5f
Diff: https://reviews.apache.org/r/40857/diff/
Testing
---
Thanks,
Aleksandar Bircakovic
Review Request 41874: SAMZA-837: Support Gradle 2.9
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41874/ --- Review request for samza. Repository: samza Description --- Formatted code in some Java classes so now gradle clean check runs with no checkstyle errors with new versions of Gradle. Diffs - samza-api/src/main/java/org/apache/samza/storage/StorageEngineFactory.java 963ccf2 samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java 429573b samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java 0e73e18 Diff: https://reviews.apache.org/r/41874/diff/ Testing --- gradle clean check - BUILD SUCCESSFUL ./gradlew clean build - BUILD SUCCESSFUL Thanks, Aleksandar Bircakovic
Re: Review Request 41874: SAMZA-837: Support Gradle 2.9
> On Jan. 5, 2016, 12:21 a.m., Jake Maes wrote: > > Ship It! > > Jake Maes wrote: > Any idea why the checkstyle behavior changed with the newer gradle? Gradle 2.7 or newer is required for Checkstyle versions 6.8 - 6.12.1. In my opinion Gradle will by default try to use new supported version of Checkstyle and new version brings more validations. - Aleksandar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41874/#review112705 --- On Jan. 4, 2016, 9:59 a.m., Aleksandar Bircakovic wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41874/ > --- > > (Updated Jan. 4, 2016, 9:59 a.m.) > > > Review request for samza. > > > Repository: samza > > > Description > --- > > Formatted code in some Java classes so now gradle clean check runs with no > checkstyle errors with new versions of Gradle. > > > Diffs > - > > samza-api/src/main/java/org/apache/samza/storage/StorageEngineFactory.java > 963ccf2 > > samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java > 429573b > > samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java > 0e73e18 > > Diff: https://reviews.apache.org/r/41874/diff/ > > > Testing > --- > > gradle clean check - BUILD SUCCESSFUL > ./gradlew clean build - BUILD SUCCESSFUL > > > Thanks, > > Aleksandar Bircakovic > >
