Hi Jeremiah, >> It looks like my configuration for changelog.replication.factor is not being applied. Instead the default seems to be applied. I have messages on 2/3 partitions. I am not seeing what I have incorrectly configured.
The "replication.factor" config you mention determines how many brokers each change-log partition is replicated to. This is un-related to the behavior you are observing - where a single partition is empty. For context, each Samza task has its own store and writes to a single change-log partition. An empty change-log partition means its corresponding task did not perform writes to its store. Can you check if there are *input* topic-partitions that don't receive messages? If so, tasks consuming from them wouldn't have data to process. This results in their store and the corresponding change-log partition to be empty. Also, can you check for any exceptions in your logs? For details on Samza's state management, I'd recommend the below resources: Videos: Samza Architecture Part-1 <https://www.youtube.com/watch?v=F7WJNqP2nCM&index=4&list=PLZDyxA22zzGyNgtBMUIXAgIaO5Ok3PR-x&t=0s> Samza Architecture Part-2 <https://www.youtube.com/watch?v=Uafd_tJgAWs&index=4&list=PLZDyxA22zzGyNgtBMUIXAgIaO5Ok3PR-x> Docs: Architecture overview <https://samza.apache.org/learn/documentation/latest/architecture/architecture-overview.html> Best, Jagadish On Mon, Mar 25, 2019 at 8:30 AM Jeremiah Adams <jad...@helixeducation.com> wrote: > It looks like my configuration for changelog.replication.factor is not > being applied. Instead the default seems to be applied. I have messages on > 2/3 partitions. I am not seeing what I have incorrectly configured. > > > stores.redelivery-store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory > stores.redelivery-store.changelog=kafka.delivery-changelog > stores.default.changelog.replication.factor=3 > > > > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > ________________________________________ > From: Jagadish Venkatraman <jagadish1...@gmail.com> > Sent: Friday, March 22, 2019 3:01 PM > To: dev@samza.apache.org > Subject: Re: Empty Kafka topic partition Warning > > Hi Jeremiah, > > >> why is the offset 0? > > This likely means that the change-log is empty and does not have any > messages. > > Can you try consuming from partition-number: 0 using a KafkaConsumer? > > Best, > Jagadish > > > > > > On Fri, Mar 22, 2019 at 11:45 AM Jeremiah Adams <jad...@helixeducation.com > > > wrote: > > > I'm seeing these in our log periodically and havn't seen them before. > > Does this imply that the topic associated with the change log is being > > replayed from the beginning? > > > > > > Also, why is the offset 0? It definitely should not be. We have messages > > across all partitions. > > > > > > 2019-03-22 18:30:24 KafkaSystemAdmin [INFO] Fetching SSP metadata for: > > [SystemStreamPartition [kafka, delivery-changelog, 0]] > > 2019-03-22 18:30:24 KafkaSystemAdmin [WARN] Empty Kafka topic partition > > delivery-changelog-0 with upcoming offset 0. Skipping newest offset and > > setting oldest offset to 0 to consume from beginning > > 2019-03-22 18:30:52 KafkaSystemAdmin [INFO] Fetching SSP metadata for: > > [SystemStreamPartition [kafka, delivery-changelog, 0]] > > 2019-03-22 18:30:52 KafkaSystemAdmin [WARN] Empty Kafka topic partition > > delivery-changelog-0 with upcoming offset 0. Skipping newest offset and > > setting oldest offset to 0 to consume from beginning > > > > > > Jeremiah Adams > > Software Engineer > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > < > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~ > > > > Blog< > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~> > | Twitter< > > > https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~> > | Facebook< > > > https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~> > | LinkedIn< > > > https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0 > > > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
