Hey Roger,
We haven't thought about this in great detail. People do all kinds of wacky
things in practice. We have some that are like, AdViewsByMemberId. There
are various permutations of that.
One thing I haven't seen, but might be relevant, is including partition
counts in the topic. If you're
Hey Chinmay,
Cool, this is good feedback. I didn't think I was *that* crazy. :)
Cheers,
Chris
On Wed, Mar 18, 2015 at 6:10 PM, Chinmay Soman chinmay.cere...@gmail.com
wrote:
Thats what we're doing as well - appending partition count to the kafka
topic name. This actually helps keep track of
Hey Jakob,
Yeah, but then if you change the partition count later on, you've got
incorrect
information forever.
You're right. But IMO this further reinforces that you *can't* change
partition counts on a topic that you're using for a JOIN. This completely
breaks the operation.
Agree that
On March 18, 2015, 9:01 p.m., Navina Ramesh wrote:
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala,
line 49
https://reviews.apache.org/r/32155/diff/1/?file=897641#file897641line49
Can you explain the purpose of failedSources?
It is used to
Thats what we're doing as well - appending partition count to the kafka
topic name. This actually helps keep track of the #partitions for each
topic (since Kafka doesn't have a Metadata store yet).
In case of topic expansion - we actually just resort to creating a new
topic. Although that is an
Hi,
Wondering what naming conventions people are using for topics in Kafka.
When there's re-partitioning involved, you can end up with multiple topics
that have the exact same data but are partitioned differently. How do you
name them?
Thanks,
Roger
Yeah ! It does seem a bit hackish - but I think this approach promises less
config/operation errors.
Although I think some of these checks can be built within Samza - assuming
Kafka has a metadata store in the near future - the Samza container can
validate the #topics against this store.
On Wed,
Thanks, guys. I was also playing around with including partition count and
even the partition key in the topic name. My thought was that topics may
have the same data and number of partitions but only differ by partition
key. After a while, the naming does get crazy (too long and ugly). We
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32188/
---
(Updated March 18, 2015, 9:48 p.m.)
Review request for samza.
Repository:
