Wouldn't it be a better choice to store the logs offline somewhere? HDFS and S3
are both good choices...
-Mark
> On Feb 27, 2015, at 16:12, Warren Kiser wrote:
>
> Does anyone know how to achieve unlimited log retention either globally or
> on a per topic basis? I tried explicitly setting the
Kafka is more of a message queue than a data store. You can use it to store
history of the queue (certainly a powerful use case for disaster recovery),
but it's still not really a data store.
>From the Kafka website (kafka.apache.org):
Apache Kafka is a publish-subscribe messaging [queue] rethough
nsumer, so people
> > > who use Kafka's built-in clients will know which version they have in
> the
> > > environment.
> > >
> > >
> > >
> > > On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts
> wrote:
> > >
> > >>
Just to be clear: this is going to be exposed via some Api the clients can call
at startup?
> On Nov 12, 2014, at 08:59, Guozhang Wang wrote:
>
> Sounds great, +1 on this.
>
>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira wrote:
>>
>> So it looks like we can use Gradle to add properties to
I think it will depend on how your producer application logs things, but
yes I have historically seen exceptions in the producer logs when they
exceed the max message size.
-Mark
On Mon, Oct 27, 2014 at 10:19 AM, Chen Wang
wrote:
> Hello folks,
> I recently noticed our message amount in kafka s
If you use Kafka for the first bulk load, you will test your new
Teradata->Kafka->Hive pipeline, as well as have the ability to blow away
the data in Hive and reflow it from Kafka without an expensive full
re-export from Teradata. As for whether Kafka can handle hundreds of GB of
data: Yes, absolu
Did this mailing list ever get created? Was there consensus that it did or
didn't need created?
-Mark
> On Jul 18, 2014, at 14:34, Jay Kreps wrote:
>
> A question was asked in another thread about what was an effective way
> to contribute to the Kafka project for people who weren't very
> enth
Hi all,
As a client engineer on the python client, I would really appreciate a
separate mailing list for client implementation discussion and a language
agnostic test suite. What might also be really useful is an enumerated
list of error conditions and the expected behavior to come out of them.
When we were in testing phase, we would either create a new topic with the
correct details or shut the cluster down and hard kill the topic in zookeeper +
local disk. In prod we have the cluster configured via configuration
management and auto create turned off.
The ability to delete a topic i
You would ship the contents of the file across as a message. In general this
would mean that your maximum file size must be smaller than your maximum
message size. It would generally be a better choice to put a pointer to the
file in some shared location on the queue.
-Mark
> On Jun 15, 2014,
10 matches
Mail list logo
