[
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15780697#comment-15780697
]
Bill Warshaw commented on KAFKA-2260:
-
[~ijuma] it seems like we would be abl
[
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768086#comment-15768086
]
Bill Warshaw commented on KAFKA-2260:
-
I'd like to revive the discussion on
Github user bill-warshaw closed the pull request at:
https://github.com/apache/kafka/pull/1972
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
add
> a trim() api like the following that will trim the log up to the specified
> offsets. This addresses both of the above issues that I mentioned. Will
> that work for you?
>
> void trim(Map offsetsToTruncate)
>
> Thanks,
>
> Jun
>
> On Wed, Oct 5, 2016 at 1:55 PM,
Bumping for visibility. KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy
On Wed, Aug 24, 2016 at 2:32 PM Bill Warshaw wrote:
> Hello Guozhang,
>
> KIP-71 seems unrelated to this KIP. KIP-47 is just adding a new deletion
GitHub user bill-warshaw opened a pull request:
https://github.com/apache/kafka/pull/1972
KAFKA-3224: New log deletion policy based on timestamp
* adds a new topic-level broker configuration, `log.retention.min.timestamp`
* if unset, this setting is ignored
* setting
ke to double check with you.
>
>
> Guozhang
>
>
> On Wed, Aug 24, 2016 at 11:05 AM, Bill Warshaw
> wrote:
>
> > I'd like to re-awaken this voting thread now that KIP-33 has merged.
> This
> > KIP is now completely unblocked. I have a working branch off of tru
d. From that point of view, I feel that current KIP
> still has value to be accepted.
>
> Guozhang
>
>
> On Mon, May 2, 2016 at 2:37 PM, Bill Warshaw wrote:
>
> > Yes, I'd agree that offset is a more precise configuration than
> timestamp.
> > If ther
We are running a 3-node deployment of Kafka, and on several of our testing
sites we have seen the following scenario occur:
- "auto.offset.reset" is set to "earliest"
- A client is reading from Kafka, and at some point the broker throws an
OffsetOutOfRangeException, causing the consumer to seek to
33 PM, Jay Kreps wrote:
> Gotcha, good point. But barring that limitation, you agree that that makes
> more sense?
>
> -Jay
>
> On Mon, May 2, 2016 at 2:29 PM, Bill Warshaw wrote:
>
> > The problem with offset as a config option is that offsets are
> > partition-sp
rent: in
> the proposal you have where there is a config that controls retention that
> the user would update, wouldn't it make more sense for this config to be
> based on offset rather than timestamp?
>
> -Jay
>
> On Mon, May 2, 2016 at 12:53 PM, Bill Warshaw wrote:
>
>
.
>2. Is this mechanism practical to use at scale? It requires several ZK
>writes per config change, so I guess that depends on how frequently the
>consumers would update the value and how many consumers there are...any
> thoughts on this?
>
> -Jay
>
> On Thu, Ap
e segment list may not be huge, but if it is straight-forward
> to do I'd suggest choose this option.
>
>
> Guozhang
>
>
> On Mon, May 2, 2016 at 7:15 AM, Bill Warshaw wrote:
>
> > Conditions 1, 2 and 3 will all be checked sequentially. If any of the
> > thre
016 at 8:42 PM, Gwen Shapira wrote:
> >
> > > For convenience, the KIP is here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy
> > >
> > > Do you mind updating the KI
with time formats we plan on supporting
> in the configuration?
>
> On Wed, Mar 9, 2016 at 11:44 AM, Bill Warshaw wrote:
> > Hello,
> >
> > I'd like to initiate the vote for KIP-47.
> >
> > Thanks,
> > Bill Warshaw
>
Hello,
I'd like to initiate the vote for KIP-47.
Thanks,
Bill Warshaw
deletion as well right? I think it would be
> useful for me if you or anyone else can go over the exact
> mechanics/workflow for accomplishing precise purges at today's KIP meeting.
>
> Thanks,
>
> Joel
>
> On Monday, February 22, 2016, Bill Warshaw wrote:
>
> >
you submitted KIP-33, are you actively working on that? If so, it
> > would make sense to implement KIP-47 after KIP-33 so that it works for
> both
> > CreateTime and LogAppendTime.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On
p of a segment is older than an absolute timestamp? If so,
> could you update the wiki accordingly?
>
> Thanks,
>
> Jun
>
> On Fri, Feb 19, 2016 at 2:57 PM, Bill Warshaw wrote:
>
> > Hello all,
> >
> > What is the next step with this proposal? The work for
heck that will cause a segment to be deleted
> > when the timestamp of a segment is older than an absolute timestamp? If
> so,
> > could you update the wiki accordingly?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Feb 19, 2016 at 2:57 PM, Bill Wa
is older than an absolute timestamp? If so,
> could you update the wiki accordingly?
>
> Jun
>
>
>
> On Sat, Feb 13, 2016 at 3:23 PM, Bill Warshaw wrote:
>
> > Hello,
> >
> > That is a good catch, thanks for pointing it out. If this KIP is
[
https://issues.apache.org/jira/browse/KAFKA-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Warshaw updated KAFKA-3224:
Description:
One of Kafka's officially-described use cases is a distributed commit log
is proposal
> makes
> > sense. Thx!
> >
> > On Wed, Feb 10, 2016 at 3:35 PM, Jay Kreps wrote:
> >
> >> I think this makes a lot of sense and won't be hard to implement and
> >> doesn't create too much in the way of new interfaces.
> >
; > > From: n...@confluent.io
> > > To: dev@kafka.apache.org
> > >
> > > Adding a timestamp based auto-expiration is useful and this proposal
> > makes
> > > sense. Thx!
> > >
> > > On Wed, Feb 10, 2016 at 3:35 PM, Jay Kreps wrot
Hello,
I just submitted KIP-47 for adding a new log deletion policy based on a
minimum timestamp of messages to retain.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy
I'm open to any comments or suggestions.
Thanks,
Bill Warshaw
[
https://issues.apache.org/jira/browse/KAFKA-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Warshaw updated KAFKA-3224:
Description:
One of Kafka's officially-described use cases is a distributed commit log
Bill Warshaw created KAFKA-3224:
---
Summary: Add timestamp-based log deletion policy
Key: KAFKA-3224
URL: https://issues.apache.org/jira/browse/KAFKA-3224
Project: Kafka
Issue Type: Improvement
hanks,
Bill Warshaw
On Mon, Feb 1, 2016 at 12:35 PM, Becket Qin wrote:
> Hi Bill,
>
> The PR is still under review. It might take some more time because it
> touches a bunch of files. You can watch KAFKA-3025 so once it gets closed
> you will get email notification.
> Looki
[
https://issues.apache.org/jira/browse/KAFKA-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Warshaw resolved KAFKA-3178.
-
Resolution: Won't Fix
> Expose a method in AdminUtils to manually truncate a specific p
[
https://issues.apache.org/jira/browse/KAFKA-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129513#comment-15129513
]
Bill Warshaw commented on KAFKA-3178:
-
[~vahid] fyi this ticket may end up clo
pull request?
Bill Warshaw
On Fri, Jan 22, 2016 at 12:41 AM, Becket Qin wrote:
> I agree with Guozhang that this seems better to be a separate tool.
>
> Also, I am wondering if KIP-32 can be used here. We can have a timestamp
> based compaction policy if needed, for example, keep
wdwars...@gmail.com
Bill Warshaw
Thanks!
On Mon, Feb 1, 2016 at 1:08 AM, Gwen Shapira wrote:
> What is your wiki user name?
>
> On Sat, Jan 30, 2016 at 11:35 PM, Bill Warshaw
> wrote:
>
> > Hello again,
> >
> > Is there anyone on this thread who has admin
[
https://issues.apache.org/jira/browse/KAFKA-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Warshaw updated KAFKA-3178:
Description:
h3. Description
One of Kafka's officially-described use cases is a distributed c
Bill Warshaw created KAFKA-3178:
---
Summary: Expose a method in AdminUtils to manually truncate a
specific partition to a particular offset
Key: KAFKA-3178
URL: https://issues.apache.org/jira/browse/KAFKA-3178
AFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
>
> Guozhang
>
>
>
> On Fri, Jan 22, 2016 at 10:58 AM, Bill Warshaw
> wrote:
>
> > A function such as "deleteUpToOffset(TopicPartition tp, long
> > minOffsetToRetain)" exposed through AdminUt
Hello,
I'm planning to create a new KIP, but I don't think I have the proper
permissions to actually create a child page on the Confluence wiki. Could
an administrator give me access to create pages, or am I missing something
obvious?
Thanks,
Bill Warshaw
--
<http://appianwo
rmark will effectively be a no-op, so I feel your scenario will be
> better fit as a one-time admin tool to cleanup the logs rather than
> customizing the periodic cleaning policy. Does this sound reasonable to
> you?
>
>
> Guozhang
>
>
> On Wed, Jan 20, 2016 at 7:09 PM, Bil
ng for all other properties.
On Wed, Jan 20, 2016 at 9:23 PM, Guozhang Wang wrote:
> So do you need to periodically update the key-value pairs to "advance the
> threshold for each topic"?
>
> Guozhang
>
> On Wed, Jan 20, 2016 at 5:51 PM, Bill Warshaw
> wrote:
>
&
is the latter case, how will you advance the "threshold
> transaction_id" each time when it executes?
>
> Guozhang
>
>
> On Wed, Jan 20, 2016 at 1:50 PM, Bill Warshaw
> wrote:
>
> > Damian, I appreciate your quick response.
> >
> > Our transa
would delete all messages whose key was less
than a given transaction_id that we passed in. I can imagine a wide
variety of other custom policies that could be used for retention based on
the key and value of the message.
On Wed, Jan 20, 2016 at 1:35 PM, Bill Warshaw
wrote:
> Hello,
>
> I&
f so, are there other approaches we could take that come to
mind?
Thanks for your time,
Bill Warshaw
--
<http://appianworld.com>
This message and any attachments are solely for the intended recipient. If
you are not the intended recipient, disclosure, copying, use, or
distribution of th
41 matches
Mail list logo
