[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575850#comment-15575850
]
Jeff Widman commented on KAFKA-4254:
Does this affect 10.0 or just 10.1?
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.0
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574020#comment-15574020
]
ASF GitHub Bot commented on KAFKA-4254:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/1995
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.0
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557526#comment-15557526
]
Konstantine Karantasis commented on KAFKA-4254:
---
A patch has been submitted, taking into account both the main considerations
that were raised.
A single metadata request is issued if a partition higher than the upper bound
of partition range is requested, and an UnknownTopicOrPartitionException is
thrown instead of IllegalArgumentException when a requested partition is out of
bounds.
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557508#comment-15557508
]
ASF GitHub Bot commented on KAFKA-4254:
---
GitHub user kkonstantine opened a pull request:
https://github.com/apache/kafka/pull/1995
KAFKA-4254: Update producers metadata before failing on non-existent …
…partition
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kkonstantine/kafka
KAFKA-4254-Update-producers-metadata-before-failing-on-non-existent-partition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1995.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1995
commit d6ea6b77b879ccd5b2d4980845b991052fd09b88
Author: Konstantine Karantasis
Date: 2016-10-08T08:00:00Z
KAFKA-4254: Update producers metadata before failing on non-existent
partition
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553410#comment-15553410
]
Jason Gustafson commented on KAFKA-4254:
[~jjkoshy] Yeah, good point.
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553228#comment-15553228
]
Joel Koshy commented on KAFKA-4254:
---
It seems (2) would still help as there are use-cases which set {{max.block.ms}}
to zero. So we can refresh metadata but also return a more specific exception.
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549298#comment-15549298
]
Ismael Juma commented on KAFKA-4254:
Yeah, one refresh makes sense to me.
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549204#comment-15549204
]
Jason Gustafson commented on KAFKA-4254:
[~ijuma] I'm sympathetic to your concern, but it seems a little difficult to
justify the current behavior of raising an exception based on stale metadata.
As a user, I expect that I can produce to an existing partition even if it has
just been created, so requiring me to catch exceptions until the producer's
metadata refresh interval expires seems a bit clunky. Of course, the producer
doesn't necessarily need to block the full {{max.block.ms}} until the partition
is created, maybe one metadata refresh prior to raising would be sufficient?
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549203#comment-15549203
]
Jason Gustafson commented on KAFKA-4254:
[~ijuma] I'm sympathetic to your concern, but it seems a little difficult to
justify the current behavior of raising an exception based on stale metadata.
As a user, I expect that I can produce to an existing partition even if it has
just been created, so requiring me to catch exceptions until the producer's
metadata refresh interval expires seems a bit clunky. Of course, the producer
doesn't necessarily need to block the full {{max.block.ms}} until the partition
is created, maybe one metadata refresh prior to raising would be sufficient?
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
[
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547518#comment-15547518
]
Ismael Juma commented on KAFKA-4254:
Do we really want to have more cases where we fail slow on the off chance that
some very rare operation is taking place? Adding partitions to a topic is
pretty rare, so I think the second option may be better.
> Questionable handling of unknown partitions in KafkaProducer
>
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user
> attempts to write to a partition which has just been created. This is caused
> by the fact that the producer does not attempt to refetch topic metadata in
> this case, which means that its check for partition validity is based on
> stale metadata.
> If the topic for the partition did not already exist, it works fine because
> the producer will block until it has metadata for the topic, so this case is
> primarily hit when the number of partitions is dynamically increased.
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the
> partition doesn't exist, we refetch metadata and try again (timing out when
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle
> the error. Raising {{IllegalArgumentException}} is not helpful in practice
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to
> tell the user that the partition is invalid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
