[jira] [Commented] (KAFKA-8733) Offline partitions occur when leader's disk is slow in reads while responding to follower fetch requests.
[
https://issues.apache.org/jira/browse/KAFKA-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220243#comment-17220243
]
GEORGE LI commented on KAFKA-8733:
--
for lossy clusters, setting unclean.leader.election.enable=true will help. we
are also rolling out replica.lag.time.max.ms=3 .
> Offline partitions occur when leader's disk is slow in reads while responding
> to follower fetch requests.
> -
>
> Key: KAFKA-8733
> URL: https://issues.apache.org/jira/browse/KAFKA-8733
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.2, 2.4.0
>Reporter: Satish Duggana
>Assignee: Satish Duggana
>Priority: Critical
> Attachments: weighted-io-time-2.png, wio-time.png
>
>
> We found offline partitions issue multiple times on some of the hosts in our
> clusters. After going through the broker logs and hosts’s disk stats, it
> looks like this issue occurs whenever the read/write operations take more
> time on that disk. In a particular case where read time is more than the
> replica.lag.time.max.ms, follower replicas will be out of sync as their
> earlier fetch requests are stuck while reading the local log and their fetch
> status is not yet updated as mentioned in the below code of `ReplicaManager`.
> If there is an issue in reading the data from the log for a duration more
> than replica.lag.time.max.ms then all the replicas will be out of sync and
> partition becomes offline if min.isr.replicas > 1 and unclean.leader.election
> is false.
>
> {code:java}
> def readFromLog(): Seq[(TopicPartition, LogReadResult)] = {
> val result = readFromLocalLog( // this call took more than
> `replica.lag.time.max.ms`
> replicaId = replicaId,
> fetchOnlyFromLeader = fetchOnlyFromLeader,
> readOnlyCommitted = fetchOnlyCommitted,
> fetchMaxBytes = fetchMaxBytes,
> hardMaxBytesLimit = hardMaxBytesLimit,
> readPartitionInfo = fetchInfos,
> quota = quota,
> isolationLevel = isolationLevel)
> if (isFromFollower) updateFollowerLogReadResults(replicaId, result). //
> fetch time gets updated here, but mayBeShrinkIsr should have been already
> called and the replica is removed from isr
> else result
> }
> val logReadResults = readFromLog()
> {code}
> Attached the graphs of disk weighted io time stats when this issue occurred.
> I will raise [KIP-501|https://s.apache.org/jhbpn] describing options on how
> to handle this scenario.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kafka] RamanVerma commented on a change in pull request #9364: KAFKA-10471 Mark broker crash during log loading as unclean shutdown
RamanVerma commented on a change in pull request #9364:
URL: https://github.com/apache/kafka/pull/9364#discussion_r511545890
##
File path: core/src/test/scala/unit/kafka/log/LogTest.scala
##
@@ -1257,7 +1328,7 @@ class LogTest {
log.close()
// After reloading log, producer state should not be regenerated
-val reloadedLog = createLog(logDir, logConfig, logStartOffset = 1L)
+val reloadedLog = createLog(logDir, logConfig, logStartOffset = 1L,
lastShutdownClean = false)
Review comment:
Jun, this test was not creating a clean shut down file before opening
the log again. So, it would have been going through log recovery code path
earlier as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] RamanVerma commented on a change in pull request #9364: KAFKA-10471 Mark broker crash during log loading as unclean shutdown
RamanVerma commented on a change in pull request #9364:
URL: https://github.com/apache/kafka/pull/9364#discussion_r511545712
##
File path: core/src/test/scala/unit/kafka/log/LogTest.scala
##
@@ -2131,12 +2202,12 @@ class LogTest {
assertEquals("Should have same number of time index entries as before.",
numTimeIndexEntries, log.activeSegment.timeIndex.entries)
}
-log = createLog(logDir, logConfig, recoveryPoint = lastOffset)
+log = createLog(logDir, logConfig, recoveryPoint = lastOffset,
lastShutdownClean = false)
Review comment:
Jun, this test was not creating a clean shut down file before opening
the log again. So, it would have gone through the recovery path. Hence, I have
set `lastShutdownClean` parameter to `false`. Similarly, for line 2210.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #9374: MINOR: Fix NPE in KafkaAdminClient.describeUserScramCredentials
chia7712 commented on pull request #9374: URL: https://github.com/apache/kafka/pull/9374#issuecomment-716090509 We can merge this patch so as to avoid making a release with (known) unavailable Public APIs. Honoring the protocol (null handle) on server-side can be a follow-up. Just my two cents. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-7870) Error sending fetch request (sessionId=1578860481, epoch=INITIAL) to node 2: java.io.IOException: Connection to 2 was disconnected before the response was read.
[
https://issues.apache.org/jira/browse/KAFKA-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220237#comment-17220237
]
Eran Levy commented on KAFKA-7870:
--
Experiencing that with 2.6.0
{"timestamp":"2020-10-24T17:40:26.019Z","level":"INFO","thread":"-ecc9d1ed-83e7-4d10-882e-12ed84aa1d84-StreamThread-1","logger":"org.apache.kafka.clients.FetchSessionHandler","message":"[Consumer
clientId=...-ecc9d1ed-83e7-4d10-882e-12ed84aa1d84-StreamThread-1-restore-consumer,
groupId=null] Error sending fetch request (sessionId=1026275111, epoch=10044)
to node
2:","context":"default","exception":"org.apache.kafka.common.errors.DisconnectException:
null\n"}
The only thing that looks suspicious is that leaders and partitions aren't
balanced between the brokers - node 2 is that one that isn't balanced at all.
All metrics show that this broker is healthy but it do looks like it has only
10% of partitions and leaders which is very suspicious.
Any idea?
> Error sending fetch request (sessionId=1578860481, epoch=INITIAL) to node 2:
> java.io.IOException: Connection to 2 was disconnected before the response was
> read.
>
>
> Key: KAFKA-7870
> URL: https://issues.apache.org/jira/browse/KAFKA-7870
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 2.1.0
>Reporter: Chakhsu Lau
>Priority: Blocker
>
> We build a kafka cluster with 5 brokers. But one of brokers suddenly stopped
> running during the run. And it happened twice in the same broker. Here is the
> log and is this a bug in kafka ?
> {code:java}
> [2019-01-25 12:57:14,686] INFO [ReplicaFetcher replicaId=3, leaderId=2,
> fetcherId=0] Error sending fetch request (sessionId=1578860481,
> epoch=INITIAL) to node 2: java.io.IOException: Connection to 2 was
> disconnected before the response was read.
> (org.apache.kafka.clients.FetchSessionHandler)
> [2019-01-25 12:57:14,687] WARN [ReplicaFetcher replicaId=3, leaderId=2,
> fetcherId=0] Error in response for fetch request (type=FetchRequest,
> replicaId=3, maxWait=500, minBytes=1, maxBytes=10485760,
> fetchData={api-result-bi-heatmap-8=(offset=0, logStartOffset=0,
> maxBytes=1048576, currentLeaderEpoch=Optional[4]),
> api-result-bi-heatmap-save-12=(offset=0, logStartOffset=0, maxBytes=1048576,
> currentLeaderEpoch=Optional[4]), api-result-bi-heatmap-task-2=(offset=2,
> logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[4]),
> api-result-bi-flow-39=(offset=1883206, logStartOffset=0, maxBytes=1048576,
> currentLeaderEpoch=Optional[4]), __consumer_offsets-47=(offset=349437,
> logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[4]),
> api-result-bi-heatmap-track-6=(offset=1039889, logStartOffset=0,
> maxBytes=1048576, currentLeaderEpoch=Optional[4]),
> api-result-bi-heatmap-task-17=(offset=0, logStartOffset=0, maxBytes=1048576,
> currentLeaderEpoch=Optional[4]), __consumer_offsets-2=(offset=0,
> logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[4]),
> api-result-bi-heatmap-aggs-19=(offset=1255056, logStartOffset=0,
> maxBytes=1048576, currentLeaderEpoch=Optional[4])},
> isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1578860481,
> epoch=INITIAL)) (kafka.server.ReplicaFetcherThread)
> java.io.IOException: Connection to 2 was disconnected before the response was
> read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
> at
> kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130)
> at
> kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kafka] rondagostino commented on pull request #9374: MINOR: Fix NPE in KafkaAdminClient.describeUserScramCredentials
rondagostino commented on pull request #9374: URL: https://github.com/apache/kafka/pull/9374#issuecomment-716065197 This isn’t a regression, so I don’t think it is a blocker. NPE would only occur if someone were to code against the new API. The kafka-configs.sh CLI works fine because it doesn’t exercise the API in a way that generates the NPE. Would I like to t this fix in? Yes. But is it a blocker? I don’t think so. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] rondagostino edited a comment on pull request #9374: MINOR: Fix NPE in KafkaAdminClient.describeUserScramCredentials
rondagostino edited a comment on pull request #9374: URL: https://github.com/apache/kafka/pull/9374#issuecomment-716065197 This isn’t a regression, so I don’t think it is a blocker. NPE would only occur if someone were to code against the new API. The kafka-configs.sh CLI works fine because it doesn’t exercise the API in a way that generates the NPE. Would I like to get this fix in? Yes. But is it a blocker? I don’t think so. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #9374: MINOR: Fix NPE in KafkaAdminClient.describeUserScramCredentials
dajac commented on pull request #9374: URL: https://github.com/apache/kafka/pull/9374#issuecomment-716061853 @rondagostino Is this issue a blocker for 2.7? cc @bbejeck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] afalko commented on pull request #8794: KAFKA-10092: Remove unused code branches in NioEchoServer
afalko commented on pull request #8794: URL: https://github.com/apache/kafka/pull/8794#issuecomment-716058925 > @afalko Could you rebase PR to trigger QA? Thanks! Looks like it partially passed. I'll merge master against and see if it does better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy commented on pull request #9496: MINOR: fix error in quota_test.py system tests
omkreddy commented on pull request #9496: URL: https://github.com/apache/kafka/pull/9496#issuecomment-716048849 cc @rondagostino This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy opened a new pull request #9496: MINOR: fix error in quota_test.py system tests
omkreddy opened a new pull request #9496: URL: https://github.com/apache/kafka/pull/9496 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy commented on pull request #9480: KAFKA-10592: Fix vagrant for a system tests with python3
omkreddy commented on pull request #9480: URL: https://github.com/apache/kafka/pull/9480#issuecomment-716045146 Merging to trunk and 2.7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy closed pull request #9480: KAFKA-10592: Fix vagrant for a system tests with python3
omkreddy closed pull request #9480: URL: https://github.com/apache/kafka/pull/9480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KAFKA-10641) ACL Command hangs with SSL as not existing with proper error code
[ https://issues.apache.org/jira/browse/KAFKA-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Senthilnathan Muthusamy updated KAFKA-10641: Summary: ACL Command hangs with SSL as not existing with proper error code (was: ACL Command hands with SSL as not existing with proper error code) > ACL Command hangs with SSL as not existing with proper error code > - > > Key: KAFKA-10641 > URL: https://issues.apache.org/jira/browse/KAFKA-10641 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2, > 2.4.0, 2.3.1, 2.5.0, 2.4.1, 2.6.0, 2.5.1 >Reporter: Senthilnathan Muthusamy >Assignee: Senthilnathan Muthusamy >Priority: Minor > Fix For: 2.7.0 > > > When using ACL Command with SSL mode, the process is not terminating after > successful ACL operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] senthilm-ms opened a new pull request #9495: KAFKA-10642: Expose the real stack trace if any exception occurred during SSL Client Trust Verification in extension
senthilm-ms opened a new pull request #9495: URL: https://github.com/apache/kafka/pull/9495 If there is any exception occurred in the custom implementation of client trust verification (i.e. using security.provider), the inner exception is suppressed or hidden and not logged to the log file... @junrao @mjsax @guozhangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-10642) Expose the real stack trace if any exception occurred during SSL Client Trust Verification in extension
Senthilnathan Muthusamy created KAFKA-10642: --- Summary: Expose the real stack trace if any exception occurred during SSL Client Trust Verification in extension Key: KAFKA-10642 URL: https://issues.apache.org/jira/browse/KAFKA-10642 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 2.5.1, 2.6.0, 2.4.1, 2.5.0, 2.3.1, 2.4.0, 2.3.0 Reporter: Senthilnathan Muthusamy Assignee: Senthilnathan Muthusamy Fix For: 2.7.0 If there is any exception occurred in the custom implementation of client trust verification (i.e. using security.provider), the inner exception is suppressed or hidden and not logged to the log file... Below is an example stack trace not showing actual exception from the extension/custom implementation. [2020-05-13 14:30:26,892] ERROR [KafkaServer id=423810470] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)[2020-05-13 14:30:26,892] ERROR [KafkaServer id=423810470] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: org.apache.kafka.common.config.ConfigException: Invalid value java.lang.RuntimeException: Delegated task threw Exception/Error for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings. at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85) at kafka.network.Processor.(SocketServer.scala:753) at kafka.network.SocketServer.newProcessor(SocketServer.scala:394) at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:279) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:278) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:241) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:238) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:238) at kafka.network.SocketServer.startup(SocketServer.scala:121) at kafka.server.KafkaServer.startup(KafkaServer.scala:265) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:84) at kafka.Kafka.main(Kafka.scala)Caused by: org.apache.kafka.common.config.ConfigException: Invalid value java.lang.RuntimeException: Delegated task threw Exception/Error for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings. at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ... 18 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] joshuagrisham commented on pull request #9493: KAFKA-10640: Add recursive support to Connect Cast and ReplaceField transforms, and support for casting complex types to either a native
joshuagrisham commented on pull request #9493: URL: https://github.com/apache/kafka/pull/9493#issuecomment-716041356 I cleaned up most of the checkstyle issues before creating this new PR but it is still failing in 3 places due to CyclomaticComplexity failure. This is basically because the code is recursive and everything has been set up in "pieces" that can be called recursively so sort of through design it can recurs quite deeply into different methods. So to get past this check, unfortunately I will need to refactor it a bit, to consolidate some of the functionality into larger methods or figure out a slightly different solution. Unless anyone else has a better idea... ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] senthilm-ms commented on pull request #9494: KAFKA-10641: ACL Command Exit properly always with error code
senthilm-ms commented on pull request #9494: URL: https://github.com/apache/kafka/pull/9494#issuecomment-716040837 @junrao @mjsax @guozhangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] senthilm-ms opened a new pull request #9494: KAFKA-10641: ACL Command Exit properly always with error code
senthilm-ms opened a new pull request #9494: URL: https://github.com/apache/kafka/pull/9494 When using ACL Command with SSL mode, the process is not terminating after successful ACL operation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-10641) ACL Command hands with SSL as not existing with proper error code
Senthilnathan Muthusamy created KAFKA-10641: --- Summary: ACL Command hands with SSL as not existing with proper error code Key: KAFKA-10641 URL: https://issues.apache.org/jira/browse/KAFKA-10641 Project: Kafka Issue Type: Bug Components: core Affects Versions: 2.5.1, 2.6.0, 2.4.1, 2.5.0, 2.3.1, 2.4.0, 2.2.2, 2.2.1, 2.3.0, 2.1.1, 2.2.0, 2.1.0, 2.0.1, 2.0.0 Reporter: Senthilnathan Muthusamy Assignee: Senthilnathan Muthusamy Fix For: 2.7.0 When using ACL Command with SSL mode, the process is not terminating after successful ACL operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] joshuagrisham commented on pull request #9470: Add recursive support to Connect Cast and ReplaceField transforms, and support for casting complex types to either a native or JSON stri
joshuagrisham commented on pull request #9470: URL: https://github.com/apache/kafka/pull/9470#issuecomment-716030144 Closing this PR in favor of #9493 -- I had to do some cleanup and move this out of my repo's trunk branch and into its own branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] joshuagrisham closed pull request #9470: Add recursive support to Connect Cast and ReplaceField transforms, and support for casting complex types to either a native or JSON string.
joshuagrisham closed pull request #9470: URL: https://github.com/apache/kafka/pull/9470 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] joshuagrisham opened a new pull request #9492: KAFKA-10627: Added support for Connect TimestampConverter to convert multiple fields using a comma-separated list, and changed the Strin
joshuagrisham opened a new pull request #9492:
URL: https://github.com/apache/kafka/pull/9492
I have made an update to **TimestampConverter** Connect transform to address
the main issues that I logged in
[KAFKA-10627](https://issues.apache.org/jira/browse/KAFKA-10627).
I realized that kafka is using `java.util.Date` everywhere and as part of
its core types (including in Schemas, values, etc). In theory it would be good
over time to upgrade to `java.time` classes but on first reflection it seems
like quite a big overhaul to do this.
So instead I focused on the specific problem at hand: parsing strings into
`Date` where the strings can come in different formats. So for this part alone
I changed to use `DateTimeFormatter` so we can use multiple patterns to match
input strings and convert them to a `java.util.Date` after.
I also updated some of the way the Config parameters and values work, to
bring in line with the other classes and similar to what I did with #9470.
String Input and Output Timestamp Format updates
Because now for input formats we allow multiple different possibilities
using pattern matching, this does not work for the output format of a Timestamp
to a String (which was another possibility of this transform). So I have
changed the configuration a bit... now there are three parameters:
- `format` which is the original one. You can still use this one, and it
will set both input (parsing) and output (Date/Timestamp to string format)
based on this format.
- `format.input` is a new parameter, where you can specify a
DateTimeFormatter-compatible pattern string that supports multiple different
formats in case you have a mix in your data. For just one example, now you can
use something like this as `format.input` and it will catch a lot of different
variations which you might see in one timestamp field: `"[-MM-dd[['T'][
]HH:mm:ss[.SSSz][.SSS[XXX][X"`
- `format.output` is a new parameter which only controls the output of a
Date/Timestamp to target type of `string`. This is the same as before and still
uses `SimpleDateFormat` to create the output string, it is just controlled in a
separate parameter now.
I also added some code which checks the value of each of these three.
Basically it forces you to use either `format`, or one or both of the new
parameters -- you cannot mix the old and new together. In the end,
`format.input` and `format.output` are the ones used in the rest of the logic,
but the code first compares `format` against these values and sets the value
for both of the new parameters depending on what was sent in the config.
Support for multiple fields instead of one single field
I changed the `field` parameter to now be called `fields` and supports
multiple values as a comma-separated list. I used this new
`ConfigUtils.translateDeprecatedConfigs` method to provide automatic
translation of of the old parameter to the new one as well.
With this change I also updated the `apply` methods so that they loop
through each field and check against the list of `fields`. Now you can specify
a comma-separated list of multiple fields to have the same input format/output
type applied.
Unit tests have been added for both new updates (string formatting and
multiple field support).
As I looked at this one then I realized that maybe it would be good to add
`recursive` support similar to what I have done in #9470 but I guess that can
come at another day!
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-8733) Offline partitions occur when leader's disk is slow in reads while responding to follower fetch requests.
[
https://issues.apache.org/jira/browse/KAFKA-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220139#comment-17220139
]
Ming Liu commented on KAFKA-8733:
-
One observation is after moving to 2.5 (so
_[replica.lag.time.max.ms|http://replica.lag.time.max.ms/]_ is changed from 10
second to 30 seconds), Offline partitions when leader's disk went bad occurs
much less frequently.
> Offline partitions occur when leader's disk is slow in reads while responding
> to follower fetch requests.
> -
>
> Key: KAFKA-8733
> URL: https://issues.apache.org/jira/browse/KAFKA-8733
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.2, 2.4.0
>Reporter: Satish Duggana
>Assignee: Satish Duggana
>Priority: Critical
> Attachments: weighted-io-time-2.png, wio-time.png
>
>
> We found offline partitions issue multiple times on some of the hosts in our
> clusters. After going through the broker logs and hosts’s disk stats, it
> looks like this issue occurs whenever the read/write operations take more
> time on that disk. In a particular case where read time is more than the
> replica.lag.time.max.ms, follower replicas will be out of sync as their
> earlier fetch requests are stuck while reading the local log and their fetch
> status is not yet updated as mentioned in the below code of `ReplicaManager`.
> If there is an issue in reading the data from the log for a duration more
> than replica.lag.time.max.ms then all the replicas will be out of sync and
> partition becomes offline if min.isr.replicas > 1 and unclean.leader.election
> is false.
>
> {code:java}
> def readFromLog(): Seq[(TopicPartition, LogReadResult)] = {
> val result = readFromLocalLog( // this call took more than
> `replica.lag.time.max.ms`
> replicaId = replicaId,
> fetchOnlyFromLeader = fetchOnlyFromLeader,
> readOnlyCommitted = fetchOnlyCommitted,
> fetchMaxBytes = fetchMaxBytes,
> hardMaxBytesLimit = hardMaxBytesLimit,
> readPartitionInfo = fetchInfos,
> quota = quota,
> isolationLevel = isolationLevel)
> if (isFromFollower) updateFollowerLogReadResults(replicaId, result). //
> fetch time gets updated here, but mayBeShrinkIsr should have been already
> called and the replica is removed from isr
> else result
> }
> val logReadResults = readFromLog()
> {code}
> Attached the graphs of disk weighted io time stats when this issue occurred.
> I will raise [KIP-501|https://s.apache.org/jhbpn] describing options on how
> to handle this scenario.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kafka] omkreddy closed pull request #9490: Pin ducktape to version 0.7.10
omkreddy closed pull request #9490: URL: https://github.com/apache/kafka/pull/9490 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy commented on pull request #9490: Pin ducktape to version 0.7.10
omkreddy commented on pull request #9490: URL: https://github.com/apache/kafka/pull/9490#issuecomment-715967601 Merging to 2.6 and below. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy commented on pull request #9490: Pin ducktape to version 0.7.10
omkreddy commented on pull request #9490: URL: https://github.com/apache/kafka/pull/9490#issuecomment-715967307 test results: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2020-10-24--001.1603547932--stan-confluent--ducktape-710-26--c39b5894d/report.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-10640) Connect Cast and ReplaceField transforms only support top-level fields and Cast does not support complex types.
Joshua Grisham created KAFKA-10640: -- Summary: Connect Cast and ReplaceField transforms only support top-level fields and Cast does not support complex types. Key: KAFKA-10640 URL: https://issues.apache.org/jira/browse/KAFKA-10640 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Joshua Grisham When working with anything but the most simple of messages, you will inevitably run into nested structures and nested fields within other structures. These two transforms are very important when working with data and if we want to use Connect for anything except the most simple scenarios then they should support working with more complicated structures. Also, in case the messages have more complicated types such as Arrays or Maps that are not supported by the Flatten transform, it is not possible to do many "standard" feeling things with the messages such as using a JdbcSinkConnector to put the data into a database and use it in a database-driven system. So it would be good if at a minimum we can Cast complex types as a string so that they can at least be sent along and stored in the database and then parsed later. This is opposed to them just being completely blocked like is the current state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] dajac closed pull request #3051: KAFKA-5235: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimized number of requests to server
dajac closed pull request #3051: URL: https://github.com/apache/kafka/pull/3051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #3051: KAFKA-5235: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimized number of requests to server
dajac commented on pull request #3051: URL: https://github.com/apache/kafka/pull/3051#issuecomment-715787837 Closing as this is addressed in https://github.com/apache/kafka/pull/9430 (KIP-635). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dajac closed pull request #5577: Update InFlightRequests.java
dajac closed pull request #5577: URL: https://github.com/apache/kafka/pull/5577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #5577: Update InFlightRequests.java
dajac commented on pull request #5577: URL: https://github.com/apache/kafka/pull/5577#issuecomment-715767796 Thanks for the PR. "iff" is actually correct here. It means "if and only if". Closing the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dajac closed pull request #7134: Test1
dajac closed pull request #7134: URL: https://github.com/apache/kafka/pull/7134 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
