[jira] [Created] (KAFKA-14146) KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Alexandre Garnier created KAFKA-14146: - Summary: KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer Key: KAFKA-14146 URL: https://issues.apache.org/jira/browse/KAFKA-14146 Project: Kafka Issue Type: Improvement Components: tools Affects Versions: 3.2.1 Reporter: Alexandre Garnier Assignee: Alexandre Garnier Jira for [KIP-840|https://cwiki.apache.org/confluence/x/bBqhD] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14099) No REST API request logs in Kafka connect
Alexandre Garnier created KAFKA-14099:
-
Summary: No REST API request logs in Kafka connect
Key: KAFKA-14099
URL: https://issues.apache.org/jira/browse/KAFKA-14099
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.2.0
Reporter: Alexandre Garnier
Prior to 2.2.1, when an REST API request was performed, there was a request log
in the log file:
{code}
[2022-07-23 07:18:16,128] INFO 172.18.0.1 - - [23/Jul/2022:07:18:16 +] "GET
/connectors HTTP/1.1" 200 2 "-" "curl/7.81.0" 66
(org.apache.kafka.connect.runtime.rest.RestServer:62)
{code}
With a bisect, I found the problem comes from [PR
6651|https://github.com/apache/kafka/pull/6651] to fix KAFKA-8304
>From what I understand of the problem, the ContextHandlerCollection is added
>in the Server
>(https://github.com/dongjinleekr/kafka/blob/63a6130af30536d67fca5802005695a84c875b5e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java#L195)
> before handlers are really added in the ContextHandlerCollection
>(https://github.com/dongjinleekr/kafka/blob/63a6130af30536d67fca5802005695a84c875b5e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java#L296).
I don't know the impact on other handlers, but clearly it doesn't work for the
RequestLogHandler
A solution I found for the logging issue is to set the RequestLog directly in
the server without using an handlers:
{code}
diff --git
i/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
w/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
index ab18419efc..4d09cc0e6c 100644
---
i/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
+++
w/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
@@ -187,6 +187,11 @@ public class RestServer {
public void initializeServer() {
log.info("Initializing REST server");
+Slf4jRequestLogWriter slf4jRequestLogWriter = new
Slf4jRequestLogWriter();
+
slf4jRequestLogWriter.setLoggerName(RestServer.class.getCanonicalName());
+CustomRequestLog requestLog = new
CustomRequestLog(slf4jRequestLogWriter, CustomRequestLog.EXTENDED_NCSA_FORMAT +
" %{ms}T");
+jettyServer.setRequestLog(requestLog);
+
/* Needed for graceful shutdown as per `setStopTimeout` documentation
*/
StatisticsHandler statsHandler = new StatisticsHandler();
statsHandler.setHandler(handlers);
@@ -275,14 +280,7 @@ public class RestServer {
configureHttpResponsHeaderFilter(context);
}
-RequestLogHandler requestLogHandler = new RequestLogHandler();
-Slf4jRequestLogWriter slf4jRequestLogWriter = new
Slf4jRequestLogWriter();
-
slf4jRequestLogWriter.setLoggerName(RestServer.class.getCanonicalName());
-CustomRequestLog requestLog = new
CustomRequestLog(slf4jRequestLogWriter, CustomRequestLog.EXTENDED_NCSA_FORMAT +
" %{ms}T");
-requestLogHandler.setRequestLog(requestLog);
-
contextHandlers.add(new DefaultHandler());
-contextHandlers.add(requestLogHandler);
handlers.setHandlers(contextHandlers.toArray(new Handler[0]));
try {
{code}
Same issue raised on StackOverflow:
https://stackoverflow.com/questions/67699702/no-rest-api-logs-in-kafka-connect
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hello! A little ping on this vote. Thanks. Le jeu. 16 juin 2022 à 16:36, Alexandre Garnier a écrit : > Hi everyone. > > Anyone wants to give a last binding vote for this KIP? > > Thanks. > > Le mar. 7 juin 2022 à 14:53, Alexandre Garnier a > écrit : > >> Hi! >> >> A little reminder to vote for this KIP. >> >> Thanks. >> >> >> Le mer. 1 juin 2022 à 10:58, Alexandre Garnier a >> écrit : >> > >> > Hi everyone! >> > >> > I propose to start voting for KIP-840: >> > https://cwiki.apache.org/confluence/x/bBqhD >> > >> > Thanks, >> > -- >> > Alex >> >
Re: [VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hi everyone. Anyone wants to give a last binding vote for this KIP? Thanks. Le mar. 7 juin 2022 à 14:53, Alexandre Garnier a écrit : > Hi! > > A little reminder to vote for this KIP. > > Thanks. > > > Le mer. 1 juin 2022 à 10:58, Alexandre Garnier a écrit > : > > > > Hi everyone! > > > > I propose to start voting for KIP-840: > > https://cwiki.apache.org/confluence/x/bBqhD > > > > Thanks, > > -- > > Alex >
Re: [DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hi Luke, Thanks for the feedback. 1. Usually, in "Proposed Changes" section, we won't put PR link there. > Maybe you can take KIP-824 as a reference. > I don't know what to add more here than "Add the new option", it's a really simple straightforward modification. 2. Also, it'd be better you can provide some example reader-config and > formatter-config file. > And how they work within the script. > I did add examples. 3. If user provide both `--reader-config or --formatter-config` and > `--property` at the same time, how will we handle this case? > Could you add that into the KIP? > This was covered by the sentence "As for --producer-property/--consumer-property with --consumer.config/--producer.config, any value from option --property would override value from config file." I did add an example for this situation. Le jeu. 9 juin 2022 à 05:21, Luke Chen a écrit : > Hi Alexandre, > > Thanks for the KIP. > > Some comments: > > 1. Usually, in "Proposed Changes" section, we won't put PR link there. > Maybe you can take KIP-824 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-824%3A+Allowing+dumping+segmentlogs+limiting+the+batches+in+the+output> > as a reference. > > 2. Also, it'd be better you can provide some example reader-config and > formatter-config file. > And how they work within the script. > > 3. If user provide both `--reader-config or --formatter-config` and > `--property` at the same time, how will we handle this case? > Could you add that into the KIP? > > Thank you. > Luke > > > On Sun, May 29, 2022 at 2:32 AM Alexandre Garnier > wrote: > >> Hi! >> >> Thanks for the feedback. >> It's a good point, I updated KIP accordingly and did put the >> dot-separated option in rejected alternatives. >> >> Le ven. 27 mai 2022 à 10:22, deng ziming a >> écrit : >> > >> > Thanks for the KIP, this is a good improvement. I only have one minor >> suggestion. >> > >> > Currently many command line tools supports config file argument, but >> their name style is not unified, for example, most newly added tools are >> using --command-config, but ConsoleConsumer use —consumer.config。 I think >> we should unify the naming style from now on, I recommend us to use >> --reader-config and --formatter-config for the newly added arguments. >> > >> > -- >> > Best, >> > Ziming >> > >> > >> > > On May 26, 2022, at 4:36 PM, Alexandre Garnier >> wrote: >> > > >> > > Hello everyone, >> > > >> > > Any feedback on this KIP https://cwiki.apache.org/confluence/x/bBqhD? >> > > It is a straightforward improvement without any impact on existing >> users, >> > > so not much to discuss besides maybe the option name. >> > > >> > > -- >> > > Alex >> > > >> > > >> > > Le mer. 18 mai 2022 à 10:44, Alexandre Garnier a >> écrit : >> > > >> > >> Hi everyone, >> > >> >> > >> I created a KIP to add a config file option of reader/formatter for >> > >> kafka-console-(consumer|producer).sh tools. >> > >> https://cwiki.apache.org/confluence/x/bBqhD >> > >> >> > >> Thanks for your feedback, >> > >> -- >> > >> Alex >> > >> >> > >> >
Re: [VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hi! A little reminder to vote for this KIP. Thanks. Le mer. 1 juin 2022 à 10:58, Alexandre Garnier a écrit : > > Hi everyone! > > I propose to start voting for KIP-840: > https://cwiki.apache.org/confluence/x/bBqhD > > Thanks, > -- > Alex
[VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hi everyone! I propose to start voting for KIP-840: https://cwiki.apache.org/confluence/x/bBqhD Thanks, -- Alex
Re: [DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hi! Thanks for the feedback. It's a good point, I updated KIP accordingly and did put the dot-separated option in rejected alternatives. Le ven. 27 mai 2022 à 10:22, deng ziming a écrit : > > Thanks for the KIP, this is a good improvement. I only have one minor > suggestion. > > Currently many command line tools supports config file argument, but their > name style is not unified, for example, most newly added tools are using > --command-config, but ConsoleConsumer use —consumer.config。 I think we should > unify the naming style from now on, I recommend us to use --reader-config and > --formatter-config for the newly added arguments. > > -- > Best, > Ziming > > > > On May 26, 2022, at 4:36 PM, Alexandre Garnier wrote: > > > > Hello everyone, > > > > Any feedback on this KIP https://cwiki.apache.org/confluence/x/bBqhD? > > It is a straightforward improvement without any impact on existing users, > > so not much to discuss besides maybe the option name. > > > > -- > > Alex > > > > > > Le mer. 18 mai 2022 à 10:44, Alexandre Garnier a écrit : > > > >> Hi everyone, > >> > >> I created a KIP to add a config file option of reader/formatter for > >> kafka-console-(consumer|producer).sh tools. > >> https://cwiki.apache.org/confluence/x/bBqhD > >> > >> Thanks for your feedback, > >> -- > >> Alex > >> >
Re: [DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hello everyone, Any feedback on this KIP https://cwiki.apache.org/confluence/x/bBqhD? It is a straightforward improvement without any impact on existing users, so not much to discuss besides maybe the option name. -- Alex Le mer. 18 mai 2022 à 10:44, Alexandre Garnier a écrit : > Hi everyone, > > I created a KIP to add a config file option of reader/formatter for > kafka-console-(consumer|producer).sh tools. > https://cwiki.apache.org/confluence/x/bBqhD > > Thanks for your feedback, > -- > Alex >
Permission request to contribute
Hi there! I'd like to request permission to contribute to Kafka. Wiki ID: zigarn Jira ID: zigarn Thanks, -- Alex
Permission request to contribute
Hi there! I'd like to request permission to contribute to Kafka. Wiki ID: zigarn Jira ID: zigarn Thanks, -- Alex
[DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
Hi everyone, I created a KIP to add a config file option of reader/formatter for kafka-console-(consumer|producer).sh tools. https://cwiki.apache.org/confluence/x/bBqhD Thanks for your feedback, -- Alex
Permission request to contribute
Hi there! I'd like to request permission to contribute to Kafka. Wiki ID: zigarn Jira ID: zigarn Thanks, -- Alex
