[GitHub] [kafka] OneCricketeer commented on pull request #7965: KAFKA-9436: New Kafka Connect SMT for plainText => Struct(or Map)

2021-12-17 Thread GitBox 


OneCricketeer commented on pull request #7965:
URL: https://github.com/apache/kafka/pull/7965#issuecomment-996944487


   > suggest using grok patterns
   
   I knew I'd seen this somewhere before, but finally found it again
   
   https://github.com/streamthoughts/kafka-connect-transform-grok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] OneCricketeer commented on pull request #7965: KAFKA-9436: New Kafka Connect SMT for plainText => Struct(or Map)

2021-12-12 Thread GitBox 


OneCricketeer commented on pull request #7965:
URL: https://github.com/apache/kafka/pull/7965#issuecomment-991934558


   > a custom value.converter may work. 
   
   
https://www.confluent.io/blog/kafka-connect-single-message-transformation-tutorial-with-examples/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] OneCricketeer commented on pull request #7965: KAFKA-9436: New Kafka Connect SMT for plainText => Struct(or Map)

2021-12-12 Thread GitBox 


OneCricketeer commented on pull request #7965:
URL: https://github.com/apache/kafka/pull/7965#issuecomment-991931167


   The class provided needs to be part of consumer code and must be 
`value.deserializer` not the Connect API since it's not a converter. You said 
you're consuming strings, not JSON, which would imply you're using 
StringConverter already, rather than JSONConverter or something else 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] OneCricketeer commented on pull request #7965: KAFKA-9436: New Kafka Connect SMT for plainText => Struct(or Map)

2021-12-12 Thread GitBox 


OneCricketeer commented on pull request #7965:
URL: https://github.com/apache/kafka/pull/7965#issuecomment-991928166


   Well, Kafka record have their own timestamps since version 0.10.2, so they 
shouldn't need to be part of the record string... 
   
   > no idea how or where to begin with that 
   
   Assuming Java, then
   
   ```
   public class MyDeserializer extends StringDeserializer {
 private int dateLength = "mmdd".length; // for example 
 @Override
 public void deserialize(String topic, byte[] data) {
   String s = super.deserialize(data);
   return s.substring(dateLength);
} 
   }
   ```
   
   In the consumer config, set `value.deserializer` to `MyDeserializer.class`
   
   In your poll loop, accept strings and use a JSON library to further process 
the record 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka] OneCricketeer commented on pull request #7965: KAFKA-9436: New Kafka Connect SMT for plainText => Struct(or Map)

2021-12-12 Thread GitBox 


OneCricketeer commented on pull request #7965:
URL: https://github.com/apache/kafka/pull/7965#issuecomment-991925594


   >  it puts a timestamp in front of the object
   
   If that's the case, I don't think this specific transform will help as it'd 
make a Struct wrapper around the extracted json object within the string.
   
   My recommendation instead is to define your own subclass of 
StringDeserializer that substrings the message to remove the date prefix 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org