FernandoDorado opened a new issue #1291:
URL: https://github.com/apache/camel-kafka-connector/issues/1291


   Good afternoon, 
   
   I have been doing some testing of the kafka-hdfs connector, especially on 
the way it stacks files within HDFS. The used configuration for the connector 
is as follows: 
   ```json
   {
       "name":"CamelHdfsSinkConnector",
       "config":{
           
"connector.class":"org.apache.camel.kafkaconnector.hdfs.CamelHdfsSinkConnector",
           "task.max":2,
           "key.converter":"org.apache.kafka.connect.storage.StringConverter",
           "value.converter":"org.apache.kafka.connect.storage.StringConverter",
           "topics":"office-topic",
           "camel.sink.path.hostName": "HAcluster",
           "camel.sink.endpoint.namedNodes": "namenode-1:8020,namenode-2:8020",
           "camel.sink.endpoint.splitStrategy": "IDLE:100000,BYTES:10000000",
           "camel.sink.path.path": "user/kafka/"
       }
   }
   ```
   I have verified that messages do indeed arrive and stack correctly in HDFS. 
My problem comes because these messages are stacked without any line break to 
separate them in order to be able to use them for example with Hive. An example 
of a msg in HDFS is: 
   
   
   
   
   
   I have seen that this was not a problem that happened only to me, as there 
is a [post on 
StackOverflow](https://stackoverflow.com/questions/67374326/append-a-new-line-character-at-every-record-passing-through-camelkafkaazureblobs)
 with this problem. 
   
   I have tried the proposed solution, but since I am not a Java expert, I have 
not obtained good results. I have modified the Apply method of 
CamelTypeConverTransform, but I don't know if the post refers to this or 
another one. This would be the method I have tried: 
   ```java
       @Override
       public R apply(R record) {
           final Schema schema = operatingSchema(record);
           final Object value = operatingValue(record);
   
           final Object convertedValue = 
convertValueWithCamelTypeConverter(value);
           final Object updatedValue = convertedValue + System.lineSeparator();
   
           final Schema updatedSchema = getOrBuildRecordSchema(schema, 
updatedValue);
   
           return newRecord(record, updatedSchema, updatedValue);
       }
   ```
   A single msg has a format: 
   ```json
   {
     "fields": {
       "year": 2021,
       "I1": 0,
       "I2": 8.5143e-41,
       "I3": 6.234535,
       "PHI_UI1": 0,
       "sec": 54,
       "PHI_UI2": 8.6209e-41,
       "P": 1335.8148,
       "PHI_UI3": -20.375,
       "Q": -584.91974,
       "min": 42,
       "month": 11,
       "hour": 13,
       "FREQ": 50,
       "I_N": 5.915098,
       "U1": 234.59723,
       "UUID": 35697,
       "day": 19,
       "U2": 233.06522,
       "U3": 234.024
     },
     "timestamp": 1637329377581,
     "expires": null
   }
   ```
   
   An extracted portion of data from the HDFS file: 
   ```json
   
{"fields":{"year":2021,"I1":0.0,"I2":4.8611E-41,"I3":6.2057714,"PHI_UI1":0.0,"sec":14,"PHI_UI2":4.104E-42,"P":1325.8503,"PHI_UI3":-20.5,"Q":-583.9375,"min":52,"month":11,"hour":13,"FREQ":49.75,"I_N":5.9074244,"U1":233.73982,"UUID":35697,"day":19,"U2":232.20639,"U3":233.972},"timestamp":1637329937662,"expires":null}{"fields":{"year":2021,"I1":0.0,"I2":4.1306E-41,"I3":6.2395077,"PHI_UI1":0.0,"sec":15,"PHI_UI2":5.0996E-41,"P":1323.7439,"PHI_UI3":-20.75,"Q":-599.9375,"min":52,"month":11,"hour":13,"FREQ":49.75,"I_N":5.9192123,"U1":233.64069,"UUID":35697,"day":19,"U2":232.09172,"U3":233.976},"timestamp":1637329938420,"expires":null}{"fields":{"year":2021,"I1":0.0,"I2":1.208E-41,"I3":6.243064,"PHI_UI1":0.0,"sec":15,"PHI_UI2":3.6516E-41,"P":1317.8503,"PHI_UI3":-20.5,"Q":-581.44635,"min":52,"month":11,"hour":13,"FREQ":49.75,"I_N":5.9074244,"U1":233.63208,"UUID":35697,"day":19,"U2":232.06783,"U3":233.976},"timestamp":1637329939102,"expires":null}
   ```
   
   Where it can be seen that there is no separation between different msgs. Is 
there some way to fix this problem? 
   
   
   Thank you in advance


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to