[
https://issues.apache.org/jira/browse/KYLIN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-4866:
-----------------------------------
Assignee: Xiaoxiang Yu
> Streaming from Kafka with nested timestamp column
> -------------------------------------------------
>
> Key: KYLIN-4866
> URL: https://issues.apache.org/jira/browse/KYLIN-4866
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
> Affects Versions: v3.1.0
> Environment: Docker setup
> Reporter: Hélder Hugo Ferreira
> Assignee: Xiaoxiang Yu
> Priority: Major
> Fix For: v3.1.2
>
> Attachments: image-2021-01-08-16-16-51-615.png
>
>
> We have an IOT Data Platform which saves in Kafka Json messages with the
> following format:
> {code:json}
> {
> "SysProperties": {
> "EventId": "1610120564321723276",
> "EnqueueTime": "2021-01-08T15:42:44.045Z",
> "UserName": "dummyuser"
> },
> "AppProperties": {
> "EventDefinition": "EquipmentData",
> "EventTime": "2021-01-08T15:42:43.26+00:00",
> "ApplicationName": "ConnectIoTSimulator",
> "ApplicationContext": null
> },
> "Data": {
> "Equipment": "DUMMY-01",
> "Material": "DUMMYZA9983884",
> "MeasurementData": {
> "Temp": 57,
> "Pressure": 48.379,
> "Energy": 47,
> "Humidity": 31.727
> }
> }
> {code}
> From these messages, we are trying to build a cube in which the
> "AppProperties.EventTime" shall be the timestamp column (using format
> yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp",
> "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and
> "Data.MeasurementData.Humidity" shall be the metrics and all other fields
> dimensions.
> We can setup everything fine in the kylin portal (like in the tutorial),
> however on runtime we have the following exceptions logged by the streaming
> receiver component:
> !image-2021-01-08-16-16-51-615.png!
> After some investigation, we figured out it only works if the timestamp field
> within the Json messages is not nested inside inner objects, like in the
> following example:
> {code:json}
> {
> "timestamp": 1610038228492,
> "nestedData": {
> "Equipment": "EPFTES-01",
> "Material": "ZA9983884",
> "Temp": 10,
> "Pressure": 20.459,
> "Energy": 30,
> "Humidity": 80.967
> }
> }{code}
> Therefore, we believe there is a bug handling the timestamp column when it is
> nested within Json messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
