[
https://issues.apache.org/jira/browse/BAHIR-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803676#comment-16803676
]
ASF GitHub Bot commented on BAHIR-183:
--------------------------------------
lukasz-antoniak commented on pull request #84: [BAHIR-183] [WIP] HDFS based
MQTT client persistence
URL: https://github.com/apache/bahir/pull/84
As discussed in the JIRA ticket
(https://issues.apache.org/jira/browse/BAHIR-183), revisited version of HDFS
persistence of MQTT messages, which should ease recovery in case of device
failure. The design discussion may be continued in JIRA, but I decided to open
WIP PR for @yanlin-Lynn to review and comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Using HDFS for saving message for mqtt source
> ---------------------------------------------
>
> Key: BAHIR-183
> URL: https://issues.apache.org/jira/browse/BAHIR-183
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Structured Streaming Connectors
> Affects Versions: Spark-2.2.0
> Reporter: Wang Yanlin
> Assignee: Wang Yanlin
> Priority: Major
> Fix For: Spark-2.4.0
>
>
> Currently in spark-sql-streaming-mqtt, the received mqtt message is saved in
> a local file by driver, this will have the risks of losing data for cluster
> mode when application master failover occurs. So saving in-coming mqtt
> messages using a director in checkpoint will solve this problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
