[jira] [Commented] (BAHIR-183) Using HDFS for saving message for mqtt source
[ https://issues.apache.org/jira/browse/BAHIR-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706075#comment-16706075 ] ASF GitHub Bot commented on BAHIR-183: -- GitHub user yanlin-Lynn opened a pull request: https://github.com/apache/bahir/pull/72 [BAHIR-183]Using HDFS for saving message for mqtt source. Currently in spark-sql-streaming-mqtt, the received mqtt message is saved in a local file by driver, this will have the risks of losing data for cluster mode when application master failover occurs. So add a hdfs-based mqtt source to solve this problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yanlin-Lynn/bahir bahir-183 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bahir/pull/72.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #72 commit eb33741f8b77828815c8f834ec0951d6c39728fc Author: wangyanlin01 Date: 2018-12-02T03:00:21Z [BAHIR-183]Using HDFS for saving message for mqtt source. > Using HDFS for saving message for mqtt source > - > > Key: BAHIR-183 > URL: https://issues.apache.org/jira/browse/BAHIR-183 > Project: Bahir > Issue Type: Improvement > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.2.0 >Reporter: Wang Yanlin >Priority: Major > Fix For: Spark-2.2.1 > > > Currently in spark-sql-streaming-mqtt, the received mqtt message is saved in > a local file by driver, this will have the risks of losing data for cluster > mode when application master failover occurs. So saving in-coming mqtt > messages using a director in checkpoint will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BAHIR-183) Using HDFS for saving message for mqtt source
Wang Yanlin created BAHIR-183: - Summary: Using HDFS for saving message for mqtt source Key: BAHIR-183 URL: https://issues.apache.org/jira/browse/BAHIR-183 Project: Bahir Issue Type: Improvement Components: Spark Structured Streaming Connectors Affects Versions: Spark-2.2.0 Reporter: Wang Yanlin Fix For: Spark-2.2.1 Currently in spark-sql-streaming-mqtt, the received mqtt message is saved in a local file by driver, this will have the risks of losing data for cluster mode when application master failover occurs. So saving in-coming mqtt messages using a director in checkpoint will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Apache Bahir 2.3.1 (RC1)
+1 Ran test suite which passed. On Fri, Nov 30, 2018 at 9:13 PM Jean-Baptiste Onofré wrote: > +1 (binding) > > Regards > JB > > On 30/11/2018 17:11, Luciano Resende wrote: > > Dear community member, > > > > Please vote to approve the release of Apache Bahir 2.3.1 (RC1) based on > > Apache Spark 2.3.1. > > > > Tag: v2.3.1-rc1 (6c4d67e0a99bcbbad199b7d1d26f3624491070b4) > > > > https://github.com/apache/bahir/tree/v2.3.1-rc1 > > > > Release files: > > > > https://repository.apache.org/content/repositories/orgapachebahir-1026 > > > > Source distribution: > > > > https://dist.apache.org/repos/dist/dev/bahir/bahir-spark/2.3.1-rc1/ > > > > > > The vote is open for at least 72 hours and passes if a majority of at > least > > 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Bahir 2.3.1 > > [ ] -1 Do not release this package because ... > > > > > > Thanks for your vote! > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
Re: [VOTE] Apache Bahir 2.3.2 (RC1)
+1 Ran test suite which passed On Fri, Nov 30, 2018 at 12:57 PM Luciano Resende wrote: > Dear community member, > > Please vote to approve the release of Apache Bahir 2.3.2 (RC1) based on > Apache Spark 2.3.2. > > Tag: v2.3.2-rc1 (6d24b270fd713352faf1c93f08f835e39e496489) > > https://github.com/apache/bahir/tree/v2.3.2-rc1 > > Release files: > > https://repository.apache.org/content/repositories/orgapachebahir-1027 > > Source distribution: > > https://dist.apache.org/repos/dist/dev/bahir/bahir-spark/2.3.2-rc1/ > > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Bahir 2.3.2 > [ ] -1 Do not release this package because ... > > > Thanks for your vote! > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ >
