[jira] [Closed] (BAHIR-179) fail silently when tests need docker image to be running
[ https://issues.apache.org/jira/browse/BAHIR-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joao Boto closed BAHIR-179. --- Resolution: Fixed Fix Version/s: Flink-Next > fail silently when tests need docker image to be running > > > Key: BAHIR-179 > URL: https://issues.apache.org/jira/browse/BAHIR-179 > Project: Bahir > Issue Type: Test > Components: Flink Streaming Connectors >Reporter: Joao Boto >Assignee: Joao Boto >Priority: Minor > Fix For: Flink-Next > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BAHIR-190) ActiveMQ connector stops on empty queue
Stephan Brosinski created BAHIR-190:
---
Summary: ActiveMQ connector stops on empty queue
Key: BAHIR-190
URL: https://issues.apache.org/jira/browse/BAHIR-190
Project: Bahir
Issue Type: Bug
Components: Flink Streaming Connectors
Affects Versions: Flink-1.0
Reporter: Stephan Brosinski
I tried the ActiveMQ Flink Connector. Reading from an ActiveMQ queue, it seems
to connector exits once there are no more messages in the queue. This ends the
Flink job processing the stream.
To me it seems, that the while loop inside the run method (AMQSource.java, line
222) should not do a return, but a continue if the message is no instance of
ByteMessage, e.g. null.
If I'm right, I can create a pull request showing the change.
To reproduce:
{code:java}
ActiveMQConnectionFactory connectionFactory = new
ActiveMQConnectionFactory("xxx", "xxx",
"tcp://localhost:61616?jms.redeliveryPolicy.maximumRedeliveries=1");
AMQSourceConfig amqConfig = new
AMQSourceConfig.AMQSourceConfigBuilder()
.setConnectionFactory(connectionFactory)
.setDestinationName("test")
.setDestinationType(DestinationType.QUEUE)
.setDeserializationSchema(new SimpleStringSchema())
.build();
AMQSource amqSource = new AMQSource<>(amqConfig);
env.addSource(amqSource).print()
env.setParallelism(1).execute("ActiveMQ Consumer");{code}
Then point the Flink job at an empty ActiveMQ queue.
Not sure if this is a bug, but it's not what I expected when I used the
connector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BAHIR-191) update flume version to 1.9.0
Joao Boto created BAHIR-191: --- Summary: update flume version to 1.9.0 Key: BAHIR-191 URL: https://issues.apache.org/jira/browse/BAHIR-191 Project: Bahir Issue Type: Improvement Reporter: Joao Boto Assignee: Joao Boto flume has new version.. update to last one https://flume.apache.org/releases/1.9.0.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-180) kudu connector improve real-time data performance
[ https://issues.apache.org/jira/browse/BAHIR-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743112#comment-16743112 ] ASF GitHub Bot commented on BAHIR-180: -- GitHub user eskabetxe opened a pull request: https://github.com/apache/bahir-flink/pull/40 [BAHIR-180] changes to improve eventual consistence You can merge this pull request into a Git repository by running: $ git pull https://github.com/eskabetxe/bahir-flink BAHIR-180_newVersion Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bahir-flink/pull/40.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #40 commit d9c969076a349bf592729d10a2cb9acb1a99743e Author: eskabetxe Date: 2019-01-15T12:05:29Z BAHIR-180: added serialization and change eventual time commit f875ae2118c452050e8401ec3fd7f93918cf25da Author: eskabetxe Date: 2019-01-15T12:15:30Z BAHIR-180: added serialization and change eventual time commit 9ce5b32d352864c946bb73a8c0b628097db5a7f6 Author: eskabetxe Date: 2019-01-15T12:31:25Z BAHIR-180: added serialization and change eventual time commit cb9fbda0f3f7b4216be81756ed7e5c71aeab95f0 Author: eskabetxe Date: 2019-01-15T14:11:04Z BAHIR-180: added serialization and change eventual time commit c35ff0112111ddaab400ce76f2d78610b4595597 Author: eskabetxe Date: 2019-01-15T14:17:42Z BAHIR-180: added serialization and change eventual time > kudu connector improve real-time data performance > - > > Key: BAHIR-180 > URL: https://issues.apache.org/jira/browse/BAHIR-180 > Project: Bahir > Issue Type: Improvement > Components: Flink Streaming Connectors >Affects Versions: Flink-1.0 >Reporter: mei jie >Assignee: Joao Boto >Priority: Major > Fix For: Flink-Next > > > kudu connector has poof performance to write real-time data with either > strong consistency or eventual consistency. at my environment(3server with > 250G memory, 5 disk, and the parallelism of flink is 60), the speed is about > 20,000/s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-191) update flume version to 1.9.0
[ https://issues.apache.org/jira/browse/BAHIR-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743120#comment-16743120 ] ASF GitHub Bot commented on BAHIR-191: -- GitHub user eskabetxe opened a pull request: https://github.com/apache/bahir-flink/pull/41 [BAHIR-191] updated flume version to 1.9.0 @lresende clould you check this You can merge this pull request into a Git repository by running: $ git pull https://github.com/eskabetxe/bahir-flink BAHIR-191 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bahir-flink/pull/41.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #41 commit cc03100ef7c22b2848cd6a442bd2073d569c76d6 Author: eskabetxe Date: 2019-01-15T14:51:14Z BAHIR-191: uddate flume version to 1.9.0 > update flume version to 1.9.0 > - > > Key: BAHIR-191 > URL: https://issues.apache.org/jira/browse/BAHIR-191 > Project: Bahir > Issue Type: Improvement >Reporter: Joao Boto >Assignee: Joao Boto >Priority: Major > > flume has new version.. > update to last one > > https://flume.apache.org/releases/1.9.0.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BAHIR-180) kudu connector improve real-time data performance
[ https://issues.apache.org/jira/browse/BAHIR-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joao Boto updated BAHIR-180: Affects Version/s: (was: Flink-1.0) Flink-Next > kudu connector improve real-time data performance > - > > Key: BAHIR-180 > URL: https://issues.apache.org/jira/browse/BAHIR-180 > Project: Bahir > Issue Type: Improvement > Components: Flink Streaming Connectors >Affects Versions: Flink-Next >Reporter: mei jie >Assignee: Joao Boto >Priority: Major > Fix For: Flink-Next > > > kudu connector has poof performance to write real-time data with either > strong consistency or eventual consistency. at my environment(3server with > 250G memory, 5 disk, and the parallelism of flink is 60), the speed is about > 20,000/s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-180) kudu connector improve real-time data performance
[ https://issues.apache.org/jira/browse/BAHIR-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743115#comment-16743115 ] ASF GitHub Bot commented on BAHIR-180: -- Github user eskabetxe commented on the issue: https://github.com/apache/bahir-flink/pull/40 @lresende cuando puedas > kudu connector improve real-time data performance > - > > Key: BAHIR-180 > URL: https://issues.apache.org/jira/browse/BAHIR-180 > Project: Bahir > Issue Type: Improvement > Components: Flink Streaming Connectors >Affects Versions: Flink-1.0 >Reporter: mei jie >Assignee: Joao Boto >Priority: Major > Fix For: Flink-Next > > > kudu connector has poof performance to write real-time data with either > strong consistency or eventual consistency. at my environment(3server with > 250G memory, 5 disk, and the parallelism of flink is 60), the speed is about > 20,000/s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-99) Kudu connector to read/write from/to Kudu
[ https://issues.apache.org/jira/browse/BAHIR-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743140#comment-16743140 ] ASF GitHub Bot commented on BAHIR-99: - Github user eskabetxe commented on the issue: https://github.com/apache/bahir-flink/pull/17 @lresende this could be closed.. we have another contribution with this connector > Kudu connector to read/write from/to Kudu > - > > Key: BAHIR-99 > URL: https://issues.apache.org/jira/browse/BAHIR-99 > Project: Bahir > Issue Type: New Feature > Components: Flink Streaming Connectors >Affects Versions: Flink-1.0 >Reporter: Rubén Casado >Assignee: Joao Boto >Priority: Major > Fix For: Flink-Next > > > Java library to integrate Apache Kudu and Apache Flink. Main goal is to be > able to read/write data from/to Kudu using the DataSet and DataStream Flink's > APIs. > Data flows patterns: > Batch > - Kudu -> DataSet -> Kudu > - Kudu -> DataSet -> other source > - Other source -> DataSet -> other source > Stream > - Other source -> DataStream -> Kudu > Code is available in https://github.com/rubencasado/Flink-Kudu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-99) Kudu connector to read/write from/to Kudu
[ https://issues.apache.org/jira/browse/BAHIR-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743173#comment-16743173 ] ASF GitHub Bot commented on BAHIR-99: - Github user asfgit closed the pull request at: https://github.com/apache/bahir-flink/pull/17 > Kudu connector to read/write from/to Kudu > - > > Key: BAHIR-99 > URL: https://issues.apache.org/jira/browse/BAHIR-99 > Project: Bahir > Issue Type: New Feature > Components: Flink Streaming Connectors >Affects Versions: Flink-1.0 >Reporter: Rubén Casado >Assignee: Joao Boto >Priority: Major > Fix For: Flink-Next > > > Java library to integrate Apache Kudu and Apache Flink. Main goal is to be > able to read/write data from/to Kudu using the DataSet and DataStream Flink's > APIs. > Data flows patterns: > Batch > - Kudu -> DataSet -> Kudu > - Kudu -> DataSet -> other source > - Other source -> DataSet -> other source > Stream > - Other source -> DataStream -> Kudu > Code is available in https://github.com/rubencasado/Flink-Kudu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-107) Build and test Bahir against Scala 2.12
[ https://issues.apache.org/jira/browse/BAHIR-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743176#comment-16743176 ] ASF GitHub Bot commented on BAHIR-107: -- Github user lresende commented on the issue: https://github.com/apache/bahir/pull/76 Could you start a conversation on the dev list regarding the direction this is going to take, are we droping 2.10 and continue support with 211 and 212 but making 212 the default, and all other details... at least to be informative to the dev community > Build and test Bahir against Scala 2.12 > --- > > Key: BAHIR-107 > URL: https://issues.apache.org/jira/browse/BAHIR-107 > Project: Bahir > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Spark has started effort for accommodating Scala 2.12 > See SPARK-14220 . > This JIRA is to track requirements for building Bahir on Scala 2.12.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-186) Support SSL connection in MQTT SQL Streaming
[ https://issues.apache.org/jira/browse/BAHIR-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743179#comment-16743179 ] ASF GitHub Bot commented on BAHIR-186: -- Github user lresende commented on the issue: https://github.com/apache/bahir/pull/75 @emlaver could you please rebase and force a new build > Support SSL connection in MQTT SQL Streaming > > > Key: BAHIR-186 > URL: https://issues.apache.org/jira/browse/BAHIR-186 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Lukasz Antoniak >Assignee: Lukasz Antoniak >Priority: Major > Fix For: Spark-2.4.0 > > > Mailing list discussion: > https://www.mail-archive.com/user@bahir.apache.org/msg00022.html. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-186) Support SSL connection in MQTT SQL Streaming
[ https://issues.apache.org/jira/browse/BAHIR-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743209#comment-16743209 ] ASF GitHub Bot commented on BAHIR-186: -- Github user emlaver commented on the issue: https://github.com/apache/bahir/pull/75 @lresende Ok, rebased in 42d9ec2. > Support SSL connection in MQTT SQL Streaming > > > Key: BAHIR-186 > URL: https://issues.apache.org/jira/browse/BAHIR-186 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Lukasz Antoniak >Assignee: Lukasz Antoniak >Priority: Major > Fix For: Spark-2.4.0 > > > Mailing list discussion: > https://www.mail-archive.com/user@bahir.apache.org/msg00022.html. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
