Re: BEAM-6516 - RabbitMqIO.Read - Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method(reply-code=406, reply-text=PRECONDITION_FAILED - unknow

[Jean-Baptiste Onofre]

Hi,

Let me take a look.

Regards
JB

> Le 15 janv. 2021 à 20:43, Rafael Ribeiro  a écrit :
> 
> hi All,
> 
> I'm facing the problem reported on 
> https://issues.apache.org/jira/browse/BEAM-6516 
> 
> 
> running a flex template based on kafka-to-bigquery and adapted to rabbitmq
> 
> java.lang.RuntimeException: Exception while finalizing checkpoint
> at 
> org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.lambda$flushState$0
>  (StreamingModeExecutionContext.java:403 
> )
> at 
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.lambda$callFinalizeCallbacks$3
>  (StreamingDataflowWorker.java:1218 
> )
> at java.util.concurrent.ThreadPoolExecutor.runWorker 
> (ThreadPoolExecutor.java:1149 
> )
> at java.util.concurrent.ThreadPoolExecutor$Worker.run 
> (ThreadPoolExecutor.java:624 
> )
> at java.lang.Thread.run (Thread.java:748 
> )
> Caused by: java.io.IOException
> at com.rabbitmq.client.impl.AMQChannel.wrap (AMQChannel.java:129 
> )
> at com.rabbitmq.client.impl.AMQChannel.wrap (AMQChannel.java:125 
> )
> at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc (AMQChannel.java:147 
> )
> at com.rabbitmq.client.impl.ChannelN.txCommit (ChannelN.java:1540 
> )
> at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.txCommit 
> (AutorecoveringChannel.java:663 
> )
> Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; 
> protocol method: #method(reply-code=406, 
> reply-text=PRECONDITION_FAILED - unknown delivery tag 2, class-id=60, 
> method-id=80)
> at com.rabbitmq.utility.ValueOrException.getValue (ValueOrException.java:66 
> )
> at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue 
> (BlockingValueOrException.java:36 
> )
> at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply 
> (AMQChannel.java:502 
> )
> at com.rabbitmq.client.impl.AMQChannel.privateRpc (AMQChannel.java:293 
> )
> at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc (AMQChannel.java:141 
> )
> Caused by: 

Re: [VOTE] Release 2.27.0, release candidate #4

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 6 janv. 2021 à 08:17, Pablo Estrada  a écrit :
> 
> Hi everyone,
> Please review and vote on the release candidate #4 for the version 2.27.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> NOTE. What happened to RC #2? I started building RC2 before completing all 
> the cherry-picks, so the tag for RC2 was created on an incorrect commit.
> 
> NOTE. What happened to RC #3? I started building RC3, but a new bug was 
> discovered (BEAM-11569) that required amending the branch. Thus this is now 
> RC4.
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1
>  if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> C79DDD47DAF3808F0B9DDFAC02B2D9F742008494 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.27.0-RC4" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.27.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours, but given the holidays, we will 
> likely extend for a few more days. The release will be adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> -P.
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349380
>  
> 
>  
> [2] https://dist.apache.org/repos/dist/dev/beam/2.27.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1149/ 
>  
> [5] https://github.com/apache/beam/tree/v2.27.0-RC4 
>  
> [6] https://github.com/apache/beam/pull/13602 
>  
> [7] https://github.com/apache/beam-site/pull/610 
>  
> [8] https://github.com/apache/beam/pull/13603 
>  
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=194829106
>  
> 
>  
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 

Re: [VOTE] Release 2.27.0, release candidate #1

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 23 déc. 2020 à 06:46, Pablo Estrada  a écrit :
> 
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 2.27.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1
>  if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> C79DDD47DAF3808F0B9DDFAC02B2D9F742008494 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.27.0-RC1" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.27.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours, but given the holidays, we will 
> likely extend for a few more days. The release will be adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> -P.
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349380
>  
> 
>  
> [2] https://dist.apache.org/repos/dist/dev/beam/2.27.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1145/ 
> 
> [5] https://github.com/apache/beam/tree/v2.27 
> .0-RC1
> [6] https://github.com/apache/beam/pull/13602 
>  
> [7] https://github.com/apache/beam-site/pull/610 
>  
> [8] https://github.com/apache/beam/pull/13603 
>  
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=194829106
>  
> 
>  
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 

Re: Hello Beam Community!

[Jean-Baptiste Onofre]

Welcome Brittany !

Regards
JB

> Le 13 mars 2020 à 02:31, Brittany Hermann  a écrit :
> 
> Hello Beam Community!
> 
> My name is Brittany Hermann and I recently joined the Open Source team in 
> Data Analytics at Google. As a Program Manager, I will be focusing on 
> community engagement while getting to work on Apache Beam and Airflow 
> projects! I have always thrived on creating healthy, diverse, and overall 
> happy communities and am excited to bring that to the team. For a fun fact, I 
> am a big Wisconsin Badgers Football fan and have a goldendoodle puppy named 
> Ollie!
> 
> I look forward to collaborating with you all! 
> 
> Kind regards,
> 
> Brittany Hermann 
> 
> -- 
> 
> Brittany Hermann 
> Open Source Program Manager (Provided by Adecco Staffing)
> 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089 
> 
> 

Re: A new reworked Elasticsearch 7+ IO module

[Jean-Baptiste Onofre]

Hi,

I think WARN makes sense and the safest approach. It allows users to be notify 
and eventually update or back on previous Beam IO version.

Regards
JB

> Le 6 mars 2020 à 18:49, Kenneth Knowles  a écrit :
> 
> Since the user provides backendVersion, here are some possible levels of 
> things to add in expand() based on that (these are extra niceties beyond the 
> agreed number of releases to remove)
> 
>  - WARN for backendVersion < n
>  - reject for backendVersion < n with opt-in pipeline option to keep it 
> working one more version (gets their attention and indicates urgency)
>  - reject completely
> 
> Kenn
> 
> On Fri, Mar 6, 2020 at 2:26 AM Etienne Chauchot  > wrote:
> Hi all, 
> 
> it's been 3 weeks since the survey on ES versions the users use. 
> 
> The survey received very few responses: only 9 responses for now (multiple 
> versions possible of course). The responses are the following:
> 
> ES2: 0 clients, ES5: 1, ES6: 5, ES7: 8 
> 
> It tends to go toward a drop of ES2 support but for now it is still not very 
> representative.
> 
> I'm cross-posting to @users to let you know that I'm closing the survey 
> within 1 or 2 weeks. So please respond if you're using ESIO.
> 
> Best
> 
> Etienne
> 
> On 13/02/2020 12:37, Etienne Chauchot wrote:
>> Hi Cham, thanks for your comments !
>> 
>> I just sent an email to user ML with a survey link to count ES uses per 
>> version:
>> 
>> https://lists.apache.org/thread.html/rc8185afb8af86a2a032909c13f569e18bd89e75a5839894d5b5d4082%40%3Cuser.beam.apache.org%3E
>>  
>> 
>> Best
>> 
>> Etienne
>> 
>> On 10/02/2020 19:46, Chamikara Jayalath wrote:
>>> 
>>> 
>>> On Thu, Feb 6, 2020 at 8:13 AM Etienne Chauchot >> > wrote:
>>> Hi,
>>> 
>>> please see my comments inline
>>> 
>>> On 06/02/2020 16:24, Alexey Romanenko wrote:
 Please, see my comments inline.
 
> On 6 Feb 2020, at 10:50, Etienne Chauchot  > wrote:
 1. regarding version support: ES v2 is no more maintained by Elastic 
 since 2018/02 so we plan to remove it from the IO. In the past we 
 already retired versions (like spark 1.6 for instance).
 
>> 
>> 
>> My only concern here is that there might be users who use the existing 
>> module who might not be able to easily upgrade the Beam version if we 
>> remove it. But given that V2 is 5 versions behind the latest release 
>> this might be OK.
>> 
>> It seems we have a consensus on this.
>> I think there should be another general discussion on the long term 
>> support of our prefered tool IO modules.
> => yes, consensus, let's drop ESV2
> 
 We had (and still have) a similar problem with KafkaIO to support 
 different versions of Kafka, especially very old version 0.9. We raised 
 this question on user@ and it appears that there are users who for some 
 reasons still use old Kafka versions. So, before dropping a support of any 
 ES versions, I’d suggest to ask it user@ and see if any people will be 
 affected by this.
>>> Yes we can do a survey among users but the question is, should we support 
>>> an ES version that is no more supported by Elastic themselves ?
>>> 
>>> +1 for asking in the user list. I guess this is more about whether users 
>>> need this specific version that we hope to drop support for. Whether we 
>>> need to support unsupported versions is a more generic question that should 
>>> prob. be addressed in the dev list. (and I personally don't think we should 
>>> unless there's a large enough user base for a given version).
>>> 
> 
 2. regarding the user: the aim is to unlock some new features (listed 
 by Ludovic) and give the user more flexibility on his request. For 
 that, it requires to use high level java ES client in place of the low 
 level REST client (that was used because it is the only one compatible 
 with all ES versions). We plan to replace the API (json document in 
 and out) by more complete standard ES objects that contain de request 
 logic (insert/update, doc routing etc...) and the data. There are 
 already IOs like SpannerIO that use similar objects in input 
 PCollection rather than pure POJOs. 
 
>> 
>> 
>> Won't this be a breaking change for all users ? IMO using POJOs in 
>> PCollections is safer since we have to worry about changes to the 
>> underlying client library API. Exception would be when underlying client 
>> library offers a backwards compatibility guarantee that we can rely on 
>> for the foreseeable future (for example, BQ TableRow).
>> 
>> Agreed but actually, there will be POJOs in order to abstract 
>> Elasticsearch's version support. The 

Re: Apache Beam with Hive

[Jean-Baptiste Onofre]

Hi,

If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in 
the javadoc:

https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82
 


Regards
JB

> Le 11 févr. 2020 à 07:03, Gershi, Noam  a écrit :
> 
> Hi,
>  
> I am searching for a detailed example how to use Apache Beam with Hive and/or 
> Hcatalog?
> Creating tables, inserting data and fetching it…
>  
>  
>   Noam Gershi Software Developer
>   ICG Technology – TLV Lab
>   T: +972 (3) 7405718
>  
>