Great news! I can’t wait to also have support for this for the Flink Runner.
Which is partially blocked by me or others working on the Flink Runner, I
guess… :-(
> On 22 Mar 2017, at 05:15, Jean-Baptiste Onofré <[email protected]> wrote:
>
> Awesome !!! Great news !
>
> Thanks guys for that !
>
> I started to implement IT in JMS, MQTT, Redis, Cassandra IOs. I keep you
> posted.
>
> Regards
> JB
>
> On 03/21/2017 11:01 PM, Stephen Sisk wrote:
>> I'm really excited to see these tests are running!
>>
>> These Jdbc tests are testing against a postgres instance - that instance is
>> running on the kubernetes cluster I've set up for beam IO ITs as discussed
>> in the "Hosting data stores for IO transform testing" thread[0]. I set up
>> that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
>> run their own kubernetes cluster and do the same thing for themselves to
>> run the ITs. (I'd actually to love to hear about that if anyone does it.)
>>
>> I'm excited to get a few more ITs using this infrastructure so we can test
>> it out/smooth out the remaining rough edges in creating ITs. I'm happy to
>> answer questions about that on the mailing list, but we obviously have to
>> have the process written down - the Testing IO Transforms in Apache Beam
>> doc [2] covers how to do this, but is still rough. I'm working on getting
>> that up on the website and ironing out the rough edges [3], but generally
>> reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
>> work should give you a sense of how to get it working. I'm also thinking we
>> might want to simplify the way we do data loading, so I don't consider this
>> process fully stabilized, but I'll port code written according to the
>> current standards to the new standards if we make changes.
>>
>> ElasticsearchIO has all the prerequisites, so I'd like to get them going in
>> the near future. I know JB has started on this in his RedisIO PR, and the
>> HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
>> For now, each datastore has to be manually set up, but I'd like to automate
>> that process - I'll file a JIRA ticket shortly for that.
>>
>> Thanks,
>> Stephen
>> [0] Hosting data stores for IO transform testing -
>> https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
>> [1] Postgres k8 scripts -
>> https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes
>> [2] IO testing guide -
>> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
>> [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
>>
>> On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <[email protected]>
>> wrote:
>>
>>> Hi all,
>>>
>>> Exciting news! As of yesterday, we have checked in the Jenkins
>>> configuration for our first continuously running IO Integration Test! You
>>> can check it out in Jenkins here[1]. We’re also publishing results to a
>>> database, and we’ve turned up a basic dashboarding system where you can see
>>> the results here[2]. Caveat: there are only two runs, and we’ll be tweaking
>>> the underlying system still, so don’t panic that we’re up and to the right
>>> currently. ;)
>>>
>>> This is the first test running continuously on top of the performance / IO
>>> testing infrastructure described in this doc[3]. Initial support for Beam
>>> is now present in PerfKit Benchmarker; given what they had already, it was
>>> easiest to add support for Dataflow and Java. We need your help to add
>>> additional support! The doc lists a number of JIRA issues to build out
>>> support for other systems. I’m happy to work with people to help them
>>> understand what is necessary for these tasks; just send an email to the
>>> list if you need help and I’ll help you move forwards.
>>>
>>> Looking forward to it!
>>>
>>> Jason
>>>
>>> [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
>>> [2]
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
>>> [3]
>>>
>>> https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
>>>
>>> --
>>> -------
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>
>
> --
> Jean-Baptiste Onofré
> [email protected]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
