Excellent news, I am eager to see more IOs/Runners been included in the Integration Tests, and I will be glad to contribute in anything I can.
Congratulations for this important milestone. Ismaël ps. I will try to reproduce the Kubernetes setup so I will be eventually annoying you with questions. On Wed, Mar 22, 2017 at 11:28 AM, Aljoscha Krettek <[email protected]> wrote: > Great news! I can’t wait to also have support for this for the Flink Runner. > Which is partially blocked by me or others working on the Flink Runner, I > guess… :-( >> On 22 Mar 2017, at 05:15, Jean-Baptiste Onofré <[email protected]> wrote: >> >> Awesome !!! Great news ! >> >> Thanks guys for that ! >> >> I started to implement IT in JMS, MQTT, Redis, Cassandra IOs. I keep you >> posted. >> >> Regards >> JB >> >> On 03/21/2017 11:01 PM, Stephen Sisk wrote: >>> I'm really excited to see these tests are running! >>> >>> These Jdbc tests are testing against a postgres instance - that instance is >>> running on the kubernetes cluster I've set up for beam IO ITs as discussed >>> in the "Hosting data stores for IO transform testing" thread[0]. I set up >>> that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can >>> run their own kubernetes cluster and do the same thing for themselves to >>> run the ITs. (I'd actually to love to hear about that if anyone does it.) >>> >>> I'm excited to get a few more ITs using this infrastructure so we can test >>> it out/smooth out the remaining rough edges in creating ITs. I'm happy to >>> answer questions about that on the mailing list, but we obviously have to >>> have the process written down - the Testing IO Transforms in Apache Beam >>> doc [2] covers how to do this, but is still rough. I'm working on getting >>> that up on the website and ironing out the rough edges [3], but generally >>> reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests >>> work should give you a sense of how to get it working. I'm also thinking we >>> might want to simplify the way we do data loading, so I don't consider this >>> process fully stabilized, but I'll port code written according to the >>> current standards to the new standards if we make changes. >>> >>> ElasticsearchIO has all the prerequisites, so I'd like to get them going in >>> the near future. I know JB has started on this in his RedisIO PR, and the >>> HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe. >>> For now, each datastore has to be manually set up, but I'd like to automate >>> that process - I'll file a JIRA ticket shortly for that. >>> >>> Thanks, >>> Stephen >>> [0] Hosting data stores for IO transform testing - >>> https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E >>> [1] Postgres k8 scripts - >>> https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes >>> [2] IO testing guide - >>> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing >>> [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025 >>> >>> On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster >>> <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> Exciting news! As of yesterday, we have checked in the Jenkins >>>> configuration for our first continuously running IO Integration Test! You >>>> can check it out in Jenkins here[1]. We’re also publishing results to a >>>> database, and we’ve turned up a basic dashboarding system where you can see >>>> the results here[2]. Caveat: there are only two runs, and we’ll be tweaking >>>> the underlying system still, so don’t panic that we’re up and to the right >>>> currently. ;) >>>> >>>> This is the first test running continuously on top of the performance / IO >>>> testing infrastructure described in this doc[3]. Initial support for Beam >>>> is now present in PerfKit Benchmarker; given what they had already, it was >>>> easiest to add support for Dataflow and Java. We need your help to add >>>> additional support! The doc lists a number of JIRA issues to build out >>>> support for other systems. I’m happy to work with people to help them >>>> understand what is necessary for these tasks; just send an email to the >>>> list if you need help and I’ll help you move forwards. >>>> >>>> Looking forward to it! >>>> >>>> Jason >>>> >>>> [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/ >>>> [2] >>>> https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696 >>>> [3] >>>> >>>> https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73 >>>> >>>> -- >>>> ------- >>>> Jason Kuster >>>> Apache Beam / Google Cloud Dataflow >>>> >>> >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >
