This is a really exciting development! I would definitely like to help out. Still ingesting the docs and JIRAs.
On Tue, Mar 21, 2017 at 3:01 PM, Stephen Sisk <[email protected]> wrote: > I'm really excited to see these tests are running! > > These Jdbc tests are testing against a postgres instance - that instance is > running on the kubernetes cluster I've set up for beam IO ITs as discussed > in the "Hosting data stores for IO transform testing" thread[0]. I set up > that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can > run their own kubernetes cluster and do the same thing for themselves to > run the ITs. (I'd actually to love to hear about that if anyone does it.) > > I'm excited to get a few more ITs using this infrastructure so we can test > it out/smooth out the remaining rough edges in creating ITs. I'm happy to > answer questions about that on the mailing list, but we obviously have to > have the process written down - the Testing IO Transforms in Apache Beam > doc [2] covers how to do this, but is still rough. I'm working on getting > that up on the website and ironing out the rough edges [3], but generally > reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests > work should give you a sense of how to get it working. I'm also thinking we > might want to simplify the way we do data loading, so I don't consider this > process fully stabilized, but I'll port code written according to the > current standards to the new standards if we make changes. > > ElasticsearchIO has all the prerequisites, so I'd like to get them going in > the near future. I know JB has started on this in his RedisIO PR, and the > HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe. > For now, each datastore has to be manually set up, but I'd like to automate > that process - I'll file a JIRA ticket shortly for that. > > Thanks, > Stephen > [0] Hosting data stores for IO transform testing - > https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6a > c438b851818b639aba644607af@%3Cdev.beam.apache.org%3E > [1] Postgres k8 scripts - > https://github.com/apache/beam/tree/master/sdks/java/io/ > jdbc/src/test/resources/kubernetes > [2] IO testing guide - > https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg- > NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing > [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025 > > On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <[email protected]. > invalid> > wrote: > > > Hi all, > > > > Exciting news! As of yesterday, we have checked in the Jenkins > > configuration for our first continuously running IO Integration Test! You > > can check it out in Jenkins here[1]. We’re also publishing results to a > > database, and we’ve turned up a basic dashboarding system where you can > see > > the results here[2]. Caveat: there are only two runs, and we’ll be > tweaking > > the underlying system still, so don’t panic that we’re up and to the > right > > currently. ;) > > > > This is the first test running continuously on top of the performance / > IO > > testing infrastructure described in this doc[3]. Initial support for > Beam > > is now present in PerfKit Benchmarker; given what they had already, it > was > > easiest to add support for Dataflow and Java. We need your help to add > > additional support! The doc lists a number of JIRA issues to build out > > support for other systems. I’m happy to work with people to help them > > understand what is necessary for these tasks; just send an email to the > > list if you need help and I’ll help you move forwards. > > > > Looking forward to it! > > > > Jason > > > > [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/ > > [2] > > https://apache-beam-testing.appspot.com/explore?dashboard= > 5714163003293696 > > [3] > > > > https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOz > ph5FnL2DhaRDz0/edit?ts=58a78e73 > > > > -- > > ------- > > Jason Kuster > > Apache Beam / Google Cloud Dataflow > > >
