boyuanzz commented on a change in pull request #13282:
URL: https://github.com/apache/beam/pull/13282#discussion_r520978104
##########
File path: .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
##########
@@ -61,14 +61,38 @@ job(jobName) {
autoscalingAlgorithm : 'NONE'
]
+ Map runnerV2SdfWrapperPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-runnerv2',
+ bigQueryTable : 'kafkaioit_results_sdf_wrapper',
+ influxMeasurement : 'kafkaioit_results_sdf_wrapper',
+ experiments :
'beam_fn_api,use_runner_v2,use_unified_worker',
+ ]
+
+ Map runnerV2SdfPipelineOptions = pipelineOptions + [
+ kafkaTopic : 'beam-sdf',
+ bigQueryTable : 'kafkaioit_results_runner_v2',
+ influxMeasurement : 'kafkaioit_results_runner_v2',
+ experiments :
'beam_fn_api,use_runner_v2,use_unified_worker,use_sdf_kafka_read',
+ ]
+
steps {
gradle {
rootBuildScriptDir(common.checkoutDir)
common.setGradleSwitches(delegate)
switches("--info")
-
switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+
switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(runnerV2SdfWrapperPipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ switches("-Dexperiment=use_runner_v2")
Review comment:
`runner v2` is a new version of Dataflow. We are under migration from
old Dataflow runner to this new runner. We(dataflow) are more interested in
making sure SDF and KafkaIO work well on the new runner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
