[jira] [Updated] (BEAM-9979) Fix race condition where the read index maybe reported from the last executed bundle

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9979: Description: When the BeamFnDataReadRunner/DataInputOperation is reused there is a short period of time

[jira] [Updated] (BEAM-9979) Fix race condition where the read index maybe reported from the last executed bundle

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9979: Status: Open (was: Triage Needed) > Fix race condition where the read index maybe reported from the last

[jira] [Created] (BEAM-9979) Fix race condition where the read index maybe reported from the last executed bundle

2020-05-12 Thread Luke Cwik (Jira)
Luke Cwik created BEAM-9979: --- Summary: Fix race condition where the read index maybe reported from the last executed bundle Key: BEAM-9979 URL: https://issues.apache.org/jira/browse/BEAM-9979 Project: Beam

[jira] [Comment Edited] (BEAM-9945) Use consistent element count for progress counter.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105926#comment-17105926 ] Luke Cwik edited comment on BEAM-9945 at 5/13/20, 3:23 AM: --- When testing the

[jira] [Commented] (BEAM-9945) Use consistent element count for progress counter.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105937#comment-17105937 ] Luke Cwik commented on BEAM-9945: - Fix in https://github.com/apache/beam/pull/11689 > Use consistent

[jira] [Updated] (BEAM-9945) Use consistent element count for progress counter.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9945: Status: Open (was: Triage Needed) > Use consistent element count for progress counter. >

[jira] [Reopened] (BEAM-9945) Use consistent element count for progress counter.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik reopened BEAM-9945: - When testing the implementation against runner v2, I found that the Go SDK's read index ends at *# elements*

[jira] [Closed] (BEAM-8964) DeprecationWarning: Flags not at the start of the expression

2020-05-12 Thread Udi Meiri (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-8964. --- Fix Version/s: Not applicable Resolution: Fixed > DeprecationWarning: Flags not at the start of the

[jira] [Assigned] (BEAM-8964) DeprecationWarning: Flags not at the start of the expression

2020-05-12 Thread Udi Meiri (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-8964: --- Assignee: Udi Meiri > DeprecationWarning: Flags not at the start of the expression >

[jira] [Commented] (BEAM-8964) DeprecationWarning: Flags not at the start of the expression

2020-05-12 Thread Udi Meiri (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105915#comment-17105915 ] Udi Meiri commented on BEAM-8964: - Oops I fixed this in https://github.com/apache/beam/pull/11016, but

[jira] [Updated] (BEAM-9978) Add offset range restrictions to the Go SDK proper.

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9978: -- Status: Open (was: Triage Needed) > Add offset range restrictions to the Go SDK proper. >

[jira] [Updated] (BEAM-9978) Add offset range restrictions to the Go SDK.

2020-05-12 Thread Daniel Oliveira (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9978: -- Summary: Add offset range restrictions to the Go SDK. (was: Add offset range restrictions to

[jira] [Created] (BEAM-9978) Add offset range restrictions to the Go SDK proper.

2020-05-12 Thread Daniel Oliveira (Jira)
Daniel Oliveira created BEAM-9978: - Summary: Add offset range restrictions to the Go SDK proper. Key: BEAM-9978 URL: https://issues.apache.org/jira/browse/BEAM-9978 Project: Beam Issue Type:

[jira] [Comment Edited] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Ahmet Altay (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105894#comment-17105894 ] Ahmet Altay edited comment on BEAM-9975 at 5/13/20, 2:26 AM: - Could we add a

[jira] [Resolved] (BEAM-9775) Add capability to run SDFs on Flink Runner and Python FnApiRunner.

2020-05-12 Thread Daniel Oliveira (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9775. --- Fix Version/s: Not applicable Resolution: Fixed This seems complete now. We have an

[jira] [Commented] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Ahmet Altay (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105894#comment-17105894 ] Ahmet Altay commented on BEAM-9975: --- Could we add a log error before the raise and log the value instead

[jira] [Resolved] (BEAM-9799) Add automated RTracker validation to Go SDF

2020-05-12 Thread Daniel Oliveira (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9799. --- Fix Version/s: Not applicable Resolution: Fixed > Add automated RTracker validation to

[jira] [Commented] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105866#comment-17105866 ] Luke Cwik commented on BEAM-9934: - 2.22 > Resolve differences in beam:metric:element_count:v1

[jira] [Updated] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9934: Fix Version/s: (was: 2.21.0) 2.22.0 > Resolve differences in

[jira] [Commented] (BEAM-9767) test_streaming_wordcount flaky timeouts

2020-05-12 Thread Ahmet Altay (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105858#comment-17105858 ] Ahmet Altay commented on BEAM-9767: --- https://github.com/apache/beam/pull/11663 is merged. Does this

[jira] [Commented] (BEAM-9934) Resolve differences in beam:metric:element_count:v1 implementations

2020-05-12 Thread Kyle Weaver (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105846#comment-17105846 ] Kyle Weaver commented on BEAM-9934: --- What's the status on this one? Are we going to try to fix this for

[jira] [Commented] (BEAM-9935) Resolve differences in allowed_split_point implementations

2020-05-12 Thread Kyle Weaver (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105842#comment-17105842 ] Kyle Weaver commented on BEAM-9935: --- https://github.com/apache/beam/pull/11671 solved this for Python,

[jira] [Updated] (BEAM-9935) Resolve differences in allowed_split_point implementations

2020-05-12 Thread Kyle Weaver (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9935: -- Fix Version/s: (was: 2.21.0) 2.22.0 > Resolve differences in allowed_split_point

[jira] [Resolved] (BEAM-9945) Use consistent element count for progress counter.

2020-05-12 Thread Kyle Weaver (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver resolved BEAM-9945. --- Resolution: Fixed > Use consistent element count for progress counter. >

[jira] [Commented] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Kyle Weaver (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105836#comment-17105836 ] Kyle Weaver commented on BEAM-9975: --- The error message makes it impossible to tell which option is

[jira] [Commented] (BEAM-9707) Hardcode runner harness container image for unified worker

2020-05-12 Thread Brian Hulette (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105830#comment-17105830 ] Brian Hulette commented on BEAM-9707: - What happens if 2.21 isn't released before the 2.22 cut date?

[jira] [Resolved] (BEAM-9622) Support for consuming tagged PCollections in Python SqlTransform

2020-05-12 Thread Brian Hulette (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette resolved BEAM-9622. - Fix Version/s: 2.22.0 Resolution: Fixed > Support for consuming tagged PCollections in

[jira] [Updated] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Kyle Weaver (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9975: -- Description: Error looks similar to the one in BEAM-9907. Example from

[jira] [Assigned] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Brian Hulette (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette reassigned BEAM-9975: --- Assignee: (was: Brian Hulette) > PortableRunnerTest flake "ParseError: Unexpected type

[jira] [Created] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn

2020-05-12 Thread Boyuan Zhang (Jira)
Boyuan Zhang created BEAM-9977: -- Summary: Build Kafka Read on top of Java SplittableDoFn Key: BEAM-9977 URL: https://issues.apache.org/jira/browse/BEAM-9977 Project: Beam Issue Type: New

[jira] [Updated] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9977: -- Status: Open (was: Triage Needed) > Build Kafka Read on top of Java SplittableDoFn >

[jira] [Updated] (BEAM-9488) Python SDK sending unexpected MonitoringInfo

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9488: Fix Version/s: (was: 2.22.0) 2.21.0 > Python SDK sending unexpected MonitoringInfo >

[jira] [Commented] (BEAM-9959) Mistakes Computing Composite Inputs and Outputs

2020-05-12 Thread Robert Burke (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105750#comment-17105750 ] Robert Burke commented on BEAM-9959: The right overall fix for that is to check for cycles WRT the

[jira] [Created] (BEAM-9976) FlinkSavepointTest timeout flake

2020-05-12 Thread Kyle Weaver (Jira)
Kyle Weaver created BEAM-9976: - Summary: FlinkSavepointTest timeout flake Key: BEAM-9976 URL: https://issues.apache.org/jira/browse/BEAM-9976 Project: Beam Issue Type: Bug Components:

[jira] [Commented] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Brian Hulette (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105743#comment-17105743 ] Brian Hulette commented on BEAM-9975: - In BEAM-9907 the underlying issue was that PipelineOptions

[jira] [Created] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Brian Hulette (Jira)
Brian Hulette created BEAM-9975: --- Summary: PortableRunnerTest flake "ParseError: Unexpected type for Value message." Key: BEAM-9975 URL: https://issues.apache.org/jira/browse/BEAM-9975 Project: Beam

[jira] [Updated] (BEAM-9975) PortableRunnerTest flake "ParseError: Unexpected type for Value message."

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9975: -- Status: Open (was: Triage Needed) > PortableRunnerTest flake "ParseError: Unexpected type for

[jira] [Commented] (BEAM-9843) Flink UnboundedSourceWrapperTest flaky due to a timeout

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105723#comment-17105723 ] Maximilian Michels commented on BEAM-9843: -- Thanks for reporting this. This should be fixed. Feel

[jira] [Resolved] (BEAM-9164) [PreCommit_Java] [Flake] UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels resolved BEAM-9164. -- Fix Version/s: Not applicable Resolution: Fixed This should be fixed but feel

[jira] [Updated] (BEAM-9969) Beam ZetaSQL supports pure SQL user-defined table functions

2020-05-12 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated BEAM-9969: --- Description: There are four types of implementation of table-valued function: 1. FixedSchemaOutput. The

[jira] [Created] (BEAM-9974) beam_PostRelease_NightlySnapshot failing

2020-05-12 Thread Kyle Weaver (Jira)
Kyle Weaver created BEAM-9974: - Summary: beam_PostRelease_NightlySnapshot failing Key: BEAM-9974 URL: https://issues.apache.org/jira/browse/BEAM-9974 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-9973) beam_PostCommit_Py_ValCont flakes (before running tests)

2020-05-12 Thread Kyle Weaver (Jira)
Kyle Weaver created BEAM-9973: - Summary: beam_PostCommit_Py_ValCont flakes (before running tests) Key: BEAM-9973 URL: https://issues.apache.org/jira/browse/BEAM-9973 Project: Beam Issue Type:

[jira] [Created] (BEAM-9972) Test pure SQL UDF for all data types

2020-05-12 Thread Rui Wang (Jira)
Rui Wang created BEAM-9972: -- Summary: Test pure SQL UDF for all data types Key: BEAM-9972 URL: https://issues.apache.org/jira/browse/BEAM-9972 Project: Beam Issue Type: Task Components:

[jira] [Updated] (BEAM-9969) Beam ZetaSQL supports pure SQL user-defined table functions

2020-05-12 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated BEAM-9969: --- Status: Open (was: Triage Needed) > Beam ZetaSQL supports pure SQL user-defined table functions >

[jira] [Updated] (BEAM-9971) beam_PostCommit_Java_PVR_Spark_Batch flakes (no such file)

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9971: -- Status: Open (was: Triage Needed) > beam_PostCommit_Java_PVR_Spark_Batch flakes (no such file)

[jira] [Created] (BEAM-9971) beam_PostCommit_Java_PVR_Spark_Batch flakes (no such file)

2020-05-12 Thread Kyle Weaver (Jira)
Kyle Weaver created BEAM-9971: - Summary: beam_PostCommit_Java_PVR_Spark_Batch flakes (no such file) Key: BEAM-9971 URL: https://issues.apache.org/jira/browse/BEAM-9971 Project: Beam Issue Type:

[jira] [Created] (BEAM-9970) beam_PostRelease_NightlySnapshot failing

2020-05-12 Thread Kyle Weaver (Jira)
Kyle Weaver created BEAM-9970: - Summary: beam_PostRelease_NightlySnapshot failing Key: BEAM-9970 URL: https://issues.apache.org/jira/browse/BEAM-9970 Project: Beam Issue Type: Bug

[jira] [Assigned] (BEAM-9968) beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-9968: - Assignee: Luke Cwik > beam_PreCommit_Java_Cron >

[jira] [Commented] (BEAM-9968) beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105664#comment-17105664 ] Kenneth Knowles commented on BEAM-9968: --- Actually stderr is flooded with a bunch of errors. Not

[jira] [Created] (BEAM-9969) Beam ZetaSQL supports pure SQL user-defined table functions

2020-05-12 Thread Rui Wang (Jira)
Rui Wang created BEAM-9969: -- Summary: Beam ZetaSQL supports pure SQL user-defined table functions Key: BEAM-9969 URL: https://issues.apache.org/jira/browse/BEAM-9969 Project: Beam Issue Type: Task

[jira] [Work started] (BEAM-9622) Support for consuming tagged PCollections in Python SqlTransform

2020-05-12 Thread Brian Hulette (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9622 started by Brian Hulette. --- > Support for consuming tagged PCollections in Python SqlTransform >

[jira] [Assigned] (BEAM-9622) Support for consuming tagged PCollections in Python SqlTransform

2020-05-12 Thread Brian Hulette (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette reassigned BEAM-9622: --- Assignee: Brian Hulette > Support for consuming tagged PCollections in Python SqlTransform >

[jira] [Commented] (BEAM-9968) beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105661#comment-17105661 ] Kenneth Knowles commented on BEAM-9968: --- I did not see an obvious suspect commit. It has failed

[jira] [Updated] (BEAM-9968) beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9968: -- Description: _Use this form to file an issue for test failure:_ *

[jira] [Commented] (BEAM-9968) beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105660#comment-17105660 ] Kenneth Knowles commented on BEAM-9968: --- CC [~lcwik] and [~chamikara]. Any ideas on this one? >

[jira] [Created] (BEAM-9968) beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure

2020-05-12 Thread Kenneth Knowles (Jira)
Kenneth Knowles created BEAM-9968: - Summary: beam_PreCommit_Java_Cron org.apache.beam.runners.fnexecution.control.RemoteExecutionTest.testSplit Failure Key: BEAM-9968 URL:

[jira] [Updated] (BEAM-9954) Beam ZetaSQL supports pure SQL user-defined aggregation functions

2020-05-12 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated BEAM-9954: --- Description: One naive example is {code:sql} CREATE AGGREGATE FUNCTION fun (a INT64) AS (SUM(a)); {code}

[jira] [Updated] (BEAM-9954) Beam ZetaSQL supports pure SQL user-defined aggregation functions

2020-05-12 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated BEAM-9954: --- Description: One naive example is {code:sql} CREATE AGGREGATE FUNCTION fun (a INT64, b INT64) AS

[jira] [Updated] (BEAM-9967) Add support for BigQuery job labels on ReadFrom/WriteTo(BigQuery) transforms

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9967: -- Status: Open (was: Triage Needed) > Add support for BigQuery job labels on

[jira] [Created] (BEAM-9967) Add support for BigQuery job labels on ReadFrom/WriteTo(BigQuery) transforms

2020-05-12 Thread Pablo Estrada (Jira)
Pablo Estrada created BEAM-9967: --- Summary: Add support for BigQuery job labels on ReadFrom/WriteTo(BigQuery) transforms Key: BEAM-9967 URL: https://issues.apache.org/jira/browse/BEAM-9967 Project: Beam

[jira] [Updated] (BEAM-9807) PostRelease_NightlySnapshot failing due to missing "region" flag

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9807: -- Priority: Critical (was: Major) > PostRelease_NightlySnapshot failing due to missing "region"

[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?

2020-05-12 Thread Alexey Romanenko (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105641#comment-17105641 ] Alexey Romanenko commented on BEAM-9760: Hi [~wongkawah], thanks for details. I see your point but

[jira] [Updated] (BEAM-9966) Investigate variance in checkpoint duration of ParDo streaming tests

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9966: -- Status: Open (was: Triage Needed) > Investigate variance in checkpoint duration of ParDo

[jira] [Created] (BEAM-9966) Investigate variance in checkpoint duration of ParDo streaming tests

2020-05-12 Thread Maximilian Michels (Jira)
Maximilian Michels created BEAM-9966: Summary: Investigate variance in checkpoint duration of ParDo streaming tests Key: BEAM-9966 URL: https://issues.apache.org/jira/browse/BEAM-9966 Project:

[jira] [Commented] (BEAM-6327) Don't attempt to fuse subtransforms of primitive/known transforms.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105629#comment-17105629 ] Luke Cwik commented on BEAM-6327: - I'm not sure if we want to add it to the fuser since the pipeline

[jira] [Commented] (BEAM-6327) Don't attempt to fuse subtransforms of primitive/known transforms.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105627#comment-17105627 ] Luke Cwik commented on BEAM-6327: - I think this shouldn't be considered "trimming" but native transform

[jira] [Assigned] (BEAM-6327) Don't attempt to fuse subtransforms of primitive/known transforms.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik reassigned BEAM-6327: --- Assignee: Kyle Weaver (was: Luke Cwik) > Don't attempt to fuse subtransforms of primitive/known

[jira] [Assigned] (BEAM-6327) Don't attempt to fuse subtransforms of primitive/known transforms.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik reassigned BEAM-6327: --- Assignee: Luke Cwik (was: Kyle Weaver) > Don't attempt to fuse subtransforms of primitive/known

[jira] [Commented] (BEAM-9900) Remove the need for shutdownSourcesOnFinalWatermark flag

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105620#comment-17105620 ] Maximilian Michels commented on BEAM-9900: -- For context:

[jira] [Created] (BEAM-9965) Explain direct runner running modes

2020-05-12 Thread Kyle Weaver (Jira)
Kyle Weaver created BEAM-9965: - Summary: Explain direct runner running modes Key: BEAM-9965 URL: https://issues.apache.org/jira/browse/BEAM-9965 Project: Beam Issue Type: Improvement

[jira] [Updated] (BEAM-9959) Mistakes Computing Composite Inputs and Outputs

2020-05-12 Thread Robert Burke (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Burke updated BEAM-9959: --- Description: The Go SDK uses a Scope object to manage beam Composites. A bug was discovered when

[jira] [Updated] (BEAM-9964) Setting workerCacheMb to make its way to the WindmillStateCache Constructor

2020-05-12 Thread Omar Ismail (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omar Ismail updated BEAM-9964: -- Description: Setting --workerCacheMB seems to affect batch pipelines only. For Streaming, the cache

[jira] [Created] (BEAM-9964) Setting workerCacheMb to make its way to the WindmillStateCache Constructor

2020-05-12 Thread Omar Ismail (Jira)
Omar Ismail created BEAM-9964: - Summary: Setting workerCacheMb to make its way to the WindmillStateCache Constructor Key: BEAM-9964 URL: https://issues.apache.org/jira/browse/BEAM-9964 Project: Beam

[jira] [Commented] (BEAM-9945) Use consistent element count for progress counter.

2020-05-12 Thread Luke Cwik (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105512#comment-17105512 ] Luke Cwik commented on BEAM-9945: - PR 11652 needs to be cherry picked into 2.21 release branch > Use

[jira] [Assigned] (BEAM-8133) Create metrics publisher in Java SDK

2020-05-12 Thread Kamil Wasilewski (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Wasilewski reassigned BEAM-8133: -- Assignee: Pawel Pasterz > Create metrics publisher in Java SDK >

[jira] [Closed] (BEAM-8132) Create metrics publisher in Python SDK

2020-05-12 Thread Kamil Wasilewski (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Wasilewski closed BEAM-8132. -- > Create metrics publisher in Python SDK > -- > >

[jira] [Resolved] (BEAM-8132) Create metrics publisher in Python SDK

2020-05-12 Thread Kamil Wasilewski (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Wasilewski resolved BEAM-8132. Fix Version/s: Not applicable Resolution: Fixed > Create metrics publisher in Python

[jira] [Closed] (BEAM-8133) Create metrics publisher in Java SDK

2020-05-12 Thread Kamil Wasilewski (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Wasilewski closed BEAM-8133. -- Fix Version/s: Not applicable Resolution: Fixed > Create metrics publisher in Java SDK >

[jira] [Closed] (BEAM-8131) Provide Kubernetes setup with Prometheus

2020-05-12 Thread Kamil Wasilewski (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Wasilewski closed BEAM-8131. -- > Provide Kubernetes setup with Prometheus > > >

[jira] [Updated] (BEAM-8742) Add stateful processing to ParDo load test

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels updated BEAM-8742: - Description: So far, the ParDo load test is not stateful. We should add a basic counter

[jira] [Commented] (BEAM-8742) Add stateful processing to ParDo load test

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105339#comment-17105339 ] Maximilian Michels commented on BEAM-8742: -- Cron:

[jira] [Resolved] (BEAM-9963) Flink streaming load tests fail with TypeError

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels resolved BEAM-9963. -- Fix Version/s: Not applicable Resolution: Fixed > Flink streaming load tests

[jira] [Updated] (BEAM-9963) Flink streaming load tests fail with TypeError

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels updated BEAM-9963: - Description: The newly added load tests now fail on the latest master.

[jira] [Created] (BEAM-9963) Flink streaming load tests fail with TypeError

2020-05-12 Thread Maximilian Michels (Jira)
Maximilian Michels created BEAM-9963: Summary: Flink streaming load tests fail with TypeError Key: BEAM-9963 URL: https://issues.apache.org/jira/browse/BEAM-9963 Project: Beam Issue

[jira] [Updated] (BEAM-9963) Flink streaming load tests fail with TypeError

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9963: -- Status: Open (was: Triage Needed) > Flink streaming load tests fail with TypeError >

[jira] [Resolved] (BEAM-9947) Timer coder contains a faulty key coder leading to a corrupted encoding

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels resolved BEAM-9947. -- Resolution: Fixed > Timer coder contains a faulty key coder leading to a corrupted

[jira] [Updated] (BEAM-9917) BigQueryBatchFileLoads dynamic destination

2020-05-12 Thread Jira
[ https://issues.apache.org/jira/browse/BEAM-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tord Sætren updated BEAM-9917: -- Priority: Minor (was: Major) > BigQueryBatchFileLoads dynamic destination >

[jira] [Resolved] (BEAM-6710) Add Landing page to community metrics dashboard

2020-05-12 Thread Kamil Wasilewski (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Wasilewski resolved BEAM-6710. Fix Version/s: Not applicable Resolution: Fixed > Add Landing page to community

[jira] [Commented] (BEAM-8742) Add stateful processing to ParDo load test

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105273#comment-17105273 ] Maximilian Michels commented on BEAM-8742: -- I just saw your comment here. Thank you. I added

[jira] [Created] (BEAM-9962) CI/CD for Beam

2020-05-12 Thread Ankur Goenka (Jira)
Ankur Goenka created BEAM-9962: -- Summary: CI/CD for Beam Key: BEAM-9962 URL: https://issues.apache.org/jira/browse/BEAM-9962 Project: Beam Issue Type: Bug Components: sdk-ideas

[jira] [Commented] (BEAM-7438) Distribution and Gauge metrics are not being exported to Flink dashboard neither Prometheus IO

2020-05-12 Thread Akshay Iyangar (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105192#comment-17105192 ] Akshay Iyangar commented on BEAM-7438: -- [~mxm] - Facing the same issue will have a PR out for this

[jira] [Commented] (BEAM-9164) [PreCommit_Java] [Flake] UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest

2020-05-12 Thread Maximilian Michels (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105184#comment-17105184 ] Maximilian Michels commented on BEAM-9164: -- I was not aware that tests had been disabled. I'm a

[jira] [Assigned] (BEAM-7438) Distribution and Gauge metrics are not being exported to Flink dashboard neither Prometheus IO

2020-05-12 Thread Akshay Iyangar (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay Iyangar reassigned BEAM-7438: Assignee: Akshay Iyangar > Distribution and Gauge metrics are not being exported to Flink

[jira] [Created] (BEAM-9961) Python MongoDBIO does not apply projection

2020-05-12 Thread Corvin Deboeser (Jira)
Corvin Deboeser created BEAM-9961: - Summary: Python MongoDBIO does not apply projection Key: BEAM-9961 URL: https://issues.apache.org/jira/browse/BEAM-9961 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-9960) Python MongoDBIO fails when response of split vector command is larger than 16mb

2020-05-12 Thread Corvin Deboeser (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corvin Deboeser updated BEAM-9960: -- Description: When using MongoDBIO on a large collection and the source bundle size was

[jira] [Created] (BEAM-9960) Python MongoDBIO fails when response of split vector command is larger than 16mb

2020-05-12 Thread Corvin Deboeser (Jira)
Corvin Deboeser created BEAM-9960: - Summary: Python MongoDBIO fails when response of split vector command is larger than 16mb Key: BEAM-9960 URL: https://issues.apache.org/jira/browse/BEAM-9960

[jira] [Updated] (BEAM-9959) Mistakes Computing Composite Inputs and Outputs

2020-05-12 Thread Kenneth Knowles (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9959: -- Status: Open (was: Triage Needed) > Mistakes Computing Composite Inputs and Outputs >

[jira] [Created] (BEAM-9959) Mistakes Computing Composite Inputs and Outputs

2020-05-12 Thread Robert Burke (Jira)
Robert Burke created BEAM-9959: -- Summary: Mistakes Computing Composite Inputs and Outputs Key: BEAM-9959 URL: https://issues.apache.org/jira/browse/BEAM-9959 Project: Beam Issue Type: Bug