[beam] branch master updated (2f2e4fa -> 7ebccbc)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 2f2e4fa Merge pull request #15922 from ibzib/es-flake
add 7ebccbc [BEAM-11097] Create hook to enable cross-bundle side input
caching (#15896)
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 1 +
.../runtime/harness/{sampler.go => cache_hooks.go} | 45 --
sdks/go/pkg/beam/core/runtime/harness/harness.go | 14 ---
.../core/runtime/harness/statecache/statecache.go | 20 +-
.../runtime/harness/statecache/statecache_test.go | 13 ++-
sdks/go/pkg/beam/util/harnessopts/cache.go | 42
.../harnessopts/cache_test.go} | 43 ++---
7 files changed, 128 insertions(+), 50 deletions(-)
copy sdks/go/pkg/beam/core/runtime/harness/{sampler.go => cache_hooks.go} (58%)
create mode 100644 sdks/go/pkg/beam/util/harnessopts/cache.go
copy sdks/go/pkg/beam/{core/runtime/xlangx/payload_test.go =>
util/harnessopts/cache_test.go} (51%)
[beam] branch master updated (7ebccbc -> 39bd3a4)
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7ebccbc [BEAM-11097] Create hook to enable cross-bundle side input caching (#15896) add 591f00e [BEAM-13056] Expose FieldAccess in DoFnSchemaInformation add 39bd3a4 Merge pull request #15886: [BEAM-13056] Expose FieldAccess in DoFnSchemaInformation No new revisions were added by this update. Summary of changes: .../beam/sdk/transforms/DoFnSchemaInformation.java | 20 .../java/org/apache/beam/sdk/transforms/ParDo.java | 11 ++ .../org/apache/beam/sdk/transforms/ParDoTest.java | 126 + .../extensions/sql/impl/rel/BeamCalcRelTest.java | 29 +++-- .../sql/zetasql/BeamZetaSqlCalcRelTest.java| 29 +++-- 5 files changed, 183 insertions(+), 32 deletions(-)
[beam] branch master updated (39bd3a4 -> 57e331b)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 39bd3a4 Merge pull request #15886: [BEAM-13056] Expose FieldAccess in DoFnSchemaInformation add 57e331b Merge pull request #15804 from [BEAM-13109][Playground] Add processing of timeout for RunCode API method No new revisions were added by this update. Summary of changes: playground/api/v1/api.proto| 16 +- playground/backend/cmd/server/controller.go| 131 +++--- playground/backend/cmd/server/controller_test.go | 19 ++- playground/backend/cmd/server/server.go| 3 + playground/backend/internal/api/v1/api.pb.go | 189 +++-- .../environment/environment_service_test.go| 8 +- playground/backend/internal/executors/executor.go | 31 ++-- .../backend/internal/executors/executor_test.go| 5 +- playground/frontend/lib/api/v1/api.pbenum.dart | 18 +- playground/frontend/lib/api/v1/api.pbgrpc.dart | 9 +- playground/frontend/lib/api/v1/api.pbjson.dart | 18 +- 11 files changed, 294 insertions(+), 153 deletions(-)
[beam] branch master updated (57e331b -> e237c2f)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 57e331b Merge pull request #15804 from [BEAM-13109][Playground] Add processing of timeout for RunCode API method add e237c2f Improve FhirIO LRO Counters + minor fixes (#15921) No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/gcp/healthcare/FhirIO.java | 130 ++--- .../io/gcp/healthcare/HttpHealthcareApiClient.java | 2 +- 2 files changed, 89 insertions(+), 43 deletions(-)
[beam] branch master updated (e237c2f -> 6c954ca)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e237c2f Improve FhirIO LRO Counters + minor fixes (#15921) new c4feaff Minor HL7v2IO Fixes new a1bfd08 Merge branch 'apache:master' into PIPEWHEEL new 6c954ca Merge pull request #15868 from msbukal/PIPEWHEEL The 33480 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java | 81 +++--- 1 file changed, 40 insertions(+), 41 deletions(-)
[beam] branch master updated (6c954ca -> 9b1512d)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6c954ca Merge pull request #15868 from msbukal/PIPEWHEEL add 9b1512d Merge pull request #15771 from Enable BQ Standard SQL dialect style table specifications. No new revisions were added by this update. Summary of changes: .../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 11 +++ .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 2 +- .../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 23 +- 3 files changed, 30 insertions(+), 6 deletions(-)
[beam] branch master updated (9b1512d -> 585e081)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9b1512d Merge pull request #15771 from Enable BQ Standard SQL dialect style table specifications. new cb54dca [BEAM-12976] Add Java interfaces for projection pushdown. new cd2ac1f fix cast new 3f10676 [BEAM-12976] Address review comments. new 2249c9f Hard-code output tag instead of making it public. new 72a5054 revert MAIN_OUTPUT_TAG to private new 31b66f4 Remove WITHOUT_FIELD_ORDERING. Field reordering will be a prerequisite for pushdown support. new 585e081 Merge pull request #15712 from ibzib/generic-pushdown The 33488 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/schemas/ProjectionConsumer.java | 41 ++ .../beam/sdk/schemas/ProjectionProducer.java | 46 + .../beam/sdk/schemas/io/PushdownProjector.java | 48 -- .../provider/SchemaIOTableProviderWrapper.java | 19 + .../provider/TestSchemaIOTableProviderWrapper.java | 25 ++- 5 files changed, 112 insertions(+), 67 deletions(-) create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/ProjectionConsumer.java create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/ProjectionProducer.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/PushdownProjector.java
[beam] branch master updated: [BEAM-11217] Metrics querying for Pcol metrics (#15923)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ededcb5 [BEAM-11217] Metrics querying for Pcol metrics (#15923)
ededcb5 is described below
commit ededcb50149eab212605690211b90c9eeb747fcb
Author: Ritesh Ghorse
AuthorDate: Tue Nov 9 17:32:30 2021 -0500
[BEAM-11217] Metrics querying for Pcol metrics (#15923)
---
sdks/go/pkg/beam/core/metrics/metrics.go | 89 --
sdks/go/pkg/beam/core/metrics/metrics_test.go | 37 +
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 62 ---
.../beam/core/runtime/metricsx/metricsx_test.go| 9 ++-
.../beam/runners/dataflow/dataflowlib/metrics.go | 2 +-
.../beam/runners/universal/runnerlib/execute.go| 6 +-
6 files changed, 180 insertions(+), 25 deletions(-)
diff --git a/sdks/go/pkg/beam/core/metrics/metrics.go
b/sdks/go/pkg/beam/core/metrics/metrics.go
index 2c636e2..0598c7e 100644
--- a/sdks/go/pkg/beam/core/metrics/metrics.go
+++ b/sdks/go/pkg/beam/core/metrics/metrics.go
@@ -479,11 +479,17 @@ func (m *executionState) kind() kind {
return kindDoFnMsec
}
-// MsecValue is the value of a single msec metric
+// MsecValue is the value of a single msec metric.
type MsecValue struct {
Start, Process, Finish, Total time.Duration
}
+// PColValue is the value of a single PCollection metric.
+type PColValue struct {
+ ElementCountint64
+ SampledByteSize DistributionValue
+}
+
// Results represents all metrics gathered during the job's execution.
// It allows for querying metrics using a provided filter.
type Results struct {
@@ -491,6 +497,7 @@ type Results struct {
distributions []DistributionResult
gauges[]GaugeResult
msecs []MsecResult
+ pCols []PColResult
}
// NewResults creates a new Results.
@@ -498,17 +505,16 @@ func NewResults(
counters []CounterResult,
distributions []DistributionResult,
gauges []GaugeResult,
- msecs []MsecResult) *Results {
- return &Results{counters, distributions, gauges, msecs}
+ msecs []MsecResult,
+ pCols []PColResult) *Results {
+ return &Results{counters, distributions, gauges, msecs, pCols}
}
// AllMetrics returns all metrics from a Results instance.
func (mr Results) AllMetrics() QueryResults {
- return QueryResults{mr.counters, mr.distributions, mr.gauges, mr.msecs}
+ return QueryResults{mr.counters, mr.distributions, mr.gauges, mr.msecs,
mr.pCols}
}
-// TODO(BEAM-11217): Implement querying metrics by DoFn
-
// SingleResult interface facilitates metrics query filtering methods.
type SingleResult interface {
Name() string
@@ -525,6 +531,7 @@ func (mr Results) Query(f func(SingleResult) bool)
QueryResults {
distributions := []DistributionResult{}
gauges := []GaugeResult{}
msecs := []MsecResult{}
+ pCols := []PColResult{}
for _, counter := range mr.counters {
if f(counter) {
@@ -546,7 +553,12 @@ func (mr Results) Query(f func(SingleResult) bool)
QueryResults {
msecs = append(msecs, msec)
}
}
- return QueryResults{counters: counters, distributions: distributions,
gauges: gauges, msecs: msecs}
+ for _, pCol := range mr.pCols {
+ if f(pCol) {
+ pCols = append(pCols, pCol)
+ }
+ }
+ return QueryResults{counters: counters, distributions: distributions,
gauges: gauges, msecs: msecs, pCols: pCols}
}
// QueryResults is the result of a query. Allows accessing all of the
@@ -556,6 +568,7 @@ type QueryResults struct {
distributions []DistributionResult
gauges[]GaugeResult
msecs []MsecResult
+ pCols []PColResult
}
// Counters returns a slice of counter metrics.
@@ -586,6 +599,13 @@ func (qr QueryResults) Msecs() []MsecResult {
return out
}
+// PCols returns a slice of PCollection metrics.
+func (qr QueryResults) PCols() []PColResult {
+ out := make([]PColResult, len(qr.pCols))
+ copy(out, qr.pCols)
+ return out
+}
+
// CounterResult is an attempted and a commited value of a counter metric plus
// key.
type CounterResult struct {
@@ -725,6 +745,61 @@ func (r GaugeResult) Namespace() string {
// Transform returns the Transform step for this GaugeResult.
func (r GaugeResult) Transform() string { return r.Key.Step }
+// PColResult is an attempted and a commited value of a pcollection
+// metric plus key.
+type PColResult struct {
+ Attempted, Committed PColValue
+ Key StepKey
+}
+
+// Result returns committed metrics. Falls back to attempted metrics if
committed
+// are not populated (e.g. due to not being
[beam] branch master updated (ededcb5 -> 8da177f)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ededcb5 [BEAM-11217] Metrics querying for Pcol metrics (#15923) add ca6b25a Add detection of BundleFinalizers to DoFnSignatures add 90fbace Add enable_prime to the unified worker experiment list. add d7a87d0 Raise an exception before trying to launch a pipeline using BundleFinalizers on dataflow when runnerv2 is not in use add 8a1addc Fix detection to properly null check. new 8da177f Merge pull request #15904: Raise an exception before trying to launch a pipeline using BundleFinalizers on dataflow when runnerv2 is not in use The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/runners/dataflow/DataflowRunner.java | 11 ++- .../beam/sdk/transforms/reflect/DoFnSignatures.java | 20 2 files changed, 30 insertions(+), 1 deletion(-)
[beam] 01/01: Merge pull request #15904: Raise an exception before trying to launch a pipeline using BundleFinalizers on dataflow when runnerv2 is not in use
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8da177f64d314cf72e89a51e51fb0915f706a784 Merge: ededcb5 8a1addc Author: Kenn Knowles AuthorDate: Tue Nov 9 15:45:46 2021 -0800 Merge pull request #15904: Raise an exception before trying to launch a pipeline using BundleFinalizers on dataflow when runnerv2 is not in use .../apache/beam/runners/dataflow/DataflowRunner.java | 11 ++- .../beam/sdk/transforms/reflect/DoFnSignatures.java | 20 2 files changed, 30 insertions(+), 1 deletion(-)
[beam] branch nightly-refs/heads/master updated (2f2e4fa -> 8da177f)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 2f2e4fa Merge pull request #15922 from ibzib/es-flake
add 7ebccbc [BEAM-11097] Create hook to enable cross-bundle side input
caching (#15896)
add 591f00e [BEAM-13056] Expose FieldAccess in DoFnSchemaInformation
add 39bd3a4 Merge pull request #15886: [BEAM-13056] Expose FieldAccess in
DoFnSchemaInformation
add 57e331b Merge pull request #15804 from [BEAM-13109][Playground] Add
processing of timeout for RunCode API method
add e237c2f Improve FhirIO LRO Counters + minor fixes (#15921)
add c4feaff Minor HL7v2IO Fixes
add a1bfd08 Merge branch 'apache:master' into PIPEWHEEL
add 6c954ca Merge pull request #15868 from msbukal/PIPEWHEEL
add 9b1512d Merge pull request #15771 from Enable BQ Standard SQL dialect
style table specifications.
add cb54dca [BEAM-12976] Add Java interfaces for projection pushdown.
add cd2ac1f fix cast
add 3f10676 [BEAM-12976] Address review comments.
add 2249c9f Hard-code output tag instead of making it public.
add 72a5054 revert MAIN_OUTPUT_TAG to private
add 31b66f4 Remove WITHOUT_FIELD_ORDERING. Field reordering will be a
prerequisite for pushdown support.
add 585e081 Merge pull request #15712 from ibzib/generic-pushdown
add ededcb5 [BEAM-11217] Metrics querying for Pcol metrics (#15923)
add ca6b25a Add detection of BundleFinalizers to DoFnSignatures
add 90fbace Add enable_prime to the unified worker experiment list.
add d7a87d0 Raise an exception before trying to launch a pipeline using
BundleFinalizers on dataflow when runnerv2 is not in use
add 8a1addc Fix detection to properly null check.
add 8da177f Merge pull request #15904: Raise an exception before trying
to launch a pipeline using BundleFinalizers on dataflow when runnerv2 is not in
use
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 1 +
playground/api/v1/api.proto| 16 +-
playground/backend/cmd/server/controller.go| 131 +++---
playground/backend/cmd/server/controller_test.go | 19 ++-
playground/backend/cmd/server/server.go| 3 +
playground/backend/internal/api/v1/api.pb.go | 189 +++--
.../environment/environment_service_test.go| 8 +-
playground/backend/internal/executors/executor.go | 31 ++--
.../backend/internal/executors/executor_test.go| 5 +-
playground/frontend/lib/api/v1/api.pbenum.dart | 18 +-
playground/frontend/lib/api/v1/api.pbgrpc.dart | 9 +-
playground/frontend/lib/api/v1/api.pbjson.dart | 18 +-
.../beam/runners/dataflow/DataflowRunner.java | 11 +-
sdks/go/pkg/beam/core/metrics/metrics.go | 89 +-
sdks/go/pkg/beam/core/metrics/metrics_test.go | 37
.../runtime/harness/{sampler.go => cache_hooks.go} | 45 ++---
sdks/go/pkg/beam/core/runtime/harness/harness.go | 14 +-
.../core/runtime/harness/statecache/statecache.go | 20 ++-
.../runtime/harness/statecache/statecache_test.go | 13 +-
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 62 +--
.../beam/core/runtime/metricsx/metricsx_test.go| 9 +-
.../beam/runners/dataflow/dataflowlib/metrics.go | 2 +-
.../beam/runners/universal/runnerlib/execute.go| 6 +-
sdks/go/pkg/beam/util/harnessopts/cache.go | 42 +
.../harnessopts/cache_test.go} | 43 +++--
.../beam/sdk/schemas/ProjectionConsumer.java | 41 +
.../beam/sdk/schemas/ProjectionProducer.java | 46 +
.../beam/sdk/schemas/io/PushdownProjector.java | 48 --
.../beam/sdk/transforms/DoFnSchemaInformation.java | 20 +++
.../java/org/apache/beam/sdk/transforms/ParDo.java | 11 ++
.../sdk/transforms/reflect/DoFnSignatures.java | 20 +++
.../org/apache/beam/sdk/transforms/ParDoTest.java | 126 ++
.../provider/SchemaIOTableProviderWrapper.java | 19 ++-
.../extensions/sql/impl/rel/BeamCalcRelTest.java | 29 ++--
.../provider/TestSchemaIOTableProviderWrapper.java | 25 +--
.../sql/zetasql/BeamZetaSqlCalcRelTest.java| 29 ++--
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 11 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 2 +-
.../apache/beam/sdk/io/gcp/healthcare/FhirIO.java | 130 +-
.../apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java | 81 +
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 23 ++-
42 files changed, 1086 insertions(+), 418 deletions(-)
copy sdks/go/pkg/beam/core/runtime/harness/{sampler.go => cache_hooks.go} (58%)
create mode 100644 sdks/go/pkg/beam/util/harnessopts/cache.go
copy sdks/go/pkg/beam/{cor
