See
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/4372/display/redirect?page=changes>
Changes:
[herohde] Initial sketches of a Go SDK
[herohde] Initial version of the direct style w/ direct runner. Incomplete.
[herohde] Add Data as UserFn context w/ immediate value.
[herohde] Added no-I/O wordcount for profiling.
[herohde] Fleshed out possible approach to generic transformations.
[herohde] Add “dag” example that use multiplexing and side input.
[herohde] Added a more complex DAG example.
[herohde] Add yatzy example with more complex construction-time setup
[herohde] Add proto for Fn API
[herohde] Add beam.Composite helper for the most common pattern to align with
java
[herohde] Move pipeline-construction time errors into an accumulator
[herohde] Add Dataflow job and Fn API clients. Incomplete.
[herohde] Add binary cross-compile and upload to Dataflow runner. Incomplete.
[herohde] Add tentative runner indirection (default: local).
[herohde] Made data flow runner detect user main for cross-compilation.
[herohde] Remove error accumulation in favor of panic.
[herohde] Improve Dataflow translation of coders, side input and composite
names.
[herohde] Fix name for AsView.
[herohde] Use 2 grpc endpoints in harness
[herohde] Add gRPC harness logging
[herohde] Flesh out harness and serialization further.
[herohde] Made the dataflow runner wait for job termination by default
[herohde] beam:
[herohde] beam:
[herohde] combinefn.go: fix compilation issues
[herohde] Improve dataflow serialization and execution. Incomplete.
[herohde] Sleep 30 sec in wordcap to allow logs to propagate to Cloud Logging.
[herohde] Move the 30s sleep for logging to the harness instead of in WordCap.
[herohde] Post-review updates.
[herohde] Doc updates.
[herohde] Flesh out coders. Incomplete.
[herohde] Added prototype implementation of more coders and the runner source.
[herohde] dofn: illustrates how dofns are written.
[herohde] beam: add viewfn and windowfn to side inputs match support Beam 1.0
[herohde] dofn: timers
[herohde] Complete revamp: coders, graph and execution use element-wise
[herohde] Fix coder encoding for Dataflow side input. Otherwise, the job is
[herohde] Added more godoc comments to graph types.
[herohde] Added more comments plus made local GBK use coder equality.
[herohde] Added Flatten support and “forest” example that uses it.
[herohde] Move bigqueryio to defunct
[herohde] Make forest example print less
[herohde] Add external struct registry and serialization.
[herohde] Updated comments in node.go.
[herohde] Replace real type with 'full type' since that's the current term.
[herohde] Refactor Fn API dependency.
[herohde] Added more comments to the runner/dataflow and runner/beamexec
packages
[herohde] Fix most go vet issues
[herohde] Make core operations panic to cut down on the error propagation
[herohde] Add more comments to the graph package.
[herohde] Add DoFn wrapper to handle either function or (ptr to) struct
[herohde] Fix remaining go vet warnings.
[herohde] Code review for beam/graph/coder package.
[herohde] Code review of the runtime/graphx package.
[herohde] Remove Data options in favor of using a Fn struct
[herohde] Code review of the beam/graph/userfn package.
[herohde] Code review for beam/graph package.
[herohde] godoc for runtime/graphx
[herohde] Add support for []T and Combine functions
[herohde] Add adapted documentation from the Java SDK to the beam package
[herohde] Update snapshot of Fn API.
[herohde] Add experiments flag to the Dataflow runner
[herohde] Remove context arg from beamexec.Init
[herohde] Migration to Runner API.
[herohde] Add support for creating DOT graphs.
[herohde] Make pretty printing of types and coders more concise
[herohde] Add flexible Signature to aid type checking
[herohde] Adding unit testability to harness translation.
[herohde] Fix crash due to initialization order
[herohde] Add CreateValues and Impulse
[herohde] Add Runner API support for WindowingStrategy.
[herohde] Run goimports on baseline.
[herohde] Fix encoding of global window strategy.
[herohde] Ensure the windowed value is atomically encoded.
[herohde] Limit gRPC messages to max size.
[herohde] Developer conveniences for running jobs.
[herohde] Fix sends to not close the network channel.
[herohde] Add re-iterable side input
[herohde] Add per-key Combine
[herohde] Add Min
[herohde] Reorganize non-user-facing code into core
[herohde] Make type register reject unnamed or predeclared types
[herohde] Add type specialization tool
[herohde] Don't run grpc plugin in generate phase.
[herohde] Fix import reference path for runner API proto.
[herohde] Revamp runner registration as _ imports
[herohde] Add stats.Max and Mean
[herohde] Add global pipeline options
[herohde] Unify global and per-key combiners
[herohde] Add beam convenience wrapper for imports and runner selection
[herohde] Add session recording and CPU profiling to harness.
[herohde] Add ptest and passert for testing pipelines
[herohde] Add GCS and glob support to textio
[herohde] Add BigQuery IO and examples
[herohde] Adds a session runner for testing.
[herohde] Add Partition and dynamic functions
[herohde] Adding example that returns 10 words that contain provided search
[herohde] Remove duplicate LOG line
[herohde] Enable Combine Fns in Dataflow runner by modifying translation.
[herohde] Fixing type bug by dropping T and using underlying type of value in
[herohde] Adding graph validation at build time.
[herohde] Import the Fn API changes.
[herohde] Simple changes to support new Fn API coder changes.
[herohde] Update translator to work with new Fn API changes.
[herohde] Use appropriate equality tests.
[herohde] Fix test to not use path of package.
[herohde] Renaming directory to match package name.
[herohde] Fixing random nits in comments.
[herohde] Modify build command to avoid bash.
[herohde] Fixing selected golint issues.
[herohde] Addressing import review comments.
[herohde] Add coder specialization for bytes/strings.
[herohde] Adding unit tests to stats.
[herohde] Fixing typo.
[herohde] Add beam.External
[herohde] Fix grpc.Dial calls to block properly.
[herohde] Creates a symtab verifier by running Sym2Addr and Addr2Sym in a
binary.
[herohde] Add spec field to help interpretation of payload.
[herohde] Use beam.T alias for typex.T etc with Go 1.9
[herohde] Move shared GCP options to a separate package
[herohde] Update portability protos
[herohde] Remove old source/sink from beam package
[herohde] Add context-aware logging for both pipeline-submission time and
runtime
[herohde] Fix coder inference for strings.
[herohde] Improve tornadoes example
[herohde] Fix beam.External to map arguments correctly.
[herohde] Added comments to yatzy and forest
[herohde] Add comments to tornadoes from the java counterpart
[herohde] Rename Pipeline Composite to Scope
[herohde] Add 3 progressive wordcount examples
[herohde] Clarify comments in wordcount pipelines
[herohde] Add apache 2.0 license to files
[herohde] Updates to examples.
[herohde] Adding more godoc for the main beam package.
[herohde] Update to new proto structure
[herohde] Split Combine and fields in to global and per-key variants
[herohde] Refactor Flatten of a single []T into Explode
[herohde] Rename local runner to direct runner
[herohde] Fix argument index error in ParDo execution
[herohde] Add Apache copyright header to files that need it.
[herohde] Made debug.Head not just work per bundle
[herohde] Impose a total ordering on Fn parameters.
[herohde] Rename Dedup to Distinct for consistency with other SDKs
[herohde] Add coder to model coder translation
[herohde] Simplify harness coder translation
[herohde] Split Pipeline into Pipeline and Scope
[herohde] Relocate Go SDK code
[herohde] Fix Go SDK maven build
[herohde] Move Go SKD to latest version of bigquery
[herohde] Add Go SDK container image
[herohde] Add Go SDK README
[herohde] Update version for Go Dataflow pipelines
[herohde] Make Scope a value type
[herohde] Add Go graph/pipeline translation
[herohde] Stage Go model pipeline for Dataflow
[herohde] Use pipeline unmarhaller in runtime harness
[herohde] CR: [BEAM-3287] Use model pipelines in Go SDK
[herohde] CR: [BEAM-3287] Use model pipelines in Go SDK
[herohde] Fix name of syscallx ErrUnsupported
[herohde] Allow any named type to be registered and serialized as external
[herohde] Add more package comments for core packages
[herohde] Make Go SDK External a graph primitive
[herohde] Cache Go runtime symbol lookups
[wcn] Fix storagePath variable.
[wcn] BEAM-3368 fix translation for external
[robertwb] [BEAM-3356] Add Go SDK int and varint custom coders (#4276)
[lcwik] BEAM-3361 Increase Go gRPC message size
[herohde] Go SDK runtime revamp
[lcwik] Add a few function call overhead benchmarks
[lcwik] Add type-specialized emitters
[lcwik] BEAM-3324 improve symtab memory usage
[lcwik] BEAM-3324 improve symtab memory usage
[lcwik] BEAM-3324 improve symtab memory usage
[lcwik] Store objects in pool so they can be reused.
[lcwik] Add builtin varint coder
[herohde] Type-specialize custom decoders and encoders in Go SDK runtime
[herohde] Type-specialize iterators and side input in the Go SDK
[herohde] Add warnings if Go runtime registrations are overwritten
[herohde] Add reusable element coders in Go SDK runtime
[wcn] Updated translater to preserve payload and its URN.
[herohde] Initial version of type-specialized general reflective calls
[herohde] Add general-purpose untyped callers in Go SDK runtime
[herohde] Use fast caller for filter transform predicate
[herohde] CR: Clarified comment on caller template
[herohde] Fix value encoding for Create
[lcwik] BEAM-3473: Fix GroupByKey iterators to be initialized.
[lcwik] BEAM-3474 Include stacks in panic messages.
[lcwik] BEAM-3299: Add source reporting support.
[lcwik] Remove GetId() call from under lock.
[lcwik] Add additional comments about concurrency invariants.
[lcwik] Add initialization of active plans map.
[lcwik] Renamed Go runtime Caller to Func and added name
[lcwik] Use reflectx.Func as the fundamental function representation
[lcwik] CR: fix DynFn comments
[lcwik] CR: fix comments
[herohde] Avoid reflect.Value conversions in Go runtime
[wcn] Allow grpcx.Dial to support overrides.
[rober] Use a typeswitch instead of reflect.Convert when encoding strings or
[github] Update coder.go
[robert] Replace reflective convert to direct convert.
[rober] Fix beam.Combine to combine globally
[herohde] Add optional function registration to Go SDK runtime
[rober] fixup! Remove reflection from varint codecs
[herohde] Changed core GBK to CoGBK
[herohde] Add CoGBK support to direct runner and Join example
[herohde] [BEAM-3316] Translate bundle descriptors directly to execution plans
in
[herohde] Translate CoGBK into GBK for Dataflow and model pipeline runners
[herohde] CR: [BEAM-3302] Support CoGBK in the Go SDK
[herohde] [BEAM-3579] Fix textio.Write
[herohde] CR: fix Go SDK textio.Write
[wcn] Fixing filename.
[ehudm] Integration test for Python HDFS implementation.
[wcn] Improve rendering of DOT diagrams.
[herohde] Update Go SDK coder constants
[tgroh] Make Impulse#create() visible
[tgroh] Update QueryablePipeline Factory Method Name
[tgroh] Add JavaReadViaImpulse to core-construction
[amyrvold] Beam runner inventory, run as a cron job, not on each CL
------------------------------------------
[...truncated 1.01 MB...]
copying
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py ->
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline_test.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream_test.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils_test.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/util.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/util_test.py ->
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/data/standard_coders.yaml ->
apache-beam-2.5.0.dev0/apache_beam/testing/data
copying apache_beam/testing/data/trigger_transcripts.yaml ->
apache-beam-2.5.0.dev0/apache_beam/testing/data
copying apache_beam/transforms/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/core.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/create_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.pxd ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/timeutil.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/window.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/window_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/write_ptransform_test.py ->
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/typehints/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/native_type_compatibility.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/native_type_compatibility_test.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/opcodes.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test.py ->
apache-beam-2.5.0.dev0/apache_beam/typehints
copying apache_beam/utils/__init__.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations_test.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.pxd ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/counters_test.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/plugin.py -> apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/processes.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/processes_test.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/profiler.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/proto_utils.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/retry.py -> apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/retry_test.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp_test.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/urns.py -> apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.pxd ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value_test.py ->
apache-beam-2.5.0.dev0/apache_beam/utils
Writing apache-beam-2.5.0.dev0/setup.cfg
creating dist
Creating tar archive
removing 'apache-beam-2.5.0.dev0' (and everything under it)
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
find dist/apache-beam-*.tar.gz
# Run integration tests on the Google Cloud Dataflow service
# and validate that jobs finish successfully.
echo ">>> RUNNING TEST DATAFLOW RUNNER it tests"
>>> RUNNING TEST DATAFLOW RUNNER it tests
python setup.py nosetests \
--attr IT \
--nocapture \
--processes=4 \
--process-timeout=900 \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-it \
--temp_location=$GCS_LOCATION/temp-it \
--output=$GCS_LOCATION/py-it-cloud/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
--sleep_secs=20"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
UserWarning: Normalizing '2.5.0.dev' to '2.5.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT)
... ok
test_bigquery_tornadoes_it
(apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT)
... ERROR
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
======================================================================
ERROR: test_bigquery_tornadoes_it
(apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py";,>
line 812, in run
test(orig)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py";,>
line 45, in __call__
return self.run(*arg, **kwarg)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py";,>
line 133, in run
self.runTest(result)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py";,>
line 151, in runTest
test(result)
File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py";,>
line 69, in test_bigquery_tornadoes_it
test_pipeline.get_full_options_as_args(**extra_opts))
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py";,>
line 90, in run
write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 389, in __exit__
self.run().wait_until_finish()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 987, in wait_until_finish
time.sleep(5.0)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py";,>
line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_bigquery_tornadoes_it
(apache_beam.examples.cookbook.bigquery_tornadoes_it_test.BigqueryTornadoesIT)'
======================================================================
ERROR: test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py";,>
line 812, in run
test(orig)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py";,>
line 45, in __call__
return self.run(*arg, **kwarg)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py";,>
line 133, in run
self.runTest(result)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py";,>
line 151, in runTest
test(result)
File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 66, in test_wordcount_it
wordcount.run(test_pipeline.get_full_options_as_args(**extra_opts))
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/examples/wordcount.py";,>
line 116, in run
result.wait_until_finish()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 987, in wait_until_finish
time.sleep(5.0)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py";,>
line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_wordcount_it
(apache_beam.examples.wordcount_it_test.WordCountIT)'
----------------------------------------------------------------------
Ran 3 tests in 901.267s
FAILED (errors=2)
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user amyrv...@google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user eh...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user xuming...@users.noreply.github.com
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user samuel.waggo...@healthsparq.com
Not sending mail to unregistered user ccla...@bluewin.ch
Not sending mail to unregistered user rob...@frantil.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
