[beam] 01/01: Merge pull request #12376: [BEAM-10575] Eliminate legacy rawtypes from GCP IOs and some others; enable -Wrawtypes -Werror
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c2eb01df5098a8c4a5422dc7e5e5c890edc58bc7 Merge: 1019ef3 ab29122 Author: Kenn Knowles AuthorDate: Wed Jul 29 19:22:30 2020 -0700 Merge pull request #12376: [BEAM-10575] Eliminate legacy rawtypes from GCP IOs and some others; enable -Wrawtypes -Werror .../apache/beam/examples/snippets/Snippets.java| 2 +- .../beam/examples/complete/AutoCompleteTest.java | 2 +- .../complete/game/StatefulTeamScoreTest.java | 1 - .../org/apache/beam/runners/core/DoFnRunner.java | 8 +- .../apache/beam/runners/core/PaneInfoTracker.java | 3 + .../org/apache/beam/runners/core/StateMerging.java | 2 + .../apache/beam/runners/core/SystemReduceFn.java | 2 + .../apache/beam/runners/core/WatermarkHold.java| 6 + .../direct/ExecutorServiceParallelExecutor.java| 13 + .../runners/direct/ParDoMultiOverrideFactory.java | 4 + .../beam/runners/flink/metrics/FileReporter.java | 3 + .../beam/runners/dataflow/TestDataflowRunner.java | 2 + .../runners/dataflow/util/OutputReference.java | 2 + .../beam/runners/dataflow/worker/graph/Nodes.java | 4 +- .../JulHandlerPrintStreamAdapterFactory.java | 15 +- .../jobsubmission/PortablePipelineJarCreator.java | 4 +- .../beam/runners/spark/metrics/sink/CsvSink.java | 3 + .../runners/spark/metrics/sink/GraphiteSink.java | 3 + .../src/main/resources/beam/spotbugs-filter.xml| 426 ++--- .../java/org/apache/beam/sdk/coders/AvroCoder.java | 5 + .../apache/beam/sdk/coders/SerializableCoder.java | 3 + .../sdk/expansion/ExternalTransformRegistrar.java | 2 +- .../apache/beam/sdk/io/ClassLoaderFileSystem.java | 2 +- .../apache/beam/sdk/io/FileSystemRegistrar.java| 2 +- .../org/apache/beam/sdk/io/GenerateSequence.java | 2 +- .../beam/sdk/io/LocalFileSystemRegistrar.java | 2 +- .../beam/sdk/options/ProxyInvocationHandler.java | 10 + .../java/org/apache/beam/sdk/testing/PAssert.java | 2 + .../beam/sdk/testing/SerializableMatchers.java | 2 + .../apache/beam/sdk/testing/WindowSupplier.java| 3 + .../java/org/apache/beam/sdk/transforms/Mean.java | 4 + .../org/apache/beam/sdk/util/CombineFnUtil.java| 4 + .../sdk/util/ExposedByteArrayOutputStream.java | 2 + .../beam/sdk/io/LocalFileSystemRegistrarTest.java | 3 +- .../sdk/expansion/service/ExpansionService.java| 14 +- .../google-cloud-platform-core/build.gradle| 1 - .../gcp/storage/GcsFileSystemRegistrar.java| 2 +- .../beam/sdk/extensions/gcp/util/GcsUtil.java | 3 + .../gcp/storage/GcsFileSystemRegistrarTest.java| 3 +- .../expansion/ExternalSqlTransformRegistrar.java | 2 +- .../beam/sdk/extensions/sql/jdbc/BeamSqlLine.java | 5 +- .../provider/bigquery/BigQueryIOPushDownIT.java| 3 +- .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 31 +- .../extensions/sql/impl/CalciteQueryPlanner.java | 17 +- .../beam/sdk/extensions/sql/impl/QueryPlanner.java | 6 + .../extensions/sql/impl/planner/BeamRuleSets.java | 19 +- .../sql/impl/rule/IOPushDownRuleTest.java | 3 +- .../sql/impl/rule/JoinReorderingTest.java | 4 +- ...tTableProviderWithFilterAndProjectPushDown.java | 3 +- .../test/TestTableProviderWithFilterPushDown.java | 3 +- .../test/TestTableProviderWithProjectPushDown.java | 3 +- .../sql/zetasql/ZetaSQLQueryPlanner.java | 31 +- .../sql/zetasql/ZetaSQLPushDownTest.java | 3 +- .../extensions/sql/zetasql/ZetaSqlTestBase.java| 3 +- .../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java | 2 +- sdks/java/io/azure/build.gradle| 3 +- .../AzureBlobStoreFileSystemRegistrar.java | 2 +- .../sdk/io/azure/blobstore/AzfsResourceIdTest.java | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 1 - .../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 6 +- .../gcp/bigquery/BigQueryStorageStreamSource.java | 2 +- .../beam/sdk/io/gcp/bigquery/BigQueryUtils.java| 11 +- .../sdk/io/gcp/bigquery/TableRowInfoCoder.java | 4 +- .../beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java | 5 +- .../beam/sdk/io/gcp/bigquery/WriteTables.java | 4 +- .../io/gcp/healthcare/HealthcareIOErrorCoder.java | 2 +- .../beam/sdk/io/gcp/pubsub/ExternalRead.java | 2 +- .../beam/sdk/io/gcp/pubsub/ExternalWrite.java | 2 +- .../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 4 +- .../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 6 +- .../sdk/io/gcp/testing/FakeBigQueryServices.java | 2 +- .../sdk/io/gcp/bigquery/BigQueryIOReadTest.java| 6 +- .../gcp/bigquery/BigQueryIOStorageQueryTest.java | 2 +- .../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 2 +- .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 4
[beam] branch master updated (1019ef3 -> c2eb01d)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1019ef3 Prepare a website update for 2.23.0 (#12212) add a443fdd Refactor Beam SQL QueryPlanner instantiation to be more type safe and robust add 8084d1b Move individual spotbug suppressions out of spotbugs-filter.xml and remove some global suppressions add 12631cb [BEAM-10575] Fix rawtypes in FileSystemRegistrar and implementations add 45d1a5c [BEAM-10556] Eliminate remaining trivial rawtypes in sdks/java/io/hadoop-file-system add eb20f3a [BEAM-10577] Eliminate remaining trivial rawtypes in sdks/java/io/azure add ab29122 [BEAM-10575] Eliminate rawtypes from GCP IOs and enable -Wrawtypes new c2eb01d Merge pull request #12376: [BEAM-10575] Eliminate legacy rawtypes from GCP IOs and some others; enable -Wrawtypes -Werror The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/examples/snippets/Snippets.java| 2 +- .../beam/examples/complete/AutoCompleteTest.java | 2 +- .../complete/game/StatefulTeamScoreTest.java | 1 - .../org/apache/beam/runners/core/DoFnRunner.java | 8 +- .../apache/beam/runners/core/PaneInfoTracker.java | 3 + .../org/apache/beam/runners/core/StateMerging.java | 2 + .../apache/beam/runners/core/SystemReduceFn.java | 2 + .../apache/beam/runners/core/WatermarkHold.java| 6 + .../direct/ExecutorServiceParallelExecutor.java| 13 + .../runners/direct/ParDoMultiOverrideFactory.java | 4 + .../beam/runners/flink/metrics/FileReporter.java | 3 + .../beam/runners/dataflow/TestDataflowRunner.java | 2 + .../runners/dataflow/util/OutputReference.java | 2 + .../beam/runners/dataflow/worker/graph/Nodes.java | 4 +- .../JulHandlerPrintStreamAdapterFactory.java | 15 +- .../jobsubmission/PortablePipelineJarCreator.java | 4 +- .../beam/runners/spark/metrics/sink/CsvSink.java | 3 + .../runners/spark/metrics/sink/GraphiteSink.java | 3 + .../src/main/resources/beam/spotbugs-filter.xml| 426 ++--- .../java/org/apache/beam/sdk/coders/AvroCoder.java | 5 + .../apache/beam/sdk/coders/SerializableCoder.java | 3 + .../sdk/expansion/ExternalTransformRegistrar.java | 2 +- .../apache/beam/sdk/io/ClassLoaderFileSystem.java | 2 +- .../apache/beam/sdk/io/FileSystemRegistrar.java| 2 +- .../org/apache/beam/sdk/io/GenerateSequence.java | 2 +- .../beam/sdk/io/LocalFileSystemRegistrar.java | 2 +- .../beam/sdk/options/ProxyInvocationHandler.java | 10 + .../java/org/apache/beam/sdk/testing/PAssert.java | 2 + .../beam/sdk/testing/SerializableMatchers.java | 2 + .../apache/beam/sdk/testing/WindowSupplier.java| 3 + .../java/org/apache/beam/sdk/transforms/Mean.java | 4 + .../org/apache/beam/sdk/util/CombineFnUtil.java| 4 + .../sdk/util/ExposedByteArrayOutputStream.java | 2 + .../beam/sdk/io/LocalFileSystemRegistrarTest.java | 3 +- .../sdk/expansion/service/ExpansionService.java| 14 +- .../google-cloud-platform-core/build.gradle| 1 - .../gcp/storage/GcsFileSystemRegistrar.java| 2 +- .../beam/sdk/extensions/gcp/util/GcsUtil.java | 3 + .../gcp/storage/GcsFileSystemRegistrarTest.java| 3 +- .../expansion/ExternalSqlTransformRegistrar.java | 2 +- .../beam/sdk/extensions/sql/jdbc/BeamSqlLine.java | 5 +- .../provider/bigquery/BigQueryIOPushDownIT.java| 3 +- .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 31 +- .../extensions/sql/impl/CalciteQueryPlanner.java | 17 +- .../beam/sdk/extensions/sql/impl/QueryPlanner.java | 6 + .../extensions/sql/impl/planner/BeamRuleSets.java | 19 +- .../sql/impl/rule/IOPushDownRuleTest.java | 3 +- .../sql/impl/rule/JoinReorderingTest.java | 4 +- ...tTableProviderWithFilterAndProjectPushDown.java | 3 +- .../test/TestTableProviderWithFilterPushDown.java | 3 +- .../test/TestTableProviderWithProjectPushDown.java | 3 +- .../sql/zetasql/ZetaSQLQueryPlanner.java | 31 +- .../sql/zetasql/ZetaSQLPushDownTest.java | 3 +- .../extensions/sql/zetasql/ZetaSqlTestBase.java| 3 +- .../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java | 2 +- sdks/java/io/azure/build.gradle| 3 +- .../AzureBlobStoreFileSystemRegistrar.java | 2 +- .../sdk/io/azure/blobstore/AzfsResourceIdTest.java | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 1 - .../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 6 +- .../gcp/bigquery/BigQueryStorageStreamSource.java | 2 +- .../beam/sdk/io/gcp/bigquery/BigQueryUtils.java| 11 +-
svn commit: r40762 - in /dev/beam: 2.23.0/ latest
Author: pabloem Date: Wed Jul 29 22:50:59 2020 New Revision: 40762 Log: Cleaning 2.23.0 from dev artifacts. Removed: dev/beam/2.23.0/ dev/beam/latest
svn commit: r40761 - in /release/beam: ./ 2.23.0/ 2.23.0/python/
Author: pabloem Date: Wed Jul 29 22:48:50 2020 New Revision: 40761 Log: Adding 2.23.0 to release artifacts. Added: release/beam/2.23.0/ release/beam/2.23.0/apache-beam-2.23.0-source-release.zip (with props) release/beam/2.23.0/apache-beam-2.23.0-source-release.zip.asc release/beam/2.23.0/apache-beam-2.23.0-source-release.zip.sha512 release/beam/2.23.0/python/ release/beam/2.23.0/python/apache-beam-2.23.0.zip (with props) release/beam/2.23.0/python/apache-beam-2.23.0.zip.asc release/beam/2.23.0/python/apache-beam-2.23.0.zip.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-macosx_10_9_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-macosx_10_9_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-macosx_10_9_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux1_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux1_i686.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux1_i686.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux1_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux1_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux1_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux2010_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux2010_i686.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux2010_i686.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux2010_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux2010_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27m-manylinux2010_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux1_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux1_i686.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux1_i686.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux1_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux1_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux1_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux2010_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux2010_i686.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux2010_i686.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux2010_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux2010_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp27-cp27mu-manylinux2010_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-macosx_10_9_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-macosx_10_9_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-macosx_10_9_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux1_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux1_i686.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux1_i686.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux1_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux1_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux1_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux2010_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux2010_i686.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux2010_i686.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux2010_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux2010_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp35-cp35m-manylinux2010_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp36-cp36m-macosx_10_9_x86_64.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp36-cp36m-macosx_10_9_x86_64.whl.asc release/beam/2.23.0/python/apache_beam-2.23.0-cp36-cp36m-macosx_10_9_x86_64.whl.sha512 release/beam/2.23.0/python/apache_beam-2.23.0-cp36-cp36m-manylinux1_i686.whl (with props) release/beam/2.23.0/python/apache_beam-2.23.0-cp36-cp36m-manylinux1_i686.whl.asc
[beam] annotated tag v2.23.0 updated (696fc99 -> 3c6c354)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to annotated tag v2.23.0 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag v2.23.0 was modified! *** from 696fc99 (commit) to 3c6c354 (tag) tagging c33a8100cadbdb4de393c6bcb4146b98a5e23054 (tag) length 200 bytes by Valentyn Tymofieiev on Wed Jul 29 15:18:21 2020 -0700 - Log - Tag for 2.23.0 release. -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEkHE5Z3XlzmG0rN/DHfUGAyJdKaQFAl8h9bYACgkQHfUGAyJd KaTCKg/9Hk7sWflNTI1e/lrC7cfO2sFDFr/l0p8pKf4XGUQV6i34X9OFYHXeRZun Av0ReyaWj0x63g2ctSPzOXSy675zKyqgVdiS9EFT/oZNOcV5MhjgDvtJOeRfw77B HIUtxp+Z2DyH7kw/rf5xWMHTjsEVWqyS9hW/VHHQBxRTgMO/xUQF0/m7GmkfSilt a15/uQ80CuyswyjEO9U+PE4fchSCNBIL5s2lY/Vq6w58iAJ2UoEjWBLteqyGgVuN wxWOC1AJ8g7eyuKk5jzTwX86WrE5dPqwVsgKFYOI681mt7Ih+axr6O0GGuUoL0WI NMSNoBDtVu8JIZ1QiC9qL3Lippo3/C1tu7os7Qnmg04rMv671J22yPeDfVYTbKRz 30wYelL+1tp/6+oZW41GJNQtq95Dig6FpTlB6uFzY4/W1gLSyJ7NiWr57CvGu+zz 2bal5mbW6+GGjIhHUYVONeCQS880h1D4XZmu6htBMPbeBF/unF6vKGjya7ekuG66 pYW1/DPzp2NyK0qyuFeBINVRTbV11x+PwR7nTCBJE8bxt1Sv3W0Wk2gOip1QqZ5o w5a0z08Qd/Sfwm03oEC9cBAuSiql6W61aECXpCxUgeHT8PCuZn/KHSjyuwHFC0MN 9uzr8vIaLq/K2G96zCiif07BB7xKtzBYyE3bIQH8Cl3YuYRiY6M= =OJNz -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] branch master updated (2496fc4 -> 1019ef3)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2496fc4 Merge pull request #11974 [BEAM-9547] Add more methods to deferred dataframes. add 1019ef3 Prepare a website update for 2.23.0 (#12212) No new revisions were added by this update. Summary of changes: website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.23.0.md | 2 +- website/www/site/content/en/get-started/downloads.md | 14 +++--- website/www/site/static/.htaccess| 2 +- 4 files changed, 14 insertions(+), 6 deletions(-)
[beam] branch master updated (2496fc4 -> 1019ef3)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2496fc4 Merge pull request #11974 [BEAM-9547] Add more methods to deferred dataframes. add 1019ef3 Prepare a website update for 2.23.0 (#12212) No new revisions were added by this update. Summary of changes: website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.23.0.md | 2 +- website/www/site/content/en/get-started/downloads.md | 14 +++--- website/www/site/static/.htaccess| 2 +- 4 files changed, 14 insertions(+), 6 deletions(-)
[beam] branch master updated (da49b24 -> 2496fc4)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from da49b24 Support creation of empty PCollection in beam CreateList add 36dc765 Extend elementwise to generic proxy fn. add ce5d75b Add a more methods to dataframes. add a37170e Add support for dataframe scalar values. add 8b08bf2 More dataframes methods and test filters. add b365608 Reviewer comments. add de7fa96 Skip dict-order-sensitive tests on 3.5. new 2496fc4 Merge pull request #11974 [BEAM-9547] Add more methods to deferred dataframes. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/dataframe/doctests.py | 62 -- sdks/python/apache_beam/dataframe/frame_base.py| 109 +- sdks/python/apache_beam/dataframe/frames.py| 240 - .../apache_beam/dataframe/pandas_doctests_test.py | 76 ++- sdks/python/apache_beam/dataframe/transforms.py| 47 ++-- .../apache_beam/dataframe/transforms_test.py | 40 +++- 6 files changed, 454 insertions(+), 120 deletions(-)
[beam] 01/01: Merge pull request #11974 [BEAM-9547] Add more methods to deferred dataframes.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2496fc40bc746dfc8dab41b7030592630f9ce253 Merge: da49b24 de7fa96 Author: Robert Bradshaw AuthorDate: Wed Jul 29 14:25:41 2020 -0700 Merge pull request #11974 [BEAM-9547] Add more methods to deferred dataframes. sdks/python/apache_beam/dataframe/doctests.py | 62 -- sdks/python/apache_beam/dataframe/frame_base.py| 109 +- sdks/python/apache_beam/dataframe/frames.py| 240 - .../apache_beam/dataframe/pandas_doctests_test.py | 76 ++- sdks/python/apache_beam/dataframe/transforms.py| 47 ++-- .../apache_beam/dataframe/transforms_test.py | 40 +++- 6 files changed, 454 insertions(+), 120 deletions(-)
[beam] branch master updated (890bedc -> da49b24)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 890bedc [BEAM-10595] Remove rules that not work well with Java UDF (#12400) add c92e57b Support creation of empty PCollection in beam CreateList add 5f9848f Update docstrings add 1e651d8 Refactor CreateList add 57133f8 Merge branch 'master' into beam-go-create-empty-pcollection add 631bbb8 Refactor CreateList add da49b24 Support creation of empty PCollection in beam CreateList No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/create.go | 51 + sdks/go/pkg/beam/create_test.go | 26 + 2 files changed, 62 insertions(+), 15 deletions(-)
[beam] branch master updated (88acc52 -> 890bedc)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 88acc52 [BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924) add 890bedc [BEAM-10595] Remove rules that not work well with Java UDF (#12400) No new revisions were added by this update. Summary of changes: .../beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java | 10 ++ 1 file changed, 10 insertions(+)
[beam] branch master updated: [BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 88acc52 [BEAM-8543] Dataflow streaming timers are not strictly time
ordered when set earlier mid-bundle (#11924)
88acc52 is described below
commit 88acc5267f759d81e9836a9db17b9e0ee521c785
Author: Rehman Murad Ali
AuthorDate: Thu Jul 30 01:06:52 2020 +0500
[BEAM-8543] Dataflow streaming timers are not strictly time ordered when
set earlier mid-bundle (#11924)
---
CHANGES.md | 1 +
.../worker/StreamingModeExecutionContext.java | 75 +-
.../dataflow/worker/WindmillTimerInternals.java| 25
.../org/apache/beam/sdk/transforms/ParDoTest.java | 69
4 files changed, 141 insertions(+), 29 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 1cfc52c..c2665e7 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -146,6 +146,7 @@
* Upgrade Sphinx to 3.0.3 for building PyDoc.
* Added a PTransform for image annotation using Google Cloud AI image
processing service
([BEAM-9646](https://issues.apache.org/jira/browse/BEAM-9646))
+* Dataflow streaming timers are not strictly time ordered when set earlier
mid-bundle ([BEAM-8543](https://issues.apache.org/jira/browse/BEAM-8543)).
## Breaking Changes
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
index 7ed390b..334f145 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
@@ -24,10 +24,12 @@ import com.google.api.services.dataflow.model.CounterUpdate;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.io.Closeable;
import java.io.IOException;
+import java.util.Comparator;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import java.util.PriorityQueue;
import java.util.Set;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.atomic.AtomicLong;
@@ -519,7 +521,7 @@ public class StreamingModeExecutionContext extends
DataflowExecutionContext cachedFiredUserTimers = null;
+private PriorityQueue toBeFiredTimersOrdered = null;
+
+// to track if timer is reset earlier mid-bundle.
+// Map of timer's id to timer's firing time to check
+// the actual firing time of a timer.
+private Map firedTimer = new HashMap<>();
public TimerData getNextFiredUserTimer(Coder
windowCoder) {
- if (cachedFiredUserTimers == null) {
-cachedFiredUserTimers =
-
FluentIterable.from(StreamingModeExecutionContext.this.getFiredTimers())
-.filter(
-timer ->
-WindmillTimerInternals.isUserTimer(timer)
-&& timer.getStateFamily().equals(stateFamily))
-.transform(
-timer ->
-WindmillTimerInternals.windmillTimerToTimerData(
-WindmillNamespacePrefix.USER_NAMESPACE_PREFIX,
timer, windowCoder))
-.iterator();
+ if (toBeFiredTimersOrdered == null) {
+
+toBeFiredTimersOrdered = new
PriorityQueue<>(Comparator.comparing(TimerData::getTimestamp));
+
FluentIterable.from(StreamingModeExecutionContext.this.getFiredTimers())
+.filter(
+timer ->
+WindmillTimerInternals.isUserTimer(timer)
+&& timer.getStateFamily().equals(stateFamily))
+.transform(
+timer ->
+WindmillTimerInternals.windmillTimerToTimerData(
+WindmillNamespacePrefix.USER_NAMESPACE_PREFIX, timer,
windowCoder))
+.iterator()
+.forEachRemaining(
+timerData -> {
+ firedTimer.put(
+ timerData.getTimerId() + '+' +
timerData.getTimerFamilyId(),
+ timerData.getTimestamp());
+ toBeFiredTimersOrdered.add(timerData);
+});
}
- if (!cachedFiredUserTimers.hasNext()) {
+ Instant currentInputWatermark =
userTimerInternals.currentInputWatermarkTime();
+
+ if (userTimerInternals.hasTimerBefore(currentInputWatermark)) {
+List currentTimers = userTimerInternals.getCurrentTimers();
+
+for (TimerData timerData : currentTimers) {
+ firedTimer.put(
+ timerData.getTimerId() + '+' +
[beam] branch master updated (f95ec86 -> b976e7b)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f95ec86 [BEAM-7390] Add latest code snippets (#12394) add b976e7b Add a blogpost for Apache Beam 2.23.0. (#12213) No new revisions were added by this update. Summary of changes: CHANGES.md | 4 +- website/www/site/content/en/blog/beam-2.23.0.md | 68 + 2 files changed, 69 insertions(+), 3 deletions(-) create mode 100644 website/www/site/content/en/blog/beam-2.23.0.md
[beam] branch master updated (780cc4f -> f95ec86)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 780cc4f [BEAM-8494] Add python3.8 to the list of container images to publish. (#12401) add f95ec86 [BEAM-7390] Add latest code snippets (#12394) No new revisions were added by this update. Summary of changes: .../transforms/python/aggregation/latest.md| 57 -- .../en/documentation/transforms/python/overview.md | 2 +- .../partials/section-menu/en/documentation.html| 1 + 3 files changed, 56 insertions(+), 4 deletions(-)
[beam-wheels] branch master updated: Add deprecation warning to README.md
This is an automated email from the ASF dual-hosted git repository. altay pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam-wheels.git The following commit(s) were added to refs/heads/master by this push: new 469543d Add deprecation warning to README.md new a556195 Merge pull request #18 from TobKed/deprecate 469543d is described below commit 469543dd8f87da42010b131543543c1739b7c1e6 Author: Tobiasz Kędzierski AuthorDate: Wed Jul 1 16:06:55 2020 +0200 Add deprecation warning to README.md It can be deprecated after using in release process python source distribution and wheels built within apache/beam repository itself. https://github.com/apache/beam/pull/12150 --- README.md | 4 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 87c1a63..e7ffbf4 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,10 @@ under the License. --> +:no_entry: [DEPRECATED] + +This repository is deprecated since python wheels for [Apache Beam](https://github.com/apache/beam) are build with use of GitHub Actions in Apache Beam repository itself. + # Apache Beam Wheels ## Overview
[GitHub] [beam-wheels] aaltay commented on pull request #18: Deprecate repository, not used in release process anymore
aaltay commented on pull request #18: URL: https://github.com/apache/beam-wheels/pull/18#issuecomment-665798992 Nice. LGTM. @tvalentyn - Could you verify that 2.23 release did not use this repo? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam-wheels] TobKed commented on pull request #18: Deprecate repository (depends on https://github.com/apache/beam/pull/12150)
TobKed commented on pull request #18: URL: https://github.com/apache/beam-wheels/pull/18#issuecomment-665693872 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
