[GitHub] [beam] mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601567693 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] alexvanboxel commented on issue #11008: Update comment to tell user this is not secure
alexvanboxel commented on issue #11008: Update comment to tell user this is not secure URL: https://github.com/apache/beam/pull/11008#issuecomment-601569348 > What should be the next action here? Should we remove it? Or add this comment? Personally I would close this PR as the original author didn't take actions on the commit. It doesn't add a lot of value and just adds to our cognitive load being here. It did bring too our attention that it's maybe interesting to think about a unified way of providing TLS for all IO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601569728 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601569767 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] piotr-szuberski commented on a change in pull request #11122: [BEAM-9346] Improve the efficiency of TFRecordIO
piotr-szuberski commented on a change in pull request #11122: [BEAM-9346]
Improve the efficiency of TFRecordIO
URL: https://github.com/apache/beam/pull/11122#discussion_r395475008
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
##
@@ -410,13 +412,44 @@ private GatherResults(Coder resultCoder) {
} else {
// Pass results via a side input rather than reshuffle, because we
need to get an empty
// iterable to finalize if there are no results.
-return input
-.getPipeline()
-.apply(Reify.viewInGlobalWindow(input.apply(View.asList()),
ListCoder.of(resultCoder)));
+return input.apply("ToList", Combine.globally(new
ToListCombineFn<>()));
}
}
}
+ public static class ToListCombineFn
Review comment:
You're right, I just copy pasted the code from Jira. It was my first PR in
Beam and I wanted to get the workflow without digging the code proposed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-591959468 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-586256259 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-586964776 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-580198417 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-591954597 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-580728679 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-586261220 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-586959242 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-580731061 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592441414 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592452480 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600052646 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-599915764 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600037517 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-591962646 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600076078 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600070865 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592476627 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600031195 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592457334 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600106774 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-599928528 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-599965509 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600131618 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600109497 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600081286 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600559891 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600135623 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600656596 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600029608 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600527389 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600564536 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600613410 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600602229 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600613470 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600602288 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600564484 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600524672 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600573249 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600586885 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601048332 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600658641 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-600133615 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601110114 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601130890 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-60079 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601107028 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601093845 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601057353 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601158639 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601094876 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601569728 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601569767 Run Load Tests Java Combine Portable Flink Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601185879 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia removed a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601567693 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601597356 @mxm I ran into a problem with TaskManagers timing out in a streaming case, the latest failed test is an example of this. Do you have any insight into why this might have happened? Pinging @lukecwik as well. Also there's an issue of intermittent `OutOfDirectMemoryError` getting thrown by gRPC on batch as well (eg. [here](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch_PR/30/console) ) Thanks for your help! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (e9430a0 -> e91659f)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e9430a0 Merge pull request #11167 from lukecwik/proto5 add e03a992 [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark add e91659f Merge pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark No new revisions were added by this update. Summary of changes: CHANGES.md | 1 + 1 file changed, 1 insertion(+)
[GitHub] [beam] iemejia merged pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
iemejia merged pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
iemejia commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#issuecomment-601605616 I am not sure if this means now that we can resolve the issue. Please do it yourself if it is the case @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mxm commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601606381 >@mxm I ran into a problem with TaskManagers timing out in a streaming case, the latest failed test is an example of this. Do you have any insight into why this might have happened? @mwalenia Not from the top of my head. I would need a stack trace or log to see what happened. >Also there's an issue of intermittent OutOfDirectMemoryError getting thrown by gRPC on batch as well (eg. here) > Also there's an issue of intermittent `OutOfDirectMemoryError` getting thrown by gRPC on batch as well (eg. [here](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch_PR/30/console) ) Flink 1.10 configures has changed the memory configuration. It looks like GRPC's direct memory allocation is competing with Flink's managed memory. Flink allocates a fix amount of direct memory upfront leaving only a small amount left. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration, especially `taskmanager.memory.framework.off-heap.size` and `taskmanager.memory.jvm-overhead.fraction`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm edited a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mxm edited a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601606381 >@mxm I ran into a problem with TaskManagers timing out in a streaming case, the latest failed test is an example of this. Do you have any insight into why this might have happened? @mwalenia Not from the top of my head. I would need a stack trace or log to see what happened. > Also there's an issue of intermittent `OutOfDirectMemoryError` getting thrown by gRPC on batch as well (eg. [here](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch_PR/30/console) ) Flink 1.10 has changed the memory configuration. It looks like GRPC's direct memory allocation is competing with Flink's managed memory. Flink allocates a fix amount of direct memory upfront leaving only a small amount left. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration, especially `taskmanager.memory.framework.off-heap.size` and `taskmanager.memory.jvm-overhead.fraction`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm edited a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mxm edited a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601606381 >@mxm I ran into a problem with TaskManagers timing out in a streaming case, the latest failed test is an example of this. Do you have any insight into why this might have happened? @mwalenia Not from the top of my head. I would need a stack trace or log to see what happened. > Also there's an issue of intermittent `OutOfDirectMemoryError` getting thrown by gRPC on batch as well (eg. [here](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch_PR/30/console) ) Flink 1.10 configures has changed the memory configuration. It looks like GRPC's direct memory allocation is competing with Flink's managed memory. Flink allocates a fix amount of direct memory upfront leaving only a small amount left. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration, especially `taskmanager.memory.framework.off-heap.size` and `taskmanager.memory.jvm-overhead.fraction`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm edited a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mxm edited a comment on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601606381 >@mxm I ran into a problem with TaskManagers timing out in a streaming case, the latest failed test is an example of this. Do you have any insight into why this might have happened? @mwalenia Not from the top of my head. I would need a stack trace or log to see what happened. > Also there's an issue of intermittent `OutOfDirectMemoryError` getting thrown by gRPC on batch as well (eg. [here](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch_PR/30/console) ) Flink 1.10 has changed the memory configuration. It looks like GRPC's direct memory allocation is competing with Flink's managed memory. Flink allocates a fix amount of direct memory upfront leaving only a small amount left. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration, especially `taskmanager.memory.framework.off-heap.size` and `taskmanager.memory.task.off-heap.size`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
mxm commented on a change in pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#discussion_r395523161 ## File path: CHANGES.md ## @@ -58,6 +58,7 @@ ## Breaking Changes * HBaseIO.ReadAll now requires a PCollection of HBaseIO.Read objects instead of HBaseQuery objects ([BEAM-9279](https://issues.apache.org/jira/browse/BEAM-9279)). +* ProcessContext.updateWatermark has been removed in favor of using a WatermarkEstimator ([BEAM-9430](https://issues.apache.org/jira/browse/BEAM-9430)). Review comment: Do we plan to add some end-user documentation for WatermarkEstimators or SDF in general? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
mxm commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#issuecomment-601608697 Thanks @lukecwik. @iemejia Why the quick merge if a review was still pending on me? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] piotr-szuberski opened a new pull request #11180: [BEAM-9563] Change ToListCombineFn access level to private
piotr-szuberski opened a new pull request #11180: [BEAM-9563] Change ToListCombineFn access level to private URL: https://github.com/apache/beam/pull/11180 The access level of ToListCombineFn can be private. But more important things were in the discussion below #11122 there were voices to revert this change or to modify it. Let's discuss more what should be done. @lukecwik @chamikaramj @aaltay @mwalenia Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostC
[GitHub] [beam] mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601616918 cc @kamilwu : Memory management changed in Flink 1.10, which might have some impact on Flink jobs we're seeing in tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601617591 @mxm I'm guessing this: ``` 08:59:51 Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The Runner experienced the following error during execution: 08:59:51 java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e01_1584690308819_0001_01_02 timed out. 08:59:51 at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:98) 08:59:51 at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:99) 08:59:51 at org.apache.beam.sdk.loadtests.CombineLoadTest.run(CombineLoadTest.java:66) 08:59:51 at org.apache.beam.sdk.loadtests.CombineLoadTest.main(CombineLoadTest.java:169) 08:59:51 Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The Runner experienced the following error during execution: 08:59:51 java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e01_1584690308819_0001_01_02 timed out. 08:59:51 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) 08:59:51 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) 08:59:51 at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:90) 08:59:51 ... 3 more 08:59:51 Caused by: java.lang.RuntimeException: The Runner experienced the following error during execution: 08:59:51 java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e01_1584690308819_0001_01_02 timed out. 08:59:51 at org.apache.beam.runners.portability.JobServicePipelineResult.propagateErrors(JobServicePipelineResult.java:165) 08:59:51 at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:110) 08:59:51 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) 08:59:51 at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596) 08:59:51 at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) 08:59:51 at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) 08:59:51 at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) 08:59:51 at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) 08:59:51 08:59:51 > Task :sdks:java:testing:load-tests:run FAILED ``` is too little to diagnose anything. How can I find more logs from the failure? I can launch my own Flink cluster, but I'm not sure how to gather logs from TaskManagers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #11137: [BEAM-8173] Fix shadowing the actual failure exception in Filesystems…
mwalenia commented on issue #11137: [BEAM-8173] Fix shadowing the actual failure exception in Filesystems… URL: https://github.com/apache/beam/pull/11137#issuecomment-601618228 LGTM, but to be on the safe side: @kennknowles can you take a look? Won't this have any adverse or unforseen side effects? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java
mxm commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601620962 If you run your own cluster, there is a log file for each task manager in the /logs directory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] kamilwu edited a comment on issue #11136: [BEAM-7505] Add side input load test to Python SDK
kamilwu edited a comment on issue #11136: [BEAM-7505] Add side input load test to Python SDK URL: https://github.com/apache/beam/pull/11136#issuecomment-599565761 Also a note: although this PR might look like a rework of the existing side input test, it's better to treat it as a completely different, new test (a lot has changed) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #10949: [BEAM-9371] Add SideInputLoadTest to Java SDK
mwalenia commented on issue #10949: [BEAM-9371] Add SideInputLoadTest to Java SDK URL: https://github.com/apache/beam/pull/10949#issuecomment-601645586 @Ardagan @kennknowles Pinging again This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia commented on a change in pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
iemejia commented on a change in pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#discussion_r395572143 ## File path: CHANGES.md ## @@ -58,6 +58,7 @@ ## Breaking Changes * HBaseIO.ReadAll now requires a PCollection of HBaseIO.Read objects instead of HBaseQuery objects ([BEAM-9279](https://issues.apache.org/jira/browse/BEAM-9279)). +* ProcessContext.updateWatermark has been removed in favor of using a WatermarkEstimator ([BEAM-9430](https://issues.apache.org/jira/browse/BEAM-9430)). Review comment: Oh sorry it was just obviously good so I proceeded. Next time I will wait. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia commented on a change in pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
iemejia commented on a change in pull request #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#discussion_r395572143 ## File path: CHANGES.md ## @@ -58,6 +58,7 @@ ## Breaking Changes * HBaseIO.ReadAll now requires a PCollection of HBaseIO.Read objects instead of HBaseQuery objects ([BEAM-9279](https://issues.apache.org/jira/browse/BEAM-9279)). +* ProcessContext.updateWatermark has been removed in favor of using a WatermarkEstimator ([BEAM-9430](https://issues.apache.org/jira/browse/BEAM-9430)). Review comment: Oh sorry it was just obviously good so I proceeded. Next time I will wait. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
iemejia commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#issuecomment-601649153 Oh sorry it was just obviously good so I proceeded. Next time I will wait. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark
mxm commented on issue #11178: [BEAM-9430] Update CHANGES.md to reflect removal of ProcessContext#updateWatermark URL: https://github.com/apache/beam/pull/11178#issuecomment-601652327 If there was nobody assigned I think that would have been fair, but given that less than a day had passed and a review was explicitly requested, I think it is better to wait. There is no need to rush such things. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side input load test to Python SDK
kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side
input load test to Python SDK
URL: https://github.com/apache/beam/pull/11136#discussion_r395594161
##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -47,65 +49,154 @@
or:
./gradlew -PloadTest.args="
---publish_to_big_query=true
---project=...
---metrics_dataset=python_load_tests
---metrics_table=side_input
+--side_input_type=iter
--input_options='{
- \"num_records\": 1,
- \"key_size\": 1,
- \"value_size\": 1}'
---runner=DirectRunner" \
+ \"num_records\": 300,
+ \"key_size\": 5,
+ \"value_size\": 15}'" \
-PloadTest.mainClass=apache_beam.testing.load_tests.sideinput_test \
-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
"""
# pytype: skip-file
from __future__ import absolute_import
+from __future__ import division
import logging
import apache_beam as beam
-from apache_beam.pvalue import AsIter
from apache_beam.testing.load_tests.load_test import LoadTest
from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
from apache_beam.testing.synthetic_pipeline import SyntheticSource
class SideInputTest(LoadTest):
+ SIDE_INPUT_TYPES = {
+ 'iter': beam.pvalue.AsIter,
+ 'list': beam.pvalue.AsList,
+ 'dict': beam.pvalue.AsDict,
+ }
+
def __init__(self):
super(SideInputTest, self).__init__()
-self.iterations = self.get_option_or_default(
-'number_of_counter_operations', 1)
+self.windows = self.get_option_or_default('window_count', default=0)
+self.access_percentage = self.get_option_or_default(
+'access_percentage', default=100)
+if self.access_percentage < 0 or self.access_percentage > 100:
+ raise ValueError(
+ 'access_percentage: Invalid value. Should be in range '
+ 'from 0 to 100, got {} instead'.format(self.access_percentage))
+
+self.side_input_size = self.get_option_or_default(
+'side_input_size', default=0)
+if self.side_input_size == 0:
+ self.side_input_size = self.input_options.get('num_records')
+
+self.side_input_type = self.pipeline.get_option('side_input_type')
+if self.side_input_type is None:
+ raise ValueError('side_input_type is required')
+
+ def materialize_as(self):
+try:
+ return self.SIDE_INPUT_TYPES[self.side_input_type]
+except KeyError:
+ raise ValueError(
+ 'Unknown side input type. You have to provide one of '
+ 'these: {}'.format(list(self.SIDE_INPUT_TYPES.keys(
def test(self):
-def join_fn(element, side_input, iterations):
- result = []
- for i in range(iterations):
-for key, value in side_input:
- if i == iterations - 1:
-result.append({key: element[1] + value})
- yield result
-
-main_input = (
+class SequenceSideInputTestDoFn(beam.DoFn):
+ """Iterate over first n side_input elements. Iterate over all
+ elements if `first_n` is :data:`None`."""
+ def __init__(self, first_n=None):
+self._first_n = first_n
+
+ def process(self, unused_element, side_input):
+for i, _ in enumerate(side_input):
+ if self._first_n and i >= self._first_n:
+return
+ # No-op. We only make sure that the element is accessed.
Review comment:
No, as soon as `i == self._first_n`, the `if` statement is hit and the
function returns immediately.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side input load test to Python SDK
kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side
input load test to Python SDK
URL: https://github.com/apache/beam/pull/11136#discussion_r395594609
##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -47,65 +49,154 @@
or:
./gradlew -PloadTest.args="
---publish_to_big_query=true
---project=...
---metrics_dataset=python_load_tests
---metrics_table=side_input
+--side_input_type=iter
--input_options='{
- \"num_records\": 1,
- \"key_size\": 1,
- \"value_size\": 1}'
---runner=DirectRunner" \
+ \"num_records\": 300,
+ \"key_size\": 5,
+ \"value_size\": 15}'" \
-PloadTest.mainClass=apache_beam.testing.load_tests.sideinput_test \
-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
"""
# pytype: skip-file
from __future__ import absolute_import
+from __future__ import division
import logging
import apache_beam as beam
-from apache_beam.pvalue import AsIter
from apache_beam.testing.load_tests.load_test import LoadTest
from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
from apache_beam.testing.synthetic_pipeline import SyntheticSource
class SideInputTest(LoadTest):
+ SIDE_INPUT_TYPES = {
+ 'iter': beam.pvalue.AsIter,
+ 'list': beam.pvalue.AsList,
+ 'dict': beam.pvalue.AsDict,
+ }
+
def __init__(self):
super(SideInputTest, self).__init__()
-self.iterations = self.get_option_or_default(
-'number_of_counter_operations', 1)
+self.windows = self.get_option_or_default('window_count', default=0)
+self.access_percentage = self.get_option_or_default(
+'access_percentage', default=100)
+if self.access_percentage < 0 or self.access_percentage > 100:
+ raise ValueError(
+ 'access_percentage: Invalid value. Should be in range '
+ 'from 0 to 100, got {} instead'.format(self.access_percentage))
+
+self.side_input_size = self.get_option_or_default(
+'side_input_size', default=0)
+if self.side_input_size == 0:
+ self.side_input_size = self.input_options.get('num_records')
+
+self.side_input_type = self.pipeline.get_option('side_input_type')
+if self.side_input_type is None:
+ raise ValueError('side_input_type is required')
+
+ def materialize_as(self):
+try:
+ return self.SIDE_INPUT_TYPES[self.side_input_type]
+except KeyError:
+ raise ValueError(
+ 'Unknown side input type. You have to provide one of '
+ 'these: {}'.format(list(self.SIDE_INPUT_TYPES.keys(
def test(self):
-def join_fn(element, side_input, iterations):
- result = []
- for i in range(iterations):
-for key, value in side_input:
- if i == iterations - 1:
-result.append({key: element[1] + value})
- yield result
-
-main_input = (
+class SequenceSideInputTestDoFn(beam.DoFn):
+ """Iterate over first n side_input elements. Iterate over all
+ elements if `first_n` is :data:`None`."""
+ def __init__(self, first_n=None):
+self._first_n = first_n
+
+ def process(self, unused_element, side_input):
+for i, _ in enumerate(side_input):
+ if self._first_n and i >= self._first_n:
+return
+ # No-op. We only make sure that the element is accessed.
+
+class MappingSideInputTestDoFn(beam.DoFn):
+ """Take a sequence of keys as an additional side input and for each
+ key in the sequence checks the value for key in the dictionary."""
+ def process(self, unused_element, dict_side_input, keys_to_check):
+for key in keys_to_check:
+ # No-op. We only make sure that the element is accessed.
+ dict_side_input[key]
+
+class GetRandomKeys(beam.DoFn):
+ def __init__(self, n):
+self._n = n
+
+ def process(self, unused_element, dict_side_input):
+import random
+n = min(self._n, len(dict_side_input))
+return random.sample(dict_side_input.keys(), n)
+
+class AddEventTimestamps(beam.DoFn):
+ """Assign timestamp to each element of PCollection, starting from the
+ current Unix seconds-since-epoch timestamp."""
+ def setup(self):
+import time
+self.current_time = int(time.time())
Review comment:
No particular reason. I'll change it to start at 0.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mwalenia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
mwalenia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods URL: https://github.com/apache/beam/pull/11154#issuecomment-601684571 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
mwalenia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods URL: https://github.com/apache/beam/pull/11154#issuecomment-601684734 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
mwalenia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods URL: https://github.com/apache/beam/pull/11154#issuecomment-601684571 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key
should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395633155
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
##
@@ -361,6 +361,7 @@ private void processUserTimer(TimerData timer) throws
Exception {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ this.stepContext.stateInternals().getKey(),
Review comment:
you'll need the change to StreamingDataflowWorker as well, without it, the
key will always be null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key
should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395633155
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
##
@@ -361,6 +361,7 @@ private void processUserTimer(TimerData timer) throws
Exception {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ this.stepContext.stateInternals().getKey(),
Review comment:
you'll need the change to StreamingDataflowWorker as well, without it, the
key will be null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11180: [BEAM-9563] Change ToListCombineFn access level to private
chamikaramj commented on issue #11180: [BEAM-9563] Change ToListCombineFn access level to private URL: https://github.com/apache/beam/pull/11180#issuecomment-601701191 Thanks. LGTM. Agree on discussing more regarding the original change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key
should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395644552
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
##
@@ -361,6 +361,7 @@ private void processUserTimer(TimerData timer) throws
Exception {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ this.stepContext.stateInternals().getKey(),
Review comment:
@steveniemitz added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11122: [BEAM-9346] Improve the efficiency of TFRecordIO
chamikaramj commented on a change in pull request #11122: [BEAM-9346] Improve
the efficiency of TFRecordIO
URL: https://github.com/apache/beam/pull/11122#discussion_r395658938
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
##
@@ -410,13 +412,44 @@ private GatherResults(Coder resultCoder) {
} else {
// Pass results via a side input rather than reshuffle, because we
need to get an empty
// iterable to finalize if there are no results.
-return input
-.getPipeline()
-.apply(Reify.viewInGlobalWindow(input.apply(View.asList()),
ListCoder.of(resultCoder)));
+return input.apply("ToList", Combine.globally(new
ToListCombineFn<>()));
Review comment:
Thanks. Agree with Luke. Combine globally has shuffle (GBK) inside hence
breaks above statements. This could be a regression at least for Dataflow when
there are no outputs. We should try following cases.
(1) Dataflow (and possibly other runners may have similar regressions ?)
with an empty output.
(2) Writing using a non-global window while WriteFiles.withWindowedWrites()
not set.
Luke, does that make sense ? Anything else to try out to make sure there's
no regression here ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side input load test to Python SDK
kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side
input load test to Python SDK
URL: https://github.com/apache/beam/pull/11136#discussion_r395594161
##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -47,65 +49,154 @@
or:
./gradlew -PloadTest.args="
---publish_to_big_query=true
---project=...
---metrics_dataset=python_load_tests
---metrics_table=side_input
+--side_input_type=iter
--input_options='{
- \"num_records\": 1,
- \"key_size\": 1,
- \"value_size\": 1}'
---runner=DirectRunner" \
+ \"num_records\": 300,
+ \"key_size\": 5,
+ \"value_size\": 15}'" \
-PloadTest.mainClass=apache_beam.testing.load_tests.sideinput_test \
-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
"""
# pytype: skip-file
from __future__ import absolute_import
+from __future__ import division
import logging
import apache_beam as beam
-from apache_beam.pvalue import AsIter
from apache_beam.testing.load_tests.load_test import LoadTest
from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
from apache_beam.testing.synthetic_pipeline import SyntheticSource
class SideInputTest(LoadTest):
+ SIDE_INPUT_TYPES = {
+ 'iter': beam.pvalue.AsIter,
+ 'list': beam.pvalue.AsList,
+ 'dict': beam.pvalue.AsDict,
+ }
+
def __init__(self):
super(SideInputTest, self).__init__()
-self.iterations = self.get_option_or_default(
-'number_of_counter_operations', 1)
+self.windows = self.get_option_or_default('window_count', default=0)
+self.access_percentage = self.get_option_or_default(
+'access_percentage', default=100)
+if self.access_percentage < 0 or self.access_percentage > 100:
+ raise ValueError(
+ 'access_percentage: Invalid value. Should be in range '
+ 'from 0 to 100, got {} instead'.format(self.access_percentage))
+
+self.side_input_size = self.get_option_or_default(
+'side_input_size', default=0)
+if self.side_input_size == 0:
+ self.side_input_size = self.input_options.get('num_records')
+
+self.side_input_type = self.pipeline.get_option('side_input_type')
+if self.side_input_type is None:
+ raise ValueError('side_input_type is required')
+
+ def materialize_as(self):
+try:
+ return self.SIDE_INPUT_TYPES[self.side_input_type]
+except KeyError:
+ raise ValueError(
+ 'Unknown side input type. You have to provide one of '
+ 'these: {}'.format(list(self.SIDE_INPUT_TYPES.keys(
def test(self):
-def join_fn(element, side_input, iterations):
- result = []
- for i in range(iterations):
-for key, value in side_input:
- if i == iterations - 1:
-result.append({key: element[1] + value})
- yield result
-
-main_input = (
+class SequenceSideInputTestDoFn(beam.DoFn):
+ """Iterate over first n side_input elements. Iterate over all
+ elements if `first_n` is :data:`None`."""
+ def __init__(self, first_n=None):
+self._first_n = first_n
+
+ def process(self, unused_element, side_input):
+for i, _ in enumerate(side_input):
+ if self._first_n and i >= self._first_n:
+return
+ # No-op. We only make sure that the element is accessed.
Review comment:
No, as soon as `i == self._first_n`, the `if` statement is hit and the
function returns immediately.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side input load test to Python SDK
kamilwu commented on a change in pull request #11136: [BEAM-7505] Add side
input load test to Python SDK
URL: https://github.com/apache/beam/pull/11136#discussion_r395678590
##
File path: sdks/python/apache_beam/testing/load_tests/sideinput_test.py
##
@@ -47,65 +49,154 @@
or:
./gradlew -PloadTest.args="
---publish_to_big_query=true
---project=...
---metrics_dataset=python_load_tests
---metrics_table=side_input
+--side_input_type=iter
--input_options='{
- \"num_records\": 1,
- \"key_size\": 1,
- \"value_size\": 1}'
---runner=DirectRunner" \
+ \"num_records\": 300,
+ \"key_size\": 5,
+ \"value_size\": 15}'" \
-PloadTest.mainClass=apache_beam.testing.load_tests.sideinput_test \
-Prunner=DirectRunner :sdks:python:apache_beam:testing:load_tests:run
"""
# pytype: skip-file
from __future__ import absolute_import
+from __future__ import division
import logging
import apache_beam as beam
-from apache_beam.pvalue import AsIter
from apache_beam.testing.load_tests.load_test import LoadTest
from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
from apache_beam.testing.synthetic_pipeline import SyntheticSource
class SideInputTest(LoadTest):
+ SIDE_INPUT_TYPES = {
+ 'iter': beam.pvalue.AsIter,
+ 'list': beam.pvalue.AsList,
+ 'dict': beam.pvalue.AsDict,
+ }
+
def __init__(self):
super(SideInputTest, self).__init__()
-self.iterations = self.get_option_or_default(
-'number_of_counter_operations', 1)
+self.windows = self.get_option_or_default('window_count', default=0)
+self.access_percentage = self.get_option_or_default(
+'access_percentage', default=100)
+if self.access_percentage < 0 or self.access_percentage > 100:
+ raise ValueError(
+ 'access_percentage: Invalid value. Should be in range '
+ 'from 0 to 100, got {} instead'.format(self.access_percentage))
+
+self.side_input_size = self.get_option_or_default(
+'side_input_size', default=0)
+if self.side_input_size == 0:
+ self.side_input_size = self.input_options.get('num_records')
+
+self.side_input_type = self.pipeline.get_option('side_input_type')
+if self.side_input_type is None:
+ raise ValueError('side_input_type is required')
+
+ def materialize_as(self):
+try:
+ return self.SIDE_INPUT_TYPES[self.side_input_type]
+except KeyError:
+ raise ValueError(
+ 'Unknown side input type. You have to provide one of '
+ 'these: {}'.format(list(self.SIDE_INPUT_TYPES.keys(
def test(self):
-def join_fn(element, side_input, iterations):
- result = []
- for i in range(iterations):
-for key, value in side_input:
- if i == iterations - 1:
-result.append({key: element[1] + value})
- yield result
-
-main_input = (
+class SequenceSideInputTestDoFn(beam.DoFn):
+ """Iterate over first n side_input elements. Iterate over all
+ elements if `first_n` is :data:`None`."""
+ def __init__(self, first_n=None):
+self._first_n = first_n
+
+ def process(self, unused_element, side_input):
+for i, _ in enumerate(side_input):
+ if self._first_n and i >= self._first_n:
+return
+ # No-op. We only make sure that the element is accessed.
Review comment:
Ohh that's right. Thanks.
I'll use a while loop and iter(), so that we'd be able to treat list and
iterable in the same way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build
je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168#issuecomment-601750634 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build
je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168#issuecomment-601750689 Run Java HadoopFormatIO Performance Test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build
je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168#issuecomment-601750794 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build
je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168#issuecomment-601750733 Run BigQueryIO Streaming Performance Test Java This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build
je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168#issuecomment-601750952 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build
je-ik commented on issue #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168#issuecomment-601750996 Run SQL Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] piotr-szuberski commented on issue #11181: [BEAM-9500] Refactor load tests
piotr-szuberski commented on issue #11181: [BEAM-9500] Refactor load tests URL: https://github.com/apache/beam/pull/11181#issuecomment-601761169 @mwalenia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
