[jira] [Resolved] (BEAM-9507) Beam dependency check failing
[
https://issues.apache.org/jira/browse/BEAM-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Szuberski resolved BEAM-9507.
---
Fix Version/s: Not applicable
Resolution: Fixed
> Beam dependency check failing
> -
>
> Key: BEAM-9507
> URL: https://issues.apache.org/jira/browse/BEAM-9507
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Michał Walenia
>Assignee: Piotr Szuberski
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Here are the logs:
> [https://builds.apache.org/job/beam_Dependency_Check/257/console]
>
>from grpc_tools import protoc*13:04:25* ImportError: No module
> named 'grpc_tools'*13:04:25* *13:04:25* During handling of the above
> exception, another exception occurred:*13:04:25* *13:04:25* Traceback
> (most recent call last):*13:04:25* File
> "/usr/lib/python3.5/multiprocessing/process.py", line 249, in
> _bootstrap*13:04:25* self.run()*13:04:25* File
> "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run*13:04:25*
> self._target(*self._args, **self._kwargs)*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 378, in _install_grpcio_tools_and_generate_proto_files*13:04:25*
> generate_proto_files(force=force)*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 315, in generate_proto_files*13:04:25* protoc_gen_mypy =
> _find_protoc_gen_mypy()*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 233, in _find_protoc_gen_mypy*13:04:25* (fname, ',
> '.join(search_paths)))*13:04:25* RuntimeError: Could not find
> protoc-gen-mypy in
> /home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/sdks/python/bin,
>
> /home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/sdks/python/bin,
> /home/jenkins/tools/java/latest1.8/bin, /usr/local/sbin, /usr/local/bin,
> /usr/sbin, /usr/bin, /sbin, /bin, /usr/games, /usr/local/games*13:04:25*
> Traceback (most recent call last):*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 292, in generate_proto_files*13:04:25* from grpc_tools import
> protoc*13:04:25* ImportError: No module named 'grpc_tools'*13:04:25*
> *13:04:25* During handling of the above exception, another exception
> occurred:*13:04:25* *13:04:25* Traceback (most recent call
> last):*13:04:25* File "", line 1, in *13:04:25*
> File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/setup.py",
> line 315, in *13:04:25* 'mypy':
> generate_protos_first(mypy),*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/sdks/python/lib/python3.5/site-packages/setuptools/__init__.py",
> line 144, in setup*13:04:25* return
> distutils.core.setup(**attrs)*13:04:25* File
> "/usr/lib/python3.5/distutils/core.py", line 148, in setup*13:04:25*
> dist.run_commands()*13:04:25* File
> "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands*13:04:25*
> self.run_command(cmd)*13:04:25* File
> "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command*13:04:25*
> cmd_obj.run()*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/setup.py",
> line 239, in run*13:04:25*
> gen_protos.generate_proto_files()*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 310, in generate_proto_files*13:04:25* raise ValueError("Proto
> generation failed (see log for details).")*13:04:25* ValueError: Proto
> generation failed (see log for details).*13:04:25*
> *13:04:25* ERROR: Command errored out
> with exit status 1: python setup.py egg_info Check the logs for full command
> output.*13:04:25* *13:04:25* >
> *Task :sdks:python:dependencyUpdates*
> FAILED*13:04:25* *13:04:25* FAILURE: Build failed with an
> exception.*13:04:25* *13:04:25* * Where:*13:04:25* Build file
> '/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/build.gradle'
> line: 94*13:04:25* *13:04:25* * What went wrong:*13:04:25* Execution failed
> for task ':sdks:python:dependencyUpdates'.*13:04:25* > Process 'command 'sh''
> finished with non-zero exit value 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?focusedWorklogId=409368&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409368 ] ASF GitHub Bot logged work on BEAM-9573: Author: ASF GitHub Bot Created on: 25/Mar/20 08:22 Start Date: 25/Mar/20 08:22 Worklog Time Spent: 10m Work Description: mxm commented on issue #11200: [BEAM-9573] Correct computing of watermark hold for timer output timestamp URL: https://github.com/apache/beam/pull/11200#issuecomment-603704355 Java PreCommit is flaky due to https://issues.apache.org/jira/browse/BEAM-9578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409368) Time Spent: 5h 10m (was: 5h) > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?focusedWorklogId=409369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409369 ] ASF GitHub Bot logged work on BEAM-9573: Author: ASF GitHub Bot Created on: 25/Mar/20 08:23 Start Date: 25/Mar/20 08:23 Worklog Time Spent: 10m Work Description: mxm commented on pull request #11200: [BEAM-9573] Correct computing of watermark hold for timer output timestamp URL: https://github.com/apache/beam/pull/11200 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409369) Time Spent: 5h 20m (was: 5h 10m) > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9436) Try to avoid elements list materialization in GBK
[
https://issues.apache.org/jira/browse/BEAM-9436?focusedWorklogId=409388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409388
]
ASF GitHub Bot logged work on BEAM-9436:
Author: ASF GitHub Bot
Created on: 25/Mar/20 09:04
Start Date: 25/Mar/20 09:04
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #11055: [BEAM-9436]
Improve GBK in spark structured streaming runner
URL: https://github.com/apache/beam/pull/11055#issuecomment-603725379
> > I think you missed this comment 8 days ago: #11055 (comment)
>
> Once again, I saw your results (and even mentioned them in my previous
comment) but I asked for more details since it was not clear what kind of tests
have been running and, before, we usually provided Nexmark results in such kind
of optimisations.
My bad, the results comment was not clear enough: when the PR contained the
jenkins scripts of the load tests it was clear but now that I have separated
them in a different PR, it is no more clear: the test that I ran was
`GroupByKeyLoadTest` and the CL:
`-Xms12g -Xmx20g -Prunner=":runners:spark"
-PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest"
-PloadTest.args="--fanout=1 --iterations=1 --streaming=false
--runner=SparkStructuredStreamingRunner
--sourceOptions={\"numRecords\":20,\"keySizeBytes\":10,\"valueSizeBytes\":10}"`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409388)
Time Spent: 12h 20m (was: 12h 10m)
> Try to avoid elements list materialization in GBK
> -
>
> Key: BEAM-9436
> URL: https://issues.apache.org/jira/browse/BEAM-9436
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Labels: structured-streaming
> Time Spent: 12h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9371) Implement SideInput load test in Java
[ https://issues.apache.org/jira/browse/BEAM-9371?focusedWorklogId=409389&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409389 ] ASF GitHub Bot logged work on BEAM-9371: Author: ASF GitHub Bot Created on: 25/Mar/20 09:06 Start Date: 25/Mar/20 09:06 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10949: [BEAM-9371] Add SideInputLoadTest to Java SDK URL: https://github.com/apache/beam/pull/10949#issuecomment-603726033 @Ardagan Pinging again :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409389) Time Spent: 4h 10m (was: 4h) > Implement SideInput load test in Java > - > > Key: BEAM-9371 > URL: https://issues.apache.org/jira/browse/BEAM-9371 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9416) BIP-1: Convert avro metadata to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9416?focusedWorklogId=409393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409393 ] ASF GitHub Bot logged work on BEAM-9416: Author: ASF GitHub Bot Created on: 25/Mar/20 09:19 Start Date: 25/Mar/20 09:19 Worklog Time Spent: 10m Work Description: alexvanboxel commented on pull request #11217: [BEAM-9416] BIP-1: Convert Avro metadata to Schema options URL: https://github.com/apache/beam/pull/11217 Convert Avro properties to Typed Beam Schema Options. The avro implementation of the schema options Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [ Try to avoid elements list materialization in GBK
[
https://issues.apache.org/jira/browse/BEAM-9436?focusedWorklogId=409406&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409406
]
ASF GitHub Bot logged work on BEAM-9436:
Author: ASF GitHub Bot
Created on: 25/Mar/20 09:34
Start Date: 25/Mar/20 09:34
Worklog Time Spent: 10m
Work Description: echauchot commented on pull request #11055: [BEAM-9436]
Improve GBK in spark structured streaming runner
URL: https://github.com/apache/beam/pull/11055#discussion_r397716574
##
File path:
runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/GroupAlsoByWindowViaOutputBufferFn.java
##
@@ -65,9 +65,15 @@ public GroupAlsoByWindowViaOutputBufferFn(
@Override
public Iterator>>> call(
- KV>> kv) throws Exception {
-K key = kv.getKey();
-Iterable> values = kv.getValue();
+ K key, Iterator>> iterator) throws Exception
{
+
+// we have to meterialize the Iterator because
ReduceFnRunner.processElements expects
+// ArrayList> and not Iterator>>
+ArrayList> values = new ArrayList<>();
+while (iterator.hasNext()) {
+ WindowedValue> wv = iterator.next();
+ values.add(wv.withValue(wv.getValue().getValue()));
Review comment:
Yes sure, I agree about hot keys. But if spark spills to disk it is because
it is out of memory. And this PR does not claim to fix the OOM issue that also
existed in the previous version of GBK but it claims to remove a step in the
translation and avoid memory consumption. I could profile the load test to give
you numbers on memory because it is not integrated in load tests
Also see my comment
https://github.com/apache/beam/pull/11055#discussion_r397246846 regarding
materialization.
Anyway there is hotkey configuration in GBKLoadTest. I'll configure it and
post the results
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409406)
Time Spent: 12.5h (was: 12h 20m)
> Try to avoid elements list materialization in GBK
> -
>
> Key: BEAM-9436
> URL: https://issues.apache.org/jira/browse/BEAM-9436
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Labels: structured-streaming
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng closed BEAM-9573. - Resolution: Fixed Close it as the PR is already merged :) > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reopened BEAM-9573: Reopening it until it is cherry picked. > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9573: --- Status: Open (was: Triage Needed) > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2546) Create InfluxDbIO
[ https://issues.apache.org/jira/browse/BEAM-2546?focusedWorklogId=409411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409411 ] ASF GitHub Bot logged work on BEAM-2546: Author: ASF GitHub Bot Created on: 25/Mar/20 10:00 Start Date: 25/Mar/20 10:00 Worklog Time Spent: 10m Work Description: iemejia commented on issue #11028: BEAM-2546 Beam IO for InfluxDB URL: https://github.com/apache/beam/pull/11028#issuecomment-603750036 Sorry for the delay and thanks for your patience @bipinupd , my review queue at least is finally advancing. I added @mwalenia so he can help me verfify that the CI / IT part is consistent with Beam common patterns and it works. I will give a round to the code and comment later on today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409411) Time Spent: 8h 50m (was: 8h 40m) > Create InfluxDbIO > - > > Key: BEAM-2546 > URL: https://issues.apache.org/jira/browse/BEAM-2546 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Bipin Upadhyaya >Priority: Major > Fix For: 2.21.0 > > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9598) _CustomBigQuerySource checks valueprovider when it's not needed
[ https://issues.apache.org/jira/browse/BEAM-9598?focusedWorklogId=409415&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409415 ] ASF GitHub Bot logged work on BEAM-9598: Author: ASF GitHub Bot Created on: 25/Mar/20 10:03 Start Date: 25/Mar/20 10:03 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11206: [BEAM-9598] Ensuring _CustomBigQuerySource ValueProvider is only checked when strictly needed URL: https://github.com/apache/beam/pull/11206#issuecomment-603751295 @pabloem Thanks for the fix :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409415) Time Spent: 1h 40m (was: 1.5h) > _CustomBigQuerySource checks valueprovider when it's not needed > --- > > Key: BEAM-9598 > URL: https://issues.apache.org/jira/browse/BEAM-9598 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, test-failures >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9044) BIP-1: Convert protobuf options to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9044 started by Alex Van Boxel. > BIP-1: Convert protobuf options to Schema options > - > > Key: BEAM-9044 > URL: https://issues.apache.org/jira/browse/BEAM-9044 > Project: Beam > Issue Type: Sub-task > Components: extensions-java-protobuf >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > Protobuf has a rich metadata system called options. This system is fully > typed and matches Beams Schema Option system. For now we can only convert the > following protobuf options: > * File Options -> _Beam doesn't have this concept_ > * Message Options -> *Beam Schema Options* > * Field Options -> *Beam Schema Options* > * Enum Options -> _This can only be done when logical type options are > available_ > * EnumValue Options -> _This can only be done when logical type options are > available_ > * Service Options -> _Beam doesn't have this concept_ > * Method Options -> _Beam doesn't have this concept_ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9035) BIP-1: Typed options for Row Schema and Fields
[ https://issues.apache.org/jira/browse/BEAM-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel updated BEAM-9035: - Fix Version/s: (was: 2.19.0) 2.20.0 > BIP-1: Typed options for Row Schema and Fields > -- > > Key: BEAM-9035 > URL: https://issues.apache.org/jira/browse/BEAM-9035 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Fix For: 2.20.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > This is the first issue of a multipart commit: this ticket implements the > basic infrastructure of options on row and field. > Full explanation: > Introduce the concept of Options in Beam Schema’s to add extra context to > fields and schema. In contracts to metadata, options would be added to > fields, logical types and rows. In the options schema convertors can add > options/annotations/decorators that were in the original schema, this context > can be used in the rest of the pipeline for specific transformations or > augment the end schema in the target output. > Examples of options are: > * informational: like the source of the data, ... > * drive decisions further in the pipeline: flatten a row into another, > rename a field, ... > * influence something in the output: like cluster index, primary key, ... > * logical type information > And option is a key/typed value combination. The advantages of having the > value types is: > * Having strongly typed options would give a *portable way of Logical Types* > to have structured information that could be shared over different languages. > * This could keep the type intact when mapping from a formats that have > strongly typed options (example: Protobuf). > This is part of a multi ticket implementation. The following tickets are > related: > # Typed options for Row Schema and Fields > # Convert Proto Options to Beam Schema options > # Convert Avro extra information for Beam string options > # Replace meta data with Logical Type options > # Extract meta data in Calcite SQL to Beam options > # Extract meta data in Zeta SQL to Beam options > # Add java example of using option in a transform > This feature is discussed with Reuven Lax, Brian Hulette -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9035) BIP-1: Typed options for Row Schema and Fields
[ https://issues.apache.org/jira/browse/BEAM-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel updated BEAM-9035: - Fix Version/s: (was: 2.20.0) 2.21.0 > BIP-1: Typed options for Row Schema and Fields > -- > > Key: BEAM-9035 > URL: https://issues.apache.org/jira/browse/BEAM-9035 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Fix For: 2.21.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > This is the first issue of a multipart commit: this ticket implements the > basic infrastructure of options on row and field. > Full explanation: > Introduce the concept of Options in Beam Schema’s to add extra context to > fields and schema. In contracts to metadata, options would be added to > fields, logical types and rows. In the options schema convertors can add > options/annotations/decorators that were in the original schema, this context > can be used in the rest of the pipeline for specific transformations or > augment the end schema in the target output. > Examples of options are: > * informational: like the source of the data, ... > * drive decisions further in the pipeline: flatten a row into another, > rename a field, ... > * influence something in the output: like cluster index, primary key, ... > * logical type information > And option is a key/typed value combination. The advantages of having the > value types is: > * Having strongly typed options would give a *portable way of Logical Types* > to have structured information that could be shared over different languages. > * This could keep the type intact when mapping from a formats that have > strongly typed options (example: Protobuf). > This is part of a multi ticket implementation. The following tickets are > related: > # Typed options for Row Schema and Fields > # Convert Proto Options to Beam Schema options > # Convert Avro extra information for Beam string options > # Replace meta data with Logical Type options > # Extract meta data in Calcite SQL to Beam options > # Extract meta data in Zeta SQL to Beam options > # Add java example of using option in a transform > This feature is discussed with Reuven Lax, Brian Hulette -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9044) BIP-1: Convert protobuf options to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel resolved BEAM-9044. -- Fix Version/s: 2.21.0 Resolution: Fixed > BIP-1: Convert protobuf options to Schema options > - > > Key: BEAM-9044 > URL: https://issues.apache.org/jira/browse/BEAM-9044 > Project: Beam > Issue Type: Sub-task > Components: extensions-java-protobuf >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Protobuf has a rich metadata system called options. This system is fully > typed and matches Beams Schema Option system. For now we can only convert the > following protobuf options: > * File Options -> _Beam doesn't have this concept_ > * Message Options -> *Beam Schema Options* > * Field Options -> *Beam Schema Options* > * Enum Options -> _This can only be done when logical type options are > available_ > * EnumValue Options -> _This can only be done when logical type options are > available_ > * Service Options -> _Beam doesn't have this concept_ > * Method Options -> _Beam doesn't have this concept_ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9416) BIP-1: Convert avro metadata to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel resolved BEAM-9416. -- Resolution: Fixed > BIP-1: Convert avro metadata to Schema options > -- > > Key: BEAM-9416 > URL: https://issues.apache.org/jira/browse/BEAM-9416 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Fix For: 2.21.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Avro has some metadata that can be added to the normal type information. It > is based on json typing, so the conversion will be best effort (probably we > can bet int, string and float out of it). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9604) BIP-1: Remove schema metadata usage for Protobuf extension
Alex Van Boxel created BEAM-9604: Summary: BIP-1: Remove schema metadata usage for Protobuf extension Key: BEAM-9604 URL: https://issues.apache.org/jira/browse/BEAM-9604 Project: Beam Issue Type: Task Components: extensions-java-protobuf Reporter: Alex Van Boxel Assignee: Alex Van Boxel Replace the schema metadata usage and replace it with using the options. This will probably mean: * Moving the message_name metadata to a Schema option (for field, map key and value) * Replace the proto_number to a Field option -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9604) BIP-1: Remove schema metadata usage for Protobuf extension
[ https://issues.apache.org/jira/browse/BEAM-9604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel updated BEAM-9604: - Parent: BEAM-9275 Issue Type: Sub-task (was: Task) > BIP-1: Remove schema metadata usage for Protobuf extension > -- > > Key: BEAM-9604 > URL: https://issues.apache.org/jira/browse/BEAM-9604 > Project: Beam > Issue Type: Sub-task > Components: extensions-java-protobuf >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > > Replace the schema metadata usage and replace it with using the options. This > will probably mean: > * Moving the message_name metadata to a Schema option (for field, map key > and value) > * Replace the proto_number to a Field option -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9044) BIP-1: Convert protobuf options to Schema options
[
https://issues.apache.org/jira/browse/BEAM-9044?focusedWorklogId=409423&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409423
]
ASF GitHub Bot logged work on BEAM-9044:
Author: ASF GitHub Bot
Created on: 25/Mar/20 10:27
Start Date: 25/Mar/20 10:27
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on pull request #10529:
[BEAM-9044] Protobuf options to Schema options
URL: https://github.com/apache/beam/pull/10529#discussion_r397748991
##
File path:
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java
##
@@ -205,10 +205,12 @@ static Schema getSchema(Descriptors.Descriptor
descriptor) {
// Store proto field number in metadata.
FieldType fieldType =
withMetaData(beamFieldTypeFromProtoField(fieldDescriptor),
fieldDescriptor);
Review comment:
Now all the options are prefix. I moved the removal of the metadata to
another ticket, because it's more involved:
[BEAM-9604] BIP-1: Remove schema metadata usage for Protobuf extension
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409423)
Time Spent: 1h 40m (was: 1.5h)
> BIP-1: Convert protobuf options to Schema options
> -
>
> Key: BEAM-9044
> URL: https://issues.apache.org/jira/browse/BEAM-9044
> Project: Beam
> Issue Type: Sub-task
> Components: extensions-java-protobuf
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.21.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Protobuf has a rich metadata system called options. This system is fully
> typed and matches Beams Schema Option system. For now we can only convert the
> following protobuf options:
> * File Options -> _Beam doesn't have this concept_
> * Message Options -> *Beam Schema Options*
> * Field Options -> *Beam Schema Options*
> * Enum Options -> _This can only be done when logical type options are
> available_
> * EnumValue Options -> _This can only be done when logical type options are
> available_
> * Service Options -> _Beam doesn't have this concept_
> * Method Options -> _Beam doesn't have this concept_
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066588#comment-17066588 ] Maximilian Michels commented on BEAM-9573: -- Yes, the issue was open on purpose until the "Fix Version" target is reached. > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2546) Create InfluxDbIO
[
https://issues.apache.org/jira/browse/BEAM-2546?focusedWorklogId=409431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409431
]
ASF GitHub Bot logged work on BEAM-2546:
Author: ASF GitHub Bot
Created on: 25/Mar/20 10:47
Start Date: 25/Mar/20 10:47
Worklog Time Spent: 10m
Work Description: mwalenia commented on pull request #11028: BEAM-2546
Beam IO for InfluxDB
URL: https://github.com/apache/beam/pull/11028#discussion_r397759342
##
File path: .test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import CommonJobProperties as common
+import Kubernetes
+
+String jobName = "beam_PerformanceTests_InfluxDBIO_IT"
+
+job(jobName) {
+ common.setTopLevelMainJobProperties(delegate)
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Java InfluxDBIO Performance Test',
+ 'Run Java InfluxDBIO Performance Test')
+
+ String namespace = common.getKubernetesNamespace(jobName)
+ String kubeconfigPath = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfigPath, namespace)
+
+
k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/influxdb/influxdb.yml"))
+ String influxDBHostName = "LOAD_BALANCER_IP"
+ k8s.loadBalancerIP("influxdb-load-balancer-service", influxDBHostName)
+ Map pipelineOptions = [
+ influxDBURL : "http://\$${influxDBHostName}:8086";,
+ influxDBUserName : "superadmin",
+ influxDBPassword : "supersecretpassword",
+ databaseName : "db1"
+ ]
+
+ steps {
+gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+
switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=direct")
Review comment:
I think this should be executed on Dataflow so that we can test integration
of a real runner with a real service.
Remember about the appropriate options relevant to the Dataflow runner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409431)
Time Spent: 9h (was: 8h 50m)
> Create InfluxDbIO
> -
>
> Key: BEAM-2546
> URL: https://issues.apache.org/jira/browse/BEAM-2546
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Bipin Upadhyaya
>Priority: Major
> Fix For: 2.21.0
>
> Time Spent: 9h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9598) _CustomBigQuerySource checks valueprovider when it's not needed
[ https://issues.apache.org/jira/browse/BEAM-9598?focusedWorklogId=409439&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409439 ] ASF GitHub Bot logged work on BEAM-9598: Author: ASF GitHub Bot Created on: 25/Mar/20 11:08 Start Date: 25/Mar/20 11:08 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11206: [BEAM-9598] Ensuring _CustomBigQuerySource ValueProvider is only checked when strictly needed URL: https://github.com/apache/beam/pull/11206#issuecomment-603780062 Cheers for the quick fix @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409439) Time Spent: 1h 50m (was: 1h 40m) > _CustomBigQuerySource checks valueprovider when it's not needed > --- > > Key: BEAM-9598 > URL: https://issues.apache.org/jira/browse/BEAM-9598 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, test-failures >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9044) BIP-1: Convert protobuf options to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9044?focusedWorklogId=409444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409444 ] ASF GitHub Bot logged work on BEAM-9044: Author: ASF GitHub Bot Created on: 25/Mar/20 11:11 Start Date: 25/Mar/20 11:11 Worklog Time Spent: 10m Work Description: alexvanboxel commented on issue #10529: [BEAM-9044] Protobuf options to Schema options URL: https://github.com/apache/beam/pull/10529#issuecomment-603781120 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409444) Time Spent: 1h 50m (was: 1h 40m) > BIP-1: Convert protobuf options to Schema options > - > > Key: BEAM-9044 > URL: https://issues.apache.org/jira/browse/BEAM-9044 > Project: Beam > Issue Type: Sub-task > Components: extensions-java-protobuf >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Protobuf has a rich metadata system called options. This system is fully > typed and matches Beams Schema Option system. For now we can only convert the > following protobuf options: > * File Options -> _Beam doesn't have this concept_ > * Message Options -> *Beam Schema Options* > * Field Options -> *Beam Schema Options* > * Enum Options -> _This can only be done when logical type options are > available_ > * EnumValue Options -> _This can only be done when logical type options are > available_ > * Service Options -> _Beam doesn't have this concept_ > * Method Options -> _Beam doesn't have this concept_ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9214) TypeError: GetJobMetrics() missing 1 required positional argument: 'context'
[ https://issues.apache.org/jira/browse/BEAM-9214?focusedWorklogId=409448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409448 ] ASF GitHub Bot logged work on BEAM-9214: Author: ASF GitHub Bot Created on: 25/Mar/20 11:15 Start Date: 25/Mar/20 11:15 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11213: [BEAM-9214] Add missing argument to GetJobMetrics. URL: https://github.com/apache/beam/pull/11213#issuecomment-603783082 Could you elaborate on why this change is needed? I see that proto representation of `GetJobMetricsRequest` hasn't changed recently: https://github.com/apache/beam/blob/af8559da52e9d67bd3a069da4d51698108336723/model/job-management/src/main/proto/beam_job_api.proto#L257-L259 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409448) Time Spent: 20m (was: 10m) > TypeError: GetJobMetrics() missing 1 required positional argument: 'context' > > > Key: BEAM-9214 > URL: https://issues.apache.org/jira/browse/BEAM-9214 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 20m > Remaining Estimate: 0h > > I've seen this error myself, and had multiple users report it, but I don't > know what causes it. In this case, it happened when using > --flink_submit_uber_jar. But the job succeeded when retried. (EDIT: User > reported job worked when retried; I have no idea why that might happen.) > INFO:root:Started Flink job as 2d1bf5003307c580900abe6bbe56d9e8 > INFO:root:Job state changed to STOPPED > INFO:root:Job state changed to RUNNING > Traceback (most recent call last): > File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main > "__main__", mod_spec) > File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code > exec(code, run_globals) > File > "/usr/local/lib/python3.7/site-packages/apache_beam/examples/wordcount.py", > line 138, in > run() > File > "/usr/local/lib/python3.7/site-packages/apache_beam/examples/wordcount.py", > line 124, in run > query_result = result.metrics().query(empty_lines_filter) > File > "/usr/local/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 390, in metrics > beam_job_api_pb2.GetJobMetricsRequest(job_id=self._job_id)) > TypeError: GetJobMetrics() missing 1 required positional argument: 'context' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9605) BIP-1: Rename setRowOption to setOption on Option builder
Alex Van Boxel created BEAM-9605: Summary: BIP-1: Rename setRowOption to setOption on Option builder Key: BEAM-9605 URL: https://issues.apache.org/jira/browse/BEAM-9605 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Alex Van Boxel Assignee: Alex Van Boxel Rename setRowOption to setOption on Option builder as setRowOption name is too confusing. It sets an option as a Row, not an option on a Row. Using setOption is better and doesn't conflict with the other setOption with 3 parameters and explicit type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[ https://issues.apache.org/jira/browse/BEAM-9420?focusedWorklogId=409451&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409451 ] ASF GitHub Bot logged work on BEAM-9420: Author: ASF GitHub Bot Created on: 25/Mar/20 11:23 Start Date: 25/Mar/20 11:23 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #11099: [BEAM-9420] Configurable timeout for blocking kafka API call(s) URL: https://github.com/apache/beam/pull/11099#issuecomment-603786407 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409451) Time Spent: 1h 40m (was: 1.5h) > Configurable timeout for Kafka setupInitialOffset() > --- > > Key: BEAM-9420 > URL: https://issues.apache.org/jira/browse/BEAM-9420 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.19.0 >Reporter: Jozef Vilcek >Assignee: Jozef Vilcek >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > If bootstrap brokers does contain an unhealthy server, it can break the start > of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for > `setupInitialOffset()`. Wait timeout is either a double time of `request. > timeout.ms` or some default constant. In both cases, it might not be enough > time for kafka-client to initiate fallback and retry metadata discovery via > another broker from given bootstrap list. > The client should be able to specify timeout for `setupInitialOffset()` > explicitly as a setting to KafkaIO read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9550) beam_PostCommit_Python_Chicago_Taxi_Flink OOM
[ https://issues.apache.org/jira/browse/BEAM-9550?focusedWorklogId=409473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409473 ] ASF GitHub Bot logged work on BEAM-9550: Author: ASF GitHub Bot Created on: 25/Mar/20 12:20 Start Date: 25/Mar/20 12:20 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors. URL: https://github.com/apache/beam/pull/11193#issuecomment-603808862 Run Chicago Taxi on Flink This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409473) Time Spent: 1h 40m (was: 1.5h) > beam_PostCommit_Python_Chicago_Taxi_Flink OOM > - > > Key: BEAM-9550 > URL: https://issues.apache.org/jira/browse/BEAM-9550 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kyle Weaver >Assignee: Kamil Wasilewski >Priority: Major > Labels: currently-failing > Time Spent: 1h 40m > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/ > The following error has been occurring consistently for several days: > 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace > 07:57:27 Traceback (most recent call last): > 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in > 07:57:27 main() > 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main > 07:57:27 project=known_args.metric_reporting_project) > 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats > 07:57:27 result.wait_until_finish() > 07:57:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 545, in wait_until_finish > 07:57:27 (self._job_id, self._state, self._last_error_message())) > 07:57:27 RuntimeError: Pipeline > chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb > failed in state FAILED: java.lang.OutOfMemoryError: Metaspace -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9557) Error setting processing time timers near end-of-window
[
https://issues.apache.org/jira/browse/BEAM-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066643#comment-17066643
]
Steve Niemitz commented on BEAM-9557:
-
ah ok, yeah I agree!
> Error setting processing time timers near end-of-window
> ---
>
> Key: BEAM-9557
> URL: https://issues.apache.org/jira/browse/BEAM-9557
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Reporter: Steve Niemitz
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.20.0
>
>
> Previously, it was possible to set a processing time timer past the end of a
> window, and it would simply not fire.
> However, now, this results in an error:
> {code:java}
> java.lang.IllegalArgumentException: Attempted to set event time timer that
> outputs for 2020-03-19T18:01:35.000Z but that is after the expiration of
> window 2020-03-19T17:59:59.999Z
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:440)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setAndVerifyOutputTimestamp(SimpleDoFnRunner.java:1011)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setRelative(SimpleDoFnRunner.java:934)
> .processElement(???.scala:187)
> {code}
>
> I think the regression was introduced in commit
> a005fd765a762183ca88df90f261f6d4a20cf3e0. Also notice that the error message
> is wrong, it says that "event time timer" but the timer is in the processing
> time domain.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9605) BIP-1: Rename setRowOption to setOption on Option builder
[ https://issues.apache.org/jira/browse/BEAM-9605?focusedWorklogId=409503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409503 ] ASF GitHub Bot logged work on BEAM-9605: Author: ASF GitHub Bot Created on: 25/Mar/20 13:27 Start Date: 25/Mar/20 13:27 Worklog Time Spent: 10m Work Description: alexvanboxel commented on pull request #11218: [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder URL: https://github.com/apache/beam/pull/11218 Rename setRowOption to setOption on Option builder as setRowOption name is too confusing. It sets an option as a Row, not an option on a Row. Using setOption is better and doesn't conflict with the other setOption with 3 parameters and explicit type. This commit also removed left over @Nullable annotations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409503) Remaining Estimate: 0h Time Spent: 10m > BIP-1: Rename setRowOption to setOption on Option builder > -- > > Key: BEAM-9605 > URL: https://issues.apache.org/jira/browse/BEAM-9605 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Rename setRowOption to setOption on Option builder as setRowOption name is > too confusing. > It sets an option as a Row, not an option on a Row. Using setOption is better > and doesn't conflict with the other setOption with 3 parameters and explicit > type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9550) beam_PostCommit_Python_Chicago_Taxi_Flink OOM
[ https://issues.apache.org/jira/browse/BEAM-9550?focusedWorklogId=409506&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409506 ] ASF GitHub Bot logged work on BEAM-9550: Author: ASF GitHub Bot Created on: 25/Mar/20 13:31 Start Date: 25/Mar/20 13:31 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors. URL: https://github.com/apache/beam/pull/11193#issuecomment-603840955 Run Load Tests Python CoGBK Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409506) Time Spent: 1h 50m (was: 1h 40m) > beam_PostCommit_Python_Chicago_Taxi_Flink OOM > - > > Key: BEAM-9550 > URL: https://issues.apache.org/jira/browse/BEAM-9550 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kyle Weaver >Assignee: Kamil Wasilewski >Priority: Major > Labels: currently-failing > Time Spent: 1h 50m > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/ > The following error has been occurring consistently for several days: > 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace > 07:57:27 Traceback (most recent call last): > 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in > 07:57:27 main() > 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main > 07:57:27 project=known_args.metric_reporting_project) > 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats > 07:57:27 result.wait_until_finish() > 07:57:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 545, in wait_until_finish > 07:57:27 (self._job_id, self._state, self._last_error_message())) > 07:57:27 RuntimeError: Pipeline > chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb > failed in state FAILED: java.lang.OutOfMemoryError: Metaspace -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9550) beam_PostCommit_Python_Chicago_Taxi_Flink OOM
[ https://issues.apache.org/jira/browse/BEAM-9550?focusedWorklogId=409507&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409507 ] ASF GitHub Bot logged work on BEAM-9550: Author: ASF GitHub Bot Created on: 25/Mar/20 13:32 Start Date: 25/Mar/20 13:32 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors. URL: https://github.com/apache/beam/pull/11193#issuecomment-603841123 Run Load Tests Python GBK Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409507) Time Spent: 2h (was: 1h 50m) > beam_PostCommit_Python_Chicago_Taxi_Flink OOM > - > > Key: BEAM-9550 > URL: https://issues.apache.org/jira/browse/BEAM-9550 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kyle Weaver >Assignee: Kamil Wasilewski >Priority: Major > Labels: currently-failing > Time Spent: 2h > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/ > The following error has been occurring consistently for several days: > 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace > 07:57:27 Traceback (most recent call last): > 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in > 07:57:27 main() > 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main > 07:57:27 project=known_args.metric_reporting_project) > 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats > 07:57:27 result.wait_until_finish() > 07:57:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 545, in wait_until_finish > 07:57:27 (self._job_id, self._state, self._last_error_message())) > 07:57:27 RuntimeError: Pipeline > chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb > failed in state FAILED: java.lang.OutOfMemoryError: Metaspace -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9550) beam_PostCommit_Python_Chicago_Taxi_Flink OOM
[ https://issues.apache.org/jira/browse/BEAM-9550?focusedWorklogId=409508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409508 ] ASF GitHub Bot logged work on BEAM-9550: Author: ASF GitHub Bot Created on: 25/Mar/20 13:32 Start Date: 25/Mar/20 13:32 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors. URL: https://github.com/apache/beam/pull/11193#issuecomment-603841228 Run Python Load Tests ParDo Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409508) Time Spent: 2h 10m (was: 2h) > beam_PostCommit_Python_Chicago_Taxi_Flink OOM > - > > Key: BEAM-9550 > URL: https://issues.apache.org/jira/browse/BEAM-9550 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kyle Weaver >Assignee: Kamil Wasilewski >Priority: Major > Labels: currently-failing > Time Spent: 2h 10m > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/ > The following error has been occurring consistently for several days: > 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace > 07:57:27 Traceback (most recent call last): > 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in > 07:57:27 main() > 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main > 07:57:27 project=known_args.metric_reporting_project) > 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats > 07:57:27 result.wait_until_finish() > 07:57:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 545, in wait_until_finish > 07:57:27 (self._job_id, self._state, self._last_error_message())) > 07:57:27 RuntimeError: Pipeline > chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb > failed in state FAILED: java.lang.OutOfMemoryError: Metaspace -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[ https://issues.apache.org/jira/browse/BEAM-9420?focusedWorklogId=409510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409510 ] ASF GitHub Bot logged work on BEAM-9420: Author: ASF GitHub Bot Created on: 25/Mar/20 13:40 Start Date: 25/Mar/20 13:40 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on pull request #11099: [BEAM-9420] Configurable timeout for blocking kafka API call(s) URL: https://github.com/apache/beam/pull/11099 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409510) Time Spent: 1h 50m (was: 1h 40m) > Configurable timeout for Kafka setupInitialOffset() > --- > > Key: BEAM-9420 > URL: https://issues.apache.org/jira/browse/BEAM-9420 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.19.0 >Reporter: Jozef Vilcek >Assignee: Jozef Vilcek >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > If bootstrap brokers does contain an unhealthy server, it can break the start > of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for > `setupInitialOffset()`. Wait timeout is either a double time of `request. > timeout.ms` or some default constant. In both cases, it might not be enough > time for kafka-client to initiate fallback and retry metadata discovery via > another broker from given bootstrap list. > The client should be able to specify timeout for `setupInitialOffset()` > explicitly as a setting to KafkaIO read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[ https://issues.apache.org/jira/browse/BEAM-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9420. Fix Version/s: 2.21.0 Resolution: Fixed > Configurable timeout for Kafka setupInitialOffset() > --- > > Key: BEAM-9420 > URL: https://issues.apache.org/jira/browse/BEAM-9420 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.19.0 >Reporter: Jozef Vilcek >Assignee: Jozef Vilcek >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > If bootstrap brokers does contain an unhealthy server, it can break the start > of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for > `setupInitialOffset()`. Wait timeout is either a double time of `request. > timeout.ms` or some default constant. In both cases, it might not be enough > time for kafka-client to initiate fallback and retry metadata discovery via > another broker from given bootstrap list. > The client should be able to specify timeout for `setupInitialOffset()` > explicitly as a setting to KafkaIO read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9605) BIP-1: Rename setRowOption to setOption on Option builder
[ https://issues.apache.org/jira/browse/BEAM-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel resolved BEAM-9605. -- Fix Version/s: 2.21.0 Resolution: Fixed > BIP-1: Rename setRowOption to setOption on Option builder > -- > > Key: BEAM-9605 > URL: https://issues.apache.org/jira/browse/BEAM-9605 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Fix For: 2.21.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Rename setRowOption to setOption on Option builder as setRowOption name is > too confusing. > It sets an option as a Row, not an option on a Row. Using setOption is better > and doesn't conflict with the other setOption with 3 parameters and explicit > type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9605) BIP-1: Rename setRowOption to setOption on Option builder
[ https://issues.apache.org/jira/browse/BEAM-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Van Boxel updated BEAM-9605: - Status: Open (was: Triage Needed) > BIP-1: Rename setRowOption to setOption on Option builder > -- > > Key: BEAM-9605 > URL: https://issues.apache.org/jira/browse/BEAM-9605 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Rename setRowOption to setOption on Option builder as setRowOption name is > too confusing. > It sets an option as a Row, not an option on a Row. Using setOption is better > and doesn't conflict with the other setOption with 3 parameters and explicit > type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9133) CassandraIOTest.classMethod test is still flaky
[
https://issues.apache.org/jira/browse/BEAM-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066687#comment-17066687
]
Alexey Romanenko commented on BEAM-9133:
Is there any update on this? It's still flaky:
[https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1924/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/]
> CassandraIOTest.classMethod test is still flaky
> ---
>
> Key: BEAM-9133
> URL: https://issues.apache.org/jira/browse/BEAM-9133
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, test-failures
>Affects Versions: 2.17.0
>Reporter: Alexey Romanenko
>Assignee: Etienne Chauchot
>Priority: Critical
>
> CassandraIOTest is still flaky. For example:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
> {code}
> Error Message
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> Stacktrace
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Han
[jira] [Resolved] (BEAM-9133) CassandraIOTest.classMethod test is still flaky
[
https://issues.apache.org/jira/browse/BEAM-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Chauchot resolved BEAM-9133.
Fix Version/s: Not applicable
Resolution: Duplicate
[~aromanenko] thanks. I know it is still flaky. That is why I reopened
BEAM-8025 some days ago. So, this ticket is duplicate. Unfortunately I have no
time to work on that right now. Maybe [~adejanovski] ?
> CassandraIOTest.classMethod test is still flaky
> ---
>
> Key: BEAM-9133
> URL: https://issues.apache.org/jira/browse/BEAM-9133
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, test-failures
>Affects Versions: 2.17.0
>Reporter: Alexey Romanenko
>Assignee: Etienne Chauchot
>Priority: Critical
> Fix For: Not applicable
>
>
> CassandraIOTest is still flaky. For example:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
> {code}
> Error Message
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> Stacktrace
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
>
[jira] [Created] (BEAM-9606) Example in gradle's combine test lacks of parameters
Piotr Szuberski created BEAM-9606: - Summary: Example in gradle's combine test lacks of parameters Key: BEAM-9606 URL: https://issues.apache.org/jira/browse/BEAM-9606 Project: Beam Issue Type: Bug Components: sdk-py-core Affects Versions: Not applicable Reporter: Piotr Szuberski Assignee: Piotr Szuberski Fix For: Not applicable The combine test example run with gradle lacks of fanout and top_count params -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9606) Example in gradle's combine test lacks of parameters
[ https://issues.apache.org/jira/browse/BEAM-9606?focusedWorklogId=409538&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409538 ] ASF GitHub Bot logged work on BEAM-9606: Author: ASF GitHub Bot Created on: 25/Mar/20 14:23 Start Date: 25/Mar/20 14:23 Worklog Time Spent: 10m Work Description: piotr-szuberski commented on pull request #11219: [BEAM-9606] Add missing parameters in combine_test.py gradle example URL: https://github.com/apache/beam/pull/11219 @kamilwu Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/
[jira] [Work logged] (BEAM-8949) Add Spanner IO Integration Test for Python
[ https://issues.apache.org/jira/browse/BEAM-8949?focusedWorklogId=409580&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409580 ] ASF GitHub Bot logged work on BEAM-8949: Author: ASF GitHub Bot Created on: 25/Mar/20 15:25 Start Date: 25/Mar/20 15:25 Worklog Time Spent: 10m Work Description: iemejia commented on issue #11210: [WIP] [BEAM-8949] SpannerIO Read integration tests URL: https://github.com/apache/beam/pull/11210#issuecomment-603904688 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409580) Time Spent: 40m (was: 0.5h) > Add Spanner IO Integration Test for Python > -- > > Key: BEAM-8949 > URL: https://issues.apache.org/jira/browse/BEAM-8949 > Project: Beam > Issue Type: Test > Components: io-py-gcp >Reporter: Shoaib Zafar >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Spanner IO (Python SDK) contains PTransform which uses the BatchAPI to read > from the spanner. Currently, it only contains direct runner unit tests. In > order to make this functionality available for the users, integration tests > also need to be added. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8949) Add Spanner IO Integration Test for Python
[ https://issues.apache.org/jira/browse/BEAM-8949?focusedWorklogId=409579&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409579 ] ASF GitHub Bot logged work on BEAM-8949: Author: ASF GitHub Bot Created on: 25/Mar/20 15:25 Start Date: 25/Mar/20 15:25 Worklog Time Spent: 10m Work Description: iemejia commented on issue #11210: [WIP] [BEAM-8949] SpannerIO Read integration tests URL: https://github.com/apache/beam/pull/11210#issuecomment-603904530 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409579) Time Spent: 0.5h (was: 20m) > Add Spanner IO Integration Test for Python > -- > > Key: BEAM-8949 > URL: https://issues.apache.org/jira/browse/BEAM-8949 > Project: Beam > Issue Type: Test > Components: io-py-gcp >Reporter: Shoaib Zafar >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spanner IO (Python SDK) contains PTransform which uses the BatchAPI to read > from the spanner. Currently, it only contains direct runner unit tests. In > order to make this functionality available for the users, integration tests > also need to be added. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8949) Add Spanner IO Integration Test for Python
[ https://issues.apache.org/jira/browse/BEAM-8949?focusedWorklogId=409581&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409581 ] ASF GitHub Bot logged work on BEAM-8949: Author: ASF GitHub Bot Created on: 25/Mar/20 15:25 Start Date: 25/Mar/20 15:25 Worklog Time Spent: 10m Work Description: iemejia commented on issue #11210: [WIP] [BEAM-8949] SpannerIO Read integration tests URL: https://github.com/apache/beam/pull/11210#issuecomment-603904688 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409581) Time Spent: 50m (was: 40m) > Add Spanner IO Integration Test for Python > -- > > Key: BEAM-8949 > URL: https://issues.apache.org/jira/browse/BEAM-8949 > Project: Beam > Issue Type: Test > Components: io-py-gcp >Reporter: Shoaib Zafar >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Spanner IO (Python SDK) contains PTransform which uses the BatchAPI to read > from the spanner. Currently, it only contains direct runner unit tests. In > order to make this functionality available for the users, integration tests > also need to be added. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9562) Remove timer from PCollection and treat timers as Elements
[
https://issues.apache.org/jira/browse/BEAM-9562?focusedWorklogId=409591&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409591
]
ASF GitHub Bot logged work on BEAM-9562:
Author: ASF GitHub Bot
Created on: 25/Mar/20 15:29
Start Date: 25/Mar/20 15:29
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11216: [BEAM-9562]
Remove TimerSpec from ParDoPayload
URL: https://github.com/apache/beam/pull/11216#discussion_r397940099
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -442,35 +442,30 @@ message ParDoPayload {
// be placed in the pipeline requirements.
map state_specs = 4;
- // (Optional) A mapping of local timer names to timer specifications.
- // If this is set, the stateful processing requirement should also
- // be placed in the pipeline requirements.
- map timer_specs = 5;
-
- // (Optional) A mapping of local timer family names to timer specifications.
- // If this is set, the stateful processing requirement should also
- // be placed in the pipeline requirements.
- map timer_family_specs = 9;
+ // (Optional) A mapping of local timer family names to timer family
+ // specifications. If this is set, the stateful processing requirement should
+ // also be placed in the pipeline requirements.
+ map timer_family_specs = 8;
Review comment:
I would suggest not changing the proto tag numbers otherwise you'll have a
more difficult import/update issue with Google.
It will be much easier as a separate change to renumber them in increasing
order, so timer_family_specs is 5, restriction_coder_id is 6,
requests_finalization is 7, ... (making the import into Google much simpler to
do).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409591)
Time Spent: 2h 40m (was: 2.5h)
> Remove timer from PCollection and treat timers as Elements
> ---
>
> Key: BEAM-9562
> URL: https://issues.apache.org/jira/browse/BEAM-9562
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-harness
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9562) Remove timer from PCollection and treat timers as Elements
[ https://issues.apache.org/jira/browse/BEAM-9562?focusedWorklogId=409589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409589 ] ASF GitHub Bot logged work on BEAM-9562: Author: ASF GitHub Bot Created on: 25/Mar/20 15:29 Start Date: 25/Mar/20 15:29 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11216: [BEAM-9562] Remove TimerSpec from ParDoPayload URL: https://github.com/apache/beam/pull/11216#discussion_r397942726 ## File path: sdks/python/apache_beam/transforms/userstate.py ## @@ -141,6 +143,13 @@ def __init__(self, name, time_domain): def __repr__(self): return '%s(%s)' % (self.__class__.__name__, self.name) + def to_timer_family_runner_api(self, context): Review comment: shouldn't this method replace `to_runner_api`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409589) Time Spent: 2.5h (was: 2h 20m) > Remove timer from PCollection and treat timers as Elements > --- > > Key: BEAM-9562 > URL: https://issues.apache.org/jira/browse/BEAM-9562 > Project: Beam > Issue Type: New Feature > Components: sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9562) Remove timer from PCollection and treat timers as Elements
[ https://issues.apache.org/jira/browse/BEAM-9562?focusedWorklogId=409590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409590 ] ASF GitHub Bot logged work on BEAM-9562: Author: ASF GitHub Bot Created on: 25/Mar/20 15:29 Start Date: 25/Mar/20 15:29 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11216: [BEAM-9562] Remove TimerSpec from ParDoPayload URL: https://github.com/apache/beam/pull/11216#discussion_r397943829 ## File path: sdks/python/apache_beam/transforms/userstate.py ## @@ -149,6 +158,26 @@ def to_runner_api(self, context): coders._TimerCoder(coders.SingletonCoder(None +# TODO(BEAM-9602): Provide support of dynamic timer Review comment: ```suggestion # TODO(BEAM-9602): Provide support for dynamic timer ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409590) Time Spent: 2h 40m (was: 2.5h) > Remove timer from PCollection and treat timers as Elements > --- > > Key: BEAM-9562 > URL: https://issues.apache.org/jira/browse/BEAM-9562 > Project: Beam > Issue Type: New Feature > Components: sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2546) Create InfluxDbIO
[
https://issues.apache.org/jira/browse/BEAM-2546?focusedWorklogId=409603&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409603
]
ASF GitHub Bot logged work on BEAM-2546:
Author: ASF GitHub Bot
Created on: 25/Mar/20 15:49
Start Date: 25/Mar/20 15:49
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #11028: BEAM-2546
Beam IO for InfluxDB
URL: https://github.com/apache/beam/pull/11028#discussion_r397962741
##
File path: .test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import CommonJobProperties as common
+import Kubernetes
+
+String jobName = "beam_PerformanceTests_InfluxDBIO_IT"
+
+job(jobName) {
+ common.setTopLevelMainJobProperties(delegate)
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Java InfluxDBIO Performance Test',
+ 'Run Java InfluxDBIO Performance Test')
+
+ String namespace = common.getKubernetesNamespace(jobName)
+ String kubeconfigPath = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfigPath, namespace)
+
+
k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/influxdb/influxdb.yml"))
+ String influxDBHostName = "LOAD_BALANCER_IP"
+ k8s.loadBalancerIP("influxdb-load-balancer-service", influxDBHostName)
+ Map pipelineOptions = [
+ influxDBURL : "http://\$${influxDBHostName}:8086";,
+ influxDBUserName : "superadmin",
+ influxDBPassword : "supersecretpassword",
+ databaseName : "db1"
+ ]
+
+ steps {
+gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+
switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=direct")
Review comment:
After looking a bit into the code we have a weird mix of Dataflow/Direct in
the runs, do you think there is a way we can do that 'choosable' when running
an IT test? (note that I agree with the idea of running on Dataflow, but of
course I understand contributors may not have access to it and it is quicker to
develop/do it with DirectRunner).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409603)
Time Spent: 9h 10m (was: 9h)
> Create InfluxDbIO
> -
>
> Key: BEAM-2546
> URL: https://issues.apache.org/jira/browse/BEAM-2546
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Bipin Upadhyaya
>Priority: Major
> Fix For: 2.21.0
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9028) bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
[
https://issues.apache.org/jira/browse/BEAM-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066781#comment-17066781
]
Kamil Wasilewski commented on BEAM-9028:
_bigquery_read_it_test_ seems to be pretty stable since January. [~udim] I'll
close this ticket if you don't mind.
> bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
> --
>
> Key: BEAM-9028
> URL: https://issues.apache.org/jira/browse/BEAM-9028
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Kamil Wasilewski
>Priority: Major
>
> It's not clear what the root cause is (there's also a "cannot allocate
> memory" error message).
> {code}
> 14:16:06 ERROR: test_iobase_source
> (apache_beam.io.gcp.bigquery_read_it_test.ReadTests)
> 14:16:06
> --
> 14:16:06 Traceback (most recent call last):
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py",
> line 154, in test_iobase_source
> 14:16:06 assert_that(result, equal_to(self.TABLE_DATA))
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 480, in __exit__
> 14:16:06 self.run().wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 460, in run
> 14:16:06 self._options).run(False)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 473, in run
> 14:16:06 return self.runner.run_pipeline(self, self._options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/flink_runner.py",
> line 45, in run_pipeline
> 14:16:06 return super(FlinkRunner, self).run_pipeline(pipeline, options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 330, in run_pipeline
> 14:16:06 result.wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 438, in wait_until_finish
> 14:16:06 for state_response in self._state_stream:
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 416, in __next__
> 14:16:06 return self._next()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 703, in _next
> 14:16:06 raise self
> 14:16:06 grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous
> of RPC that terminated with:
> 14:16:06 status = StatusCode.UNAVAILABLE
> 14:16:06 details = "Socket closed"
> 14:16:06 debug_error_string =
> "{"created":"@1577139361.436477926","description":"Error received from peer
> ipv4:127.0.0.1:51557","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket
> closed","grpc_status":14}"
> 14:16:06 >
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1210/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9606) Example in gradle's combine test lacks of parameters
[ https://issues.apache.org/jira/browse/BEAM-9606?focusedWorklogId=409608&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409608 ] ASF GitHub Bot logged work on BEAM-9606: Author: ASF GitHub Bot Created on: 25/Mar/20 16:03 Start Date: 25/Mar/20 16:03 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11219: [BEAM-9606] Add missing parameters in combine_test.py gradle example URL: https://github.com/apache/beam/pull/11219#issuecomment-603927193 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409608) Time Spent: 20m (was: 10m) > Example in gradle's combine test lacks of parameters > > > Key: BEAM-9606 > URL: https://issues.apache.org/jira/browse/BEAM-9606 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Affects Versions: Not applicable >Reporter: Piotr Szuberski >Assignee: Piotr Szuberski >Priority: Minor > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > The combine test example run with gradle lacks of fanout and top_count params -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=409615&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409615 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 25/Mar/20 16:13 Start Date: 25/Mar/20 16:13 Worklog Time Spent: 10m Work Description: udim commented on issue #11212: [BEAM-8280] Type hints via annotations snippets URL: https://github.com/apache/beam/pull/11212#issuecomment-603933281 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409615) Time Spent: 10h 40m (was: 10.5h) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7896) Rate estimation for Kafka Table
[ https://issues.apache.org/jira/browse/BEAM-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-7896. --- Fix Version/s: 2.16.0 Resolution: Fixed > Rate estimation for Kafka Table > --- > > Key: BEAM-7896 > URL: https://issues.apache.org/jira/browse/BEAM-7896 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Alireza Samadianzakaria >Assignee: Alireza Samadianzakaria >Priority: Major > Fix For: 2.16.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Currently, KafkaTable returns UNKNOWN statistics for its rate. > We can use previously arrived tuples to estimate the rate and return correct > statistics (See > [https://docs.google.com/document/d/1vi1PBBu5IqSy-qZl1Gk-49CcANOpbNs1UAud6LnOaiY|https://docs.google.com/document/d/1vi1PBBu5IqSy-qZl1Gk-49CcANOpbNs1UAud6LnOaiY/]) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9028) bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
[
https://issues.apache.org/jira/browse/BEAM-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066802#comment-17066802
]
Udi Meiri commented on BEAM-9028:
-
Sure, np
> bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
> --
>
> Key: BEAM-9028
> URL: https://issues.apache.org/jira/browse/BEAM-9028
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Kamil Wasilewski
>Priority: Major
>
> It's not clear what the root cause is (there's also a "cannot allocate
> memory" error message).
> {code}
> 14:16:06 ERROR: test_iobase_source
> (apache_beam.io.gcp.bigquery_read_it_test.ReadTests)
> 14:16:06
> --
> 14:16:06 Traceback (most recent call last):
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py",
> line 154, in test_iobase_source
> 14:16:06 assert_that(result, equal_to(self.TABLE_DATA))
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 480, in __exit__
> 14:16:06 self.run().wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 460, in run
> 14:16:06 self._options).run(False)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 473, in run
> 14:16:06 return self.runner.run_pipeline(self, self._options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/flink_runner.py",
> line 45, in run_pipeline
> 14:16:06 return super(FlinkRunner, self).run_pipeline(pipeline, options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 330, in run_pipeline
> 14:16:06 result.wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 438, in wait_until_finish
> 14:16:06 for state_response in self._state_stream:
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 416, in __next__
> 14:16:06 return self._next()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 703, in _next
> 14:16:06 raise self
> 14:16:06 grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous
> of RPC that terminated with:
> 14:16:06 status = StatusCode.UNAVAILABLE
> 14:16:06 details = "Socket closed"
> 14:16:06 debug_error_string =
> "{"created":"@1577139361.436477926","description":"Error received from peer
> ipv4:127.0.0.1:51557","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket
> closed","grpc_status":14}"
> 14:16:06 >
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1210/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-4628) Flaky post-commit test org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq
[ https://issues.apache.org/jira/browse/BEAM-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-4628. --- Fix Version/s: Not applicable Resolution: Fixed > Flaky post-commit test > org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq > --- > > Key: BEAM-4628 > URL: https://issues.apache.org/jira/browse/BEAM-4628 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Mikhail Gryzykhin >Assignee: Anton Kedin >Priority: Major > Labels: flake, sickbay > Fix For: Not applicable > > > As part of code lime we want to fix existing flaky tests. This test changed > state in 4 jobs with no code changes during ~2 days. > Link to the test history: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/844/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/|https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/844/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/?start=25] > Links to the jobs where test changed state: > ||Url|| > |[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/827/] > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/854/] > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/867/] > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/844/| > Job fails with timeout of 5 minutes, while usual runtime is around 70 > seconds. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-4161) Nested Rows flattening doesn't work
[ https://issues.apache.org/jira/browse/BEAM-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4161: - Assignee: Rui Wang (was: Anton Kedin) > Nested Rows flattening doesn't work > --- > > Key: BEAM-4161 > URL: https://issues.apache.org/jira/browse/BEAM-4161 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Rui Wang >Priority: Major > > Calcite flattens nested rows. It updates the field indices of the flattened > row so the fields are referenced correctly in the Rel Nodes. But the fields > after the flattened row don't have the indices updated, they have the > previous ordinals before the flattening. There is no way to look up the > correct index at the point when it reaches Beam SQL Rel Nodes. It will be > fixed in Calcite 1.17. > We need to update the Calcite as soon as it is released and add few > integration tests around nested Rows: > - basic nesting with fields before and after the row field; > - multi-level row nesting; > - multiple row fields; > > Calcite JIRA: CALCITE-2220 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-3694) [SQL] Update SQL documentation
[ https://issues.apache.org/jira/browse/BEAM-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-3694. --- Fix Version/s: Not applicable Resolution: Fixed > [SQL] Update SQL documentation > -- > > Key: BEAM-3694 > URL: https://issues.apache.org/jira/browse/BEAM-3694 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Anton Kedin >Priority: Major > Fix For: Not applicable > > > Update / add: > * windowing inheritance; > * HOP parameters swap; > * datetime functions; > * joins behavior; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4161) Nested Rows flattening doesn't work
[ https://issues.apache.org/jira/browse/BEAM-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066807#comment-17066807 ] Kenneth Knowles commented on BEAM-4161: --- Resolved, right? > Nested Rows flattening doesn't work > --- > > Key: BEAM-4161 > URL: https://issues.apache.org/jira/browse/BEAM-4161 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Rui Wang >Priority: Major > > Calcite flattens nested rows. It updates the field indices of the flattened > row so the fields are referenced correctly in the Rel Nodes. But the fields > after the flattened row don't have the indices updated, they have the > previous ordinals before the flattening. There is no way to look up the > correct index at the point when it reaches Beam SQL Rel Nodes. It will be > fixed in Calcite 1.17. > We need to update the Calcite as soon as it is released and add few > integration tests around nested Rows: > - basic nesting with fields before and after the row field; > - multi-level row nesting; > - multiple row fields; > > Calcite JIRA: CALCITE-2220 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-3481) Query with subquery and aggregates cannot be implemented.
[
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-3481:
-
Assignee: (was: Anton Kedin)
> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
> FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
> FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>
> {noformat}
> [nREPL-worker-5] INFO
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
> LogicalAggregate(group=[{0}])
> LogicalProject(p=[$0])
> LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}],
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu,
> 0.0 io}, id = 155
> LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount =
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
> LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
> LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]):
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
> rel#150:Subset#0.NONE.[], best=null, importance=0.6561
> rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0,
> cumulative cost={inf}
> rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
> rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0,
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
> rel#152:Subset#1.NONE.[], best=null, importance=0.7291
> rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0),
> rowcount=100.0, cumulative cost={inf}
> rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163,
> importance=0.36455
>
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
> rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
> rel#154:Subset#2.NONE.[], best=null, importance=0.81
> rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}),
> rowcount=10.0, cumulative cost={inf}
> rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
> rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
> rel#156:Subset#3.NONE.[], best=null, importance=0.9
>
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
> rowcount=1.0, cumulative cost={inf}
> rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
> importance=1.0
>
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
> rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
> (RelSubset.java:441)
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-3481) Query with subquery and aggregates cannot be implemented.
[
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066809#comment-17066809
]
Kenneth Knowles commented on BEAM-3481:
---
It has been a while, and a lot of changes - is this still a problem?
> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
> FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
> FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>
> {noformat}
> [nREPL-worker-5] INFO
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
> LogicalAggregate(group=[{0}])
> LogicalProject(p=[$0])
> LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}],
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu,
> 0.0 io}, id = 155
> LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount =
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
> LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
> LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]):
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
> rel#150:Subset#0.NONE.[], best=null, importance=0.6561
> rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0,
> cumulative cost={inf}
> rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
> rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0,
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
> rel#152:Subset#1.NONE.[], best=null, importance=0.7291
> rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0),
> rowcount=100.0, cumulative cost={inf}
> rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163,
> importance=0.36455
>
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
> rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
> rel#154:Subset#2.NONE.[], best=null, importance=0.81
> rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}),
> rowcount=10.0, cumulative cost={inf}
> rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
> rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
> rel#156:Subset#3.NONE.[], best=null, importance=0.9
>
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
> rowcount=1.0, cumulative cost={inf}
> rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
> importance=1.0
>
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
> rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
> (RelSubset.java:441)
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-3647) Default Coder/Reading Coder From File
[ https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3647: - Assignee: Reuven Lax (was: Anton Kedin) > Default Coder/Reading Coder From File > -- > > Key: BEAM-3647 > URL: https://issues.apache.org/jira/browse/BEAM-3647 > Project: Beam > Issue Type: New Feature > Components: beam-model, dsl-sql >Affects Versions: 2.2.0 >Reporter: Kishan Kumar >Assignee: Reuven Lax >Priority: Major > > *Requirement*-: Need to Run Template With Same Logics on Different Tables > Data.(Example is Given Below) > > *Need*: Default Coder is Required So According to Data It Make All Fields as > String and Read Data else Thier must be Dynamic Options to Read Coder From > GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location > Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider. > > > *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, > SUB_PRICE) > And 2 Table is (NAME, ROLL, SUB, TEST_MARKS) > > On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can > Read Coder at Run Time The Same Template Can Be Used For Different Tables at > Run Time. > > Such Situations Make Our Work Easy and Make Our job Easy. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3647) Default Coder/Reading Coder From File
[ https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-3647: -- Component/s: (was: dsl-sql) > Default Coder/Reading Coder From File > -- > > Key: BEAM-3647 > URL: https://issues.apache.org/jira/browse/BEAM-3647 > Project: Beam > Issue Type: New Feature > Components: beam-model >Affects Versions: 2.2.0 >Reporter: Kishan Kumar >Assignee: Reuven Lax >Priority: Major > > *Requirement*-: Need to Run Template With Same Logics on Different Tables > Data.(Example is Given Below) > > *Need*: Default Coder is Required So According to Data It Make All Fields as > String and Read Data else Thier must be Dynamic Options to Read Coder From > GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location > Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider. > > > *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, > SUB_PRICE) > And 2 Table is (NAME, ROLL, SUB, TEST_MARKS) > > On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can > Read Coder at Run Time The Same Template Can Be Used For Different Tables at > Run Time. > > Such Situations Make Our Work Easy and Make Our job Easy. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3647) Default Coder/Reading Coder From File
[ https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066812#comment-17066812 ] Kenneth Knowles commented on BEAM-3647: --- Is all the new schema support helping this? > Default Coder/Reading Coder From File > -- > > Key: BEAM-3647 > URL: https://issues.apache.org/jira/browse/BEAM-3647 > Project: Beam > Issue Type: New Feature > Components: beam-model >Affects Versions: 2.2.0 >Reporter: Kishan Kumar >Assignee: Reuven Lax >Priority: Major > > *Requirement*-: Need to Run Template With Same Logics on Different Tables > Data.(Example is Given Below) > > *Need*: Default Coder is Required So According to Data It Make All Fields as > String and Read Data else Thier must be Dynamic Options to Read Coder From > GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location > Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider. > > > *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, > SUB_PRICE) > And 2 Table is (NAME, ROLL, SUB, TEST_MARKS) > > On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can > Read Coder at Run Time The Same Template Can Be Used For Different Tables at > Run Time. > > Such Situations Make Our Work Easy and Make Our job Easy. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9028) bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
[
https://issues.apache.org/jira/browse/BEAM-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Wasilewski resolved BEAM-9028.
Fix Version/s: Not applicable
Resolution: Not A Problem
> bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
> --
>
> Key: BEAM-9028
> URL: https://issues.apache.org/jira/browse/BEAM-9028
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Kamil Wasilewski
>Priority: Major
> Fix For: Not applicable
>
>
> It's not clear what the root cause is (there's also a "cannot allocate
> memory" error message).
> {code}
> 14:16:06 ERROR: test_iobase_source
> (apache_beam.io.gcp.bigquery_read_it_test.ReadTests)
> 14:16:06
> --
> 14:16:06 Traceback (most recent call last):
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py",
> line 154, in test_iobase_source
> 14:16:06 assert_that(result, equal_to(self.TABLE_DATA))
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 480, in __exit__
> 14:16:06 self.run().wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 460, in run
> 14:16:06 self._options).run(False)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 473, in run
> 14:16:06 return self.runner.run_pipeline(self, self._options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/flink_runner.py",
> line 45, in run_pipeline
> 14:16:06 return super(FlinkRunner, self).run_pipeline(pipeline, options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 330, in run_pipeline
> 14:16:06 result.wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 438, in wait_until_finish
> 14:16:06 for state_response in self._state_stream:
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 416, in __next__
> 14:16:06 return self._next()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 703, in _next
> 14:16:06 raise self
> 14:16:06 grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous
> of RPC that terminated with:
> 14:16:06 status = StatusCode.UNAVAILABLE
> 14:16:06 details = "Socket closed"
> 14:16:06 debug_error_string =
> "{"created":"@1577139361.436477926","description":"Error received from peer
> ipv4:127.0.0.1:51557","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket
> closed","grpc_status":14}"
> 14:16:06 >
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1210/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-2281) call SqlFunctions in operator implementation
[
https://issues.apache.org/jira/browse/BEAM-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-2281.
---
Fix Version/s: 2.10.0
Assignee: Andrew Pilloud (was: Anton Kedin)
Resolution: Fixed
> call SqlFunctions in operator implementation
>
>
> Key: BEAM-2281
> URL: https://issues.apache.org/jira/browse/BEAM-2281
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Mingmin Xu
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Calcite has a collections of functions in
> {{org.apache.calcite.runtime.SqlFunctions}}. It sounds a good source to
> leverage when adding operators as {{BeamSqlExpression}}.
> [~xumingming] [~app-tarush], any comments?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5130) Support ALL (sub query)
[ https://issues.apache.org/jira/browse/BEAM-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066820#comment-17066820 ] Kenneth Knowles commented on BEAM-5130: --- Hi! I am checking on SQL JIRAs. Is this still an issue? Are you working on it? We can unassign and let someone else work on it. Also the parent issue (just a container issue that could be a label instead) is closed so I will detach this sub-issue. > Support ALL (sub query) > --- > > Key: BEAM-5130 > URL: https://issues.apache.org/jira/browse/BEAM-5130 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > Example: > > SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM > special_item_table). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5130) Support ALL (sub query)
[ https://issues.apache.org/jira/browse/BEAM-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5130: -- Parent: (was: BEAM-5046) Issue Type: New Feature (was: Sub-task) > Support ALL (sub query) > --- > > Key: BEAM-5130 > URL: https://issues.apache.org/jira/browse/BEAM-5130 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > Example: > > SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM > special_item_table). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5111) SUM0/SUM
[ https://issues.apache.org/jira/browse/BEAM-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066825#comment-17066825 ] Kenneth Knowles commented on BEAM-5111: --- Hi! I am checking on SQL JIRAs. Is this still an issue? Are you working on it? We can unassign and let someone else work on it. Also the parent issue (just a container issue that could be a label instead) is closed so I will detach this sub-issue. > SUM0/SUM > > > Key: BEAM-5111 > URL: https://issues.apache.org/jira/browse/BEAM-5111 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > SUM and SUM0 share the same code, either one could be wrong. Should fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4561) Create unit tests for BeamQueryPlanner.convertToBeamRel
[ https://issues.apache.org/jira/browse/BEAM-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066827#comment-17066827 ] Kenneth Knowles commented on BEAM-4561: --- Hi! I am checking on SQL JIRAs. Is this still an issue? Are you working on it? For this one maybe we can close it since it is just an idea that we might do something. > Create unit tests for BeamQueryPlanner.convertToBeamRel > --- > > Key: BEAM-4561 > URL: https://issues.apache.org/jira/browse/BEAM-4561 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > > As discussion on PR#5481, we should consider a concrete unit test for > [BeamQueryPlanner.convertToBeamRel|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L116] > when apply BeamRuleSets to optimize logical plan. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3558) aggregation expression can't apply to math or arithmetic expressions
[ https://issues.apache.org/jira/browse/BEAM-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066830#comment-17066830 ] Kenneth Knowles commented on BEAM-3558: --- I think I know that this is still an issue. Are you working on it? We can unassign and let someone else work on it. Also I will detach this sub-issue. > aggregation expression can't apply to math or arithmetic expressions > > > Key: BEAM-3558 > URL: https://issues.apache.org/jira/browse/BEAM-3558 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kai Jiang >Priority: Major > > fails when executing sql > 'select sum(c1)+2 from PCOLLECTION group by c2' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3558) aggregation expression can't apply to math or arithmetic expressions
[ https://issues.apache.org/jira/browse/BEAM-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-3558: -- Parent: (was: BEAM-3517) Issue Type: Improvement (was: Sub-task) > aggregation expression can't apply to math or arithmetic expressions > > > Key: BEAM-3558 > URL: https://issues.apache.org/jira/browse/BEAM-3558 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kai Jiang >Priority: Major > > fails when executing sql > 'select sum(c1)+2 from PCOLLECTION group by c2' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-3517) Support Built-in Aggregation Functions
[ https://issues.apache.org/jira/browse/BEAM-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-3517. --- Fix Version/s: Not applicable Resolution: Fixed No longer using the bucket task but just tracking bugs individually. > Support Built-in Aggregation Functions > -- > > Key: BEAM-3517 > URL: https://issues.apache.org/jira/browse/BEAM-3517 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: Not applicable > > > Support UDAF listed in Calcite. > https://calcite.apache.org/docs/reference.html#aggregate-functions -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9036) Failure in beam_PostCommit_SQL: testSQLReadAndWriteWithSameFlatTableDefinition
[
https://issues.apache.org/jira/browse/BEAM-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-9036.
---
Fix Version/s: Not applicable
Resolution: Fixed
> Failure in beam_PostCommit_SQL: testSQLReadAndWriteWithSameFlatTableDefinition
> --
>
> Key: BEAM-9036
> URL: https://issues.apache.org/jira/browse/BEAM-9036
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, test-failures
>Reporter: Udi Meiri
>Assignee: Daniel Collins
>Priority: Major
> Fix For: Not applicable
>
>
> Culprit seems to be:
> https://github.com/apache/beam/pull/10331#issuecomment-569309669
> Error 1:
> {code}
> java.lang.AssertionError:
> Expected: iterable with items [hasProperty("payload", is <{name=person1,
> height=80}>), hasProperty("payload", is <{name=person3, height=60}>),
> hasProperty("payload", is <{name=person5, height=40}>)] in any order
> but: no item matches: hasProperty("payload", is <{name=person1,
> height=80}>), hasProperty("payload", is <{name=person3, height=60}>),
> hasProperty("payload", is <{name=person5, height=40}>) in []
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.lambda$assertThatTopicEventuallyReceives$1(TestPubsub.java:262)
> at
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testSQLReadAndWriteWithSameFlatTableDefinition(PubsubJsonIT.java:571)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:101)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:101)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:101)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal$1.evaluate(TestPubsubSignal.java:116)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.Refle
[jira] [Resolved] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug
[ https://issues.apache.org/jira/browse/BEAM-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-9203. --- Fix Version/s: 2.20.0 Resolution: Fixed > Programmatically determine if SQL exception is user error, unsupported, or bug > -- > > Key: BEAM-9203 > URL: https://issues.apache.org/jira/browse/BEAM-9203 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Right now there are a lot exceptions thrown by the Calcite SQL dialect and > ZetaSQL dialect of Beam SQL. It is hard to catch just the errors that are > user errors, or just the errors that are unsupported operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8817) Beam SQL dialects should share table registration and name resolution logic
[ https://issues.apache.org/jira/browse/BEAM-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066840#comment-17066840 ] Kenneth Knowles commented on BEAM-8817: --- I spent some time on this. I wanted them to actually share the main planner. But actually each dialect has some of their own custom rules. So it is more of a fancy project to separate the planners. Calcite also really wants there to be just one planner so sharing logic can be hard. Going to close this as won't fix. > Beam SQL dialects should share table registration and name resolution logic > --- > > Key: BEAM-8817 > URL: https://issues.apache.org/jira/browse/BEAM-8817 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Currently the ZetaSQL and Calcite SQL code paths each have their own version > of "scrape the table names from the query" and "let the table providers know > that we will be asking for these names". There should be a common abstraction > for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8817) Beam SQL dialects should share table registration and name resolution logic
[ https://issues.apache.org/jira/browse/BEAM-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8817. --- Fix Version/s: Not applicable Resolution: Won't Fix > Beam SQL dialects should share table registration and name resolution logic > --- > > Key: BEAM-8817 > URL: https://issues.apache.org/jira/browse/BEAM-8817 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > > Currently the ZetaSQL and Calcite SQL code paths each have their own version > of "scrape the table names from the query" and "let the table providers know > that we will be asking for these names". There should be a common abstraction > for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8592) DataCatalogTableProvider should not squash table components together into a string
[
https://issues.apache.org/jira/browse/BEAM-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-8592.
---
Fix Version/s: 2.18.0
Resolution: Fixed
> DataCatalogTableProvider should not squash table components together into a
> string
> --
>
> Key: BEAM-8592
> URL: https://issues.apache.org/jira/browse/BEAM-8592
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, dsl-sql-zetasql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently, if a user writes a table name like \{{foo.`baz.bar`.bizzle}}
> representing the components \{{"foo", "baz.bar", "bizzle"}} the
> DataCatalogTableProvider will concatenate the components into a string and
> resolve the identifier as if it represented \{{"foo", "baz", "bar",
> "bizzle"}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8359) Move ZetaSQL to Calcite translator to its own module
[ https://issues.apache.org/jira/browse/BEAM-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8359. --- Fix Version/s: 2.17.0 Resolution: Fixed > Move ZetaSQL to Calcite translator to its own module > > > Key: BEAM-8359 > URL: https://issues.apache.org/jira/browse/BEAM-8359 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.17.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This module has protobuf and gRPC on its surface (for now anyhow) and is > totally independent of the core Beam SQL. Moving to its own module will > protect Beam SQL users from the protobuf and gRPC dependencies and clean the > build slightly, removing some Gradle hacks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8445) ZetaSQL translator returns null from unsupported methods
[ https://issues.apache.org/jira/browse/BEAM-8445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8445. --- Fix Version/s: 2.17.0 Resolution: Fixed > ZetaSQL translator returns null from unsupported methods > > > Key: BEAM-8445 > URL: https://issues.apache.org/jira/browse/BEAM-8445 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.17.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This is an error-prone style that was missed because our null analysis is > disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8040) NPE in table name resolver when selecting from a table that doesn't exist
[
https://issues.apache.org/jira/browse/BEAM-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-8040.
---
Fix Version/s: 2.17.0
Resolution: Fixed
> NPE in table name resolver when selecting from a table that doesn't exist
> -
>
> Key: BEAM-8040
> URL: https://issues.apache.org/jira/browse/BEAM-8040
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.17.0
>
>
> NullPointerException when selecting from a table that doesn't exist.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolverImpl.assumeLeafIsTable(TableResolverImpl.java:42)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.resolveCalciteTable(TableResolution.java:48)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.addTableToLeafCatalog(SqlAnalyzer.java:174)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.lambda$createPopulatedCatalog$0(SqlAnalyzer.java:132)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8040) NPE in table name resolver when selecting from a table that doesn't exist
[
https://issues.apache.org/jira/browse/BEAM-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066843#comment-17066843
]
Kenneth Knowles commented on BEAM-8040:
---
I think the PR I just added, plus my work on nulls.
> NPE in table name resolver when selecting from a table that doesn't exist
> -
>
> Key: BEAM-8040
> URL: https://issues.apache.org/jira/browse/BEAM-8040
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kenneth Knowles
>Priority: Major
>
> NullPointerException when selecting from a table that doesn't exist.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolverImpl.assumeLeafIsTable(TableResolverImpl.java:42)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.resolveCalciteTable(TableResolution.java:48)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.addTableToLeafCatalog(SqlAnalyzer.java:174)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.lambda$createPopulatedCatalog$0(SqlAnalyzer.java:132)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7832) ZetaSQL Dialect
[ https://issues.apache.org/jira/browse/BEAM-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7832: -- Component/s: (was: dsl-sql) > ZetaSQL Dialect > --- > > Key: BEAM-7832 > URL: https://issues.apache.org/jira/browse/BEAM-7832 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Mujuzi Moses >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > We can support ZetaSQL(https://github.com/google/zetasql) dialect in BeamSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7832) ZetaSQL Dialect
[ https://issues.apache.org/jira/browse/BEAM-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066845#comment-17066845 ] Kenneth Knowles commented on BEAM-7832: --- Now that we have a JIRA component for the ZetaSQL dialect, I think making the bugs top level is just as good and you can do more with issues when they are top level. > ZetaSQL Dialect > --- > > Key: BEAM-7832 > URL: https://issues.apache.org/jira/browse/BEAM-7832 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Mujuzi Moses >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > We can support ZetaSQL(https://github.com/google/zetasql) dialect in BeamSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9606) Example in gradle's combine test lacks of parameters
[ https://issues.apache.org/jira/browse/BEAM-9606?focusedWorklogId=409637&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409637 ] ASF GitHub Bot logged work on BEAM-9606: Author: ASF GitHub Bot Created on: 25/Mar/20 16:49 Start Date: 25/Mar/20 16:49 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11219: [BEAM-9606] Add missing parameters in combine_test.py gradle example URL: https://github.com/apache/beam/pull/11219#issuecomment-603955446 Looks good, thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409637) Time Spent: 0.5h (was: 20m) > Example in gradle's combine test lacks of parameters > > > Key: BEAM-9606 > URL: https://issues.apache.org/jira/browse/BEAM-9606 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Affects Versions: Not applicable >Reporter: Piotr Szuberski >Assignee: Piotr Szuberski >Priority: Minor > Fix For: Not applicable > > Time Spent: 0.5h > Remaining Estimate: 0h > > The combine test example run with gradle lacks of fanout and top_count params -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9606) Example in gradle's combine test lacks of parameters
[ https://issues.apache.org/jira/browse/BEAM-9606?focusedWorklogId=409639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409639 ] ASF GitHub Bot logged work on BEAM-9606: Author: ASF GitHub Bot Created on: 25/Mar/20 16:50 Start Date: 25/Mar/20 16:50 Worklog Time Spent: 10m Work Description: kamilwu commented on pull request #11219: [BEAM-9606] Add missing parameters in combine_test.py gradle example URL: https://github.com/apache/beam/pull/11219 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409639) Time Spent: 40m (was: 0.5h) > Example in gradle's combine test lacks of parameters > > > Key: BEAM-9606 > URL: https://issues.apache.org/jira/browse/BEAM-9606 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Affects Versions: Not applicable >Reporter: Piotr Szuberski >Assignee: Piotr Szuberski >Priority: Minor > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > The combine test example run with gradle lacks of fanout and top_count params -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-5829) SQL should probably not support GROUP BY or set operations on floating point numbers
[ https://issues.apache.org/jira/browse/BEAM-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-5829. --- Fix Version/s: 2.10.0 Resolution: Fixed > SQL should probably not support GROUP BY or set operations on floating point > numbers > > > Key: BEAM-5829 > URL: https://issues.apache.org/jira/browse/BEAM-5829 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > These are known to be super unreliable on most SQL engines, and generally > indicate a programming error. Floating points numbers are intended as > stand-ins for real numbers, for which equality (hence grouping and set > operations) are undecidable. And our build is broken because of it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5834) Document lack of support for GROUP BY and set operations on floating points in SQL
[ https://issues.apache.org/jira/browse/BEAM-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5834: -- Parent: (was: BEAM-5829) Issue Type: Improvement (was: Sub-task) > Document lack of support for GROUP BY and set operations on floating points > in SQL > -- > > Key: BEAM-5834 > URL: https://issues.apache.org/jira/browse/BEAM-5834 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9562) Remove timer from PCollection and treat timers as Elements
[ https://issues.apache.org/jira/browse/BEAM-9562?focusedWorklogId=409641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409641 ] ASF GitHub Bot logged work on BEAM-9562: Author: ASF GitHub Bot Created on: 25/Mar/20 16:52 Start Date: 25/Mar/20 16:52 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #11216: [BEAM-9562] Remove TimerSpec from ParDoPayload URL: https://github.com/apache/beam/pull/11216#discussion_r398011965 ## File path: sdks/python/apache_beam/transforms/userstate.py ## @@ -141,6 +143,13 @@ def __init__(self, name, time_domain): def __repr__(self): return '%s(%s)' % (self.__class__.__name__, self.name) + def to_timer_family_runner_api(self, context): Review comment: I'm working on the Java changes meanwhile having some python tests to run to catch any breakages. Changing to `to_runner_api` is the next step. We should also remove `TimerSpec` definition from proto as long as nowhere else uses it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409641) Time Spent: 2h 50m (was: 2h 40m) > Remove timer from PCollection and treat timers as Elements > --- > > Key: BEAM-9562 > URL: https://issues.apache.org/jira/browse/BEAM-9562 > Project: Beam > Issue Type: New Feature > Components: sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4561) Create unit tests for BeamQueryPlanner.convertToBeamRel
[ https://issues.apache.org/jira/browse/BEAM-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4561: -- Parent: (was: BEAM-4388) Issue Type: Improvement (was: Sub-task) > Create unit tests for BeamQueryPlanner.convertToBeamRel > --- > > Key: BEAM-4561 > URL: https://issues.apache.org/jira/browse/BEAM-4561 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > > As discussion on PR#5481, we should consider a concrete unit test for > [BeamQueryPlanner.convertToBeamRel|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L116] > when apply BeamRuleSets to optimize logical plan. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-4388) Support optimized logical plan
[ https://issues.apache.org/jira/browse/BEAM-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-4388. --- Fix Version/s: 2.10.0 Resolution: Fixed > Support optimized logical plan > -- > > Key: BEAM-4388 > URL: https://issues.apache.org/jira/browse/BEAM-4388 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: 2.10.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Before converting into Beam Pipeline physical plan, logical plan should be > optimized and it will be super helpful for efficiently executing Beam > PTransforms pipeline. > Calcite has two ways for optimizing logical plan (HepPlanner and > VolcanoPlanner). We can support VolcanoPlanner first and apply calcite > builtin optimize rules (like > FilterJoinRule.FILTER_ON_JOIN) to sql query optimize plans. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9398) Python type hints: AbstractDoFnWrapper does not wrap setup
[ https://issues.apache.org/jira/browse/BEAM-9398?focusedWorklogId=409644&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409644 ] ASF GitHub Bot logged work on BEAM-9398: Author: ASF GitHub Bot Created on: 25/Mar/20 16:57 Start Date: 25/Mar/20 16:57 Worklog Time Spent: 10m Work Description: udim commented on pull request #0: [BEAM-9398] runtime_type_check: support setup URL: https://github.com/apache/beam/pull/0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409644) Time Spent: 40m (was: 0.5h) > Python type hints: AbstractDoFnWrapper does not wrap setup > -- > > Key: BEAM-9398 > URL: https://issues.apache.org/jira/browse/BEAM-9398 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > And possibly other methods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9398) Python type hints: AbstractDoFnWrapper does not wrap setup
[ https://issues.apache.org/jira/browse/BEAM-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-9398. - Fix Version/s: 2.21.0 Resolution: Fixed > Python type hints: AbstractDoFnWrapper does not wrap setup > -- > > Key: BEAM-9398 > URL: https://issues.apache.org/jira/browse/BEAM-9398 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > And possibly other methods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409647&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409647 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 17:02 Start Date: 25/Mar/20 17:02 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-603962869 Now org.apache.beam.sdk.io.cassandra.CassandraIOTest.classMethod is failing. These are unrelated and didn't fail before. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409647) Time Spent: 3h 20m (was: 3h 10m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409648 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 17:02 Start Date: 25/Mar/20 17:02 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409648) Time Spent: 3.5h (was: 3h 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8514) ZetaSql should use cost-based optimization to take advantage of Join Reordering Rule and Push-Down Rule
[ https://issues.apache.org/jira/browse/BEAM-8514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8514. --- Fix Version/s: 2.18.0 Resolution: Fixed > ZetaSql should use cost-based optimization to take advantage of Join > Reordering Rule and Push-Down Rule > --- > > Key: BEAM-8514 > URL: https://issues.apache.org/jira/browse/BEAM-8514 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Default config should use BeamCostModel, as well as tests with custom > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7610) SELECT COALESCE(...) isn't inferred as non-nullable field
[
https://issues.apache.org/jira/browse/BEAM-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066875#comment-17066875
]
Kenneth Knowles commented on BEAM-7610:
---
Still an issue? Still working on this?
> SELECT COALESCE(...) isn't inferred as non-nullable field
> -
>
> Key: BEAM-7610
> URL: https://issues.apache.org/jira/browse/BEAM-7610
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Assignee: Kirill Kozlov
>Priority: Major
>
> In Calcite, Coalesce is described as:
> {code}
> ReturnTypes.cascade(ReturnTypes.LEAST_RESTRICTIVE,
> SqlTypeTransforms.LEAST_NULLABLE)
> {code}
> However, giving non-null constant as an argument doesn't result in a
> non-nullable expression:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true)));
> PCollection input = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = input
> .apply(SqlTransform.query("SELECT COALESCE(name, 'unknown') as name
> FROM PCOLLECTION"))
> .getSchema();
> assertEquals(
> Schema.builder().addStringField("name").build(),
> outputSchema);
> {code}
> Not sure if it's a problem in Calcite or Beam SQL.
> There are no other functions that can be used to produce a non-nullable field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8404) [SQL] Update deprecated method calls
[ https://issues.apache.org/jira/browse/BEAM-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8404. --- Fix Version/s: 2.17.0 Resolution: Fixed > [SQL] Update deprecated method calls > > > Key: BEAM-8404 > URL: https://issues.apache.org/jira/browse/BEAM-8404 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Affects Versions: 2.15.0 >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Improve code health by moving away from using deprecated methods/classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8514) ZetaSql should use cost-based optimization to take advantage of Join Reordering Rule and Push-Down Rule
[ https://issues.apache.org/jira/browse/BEAM-8514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066872#comment-17066872 ] Kenneth Knowles commented on BEAM-8514: --- I think there were some incompatibilities so some rules were disabled, but there should be a separate bug tracking anything remaining. [~amaliujia] [~apilloud] is there another Jira or is this fully resolved? > ZetaSql should use cost-based optimization to take advantage of Join > Reordering Rule and Push-Down Rule > --- > > Key: BEAM-8514 > URL: https://issues.apache.org/jira/browse/BEAM-8514 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Default config should use BeamCostModel, as well as tests with custom > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7609) SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
[
https://issues.apache.org/jira/browse/BEAM-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066876#comment-17066876
]
Kenneth Knowles commented on BEAM-7609:
---
Still an issue? Working on this?
> SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
> ---
>
> Key: BEAM-7609
> URL: https://issues.apache.org/jira/browse/BEAM-7609
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Assignee: Kirill Kozlov
>Priority: Major
>
> Works in sqlline shell:
> {code}
> Welcome to Beam SQL 2.14.0-SNAPSHOT (based on sqlline version 1.4.0)
> 0: BeamSQL> CREATE EXTERNAL TABLE s1 (id BIGINT) TYPE 'test';
> No rows affected (0.507 seconds)
> 0: BeamSQL> CREATE EXTERNAL TABLE s2 (id BIGINT) TYPE 'test';
> No rows affected (0.004 seconds)
> 0: BeamSQL> SELECT DISTINCT s1.id as lhs, s2.id as rhs FROM s1 JOIN s2 USING
> (id);
> +-+-+
> | lhs | rhs |
> +-+-+
> +-+-+
> No rows selected (2.568 seconds)
> {code}
> But doesn't work in the test:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("id", Schema.FieldType.INT32));
> PCollection i1 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> PCollection i2 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = PCollectionTuple
> .of("i1", i1)
> .and("i2", i2)
> .apply(SqlTransform.query("SELECT DISTINCT s1.id as lhs, s2.id as rhs
> FROM i1 JOIN i2 USING (id)"))
> .getSchema();
> assertEquals(ImmutableList.of("lhs", "rhs"),
> outputSchema.getFieldNames());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-3188) [Calcite SQL] Query Parametrization
[
https://issues.apache.org/jira/browse/BEAM-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066877#comment-17066877
]
Kenneth Knowles commented on BEAM-3188:
---
This is now supported in ZetaSQL dialect but not Calcite dialect, correct?
> [Calcite SQL] Query Parametrization
> ---
>
> Key: BEAM-3188
> URL: https://issues.apache.org/jira/browse/BEAM-3188
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kyle Weaver
>Priority: Major
>
> Look into SQL query parametrization/templating.
> Calcite supports parameters:
> https://issues.apache.org/jira/browse/CALCITE-2054
> Beam does not:
> {code:java}
> Caused by: java.lang.UnsupportedOperationException: class
> org.apache.calcite.rex.RexDynamicParam is not supported yet!
> at
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:430)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=409653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409653 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 25/Mar/20 17:09 Start Date: 25/Mar/20 17:09 Worklog Time Spent: 10m Work Description: udim commented on pull request #11212: [BEAM-8280] Type hints via annotations snippets URL: https://github.com/apache/beam/pull/11212 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409653) Time Spent: 10h 50m (was: 10h 40m) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
