[jira] [Reopened] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reopened BEAM-5539: - PR was rolled back > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions
[ https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647174#comment-16647174 ] Udi Meiri commented on BEAM-5663: - My investigation from April 2018 is here: https://github.com/udim/beam/commit/3c08b21a9950e39bf71b150a087ae7bf7350f7c3 I haven't test with cython yet, but I just checked and Bazel has experimental support for pip: https://github.com/bazelbuild/rules_python/tree/master/examples/version > Add tox suites for various Python 3 versions > > > Key: BEAM-5663 > URL: https://issues.apache.org/jira/browse/BEAM-5663 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Manu Zhang >Priority: Major > > Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test > failings across various Python 3 versions. It will be valuable to add tox > suites for Python 3.4, 3.5, 3.6 and 3.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5711) dockerized-jenkins broken in copy phase
Udi Meiri created BEAM-5711: --- Summary: dockerized-jenkins broken in copy phase Key: BEAM-5711 URL: https://issues.apache.org/jira/browse/BEAM-5711 Project: Beam Issue Type: Bug Components: testing Reporter: Udi Meiri Assignee: Udi Meiri Shell: $ docker build -t beamjenkins .test-infra/dockerized-jenkins/ Sending build context to Docker daemon 25.09kB Step 1/11 : FROM jenkins/jenkins:lts ---> 80fc01fa46f9 Step 2/11 : ENV JENKINS_HOME=/var/jenkins_real_home/ ---> Using cache ---> cbb533c0cf81 Step 3/11 : COPY plugins.txt /usr/share/jenkins/ref/plugins.txt ---> Using cache ---> d53002630120 Step 4/11 : RUN head -n 1 /usr/local/bin/install-plugins.sh ---> Running in 1085003841cf #!/bin/bash -eu Removing intermediate container 1085003841cf ---> ca44c9d909c9 Step 5/11 : RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt ---> Running in 38a6a3830ae9 /bin/sh: 1: cannot open /usr/share/jenkins/ref/plugins.txt: Permission denied The command '/bin/sh -c /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt' returned a non-zero code: 2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5537) Beam Dependency Update Request: google-cloud-bigquery
[ https://issues.apache.org/jira/browse/BEAM-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-5537: --- Assignee: Udi Meiri > Beam Dependency Update Request: google-cloud-bigquery > - > > Key: BEAM-5537 > URL: https://issues.apache.org/jira/browse/BEAM-5537 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > > - 2018-10-01 19:15:02.343276 > - > Please consider upgrading the dependency google-cloud-bigquery. > The current version is 0.25.0. The latest version is 1.5.1 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:08:29.646271 > - > Please consider upgrading the dependency google-cloud-bigquery. > The current version is 0.25.0. The latest version is 1.6.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-1909) BigQuery read transform fails for DirectRunner when querying non-US regions
[ https://issues.apache.org/jira/browse/BEAM-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-1909. - Resolution: Fixed Fix Version/s: 2.8.0 > BigQuery read transform fails for DirectRunner when querying non-US regions > --- > > Key: BEAM-1909 > URL: https://issues.apache.org/jira/browse/BEAM-1909 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Fix For: 2.8.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > See: > http://stackoverflow.com/questions/42135002/google-dataflow-cannot-read-and-write-in-different-locations-python-sdk-v0-5-5/42144748?noredirect=1#comment73621983_42144748 > This should be fixed by creating the temp dataset and table in the correct > region. > cc: [~sb2nov] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5457) BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong location
[
https://issues.apache.org/jira/browse/BEAM-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-5457.
-
Resolution: Fixed
Fix Version/s: 2.8.0
> BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong
> location
>
>
> Key: BEAM-5457
> URL: https://issues.apache.org/jira/browse/BEAM-5457
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: 2.8.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> I'm in the EU, if I have a
>
> {code:java}
> BigQuerySource(
> query="SELECT x, y FROM `my-other-project.mydataset.my_european_table`",
> project="myproject",
> use_standard_sql=True
> ){code}
> And then run the Pipeline through the DirectRunner I get the following
> warning and error:
> {noformat}
> 2018-09-21 11:39:52,620 WARNING root create_temporary_dataset
> Dataset myproject:temp_dataset_0bbb28f014a24225b668a67341f4f71e does not
> exist so we will create it as temporary with location=None {noformat}
> {noformat}
> HttpBadRequestError: HttpError accessing
> :
> response: <{'status': '400', 'content-length': '354', 'x-xss-protection':
> '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF',
> '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Fri, 21 Sep
> 2018 09:39:55 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443";
> ma=2592000; v="44,43,39,35"', 'content-type': 'application/json;
> charset=UTF-8'}>, content <{
> "error": {
> "code": 400,
> "message": "Cannot read and write in different locations: source: EU,
> destination: US",
> "errors": [
> {
> "message": "Cannot read and write in different locations: source: EU,
> destination: US",
> "domain": "global",
> "reason": "invalid"
> }
> ],
> "status": "INVALID_ARGUMENT"
> }
> {noformat}
> There's a TODO in the code that looks very related:
> [https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665|https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665,]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-1909) BigQuery read transform fails for DirectRunner when querying non-US regions
[ https://issues.apache.org/jira/browse/BEAM-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-1909: --- Assignee: Chamikara Jayalath (was: Ahmet Altay) > BigQuery read transform fails for DirectRunner when querying non-US regions > --- > > Key: BEAM-1909 > URL: https://issues.apache.org/jira/browse/BEAM-1909 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Fix For: 2.8.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > See: > http://stackoverflow.com/questions/42135002/google-dataflow-cannot-read-and-write-in-different-locations-python-sdk-v0-5-5/42144748?noredirect=1#comment73621983_42144748 > This should be fixed by creating the temp dataset and table in the correct > region. > cc: [~sb2nov] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-1909) BigQuery read transform fails for DirectRunner when querying non-US regions
[ https://issues.apache.org/jira/browse/BEAM-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-1909: --- Assignee: Ahmet Altay > BigQuery read transform fails for DirectRunner when querying non-US regions > --- > > Key: BEAM-1909 > URL: https://issues.apache.org/jira/browse/BEAM-1909 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Ahmet Altay >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > See: > http://stackoverflow.com/questions/42135002/google-dataflow-cannot-read-and-write-in-different-locations-python-sdk-v0-5-5/42144748?noredirect=1#comment73621983_42144748 > This should be fixed by creating the temp dataset and table in the correct > region. > cc: [~sb2nov] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5539. - Resolution: Fixed Fix Version/s: Not applicable > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5513) Upgrade google-cloud-pubsub to 0.35.4
[ https://issues.apache.org/jira/browse/BEAM-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5513. - Resolution: Fixed Fix Version/s: 2.8.0 > Upgrade google-cloud-pubsub to 0.35.4 > - > > Key: BEAM-5513 > URL: https://issues.apache.org/jira/browse/BEAM-5513 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.8.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Version 0.35.4 is not the latest version out, but it is the latest supported > on Dataflow runner. > [~markflyhigh] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4505) Archive/Retire apache/beam-site repository
[ https://issues.apache.org/jira/browse/BEAM-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642310#comment-16642310 ] Udi Meiri commented on BEAM-4505: - Things left to do: 1. Make 'master' the default branch on apache/beam-site. 2. Delete 'asf-git' branch from apache/beam-site. I don't have permission to do any of these steps. > Archive/Retire apache/beam-site repository > -- > > Key: BEAM-4505 > URL: https://issues.apache.org/jira/browse/BEAM-4505 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Scott Wegner >Priority: Major > Labels: beam-site-automation-reliability > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-4501) Update contribution guide for new website contribution process
[ https://issues.apache.org/jira/browse/BEAM-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-4501. - Resolution: Fixed Fix Version/s: Not applicable > Update contribution guide for new website contribution process > -- > > Key: BEAM-4501 > URL: https://issues.apache.org/jira/browse/BEAM-4501 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Fix For: Not applicable > > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-4502) Add redirect notice on apache/beam-site README.md
[ https://issues.apache.org/jira/browse/BEAM-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-4502. --- Resolution: Fixed Fix Version/s: Not applicable > Add redirect notice on apache/beam-site README.md > - > > Key: BEAM-4502 > URL: https://issues.apache.org/jira/browse/BEAM-4502 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Fix For: Not applicable > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640114#comment-16640114
]
Udi Meiri commented on BEAM-5442:
-
I believe PR 6557 broke integration tests using Dataflow.
Cloud console:
"Parsing unknown args:
[u'--dataflowJobId=2018-10-05_07_00_20-5526009939236014896',
u'--autoscalingAlgorithm=NONE', u'--direct_runner_use_stacked_bundle',
u'--maxNumWorkers=0', u'--style=scrambled', u'--sleep_secs=20',
u'--pipeline_type_check',
u'--gcpTempLocation=gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-1005140012-917021.1538748012.917145',
u'--numWorkers=1', u'--beam_plugins=apache_beam.io.filesystem.FileSystem',
u'--beam_plugins=apache_beam.io.hadoopfilesystem.HadoopFileSystem',
u'--beam_plugins=apache_beam.io.localfilesystem.LocalFileSystem',
u'--beam_plugins=apache_beam.io.gcp.gcsfilesystem.GCSFileSystem',
u'--beam_plugins=apache_beam.io.filesystem_test.TestingFileSystem',
u'--beam_plugins=apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer',
u'--beam_plugins=apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer',
u'--beam_plugins=apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer',
u'--beam_plugins=apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer',
u'--pipelineUrl=gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1005140012-917021.1538748012.917145/pipeline.pb']"
"Python sdk harness failed:
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
line 133, in main
sdk_pipeline_options.get_all_options(drop_default=True))
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/options/pipeline_options.py",
line 224, in get_all_options
parser.add_argument(arg.split('=', 1)[0], nargs='?')
File "/usr/lib/python2.7/argparse.py", line 1308, in add_argument
return self._add_action(action)
File "/usr/lib/python2.7/argparse.py", line 1682, in _add_action
self._optionals._add_action(action)
File "/usr/lib/python2.7/argparse.py", line 1509, in _add_action
action = super(_ArgumentGroup, self)._add_action(action)
File "/usr/lib/python2.7/argparse.py", line 1322, in _add_action
self._check_conflict(action)
File "/usr/lib/python2.7/argparse.py", line 1460, in _check_conflict
conflict_handler(action, confl_optionals)
File "/usr/lib/python2.7/argparse.py", line 1467, in _handle_conflict_error
raise ArgumentError(action, message % conflict_string)
ArgumentError: argument --beam_plugins: conflicting option string(s):
--beam_plugins"
Test output:
07:28:37 ==
07:28:37 FAIL: test_streaming_with_attributes
(apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest)
07:28:37 --
07:28:37 Traceback (most recent call last):
07:28:37 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py",
line 172, in test_streaming_with_attributes
07:28:37 self._test_streaming(with_attributes=True)
07:28:37 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py",
line 164, in _test_streaming
07:28:37 timestamp_attribute=self.TIMESTAMP_ATTRIBUTE)
07:28:37 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py",
line 91, in run_pipeline
07:28:37 result = p.run()
07:28:37 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
line 416, in run
07:28:37 return self.runner.run_pipeline(self)
07:28:37 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
line 65, in run_pipeline
07:28:37 hc_assert_that(self.result, pickler.loads(on_success_matcher))
07:28:37 AssertionError:
07:28:37 Expected: (Test pipeline expected terminated in state: RUNNING and
Expected 2 messages.)
07:28:37 but: Expected 2 messages. Got 0 messages. Diffs (item, count):
07:28:37 Expected but not in actual: [(PubsubMessage(data001-seen,
{'processed': 'IT'}), 1), (PubsubMessage(data002-seen, {'timestamp_out':
'2018-07-11T02:02:50.149000Z', 'processed': 'IT'}), 1)]
07:28:37 Unexpected: []
07:28:37 Stripped attributes: ['id', 'timestamp']
07:28:37
07:28:37 >> begin captured stdout << -
07:28:37 Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-05_07_00_20-5526009939236014896?project=apache-beam-testing.
07
[jira] [Commented] (BEAM-5513) Upgrade google-cloud-pubsub to 0.35.4
[ https://issues.apache.org/jira/browse/BEAM-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637671#comment-16637671 ] Udi Meiri commented on BEAM-5513: - https://github.com/apache/beam/pull/6564 > Upgrade google-cloud-pubsub to 0.35.4 > - > > Key: BEAM-5513 > URL: https://issues.apache.org/jira/browse/BEAM-5513 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > Version 0.35.4 is not the latest version out, but it is the latest supported > on Dataflow runner. > [~markflyhigh] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5632) Add a DirectRunner based BQ integration test
Udi Meiri created BEAM-5632: --- Summary: Add a DirectRunner based BQ integration test Key: BEAM-5632 URL: https://issues.apache.org/jira/browse/BEAM-5632 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Ahmet Altay Besides validating that BQ works, it should also verify that temporary tables for queries are created in the right location, after fixing: https://issues.apache.org/jira/browse/BEAM-1909 https://issues.apache.org/jira/browse/BEAM-5457 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5614) Using gs:// paths without first doing a "gcloud auth" gives an unhelpful error message
[ https://issues.apache.org/jira/browse/BEAM-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636321#comment-16636321 ] Udi Meiri commented on BEAM-5614: - Full output: WARNING: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/. Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) at org.apache.beam.sdk.Pipeline.create(Pipeline.java:145) at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:176) at org.apache.beam.examples.WordCount.main(WordCount.java:192) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) ... 5 more Caused by: java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:243) ... 10 more Caused by: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://XXX/staging/. at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:255) at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:232) at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592) at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533) at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:158) at com.sun.proxy.$Proxy15.getGcpTempLocation(Unknown Source) at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:241)
[jira] [Updated] (BEAM-5614) Using gs:// paths without first doing a "gcloud auth" gives an unhelpful error message
[ https://issues.apache.org/jira/browse/BEAM-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-5614: Summary: Using gs:// paths without first doing a "gcloud auth" gives an unhelpful error message (was: Using gs:// paths without first doing a "gcloud auth" give unhelpful error message) > Using gs:// paths without first doing a "gcloud auth" gives an unhelpful > error message > -- > > Key: BEAM-5614 > URL: https://issues.apache.org/jira/browse/BEAM-5614 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Udi Meiri >Assignee: Chamikara Jayalath >Priority: Major > > Users see an error like: > java.lang.IllegalArgumentException: Error constructing default value for > gcpTempLocation: tempLocation is not a valid GCS path, > gs://clouddfe-vanya/staging/. > Also reported here: > https://stackoverflow.com/questions/43026371/apache-beam-minimalwordcount-example-with-dataflow-runner-on-eclipse -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5614) Using gs:// paths without first doing a "gcloud auth" gives an unhelpful error message
[ https://issues.apache.org/jira/browse/BEAM-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-5614: Description: Users see an error like: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://bucket/path/. Also reported here: https://stackoverflow.com/questions/43026371/apache-beam-minimalwordcount-example-with-dataflow-runner-on-eclipse was: Users see an error like: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://clouddfe-vanya/staging/. Also reported here: https://stackoverflow.com/questions/43026371/apache-beam-minimalwordcount-example-with-dataflow-runner-on-eclipse > Using gs:// paths without first doing a "gcloud auth" gives an unhelpful > error message > -- > > Key: BEAM-5614 > URL: https://issues.apache.org/jira/browse/BEAM-5614 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Udi Meiri >Assignee: Chamikara Jayalath >Priority: Major > > Users see an error like: > java.lang.IllegalArgumentException: Error constructing default value for > gcpTempLocation: tempLocation is not a valid GCS path, gs://bucket/path/. > Also reported here: > https://stackoverflow.com/questions/43026371/apache-beam-minimalwordcount-example-with-dataflow-runner-on-eclipse -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5614) Using gs:// paths without first doing a "gcloud auth" give unhelpful error message
Udi Meiri created BEAM-5614: --- Summary: Using gs:// paths without first doing a "gcloud auth" give unhelpful error message Key: BEAM-5614 URL: https://issues.apache.org/jira/browse/BEAM-5614 Project: Beam Issue Type: Bug Components: io-java-gcp Reporter: Udi Meiri Assignee: Chamikara Jayalath Users see an error like: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://clouddfe-vanya/staging/. Also reported here: https://stackoverflow.com/questions/43026371/apache-beam-minimalwordcount-example-with-dataflow-runner-on-eclipse -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5457) BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong location
[
https://issues.apache.org/jira/browse/BEAM-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635963#comment-16635963
]
Udi Meiri commented on BEAM-5457:
-
Previously: https://issues.apache.org/jira/browse/BEAM-1909
> BigQuerySource(query=...) in DirectRunner creates temp dataset in the wrong
> location
>
>
> Key: BEAM-5457
> URL: https://issues.apache.org/jira/browse/BEAM-5457
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I'm in the EU, if I have a
>
> {code:java}
> BigQuerySource(
> query="SELECT x, y FROM `my-other-project.mydataset.my_european_table`",
> project="myproject",
> use_standard_sql=True
> ){code}
> And then run the Pipeline through the DirectRunner I get the following
> warning and error:
> {noformat}
> 2018-09-21 11:39:52,620 WARNING root create_temporary_dataset
> Dataset myproject:temp_dataset_0bbb28f014a24225b668a67341f4f71e does not
> exist so we will create it as temporary with location=None {noformat}
> {noformat}
> HttpBadRequestError: HttpError accessing
> :
> response: <{'status': '400', 'content-length': '354', 'x-xss-protection':
> '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF',
> '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Fri, 21 Sep
> 2018 09:39:55 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443";
> ma=2592000; v="44,43,39,35"', 'content-type': 'application/json;
> charset=UTF-8'}>, content <{
> "error": {
> "code": 400,
> "message": "Cannot read and write in different locations: source: EU,
> destination: US",
> "errors": [
> {
> "message": "Cannot read and write in different locations: source: EU,
> destination: US",
> "domain": "global",
> "reason": "invalid"
> }
> ],
> "status": "INVALID_ARGUMENT"
> }
> {noformat}
> There's a TODO in the code that looks very related:
> [https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665|https://github.com/apache/beam/blob/d691a86b8fd082efd0fd71c3cb58b7d61442717d/sdks/python/apache_beam/io/gcp/bigquery.py#L665,]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-5539: --- Assignee: Udi Meiri > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634799#comment-16634799 ] Udi Meiri commented on BEAM-5539: - Tracking issue: https://issues.apache.org/jira/browse/BEAM-5513 > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide
[ https://issues.apache.org/jira/browse/BEAM-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reopened BEAM-4498: - https://github.com/apache/beam/pull/6533 for release guide updates. > Migrate release Javadocs / Pydocs to [asf-site] branch and update release > guide > --- > > Key: BEAM-4498 > URL: https://issues.apache.org/jira/browse/BEAM-4498 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4501) Update contribution guide for new website contribution process
[ https://issues.apache.org/jira/browse/BEAM-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4501: --- Assignee: Udi Meiri (was: Scott Wegner) > Update contribution guide for new website contribution process > -- > > Key: BEAM-4501 > URL: https://issues.apache.org/jira/browse/BEAM-4501 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4502) Add redirect notice on apache/beam-site README.md
[ https://issues.apache.org/jira/browse/BEAM-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4502: --- Assignee: Udi Meiri (was: Scott Wegner) > Add redirect notice on apache/beam-site README.md > - > > Key: BEAM-4502 > URL: https://issues.apache.org/jira/browse/BEAM-4502 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5507) Pass pubsubRootUrl option to Dataflow runner.
[ https://issues.apache.org/jira/browse/BEAM-5507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5507. - Resolution: Fixed Fix Version/s: Not applicable > Pass pubsubRootUrl option to Dataflow runner. > - > > Key: BEAM-5507 > URL: https://issues.apache.org/jira/browse/BEAM-5507 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > This option will be used for testing only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5513) Upgrade google-cloud-pubsub to 0.35.4
Udi Meiri created BEAM-5513: --- Summary: Upgrade google-cloud-pubsub to 0.35.4 Key: BEAM-5513 URL: https://issues.apache.org/jira/browse/BEAM-5513 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Udi Meiri Version 0.35.4 is not the latest version out, but it is the latest supported on Dataflow runner. [~markflyhigh] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-5209) precommit timings notebook: fix illegal characters in URL
[ https://issues.apache.org/jira/browse/BEAM-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-5209. --- Resolution: Fixed Fix Version/s: Not applicable > precommit timings notebook: fix illegal characters in URL > - > > Key: BEAM-5209 > URL: https://issues.apache.org/jira/browse/BEAM-5209 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Minor > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide
[ https://issues.apache.org/jira/browse/BEAM-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-4498. --- Resolution: Won't Fix Fix Version/s: Not applicable > Migrate release Javadocs / Pydocs to [asf-site] branch and update release > guide > --- > > Key: BEAM-4498 > URL: https://issues.apache.org/jira/browse/BEAM-4498 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide
[ https://issues.apache.org/jira/browse/BEAM-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629144#comment-16629144 ] Udi Meiri commented on BEAM-4498: - Looks like we'll keep these docs in asf-site on apache/beam-site. https://lists.apache.org/thread.html/da33046a018da64c4c69b98dc415c9d3aea903d3f1a2f0dee3e913a7@%3Cdev.beam.apache.org%3E > Migrate release Javadocs / Pydocs to [asf-site] branch and update release > guide > --- > > Key: BEAM-4498 > URL: https://issues.apache.org/jira/browse/BEAM-4498 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5507) Pass pubsubRootUrl option to Dataflow runner.
Udi Meiri created BEAM-5507: --- Summary: Pass pubsubRootUrl option to Dataflow runner. Key: BEAM-5507 URL: https://issues.apache.org/jira/browse/BEAM-5507 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Udi Meiri This option will be used for testing only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5507) Pass pubsubRootUrl option to Dataflow runner.
[ https://issues.apache.org/jira/browse/BEAM-5507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628059#comment-16628059 ] Udi Meiri commented on BEAM-5507: - https://github.com/apache/beam/pull/6477 > Pass pubsubRootUrl option to Dataflow runner. > - > > Key: BEAM-5507 > URL: https://issues.apache.org/jira/browse/BEAM-5507 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > This option will be used for testing only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5486) Python: Filesystems.match(['gs://bucket/*']) fails
[ https://issues.apache.org/jira/browse/BEAM-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5486. - Resolution: Fixed Fix Version/s: 2.8.0 > Python: Filesystems.match(['gs://bucket/*']) fails > -- > > Key: BEAM-5486 > URL: https://issues.apache.org/jira/browse/BEAM-5486 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Reported here: https://github.com/apache/beam/pull/5024#issuecomment-406211816 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5486) Python: Filesystems.match(['gs://bucket/*']) fails
[ https://issues.apache.org/jira/browse/BEAM-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627656#comment-16627656 ] Udi Meiri commented on BEAM-5486: - We want GCS glob matching to match gsutil's: https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNames > Python: Filesystems.match(['gs://bucket/*']) fails > -- > > Key: BEAM-5486 > URL: https://issues.apache.org/jira/browse/BEAM-5486 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Reported here: https://github.com/apache/beam/pull/5024#issuecomment-406211816 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5486) Python: Filesystems.match(['gs://bucket/*']) fails
Udi Meiri created BEAM-5486: --- Summary: Python: Filesystems.match(['gs://bucket/*']) fails Key: BEAM-5486 URL: https://issues.apache.org/jira/browse/BEAM-5486 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Udi Meiri Reported here: https://github.com/apache/beam/pull/5024#issuecomment-406211816 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-3442) Clean up usage of deprecated BaseException.message
[ https://issues.apache.org/jira/browse/BEAM-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-3442. --- Resolution: Fixed Fix Version/s: Not applicable > Clean up usage of deprecated BaseException.message > -- > > Key: BEAM-3442 > URL: https://issues.apache.org/jira/browse/BEAM-3442 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Minor > Fix For: Not applicable > > > I believe this warning only appears in unit tests. > sdks/python/apache_beam/io/gcp/bigquery_test.py:610: DeprecationWarning: > BaseException.message has been deprecated as of Python 2.6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3442) Clean up usage of deprecated BaseException.message
[ https://issues.apache.org/jira/browse/BEAM-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626141#comment-16626141 ] Udi Meiri commented on BEAM-3442: - I haven't seen this in a while. Considering this closed. > Clean up usage of deprecated BaseException.message > -- > > Key: BEAM-3442 > URL: https://issues.apache.org/jira/browse/BEAM-3442 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Minor > Fix For: Not applicable > > > I believe this warning only appears in unit tests. > sdks/python/apache_beam/io/gcp/bigquery_test.py:610: DeprecationWarning: > BaseException.message has been deprecated as of Python 2.6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-5462:
---
Assignee: (was: Ahmet Altay)
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported}}
> {{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-5462:
---
Assignee: Ahmet Altay (was: Udi Meiri)
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Ahmet Altay
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported}}
> {{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626096#comment-16626096
]
Udi Meiri commented on BEAM-5462:
-
Discussion:
https://lists.apache.org/thread.html/7ed9a48f827bf7f196eab43c7b053e1b37cfdff676d92586577964ff@%3Cdev.beam.apache.org%3E
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported}}
> {{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-5462:
Description:
Messages look like:
{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
DeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported}}
{{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
was:
Messages look like:
{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
DeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported}}
{{ pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported}}
> {{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-5462:
Description:
Messages look like:
{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
DeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported
pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
was:
Messages look like:
/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
DeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported
pipeline.replace_all(_get_transform_overrides(pipeline.options))
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported
> pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-5462:
Description:
Messages look like:
{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
DeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported}}
{{ pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
was:
Messages look like:
{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
DeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported
pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported}}
> {{ pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5462) get rid of .options deprecation warnings in tests
Udi Meiri created BEAM-5462: --- Summary: get rid of .options deprecation warnings in tests Key: BEAM-5462 URL: https://issues.apache.org/jira/browse/BEAM-5462 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Udi Meiri Messages look like: /usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360: DeprecationWarning: options is deprecated since First stable release. References to .options will not be supported pipeline.replace_all(_get_transform_overrides(pipeline.options)) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide
[ https://issues.apache.org/jira/browse/BEAM-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4498: --- Assignee: Udi Meiri (was: Scott Wegner) > Migrate release Javadocs / Pydocs to [asf-site] branch and update release > guide > --- > > Key: BEAM-4498 > URL: https://issues.apache.org/jira/browse/BEAM-4498 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-4696) Execute Jenkins website tests in a Docker container
[ https://issues.apache.org/jira/browse/BEAM-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-4696. --- Resolution: Fixed Fix Version/s: Not applicable > Execute Jenkins website tests in a Docker container > --- > > Key: BEAM-4696 > URL: https://issues.apache.org/jira/browse/BEAM-4696 > Project: Beam > Issue Type: Improvement > Components: testing, website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > > Currently, the website tests run in a vanilla Linux environment, which > require a prerequisite step to install Ruby. The install script is flaky and > adds extra time to the job. > Instead, we should run the website pre-commits inside the pre-built ruby/2.5 > docker image so that we don't need to worry about installing extra > dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-4606. --- Resolution: Fixed Fix Version/s: Not applicable > Upper bound for pytz dependency > --- > > Key: BEAM-4606 > URL: https://issues.apache.org/jira/browse/BEAM-4606 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Do we need an upper bound for the pytz dependency? > ([https://github.com/apache/beam/blob/release-2.5.0/sdks/python/setup.py#L108)] > We typically have upper bounds, in order to avoid future breakages due to a > possibility of breaking/backward incompatible change of that depepdency. > Good practice is to upper bound either at known version, or next major > version. Do we need an exception for pytz because it does not seem to be > following semantic versioning? > cc: [~yifanzou] Is this something dependency notifier can warn on? Dependency > without upper version bounds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4790) Make it easier to select code reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-4790: Summary: Make it easier to select code reviewers for PRs (was: Make it easier to select coder reviewers for PRs) > Make it easier to select code reviewers for PRs > --- > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-4495) Create website pre-commits for apache/beam repository
[ https://issues.apache.org/jira/browse/BEAM-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-4495. - Resolution: Fixed Fix Version/s: Not applicable > Create website pre-commits for apache/beam repository > - > > Key: BEAM-4495 > URL: https://issues.apache.org/jira/browse/BEAM-4495 > Project: Beam > Issue Type: Sub-task > Components: testing, website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Fix For: Not applicable > > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5278) feature-branches.md orphaned
Udi Meiri created BEAM-5278: --- Summary: feature-branches.md orphaned Key: BEAM-5278 URL: https://issues.apache.org/jira/browse/BEAM-5278 Project: Beam Issue Type: Bug Components: website Reporter: Udi Meiri Assignee: Melissa Pashniak https://github.com/apache/beam-site/blob/asf-site/src/contribute/feature-branches.md References to this page were removed in https://github.com/apache/beam-site/pull/438 It seems it should have been deleted in that PR, but there's no other explanation of feature branches on the website, while the main contribution page (https://beam.apache.org/contribute/) lists 4 feature branches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide
[ https://issues.apache.org/jira/browse/BEAM-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597857#comment-16597857 ] Udi Meiri commented on BEAM-4498: - Release script: https://github.com/apache/beam/blob/5720c1d22771a65ad5d7be6a06ad8aa0754fa64b/release/src/main/scripts/build_release_candidate.sh#L224 > Migrate release Javadocs / Pydocs to [asf-site] branch and update release > guide > --- > > Key: BEAM-4498 > URL: https://issues.apache.org/jira/browse/BEAM-4498 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Assignee: Scott Wegner >Priority: Major > Labels: beam-site-automation-reliability > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4696) Execute Jenkins website tests in a Docker container
[ https://issues.apache.org/jira/browse/BEAM-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595396#comment-16595396 ] Udi Meiri commented on BEAM-4696: - https://github.com/apache/beam/pull/6282 is out for review, which implements pre-commits in docker containers. > Execute Jenkins website tests in a Docker container > --- > > Key: BEAM-4696 > URL: https://issues.apache.org/jira/browse/BEAM-4696 > Project: Beam > Issue Type: Improvement > Components: testing, website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > > Currently, the website tests run in a vanilla Linux environment, which > require a prerequisite step to install Ruby. The install script is flaky and > adds extra time to the job. > Instead, we should run the website pre-commits inside the pre-built ruby/2.5 > docker image so that we don't need to worry about installing extra > dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4696) Execute Jenkins website tests in a Docker container
[ https://issues.apache.org/jira/browse/BEAM-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4696: --- Assignee: Udi Meiri (was: Scott Wegner) > Execute Jenkins website tests in a Docker container > --- > > Key: BEAM-4696 > URL: https://issues.apache.org/jira/browse/BEAM-4696 > Project: Beam > Issue Type: Improvement > Components: testing, website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > > Currently, the website tests run in a vanilla Linux environment, which > require a prerequisite step to install Ruby. The install script is flaky and > adds extra time to the job. > Instead, we should run the website pre-commits inside the pre-built ruby/2.5 > docker image so that we don't need to worry about installing extra > dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5209) precommit timings notebook: fix illegal characters in URL
Udi Meiri created BEAM-5209: --- Summary: precommit timings notebook: fix illegal characters in URL Key: BEAM-5209 URL: https://issues.apache.org/jira/browse/BEAM-5209 Project: Beam Issue Type: Improvement Components: testing Reporter: Udi Meiri Assignee: Udi Meiri -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5209) precommit timings notebook: fix illegal characters in URL
[ https://issues.apache.org/jira/browse/BEAM-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590693#comment-16590693 ] Udi Meiri commented on BEAM-5209: - https://github.com/apache/beam/pull/5779 > precommit timings notebook: fix illegal characters in URL > - > > Key: BEAM-5209 > URL: https://issues.apache.org/jira/browse/BEAM-5209 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5208) Clearer Python SDK error message for streaming bigquery reads
[ https://issues.apache.org/jira/browse/BEAM-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5208. - Resolution: Fixed Fix Version/s: 2.7.0 > Clearer Python SDK error message for streaming bigquery reads > - > > Key: BEAM-5208 > URL: https://issues.apache.org/jira/browse/BEAM-5208 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.7.0 > > > BigQuery reads are currently unsupported in streaming pipelines on the Python > SDK using Dataflow. > Instead of giving a cryptic internal error message, fail sooner during > pipeline creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5208) Clearer Python SDK error message for streaming bigquery reads
[ https://issues.apache.org/jira/browse/BEAM-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590684#comment-16590684 ] Udi Meiri commented on BEAM-5208: - Fixed in https://github.com/apache/beam/pull/6192 > Clearer Python SDK error message for streaming bigquery reads > - > > Key: BEAM-5208 > URL: https://issues.apache.org/jira/browse/BEAM-5208 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > BigQuery reads are currently unsupported in streaming pipelines on the Python > SDK using Dataflow. > Instead of giving a cryptic internal error message, fail sooner during > pipeline creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5208) Clearer Python SDK error message for streaming bigquery reads
Udi Meiri created BEAM-5208: --- Summary: Clearer Python SDK error message for streaming bigquery reads Key: BEAM-5208 URL: https://issues.apache.org/jira/browse/BEAM-5208 Project: Beam Issue Type: Improvement Components: sdk-py-core Reporter: Udi Meiri Assignee: Udi Meiri BigQuery reads are currently unsupported in streaming pipelines on the Python SDK using Dataflow. Instead of giving a cryptic internal error message, fail sooner during pipeline creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4696) Execute Jenkins website tests in a Docker container
[ https://issues.apache.org/jira/browse/BEAM-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586827#comment-16586827 ] Udi Meiri commented on BEAM-4696: - I've managed to run a local Jenkins in a docker container (using [~Ardagan]'s dockerized-jenkins). I managed to set it up so that Jenkins has authorization to launch Docker jobs on the host Docker. Right now, it's managed to start the ruby:2.5 image, but it doesn't have access to /tmp on the Jenkins containers: /bin/sh: 0: Can't open /tmp/jenkins6814128182425397587.sh Current incomplete state: https://github.com/apache/beam/commit/6fb3248792a4c8a9c10868df222b8301a13a3286 > Execute Jenkins website tests in a Docker container > --- > > Key: BEAM-4696 > URL: https://issues.apache.org/jira/browse/BEAM-4696 > Project: Beam > Issue Type: Improvement > Components: testing, website >Reporter: Scott Wegner >Assignee: Scott Wegner >Priority: Major > > Currently, the website tests run in a vanilla Linux environment, which > require a prerequisite step to install Ruby. The install script is flaky and > adds extra time to the job. > Instead, we should run the website pre-commits inside the pre-built ruby/2.5 > docker image so that we don't need to worry about installing extra > dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4495) Create website pre-commits for apache/beam repository
[ https://issues.apache.org/jira/browse/BEAM-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4495: --- Assignee: Udi Meiri (was: Scott Wegner) > Create website pre-commits for apache/beam repository > - > > Key: BEAM-4495 > URL: https://issues.apache.org/jira/browse/BEAM-4495 > Project: Beam > Issue Type: Sub-task > Components: testing, website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Labels: beam-site-automation-reliability > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4790) Make it easier to select coder reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583039#comment-16583039 ] Udi Meiri commented on BEAM-4790: - pr/6235 documents code reviewers in OWNERS files. I'll add documentation to the Beam website about how to add code reviews. > Make it easier to select coder reviewers for PRs > > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5126) PreCommit filtering broken based upon PR contents
[ https://issues.apache.org/jira/browse/BEAM-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580265#comment-16580265 ] Udi Meiri commented on BEAM-5126: - Okay, I misread your bug description. I thought precommit weren't running at all. In any case, seems like it was always intended to run for all three languages: https://github.com/apache/beam/pull/5611#issuecomment-396747330 > PreCommit filtering broken based upon PR contents > - > > Key: BEAM-5126 > URL: https://issues.apache.org/jira/browse/BEAM-5126 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Luke Cwik >Assignee: Alan Myrvold >Priority: Minor > > PR precommits used to be filtered by the contents of the PR. > > Example PR that should have only spawned the Java PreCommit: > https://github.com/apache/beam/pull/6159 > > This broke work done in BEAM-4445 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5126) PreCommit filtering broken based upon PR contents
[
https://issues.apache.org/jira/browse/BEAM-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580259#comment-16580259
]
Udi Meiri commented on BEAM-5126:
-
This behavior makes sense, since all three configs have
{quote}'^runners/.*$'{quote} configured as a trigger path.
The Java precommit config is here:
https://github.com/apache/beam/blob/4c693e7aa1ba71fa83370dd383f4bcebe4690b86/.test-infra/jenkins/job_PreCommit_Java.groovy#L28
> PreCommit filtering broken based upon PR contents
> -
>
> Key: BEAM-5126
> URL: https://issues.apache.org/jira/browse/BEAM-5126
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Minor
>
> PR precommits used to be filtered by the contents of the PR.
>
> Example PR that should have only spawned the Java PreCommit:
> https://github.com/apache/beam/pull/6159
>
> This broke work done in BEAM-4445
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5126) PreCommit filtering broken based upon PR contents
[ https://issues.apache.org/jira/browse/BEAM-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580256#comment-16580256 ] Udi Meiri commented on BEAM-5126: - Just created https://github.com/apache/beam/pull/6223 which triggered all 3 precommits (go, java, python). > PreCommit filtering broken based upon PR contents > - > > Key: BEAM-5126 > URL: https://issues.apache.org/jira/browse/BEAM-5126 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Luke Cwik >Assignee: Alan Myrvold >Priority: Minor > > PR precommits used to be filtered by the contents of the PR. > > Example PR that should have only spawned the Java PreCommit: > https://github.com/apache/beam/pull/6159 > > This broke work done in BEAM-4445 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4275) Pubsub: add DirectRunner support for id_label and timestamp_attribute in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578838#comment-16578838 ] Udi Meiri commented on BEAM-4275: - DirectRunner doesn't support reading or writing label_ids, nor writing timestamp attributes. > Pubsub: add DirectRunner support for id_label and timestamp_attribute in > Python SDK > --- > > Key: BEAM-4275 > URL: https://issues.apache.org/jira/browse/BEAM-4275 > Project: Beam > Issue Type: Bug > Components: runner-direct, sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > At least for publishing (and maybe pulling) messages, non-Dataflow-based > sources and sinks for Pub/Sub use the [public > API|https://cloud.google.com/pubsub/docs/publisher] for Pub/Sub, which > doesn't support id_label and timestamp_attribute settings. > Publishing: > id_label - add an attribute to each message with a unique value > timestamp_attribute - add an attribute to each message with the publishing > time as its value > Pulling: > id_label - use the value of this message attribute to deduplicate messages > timestamp_attribute - use the value of this message attribute as the > element's timestamp > > Implementation details: could probably create a pubsubio.py module, for reuse > with other runners (i.e. implement Pub/Sub IO as PTransforms and not > NativeSinks and Sources). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5118) Make Jenkins DSL job development easier, less disruptive
Udi Meiri created BEAM-5118: --- Summary: Make Jenkins DSL job development easier, less disruptive Key: BEAM-5118 URL: https://issues.apache.org/jira/browse/BEAM-5118 Project: Beam Issue Type: Improvement Components: testing Reporter: Udi Meiri Assignee: Scott Wegner Here is my aborted attempt to add command line arguments to the gradlew invocation: https://github.com/apache/beam/pull/6188/files#diff-22460ba8f153644bf2a508681023ab54 (error: ERROR: (PrecommitJobBuilder.groovy, line 79) No such property: steps for class: javaposse.jobdsl.dsl.jobs.FreeStyleJob https://builds.apache.org/job/beam_SeedJob/2403/console ) Some possible improvements to the development process could be: 1. I used vim to edit .groovy files. Is there an IDE that could inspect these objects and do completion? 2. In order to test my I needed to: a. Commit and push my PR changes. b. Add a "run seed job" comment. c. Wait about 3m for it to succeed/fail. Another issue is that the only way to test my code is in production. Is there a test Jenkins instance (could INFRA set one up)? I know that I can run something like 'docker run -p 127.0.0.1:8080:8080 jenkins/jenkins:lts', but that requires a lot of setup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m
[ https://issues.apache.org/jira/browse/BEAM-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572478#comment-16572478 ] Udi Meiri commented on BEAM-5052: - https://github.com/apache/beam-site/pull/520 is a temporary fix > beam_PreCommit_Website_Stage times out after 100m > - > > Key: BEAM-5052 > URL: https://issues.apache.org/jira/browse/BEAM-5052 > Project: Beam > Issue Type: Bug > Components: testing, website >Reporter: Udi Meiri >Assignee: Jason Kuster >Priority: Major > > This job stages the website on GCS for easier review. > I believe this only happens on the 2nd and further runs on a specific PR. A > lot of time is spent in deleting existing files (22k) and then copying over > newer ones. > Example log: > [https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText] > A large chunk of the files are pydoc and javadoc files for version we don't > care about. > > [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc] > > [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc] > Solution is to: > 1. Delete unsupported versions' docs. > 2. Only copy over to GCS the latest pydoc and javadoc, as the rest is > irrelevant for review (for almost all PRs). This can be done by deleting the > irrelevant directories before invoking "gsutil cp". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-5058:
Summary: Python precommits should run E2E tests (was: Python precommits
don't run any E2E tests)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Priority: Major
>
> According to https://beam.apache.org/contribute/testing/ (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> {batch, streaming}x{SDK language}x{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> https://github.com/apache/beam/pull/5731
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-5058:
Description:
According to [https://beam.apache.org/contribute/testing/] (which I'm working
on), end-to-end tests should be run in precommit on each combination of
\{batch, streaming}x\{SDK language}x\{supported runner}.
At least 2 tests need to be added to Python's precommit: wordcount and
wordcount_streaming on Dataflow, and possibly on other supported runners
(direct runner and new runners plz).
These tests should be configured to run from a Gradle sub-project, so that
they're run in parallel to the unit tests.
Example that parallelizes Java precommit integration tests:
[https://github.com/apache/beam/pull/5731]
was:
According to https://beam.apache.org/contribute/testing/ (which I'm working
on), end-to-end tests should be run in precommit on each combination of {batch,
streaming}x{SDK language}x{supported runner}.
At least 2 tests need to be added to Python's precommit: wordcount and
wordcount_streaming on Dataflow, and possibly on other supported runners
(direct runner and new runners plz).
These tests should be configured to run from a Gradle sub-project, so that
they're run in parallel to the unit tests.
Example that parallelizes Java precommit integration tests:
https://github.com/apache/beam/pull/5731
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Priority: Major
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5058) Python precommits don't run any E2E tests
Udi Meiri created BEAM-5058:
---
Summary: Python precommits don't run any E2E tests
Key: BEAM-5058
URL: https://issues.apache.org/jira/browse/BEAM-5058
Project: Beam
Issue Type: Bug
Components: sdk-py-core, testing
Reporter: Udi Meiri
According to https://beam.apache.org/contribute/testing/ (which I'm working
on), end-to-end tests should be run in precommit on each combination of {batch,
streaming}x{SDK language}x{supported runner}.
At least 2 tests need to be added to Python's precommit: wordcount and
wordcount_streaming on Dataflow, and possibly on other supported runners
(direct runner and new runners plz).
These tests should be configured to run from a Gradle sub-project, so that
they're run in parallel to the unit tests.
Example that parallelizes Java precommit integration tests:
https://github.com/apache/beam/pull/5731
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-5055) TEST: Precommit slowness
[ https://issues.apache.org/jira/browse/BEAM-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-5055. --- Resolution: Won't Fix Fix Version/s: Not applicable > TEST: Precommit slowness > - > > Key: BEAM-5055 > URL: https://issues.apache.org/jira/browse/BEAM-5055 > Project: Beam > Issue Type: Task > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Labels: precommit > Fix For: Not applicable > > > please ignore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5055) TEST: Precommit slowness
[ https://issues.apache.org/jira/browse/BEAM-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-5055: --- Assignee: Udi Meiri (was: Jason Kuster) > TEST: Precommit slowness > - > > Key: BEAM-5055 > URL: https://issues.apache.org/jira/browse/BEAM-5055 > Project: Beam > Issue Type: Task > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Labels: precommit > Fix For: Not applicable > > > please ignore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5055) TEST: Precommit slowness
Udi Meiri created BEAM-5055: --- Summary: TEST: Precommit slowness Key: BEAM-5055 URL: https://issues.apache.org/jira/browse/BEAM-5055 Project: Beam Issue Type: Task Components: testing Reporter: Udi Meiri Assignee: Jason Kuster please ignore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m
[ https://issues.apache.org/jira/browse/BEAM-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564468#comment-16564468 ] Udi Meiri commented on BEAM-5052: - [~swegner] > beam_PreCommit_Website_Stage times out after 100m > - > > Key: BEAM-5052 > URL: https://issues.apache.org/jira/browse/BEAM-5052 > Project: Beam > Issue Type: Bug > Components: testing, website >Reporter: Udi Meiri >Assignee: Jason Kuster >Priority: Major > > This job stages the website on GCS for easier review. > I believe this only happens on the 2nd and further runs on a specific PR. A > lot of time is spent in deleting existing files (22k) and then copying over > newer ones. > Example log: > [https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText] > A large chunk of the files are pydoc and javadoc files for version we don't > care about. > > [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc] > > [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc] > Solution is to: > 1. Delete unsupported versions' docs. > 2. Only copy over to GCS the latest pydoc and javadoc, as the rest is > irrelevant for review (for almost all PRs). This can be done by deleting the > irrelevant directories before invoking "gsutil cp". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m
[ https://issues.apache.org/jira/browse/BEAM-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564463#comment-16564463 ] Udi Meiri commented on BEAM-5052: - [~Ardagan] [~melap] > beam_PreCommit_Website_Stage times out after 100m > - > > Key: BEAM-5052 > URL: https://issues.apache.org/jira/browse/BEAM-5052 > Project: Beam > Issue Type: Bug > Components: testing, website >Reporter: Udi Meiri >Assignee: Jason Kuster >Priority: Major > > This job stages the website on GCS for easier review. > I believe this only happens on the 2nd and further runs on a specific PR. A > lot of time is spent in deleting existing files (22k) and then copying over > newer ones. > Example log: > [https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText] > A large chunk of the files are pydoc and javadoc files for version we don't > care about. > > [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc] > > [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc] > Solution is to: > 1. Delete unsupported versions' docs. > 2. Only copy over to GCS the latest pydoc and javadoc, as the rest is > irrelevant for review (for almost all PRs). This can be done by deleting the > irrelevant directories before invoking "gsutil cp". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m
Udi Meiri created BEAM-5052: --- Summary: beam_PreCommit_Website_Stage times out after 100m Key: BEAM-5052 URL: https://issues.apache.org/jira/browse/BEAM-5052 Project: Beam Issue Type: Bug Components: testing, website Reporter: Udi Meiri Assignee: Jason Kuster This job stages the website on GCS for easier review. I believe this only happens on the 2nd and further runs on a specific PR. A lot of time is spent in deleting existing files (22k) and then copying over newer ones. Example log: [https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText] A large chunk of the files are pydoc and javadoc files for version we don't care about. [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc] [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc] Solution is to: 1. Delete unsupported versions' docs. 2. Only copy over to GCS the latest pydoc and javadoc, as the rest is irrelevant for review (for almost all PRs). This can be done by deleting the irrelevant directories before invoking "gsutil cp". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-5039) python postcommit broken in call to WriteToPubSub
[ https://issues.apache.org/jira/browse/BEAM-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-5039. --- Resolution: Fixed Fix Version/s: Not applicable Green since https://builds.apache.org/job/beam_PostCommit_Python_Verify/5613/ > python postcommit broken in call to WriteToPubSub > - > > Key: BEAM-5039 > URL: https://issues.apache.org/jira/browse/BEAM-5039 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > > ERROR: test_streaming_wordcount_it > (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) > -- > Traceback (most recent call last): > File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", > line 105, in test_streaming_wordcount_it > self.test_pipeline.get_full_options_as_args(**extra_opts)) > File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount.py", > line 90, in run > output | beam.io.WriteToPubSub(known_args.output_topic) > TypeError: __init__() takes at least 3 arguments (2 given) > https://builds.apache.org/job/beam_PostCommit_Python_Verify/5597/consoleText -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4159) Add testing for Pubsub attributes
[ https://issues.apache.org/jira/browse/BEAM-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562475#comment-16562475 ] Udi Meiri commented on BEAM-4159: - This is done for Python in: https://github.com/apache/beam/pull/5952 > Add testing for Pubsub attributes > - > > Key: BEAM-4159 > URL: https://issues.apache.org/jira/browse/BEAM-4159 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Udi Meiri >Assignee: Jason Kuster >Priority: Major > > Request is to add an integration test that exercises reading and writing > pubsub message attributes. > Platform: Java SDK > Stretch goals: ID attribute, timestamp attribute, Python SDK and Go SDK (both > using the Java runner) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5039) python postcommit broken in call to WriteToPubSub
Udi Meiri created BEAM-5039: --- Summary: python postcommit broken in call to WriteToPubSub Key: BEAM-5039 URL: https://issues.apache.org/jira/browse/BEAM-5039 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Udi Meiri ERROR: test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) -- Traceback (most recent call last): File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", line 105, in test_streaming_wordcount_it self.test_pipeline.get_full_options_as_args(**extra_opts)) File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount.py", line 90, in run output | beam.io.WriteToPubSub(known_args.output_topic) TypeError: __init__() takes at least 3 arguments (2 given) https://builds.apache.org/job/beam_PostCommit_Python_Verify/5597/consoleText -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4790) Make it easier to select coder reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559979#comment-16559979 ] Udi Meiri commented on BEAM-4790: - Summary doc: https://docs.google.com/document/d/1S8spggJsxDNYZ7aNwZN6VhLhNW372SVRezjblt-7lNQ/edit?usp=sharing This doc should be updated as we gain experience using Mention-bot and Prow. > Make it easier to select coder reviewers for PRs > > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-3744) Support full PubsubMessages
[ https://issues.apache.org/jira/browse/BEAM-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-3744. - Resolution: Fixed Fix Version/s: 2.7.0 > Support full PubsubMessages > --- > > Key: BEAM-3744 > URL: https://issues.apache.org/jira/browse/BEAM-3744 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Critical > Fix For: 2.7.0 > > Time Spent: 10.5h > Remaining Estimate: 0h > > Tracking changes to Pubsub support in Python SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4790) Make it easier to select coder reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554827#comment-16554827 ] Udi Meiri commented on BEAM-4790: - CODEOWNERS isn't a good solution. See mailing list thread above. > Make it easier to select coder reviewers for PRs > > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4790) Make it easier to select coder reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546983#comment-16546983 ] Udi Meiri commented on BEAM-4790: - [~holdenkarau] will be working on blame-based review suggestions. > Make it easier to select coder reviewers for PRs > > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4790) Make it easier to select coder reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4790: --- Assignee: Udi Meiri (was: Davor Bonaci) > Make it easier to select coder reviewers for PRs > > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-4790) Make it easier to select coder reviewers for PRs
Udi Meiri created BEAM-4790: --- Summary: Make it easier to select coder reviewers for PRs Key: BEAM-4790 URL: https://issues.apache.org/jira/browse/BEAM-4790 Project: Beam Issue Type: Improvement Components: project-management Reporter: Udi Meiri Assignee: Davor Bonaci The original idea was to use Github's CODEOWNERS feature, but that turned out to not work well for Apache Beam. Thread: https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E Document discussing this idea: https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 Mailing list thread: https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4275) Pubsub: add DirectRunner support for id_label and timestamp_attribute in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4275: --- Assignee: Udi Meiri (was: Ahmet Altay) > Pubsub: add DirectRunner support for id_label and timestamp_attribute in > Python SDK > --- > > Key: BEAM-4275 > URL: https://issues.apache.org/jira/browse/BEAM-4275 > Project: Beam > Issue Type: Bug > Components: runner-direct, sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > At least for publishing (and maybe pulling) messages, non-Dataflow-based > sources and sinks for Pub/Sub use the [public > API|https://cloud.google.com/pubsub/docs/publisher] for Pub/Sub, which > doesn't support id_label and timestamp_attribute settings. > Publishing: > id_label - add an attribute to each message with a unique value > timestamp_attribute - add an attribute to each message with the publishing > time as its value > Pulling: > id_label - use the value of this message attribute to deduplicate messages > timestamp_attribute - use the value of this message attribute as the > element's timestamp > > Implementation details: could probably create a pubsubio.py module, for reuse > with other runners (i.e. implement Pub/Sub IO as PTransforms and not > NativeSinks and Sources). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4400) Integrate Jenkins Job Cacher Plugin for improved build caching
[ https://issues.apache.org/jira/browse/BEAM-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525646#comment-16525646 ] Udi Meiri commented on BEAM-4400: - I've unassigned myself as I'm not going to work on this due to the issues above, plus that precommits are fast currently (25m) so I don't have an incentive. > Integrate Jenkins Job Cacher Plugin for improved build caching > -- > > Key: BEAM-4400 > URL: https://issues.apache.org/jira/browse/BEAM-4400 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Scott Wegner >Priority: Minor > > See discussion on > [dev|https://lists.apache.org/thread.html/c59289787c665c7732df4095bda0877637eee59e3add4d26a11f6b7f@%3Cdev.beam.apache.org%3E]. > With the new Gradle build, we are now able to build incrementally, but are > not yet taking advantage of it on Jenkins. This would give us the benefit of > much faster pre-commits for files that aren't changed. For example, A change > in the Python which doesn't touch any Java would not need to re-run Java > pre-commits. > By default, Gradle uses a build cache local to the workspace, and in Jenkins > the workspace gets nuked on each build. There is a [Jenkins Job Cacher > Plugin|https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin] which > solves this exact plugin and integrates with Gradle's build cache support. I > believe all we need to do is enable and configure this plugin to realize the > benefits of incremental builds. > To enable the plugin, we likely need to reach out the Apache INFRA team. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4400) Integrate Jenkins Job Cacher Plugin for improved build caching
[ https://issues.apache.org/jira/browse/BEAM-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4400: --- Assignee: (was: Udi Meiri) > Integrate Jenkins Job Cacher Plugin for improved build caching > -- > > Key: BEAM-4400 > URL: https://issues.apache.org/jira/browse/BEAM-4400 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Scott Wegner >Priority: Minor > > See discussion on > [dev|https://lists.apache.org/thread.html/c59289787c665c7732df4095bda0877637eee59e3add4d26a11f6b7f@%3Cdev.beam.apache.org%3E]. > With the new Gradle build, we are now able to build incrementally, but are > not yet taking advantage of it on Jenkins. This would give us the benefit of > much faster pre-commits for files that aren't changed. For example, A change > in the Python which doesn't touch any Java would not need to re-run Java > pre-commits. > By default, Gradle uses a build cache local to the workspace, and in Jenkins > the workspace gets nuked on each build. There is a [Jenkins Job Cacher > Plugin|https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin] which > solves this exact plugin and integrates with Gradle's build cache support. I > believe all we need to do is enable and configure this plugin to realize the > benefits of incremental builds. > To enable the plugin, we likely need to reach out the Apache INFRA team. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4400) Integrate Jenkins Job Cacher Plugin for improved build caching
[ https://issues.apache.org/jira/browse/BEAM-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523076#comment-16523076 ] Udi Meiri commented on BEAM-4400: - I also tried a fresh clone of the repo. Still doesn't use the cache. Weird. I'm incline to abandon this approach if precommits are fast enough (around 21m). > Integrate Jenkins Job Cacher Plugin for improved build caching > -- > > Key: BEAM-4400 > URL: https://issues.apache.org/jira/browse/BEAM-4400 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Minor > > See discussion on > [dev|https://lists.apache.org/thread.html/c59289787c665c7732df4095bda0877637eee59e3add4d26a11f6b7f@%3Cdev.beam.apache.org%3E]. > With the new Gradle build, we are now able to build incrementally, but are > not yet taking advantage of it on Jenkins. This would give us the benefit of > much faster pre-commits for files that aren't changed. For example, A change > in the Python which doesn't touch any Java would not need to re-run Java > pre-commits. > By default, Gradle uses a build cache local to the workspace, and in Jenkins > the workspace gets nuked on each build. There is a [Jenkins Job Cacher > Plugin|https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin] which > solves this exact plugin and integrates with Gradle's build cache support. I > believe all we need to do is enable and configure this plugin to realize the > benefits of incremental builds. > To enable the plugin, we likely need to reach out the Apache INFRA team. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4400) Integrate Jenkins Job Cacher Plugin for improved build caching
[ https://issues.apache.org/jira/browse/BEAM-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523014#comment-16523014 ] Udi Meiri commented on BEAM-4400: - I've spent the better of a day looking at this. Results: 1. Ran this docker container locally: https://hub.docker.com/r/gradle/build-cache-node/ 2. Configured the Gradle for Beam to use said remote cache from step 1. 3. Ran ./gradlew --info :beam-java-core-something:test Verified on the remote cache's status page that the cache was no longer empty. 4. Ran ./gradlew clean --quiet 5. Ran the command from step 3 again. Expected: remote cache to speed up gradle build. Tests should not run again. What I got: cache wasn't used, new entries written to cache (count of entries went up). I saw tests being run. I expected the cache to be used since the source hasn't changed Perhaps the 'clean' task does more than I thought it does. > Integrate Jenkins Job Cacher Plugin for improved build caching > -- > > Key: BEAM-4400 > URL: https://issues.apache.org/jira/browse/BEAM-4400 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Minor > > See discussion on > [dev|https://lists.apache.org/thread.html/c59289787c665c7732df4095bda0877637eee59e3add4d26a11f6b7f@%3Cdev.beam.apache.org%3E]. > With the new Gradle build, we are now able to build incrementally, but are > not yet taking advantage of it on Jenkins. This would give us the benefit of > much faster pre-commits for files that aren't changed. For example, A change > in the Python which doesn't touch any Java would not need to re-run Java > pre-commits. > By default, Gradle uses a build cache local to the workspace, and in Jenkins > the workspace gets nuked on each build. There is a [Jenkins Job Cacher > Plugin|https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin] which > solves this exact plugin and integrates with Gradle's build cache support. I > believe all we need to do is enable and configure this plugin to realize the > benefits of incremental builds. > To enable the plugin, we likely need to reach out the Apache INFRA team. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4630) Dataflow tests failing due to quota issues
[
https://issues.apache.org/jira/browse/BEAM-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522469#comment-16522469
]
Udi Meiri commented on BEAM-4630:
-
Some questions:
1. How can we see our current quota?
2. Is it possible to see a graph of the number of dataflow jobs over time?
> Dataflow tests failing due to quota issues
> --
>
> Key: BEAM-4630
> URL: https://issues.apache.org/jira/browse/BEAM-4630
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, testing
>Reporter: Scott Wegner
>Assignee: Rafael Fernandez
>Priority: Major
> Fix For: Not applicable
>
>
> In the last few days, we've seen a number to pre-commit and post-commit runs
> fail on Dataflow tests due to quota issues. For example
> ([jenkins|https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/6785/],
>
> [gradle|https://scans.gradle.com/s/ouxnqbymallzi/tests/ubfk4psvvdijy-brhdrcaeda7dm]):
> {{:beam-runners-google-cloud-dataflow-java:validatesRunnerTestorg.apache.beam.sdk.transforms.CombineTest$WindowingTests
> » testGlobalCombineWithDefaultsAndTriggers}}
> {{Failed to create a workflow job: (e5d646c7d1a09997): The workflow could not
> be created. Causes: (999cdad46bde76aa): Dataflow quota error for
> jobs-per-project quota. Project apache-beam-testing is running 50 jobs.
> Please check the quota usage via GCP Console. If it exceeds the limit, please
> wait for a workflow to finish or contact dataflow-feedb...@google.com to
> request an increase in quota. If it does not, contact
> dataflow-feedb...@google.com.}}
> We should disable these tests for now, and work on either increasing Dataflow
> quota, or reducing parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-4400) Integrate Jenkins Job Cacher Plugin for improved build caching
[ https://issues.apache.org/jira/browse/BEAM-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-4400: --- Assignee: Udi Meiri > Integrate Jenkins Job Cacher Plugin for improved build caching > -- > > Key: BEAM-4400 > URL: https://issues.apache.org/jira/browse/BEAM-4400 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Minor > > See discussion on > [dev|https://lists.apache.org/thread.html/c59289787c665c7732df4095bda0877637eee59e3add4d26a11f6b7f@%3Cdev.beam.apache.org%3E]. > With the new Gradle build, we are now able to build incrementally, but are > not yet taking advantage of it on Jenkins. This would give us the benefit of > much faster pre-commits for files that aren't changed. For example, A change > in the Python which doesn't touch any Java would not need to re-run Java > pre-commits. > By default, Gradle uses a build cache local to the workspace, and in Jenkins > the workspace gets nuked on each build. There is a [Jenkins Job Cacher > Plugin|https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin] which > solves this exact plugin and integrates with Gradle's build cache support. I > believe all we need to do is enable and configure this plugin to realize the > benefits of incremental builds. > To enable the plugin, we likely need to reach out the Apache INFRA team. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518773#comment-16518773 ] Udi Meiri commented on BEAM-4606: - A quick search yields that the versioning of pytz [tracks the IANA timezone database|https://www.carsonlam.ca/python-pytz-versioning/]. Thus there is no concept of major version for pytz, AFAICT. We currently only use pytz.utc in two places in this file: https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/utils/timestamp.py#L85 I don't think we care about keeping the database up to date for this use case. so I am fine with pinning to a single version (such as ==2018.3). > Upper bound for pytz dependency > --- > > Key: BEAM-4606 > URL: https://issues.apache.org/jira/browse/BEAM-4606 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Major > > Do we need an upper bound for the pytz dependency? > ([https://github.com/apache/beam/blob/release-2.5.0/sdks/python/setup.py#L108)] > We typically have upper bounds, in order to avoid future breakages due to a > possibility of breaking/backward incompatible change of that depepdency. > Good practice is to upper bound either at known version, or next major > version. Do we need an exception for pytz because it does not seem to be > following semantic versioning? > cc: [~yifanzou] Is this something dependency notifier can warn on? Dependency > without upper version bounds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4589) Test PubsubJsonClient using messages without data
[ https://issues.apache.org/jira/browse/BEAM-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-4589: Description: I would like to extend our integration tests to send/receive pubsub messages that have attributes but no data. This pull requests fixes a NullPointerException for such messages: [https://github.com/apache/beam/pull/5016] was:NPE bug: https://github.com/apache/beam/pull/5016 > Test PubsubJsonClient using messages without data > - > > Key: BEAM-4589 > URL: https://issues.apache.org/jira/browse/BEAM-4589 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Udi Meiri >Assignee: Chamikara Jayalath >Priority: Major > > I would like to extend our integration tests to send/receive pubsub messages > that have attributes but no data. > This pull requests fixes a NullPointerException for such messages: > [https://github.com/apache/beam/pull/5016] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-4589) Test PubsubJsonClient using messages without data
Udi Meiri created BEAM-4589: --- Summary: Test PubsubJsonClient using messages without data Key: BEAM-4589 URL: https://issues.apache.org/jira/browse/BEAM-4589 Project: Beam Issue Type: Improvement Components: io-java-gcp Reporter: Udi Meiri Assignee: Chamikara Jayalath NPE bug: https://github.com/apache/beam/pull/5016 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4569) Jenkins: increase build history retention to 4 weeks
[ https://issues.apache.org/jira/browse/BEAM-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516301#comment-16516301 ] Udi Meiri commented on BEAM-4569: - [~jbonofre] This should have been fixed in https://github.com/apache/beam/pull/5661 > Jenkins: increase build history retention to 4 weeks > > > Key: BEAM-4569 > URL: https://issues.apache.org/jira/browse/BEAM-4569 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Udi Meiri >Assignee: Jean-Baptiste Onofré >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > We would like to increase build history to 4 weeks in order to track build > time trends (we're trying to keep them fast). > Example job: > https://builds.apache.org/job/beam_PreCommit_Python_GradleBuild/buildTimeTrend > > If you scroll to the bottom of the page, the last build listed is 2 weeks > old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4569) Jenkins: increase build history retention to 4 weeks
[ https://issues.apache.org/jira/browse/BEAM-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514395#comment-16514395 ] Udi Meiri commented on BEAM-4569: - Previous INFRA issue: https://issues.apache.org/jira/browse/INFRA-16652 > Jenkins: increase build history retention to 4 weeks > > > Key: BEAM-4569 > URL: https://issues.apache.org/jira/browse/BEAM-4569 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > We would like to increase build history to 4 weeks in order to track build > time trends (we're trying to keep them fast). > Example job: > https://builds.apache.org/job/beam_PreCommit_Python_GradleBuild/buildTimeTrend > > If you scroll to the bottom of the page, the last build listed is 2 weeks > old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-4569) Jenkins: increase build history retention to 4 weeks
Udi Meiri created BEAM-4569: --- Summary: Jenkins: increase build history retention to 4 weeks Key: BEAM-4569 URL: https://issues.apache.org/jira/browse/BEAM-4569 Project: Beam Issue Type: Improvement Components: build-system Reporter: Udi Meiri Assignee: Udi Meiri We would like to increase build history to 4 weeks in order to track build time trends (we're trying to keep them fast). Example job: https://builds.apache.org/job/beam_PreCommit_Python_GradleBuild/buildTimeTrend If you scroll to the bottom of the page, the last build listed is 2 weeks old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4535) Python tests are failing for Windows
[ https://issues.apache.org/jira/browse/BEAM-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510401#comment-16510401 ] Udi Meiri commented on BEAM-4535: - Fixed in https://github.com/apache/beam/pull/5476 > Python tests are failing for Windows > > > Key: BEAM-4535 > URL: https://issues.apache.org/jira/browse/BEAM-4535 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Udi Meiri >Priority: Major > > Error is: > Traceback (most recent call last): > File > "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba > sedsource_test.py", line 532, in test_read_auto_pattern > compression_type=CompressionTypes.AUTO)) > File > "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba > sedsource.py", line 119, in __init__ > self._validate() > File > "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\options\v > alue_provider.py", line 133, in _f > return fnc(self, *args, **kwargs) > File > "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba > sedsource.py", line 179, in _validate > 'No files found based on the file pattern %s' % pattern) > IOError: No files found based on the file pattern > c:\windows\temp\tmpwon5_g\mytemp* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
