[jira] [Created] (BEAM-3754) Can't have commitOffsetsInFinalizeEnabled set to false with KafkaIO.readBytes()
Benjamin BENOIST created BEAM-3754: -- Summary: Can't have commitOffsetsInFinalizeEnabled set to false with KafkaIO.readBytes() Key: BEAM-3754 URL: https://issues.apache.org/jira/browse/BEAM-3754 Project: Beam Issue Type: Bug Components: io-java-kafka Affects Versions: 2.3.0 Environment: Dataflow pipeline using Kafka as a Sink Reporter: Benjamin BENOIST Assignee: Raghu Angadi Beam v2.3 introduces finalized offsets, in order to reduce the gaps or duplicate processing of records while restarting a pipeline. _read()_ sets this parameter to false [by default|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L307] but _readBytes()_ [doesn't|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L282], thus creating an exception: {noformat} Exception in thread "main" java.lang.IllegalStateException: Missing required properties: commitOffsetsInFinalizeEnabled at org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Read$Builder.build(AutoValue_KafkaIO_Read.java:344) at org.apache.beam.sdk.io.kafka.KafkaIO.readBytes(KafkaIO.java:291){noformat} The parameter can be set to true with _commitOffsetsInFinalize()_ but never to false. Using _read()_ in the definition of _readBytes()_ could prevent this kind of error in the future: {code:java} public static Read readBytes() { return read() .setKeyDeserializer(ByteArrayDeserializer.class) .setValueDeserializer(ByteArrayDeserializer.class) .build(); }{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3754) KAFKA - Can't set commitOffsetsInFinalizeEnabled to false with KafkaIO.readBytes()
[ https://issues.apache.org/jira/browse/BEAM-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3754: --- Summary: KAFKA - Can't set commitOffsetsInFinalizeEnabled to false with KafkaIO.readBytes() (was: KAFKA - Can't have commitOffsetsInFinalizeEnabled set to false with KafkaIO.readBytes()) > KAFKA - Can't set commitOffsetsInFinalizeEnabled to false with > KafkaIO.readBytes() > -- > > Key: BEAM-3754 > URL: https://issues.apache.org/jira/browse/BEAM-3754 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.3.0 > Environment: Dataflow pipeline using Kafka as a Sink >Reporter: Benjamin BENOIST >Assignee: Raghu Angadi >Priority: Minor > Labels: patch > Original Estimate: 2h > Remaining Estimate: 2h > > Beam v2.3 introduces finalized offsets, in order to reduce the gaps or > duplicate processing of records while restarting a pipeline. > _read()_ sets this parameter to false [by > default|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L307] > but _readBytes()_ > [doesn't|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L282], > thus creating an exception: > {noformat} > Exception in thread "main" java.lang.IllegalStateException: Missing required > properties: commitOffsetsInFinalizeEnabled > at > org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Read$Builder.build(AutoValue_KafkaIO_Read.java:344) > at > org.apache.beam.sdk.io.kafka.KafkaIO.readBytes(KafkaIO.java:291){noformat} > The parameter can be set to true with _commitOffsetsInFinalize()_ but never > to false. > Using _read()_ in the definition of _readBytes()_ could prevent this kind of > error in the future: > {code:java} > public static Read readBytes() { > return read() > .setKeyDeserializer(ByteArrayDeserializer.class) > .setValueDeserializer(ByteArrayDeserializer.class) > .build(); > }{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3754) KAFKA - Can't have commitOffsetsInFinalizeEnabled set to false with KafkaIO.readBytes()
[ https://issues.apache.org/jira/browse/BEAM-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3754: --- Summary: KAFKA - Can't have commitOffsetsInFinalizeEnabled set to false with KafkaIO.readBytes() (was: Can't have commitOffsetsInFinalizeEnabled set to false with KafkaIO.readBytes()) > KAFKA - Can't have commitOffsetsInFinalizeEnabled set to false with > KafkaIO.readBytes() > --- > > Key: BEAM-3754 > URL: https://issues.apache.org/jira/browse/BEAM-3754 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.3.0 > Environment: Dataflow pipeline using Kafka as a Sink >Reporter: Benjamin BENOIST >Assignee: Raghu Angadi >Priority: Minor > Labels: patch > Original Estimate: 2h > Remaining Estimate: 2h > > Beam v2.3 introduces finalized offsets, in order to reduce the gaps or > duplicate processing of records while restarting a pipeline. > _read()_ sets this parameter to false [by > default|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L307] > but _readBytes()_ > [doesn't|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L282], > thus creating an exception: > {noformat} > Exception in thread "main" java.lang.IllegalStateException: Missing required > properties: commitOffsetsInFinalizeEnabled > at > org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Read$Builder.build(AutoValue_KafkaIO_Read.java:344) > at > org.apache.beam.sdk.io.kafka.KafkaIO.readBytes(KafkaIO.java:291){noformat} > The parameter can be set to true with _commitOffsetsInFinalize()_ but never > to false. > Using _read()_ in the definition of _readBytes()_ could prevent this kind of > error in the future: > {code:java} > public static Read readBytes() { > return read() > .setKeyDeserializer(ByteArrayDeserializer.class) > .setValueDeserializer(ByteArrayDeserializer.class) > .build(); > }{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3766) BigQueryIO - Must specify numFileShards when using FILE_LOADS with unbounded PCollection
Benjamin BENOIST created BEAM-3766: -- Summary: BigQueryIO - Must specify numFileShards when using FILE_LOADS with unbounded PCollection Key: BEAM-3766 URL: https://issues.apache.org/jira/browse/BEAM-3766 Project: Beam Issue Type: Bug Components: io-java-gcp Affects Versions: 2.3.0 Reporter: Benjamin BENOIST Assignee: Chamikara Jayalath Since Beam v2.2 it's possible to use [FILE_LOADS|https://beam.apache.org/documentation/sdks/javadoc/2.2.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html#FILE_LOADS]. The documentation states that we have to specify _withTriggeringFrequency_ when using it, but doesn't talk about _withNumFileShards_, whereas if we don't specify it we get the below exception: {code:java} Exception in thread "main" java.lang.IllegalArgumentException at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) at org.apache.beam.sdk.io.gcp.bigquery.BatchLoads.expandTriggered(BatchLoads.java:209) at org.apache.beam.sdk.io.gcp.bigquery.BatchLoads.expand(BatchLoads.java:546) at org.apache.beam.sdk.io.gcp.bigquery.BatchLoads.expand(BatchLoads.java:79) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472) at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expandTyped(BigQueryIO.java:1550) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expand(BigQueryIO.java:1497) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expand(BigQueryIO.java:980) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491) at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299) at com.travelaudience.data.job.rtbtobigquery.Main$.main(Main.scala:74) at com.travelaudience.data.job.rtbtobigquery.Main.main(Main.scala) {code} Either default _numFileShards_ should be used or it should be precised in the documentation that this has to be set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3772) BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS
Benjamin BENOIST created BEAM-3772: -- Summary: BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS Key: BEAM-3772 URL: https://issues.apache.org/jira/browse/BEAM-3772 Project: Beam Issue Type: Bug Components: io-java-gcp Affects Versions: 2.3.0, 2.2.0 Environment: Dataflow streaming pipeline Reporter: Benjamin BENOIST Assignee: Chamikara Jayalath My workflow : KAFKA -> Dataflow streaming -> BigQuery Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_, which is a table with the current hour as a suffix. This _BigQueryIO.Write_ is configured like this : {code:java} .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED) .withMethod(Method.FILE_LOADS) .withTriggeringFrequency(triggeringFrequency) .withNumFileShards(100) {code} The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions: {code:java} (99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job with id prefix 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_1_00023, reached max retries: 3, last failed load job: { "configuration" : { "load" : { "createDisposition" : "CREATE_NEVER", "destinationTable" : { "datasetId" : "dev_mydataset", "projectId" : "myproject-id", "tableId" : "mytable_20180302_16" }, {code} The _CreateDisposition_ used is _CREATE_NEVER_, contrary as _CREATE_IF_NEEDED_ as specified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3224) Add support for path with braces for Google Cloud Storage
Benjamin BENOIST created BEAM-3224: -- Summary: Add support for path with braces for Google Cloud Storage Key: BEAM-3224 URL: https://issues.apache.org/jira/browse/BEAM-3224 Project: Beam Issue Type: Improvement Components: sdk-java-gcp Reporter: Benjamin BENOIST Assignee: Chamikara Jayalath Priority: Minor At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage
[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3} }} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} was: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} > Add support for path with braces for Google Cloud Storage > - > > Key: BEAM-3224 > URL: https://issues.apache.org/jira/browse/BEAM-3224 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Benjamin BENOIST >Assignee: Chamikara Jayalath >Priority: Minor > Original Estimate: 3h > Remaining Estimate: 3h > > At the moment we can not use braces in Google Cloud Storage paths, as > explained > [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. > The path is backed by a file pattern defined as a Java glob and is then then > expanded to a regex in > _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ > in the _wildcardToRegexp_ function. > {{gs://bucket/{file1,file2,file3} }} should match {{gs://bucket/file1}}, > {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage
[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3}. }} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} was: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3} }} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} > Add support for path with braces for Google Cloud Storage > - > > Key: BEAM-3224 > URL: https://issues.apache.org/jira/browse/BEAM-3224 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Benjamin BENOIST >Assignee: Chamikara Jayalath >Priority: Minor > Original Estimate: 3h > Remaining Estimate: 3h > > At the moment we can not use braces in Google Cloud Storage paths, as > explained > [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. > The path is backed by a file pattern defined as a Java glob and is then then > expanded to a regex in > _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ > in the _wildcardToRegexp_ function. > {{gs://bucket/{file1,file2,file3}. }} should match {{gs://bucket/file1}}, > {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage
[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/\{file1,file2,file3\}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} was: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3}. }} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} > Add support for path with braces for Google Cloud Storage > - > > Key: BEAM-3224 > URL: https://issues.apache.org/jira/browse/BEAM-3224 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Benjamin BENOIST >Assignee: Chamikara Jayalath >Priority: Minor > Original Estimate: 3h > Remaining Estimate: 3h > > At the moment we can not use braces in Google Cloud Storage paths, as > explained > [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. > The path is backed by a file pattern defined as a Java glob and is then then > expanded to a regex in > _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ > in the _wildcardToRegexp_ function. > {{gs://bucket/\{file1,file2,file3\}}} should match {{gs://bucket/file1}}, > {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage
[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/\{file1,file2,file3\\}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} was: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/\{file1,file2,file3\} }} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} > Add support for path with braces for Google Cloud Storage > - > > Key: BEAM-3224 > URL: https://issues.apache.org/jira/browse/BEAM-3224 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Benjamin BENOIST >Assignee: Chamikara Jayalath >Priority: Minor > Original Estimate: 3h > Remaining Estimate: 3h > > At the moment we can not use braces in Google Cloud Storage paths, as > explained > [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. > The path is backed by a file pattern defined as a Java glob and is then then > expanded to a regex in > _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ > in the _wildcardToRegexp_ function. > {{gs://bucket/\{file1,file2,file3\\}}} should match {{gs://bucket/file1}}, > {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage
[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/\{file1,file2,file3\} }} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} was: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/\{file1,file2,file3\}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} > Add support for path with braces for Google Cloud Storage > - > > Key: BEAM-3224 > URL: https://issues.apache.org/jira/browse/BEAM-3224 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Benjamin BENOIST >Assignee: Chamikara Jayalath >Priority: Minor > Original Estimate: 3h > Remaining Estimate: 3h > > At the moment we can not use braces in Google Cloud Storage paths, as > explained > [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. > The path is backed by a file pattern defined as a Java glob and is then then > expanded to a regex in > _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ > in the _wildcardToRegexp_ function. > {{gs://bucket/\{file1,file2,file3\} }} should match {{gs://bucket/file1}}, > {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage
[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/{file1,file2,file3}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} was: At the moment we can not use braces in Google Cloud Storage paths, as explained [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. The path is backed by a file pattern defined as a Java glob and is then then expanded to a regex in _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ in the _wildcardToRegexp_ function. {{gs://bucket/\{file1,file2,file3\\}}} should match {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} > Add support for path with braces for Google Cloud Storage > - > > Key: BEAM-3224 > URL: https://issues.apache.org/jira/browse/BEAM-3224 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Benjamin BENOIST >Assignee: Chamikara Jayalath >Priority: Minor > Original Estimate: 3h > Remaining Estimate: 3h > > At the moment we can not use braces in Google Cloud Storage paths, as > explained > [here|https://stackoverflow.com/questions/46977552/filebasedsource-not-able-to-understand-a-glob-corresponding-to-several-specific]. > The path is backed by a file pattern defined as a Java glob and is then then > expanded to a regex in > _sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java_ > in the _wildcardToRegexp_ function. > {{gs://bucket/{file1,file2,file3}}} should match > {{gs://bucket/file1}}, {{gs://bucket/file2}} and {{gs://bucket/file3}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3772) BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS
[ https://issues.apache.org/jira/browse/BEAM-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405139#comment-16405139 ] Benjamin BENOIST commented on BEAM-3772: I had the issue with both Beam 2.2.0 and 2.3.0 > BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded > PCollection and FILE_LOADS > > > Key: BEAM-3772 > URL: https://issues.apache.org/jira/browse/BEAM-3772 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.2.0, 2.3.0 > Environment: Dataflow streaming pipeline >Reporter: Benjamin BENOIST >Assignee: Eugene Kirpichov >Priority: Major > > My workflow : KAFKA -> Dataflow streaming -> BigQuery > Given that having low-latency isn't important in my case, I use FILE_LOADS to > reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_, > which is a table with the current hour as a suffix. > This _BigQueryIO.Write_ is configured like this : > {code:java} > .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED) > .withMethod(Method.FILE_LOADS) > .withTriggeringFrequency(triggeringFrequency) > .withNumFileShards(100) > {code} > The first table is successfully created and is written to. But then the > following tables are never created and I get these exceptions: > {code:java} > (99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job > with id prefix > 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_1_00023, > reached max retries: 3, last failed load job: { > "configuration" : { > "load" : { > "createDisposition" : "CREATE_NEVER", > "destinationTable" : { > "datasetId" : "dev_mydataset", > "projectId" : "myproject-id", > "tableId" : "mytable_20180302_16" > }, > {code} > The _CreateDisposition_ used is _CREATE_NEVER_, contrary as > _CREATE_IF_NEEDED_ as specified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)