[jira] [Created] (BEAM-1137) Empty string values should be allowed for ValueProvider of all supported types (Collection, Array, Enum)
Vikas Kedigehalli created BEAM-1137: --- Summary: Empty string values should be allowed for ValueProvider of all supported types (Collection, Array, Enum) Key: BEAM-1137 URL: https://issues.apache.org/jira/browse/BEAM-1137 Project: Beam Issue Type: Improvement Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1137) Empty string values should be allowed for ValueProvider of all supported types (Collection, Array, Enum)
[ https://issues.apache.org/jira/browse/BEAM-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli updated BEAM-1137: Component/s: sdk-java-core > Empty string values should be allowed for ValueProvider of all supported > types (Collection, Array, Enum) > > > Key: BEAM-1137 > URL: https://issues.apache.org/jira/browse/BEAM-1137 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1136) Empty string value should be allowed for ValueProvider
Vikas Kedigehalli created BEAM-1136: --- Summary: Empty string value should be allowed for ValueProvider Key: BEAM-1136 URL: https://issues.apache.org/jira/browse/BEAM-1136 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1080) python sdk apiclient needs proper unit tests
Vikas Kedigehalli created BEAM-1080: --- Summary: python sdk apiclient needs proper unit tests Key: BEAM-1080 URL: https://issues.apache.org/jira/browse/BEAM-1080 Project: Beam Issue Type: New Feature Components: sdk-py Reporter: Vikas Kedigehalli Assignee: Frances Perry There is only one unit test right now that tries to fetch actual gcp credentials instead of mocking. This test fails when the credentials are not available on the machine in which it is running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-993) Python SDK build broken due to datastore dependency
[ https://issues.apache.org/jira/browse/BEAM-993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-993. Resolution: Fixed Fix Version/s: Not applicable > Python SDK build broken due to datastore dependency > --- > > Key: BEAM-993 > URL: https://issues.apache.org/jira/browse/BEAM-993 > Project: Beam > Issue Type: Bug > Components: sdk-py >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: Not applicable > > > https://github.com/apache/incubator-beam/pull/1310 introduced the > googledatastore dependency, which then depends on pyOpenSSL. pyOpenSSL pip > install runs into issues of not having c modules like libffi. Working with > datastore team to get this fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1044) tests run before install fails
[ https://issues.apache.org/jira/browse/BEAM-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716158#comment-15716158 ] Vikas Kedigehalli commented on BEAM-1044: - A new googledatastore release fixes this issue. https://github.com/apache/incubator-beam/pull/1454 > tests run before install fails > -- > > Key: BEAM-1044 > URL: https://issues.apache.org/jira/browse/BEAM-1044 > Project: Beam > Issue Type: Bug > Components: sdk-py >Reporter: Ahmet Altay >Assignee: Vikas Kedigehalli > > `python setup.py test` fails for datastore tests when run in a new virtual > environment. Running `python setup.py install` fixes the problem but that > should not be necessary. Stack for one of the failing tests: > == > ERROR: Failure: ImportError (cannot import name descriptor) > -- > Traceback (most recent call last): > File > "/usr/local/google/home/altay/Desktop/beam/temp/incubator-beam/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/loader.py", > line 418, in loadTestsFromName > addr.filename, addr.module) > File > "/usr/local/google/home/altay/Desktop/beam/temp/incubator-beam/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/importer.py", > line 47, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/usr/local/google/home/altay/Desktop/beam/temp/incubator-beam/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/importer.py", > line 94, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/usr/local/google/home/altay/Desktop/beam/temp/incubator-beam/sdks/python/apache_beam/io/datastore/v1/query_splitter_test.py", > line 25, in > from apache_beam.io.datastore.v1 import fake_datastore > File > "/usr/local/google/home/altay/Desktop/beam/temp/incubator-beam/sdks/python/apache_beam/io/datastore/v1/fake_datastore.py", > line 21, in > from google.datastore.v1 import datastore_pb2 > File "build/bdist.linux-x86_64/egg/google/datastore/v1/datastore_pb2.py", > line 6, in > ImportError: cannot import name descriptor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1076) Datastore Delete template
Vikas Kedigehalli created BEAM-1076: --- Summary: Datastore Delete template Key: BEAM-1076 URL: https://issues.apache.org/jira/browse/BEAM-1076 Project: Beam Issue Type: New Feature Components: sdk-java-gcp Affects Versions: 0.4.0-incubating Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1069) Add CountingInput Transform to python sdk
Vikas Kedigehalli created BEAM-1069: --- Summary: Add CountingInput Transform to python sdk Key: BEAM-1069 URL: https://issues.apache.org/jira/browse/BEAM-1069 Project: Beam Issue Type: Improvement Components: sdk-py Reporter: Vikas Kedigehalli Assignee: Frances Perry Priority: Minor Similar to java sdk, https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-1021) DatastoreIO for python
Vikas Kedigehalli created BEAM-1021: --- Summary: DatastoreIO for python Key: BEAM-1021 URL: https://issues.apache.org/jira/browse/BEAM-1021 Project: Beam Issue Type: New Feature Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-992) Python SDK build broken due to datastore dependency
[ https://issues.apache.org/jira/browse/BEAM-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-992. Resolution: Fixed Fix Version/s: Not applicable > Python SDK build broken due to datastore dependency > --- > > Key: BEAM-992 > URL: https://issues.apache.org/jira/browse/BEAM-992 > Project: Beam > Issue Type: Bug > Components: sdk-py >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: Not applicable > > > https://github.com/apache/incubator-beam/pull/1310 introduced the > googledatastore dependency, which then depends on pyOpenSSL. pyOpenSSL pip > install runs into issues of not having c modules like libffi. Working with > datastore team to get this fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-943) Implement Datastore query splitter for python
[ https://issues.apache.org/jira/browse/BEAM-943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-943. Resolution: Fixed Fix Version/s: Not applicable > Implement Datastore query splitter for python > - > > Key: BEAM-943 > URL: https://issues.apache.org/jira/browse/BEAM-943 > Project: Beam > Issue Type: New Feature > Components: sdk-py >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BEAM-991) DatastoreIO Write should flush early for large batches
[ https://issues.apache.org/jira/browse/BEAM-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679617#comment-15679617 ] Vikas Kedigehalli edited comment on BEAM-991 at 11/19/16 5:53 PM: -- Joshua, all good solutions. I would prefer 3rd one, using 'getSerializedSize' to measure the approximate byte size and flush when it reaches ~10MB (https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L863) Computing getSerializedSize shouldn't be problem because that value is memoized by protobuf and protobuf will anyway compute that later for serializing, so we shouldn't hit any additional performance penalty. PS: You more than welcome to submit a Pull Request to Apache Beam if you are interested to contribute. :) was (Author: vikasrk): Joshua, all good solutions. I would prefer 3rd one, using 'getSerializedSize' to measure the approximate byte size and flush when it reaches ~10MB (https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L863) Computing getSerializedSize shouldn't be problem because that value is memoized by protobuf and protobuf will anyway compute that later for serializing, so we shouldn't hit any additional performance penalty. PS: If you more than welcome to submit a Pull Request to Apache Beam if you are interested to contribute. :) > DatastoreIO Write should flush early for large batches > -- > > Key: BEAM-991 > URL: https://issues.apache.org/jira/browse/BEAM-991 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > > If entities are large (avg size > 20KB) then the a single batched write (500 > entities) would exceed the Datastore size limit of a single request (10MB) > from https://cloud.google.com/datastore/docs/concepts/limits. > First reported in: > http://stackoverflow.com/questions/40156400/why-does-dataflow-erratically-fail-in-datastore-access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-991) DatastoreIO Write should flush early for large batches
[ https://issues.apache.org/jira/browse/BEAM-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679617#comment-15679617 ] Vikas Kedigehalli commented on BEAM-991: Joshua, all good solutions. I would prefer 3rd one, using 'getSerializedSize' to measure the approximate byte size and flush when it reaches ~10MB (https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L863) Computing getSerializedSize shouldn't be problem because that value is memoized by protobuf and protobuf will anyway compute that later for serializing, so we shouldn't hit any additional performance penalty. PS: If you more than welcome to submit a Pull Request to Apache Beam if you are interested to contribute. :) > DatastoreIO Write should flush early for large batches > -- > > Key: BEAM-991 > URL: https://issues.apache.org/jira/browse/BEAM-991 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > > If entities are large (avg size > 20KB) then the a single batched write (500 > entities) would exceed the Datastore size limit of a single request (10MB) > from https://cloud.google.com/datastore/docs/concepts/limits. > First reported in: > http://stackoverflow.com/questions/40156400/why-does-dataflow-erratically-fail-in-datastore-access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-992) Python SDK build broken due to datastore dependency
Vikas Kedigehalli created BEAM-992: -- Summary: Python SDK build broken due to datastore dependency Key: BEAM-992 URL: https://issues.apache.org/jira/browse/BEAM-992 Project: Beam Issue Type: Bug Components: sdk-py Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli https://github.com/apache/incubator-beam/pull/1310 introduced the googledatastore dependency, which then depends on pyOpenSSL. pyOpenSSL pip install runs into issues of not having c modules like libffi. Working with datastore team to get this fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-993) Python SDK build broken due to datastore dependency
Vikas Kedigehalli created BEAM-993: -- Summary: Python SDK build broken due to datastore dependency Key: BEAM-993 URL: https://issues.apache.org/jira/browse/BEAM-993 Project: Beam Issue Type: Bug Components: sdk-py Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli https://github.com/apache/incubator-beam/pull/1310 introduced the googledatastore dependency, which then depends on pyOpenSSL. pyOpenSSL pip install runs into issues of not having c modules like libffi. Working with datastore team to get this fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-991) DatastoreIO Write should flush early for large batches
Vikas Kedigehalli created BEAM-991: -- Summary: DatastoreIO Write should flush early for large batches Key: BEAM-991 URL: https://issues.apache.org/jira/browse/BEAM-991 Project: Beam Issue Type: Bug Components: sdk-java-gcp Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli If entities are large (avg size > 20KB) then the a single batched write (500 entities) would exceed the Datastore size limit of a single request (10MB) from https://cloud.google.com/datastore/docs/concepts/limits. First reported in: http://stackoverflow.com/questions/40156400/why-does-dataflow-erratically-fail-in-datastore-access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-976) Update examples README.md to fix instructions to run pipelines
Vikas Kedigehalli created BEAM-976: -- Summary: Update examples README.md to fix instructions to run pipelines Key: BEAM-976 URL: https://issues.apache.org/jira/browse/BEAM-976 Project: Beam Issue Type: Task Components: examples-java Reporter: Vikas Kedigehalli Assignee: Kenneth Knowles Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-943) Implement Datastore query splitter for python
Vikas Kedigehalli created BEAM-943: -- Summary: Implement Datastore query splitter for python Key: BEAM-943 URL: https://issues.apache.org/jira/browse/BEAM-943 Project: Beam Issue Type: New Feature Components: sdk-py Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-872) Add ProtoCoder in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-872. Resolution: Fixed Fix Version/s: Not applicable > Add ProtoCoder in Python SDK > > > Key: BEAM-872 > URL: https://issues.apache.org/jira/browse/BEAM-872 > Project: Beam > Issue Type: Improvement > Components: sdk-py >Affects Versions: Not applicable >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli >Priority: Minor > Fix For: Not applicable > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-923) Add localhost option for DatastoreIO
Vikas Kedigehalli created BEAM-923: -- Summary: Add localhost option for DatastoreIO Key: BEAM-923 URL: https://issues.apache.org/jira/browse/BEAM-923 Project: Beam Issue Type: New Feature Components: sdk-java-gcp Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli Priority: Minor Adding locahost option enables to connect to a Datastore emulator that is running locally -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-872) Add ProtoCoder in Python SDK
Vikas Kedigehalli created BEAM-872: -- Summary: Add ProtoCoder in Python SDK Key: BEAM-872 URL: https://issues.apache.org/jira/browse/BEAM-872 Project: Beam Issue Type: Improvement Components: sdk-py Affects Versions: Not applicable Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-823) Improve DatastoreIO Documentation
Vikas Kedigehalli created BEAM-823: -- Summary: Improve DatastoreIO Documentation Key: BEAM-823 URL: https://issues.apache.org/jira/browse/BEAM-823 Project: Beam Issue Type: Improvement Components: sdk-java-gcp Affects Versions: 0.3.0-incubating Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli Priority: Minor Few things to be added to the DatastoreIO documentation, 1. Inequality Filter queries are not splittable. 2. Clarify Source is Batch only, while Sink support both Batch and Streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-473) DatastoreIO querying latest timestamp statistics doesn't work with namespace
[ https://issues.apache.org/jira/browse/BEAM-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-473. Resolution: Fixed Fix Version/s: 0.3.0-incubating > DatastoreIO querying latest timestamp statistics doesn't work with namespace > > > Key: BEAM-473 > URL: https://issues.apache.org/jira/browse/BEAM-473 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.3.0-incubating > > > http://stackoverflow.com/questions/38463776/defect-in-datastoreio-read-kind-within-a-namespace > When querying for latest timestamp stats, DatastoreIO doesn't pick the right > stat Kind name when namespace is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-535) Cloud Datastore naming clean-up
[ https://issues.apache.org/jira/browse/BEAM-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-535. Resolution: Fixed Fix Version/s: 0.3.0-incubating > Cloud Datastore naming clean-up > --- > > Key: BEAM-535 > URL: https://issues.apache.org/jira/browse/BEAM-535 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Affects Versions: 0.2.0-incubating >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli >Priority: Minor > Fix For: 0.3.0-incubating > > > Clean up documentation to be always use "Cloud Datastore" instead of > "Datastore" and make it consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-580) Add a Datastore delete example
Vikas Kedigehalli created BEAM-580: -- Summary: Add a Datastore delete example Key: BEAM-580 URL: https://issues.apache.org/jira/browse/BEAM-580 Project: Beam Issue Type: Improvement Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-571) Migrate DatastoreIO from v1beta3 to v1 version
[ https://issues.apache.org/jira/browse/BEAM-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-571. Resolution: Fixed Fix Version/s: 0.3.0-incubating > Migrate DatastoreIO from v1beta3 to v1 version > -- > > Key: BEAM-571 > URL: https://issues.apache.org/jira/browse/BEAM-571 > Project: Beam > Issue Type: Bug >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.3.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-571) Migrate DatastoreIO from v1beta3 to v1 version
Vikas Kedigehalli created BEAM-571: -- Summary: Migrate DatastoreIO from v1beta3 to v1 version Key: BEAM-571 URL: https://issues.apache.org/jira/browse/BEAM-571 Project: Beam Issue Type: Bug Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-565) Datastore Sink should support deletes
[ https://issues.apache.org/jira/browse/BEAM-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-565. Resolution: Fixed Fix Version/s: 0.3.0-incubating > Datastore Sink should support deletes > - > > Key: BEAM-565 > URL: https://issues.apache.org/jira/browse/BEAM-565 > Project: Beam > Issue Type: Improvement >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.3.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-565) Datastore Sink should support deletes
Vikas Kedigehalli created BEAM-565: -- Summary: Datastore Sink should support deletes Key: BEAM-565 URL: https://issues.apache.org/jira/browse/BEAM-565 Project: Beam Issue Type: Improvement Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-550) Datastore should support writes for Unbounded PCollections
[ https://issues.apache.org/jira/browse/BEAM-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-550. Resolution: Fixed Fix Version/s: 0.3.0-incubating > Datastore should support writes for Unbounded PCollections > --- > > Key: BEAM-550 > URL: https://issues.apache.org/jira/browse/BEAM-550 > Project: Beam > Issue Type: Bug >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.3.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-559) DoFnTester should handle Setup / TearDown
[ https://issues.apache.org/jira/browse/BEAM-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli updated BEAM-559: --- Assignee: Daniel Halperin > DoFnTester should handle Setup / TearDown > - > > Key: BEAM-559 > URL: https://issues.apache.org/jira/browse/BEAM-559 > Project: Beam > Issue Type: Improvement >Reporter: Vikas Kedigehalli >Assignee: Daniel Halperin > > Now that DoFn supports setup and teardown, it would be nice for DoFnTester to > add them to its lifecycle so as to avoid calling these methods explicitly in > DoFn unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-559) DoFnTester should handle Setup / TearDown
Vikas Kedigehalli created BEAM-559: -- Summary: DoFnTester should handle Setup / TearDown Key: BEAM-559 URL: https://issues.apache.org/jira/browse/BEAM-559 Project: Beam Issue Type: Improvement Reporter: Vikas Kedigehalli Now that DoFn supports setup and teardown, it would be nice for DoFnTester to add them to its lifecycle so as to avoid calling these methods explicitly in DoFn unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-550) Datastore should support writes for Unbounded PCollections
Vikas Kedigehalli created BEAM-550: -- Summary: Datastore should support writes for Unbounded PCollections Key: BEAM-550 URL: https://issues.apache.org/jira/browse/BEAM-550 Project: Beam Issue Type: Bug Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-533) Autocomplete Example should use Datastore AncestorKey for strong consistency
[ https://issues.apache.org/jira/browse/BEAM-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-533. Resolution: Fixed Fix Version/s: 0.3.0-incubating > Autocomplete Example should use Datastore AncestorKey for strong consistency > > > Key: BEAM-533 > URL: https://issues.apache.org/jira/browse/BEAM-533 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.3.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-535) Cloud Datastore naming clean-up
Vikas Kedigehalli created BEAM-535: -- Summary: Cloud Datastore naming clean-up Key: BEAM-535 URL: https://issues.apache.org/jira/browse/BEAM-535 Project: Beam Issue Type: Improvement Components: sdk-java-gcp Affects Versions: 0.2.0-incubating Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli Priority: Minor Clean up documentation to be always use "Cloud Datastore" instead of "Datastore" and make it consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-533) Autocomplete Example should use Datastore AncestorKey for strong consistency
Vikas Kedigehalli created BEAM-533: -- Summary: Autocomplete Example should use Datastore AncestorKey for strong consistency Key: BEAM-533 URL: https://issues.apache.org/jira/browse/BEAM-533 Project: Beam Issue Type: Bug Components: examples-java Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-484) Datastore Source should support Dynamic Splitting
[ https://issues.apache.org/jira/browse/BEAM-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-484. Resolution: Fixed Fix Version/s: 0.2.0-incubating > Datastore Source should support Dynamic Splitting > - > > Key: BEAM-484 > URL: https://issues.apache.org/jira/browse/BEAM-484 > Project: Beam > Issue Type: Improvement >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.2.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-484) Datastore Source should support Dynamic Splitting
Vikas Kedigehalli created BEAM-484: -- Summary: Datastore Source should support Dynamic Splitting Key: BEAM-484 URL: https://issues.apache.org/jira/browse/BEAM-484 Project: Beam Issue Type: Improvement Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API
[ https://issues.apache.org/jira/browse/BEAM-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-354. Resolution: Fixed Fix Version/s: 0.2.0-incubating > Modify DatastoreIO to use Datastore v1beta3 API > --- > > Key: BEAM-354 > URL: https://issues.apache.org/jira/browse/BEAM-354 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Affects Versions: 0.2.0-incubating >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.2.0-incubating > > > Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the > DatastoreIO needs to be migrated to use the new version. Also in the process > of doing so, this is a good time to add a level of indirection via a > PTranform such that future changes in Datastore API would not result in > changing user/pipeline code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-473) DatastoreIO querying latest timestamp statistics doesn't work with namespace
Vikas Kedigehalli created BEAM-473: -- Summary: DatastoreIO querying latest timestamp statistics doesn't work with namespace Key: BEAM-473 URL: https://issues.apache.org/jira/browse/BEAM-473 Project: Beam Issue Type: Bug Components: sdk-java-gcp Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli http://stackoverflow.com/questions/38463776/defect-in-datastoreio-read-kind-within-a-namespace When querying for latest timestamp stats, DatastoreIO doesn't pick the right stat Kind name when namespace is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-379) DisplayDataEvaluator does not support source transforms of the form PTransform
[ https://issues.apache.org/jira/browse/BEAM-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli resolved BEAM-379. Resolution: Fixed Fix Version/s: 0.2.0-incubating > DisplayDataEvaluator does not support source transforms of the form > PTransform > - > > Key: BEAM-379 > URL: https://issues.apache.org/jira/browse/BEAM-379 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > Fix For: 0.2.0-incubating > > > DisplayDataEvaluator > (https://github.com/apache/incubator-beam/blob/c0efe568e5291298c1394016a12e7979b37afc44/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataEvaluator.java#L81) > takes PTranform, ? extends POutput>, but this > doesn't work for source transforms of the form PTransform PCollection>. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-379) DisplayDataEvaluator does not support source transforms of the form PTransform
Vikas Kedigehalli created BEAM-379: -- Summary: DisplayDataEvaluator does not support source transforms of the form PTransform Key: BEAM-379 URL: https://issues.apache.org/jira/browse/BEAM-379 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Vikas Kedigehalli Assignee: Vikas Kedigehalli DisplayDataEvaluator (https://github.com/apache/incubator-beam/blob/c0efe568e5291298c1394016a12e7979b37afc44/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataEvaluator.java#L81) takes PTranform, ? extends POutput>, but this doesn't work for source transforms of the form PTransform>. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API
[ https://issues.apache.org/jira/browse/BEAM-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334704#comment-15334704 ] Vikas Kedigehalli commented on BEAM-354: I propose to implement this in phases (as small well-defined Pull Requests). Roughly outlining the steps here, 1. Modify DatastoreIO to use the v1beta3 API and update tests&examples to reflect the changes. 2. Add PTransforms, DatastoreIO.V1Beta3.Read and .Write and wrap the DatastoreIO Source/Sink. Modify examples accordingly 3. Add end-to-end tests, add more documentation, validate maven dependencies etc. > Modify DatastoreIO to use Datastore v1beta3 API > --- > > Key: BEAM-354 > URL: https://issues.apache.org/jira/browse/BEAM-354 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > > Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the > DatastoreIO needs to be migrated to use the new version. Also in the process > of doing so, this is a good time to add a level of indirection via a > PTranform such that future changes in Datastore API would not result in > changing user/pipeline code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API
Vikas Kedigehalli created BEAM-354: -- Summary: Modify DatastoreIO to use Datastore v1beta3 API Key: BEAM-354 URL: https://issues.apache.org/jira/browse/BEAM-354 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Vikas Kedigehalli Assignee: Davor Bonaci Priority: Minor Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the DatastoreIO needs to be migrated to use the new version. Also in the process of doing so, this is a good time to add a level of indirection via a PTranform such that future changes in Datastore API would not result in user/pipeline code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API
[ https://issues.apache.org/jira/browse/BEAM-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli updated BEAM-354: --- Description: Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the DatastoreIO needs to be migrated to use the new version. Also in the process of doing so, this is a good time to add a level of indirection via a PTranform such that future changes in Datastore API would not result in changing user/pipeline code. (was: Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the DatastoreIO needs to be migrated to use the new version. Also in the process of doing so, this is a good time to add a level of indirection via a PTranform such that future changes in Datastore API would not result in user/pipeline code. ) > Modify DatastoreIO to use Datastore v1beta3 API > --- > > Key: BEAM-354 > URL: https://issues.apache.org/jira/browse/BEAM-354 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Vikas Kedigehalli >Assignee: Davor Bonaci >Priority: Minor > > Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the > DatastoreIO needs to be migrated to use the new version. Also in the process > of doing so, this is a good time to add a level of indirection via a > PTranform such that future changes in Datastore API would not result in > changing user/pipeline code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
