Hello fellow Apache Beam users,
I am trying to copy datastore entities to a PostgreSQL instance. As I
don't need each fields, I performed projections following [this
snippet][1]. I build the following query:
public static Query DatastoreQuery() {
Query.Builder query = Query.newBuilder();
// Add filter
query.addKindBuilder().setName("FOO");
query.setFilter(makeFilter("bar",
PropertyFilter.Operator.EQUAL, makeValue("fuz")));
// Add projections
query.addProjection(Projection.newBuilder().setProperty(PropertyReference.newBuilder().setName("createdAt")));
return query.build();
}
This query is then used in the pipeline:
pipeline.apply(DatastoreIO.v1().read().withProjectId(options.getProjectId())
.withQuery(ExtractDatastore.DatastoreQuery()));
Following
https://stackoverflow.com/questions/44987493/unable-to-addprojection-to-relation-field-in-envers-query,
I am expecting to get a `Map<String, Object>`. I would like to follow
[Apache Beam documentation][2] to insert the entities in PostgreSQL,
using a code similar to:
.apply(JdbcIO.<Map<String, Object>>write()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(
"com.google.cloud.sql.postgres",
jdbcUrl
)
.withUsername(username)
.withPassword(password))
.withQuery("<INSERT QUERY>")
.withPreparedStatementSetter(<this is what needs to be filled>));
As I have found no examples, here are my questions are the following:
1) How to handle the elements from the query ? In the example of my
projection, how do I get access to `createdAt` which is a timestamp ? In
python, I did it with : `value.timestamp_value.ToDatetime()`. Is there
an equivalent in Java ?
2) Are the entities really under the format `<Map<String, Object>>` as
described in the SO issue ?
3) Can one apply JdbcIO.write() to `Map<String, Object>` or does it
require to be under the `<KV<KeyType, ValueType>>` format as described
in the small snippet of the documentation ?
I am grateful to take any inputs on this topic.
Best regards,
Jonathan
[1]:
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-examples/src/main/java/com/google/cl%60enter%20code%20here%60oud/examples/datastore/snippets/QuerySnippets.java#L111
[2]: https://beam.apache.org/releases/javadoc/2.8.0/
