Unfortunately, I can't use a side input, because I need to use the result
as an input to a DatastoreV1.Read object. This is a PTransform rather than
DoFn, so there is no place for a side input.
But you did give me the hint I needed to get something that is close
enough. It's a bit dirty, but it's what I'll have to use until something
better comes along.
public interface MyOptions extends PipelineOptions,
DataflowPipelineOptions {
@Description("ignore")
@Default.Integer(0)
ValueProvider<Integer> getIgnored();
void setIgnored(ValueProvider<Integer> value);
}
ValueProvider<DateTime> now =
NestedValueProvider.of(options.getIgnored(), ignored ->
DateTime.now(DateTimeZone.UTC));
The fact that options.getIgnored() is not static means that the value of
now can't be either. Each time I call now.get(), the value will be
re-evaluated. This isn't quite what I wanted, (I was hoping for a single
evaluation), but it will work well enough for my situation.
On Thu, May 3, 2018 at 6:01 PM, Eugene Kirpichov <[email protected]>
wrote:
> There is no way to achieve this using ValueProvider. Its value is either
> fixed at construction time (StaticValueProvider), or completely dynamic
> (evaluated every time you call .get()).
> You'll need to implement this using a side input. E.g. take a look at
> implementation of BigQueryIO, how it generates unique job id tokens -
> https://github.com/apache/beam/blob/bf94e36f67a8bc5d24c795e40697ad
> 2504c8594c/sdks/java/io/google-cloud-platform/src/
> main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L756
>
> On Thu, May 3, 2018 at 5:42 PM Frank Yellin <[email protected]> wrote:
>
>> [Sorry, I accidentally hit send before I had finished typing . .]
>>
>> Is there any way to achieve what I'm looking for? Or is this just beyond
>> the scope of ValueProvider and templates?
>>
>>
>>
>> On Thu, May 3, 2018 at 5:36 PM, Frank Yellin <[email protected]> wrote:
>>
>>> I'm attempting to create a dataflow template, and within the template
>>> have a variable
>>> ValueProvider<DateTime> now
>>> such that now is the time the dataflow is started, note the time that
>>> the template was created.
>>>
>>> My first attempt was
>>> ValueProvider<DateTime> now = StaticValueProvider.of(
>>> DateTime.now(DateTimeZone.UTC));
>>>
>>> My second attempt was
>>>
>>> public interface MyOptions extends PipelineOptions,
>>> DataflowPipelineOptions {
>>> @Description("Now")
>>> @Default.InstanceFactory(GetNow.class)
>>> ValueProvider<DateTime> getNow();
>>> void setNow(ValueProvider<DateTime> value);
>>> }
>>>
>>> static class GetNow implements DefaultValueFactory<DateTime> {
>>> @Override
>>> public DateTime create(PipelineOptions options) {
>>> return DateTime.now(DateTimeZone.UTC);
>>> }
>>> }
>>>
>>> ValueProvider<DateTime> now = options.getNow()
>>>
>>> My final attempt was:
>>>
>>> ValueProvider<SerializableFunction<Void, DateTime>> nowFn =
>>> StaticValueProvider.of(x -> DateTime.now(DateTimeZone.UTC));
>>>
>>> ValueProvider<DateTime> now = NestedValueProvider.of(nowFn, x ->
>>> x.apply(null));
>>>
>>>
>>>
>>> In every case, it was clear that "now" was being set to
>>> template-creation time rather than actual runtime.
>>>
>>> I note that the documentation talks about a RuntimeValueProvider, but
>>> there is no user-visible constructor for this.
>>>
>>>
>>>
>>>
>>
