Hi,
I am trying to run a task using an infinite while loop with change in input
parameters as below but it creates a new job for every time. (I expect it to
use same pipeline that is created before while loop and should be in a single
job)
p = beam.Pipeline(options=PipelineOptions())
while True:
to_date = time.time()
(p
| 'create surveys' >> beam.Create(id_list)
| 'get data' >> beam.FlatMap(lambda id: get_api_data(id, from_date,
to_date))
| 'send to output' >> beam.FlatMap(lambda input:
(send_to_api(input)))
)
from_date = to_date + timedelta(microseconds=1)
time.sleep(30)
p.run().wait_until_finish()
It works properly(in only one job) when there is no while loop as below:
p = beam.Pipeline(options=PipelineOptions())
(p
| 'create surveys' >> beam.Create(id_list)
| 'get data' >> beam.FlatMap(lambda id: get_api_data(id, from_date,
to_date))
| 'send to output' >> beam.FlatMap(lambda input:
(send_to_api(input)))
)
p.run().wait_until_finish()
Could someone please suggest how to make the task run in same job instead of
creating multiple jobs.
Please let me know in case if any additional information needed.
Thanks,
Anjana
-----------------------------------------------------------------------------------------------------------------------
The information contained in this communication is intended solely for the use
of the individual or entity to whom it is addressed and others authorized to
receive it. It may contain confidential or legally privileged information. If
you are not the intended recipient you are hereby notified that any disclosure,
copying, distribution or taking any action in reliance on the contents of this
information is strictly prohibited and may be unlawful. If you are not the
intended recipient, please notify us immediately by responding to this email
and then delete it from your system. Bahwan Cybertek is neither liable for the
proper and complete transmission of the information contained in this
communication nor for any delay in its receipt.
