Thank you Luke and Reuven for helping me. Now I can see my pipeline
processing time for each record.
On Wed, Jun 3, 2020 at 9:25 AM Reuven Lax wrote:
> Note: you need to tag the timestamp parameter to @ProcessElement with
> the @Timestamp annotation.
>
> On Mon, Jun 1, 2020 at 3:31 PM Luke Cwik
Note: you need to tag the timestamp parameter to @ProcessElement with
the @Timestamp annotation.
On Mon, Jun 1, 2020 at 3:31 PM Luke Cwik wrote:
> You can configure KafkaIO to use some data from the record as the elements
> timestamp. See the KafkaIO javadoc around the TimestampPolicy[1], the
>
You can configure KafkaIO to use some data from the record as the elements
timestamp. See the KafkaIO javadoc around the TimestampPolicy[1], the
default is current processing time.
You can access the timestamp of the element by adding
"org.joda.time.Instant timestamp" as a parameter to your
Sorry for the late response. Where does the beam set that timestamp field
on element ? Is it set whenever KafkaIO reads that element ? And also I
have a windowing function on my pipeline. Does the timestamp field change
for any kind of operation ? On pipeline I have the following steps: KafkaIO
->
Dataflow provides msec counters for each transform that executes. You
should be able to get them from stackdriver and see them from the Dataflow
UI.
You need to keep track of the timestamp of the element as it flows through
the system as part of data that goes alongside the element. You can use
Yes I am trying to track how long it takes for a single element to be
ingested into the pipeline until it is output somewhere.
My pipeline is unbounded. I am using KafkaIO. I did not think about CPU
time. if there is a way to track it too, it would be useful to improve my
metrics.
On Thu, May
What do you mean by processing time?
Are you trying to track how long it takes for a single element to be
ingested into the pipeline until it is output somewhere?
Do you have a bounded pipeline and want to know how long all the processing
takes?
Do you care about how much CPU time is being
I am using Dataflow Runner. The pipeline read from kafkaIO and send Http. I
could not find any metadata field on the element to set first read time.
On Thu, May 28, 2020 at 10:44 AM Kyle Weaver wrote:
> Which runner are you using?
>
> On Thu, May 28, 2020 at 1:43 PM Talat Uyarer
> wrote:
>
>>
Which runner are you using?
On Thu, May 28, 2020 at 1:43 PM Talat Uyarer
wrote:
> Hi,
>
> I have a pipeline which has 5 steps. What is the best way to measure
> processing time for my pipeline?
>
> Thnaks
>
