The Dataflow error logs only showed 1 error which was: "The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: beamapp-XXXX-XXXXX-kt85-harness-8k2c Root cause: The worker lost contact with the service." In "Diagnostics" there were errors stating "Error syncing pod: Could not start worker docker container". The harness logs i.e. "projects/my-project/logs/ dataflow.googleapis.com%2Fharness" finally contained an error that looked suspect, which was "Check failed: kv_coder : expecting a KV coder, but had Strings", below[1] is a link to possibly a stacktrace or extra detail, but is internal to google so I don't have access.
[1] https://symbolize.corp.google.com/r/?trace=55a197abcf56,55a197abbe33,55a197abb97e,55a197abd708,55a196d4e22f,55a196d4d8d3,55a196d4da35,55a1967ec247,55a196f62b26,55a1968969b3,55a196886613,55a19696b0e6,55a196969815,55a1969693eb,55a19696916e,55a1969653bc,55a196b0150a,55a196b04e11,55a1979fc8df,7fe7736674e7,7fe7734dc22c&map=13ddc0ac8b57640c29c5016eb26ef88e:55a1956e7000-55a197bd5010,f1c96c67b57b74a4d7050f34aca016eef674f765:7fe773660000-7fe773676dac,76b955c7af655a4c1e53b8d4aaa0255f3721f95f:7fe7734a5000-7fe7736464c4 On Thu, Sep 9, 2021 at 6:46 PM Robert Bradshaw <rober...@google.com> wrote: > Huh, that's strange. Yes, the exact error on the service would be helpful. > > On Wed, Sep 8, 2021 at 10:12 AM Evan Galpin <evan.gal...@gmail.com> wrote: > > > > Thanks for the response. I've created a gist here to demonstrate a > minimal repro: > https://gist.github.com/egalpin/2d6ad2210cf9f66108ff48a9c7566ebc > > > > It seemed to run fine both on DirectRunner and PortableRunner (embed > mode), but Dataflow v2 runner raised an error at runtime seemingly > associated with the Shuffle service? I have job IDs and trace links if > those are helpful as well. > > > > Thanks, > > Evan > > > > On Tue, Sep 7, 2021 at 4:35 PM Robert Bradshaw <rober...@google.com> > wrote: > >> > >> This is not yet supported. Using a union for now is the way to go. (If > >> only the last value of the union was used, that sounds like a bug. Do > >> you have a minimal repro?) > >> > >> On Tue, Sep 7, 2021 at 1:23 PM Evan Galpin <evan.gal...@gmail.com> > wrote: > >> > > >> > Hi all, > >> > > >> > What is the recommended way to write type hints for a tagged output > DoFn where the outputs to different tags have different types? > >> > > >> > I tried using a Union to describe each of the possible output types, > but that resulted in mismatched coder errors where only the last entry in > the Union was used as the assumed type. Is there a way to associate a type > hint to a tag or something like that? > >> > > >> > Thanks, > >> > Evan >
