This is badly plaguing a pipeline I'm currently developing, where the exact same data set and code runs end-to-end on DirectRunner, but fails on DataflowRunner with either "Check failed: kv_coder : expecting a KV coder, but had Strings" or "Check failed: pair_coder Strings" hidden in the harness logs. It seems to be consistently repeatable with any TaggedOutput + GBK afterwards.
Any advice on how to proceed? Thanks, Evan On Fri, Sep 17, 2021 at 11:20 AM Evan Galpin <evan.gal...@gmail.com> wrote: > The Dataflow error logs only showed 1 error which was: "The job failed > because a work item has failed 4 times. Look in previous log entries for > the cause of each one of the 4 failures. For more information, see > https://cloud.google.com/dataflow/docs/guides/common-errors. The work > item was attempted on these workers: beamapp-XXXX-XXXXX-kt85-harness-8k2c > Root cause: The worker lost contact with the service." In "Diagnostics" > there were errors stating "Error syncing pod: Could not start worker docker > container". The harness logs i.e. "projects/my-project/logs/ > dataflow.googleapis.com%2Fharness" finally contained an error that looked > suspect, which was "Check failed: kv_coder : expecting a KV coder, but > had Strings", below[1] is a link to possibly a stacktrace or extra detail, > but is internal to google so I don't have access. > > [1] > https://symbolize.corp.google.com/r/?trace=55a197abcf56,55a197abbe33,55a197abb97e,55a197abd708,55a196d4e22f,55a196d4d8d3,55a196d4da35,55a1967ec247,55a196f62b26,55a1968969b3,55a196886613,55a19696b0e6,55a196969815,55a1969693eb,55a19696916e,55a1969653bc,55a196b0150a,55a196b04e11,55a1979fc8df,7fe7736674e7,7fe7734dc22c&map=13ddc0ac8b57640c29c5016eb26ef88e:55a1956e7000-55a197bd5010,f1c96c67b57b74a4d7050f34aca016eef674f765:7fe773660000-7fe773676dac,76b955c7af655a4c1e53b8d4aaa0255f3721f95f:7fe7734a5000-7fe7736464c4 > > On Thu, Sep 9, 2021 at 6:46 PM Robert Bradshaw <rober...@google.com> > wrote: > >> Huh, that's strange. Yes, the exact error on the service would be helpful. >> >> On Wed, Sep 8, 2021 at 10:12 AM Evan Galpin <evan.gal...@gmail.com> >> wrote: >> > >> > Thanks for the response. I've created a gist here to demonstrate a >> minimal repro: >> https://gist.github.com/egalpin/2d6ad2210cf9f66108ff48a9c7566ebc >> > >> > It seemed to run fine both on DirectRunner and PortableRunner (embed >> mode), but Dataflow v2 runner raised an error at runtime seemingly >> associated with the Shuffle service? I have job IDs and trace links if >> those are helpful as well. >> > >> > Thanks, >> > Evan >> > >> > On Tue, Sep 7, 2021 at 4:35 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >> >> >> This is not yet supported. Using a union for now is the way to go. (If >> >> only the last value of the union was used, that sounds like a bug. Do >> >> you have a minimal repro?) >> >> >> >> On Tue, Sep 7, 2021 at 1:23 PM Evan Galpin <evan.gal...@gmail.com> >> wrote: >> >> > >> >> > Hi all, >> >> > >> >> > What is the recommended way to write type hints for a tagged output >> DoFn where the outputs to different tags have different types? >> >> > >> >> > I tried using a Union to describe each of the possible output types, >> but that resulted in mismatched coder errors where only the last entry in >> the Union was used as the assumed type. Is there a way to associate a type >> hint to a tag or something like that? >> >> > >> >> > Thanks, >> >> > Evan >> >
