Eric, Could you post a stack trace of the error. We have had a few bugs in the past that prevented messages over 64MB. This is only an artificial limit based on the high-level reader and writer APIs that are used. If a low-level api is used, messages of arbitrary length (or at least up to 2GB IIRC) should be possible.
On Wed, Jan 30, 2019 at 2:47 PM Eric Goodman <[email protected]> wrote: > > Hi Tez devs, > > The current design of > VertexInitializedEvent<https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/history/events/VertexInitializedEvent.java> > contains all of the InputDataInformationEvents for a particular vertex. > We've had trouble scaling this for large vertices as Protobuf limits message > sizes to 64 MB. I'm wondering if anyone is working on a more scaleable > solution, and if not, if you guys have any suggestions for how to decompose > this event into smaller events so that Protobuf's message size limit is never > an issue. > > Thanks, > Eric
