Not sure if you received this email or not, Eric as I don't see you subscribed to the dev apache list. Re-Replying to the original message plus two extra jira links https://issues.apache.org/jira/browse/TEZ-3914 https://issues.apache.org/jira/browse/TEZ-3784
Eric, Could you post a stack trace of the error. We have had a few bugs in the past that prevented messages over 64MB. This is only an artificial limit based on the high-level reader and writer APIs that are used. If a low-level api is used, messages of arbitrary length (or at least up to 2GB IIRC) should be possible. On Mon, Feb 4, 2019 at 3:50 PM Jonathan Eagles <[email protected]> wrote: > > Eric, Could you post a stack trace of the error. We have had a few > bugs in the past that prevented messages over 64MB. This is only an > artificial limit based on the high-level reader and writer APIs that > are used. If a low-level api is used, messages of arbitrary length (or > at least up to 2GB IIRC) should be possible. > > On Wed, Jan 30, 2019 at 2:47 PM Eric Goodman > <[email protected]> wrote: > > > > Hi Tez devs, > > > > The current design of > > VertexInitializedEvent<https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/history/events/VertexInitializedEvent.java> > > contains all of the InputDataInformationEvents for a particular vertex. > > We've had trouble scaling this for large vertices as Protobuf limits > > message sizes to 64 MB. I'm wondering if anyone is working on a more > > scaleable solution, and if not, if you guys have any suggestions for how to > > decompose this event into smaller events so that Protobuf's message size > > limit is never an issue. > > > > Thanks, > > Eric
