That’s the max size of a byte array in Java, limited by the length which is defined as integer, and in most JVMS arrays can’t hold more than Int.MaxValue - 8 elements. Other way to overcome this is to create multiple broadcast variables
On Sunday, August 5, 2018, klrmowse <klrmo...@gmail.com> wrote: > i don't need more, per se... i just need to watch the size of the variable; > then, if it's within the size limit, go ahead and broadcast it; if not, > then > i won't broadcast... > > so, that would be a yes then? (2 GB, or which is it exactly?) > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Sent from my iPhone
