Re: Broadcast variable size limit?
That’s the max size of a byte array in Java, limited by the length which is defined as integer, and in most JVMS arrays can’t hold more than Int.MaxValue - 8 elements. Other way to overcome this is to create multiple broadcast variables On Sunday, August 5, 2018, klrmowse wrote: > i don't need more, per se... i just need to watch the size of the variable; > then, if it's within the size limit, go ahead and broadcast it; if not, > then > i won't broadcast... > > so, that would be a yes then? (2 GB, or which is it exactly?) > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Sent from my iPhone
Re: Broadcast variable size limit?
i don't need more, per se... i just need to watch the size of the variable; then, if it's within the size limit, go ahead and broadcast it; if not, then i won't broadcast... so, that would be a yes then? (2 GB, or which is it exactly?) -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Broadcast variable size limit?
I think if you need more then you should anyway think about something different than broadcast variable ... > On 5. Aug 2018, at 16:51, klrmowse wrote: > > is it currently still ~2GB (Integer.MAX_VALUE) ?? > > or am i misinformed, since that's what google-search and scouring this > mailing list seem to say... ? > > > > Thanks > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Broadcast variable size limit?
is it currently still ~2GB (Integer.MAX_VALUE) ?? or am i misinformed, since that's what google-search and scouring this mailing list seem to say... ? Thanks -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org