Hi all,
I have to deal with a lot of data, and I use spark for months. Now I try to use Vectors.sparse to generate a large vector of features, but the feature size may exceed 4 billion, above max of int, so I want to use BigInt or Long type to deal with it. But I read code and document that Vectors.sparse support int as size type only, why limit it to int? And how can I bypass this limit? Thanks, Alexander