In the mean time I think I can use gensim for this kind of data by doing 2
passes. It's a pity as I suspect I could fit it in memory, but I wonder
whether even unsigned int64 would be enough. I'll do the math when I see
the final size of the matrix.
Thanks!
Vlad
On Thu, Aug 29, 2013 at 12:11 PM, Olivier Grisel
<[email protected]>wrote:
> 2013/8/28 Lars Buitinck <[email protected]>:
> > 2013/8/28 Vlad Niculae <[email protected]>:
> >> Do the indices/indptr arrays need to be int32 or is this a limitation
> of the
> >> implementation?
> >
> > This is a limit in scipy.sparse, which uses signed int for all its
> > indices. Effectively, the number of rows, columns and non-zeros are
> > each limited to 2^31-1. There was a pull request for 64-bit indices a
> > few months ago, but I don't know what happened to it.
>
> That would be great if it could make it through.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>
>
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
