Re: [Scikit-learn-general] Segfault with large dataset

2013-08-24 Thread Lars Buitinck
2013/8/24 Peter Prettenhofer : > the SGD code assumes that the data is in one continouse memory segment - > does this hold for memory mapped arrays? > What if you create a memory mapped array that is larger than the available > memory? Yes, that holds, and an mmap that is larger than the available

Re: [Scikit-learn-general] Segfault with large dataset

2013-08-24 Thread Peter Prettenhofer
the SGD code assumes that the data is in one continouse memory segment - does this hold for memory mapped arrays? What if you create a memory mapped array that is larger than the available memory? there are no test cases that cover input of memory mapped arrays -- if we want to support those I str

Re: [Scikit-learn-general] Segfault with large dataset

2013-08-24 Thread Vlad Niculae
Is it maybe related to the OS, as it seems that the problem is with opening the memmapped file? Vlad On Sat, Aug 24, 2013 at 1:52 PM, Olivier Grisel wrote: > Sounds like a serious bug, could you please open an issue on github? > > -- > Olivier > > > -

Re: [Scikit-learn-general] Segfault with large dataset

2013-08-24 Thread Olivier Grisel
Sounds like a serious bug, could you please open an issue on github? -- Olivier -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, an

[Scikit-learn-general] Segfault with large dataset

2013-08-24 Thread Alan Li
Hello, I'm running into a segmentation fault with the following snippet along with the core dump output. The scikit version that I'm on is scikit_learn-0.14.1. import numpy as np data = np.memmap('/tmp/features.dat', dtype='float64', mode='r', shape=(4000,100)) target = np.memmap('/tmp/targets