Hi Andrew, On Thu, Feb 23, 2012 at 12:44:39PM -0800, akm wrote: > > I'm interested. But in the practical application I've in mind, the data > > isn't > > there it is generated on the fly by sage itself. The problem it to gather > > information (eg: the size) on huge sets from combinatorics. Those sets are > > generated by a choice tree. The exploration of the various branches can > > obviously be made in parallel. > > Would it be useful to be able to shove the data you generate on the > fly out to the distributed file system, where it could be analyzed via > whatever parallel process you use to explore it?
Maybe ! But you shouldn't take the application I've in mind too seriously. Maybe it's only because I don't have the necessary architecture, but right now, I've never passed a few dozens Gig of data. So there is clearly no need for a distributed FS to store them. The problem is more that on one core I'm generating something in the range 1ko/s-1mo/s. So having the thing distributed on a 32/64 core or on a even bigger machine could really help. > That's the kind of thing I'm hoping to get out of the integration with > Hadoop or Riak or whatever we used: being able to seamlessly treat > the distributed file system as "just a disk", that could then be used > for further analysis. If the disk is smart enough the generate the data, it would be indeed brilliant ;-) Cheers, Florent -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
