Hi Brian, Another alternative is to use a dedicated business intelligence tool.
I have used BIRT (http://www.eclipse.org/birt/) but I can't recommend it. Another free option is http://www.pentaho.com/ . It should be able to use large CSV files as datasource and it appears to have good data mining. Unfortunately most BI tools are just glorified reporting tools, but good ones are designed to manipulate and investigate large datasets. cheers, Jeromy On Oct 19, 10:38 am, Mike Nicholls - Elders Real Estate Mosman <mi...@eldersmosman.com.au> wrote: > Hi Brian > > Use one of the SQL GUI tools, (hook it up to an SQL server on your hosting > account) they can import from a spreadsheet and then use their point and > click style to avoid having to code the SQL yourself (even I the geek with > no technical skills managed it) > > here is a list of the main ones > > http://www.databasejournal.com/features/mysql/article.php/3880961/Top... > > Mike > > On Tue, Oct 19, 2010 at 10:28 AM, Andrew Stewart > <a...@universalsprout.com>wrote: > > > > > For real quick and dirty you could leave the data in a tsv or csv file and > > write some code (eg python) to parse the data and do simple calculations on > > it. This works really well if all you want to do is are things like 'count > > the number of records where column x satisfies some conditions' etc. > > Depending on your data set it could take a few seconds to execute each time, > > but it means you don't have to mess around with a database. Loading the data > > into RAM each time you execute in not good for performance, but I have found > > the data set normally needs to have over 100,000 records before I would > > think about storing it in a database to improve performance. - it does > > depend on the analysis you are doing though. > > > Cheers, > > > Andy > > > On 19 Oct 2010, at 10:05, Sriram Panyam wrote: > > > what kind of data processing would you like to do? > > > As a language R seems a good choice for statistical processing and > > analytics and from what Ive seen this data set would be handled well by R. > > > Also how much do you want to be on the "language" side and how much on the > > UI side? > > > But at 300x14000 records you are looking at around around 5mb of data? R > > over any simple relational database should be good. Sqlite "may" strain > > under this but postgres/mysql will handle it well. But you did say quick > > and dirty. So Id say R with Python would be a pretty formidable choice. > > > cheers > > Sri > > > On Tue, Oct 19, 2010 at 9:50 AM, BrianMenzies <bmenz...@gmail.com> wrote: > > >> Hi > > >> I need to do some quick and dirty analysis on large dataset (think 300 > >> fields and 14,000 records). > >> My typical tool of choice for this sort of thing is Excel but the > >> files are too large and I don't want to cut them up. > > >> Can anyone recommend any freely available tools that have the capacity > >> to chew up large (well large for my world) data sets and let me > >> analysis them (I'd like to avoid having to write any SQL)? > > >> Thanks > >> Brian > >> PS also recently been teaching myself Blender 3D for business > >> animations. > >> Anyone know of any sort of local user group for Blender? > > >> Delurk (again) > >> Was a coder in my dim dark past. > >> Now at a clean energy startup in Sydney. > >> Show my face at various Sydney startup-events. > > >> -- > >> You received this message because you are subscribed to the Silicon Beach > >> Australia mailing list. > > >> Guidelines on discussion:http://tr.im/ujKF > > >> No lurkers! It is expected that you introduce yourself:http://tr.im/ujMm > > >> To post to this group, send email to > >> silicon-beach-australia@googlegroups.com > >> To unsubscribe from this group, send email to > >> silicon-beach-australia+unsubscr...@googlegroups.com<silicon-beach-australi > >> a%2bunsubscr...@googlegroups.com> > >> For more options, visit this group at > >>http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en > > > -- > > Blog:http://panyam.wordpress.com > > Twitter: @panyam > > > -- > > You received this message because you are subscribed to the Silicon Beach > > Australia mailing list. > > > Guidelines on discussion:http://tr.im/ujKF > > > No lurkers! It is expected that you introduce yourself:http://tr.im/ujMm > > > To post to this group, send email to > > silicon-beach-australia@googlegroups.com > > To unsubscribe from this group, send email to > > silicon-beach-australia+unsubscr...@googlegroups.com > > For more options, visit this group at > >http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en > > > -- > > You received this message because you are subscribed to the Silicon Beach > > Australia mailing list. > > > Guidelines on discussion:http://tr.im/ujKF > > > No lurkers! It is expected that you introduce yourself:http://tr.im/ujMm > > > To post to this group, send email to > > silicon-beach-australia@googlegroups.com > > To unsubscribe from this group, send email to > > silicon-beach-australia+unsubscr...@googlegroups.com<silicon-beach-australi > > a%2bunsubscr...@googlegroups.com> > > For more options, visit this group at > >http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en -- You received this message because you are subscribed to the Silicon Beach Australia mailing list. Guidelines on discussion: http://tr.im/ujKF No lurkers! It is expected that you introduce yourself: http://tr.im/ujMm To post to this group, send email to silicon-beach-australia@googlegroups.com To unsubscribe from this group, send email to silicon-beach-australia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en
