Re: [julia-users] Moore foundation grant.
Hi All, The roadmap has now been posted: http://juliacomputing.com/blog/2016/01/14/stats-roadmap.html Can I suggest that comments be posted to the julia-stats thread: https://groups.google.com/d/topic/julia-stats/29l5yA87Qss/discussion -Simon On Wednesday, 13 January 2016 21:23:03 UTC, Lars Tonkard wrote: > > +1 to request for news on the stats roadmap :) > > On Sunday, January 10, 2016 at 11:45:59 AM UTC-5, Nils Gudat wrote: >> >> Any news on the JuliaStats team's roadmap? >> >> On Thursday, December 17, 2015 at 11:20:45 PM UTC, Viral Shah wrote: >>> >>> >>> The JuliaStats team will be publishing a general plan on stats+df in a >>> few days. I doubt we will have settled on all the df issues by then, but at >>> least there will be something to start with. >>> >>> >>> -viral >>> >>> >>> >>> > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: >>> > >>> > Hi Viral, >>> > >>> > Any update on this (stats + df) by chance or idea when we can get one? >>> Even a roadmap or some sort of vision or other details would help with >>> decision making regarding infrastructure. >>> > >>> > Thanks! >>> > >>> > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: >>> > Yes, we are really excited. This grant is to focus on core Julia >>> compiler infrastructure and key math libraries. Much of the libraries focus >>> will be on statistical Computing. >>> > -viral >>> > >>> >>>
Re: [julia-users] Moore foundation grant.
+1 to request for news on the stats roadmap :) On Sunday, January 10, 2016 at 11:45:59 AM UTC-5, Nils Gudat wrote: > > Any news on the JuliaStats team's roadmap? > > On Thursday, December 17, 2015 at 11:20:45 PM UTC, Viral Shah wrote: >> >> >> The JuliaStats team will be publishing a general plan on stats+df in a >> few days. I doubt we will have settled on all the df issues by then, but at >> least there will be something to start with. >> >> >> -viral >> >> >> >> > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: >> > >> > Hi Viral, >> > >> > Any update on this (stats + df) by chance or idea when we can get one? >> Even a roadmap or some sort of vision or other details would help with >> decision making regarding infrastructure. >> > >> > Thanks! >> > >> > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: >> > Yes, we are really excited. This grant is to focus on core Julia >> compiler infrastructure and key math libraries. Much of the libraries focus >> will be on statistical Computing. >> > -viral >> > >> >>
Re: [julia-users] Moore foundation grant.
Any news on the JuliaStats team's roadmap? On Thursday, December 17, 2015 at 11:20:45 PM UTC, Viral Shah wrote: > > > The JuliaStats team will be publishing a general plan on stats+df in a few > days. I doubt we will have settled on all the df issues by then, but at > least there will be something to start with. > > > -viral > > > > > On 17-Dec-2015, at 10:15 PM, Lampkld > > wrote: > > > > Hi Viral, > > > > Any update on this (stats + df) by chance or idea when we can get one? > Even a roadmap or some sort of vision or other details would help with > decision making regarding infrastructure. > > > > Thanks! > > > > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: > > Yes, we are really excited. This grant is to focus on core Julia > compiler infrastructure and key math libraries. Much of the libraries focus > will be on statistical Computing. > > -viral > > > >
Re: [julia-users] Moore foundation grant.
We need to have both. Juno/Atom and Eclipse are both great platforms. Juno/Atom is certainly the more experimental one, but I recently spoke with Mike and we are pretty close to getting a Juno/LT replacement. With Eclipse, the idea is to have something that is familiar to Eclipse users, and as Oleg said - I have heard from a lot of users that they are waiting for this. I am quite hopeful that we will have something polished soon, in the next 2-3 months. -viral > On 01-Jan-2016, at 2:08 AM, Oleg Mikulchenko wrote: > > Atom is rather the text editor (with some IDE features), while Eclipse is the > comprehensive true IDE, already in use for many languages. > > Anyway, Happy New Year! > > On Thu, Dec 31, 2015 at 9:19 AM, David Anthoff wrote: > You mean Julia Computing is working on an Eclipse solution? How does that > match with the efforts around Atom to create an IDE that are also ongoing? It > would be nice to see efforts and resources concentrated on a single IDE > solution, in the hope of getting something that is really polished some day... > > > -Original Message- > > From: julia-users@googlegroups.com [mailto:julia- > > us...@googlegroups.com] On Behalf Of Viral Shah > > Sent: Wednesday, December 30, 2015 11:00 PM > > To: julia-users@googlegroups.com > > Subject: Re: [julia-users] Moore foundation grant. > > > > > > The Eclipse Foundation doesn’t have funding, it turns out. However, we at > > Julia Computing just started on an effort recently, and hope to have > > something to show as soon as we have something minimal working. > > > > -viral > > > > > > > On 31-Dec-2015, at 6:37 AM, Oleg Mikulchenko > > wrote: > > > > > > 1. Congratulations! > > > 2. Having a good Eclipse plugin for Julia is the dream for many potential > > users, just like me. Looking forward to hear some good news on that ! > > > > > > On Wednesday, November 11, 2015 at 11:12:16 PM UTC-8, Viral Shah > > wrote: > > > > > > Independently, we are also discussing with the Eclipse Foundation to see > > how we can work with their scientific community and have a good Eclipse > > plugin for Julia. > > > > >
Re: [julia-users] Moore foundation grant.
Atom is rather the text editor (with some IDE features), while Eclipse is the comprehensive true IDE, already in use for many languages. Anyway, Happy New Year! On Thu, Dec 31, 2015 at 9:19 AM, David Anthoff wrote: > You mean Julia Computing is working on an Eclipse solution? How does that > match with the efforts around Atom to create an IDE that are also ongoing? > It would be nice to see efforts and resources concentrated on a single IDE > solution, in the hope of getting something that is really polished some > day... > > > -Original Message- > > From: julia-users@googlegroups.com [mailto:julia- > > us...@googlegroups.com] On Behalf Of Viral Shah > > Sent: Wednesday, December 30, 2015 11:00 PM > > To: julia-users@googlegroups.com > > Subject: Re: [julia-users] Moore foundation grant. > > > > > > The Eclipse Foundation doesn’t have funding, it turns out. However, we at > > Julia Computing just started on an effort recently, and hope to have > > something to show as soon as we have something minimal working. > > > > -viral > > > > > > > On 31-Dec-2015, at 6:37 AM, Oleg Mikulchenko > > wrote: > > > > > > 1. Congratulations! > > > 2. Having a good Eclipse plugin for Julia is the dream for many > potential > > users, just like me. Looking forward to hear some good news on that ! > > > > > > On Wednesday, November 11, 2015 at 11:12:16 PM UTC-8, Viral Shah > > wrote: > > > > > > Independently, we are also discussing with the Eclipse Foundation to > see > > how we can work with their scientific community and have a good Eclipse > > plugin for Julia. > > > > >
RE: [julia-users] Moore foundation grant.
You mean Julia Computing is working on an Eclipse solution? How does that match with the efforts around Atom to create an IDE that are also ongoing? It would be nice to see efforts and resources concentrated on a single IDE solution, in the hope of getting something that is really polished some day... > -Original Message- > From: julia-users@googlegroups.com [mailto:julia- > us...@googlegroups.com] On Behalf Of Viral Shah > Sent: Wednesday, December 30, 2015 11:00 PM > To: julia-users@googlegroups.com > Subject: Re: [julia-users] Moore foundation grant. > > > The Eclipse Foundation doesn’t have funding, it turns out. However, we at > Julia Computing just started on an effort recently, and hope to have > something to show as soon as we have something minimal working. > > -viral > > > > On 31-Dec-2015, at 6:37 AM, Oleg Mikulchenko > wrote: > > > > 1. Congratulations! > > 2. Having a good Eclipse plugin for Julia is the dream for many potential > users, just like me. Looking forward to hear some good news on that ! > > > > On Wednesday, November 11, 2015 at 11:12:16 PM UTC-8, Viral Shah > wrote: > > > > Independently, we are also discussing with the Eclipse Foundation to see > how we can work with their scientific community and have a good Eclipse > plugin for Julia. > >
Re: [julia-users] Moore foundation grant.
The Eclipse Foundation doesn’t have funding, it turns out. However, we at Julia Computing just started on an effort recently, and hope to have something to show as soon as we have something minimal working. -viral > On 31-Dec-2015, at 6:37 AM, Oleg Mikulchenko wrote: > > 1. Congratulations! > 2. Having a good Eclipse plugin for Julia is the dream for many potential > users, just like me. Looking forward to hear some good news on that ! > > On Wednesday, November 11, 2015 at 11:12:16 PM UTC-8, Viral Shah wrote: > > Independently, we are also discussing with the Eclipse Foundation to see how > we can work with their scientific community and have a good Eclipse plugin > for Julia. >
Re: [julia-users] Moore foundation grant.
1. Congratulations! 2. Having a good Eclipse plugin for Julia is the dream for many potential users, just like me. Looking forward to hear some good news on that ! On Wednesday, November 11, 2015 at 11:12:16 PM UTC-8, Viral Shah wrote: > > > Independently, we are also discussing with the Eclipse Foundation to see > how we can work with their scientific community and have a good Eclipse > plugin for Julia. > >
Re: [julia-users] Moore foundation grant.
Viral- Yes his stuff looks really exciting! Simon- Tenenbaum's Group at MIT (IIRC) and Ghahramani's at Cambridge are doing some prob programming stuff in Julia. Maybe its worth talking to collaborating with them. More cool stuff on that and related here: http://research.microsoft.com/apps/video/default.aspx?id=259579&l=i On Sunday, December 27, 2015 at 1:59:49 PM UTC-5, Cedric St-Jean wrote: > > I've thought a few times about reimplementing Stan in Julia. I wonder how > much of Stan's codebase is about parsing/code-generation (which would be > drastically simpler in Julia) versus fine-tuning their NUTS sampler. And > how much of that work about automatic differentiation/code generation could > be shared with the deep-learning libraries. > > On Sunday, December 27, 2015 at 1:28:53 PM UTC-5, Lampkld wrote: >> >> Viral and Symon, >> >> Since you asked, I will write out some rough and probably excessively >> abstract Ideas that have been floating around in my head below. I don't >> have time to formally polish, so please forgive the inchoate nature of >> these thoughts: >> >> Yes, composability and generality are the names of the game! I would also >> add expressiveness, scalability and fostering innovation. >> >> Part of #1 is at least reaching parity with R in terms of data cleaning >> and manipulation syntax. Part of R's popularity and its stubborn growth in >> the face of python's recent maturation is its advantage in ease of >> expressing data manipulation. If Julia is to compete, at the very least the >> ecosystem should leverage macros to emulate R's NSE, in a more measured >> manner (similar to DF meta). >> >> However I think we should be greedy and think , how can we do better? How >> can we shorten the overhead and feedback loop in exploring and >> experimenting with ideas and the data and models? I don't have many >> concrete suggestions here, but I suspect the solution would involve >> something Dplyr like with conservative and targeted use of interactive >> javascript and web gl. Can we do transforms on the data with the mouse? >> Fly through it With 3d glasses? I think we should think kinda wild here. >> Hadley has discussed his dreams regarding a "grammar of modeling". Is this >> prob programming or something else? >> >> What about plotting specifically ? I think an excellent sort of general >> exploration framework is Topological Data Analysis. >> >> Finally, I would look at maximizing diffuse innovation while maintaining >> uniformity, the strengths of R's and Python's ecosystems respectively. My >> amateur read of the complex systems science research is that the ability >> of a system to produce new ideas and process information robustly and >> quickly is correlated with a balance between looseness and diversity on one >> hand balanced with strength of connection between nodes and some >> hierarchy. How can we design the Julia ecosystem to leverage this insight >> while keeping uniformity in interface? I'm thinking an abstract interface >> with generic functions and types (similar to distributions.jl,) that can be >> easily composed together by researchers to create new models but can be >> plugged back in to an API and tooling to be easily leveraged by end users. >> Further making experimentation easy and fun (a trait that has received much >> acclaim from researchers already.) will encourage grad students to pick up >> Julia and the abstract interface will encourage use of these packages, >> further increasing incentives to produce. >> >> I know this is all very vague, but I just wanted to get my general >> vision out there. Things like passing in types instead of symbols for >> choosing methods, using multi inheritance traits to tag new models and >> solvers, using functions defined on abstract types to get tests and >> optimizes for free are some potential specifics. >> >> Specifically regarding a PPL, I would say with recent Lora.jl progress >> and Distributions.jl, and Julia's much more concise and expressive nature >> vs C++, I don't think it would take anywhere near the work of Stan to get >> something decent. Pymc 3 is pretty darn close and exceeds stan in some >> areas with much less labor and code volume... and this is just in python. >> (though also leveraging theano. >> >> What does everyone think? >> >> On Sunday, December 27, 2015 at 12:41:18 PM UTC-5, Simon Byrne wrote: >>> >>> Thanks for the suggestions, these are certainly the main areas in which >>> we're looking to address as part of this work. >>> >>> I'd be interested to hear if you have more thoughts about the model >>> specification/probabilistic programming language. A few other people have >>> requested things like this, and this would certainly play to Julia's >>> strengths (as shown by JuMP.jl). That said, a full-scale probabilistic >>> programming language might be a bit too much to ask as part of this work >>> (keep in mind t
Re: [julia-users] Moore foundation grant.
I've thought a few times about reimplementing Stan in Julia. I wonder how much of Stan's codebase is about parsing/code-generation (which would be drastically simpler in Julia) versus fine-tuning their NUTS sampler. And how much of that work about automatic differentiation/code generation could be shared with the deep-learning libraries. On Sunday, December 27, 2015 at 1:28:53 PM UTC-5, Lampkld wrote: > > Viral and Symon, > > Since you asked, I will write out some rough and probably excessively > abstract Ideas that have been floating around in my head below. I don't > have time to formally polish, so please forgive the inchoate nature of > these thoughts: > > Yes, composability and generality are the names of the game! I would also > add expressiveness, scalability and fostering innovation. > > Part of #1 is at least reaching parity with R in terms of data cleaning > and manipulation syntax. Part of R's popularity and its stubborn growth in > the face of python's recent maturation is its advantage in ease of > expressing data manipulation. If Julia is to compete, at the very least the > ecosystem should leverage macros to emulate R's NSE, in a more measured > manner (similar to DF meta). > > However I think we should be greedy and think , how can we do better? How > can we shorten the overhead and feedback loop in exploring and > experimenting with ideas and the data and models? I don't have many > concrete suggestions here, but I suspect the solution would involve > something Dplyr like with conservative and targeted use of interactive > javascript and web gl. Can we do transforms on the data with the mouse? > Fly through it With 3d glasses? I think we should think kinda wild here. > Hadley has discussed his dreams regarding a "grammar of modeling". Is this > prob programming or something else? > > What about plotting specifically ? I think an excellent sort of general > exploration framework is Topological Data Analysis. > > Finally, I would look at maximizing diffuse innovation while maintaining > uniformity, the strengths of R's and Python's ecosystems respectively. My > amateur read of the complex systems science research is that the ability > of a system to produce new ideas and process information robustly and > quickly is correlated with a balance between looseness and diversity on one > hand balanced with strength of connection between nodes and some > hierarchy. How can we design the Julia ecosystem to leverage this insight > while keeping uniformity in interface? I'm thinking an abstract interface > with generic functions and types (similar to distributions.jl,) that can be > easily composed together by researchers to create new models but can be > plugged back in to an API and tooling to be easily leveraged by end users. > Further making experimentation easy and fun (a trait that has received much > acclaim from researchers already.) will encourage grad students to pick up > Julia and the abstract interface will encourage use of these packages, > further increasing incentives to produce. > > I know this is all very vague, but I just wanted to get my general vision > out there. Things like passing in types instead of symbols for choosing > methods, using multi inheritance traits to tag new models and solvers, > using functions defined on abstract types to get tests and optimizes for > free are some potential specifics. > > Specifically regarding a PPL, I would say with recent Lora.jl progress and > Distributions.jl, and Julia's much more concise and expressive nature vs > C++, I don't think it would take anywhere near the work of Stan to get > something decent. Pymc 3 is pretty darn close and exceeds stan in some > areas with much less labor and code volume... and this is just in python. > (though also leveraging theano. > > What does everyone think? > > On Sunday, December 27, 2015 at 12:41:18 PM UTC-5, Simon Byrne wrote: >> >> Thanks for the suggestions, these are certainly the main areas in which >> we're looking to address as part of this work. >> >> I'd be interested to hear if you have more thoughts about the model >> specification/probabilistic programming language. A few other people have >> requested things like this, and this would certainly play to Julia's >> strengths (as shown by JuMP.jl). That said, a full-scale probabilistic >> programming language might be a bit too much to ask as part of this work >> (keep in mind that Stan has been 3+ year project with 2-3 full-time devs + >> volunteers), but there might be some low-hanging fruit here we can pick. >> >> -simon >> >> On Sunday, 27 December 2015 02:32:43 UTC, Lampkld wrote: >>> >>> Thanks for the response. >>> >>> Since you kindly asked, the following are two main areas in our >>> assessment of the general arc of the Julia ecosystem: >>> >>> 1. Will the roadmap obviate some of the bottlenecks for day to day >>> normal exploratory workflow? These
Re: [julia-users] Moore foundation grant.
Have you seen Simon D's viz stuff at juliacon? I believe a lot of that stuff is soon going to be ready for wider use. I am sure he will chime in further on this thread. -viral On 27 Dec 2015 11:59 pm, "Lampkld" wrote: > > Viral and Symon, > > Since you asked, I will write out some rough and probably excessively abstract Ideas that have been floating around in my head below. I don't have time to formally polish, so please forgive the inchoate nature of these thoughts: > > Yes, composability and generality are the names of the game! I would also add expressiveness, scalability and fostering innovation. > > Part of #1 is at least reaching parity with R in terms of data cleaning and manipulation syntax. Part of R's popularity and its stubborn growth in the face of python's recent maturation is its advantage in ease of expressing data manipulation. If Julia is to compete, at the very least the ecosystem should leverage macros to emulate R's NSE, in a more measured manner (similar to DF meta). > > However I think we should be greedy and think , how can we do better? How can we shorten the overhead and feedback loop in exploring and experimenting with ideas and the data and models? I don't have many concrete suggestions here, but I suspect the solution would involve something Dplyr like with conservative and targeted use of interactive javascript and web gl. Can we do transforms on the data with the mouse? Fly through it With 3d glasses? I think we should think kinda wild here. Hadley has discussed his dreams regarding a "grammar of modeling". Is this prob programming or something else? > > What about plotting specifically ? I think an excellent sort of general exploration framework is Topological Data Analysis. > > Finally, I would look at maximizing diffuse innovation while maintaining uniformity, the strengths of R's and Python's ecosystems respectively. My amateur read of the complex systems science research is that the ability of a system to produce new ideas and process information robustly and quickly is correlated with a balance between looseness and diversity on one hand balanced with strength of connection between nodes and some hierarchy. How can we design the Julia ecosystem to leverage this insight while keeping uniformity in interface? I'm thinking an abstract interface with generic functions and types (similar to distributions.jl,) that can be easily composed together by researchers to create new models but can be plugged back in to an API and tooling to be easily leveraged by end users. Further making experimentation easy and fun (a trait that has received much acclaim from researchers already.) will encourage grad students to pick up Julia and the abstract interface will encourage use of these packages, further increasing incentives to produce. > > I know this is all very vague, but I just wanted to get my general vision out there. Things like passing in types instead of symbols for choosing methods, using multi inheritance traits to tag new models and solvers, using functions defined on abstract types to get tests and optimizes for free are some potential specifics. > > Specifically regarding a PPL, I would say with recent Lora.jl progress and Distributions.jl, and Julia's much more concise and expressive nature vs C++, I don't think it would take anywhere near the work of Stan to get something decent. Pymc 3 is pretty darn close and exceeds stan in some areas with much less labor and code volume... and this is just in python. (though also leveraging theano. > > What does everyone think? > > On Sunday, December 27, 2015 at 12:41:18 PM UTC-5, Simon Byrne wrote: >> >> Thanks for the suggestions, these are certainly the main areas in which we're looking to address as part of this work. >> >> I'd be interested to hear if you have more thoughts about the model specification/probabilistic programming language. A few other people have requested things like this, and this would certainly play to Julia's strengths (as shown by JuMP.jl). That said, a full-scale probabilistic programming language might be a bit too much to ask as part of this work (keep in mind that Stan has been 3+ year project with 2-3 full-time devs + volunteers), but there might be some low-hanging fruit here we can pick. >> >> -simon >> >> On Sunday, 27 December 2015 02:32:43 UTC, Lampkld wrote: >>> >>> Thanks for the response. >>> >>> Since you kindly asked, the following are two main areas in our assessment of the general arc of the Julia ecosystem: >>> >>> 1. Will the roadmap obviate some of the bottlenecks for day to day normal exploratory workflow? These are minimal things that R and Python have and whose lack hamper any use of Julia for regular analysis. Thing like robust dataframe with data i/o into different formats, web scraping, work out nullable semantics and integration with ecosystem , robust data cleaning and tidy data, modeling with basic diagnostic tests etc >>> >>> 2. Will the roadmap jump leapfrog
Re: [julia-users] Moore foundation grant.
Viral and Symon, Since you asked, I will write out some rough and probably excessively abstract Ideas that have been floating around in my head below. I don't have time to formally polish, so please forgive the inchoate nature of these thoughts: Yes, composability and generality are the names of the game! I would also add expressiveness, scalability and fostering innovation. Part of #1 is at least reaching parity with R in terms of data cleaning and manipulation syntax. Part of R's popularity and its stubborn growth in the face of python's recent maturation is its advantage in ease of expressing data manipulation. If Julia is to compete, at the very least the ecosystem should leverage macros to emulate R's NSE, in a more measured manner (similar to DF meta). However I think we should be greedy and think , how can we do better? How can we shorten the overhead and feedback loop in exploring and experimenting with ideas and the data and models? I don't have many concrete suggestions here, but I suspect the solution would involve something Dplyr like with conservative and targeted use of interactive javascript and web gl. Can we do transforms on the data with the mouse? Fly through it With 3d glasses? I think we should think kinda wild here. Hadley has discussed his dreams regarding a "grammar of modeling". Is this prob programming or something else? What about plotting specifically ? I think an excellent sort of general exploration framework is Topological Data Analysis. Finally, I would look at maximizing diffuse innovation while maintaining uniformity, the strengths of R's and Python's ecosystems respectively. My amateur read of the complex systems science research is that the ability of a system to produce new ideas and process information robustly and quickly is correlated with a balance between looseness and diversity on one hand balanced with strength of connection between nodes and some hierarchy. How can we design the Julia ecosystem to leverage this insight while keeping uniformity in interface? I'm thinking an abstract interface with generic functions and types (similar to distributions.jl,) that can be easily composed together by researchers to create new models but can be plugged back in to an API and tooling to be easily leveraged by end users. Further making experimentation easy and fun (a trait that has received much acclaim from researchers already.) will encourage grad students to pick up Julia and the abstract interface will encourage use of these packages, further increasing incentives to produce. I know this is all very vague, but I just wanted to get my general vision out there. Things like passing in types instead of symbols for choosing methods, using multi inheritance traits to tag new models and solvers, using functions defined on abstract types to get tests and optimizes for free are some potential specifics. Specifically regarding a PPL, I would say with recent Lora.jl progress and Distributions.jl, and Julia's much more concise and expressive nature vs C++, I don't think it would take anywhere near the work of Stan to get something decent. Pymc 3 is pretty darn close and exceeds stan in some areas with much less labor and code volume... and this is just in python. (though also leveraging theano. What does everyone think? On Sunday, December 27, 2015 at 12:41:18 PM UTC-5, Simon Byrne wrote: > > Thanks for the suggestions, these are certainly the main areas in which > we're looking to address as part of this work. > > I'd be interested to hear if you have more thoughts about the model > specification/probabilistic programming language. A few other people have > requested things like this, and this would certainly play to Julia's > strengths (as shown by JuMP.jl). That said, a full-scale probabilistic > programming language might be a bit too much to ask as part of this work > (keep in mind that Stan has been 3+ year project with 2-3 full-time devs + > volunteers), but there might be some low-hanging fruit here we can pick. > > -simon > > On Sunday, 27 December 2015 02:32:43 UTC, Lampkld wrote: >> >> Thanks for the response. >> >> Since you kindly asked, the following are two main areas in our >> assessment of the general arc of the Julia ecosystem: >> >> 1. Will the roadmap obviate some of the bottlenecks for day to day normal >> exploratory workflow? These are minimal things that R and Python have and >> whose lack hamper any use of Julia for regular analysis. Thing like robust >> dataframe with data i/o into different formats, web scraping, work out >> nullable semantics and integration with ecosystem , robust data cleaning >> and tidy data, modeling with basic diagnostic tests etc >> >> 2. Will the roadmap jump leapfrog into areas and capabilities that are >> currently not covered by other stats and data science ecosystems? >> >> There are many here, but we are specifically looking at
Re: [julia-users] Moore foundation grant.
Thanks for the suggestions, these are certainly the main areas in which we're looking to address as part of this work. I'd be interested to hear if you have more thoughts about the model specification/probabilistic programming language. A few other people have requested things like this, and this would certainly play to Julia's strengths (as shown by JuMP.jl). That said, a full-scale probabilistic programming language might be a bit too much to ask as part of this work (keep in mind that Stan has been 3+ year project with 2-3 full-time devs + volunteers), but there might be some low-hanging fruit here we can pick. -simon On Sunday, 27 December 2015 02:32:43 UTC, Lampkld wrote: > > Thanks for the response. > > Since you kindly asked, the following are two main areas in our assessment > of the general arc of the Julia ecosystem: > > 1. Will the roadmap obviate some of the bottlenecks for day to day normal > exploratory workflow? These are minimal things that R and Python have and > whose lack hamper any use of Julia for regular analysis. Thing like robust > dataframe with data i/o into different formats, web scraping, work out > nullable semantics and integration with ecosystem , robust data cleaning > and tidy data, modeling with basic diagnostic tests etc > > 2. Will the roadmap jump leapfrog into areas and capabilities that are > currently not covered by other stats and data science ecosystems? > > There are many here, but we are specifically looking at the ability to > work with modeling on medium sized out of core databases. This would > include an abstract dataframe like interface to said databses MySQL and > SQLlite, and some sort of modeling capability on the same. My dream would > be separation of model specification as a DAG/ probabilistic programming > framework, from fitting the model. Thus the same model can be fit with > different sort of data and optimizers. Streaming black box variation > inference can be a means to extend this to OOC work. > > I realize Julia won't for a while have all the statistical tests and > random models of python, much less R. However, a general yet powerful and > scalable data querying and prob programming framework could arguably > suffice for most python and R use cases in Data Science while provide a > comparative advantage over other frameworks where it counts. To my > knowledge, Right now SAS and STATA are the only packages that offer general > modeling with on disk data sets, but the sort of capability I outlined > would seem to be in excess of what they offer. > > A bonus would be filling out gadfly towards Ggplot and ggvis capability. > > > > On Thursday, December 24, 2015 at 11:50:42 AM UTC-5, Viral Shah wrote: >> >> What would be helpful is to know what kind of decisions you are thinking >> of and what are the factors. >> >> I suspect within 2 weeks for sure - but it's really for the Julia stats >> folks to say. The idea is to get feedback and chart a course. >> >> -viral >> On 24 Dec 2015 10:07 p.m., "Lampkld" wrote: >> >>> Sorry to bug you, but can we expect something this or next week? Would >>> be helpful in knowing until when to push some stuff off. >>> >>> On Thursday, December 17, 2015 at 6:20:45 PM UTC-5, Viral Shah wrote: The JuliaStats team will be publishing a general plan on stats+df in a few days. I doubt we will have settled on all the df issues by then, but at least there will be something to start with. -viral > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: > > Hi Viral, > > Any update on this (stats + df) by chance or idea when we can get one? Even a roadmap or some sort of vision or other details would help with decision making regarding infrastructure. > > Thanks! > > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: > Yes, we are really excited. This grant is to focus on core Julia compiler infrastructure and key math libraries. Much of the libraries focus will be on statistical Computing. > -viral >
Re: [julia-users] Moore foundation grant.
Briefly, 1. Robust dataframes is a key thrust area for this work. At this point the work is exploratory, but we all are expecting this being one of the first areas to see rapid progress on. Julia’s db support has improved a lot, independently, and will keep getting better. As soon as there is consensus on this direction, the rest of the stats work should accelerate greatly. Web scraping is unlikely to be something that is part of this - but the rest of what you mention is fair game. 2. This is really the question and comes up repeatedly. As a goal, we certainly don’t want to just clone what other tools are doing, but do better. It is hard to outline exactly how that will happen, but one key thing that I am personally focussed on is being able to work with larger volumes of data in a composable general way. What are the other capabilities and areas where Julia could potentially leapfrog? -viral > On 27-Dec-2015, at 8:02 AM, Lampkld wrote: > > Thanks for the response. > > Since you kindly asked, the following are two main areas in our assessment of > the general arc of the Julia ecosystem: > > 1. Will the roadmap obviate some of the bottlenecks for day to day normal > exploratory workflow? These are minimal things that R and Python have and > whose lack hamper any use of Julia for regular analysis. Thing like robust > dataframe with data i/o into different formats, web scraping, work out > nullable semantics and integration with ecosystem , robust data cleaning and > tidy data, modeling with basic diagnostic tests etc > > 2. Will the roadmap jump leapfrog into areas and capabilities that are > currently not covered by other stats and data science ecosystems? > > There are many here, but we are specifically looking at the ability to work > with modeling on medium sized out of core databases. This would include an > abstract dataframe like interface to said databses MySQL and SQLlite, and > some sort of modeling capability on the same. My dream would be separation of > model specification as a DAG/ probabilistic programming framework, from > fitting the model. Thus the same model can be fit with different sort of data > and optimizers. Streaming black box variation inference can be a means to > extend this to OOC work. > > I realize Julia won't for a while have all the statistical tests and random > models of python, much less R. However, a general yet powerful and scalable > data querying and prob programming framework could arguably suffice for most > python and R use cases in Data Science while provide a comparative advantage > over other frameworks where it counts. To my knowledge, Right now SAS and > STATA are the only packages that offer general modeling with on disk data > sets, but the sort of capability I outlined would seem to be in excess of > what they offer. > > A bonus would be filling out gadfly towards Ggplot and ggvis capability. > > > > On Thursday, December 24, 2015 at 11:50:42 AM UTC-5, Viral Shah wrote: > What would be helpful is to know what kind of decisions you are thinking of > and what are the factors. > > I suspect within 2 weeks for sure - but it's really for the Julia stats folks > to say. The idea is to get feedback and chart a course. > > -viral > > On 24 Dec 2015 10:07 p.m., "Lampkld" wrote: > Sorry to bug you, but can we expect something this or next week? Would be > helpful in knowing until when to push some stuff off. > > On Thursday, December 17, 2015 at 6:20:45 PM UTC-5, Viral Shah wrote: > > The JuliaStats team will be publishing a general plan on stats+df in a few > days. I doubt we will have settled on all the df issues by then, but at least > there will be something to start with. > > > -viral > > > > > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: > > > > Hi Viral, > > > > Any update on this (stats + df) by chance or idea when we can get one? Even > > a roadmap or some sort of vision or other details would help with > > decision making regarding infrastructure. > > > > Thanks! > > > > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: > > Yes, we are really excited. This grant is to focus on core Julia compiler > > infrastructure and key math libraries. Much of the libraries focus will be > > on statistical Computing. > > -viral > > > -viral
Re: [julia-users] Moore foundation grant.
Thanks for the response. Since you kindly asked, the following are two main areas in our assessment of the general arc of the Julia ecosystem: 1. Will the roadmap obviate some of the bottlenecks for day to day normal exploratory workflow? These are minimal things that R and Python have and whose lack hamper any use of Julia for regular analysis. Thing like robust dataframe with data i/o into different formats, web scraping, work out nullable semantics and integration with ecosystem , robust data cleaning and tidy data, modeling with basic diagnostic tests etc 2. Will the roadmap jump leapfrog into areas and capabilities that are currently not covered by other stats and data science ecosystems? There are many here, but we are specifically looking at the ability to work with modeling on medium sized out of core databases. This would include an abstract dataframe like interface to said databses MySQL and SQLlite, and some sort of modeling capability on the same. My dream would be separation of model specification as a DAG/ probabilistic programming framework, from fitting the model. Thus the same model can be fit with different sort of data and optimizers. Streaming black box variation inference can be a means to extend this to OOC work. I realize Julia won't for a while have all the statistical tests and random models of python, much less R. However, a general yet powerful and scalable data querying and prob programming framework could arguably suffice for most python and R use cases in Data Science while provide a comparative advantage over other frameworks where it counts. To my knowledge, Right now SAS and STATA are the only packages that offer general modeling with on disk data sets, but the sort of capability I outlined would seem to be in excess of what they offer. A bonus would be filling out gadfly towards Ggplot and ggvis capability. On Thursday, December 24, 2015 at 11:50:42 AM UTC-5, Viral Shah wrote: > > What would be helpful is to know what kind of decisions you are thinking > of and what are the factors. > > I suspect within 2 weeks for sure - but it's really for the Julia stats > folks to say. The idea is to get feedback and chart a course. > > -viral > On 24 Dec 2015 10:07 p.m., "Lampkld" > > wrote: > >> Sorry to bug you, but can we expect something this or next week? Would >> be helpful in knowing until when to push some stuff off. >> >> On Thursday, December 17, 2015 at 6:20:45 PM UTC-5, Viral Shah wrote: >>> >>> >>> The JuliaStats team will be publishing a general plan on stats+df in a >>> few days. I doubt we will have settled on all the df issues by then, but at >>> least there will be something to start with. >>> >>> >>> -viral >>> >>> >>> >>> > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: >>> > >>> > Hi Viral, >>> > >>> > Any update on this (stats + df) by chance or idea when we can get one? >>> Even a roadmap or some sort of vision or other details would help with >>> decision making regarding infrastructure. >>> > >>> > Thanks! >>> > >>> > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: >>> > Yes, we are really excited. This grant is to focus on core Julia >>> compiler infrastructure and key math libraries. Much of the libraries focus >>> will be on statistical Computing. >>> > -viral >>> > >>> >>>
Re: [julia-users] Moore foundation grant.
What would be helpful is to know what kind of decisions you are thinking of and what are the factors. I suspect within 2 weeks for sure - but it's really for the Julia stats folks to say. The idea is to get feedback and chart a course. -viral On 24 Dec 2015 10:07 p.m., "Lampkld" wrote: > Sorry to bug you, but can we expect something this or next week? Would > be helpful in knowing until when to push some stuff off. > > On Thursday, December 17, 2015 at 6:20:45 PM UTC-5, Viral Shah wrote: >> >> >> The JuliaStats team will be publishing a general plan on stats+df in a >> few days. I doubt we will have settled on all the df issues by then, but at >> least there will be something to start with. >> >> >> -viral >> >> >> >> > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: >> > >> > Hi Viral, >> > >> > Any update on this (stats + df) by chance or idea when we can get one? >> Even a roadmap or some sort of vision or other details would help with >> decision making regarding infrastructure. >> > >> > Thanks! >> > >> > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: >> > Yes, we are really excited. This grant is to focus on core Julia >> compiler infrastructure and key math libraries. Much of the libraries focus >> will be on statistical Computing. >> > -viral >> > >> >>
Re: [julia-users] Moore foundation grant.
Sorry to bug you, but can we expect something this or next week? Would be helpful in knowing until when to push some stuff off. On Thursday, December 17, 2015 at 6:20:45 PM UTC-5, Viral Shah wrote: > > > The JuliaStats team will be publishing a general plan on stats+df in a few > days. I doubt we will have settled on all the df issues by then, but at > least there will be something to start with. > > > -viral > > > > > On 17-Dec-2015, at 10:15 PM, Lampkld > > wrote: > > > > Hi Viral, > > > > Any update on this (stats + df) by chance or idea when we can get one? > Even a roadmap or some sort of vision or other details would help with > decision making regarding infrastructure. > > > > Thanks! > > > > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: > > Yes, we are really excited. This grant is to focus on core Julia > compiler infrastructure and key math libraries. Much of the libraries focus > will be on statistical Computing. > > -viral > > > >
Re: [julia-users] Moore foundation grant.
That would be very helpful, thanks! On Thursday, December 17, 2015 at 6:20:45 PM UTC-5, Viral Shah wrote: > > > The JuliaStats team will be publishing a general plan on stats+df in a few > days. I doubt we will have settled on all the df issues by then, but at > least there will be something to start with. > > > -viral > > > > > On 17-Dec-2015, at 10:15 PM, Lampkld > > wrote: > > > > Hi Viral, > > > > Any update on this (stats + df) by chance or idea when we can get one? > Even a roadmap or some sort of vision or other details would help with > decision making regarding infrastructure. > > > > Thanks! > > > > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: > > Yes, we are really excited. This grant is to focus on core Julia > compiler infrastructure and key math libraries. Much of the libraries focus > will be on statistical Computing. > > -viral > > > >
Re: [julia-users] Moore foundation grant.
The JuliaStats team will be publishing a general plan on stats+df in a few days. I doubt we will have settled on all the df issues by then, but at least there will be something to start with. -viral > On 17-Dec-2015, at 10:15 PM, Lampkld wrote: > > Hi Viral, > > Any update on this (stats + df) by chance or idea when we can get one? Even a > roadmap or some sort of vision or other details would help with decision > making regarding infrastructure. > > Thanks! > > On Wednesday, November 11, 2015 at 3:00:50 AM UTC-5, Viral Shah wrote: > Yes, we are really excited. This grant is to focus on core Julia compiler > infrastructure and key math libraries. Much of the libraries focus will be on > statistical Computing. > -viral >
Re: [julia-users] Moore foundation grant.
Fantastic.
Re: [julia-users] Moore foundation grant.
Wow!! Congratulations! Definitely, well deserved!! I am looking forward to see what Julia can represent for Scientific Computing in a couple of years! Best, Charles On 12 November 2015 at 08:12, Viral Shah wrote: > The core focus is actually on the compiler, debugger and libraries. On the > IDE front, we’ll be working on providing hooks to integrate the debugger > and such into existing IDE efforts - the prime being the Juno work, but > other projects will use it as well. > > Independently, we are also discussing with the Eclipse Foundation to see > how we can work with their scientific community and have a good Eclipse > plugin for Julia. > > -viral > > > > > On 11-Nov-2015, at 10:42 PM, Hans-Peter wrote: > > > > Great! Congratulations!! > > > > I read about an IDE. Do you already have ideas, plans what such an IDE > could look like? And/or a word about technology? > > > > > > On Wednesday, 11 November 2015 09:00:50 UTC+1, Viral Shah wrote: > > Yes, we are really excited. This grant is to focus on core Julia > compiler infrastructure and key math libraries. Much of the libraries focus > will be on statistical Computing. > > -viral > > > > -- Um axé! :) -- Charles Novaes de Santana, PhD http://www.imedea.uib-csic.es/~charles
Re: [julia-users] Moore foundation grant.
The core focus is actually on the compiler, debugger and libraries. On the IDE front, we’ll be working on providing hooks to integrate the debugger and such into existing IDE efforts - the prime being the Juno work, but other projects will use it as well. Independently, we are also discussing with the Eclipse Foundation to see how we can work with their scientific community and have a good Eclipse plugin for Julia. -viral > On 11-Nov-2015, at 10:42 PM, Hans-Peter wrote: > > Great! Congratulations!! > > I read about an IDE. Do you already have ideas, plans what such an IDE could > look like? And/or a word about technology? > > > On Wednesday, 11 November 2015 09:00:50 UTC+1, Viral Shah wrote: > Yes, we are really excited. This grant is to focus on core Julia compiler > infrastructure and key math libraries. Much of the libraries focus will be on > statistical Computing. > -viral >
Re: [julia-users] Moore foundation grant.
Yes, that is what the plan is broadly - to push the core plumbing libraries further, consolidate efforts, etc. -viral > On 11-Nov-2015, at 10:15 PM, David Anthoff wrote: > > Fantastic news, congrats! > > When you refer to the stats focus in terms of libraries, does that mean > things like sorting out the type stability issues in DataFrames, making > NullableArrays the core of DataFrames etc? I.e. the core plumbing libraries, > or something else? Very exciting news in either way. > > Best, > David > >> -Original Message- >> From: julia-users@googlegroups.com [mailto:julia- >> us...@googlegroups.com] On Behalf Of Viral Shah >> Sent: Wednesday, November 11, 2015 12:01 AM >> To: julia-users >> Subject: [julia-users] Moore foundation grant. >> >> Yes, we are really excited. This grant is to focus on core Julia compiler >> infrastructure and key math libraries. Much of the libraries focus will be on >> statistical Computing. >> >> -viral
RE: [julia-users] Moore foundation grant.
Fantastic news, congrats! When you refer to the stats focus in terms of libraries, does that mean things like sorting out the type stability issues in DataFrames, making NullableArrays the core of DataFrames etc? I.e. the core plumbing libraries, or something else? Very exciting news in either way. Best, David > -Original Message- > From: julia-users@googlegroups.com [mailto:julia- > us...@googlegroups.com] On Behalf Of Viral Shah > Sent: Wednesday, November 11, 2015 12:01 AM > To: julia-users > Subject: [julia-users] Moore foundation grant. > > Yes, we are really excited. This grant is to focus on core Julia compiler > infrastructure and key math libraries. Much of the libraries focus will be on > statistical Computing. > > -viral
Re: [julia-users] Moore foundation grant.
We are glad that the Moore Foundation, especially Chris Mentzel, has supported Julia in a number of ways. -viral On 11 Nov 2015 3:04 p.m., "Kristoffer Carlsson" wrote: > Wow, awesome! Well deserved!
Re: [julia-users] Moore foundation grant.
Wow, awesome! Well deserved!
Re: [julia-users] Moore foundation grant.
Awesome :+1: (Thanks for the link!) On Wednesday, November 11, 2015 at 5:01:25 PM UTC+8, Mauro wrote: > > In case it is not just me who didn't spot the news, here it is: > > https://www.moore.org/newsroom/in-the-news/2015/11/10/bringing-julia-from-beta-to-1.0-to-support-data-intensive-scientific-computing > > > Nice! > > On Wed, 2015-11-11 at 09:00, Viral Shah > > wrote: > > Yes, we are really excited. This grant is to focus on core Julia > compiler infrastructure and key math libraries. Much of the libraries focus > will be on statistical Computing. > > > > -viral >
Re: [julia-users] Moore foundation grant.
In case it is not just me who didn't spot the news, here it is: https://www.moore.org/newsroom/in-the-news/2015/11/10/bringing-julia-from-beta-to-1.0-to-support-data-intensive-scientific-computing Nice! On Wed, 2015-11-11 at 09:00, Viral Shah wrote: > Yes, we are really excited. This grant is to focus on core Julia compiler > infrastructure and key math libraries. Much of the libraries focus will be on > statistical Computing. > > -viral
[julia-users] Moore foundation grant.
Yes, we are really excited. This grant is to focus on core Julia compiler infrastructure and key math libraries. Much of the libraries focus will be on statistical Computing. -viral
[julia-users] Moore foundation grant.
Congratulations on the exciting news! I have played around with Julia A bit and love the language , But Found Its Lacking Some robust stats/ml/data manipulation infrastructure. Can anyone tell me if these areas are slated to be worked on from the grant money? I'm trying to get some foresight on the ecosystem so I can make some decisions to use Julia for some projects. thanks !
