On Thu, Aug 8, 2013 at 1:19 PM, Matthew Rocklin <mrock...@gmail.com> wrote:
> Ok, Just went through this thread again. > > It should be simple to translate SymPy.Piecewise to a recursive > Theano.switch (after translating SymPy.LT to theano.lt, etc.) I'll get > on this soon. Does this sound reasonable to you Fred? > > Jason's PR allowing pass through of keyword arguments looks close to me. > I suspect that the `on_unused_input=ignore` issue will soon be possible. > Question, should this be default? Are there other defaults that should be > used? I'm curious why you both are making functions with unused inputs. > If I generate tons of long ode's I don't really care to search through them to find all of the unique arguments for each equation. We setup a multibody problem with lots of constants and lots of time varying state variables (+ others too). Every ode generated will only use a subset of these variables for the whole system. So it is just easier to pass in all the numerical values it could possibly need instead of trying to figure out which ones it does need. But we could surely be more precise about this, it's just easier to pass them all in. > > > Jason, was Fred's note on directly passing constants useful? If not why > not? It sounds like you should subs your constant symbols for numeric > values prior to creating the theano graph. > I'll try that, but subs can be time consuming for our bigger problems. > > > With regards to using an externally defined theano graph or op (theano > wrapped sympy implementation), how would one pass them to theano_function? > You wouldn't. You would use the other common interface function, > theano_code to create theano variables instead. You would then use raw > theano operations (like theano.function) to compile your graph from theano > variables. Theano_function is a convenient all-in-one interface function > for the common case. If you want to do anything custom I recommend using > SymPy and Theano natively, translating from one to the other using > theano_code instead. It does nothing except translate. > > > Does sympy.printing.theanocode.theano_function automatically optimize > the compiled graph? > Yes. theano_function translates sympy expressions into theano expressions > using theano_code. It then calls theano.function on these expressions. > theano.function optimizes by default. > > > SymPy C Codegen and Theano > > @Fred, how hard would it be to leverage SymPy's C codegen in Theano? This > might be a lot cleaner than wrapping raw SymPy operations and might > substantially extend Theano's support of scalar expressions. Do you have a > performant Bessel function op? I'll bet SymPy could be made to do this > quite well. > > @Aaron / @Ondrej, if you're reading this thread could you point us to the > best place to start looking at C codegen in SymPy? Alternatively can you > point to an active community member who would be able to do so? > > > > ODE Integration > > What are people's thoughts on scipy.integrate.odeint? Earlier this year I > was looking at nodepy <https://github.com/ketch/nodepy>, a purely > mathematical treatment/catalog of time stepping schemes. It should be easy > to generate such schemes in Theano using simple abstractions like Butcher > matrices. The code I was playing around with was > https://github.com/mrocklin/tsgen. The goal was that we could easily > experiment with and rapidly benchmark different time stepping schemes for > specific problems. > > > > > On Wed, Aug 7, 2013 at 10:37 AM, Guy Parsey <guy.par...@gmail.com> wrote: > >> Hey Fred, >> I am starting to play with making a theano.Op, and in principle, linking >> a custom theano.Op to an implemented sympy function will solve all of my >> problems (at least in the sense of getting my system running, not >> necessarily speed). My initial post spoke of both mapping Piecewise and >> interpolated functions into theano; after speaking with my advisor, due to >> wanting a smoothing criterion over our piecewise datasets, I think we will >> end up treating them as interpolated functions over smoothed data sets. My >> idea for how to use an interpolation routine will be a wrapped theano.Op, >> so I am still rather curious as to how implementing Piecewise would work in >> such way that theano can manipulate it in graph form (built out of native >> theano operations). >> >> Theano will merge duplicated subexpression automatically by default. What >> is your function? If it is too "simple", for example just working on >> scalar, Theano have by default much overhead. As said for the other >> benchmark, there is tricks to remove a big part or even all of it, but it >> ask for some work on the developer. >> >> >> The basic system/functions can be described as follows; we have some >> number of species (n_0 ...n_k) and we are creating a function for the >> vector of time derivatives. The different types of expressions come about >> from the reaction rates (K_0 ..... K_l) The whole system is described >> symbolically, for symbolic determination of the jacobian (spline >> interpolated functions are wrapped with a sympy implemented function so >> that taking a symbolic derivative also yields the derivative of the >> numerical implementation). Dummy example below with three types of base >> expressions >> d(n_i)/dt = K_0*n_2*n_0 + K_1*n_3 + K_2*n_4*n_2*n_4/(sum(n_0....n_k)) >> Each of the K terms can be described differently: >> --constant >> --analytical parameterized of some n_j (Kf ~ a0*(n_j**a1)*exp(a2/n_j) >> , a0..a2 are constants) >> --spline interpolation function of some n_j (K_interp(n_j)) >> --analytical weighted with piecewise/interpolated (new_Kf ~ >> Kf*exp(-K_interp(n_j)) >> I also have two energy equations, but these incorporate all of the same >> types of terms, just not in a nice summation as in the species equations. >> Any given base expression (eg. K_0*n_2*n_0) will appear in multiple time >> derivative equations, so if the interpolation is slow, it would be nice to >> get theano to recognize that it is a 'common expression.' I can make test >> cases with a few species and a few reactions (Ks), but the true systems I >> would like to be simulating will have 40-100 species and 300-2000 reaction >> rates. Are these functions too 'simple' to be sped by using theano? There >> is a lot of optimization that can be done due to the high number of shared >> expressions, so I was hoping to take advantage of the theano graph >> optimizations. >> >> My initial test cases were using the numpy.float64 dtype, so I will >> switch it over to ndarrays if possible. >> I hope to have questions or a working spline interpolation theano.Op in >> the next few hours. >> Cheers, >> Guy >> >> Quick Codegen question: In principle, can one create a python callback >> within a compiled c code to a python function that is not compiled >> (interpolation function)? >> >> >> On Tuesday, August 6, 2013 1:36:29 PM UTC-4, Frédéric Bastien wrote: >> >>> >>> >>> >>> On Tue, Aug 6, 2013 at 11:04 AM, Guy Parsey <guy.p...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> Theano/Sympy questions >>>> Fred, in terms of replacing sympy.Pieceiwse with a Theano equivalent, >>>> since sympy.Piecewise attempts each condition (from ExprCondPair) until one >>>> is valid, I would think that the closest equivalent would be a recursive >>>> theano ifelse (ie. sympy.Piecewise((expr1,cond1),**(expr2,cond2)) ~ >>>> theano.ifelse(cond1,expr1,**theano.ifelse(cond2,expr2,**None))) >>>> statement so that the expressions are not evaluated until the condition is >>>> achieved. This might be too much of a patch though and I am not sure how to >>>> implement it with the same argument structure as sympy.Pieceiwse. >>>> >>> >>> It would work. But for the timming, I would also try with the switch op >>> instead of ifelse. The ifelse currently have much bigger oveahead then the >>> switch op, and even if the switch op evaluate both branch, it happen >>> relatively frequently that it is faster then the ifelse. Especially if each >>> expression have small size, like a scalar or a small vector. >>> >>> >>>> With regards to using an externally defined theano graph or op (theano >>>> wrapped sympy implementation), how would one pass them to theano_function? >>>> It feels as though it would be analogous to the autowrap helpers argument, >>>> or would this be an issue of merging graphs before calling theano.function? >>>> If I can understand this step, I feel as though both of my problems would >>>> be solvable (a wrapping of a piecewise along with a wrapping of an >>>> interpolated function) >>>> >>> >>> There is 2 steps. 1) make a theano op like descrided here: >>> http://deeplearning.net/**software/theano/tutorial/** >>> extending_theano.html<http://deeplearning.net/software/theano/tutorial/extending_theano.html> >>> >>> 2) Modify theano_function to use the new Theano op for the equivalent >>> SymPy op. It could be possible to make an autowrap for this, but I don't >>> know enough of the detail to be sure. >>> >>> >>>> In all of my simplified test cases (no pieceiwse or interpolations, 9 >>>> heavily-linked non-linear ODEs), I also show that the lambdify function is >>>> faster to evaluate than my created theano function. When passing a list of >>>> expressions, if there is a commons subexpression that exists in two >>>> expressions (not twice in the same expression), is it treated as such >>>> during theano compilation or is each expression handled separately? >>>> >>> >>> Theano will merge duplicated subexpression automatically by default. >>> What is your function? If it is too "simple", for example just working on >>> scalar, Theano have by default much overhead. As said for the other >>> benchmark, there is tricks to remove a big part or even all of it, but it >>> ask for some work on the developer. >>> >>> >>>> >>>> Codegen questions (maybe a different topic): >>>> Jason, I have started playing with the sympy.printing.codegen (C for >>>> now, Fortran later) as an alternative to theano (I would very much like to >>>> get multiple methods working, and GPU acceleration makes me want to keep >>>> theano implementation). There is a size-able speedup relative to lambdify >>>> or theano_function, which is to be expected, but I arrive at the same >>>> problem as above with regards to interpolated or piecewise functions. The >>>> initial crash from using Piecewise functions comes from Routine calling >>>> sympy.tensor.index_methods get_contraction_structure which explicitly >>>> states no support for Piecewise function types. In principle, if one made a >>>> symbolic implemented function (preferably without an analytical >>>> representation), could one link the python object to the compiled code? >>>> What attributes would said implemented_function need to have? >>>> >>>> Thank you for your responses, I am glad to hear that at least I am not >>>> completely missing something. >>>> Cheers, >>>> Guy >>>> >>>> >>>> On Tuesday, August 6, 2013 9:37:41 AM UTC-4, Jason Moore wrote: >>>> >>>>> Fred, >>>>> >>>>> I think on_used_input=ignore should be a default in Matthew's theano >>>>> printing code or that arg needs to be pushed up to his layer. I hit that >>>>> issue too. >>>>> >>>>> The code I have is here: https://github.com/**PythonDynam** >>>>> ics/pydy-code-gen <https://github.com/PythonDynamics/pydy-code-gen> >>>>> >>>>> See the results.txt file for basic speed comparisons. I'm generating >>>>> the ODEs for an n-link pendulum with mechanics and then see how fast it >>>>> generates and simulates with scipy.odeint. >>>>> >>>>> The code doesn't work at the moment. I haven't touched it in a month >>>>> and looks like some things have changed in sympy and/or theano. I'll work >>>>> on the bugs now. >>>>> >>>>> But "python benchmark.py" should run it with sympy master and ?some? >>>>> version of Theano. >>>>> >>>>> >>>>> Jason >>>>> moorepants.info >>>>> +01 530-601-9791 >>>>> >>>>> >>>>> On Tue, Aug 6, 2013 at 9:25 AM, Frédéric Bastien <no...@nouiz.org>wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I don't know what is sympy.functions.elementary.****piecewise. Do >>>>>> Jason answered that part? If not, I'll look into it to know how to make >>>>>> Theano reproduce it. About converting any Sympy symbol to Theano symbol, >>>>>> when there isn't a one to one matching, you can create a one to a full >>>>>> Theano graph conversion. When this is possible, it is probably the best, >>>>>> as >>>>>> if you make a new Theano op, it work work on the GPU. But if you make a >>>>>> Theano graph, there is good change that the graph will already work on >>>>>> the >>>>>> GPU. >>>>>> >>>>>> If it is not possible to make a Theano graph for a sympy symbol, it >>>>>> is possible to make a new Theano op that just wrap the sympy >>>>>> implementation. Also, if this is a bottleneck, I recently added an >>>>>> example >>>>>> that show how to use numba with Theano so speed up the python code in a >>>>>> Theano op. >>>>>> >>>>>> >>>>>> Theano do not parallelize on the CPU, except for the call to BLAS, >>>>>> when the BLAS library is parallel. On the GPU, it is parallel. >>>>>> >>>>>> Having on_unused_input=ignore is normal for complicated generated >>>>>> code. That is why it was added. But when the code is simpler and not >>>>>> generated, but all user manually coded, most of time it mean the user >>>>>> didn't do what he wanted. If you know it is normal that you have unused >>>>>> input, there is no problem to use that flag. >>>>>> >>>>>> >>>>>> Jason, about the case where Theano is slower, can you send me the >>>>>> Theano code? I would like to look at it. I'm very surprised that Theano >>>>>> is >>>>>> slower then Sympy in this case and would like to know why it is like >>>>>> this. >>>>>> >>>>>> Fred >>>>>> >>>>>> >>>>>> On Mon, Aug 5, 2013 at 9:03 PM, Jason Moore <moore...@gmail.com>wrote: >>>>>> >>>>>>> Guy, >>>>>>> >>>>>>> We're working on the same problem for sympy.physics.mechanics. >>>>>>> Matthew Rocklin added support for matrix conversions in the theano code >>>>>>> that is in SymPy and I used that, but found that theano was slower that >>>>>>> lambdify for most of my cases (I only have two cores, so I'm not taking >>>>>>> advantage of the Theano parallel stuff). I think writing specific code >>>>>>> gen >>>>>>> for ode integration is going to be the best bet. I'm happy to >>>>>>> collaborate >>>>>>> on this with you. >>>>>>> >>>>>>> >>>>>>> Jason >>>>>>> moorepants.info >>>>>>> +01 530-601-9791 >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 5, 2013 at 3:13 PM, Guy Parsey <guy.p...@gmail.com>wrote: >>>>>>> >>>>>>>> Hello Everyone, >>>>>>>> Thank you in advance for reading through my problem and for any >>>>>>>> input you may have. I must say that I still feel like a novice >>>>>>>> programmer >>>>>>>> and my problems may be easily solvable from a different mindset. My >>>>>>>> present >>>>>>>> project entails time-integration of extremely stiff and non-linear ODEs >>>>>>>> with regards to chemical kinetics (one derivative equation for each >>>>>>>> variable species) and energy equations. Initially we were planning on >>>>>>>> using >>>>>>>> the sympy.lambdify function to create callable functions for the main >>>>>>>> function along with the jacobian and passing said functions to >>>>>>>> scipy.integrate.odeint, however this method only works for easier test >>>>>>>> cases (fewer species and/or no energy equations) before being limited >>>>>>>> by >>>>>>>> either the list recursion limit or segfaulting due to the limited stack >>>>>>>> size. I know that both of these limits can be edited, but that fact >>>>>>>> that I >>>>>>>> am reaching them makes me feel as though I am doing something extremely >>>>>>>> inefficiently. Outside of the documentation of SymPy and Theano, I have >>>>>>>> also been heavily using the BlogPost by Matthew Rocklin >>>>>>>> http://matthewrocklin.****com/blog/work/2013/03/19/**SymPy** >>>>>>>> -Theano-part-1/<http://matthewrocklin.com/blog/work/2013/03/19/SymPy-Theano-part-1/>. >>>>>>>> >>>>>>>> Presently I am trying to use the mapping between Theano and SymPy >>>>>>>> (sympy.printing.theanocode theano_function) to make my callable >>>>>>>> functions >>>>>>>> and take advantage of the optimization routines. I have two major >>>>>>>> problems >>>>>>>> and a few questions: >>>>>>>> >>>>>>>> 1st major problem: Though piecewise functions exist in SymPy >>>>>>>> (sympy.functions.elementary.**pi**ecewise) there is no counterpart >>>>>>>> in Theano. Looking at the source of the inspiration for theanocode ( >>>>>>>> https://github.com/nouiz/**thea**no_sympy/<https://github.com/nouiz/theano_sympy/> >>>>>>>> graph_translation.py) I see that some of the SymPy equivalents were >>>>>>>> defined as lambda functions. Is there an equivalent way to add Theano >>>>>>>> conditional expressions wrapped into a function to add to the mapping >>>>>>>> dictionary in theanocode.py? >>>>>>>> >>>>>>>> 2nd major problem: Similar to the problem above in that I am not >>>>>>>> sure that the Theano counterpart is; some of the terms that I use are >>>>>>>> interpolated functions (with one ODE variable as input) that we have >>>>>>>> wrapped symbolically while providing a numerical implementation (so >>>>>>>> that >>>>>>>> symbolic derivatives can be made, resulting in their own >>>>>>>> interpolations). >>>>>>>> Is it possible to recreate the interpolation function as a Theano >>>>>>>> operation >>>>>>>> for use within the system of ODEs? >>>>>>>> >>>>>>>> Remain questions: >>>>>>>> I presently have to flatten my input to theano_function to a list >>>>>>>> of expressions and then wrap to return to a form (Jacobian is a matrix >>>>>>>> not >>>>>>>> a vector); is it possible to have a matrix of different expressions as >>>>>>>> an >>>>>>>> input to theano_function with a vector output? >>>>>>>> >>>>>>>> I know that a huge amount of Theano speed up is due to >>>>>>>> parallelization of matrix operations (which I do not have), should I be >>>>>>>> focusing on SymPy Autowrap/Ufuncify or my own code generation instead >>>>>>>> of >>>>>>>> trying to get Theano to play nicely? >>>>>>>> >>>>>>>> Stupid questions: >>>>>>>> Does sympy.printing.theanocode.**thea**no_function automatically >>>>>>>> optimize the compiled graph? >>>>>>>> >>>>>>>> Minor comment: >>>>>>>> Perhaps unnecessary for most uses of the theano_function, but I >>>>>>>> needed to modify function inputs so as to be able to use the keyword >>>>>>>> argument 'on_unused_input=ignore' as opposed to 'raise' so that I did >>>>>>>> not >>>>>>>> need to have all symbols in all equations. This may be avoided by >>>>>>>> having >>>>>>>> the unused symbols somehow (I don't know how) included in each >>>>>>>> expression. >>>>>>>> >>>>>>>> Thank you again for your time in reading my problems and any >>>>>>>> potential help you may think of. I can attach code if necessary, I just >>>>>>>> didn't want to make my post more confusing. >>>>>>>> Have an excellent day. >>>>>>>> Sincerely, >>>>>>>> Guy Parsey >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "sympy" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to sympy+un...@**googlegroups.com. >>>>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>>>> >>>>>>>> Visit this group at >>>>>>>> http://groups.google.com/**group**/sympy<http://groups.google.com/group/sympy> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/**grou** >>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "sympy" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to sympy+un...@**googlegroups.com. >>>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>>> >>>>>>> Visit this group at >>>>>>> http://groups.google.com/**group**/sympy<http://groups.google.com/group/sympy> >>>>>>> . >>>>>>> For more options, visit >>>>>>> https://groups.google.com/**grou**ps/opt_out<https://groups.google.com/groups/opt_out> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "sympy" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to sympy+un...@**googlegroups.com. >>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>> >>>>>> Visit this group at >>>>>> http://groups.google.com/**group**/sympy<http://groups.google.com/group/sympy> >>>>>> . >>>>>> For more options, visit >>>>>> https://groups.google.com/**grou**ps/opt_out<https://groups.google.com/groups/opt_out> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "sympy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to sympy+un...@**googlegroups.com. >>>> To post to this group, send email to sy...@googlegroups.com. >>>> Visit this group at >>>> http://groups.google.com/**group/sympy<http://groups.google.com/group/sympy> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "sympy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sympy+unsubscr...@googlegroups.com. >> To post to this group, send email to sympy@googlegroups.com. >> Visit this group at http://groups.google.com/group/sympy. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To post to this group, send email to sympy@googlegroups.com. > Visit this group at http://groups.google.com/group/sympy. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. For more options, visit https://groups.google.com/groups/opt_out.
