Hey Fred, I am starting to play with making a theano.Op, and in principle, linking a custom theano.Op to an implemented sympy function will solve all of my problems (at least in the sense of getting my system running, not necessarily speed). My initial post spoke of both mapping Piecewise and interpolated functions into theano; after speaking with my advisor, due to wanting a smoothing criterion over our piecewise datasets, I think we will end up treating them as interpolated functions over smoothed data sets. My idea for how to use an interpolation routine will be a wrapped theano.Op, so I am still rather curious as to how implementing Piecewise would work in such way that theano can manipulate it in graph form (built out of native theano operations).
Theano will merge duplicated subexpression automatically by default. What is your function? If it is too "simple", for example just working on scalar, Theano have by default much overhead. As said for the other benchmark, there is tricks to remove a big part or even all of it, but it ask for some work on the developer. The basic system/functions can be described as follows; we have some number of species (n_0 ...n_k) and we are creating a function for the vector of time derivatives. The different types of expressions come about from the reaction rates (K_0 ..... K_l) The whole system is described symbolically, for symbolic determination of the jacobian (spline interpolated functions are wrapped with a sympy implemented function so that taking a symbolic derivative also yields the derivative of the numerical implementation). Dummy example below with three types of base expressions d(n_i)/dt = K_0*n_2*n_0 + K_1*n_3 + K_2*n_4*n_2*n_4/(sum(n_0....n_k)) Each of the K terms can be described differently: --constant --analytical parameterized of some n_j (Kf ~ a0*(n_j**a1)*exp(a2/n_j) , a0..a2 are constants) --spline interpolation function of some n_j (K_interp(n_j)) --analytical weighted with piecewise/interpolated (new_Kf ~ Kf*exp(-K_interp(n_j)) I also have two energy equations, but these incorporate all of the same types of terms, just not in a nice summation as in the species equations. Any given base expression (eg. K_0*n_2*n_0) will appear in multiple time derivative equations, so if the interpolation is slow, it would be nice to get theano to recognize that it is a 'common expression.' I can make test cases with a few species and a few reactions (Ks), but the true systems I would like to be simulating will have 40-100 species and 300-2000 reaction rates. Are these functions too 'simple' to be sped by using theano? There is a lot of optimization that can be done due to the high number of shared expressions, so I was hoping to take advantage of the theano graph optimizations. My initial test cases were using the numpy.float64 dtype, so I will switch it over to ndarrays if possible. I hope to have questions or a working spline interpolation theano.Op in the next few hours. Cheers, Guy Quick Codegen question: In principle, can one create a python callback within a compiled c code to a python function that is not compiled (interpolation function)? On Tuesday, August 6, 2013 1:36:29 PM UTC-4, Frédéric Bastien wrote: > > > > > On Tue, Aug 6, 2013 at 11:04 AM, Guy Parsey <guy.p...@gmail.com<javascript:> > > wrote: > >> Hello, >> >> Theano/Sympy questions >> Fred, in terms of replacing sympy.Pieceiwse with a Theano equivalent, >> since sympy.Piecewise attempts each condition (from ExprCondPair) until one >> is valid, I would think that the closest equivalent would be a recursive >> theano ifelse (ie. sympy.Piecewise((expr1,cond1),(expr2,cond2)) ~ >> theano.ifelse(cond1,expr1,theano.ifelse(cond2,expr2,None))) statement so >> that the expressions are not evaluated until the condition is achieved. >> This might be too much of a patch though and I am not sure how to implement >> it with the same argument structure as sympy.Pieceiwse. >> > > It would work. But for the timming, I would also try with the switch op > instead of ifelse. The ifelse currently have much bigger oveahead then the > switch op, and even if the switch op evaluate both branch, it happen > relatively frequently that it is faster then the ifelse. Especially if each > expression have small size, like a scalar or a small vector. > > >> With regards to using an externally defined theano graph or op (theano >> wrapped sympy implementation), how would one pass them to theano_function? >> It feels as though it would be analogous to the autowrap helpers argument, >> or would this be an issue of merging graphs before calling theano.function? >> If I can understand this step, I feel as though both of my problems would >> be solvable (a wrapping of a piecewise along with a wrapping of an >> interpolated function) >> > > There is 2 steps. 1) make a theano op like descrided here: > http://deeplearning.net/software/theano/tutorial/extending_theano.html > > 2) Modify theano_function to use the new Theano op for the equivalent > SymPy op. It could be possible to make an autowrap for this, but I don't > know enough of the detail to be sure. > > >> In all of my simplified test cases (no pieceiwse or interpolations, 9 >> heavily-linked non-linear ODEs), I also show that the lambdify function is >> faster to evaluate than my created theano function. When passing a list of >> expressions, if there is a commons subexpression that exists in two >> expressions (not twice in the same expression), is it treated as such >> during theano compilation or is each expression handled separately? >> > > Theano will merge duplicated subexpression automatically by default. What > is your function? If it is too "simple", for example just working on > scalar, Theano have by default much overhead. As said for the other > benchmark, there is tricks to remove a big part or even all of it, but it > ask for some work on the developer. > > >> >> Codegen questions (maybe a different topic): >> Jason, I have started playing with the sympy.printing.codegen (C for now, >> Fortran later) as an alternative to theano (I would very much like to get >> multiple methods working, and GPU acceleration makes me want to keep theano >> implementation). There is a size-able speedup relative to lambdify or >> theano_function, which is to be expected, but I arrive at the same problem >> as above with regards to interpolated or piecewise functions. The initial >> crash from using Piecewise functions comes from Routine calling >> sympy.tensor.index_methods get_contraction_structure which explicitly >> states no support for Piecewise function types. In principle, if one made a >> symbolic implemented function (preferably without an analytical >> representation), could one link the python object to the compiled code? >> What attributes would said implemented_function need to have? >> >> Thank you for your responses, I am glad to hear that at least I am not >> completely missing something. >> Cheers, >> Guy >> >> >> On Tuesday, August 6, 2013 9:37:41 AM UTC-4, Jason Moore wrote: >> >>> Fred, >>> >>> I think on_used_input=ignore should be a default in Matthew's theano >>> printing code or that arg needs to be pushed up to his layer. I hit that >>> issue too. >>> >>> The code I have is here: https://github.com/** >>> PythonDynamics/pydy-code-gen<https://github.com/PythonDynamics/pydy-code-gen> >>> >>> See the results.txt file for basic speed comparisons. I'm generating the >>> ODEs for an n-link pendulum with mechanics and then see how fast it >>> generates and simulates with scipy.odeint. >>> >>> The code doesn't work at the moment. I haven't touched it in a month and >>> looks like some things have changed in sympy and/or theano. I'll work on >>> the bugs now. >>> >>> But "python benchmark.py" should run it with sympy master and ?some? >>> version of Theano. >>> >>> >>> Jason >>> moorepants.info >>> +01 530-601-9791 >>> >>> >>> On Tue, Aug 6, 2013 at 9:25 AM, Frédéric Bastien <no...@nouiz.org>wrote: >>> >>>> Hi, >>>> >>>> I don't know what is sympy.functions.elementary.**piecewise. Do Jason >>>> answered that part? If not, I'll look into it to know how to make Theano >>>> reproduce it. About converting any Sympy symbol to Theano symbol, when >>>> there isn't a one to one matching, you can create a one to a full Theano >>>> graph conversion. When this is possible, it is probably the best, as if >>>> you >>>> make a new Theano op, it work work on the GPU. But if you make a Theano >>>> graph, there is good change that the graph will already work on the GPU. >>>> >>>> If it is not possible to make a Theano graph for a sympy symbol, it is >>>> possible to make a new Theano op that just wrap the sympy implementation. >>>> Also, if this is a bottleneck, I recently added an example that show how >>>> to >>>> use numba with Theano so speed up the python code in a Theano op. >>>> >>>> >>>> Theano do not parallelize on the CPU, except for the call to BLAS, when >>>> the BLAS library is parallel. On the GPU, it is parallel. >>>> >>>> Having on_unused_input=ignore is normal for complicated generated >>>> code. That is why it was added. But when the code is simpler and not >>>> generated, but all user manually coded, most of time it mean the user >>>> didn't do what he wanted. If you know it is normal that you have unused >>>> input, there is no problem to use that flag. >>>> >>>> >>>> Jason, about the case where Theano is slower, can you send me the >>>> Theano code? I would like to look at it. I'm very surprised that Theano is >>>> slower then Sympy in this case and would like to know why it is like this. >>>> >>>> Fred >>>> >>>> >>>> On Mon, Aug 5, 2013 at 9:03 PM, Jason Moore <moore...@gmail.com> wrote: >>>> >>>>> Guy, >>>>> >>>>> We're working on the same problem for sympy.physics.mechanics. Matthew >>>>> Rocklin added support for matrix conversions in the theano code that is >>>>> in >>>>> SymPy and I used that, but found that theano was slower that lambdify for >>>>> most of my cases (I only have two cores, so I'm not taking advantage of >>>>> the >>>>> Theano parallel stuff). I think writing specific code gen for ode >>>>> integration is going to be the best bet. I'm happy to collaborate on this >>>>> with you. >>>>> >>>>> >>>>> Jason >>>>> moorepants.info >>>>> +01 530-601-9791 >>>>> >>>>> >>>>> On Mon, Aug 5, 2013 at 3:13 PM, Guy Parsey <guy.p...@gmail.com> wrote: >>>>> >>>>>> Hello Everyone, >>>>>> Thank you in advance for reading through my problem and for any input >>>>>> you may have. I must say that I still feel like a novice programmer and >>>>>> my >>>>>> problems may be easily solvable from a different mindset. My present >>>>>> project entails time-integration of extremely stiff and non-linear ODEs >>>>>> with regards to chemical kinetics (one derivative equation for each >>>>>> variable species) and energy equations. Initially we were planning on >>>>>> using >>>>>> the sympy.lambdify function to create callable functions for the main >>>>>> function along with the jacobian and passing said functions to >>>>>> scipy.integrate.odeint, however this method only works for easier test >>>>>> cases (fewer species and/or no energy equations) before being limited by >>>>>> either the list recursion limit or segfaulting due to the limited stack >>>>>> size. I know that both of these limits can be edited, but that fact that >>>>>> I >>>>>> am reaching them makes me feel as though I am doing something extremely >>>>>> inefficiently. Outside of the documentation of SymPy and Theano, I have >>>>>> also been heavily using the BlogPost by Matthew Rocklin >>>>>> http://matthewrocklin.**com/blog/work/2013/03/19/** >>>>>> SymPy-Theano-part-1/<http://matthewrocklin.com/blog/work/2013/03/19/SymPy-Theano-part-1/>. >>>>>> >>>>>> >>>>>> Presently I am trying to use the mapping between Theano and SymPy >>>>>> (sympy.printing.theanocode theano_function) to make my callable >>>>>> functions >>>>>> and take advantage of the optimization routines. I have two major >>>>>> problems >>>>>> and a few questions: >>>>>> >>>>>> 1st major problem: Though piecewise functions exist in SymPy >>>>>> (sympy.functions.elementary.**piecewise) there is no counterpart in >>>>>> Theano. Looking at the source of the inspiration for theanocode ( >>>>>> https://github.com/nouiz/**theano_sympy/<https://github.com/nouiz/theano_sympy/> >>>>>> graph_translation.py) I see that some of the SymPy equivalents were >>>>>> defined as lambda functions. Is there an equivalent way to add Theano >>>>>> conditional expressions wrapped into a function to add to the mapping >>>>>> dictionary in theanocode.py? >>>>>> >>>>>> 2nd major problem: Similar to the problem above in that I am not sure >>>>>> that the Theano counterpart is; some of the terms that I use are >>>>>> interpolated functions (with one ODE variable as input) that we have >>>>>> wrapped symbolically while providing a numerical implementation (so that >>>>>> symbolic derivatives can be made, resulting in their own >>>>>> interpolations). >>>>>> Is it possible to recreate the interpolation function as a Theano >>>>>> operation >>>>>> for use within the system of ODEs? >>>>>> >>>>>> Remain questions: >>>>>> I presently have to flatten my input to theano_function to a list of >>>>>> expressions and then wrap to return to a form (Jacobian is a matrix not >>>>>> a >>>>>> vector); is it possible to have a matrix of different expressions as an >>>>>> input to theano_function with a vector output? >>>>>> >>>>>> I know that a huge amount of Theano speed up is due to >>>>>> parallelization of matrix operations (which I do not have), should I be >>>>>> focusing on SymPy Autowrap/Ufuncify or my own code generation instead of >>>>>> trying to get Theano to play nicely? >>>>>> >>>>>> Stupid questions: >>>>>> Does sympy.printing.theanocode.**theano_function automatically >>>>>> optimize the compiled graph? >>>>>> >>>>>> Minor comment: >>>>>> Perhaps unnecessary for most uses of the theano_function, but I >>>>>> needed to modify function inputs so as to be able to use the keyword >>>>>> argument 'on_unused_input=ignore' as opposed to 'raise' so that I did >>>>>> not >>>>>> need to have all symbols in all equations. This may be avoided by having >>>>>> the unused symbols somehow (I don't know how) included in each >>>>>> expression. >>>>>> >>>>>> Thank you again for your time in reading my problems and any >>>>>> potential help you may think of. I can attach code if necessary, I just >>>>>> didn't want to make my post more confusing. >>>>>> Have an excellent day. >>>>>> Sincerely, >>>>>> Guy Parsey >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "sympy" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to sympy+un...@**googlegroups.com. >>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>> >>>>>> Visit this group at >>>>>> http://groups.google.com/**group/sympy<http://groups.google.com/group/sympy> >>>>>> . >>>>>> For more options, visit >>>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "sympy" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to sympy+un...@**googlegroups.com. >>>>> To post to this group, send email to sy...@googlegroups.com. >>>>> >>>>> Visit this group at >>>>> http://groups.google.com/**group/sympy<http://groups.google.com/group/sympy> >>>>> . >>>>> For more options, visit >>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>>> . >>>>> >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "sympy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to sympy+un...@**googlegroups.com. >>>> To post to this group, send email to sy...@googlegroups.com. >>>> >>>> Visit this group at >>>> http://groups.google.com/**group/sympy<http://groups.google.com/group/sympy> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "sympy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sympy+un...@googlegroups.com <javascript:>. >> To post to this group, send email to sy...@googlegroups.com <javascript:> >> . >> Visit this group at http://groups.google.com/group/sympy. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. For more options, visit https://groups.google.com/groups/opt_out.
