On Thu, Aug 8, 2013 at 11:19 AM, Matthew Rocklin <mrock...@gmail.com> wrote: > Ok, Just went through this thread again. > > It should be simple to translate SymPy.Piecewise to a recursive > Theano.switch (after translating SymPy.LT to theano.lt, etc.) I'll get on > this soon. Does this sound reasonable to you Fred? > > Jason's PR allowing pass through of keyword arguments looks close to me. I > suspect that the `on_unused_input=ignore` issue will soon be possible. > Question, should this be default? Are there other defaults that should be > used? I'm curious why you both are making functions with unused inputs. > > Jason, was Fred's note on directly passing constants useful? If not why > not? It sounds like you should subs your constant symbols for numeric > values prior to creating the theano graph. > >> With regards to using an externally defined theano graph or op (theano >> wrapped sympy implementation), how would one pass them to theano_function? > You wouldn't. You would use the other common interface function, > theano_code to create theano variables instead. You would then use raw > theano operations (like theano.function) to compile your graph from theano > variables. Theano_function is a convenient all-in-one interface function > for the common case. If you want to do anything custom I recommend using > SymPy and Theano natively, translating from one to the other using > theano_code instead. It does nothing except translate. > >> Does sympy.printing.theanocode.theano_function automatically optimize the >> compiled graph? > Yes. theano_function translates sympy expressions into theano expressions > using theano_code. It then calls theano.function on these expressions. > theano.function optimizes by default. > >> SymPy C Codegen and Theano > > @Fred, how hard would it be to leverage SymPy's C codegen in Theano? This > might be a lot cleaner than wrapping raw SymPy operations and might > substantially extend Theano's support of scalar expressions. Do you have a > performant Bessel function op? I'll bet SymPy could be made to do this > quite well. > > @Aaron / @Ondrej, if you're reading this thread could you point us to the > best place to start looking at C codegen in SymPy? Alternatively can you > point to an active community member who would be able to do so?
Maybe Ondrej knows, but I don't. I would just look at the code. I believe the meat of it is in the printers, while the rest is just nice wrappers. Aaron Meurer > > >> ODE Integration > > What are people's thoughts on scipy.integrate.odeint? Earlier this year I > was looking at nodepy, a purely mathematical treatment/catalog of time > stepping schemes. It should be easy to generate such schemes in Theano > using simple abstractions like Butcher matrices. The code I was playing > around with was https://github.com/mrocklin/tsgen. The goal was that we > could easily experiment with and rapidly benchmark different time stepping > schemes for specific problems. > > > > > On Wed, Aug 7, 2013 at 10:37 AM, Guy Parsey <guy.par...@gmail.com> wrote: >> >> Hey Fred, >> I am starting to play with making a theano.Op, and in principle, linking a >> custom theano.Op to an implemented sympy function will solve all of my >> problems (at least in the sense of getting my system running, not >> necessarily speed). My initial post spoke of both mapping Piecewise and >> interpolated functions into theano; after speaking with my advisor, due to >> wanting a smoothing criterion over our piecewise datasets, I think we will >> end up treating them as interpolated functions over smoothed data sets. My >> idea for how to use an interpolation routine will be a wrapped theano.Op, so >> I am still rather curious as to how implementing Piecewise would work in >> such way that theano can manipulate it in graph form (built out of native >> theano operations). >> >> Theano will merge duplicated subexpression automatically by default. What >> is your function? If it is too "simple", for example just working on scalar, >> Theano have by default much overhead. As said for the other benchmark, there >> is tricks to remove a big part or even all of it, but it ask for some work >> on the developer. >> >> >> The basic system/functions can be described as follows; we have some >> number of species (n_0 ...n_k) and we are creating a function for the vector >> of time derivatives. The different types of expressions come about from the >> reaction rates (K_0 ..... K_l) The whole system is described symbolically, >> for symbolic determination of the jacobian (spline interpolated functions >> are wrapped with a sympy implemented function so that taking a symbolic >> derivative also yields the derivative of the numerical implementation). >> Dummy example below with three types of base expressions >> d(n_i)/dt = K_0*n_2*n_0 + K_1*n_3 + K_2*n_4*n_2*n_4/(sum(n_0....n_k)) >> Each of the K terms can be described differently: >> --constant >> --analytical parameterized of some n_j (Kf ~ a0*(n_j**a1)*exp(a2/n_j) >> , a0..a2 are constants) >> --spline interpolation function of some n_j (K_interp(n_j)) >> --analytical weighted with piecewise/interpolated (new_Kf ~ >> Kf*exp(-K_interp(n_j)) >> I also have two energy equations, but these incorporate all of the same >> types of terms, just not in a nice summation as in the species equations. >> Any given base expression (eg. K_0*n_2*n_0) will appear in multiple time >> derivative equations, so if the interpolation is slow, it would be nice to >> get theano to recognize that it is a 'common expression.' I can make test >> cases with a few species and a few reactions (Ks), but the true systems I >> would like to be simulating will have 40-100 species and 300-2000 reaction >> rates. Are these functions too 'simple' to be sped by using theano? There is >> a lot of optimization that can be done due to the high number of shared >> expressions, so I was hoping to take advantage of the theano graph >> optimizations. >> >> My initial test cases were using the numpy.float64 dtype, so I will switch >> it over to ndarrays if possible. >> I hope to have questions or a working spline interpolation theano.Op in >> the next few hours. >> Cheers, >> Guy >> >> Quick Codegen question: In principle, can one create a python callback >> within a compiled c code to a python function that is not compiled >> (interpolation function)? >> >> >> On Tuesday, August 6, 2013 1:36:29 PM UTC-4, Frédéric Bastien wrote: >>> >>> >>> >>> >>> On Tue, Aug 6, 2013 at 11:04 AM, Guy Parsey <guy.p...@gmail.com> wrote: >>>> >>>> Hello, >>>> >>>> Theano/Sympy questions >>>> Fred, in terms of replacing sympy.Pieceiwse with a Theano equivalent, >>>> since sympy.Piecewise attempts each condition (from ExprCondPair) until one >>>> is valid, I would think that the closest equivalent would be a recursive >>>> theano ifelse (ie. sympy.Piecewise((expr1,cond1),(expr2,cond2)) ~ >>>> theano.ifelse(cond1,expr1,theano.ifelse(cond2,expr2,None))) statement so >>>> that the expressions are not evaluated until the condition is achieved. >>>> This >>>> might be too much of a patch though and I am not sure how to implement it >>>> with the same argument structure as sympy.Pieceiwse. >>> >>> >>> It would work. But for the timming, I would also try with the switch op >>> instead of ifelse. The ifelse currently have much bigger oveahead then the >>> switch op, and even if the switch op evaluate both branch, it happen >>> relatively frequently that it is faster then the ifelse. Especially if each >>> expression have small size, like a scalar or a small vector. >>> >>>> >>>> With regards to using an externally defined theano graph or op (theano >>>> wrapped sympy implementation), how would one pass them to theano_function? >>>> It feels as though it would be analogous to the autowrap helpers argument, >>>> or would this be an issue of merging graphs before calling theano.function? >>>> If I can understand this step, I feel as though both of my problems would >>>> be >>>> solvable (a wrapping of a piecewise along with a wrapping of an >>>> interpolated >>>> function) >>> >>> >>> There is 2 steps. 1) make a theano op like descrided here: >>> http://deeplearning.net/software/theano/tutorial/extending_theano.html >>> >>> 2) Modify theano_function to use the new Theano op for the equivalent >>> SymPy op. It could be possible to make an autowrap for this, but I don't >>> know enough of the detail to be sure. >>> >>>> >>>> In all of my simplified test cases (no pieceiwse or interpolations, 9 >>>> heavily-linked non-linear ODEs), I also show that the lambdify function is >>>> faster to evaluate than my created theano function. When passing a list of >>>> expressions, if there is a commons subexpression that exists in two >>>> expressions (not twice in the same expression), is it treated as such >>>> during >>>> theano compilation or is each expression handled separately? >>> >>> >>> Theano will merge duplicated subexpression automatically by default. What >>> is your function? If it is too "simple", for example just working on scalar, >>> Theano have by default much overhead. As said for the other benchmark, there >>> is tricks to remove a big part or even all of it, but it ask for some work >>> on the developer. >>> >>>> >>>> >>>> Codegen questions (maybe a different topic): >>>> Jason, I have started playing with the sympy.printing.codegen (C for >>>> now, Fortran later) as an alternative to theano (I would very much like to >>>> get multiple methods working, and GPU acceleration makes me want to keep >>>> theano implementation). There is a size-able speedup relative to lambdify >>>> or >>>> theano_function, which is to be expected, but I arrive at the same problem >>>> as above with regards to interpolated or piecewise functions. The initial >>>> crash from using Piecewise functions comes from Routine calling >>>> sympy.tensor.index_methods get_contraction_structure which explicitly >>>> states >>>> no support for Piecewise function types. In principle, if one made a >>>> symbolic implemented function (preferably without an analytical >>>> representation), could one link the python object to the compiled code? >>>> What >>>> attributes would said implemented_function need to have? >>>> >>>> Thank you for your responses, I am glad to hear that at least I am not >>>> completely missing something. >>>> Cheers, >>>> Guy >>>> >>>> >>>> On Tuesday, August 6, 2013 9:37:41 AM UTC-4, Jason Moore wrote: >>>>> >>>>> Fred, >>>>> >>>>> I think on_used_input=ignore should be a default in Matthew's theano >>>>> printing code or that arg needs to be pushed up to his layer. I hit that >>>>> issue too. >>>>> >>>>> The code I have is here: >>>>> https://github.com/PythonDynamics/pydy-code-gen >>>>> >>>>> See the results.txt file for basic speed comparisons. I'm generating >>>>> the ODEs for an n-link pendulum with mechanics and then see how fast it >>>>> generates and simulates with scipy.odeint. >>>>> >>>>> The code doesn't work at the moment. I haven't touched it in a month >>>>> and looks like some things have changed in sympy and/or theano. I'll work >>>>> on >>>>> the bugs now. >>>>> >>>>> But "python benchmark.py" should run it with sympy master and ?some? >>>>> version of Theano. >>>>> >>>>> >>>>> Jason >>>>> moorepants.info >>>>> +01 530-601-9791 >>>>> >>>>> >>>>> On Tue, Aug 6, 2013 at 9:25 AM, Frédéric Bastien <no...@nouiz.org> >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I don't know what is sympy.functions.elementary.piecewise. Do Jason >>>>>> answered that part? If not, I'll look into it to know how to make Theano >>>>>> reproduce it. About converting any Sympy symbol to Theano symbol, when >>>>>> there >>>>>> isn't a one to one matching, you can create a one to a full Theano graph >>>>>> conversion. When this is possible, it is probably the best, as if you >>>>>> make a >>>>>> new Theano op, it work work on the GPU. But if you make a Theano graph, >>>>>> there is good change that the graph will already work on the GPU. >>>>>> >>>>>> If it is not possible to make a Theano graph for a sympy symbol, it is >>>>>> possible to make a new Theano op that just wrap the sympy implementation. >>>>>> Also, if this is a bottleneck, I recently added an example that show how >>>>>> to >>>>>> use numba with Theano so speed up the python code in a Theano op. >>>>>> >>>>>> >>>>>> Theano do not parallelize on the CPU, except for the call to BLAS, >>>>>> when the BLAS library is parallel. On the GPU, it is parallel. >>>>>> >>>>>> Having on_unused_input=ignore is normal for complicated generated >>>>>> code. That is why it was added. But when the code is simpler and not >>>>>> generated, but all user manually coded, most of time it mean the user >>>>>> didn't >>>>>> do what he wanted. If you know it is normal that you have unused input, >>>>>> there is no problem to use that flag. >>>>>> >>>>>> >>>>>> Jason, about the case where Theano is slower, can you send me the >>>>>> Theano code? I would like to look at it. I'm very surprised that Theano >>>>>> is >>>>>> slower then Sympy in this case and would like to know why it is like >>>>>> this. >>>>>> >>>>>> Fred >>>>>> >>>>>> >>>>>> On Mon, Aug 5, 2013 at 9:03 PM, Jason Moore <moore...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> Guy, >>>>>>> >>>>>>> We're working on the same problem for sympy.physics.mechanics. >>>>>>> Matthew Rocklin added support for matrix conversions in the theano code >>>>>>> that >>>>>>> is in SymPy and I used that, but found that theano was slower that >>>>>>> lambdify >>>>>>> for most of my cases (I only have two cores, so I'm not taking >>>>>>> advantage of >>>>>>> the Theano parallel stuff). I think writing specific code gen for ode >>>>>>> integration is going to be the best bet. I'm happy to collaborate on >>>>>>> this >>>>>>> with you. >>>>>>> >>>>>>> >>>>>>> Jason >>>>>>> moorepants.info >>>>>>> +01 530-601-9791 >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 5, 2013 at 3:13 PM, Guy Parsey <guy.p...@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hello Everyone, >>>>>>>> Thank you in advance for reading through my problem and for any >>>>>>>> input you may have. I must say that I still feel like a novice >>>>>>>> programmer >>>>>>>> and my problems may be easily solvable from a different mindset. My >>>>>>>> present >>>>>>>> project entails time-integration of extremely stiff and non-linear >>>>>>>> ODEs with >>>>>>>> regards to chemical kinetics (one derivative equation for each variable >>>>>>>> species) and energy equations. Initially we were planning on using the >>>>>>>> sympy.lambdify function to create callable functions for the main >>>>>>>> function >>>>>>>> along with the jacobian and passing said functions to >>>>>>>> scipy.integrate.odeint, however this method only works for easier test >>>>>>>> cases >>>>>>>> (fewer species and/or no energy equations) before being limited by >>>>>>>> either >>>>>>>> the list recursion limit or segfaulting due to the limited stack size. >>>>>>>> I >>>>>>>> know that both of these limits can be edited, but that fact that I am >>>>>>>> reaching them makes me feel as though I am doing something extremely >>>>>>>> inefficiently. Outside of the documentation of SymPy and Theano, I >>>>>>>> have also >>>>>>>> been heavily using the BlogPost by Matthew Rocklin >>>>>>>> http://matthewrocklin.com/blog/work/2013/03/19/SymPy-Theano-part-1/ . >>>>>>>> >>>>>>>> Presently I am trying to use the mapping between Theano and SymPy >>>>>>>> (sympy.printing.theanocode theano_function) to make my callable >>>>>>>> functions >>>>>>>> and take advantage of the optimization routines. I have two major >>>>>>>> problems >>>>>>>> and a few questions: >>>>>>>> >>>>>>>> 1st major problem: Though piecewise functions exist in SymPy >>>>>>>> (sympy.functions.elementary.piecewise) there is no counterpart in >>>>>>>> Theano. >>>>>>>> Looking at the source of the inspiration for theanocode >>>>>>>> (https://github.com/nouiz/theano_sympy/ graph_translation.py) I see >>>>>>>> that >>>>>>>> some of the SymPy equivalents were defined as lambda functions. Is >>>>>>>> there an >>>>>>>> equivalent way to add Theano conditional expressions wrapped into a >>>>>>>> function >>>>>>>> to add to the mapping dictionary in theanocode.py? >>>>>>>> >>>>>>>> 2nd major problem: Similar to the problem above in that I am not >>>>>>>> sure that the Theano counterpart is; some of the terms that I use are >>>>>>>> interpolated functions (with one ODE variable as input) that we have >>>>>>>> wrapped >>>>>>>> symbolically while providing a numerical implementation (so that >>>>>>>> symbolic >>>>>>>> derivatives can be made, resulting in their own interpolations). Is it >>>>>>>> possible to recreate the interpolation function as a Theano operation >>>>>>>> for >>>>>>>> use within the system of ODEs? >>>>>>>> >>>>>>>> Remain questions: >>>>>>>> I presently have to flatten my input to theano_function to a list of >>>>>>>> expressions and then wrap to return to a form (Jacobian is a matrix >>>>>>>> not a >>>>>>>> vector); is it possible to have a matrix of different expressions as an >>>>>>>> input to theano_function with a vector output? >>>>>>>> >>>>>>>> I know that a huge amount of Theano speed up is due to >>>>>>>> parallelization of matrix operations (which I do not have), should I be >>>>>>>> focusing on SymPy Autowrap/Ufuncify or my own code generation instead >>>>>>>> of >>>>>>>> trying to get Theano to play nicely? >>>>>>>> >>>>>>>> Stupid questions: >>>>>>>> Does sympy.printing.theanocode.theano_function automatically >>>>>>>> optimize the compiled graph? >>>>>>>> >>>>>>>> Minor comment: >>>>>>>> Perhaps unnecessary for most uses of the theano_function, but I >>>>>>>> needed to modify function inputs so as to be able to use the keyword >>>>>>>> argument 'on_unused_input=ignore' as opposed to 'raise' so that I did >>>>>>>> not >>>>>>>> need to have all symbols in all equations. This may be avoided by >>>>>>>> having the >>>>>>>> unused symbols somehow (I don't know how) included in each expression. >>>>>>>> >>>>>>>> Thank you again for your time in reading my problems and any >>>>>>>> potential help you may think of. I can attach code if necessary, I just >>>>>>>> didn't want to make my post more confusing. >>>>>>>> Have an excellent day. >>>>>>>> Sincerely, >>>>>>>> Guy Parsey >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "sympy" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to sympy+un...@googlegroups.com. >>>>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>>>> >>>>>>>> Visit this group at http://groups.google.com/group/sympy. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "sympy" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to sympy+un...@googlegroups.com. >>>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>>> >>>>>>> Visit this group at http://groups.google.com/group/sympy. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "sympy" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to sympy+un...@googlegroups.com. >>>>>> To post to this group, send email to sy...@googlegroups.com. >>>>>> >>>>>> Visit this group at http://groups.google.com/group/sympy. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "sympy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to sympy+un...@googlegroups.com. >>>> To post to this group, send email to sy...@googlegroups.com. >>>> Visit this group at http://groups.google.com/group/sympy. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "sympy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sympy+unsubscr...@googlegroups.com. >> To post to this group, send email to sympy@googlegroups.com. >> Visit this group at http://groups.google.com/group/sympy. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To post to this group, send email to sympy@googlegroups.com. > Visit this group at http://groups.google.com/group/sympy. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. For more options, visit https://groups.google.com/groups/opt_out.
