The trend with LLMs is much less structured. It doesn't use any formalism. It just guesses the next character of the input based on training on billions of examples.
That's why I think that tools like SymPy that are more structured can be useful. GPT can already write SymPy code pretty well, much better than it can do the actual mathematics. It may be as simple as automatically appending "and write SymPy code to verify this" to the end of a prompt whenever it involves mathematics. This sort of approach has already been proven to be able to solve university math problems (see https://www.pnas.org/doi/pdf/10.1073/pnas.2123433119, where they literally just take the input problem and prepend "use sympy" and the neural network model does the rest). Aaron Meurer On Thu, Dec 15, 2022 at 2:21 AM S.Y. Lee <sylee...@gmail.com> wrote: > > My hope is that tools like SymPy can be used as oracles for tools like > GPT to help them verify their mathematics. > > In the most general context, "correct mathematics" can also be considered > some "grammar". > So there should be some grammar between Type-0 grammar to Type-1 grammar > in Chomsky hierarchy <https://en.wikipedia.org/wiki/Chomsky_hierarchy>. > In this context, a parser, or a parser with sympy oracle is the solution > for such problem, > such that any other ideas to solve such problem can be isomorphic to. > > However, building up such parser is off-direction for the researches of > deep learning itself, > because it would need a lot of efforts by experts, to interpret the > sentence generated by GPT, > and design a phrase structure grammar for it. > > I also thought about an idea that they can just tag arithmetics using SKI > combinator calculus > <https://en.wikipedia.org/wiki/SKI_combinator_calculus>. > In this way, there is no wrong arithmetics for every random sequence of > alphabets. > > However, I'm not sure that if this idea is already refuted by such > contemporary researchers > because it should be pretty much well-known. > > On Thursday, December 15, 2022 at 12:45:53 AM UTC+2 asme...@gmail.com > wrote: > >> I've seen many examples on Twitter and elsewhere of chatgpt doing >> terribly on actual mathematics. I instead asked it to "Find the extreme >> values of the function x^3 + 5x^2 + 3x - 9". It got the first part with the >> derivative and the quadratic equation correct, but then did faulty >> arithmetic when plugging the values back into the expression (the correct >> values are 0 and -256/27). It makes a sign error and also gets the addition >> wrong even with the wrong sign. >> >> My hope is that tools like SymPy can be used as oracles for tools like >> GPT to help them verify their mathematics. Obviously in this case you don't >> need SymPy to just verify arithmetic, but I've also seen it do just as >> poorly on symbolic algebra problems. >> >> I've also pasted the full log below: >> >> [image: Screen Shot 2022-12-14 at 3.41.32 PM.png] >> [image: Screen Shot 2022-12-14 at 3.41.42 PM.png] >> >> To find the extreme values of a function, we need to find the critical >> points of the function, which are the points at which the derivative of the >> function is equal to 0 or does not exist. >> >> The derivative of the function x^3 + 5x^2 + 3x - 9 is 3x^2 + 10x + 3. To >> find the critical points, we need to solve the equation 3x^2 + 10x + 3 = 0. >> >> To solve this equation, we can use the quadratic formula: >> >> x = (-b ± √(b^2 - 4ac)) / 2a >> >> where a, b, and c are the coefficients of the quadratic equation ax^2 + >> bx + c = 0. In our case, a = 3, b = 10, and c = 3, so the formula becomes: >> >> x = (-10 ± √(10^2 - 4 * 3 * 3)) / 2 * 3 >> >> = (-10 ± √(100 - 36)) / 6 >> >> = (-10 ± √(64)) / 6 >> >> = (-10 ± 8) / 6 >> >> = (-18, -2) / 6 >> >> = (-3, -1/3) >> >> These are the critical points of the function x^3 + 5x^2 + 3x - 9. >> >> To find the extreme values of the function, we need to evaluate the >> function at these critical points and at any points where the derivative >> does not exist. In this case, the derivative exists for all values of x, so >> we only need to evaluate the function at the critical points. >> >> If we plug in the critical points into the function, we get: >> >> x^3 + 5x^2 + 3x - 9 = (-3)^3 + 5(-3)^2 + 3(-3) - 9 = -27 + 45 + 9 - 9 = 8 >> >> and >> >> x^3 + 5x^2 + 3x - 9 = (-1/3)^3 + 5(-1/3)^2 + 3(-1/3) - 9 = -1/27 + 5/9 + >> 1 - 9 = 5/9 - 1/27 - 8/9 = -64/27 >> >> The extreme values of the function x^3 + 5x^2 + 3x - 9 are 8 and -64/27. >> >> On Wed, Dec 14, 2022 at 3:35 PM gu...@uwosh.edu <gu...@uwosh.edu> wrote: >> >>> Just for some additional perspective. I have also tried this on some >>> general chemistry word problems. In general, I see it getting basic one >>> logical step processes correct (e.g. a single step dilution or grams -> >>> moles). Things with multiple steps or requiring understanding the physical >>> situation it does poorly on. That said, I think it does better than some of >>> my weakest students. It does not seem to be able to use significant figures >>> in computations (also a problem for my weaker students). >>> >>> It seems to be improving rapidly. If it can get to reliably >>> differentiating between correct (workable) solutions and erroneous ones, it >>> will be more useful to most people (including my students) than searches of >>> the internet or a cheating sight such as Chegg. >>> >>> My two cents worth of opinion. >>> >>> Jonathan >>> >>> On Wednesday, December 14, 2022 at 4:28:05 PM UTC-6 Francesco Bonazzi >>> wrote: >>> >>>> [image: chatgpt.sympy.matrix_diag.png] >>>> >>>> On Wednesday, December 14, 2022 at 11:26:37 p.m. UTC+1 Francesco >>>> Bonazzi wrote: >>>> >>>>> Not everything is perfect... ChatGPT misses the *convert_to( ... ) >>>>> *function >>>>> in *sympy.physics.units*, furthermore, the given code does not work: >>>>> >>>>> [image: chatgpt.sympy.unit_conv.png] >>>>> >>>>> On Wednesday, December 14, 2022 at 11:24:29 p.m. UTC+1 Francesco >>>>> Bonazzi wrote: >>>>> >>>>>> [image: chatgpt.sympy.logical_inference.png] >>>>>> >>>>>> On Wednesday, December 14, 2022 at 11:23:43 p.m. UTC+1 Francesco >>>>>> Bonazzi wrote: >>>>>> >>>>>>> https://en.wikipedia.org/wiki/ChatGPT >>>>>>> >>>>>>> Some tested examples attached as pictures to this post. Quite >>>>>>> impressive... >>>>>>> >>>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "sympy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sympy+un...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/sympy/6af62b19-1fb0-4681-9fd2-5e5fccfcb46fn%40googlegroups.com >>> <https://groups.google.com/d/msgid/sympy/6af62b19-1fb0-4681-9fd2-5e5fccfcb46fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/74847ca3-124b-414d-aa36-01eb91096310n%40googlegroups.com > <https://groups.google.com/d/msgid/sympy/74847ca3-124b-414d-aa36-01eb91096310n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6L5F_vVWYbOz2p3N%3DwVddiQQ7YNXOK%2BT2ht3uGb7PPNXg%40mail.gmail.com.
