FYI, (Posted at https://groups.google.com/g/fricas-devel and https://groups.google.com/g/sci.math.symbolic)
Independent CAS integration tests summer 2022 edition is now complete. (It took about 5 months since I initially started working on it!) <https://www.12000.org/my_notes/CAS_integration_tests/reports/summer_2022/index.htm> The following is summary of changes in this version compared to last year's edition ======================== 1. Number of integrals increased to 85,479 from 71,994. The integrals are made of the following 210 files: a. Rubi test suite. Files 1 to 208. Thanks to Alert Rich. Downloaded from <https://github.com/RuleBasedIntegration> b. IntegrateAlgebraic test file. File 209. Thanks to Sam Blake. c. Fricas test file. File 210. Thanks to Waldek Hebisch. 2. CAS version changes: Maple 2021.1 -> 2022.1 Mathematica 12.3 -> 13.01 Giac 1.7.0 -> 1.9.0-13 Sympy 1.8/python 3.8.8 -> 1.10.1/Python 3.10.4 Maxima 5.44 -> 5.46 FriCAS 1.3.7 -> 1.3.8 Sagemath 9.3 -> 9.6 No version change for Mupad at Matlab 2021a and Rubi at 4.16.1 but Rubi was run again on the current Mathematica 13.1 version. 3. General improvements to Latex and other formatting. 4. Flow chart of the test program at <https://www.12000.org/my_notes/CAS_integration_tests/images/design.svg> 5. Links to download all the 85,483 integrals in plain text in Mathematica, Rubi, Maple, Mupad, Sympy, and Sagemath (Fricas, Maxima, Giac) formats are at introduction page of the main page. 6. Regression reports comparing results in 2022 edition to 2021 edition are also now available. These reports show integrals that failed in current versions compared to last versions per CAS. <https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REGRESSION_REPORTS/index.htm> Regression per CAS ------------------ This table below gives the number of integrals that failed in this test (could be due to timeout, exception or not solved) but are solved in summer 2021 for each CAS. Number of regression per CAS is (less is better) Fricas: 16 Maxima: 17 Mathematica: 98 Maple: 114 Giac: 363 Sympy: 379 Tables give details about each integral that failed in regression. 7. Sympy, Sagemath (Fricas, Maxima, Giac) were all run on Linux inside Oracle Virtual Box. Maple, Rubi, Mupad and Mathematica were run on windows 10. All on same PC with 128 GB RAM with Intel Core i9-12900K 3.20 GHz. RESULTS ======= Overall solved percentage in summer 2022 edition ========================================= 1. Mathematica 13.1 97.99 % 2. Rubi 4.16.1 94.21 % 3. Maple 2022.1 84.53 % 4. Fricas 1.3.8 79.33 % 5. Giac 1.9.0-13 58.37 % 6. Maxima 5.46 56.86 % 7. Mupad 2021a 56.1 % 8. Sympy 1.10.1 41.89 % Grading results ================= Note that mupad is not graded. Default grade of B is given for any passed integral. 1. Rubi 91.50 % 2. Mathematica 78.58 % 3. Maple 56.34 % 4. Fricas 53.78 % 5. Maxima 43.57 % 6. Giac 42.36 % 7. Sympy 28.49 % 8. Mupad* N/A Time and leaf size Performance ============================== Sorted by time performance in average time used per integrals (seconds). Note for Maple(*) timing: the option method=_RETURNVERBOSE was used which made Maple run through all integration methods increasing average time used. For example in summer 2021 edition which did not use this option, Maple had 0.79 seconds average time on a slower PC also than the one used for the current tests. 1. Rubi 0.3 (sec) 2. Maxima 0.49 (sec) 3. Giac 0.75 (sec) 4. Fricas 1.17 (sec) 5. Mathematica 2.38 (sec) 6. Mupad 2.75 (sec) 7. Maple* 3.33 (sec) 8. Sympy 5.24 (sec) General notes ============== 1) Maxima, Fricas and Giac were called via Sagemath. Some integrals could have failed due to interface issues. These will have F(-2) error code in the section "Detailed conclusion table per each integral for all CAS systems" per each report and also in the actual integral page with the actual exception name shown. This allows one to see if the cause of the failure is due to interface issue or internal error from the CAS itself. For an example, the integral on <https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/test_cases/4_Trig_functions/4.7_Miscellaneous/139_4.7.5_x%5Em_trig-a+b_log-c_x%5En-%5Ep/reportsubsection69.htm#53> Has F(-2) on Fricas output, but looking at the exception integrate(x*sin(a+b*log(c*x^n))^(1/2),x, algorithm="fricas") "Exception raised: TypeError >> Error detected within library code: integrate: implementation incomplete (has polynomial part)" So this is not interface issue with Sagemath. The exception comes from Fricas itself. But this integral <https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/test_cases/4_Trig_functions/4.7_Miscellaneous/139_4.7.5_x%5Em_trig-a+b_log-c_x%5En-%5Ep/reportsubsection57.htm#41> Has F(-2) on Giac result, and looking at the exception it says integrate(x^2*sin(a+log(c*x^n)*(-1/n^2)^(1/2))^3,x, algorithm="giac") "Exception raised: NotImplementedError >> Unable to parse Giac output: ((-9*i)*sageVARn^4*sageVARx^3*exp((-3*i) *sageVARa)* exp((3*sageVARn*abs(sageVARn)*ln(sageVARx)+3*abs(sageVARn)* ln(sageVARc))/sageVARn^2)+27*i*sageVARn^4*sageVARx^3*exp((-i)" Which indicates it is an interface issue this time. This is verified by running this integral directly in giac and it is solved with no error. CAS integration test program currently does not distinguish between an exception generated due to a sagemath interface problem or an exception generated from the CAS itself. I am not sure if there a robust way to do this for Maxima, Fricas and Giac. If a way can be found, will add it. But there seems to be less of these interface problems in the current Sagemath version than the last one used. 2) The section "List of integrals sorted by grade for each CAS" in each report shows list of integrals for each grade per CAS. This allows one to quickly find integrals with specific grade. 3) Also as mentioned before there are a number of integrals for Maxima which fail when it asks a question in order to complete the integration as this is not supported by Sagemath or by the CAS integration test program. If an option can be added to turn this feature off that will be a good thing and will improve Maxima score. 4) Any bugs found or problems please let me know and will try my best to fix them. --Nasser -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/87fa4453-c493-4c10-908f-b43416c3180fn%40googlegroups.com.
