Re: [ANN] orgtbl-fit

[tbanelwebmin]

On 3/3/23 16:13, Ihor Radchenko wrote:



 BTW, the dollar replacement is something org-table can benefit from---a
 number of people are confused because "$" is treated specially by Calc.

I'm not sure what you mean. The dollar in spreadsheet formulas? Like:
#+TBLFM: $6=$5+1

Which means that I misread the sources. I was referring to
  | 2$ | 3$| #ERROR |
  #+tblfm: $3=$2+$1

Error in the above is because Calc handles "$" specially.


Org or Calc was waiting for something after the $, as in the last row of 
this table:


| |    | sum    | error message   |
|-+++-|
| 2$  | 3$ | #ERROR | $'s not allowed in this context |
| 2_  | 3_ | #ERROR | Expected a number   |
| 2€  | 3€ | #ERROR | Expected `)'    |
| 2㍐ | 3¥ | #ERROR | Expected `)'    |
| 2£  | 3£ | #ERROR | Expected `)'    |
| 2*  | 3+ | #ERROR | Expected a number   |
| 2   | 3  | 5  | |
| $2  | 3  | 6  | |
| 2$2 | 3  | 9  | |
 #+tblfm: $3=$2+$1




 We can, but it should be first and foremost added to GNU Calc. On Org
 side, we just need appropriate integration. Maintaining generic data
 analysis code in Org is out of Org's scope.

Absolutely

Although the latest Calc release seams to be 2.02f, dating back in January 
1992. Has it reached perfection 31
years ago?

No. It became a part of Emacs, AFAIU.
New (small) things are being added to Calc as a part of Emacs development.



Good!
So, what you said makes sense: adding features to Calc, and then giving 
access to them from Org.

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin  writes:

> BTW, the dollar replacement is something org-table can benefit from---a
> number of people are confused because "$" is treated specially by Calc.
>
> I'm not sure what you mean. The dollar in spreadsheet formulas? Like:
> #+TBLFM: $6=$5+1

Which means that I misread the sources. I was referring to
 | 2$ | 3$| #ERROR |
 #+tblfm: $3=$2+$1

Error in the above is because Calc handles "$" specially.

> We can, but it should be first and foremost added to GNU Calc. On Org
> side, we just need appropriate integration. Maintaining generic data
> analysis code in Org is out of Org's scope.
>
> Absolutely
>
> Although the latest Calc release seams to be 2.02f, dating back in January 
> 1992. Has it reached perfection 31
> years ago?

No. It became a part of Emacs, AFAIU.
New (small) things are being added to Calc as a part of Emacs development.

>From NEWS.29:

** Calc

+++
*** New user option 'calc-kill-line-numbering'.
Set it to nil to exclude line numbering from kills and copies.

>From NEWS.28:

** Calc

*** The behavior when doing forward-delete has been changed.
Previously, using the 'C-d' command would delete the final number in
the input field, no matter where point was.  This has been changed to
work more traditionally, with 'C-d' deleting the next character.
Likewise, point isn't moved to the end of the string before inserting
digits.

*** Setting the word size to zero disables word clipping.
The word size normally clips the results of certain bit-oriented
operations such as shifts and bitwise XOR.  A word size of zero, set
by 'b w', makes the operation have effect on the whole argument values
and the result is not truncated in any way.

*** The '/' operator now has higher precedence in (La)TeX input mode.
It no longer has lower precedence than '+' and '-'.

*** New user option 'calc-make-windows-dedicated'.
When this user option is non-nil, Calc will mark its windows as
dedicated.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [ANN] orgtbl-fit

[tbanelwebmin]

  
  


On 2/20/23 11:50, Ihor Radchenko wrote:


  tbanelwebmin  writes:


  

  
Examples & documentation can be read here:
https://github.com/tbanel/orgtblfit/blob/main/README.org

  
  Interesting.
Could it be somehow integrated with TBLFM formulas?
I imagine something like

? +?*year +?*passengers +?*(year-2016)*passengers

, when set as a column value in table formula, to be auto-updated with
actual coefficients upon re-calculating the table.
...


We need to specify the target column ("consumption" in this example). 
Therefore, the formula could be something like that:

$4 = fit (consumption = ? +?*year +?*passengers +?*(year-2016)*passengers)
It would benefit from other spreadsheet features, like constants and 
remote references.

  
  
Makes sense.


  
On the development side, the TBLFM handling is already quite a big chunk 
of code. We must take care that such an additional feature do not add 
complexity and maintenance burden.

  
  
>From point of view of org-table.el, adding fitting functionality is
mostly delegating the work to GNU Calc. Org side is just parsing the
formula and transforming it to appropriate Calc function call.


Absolutely.

From Org table to Calc and back to Org table is what the orgtbl-fit
package does. Currently it is around 400 lines of Elisp and 700
lines of unit tests.


  

So, given that the parsing is extended cleanly, I do not think that
maintenance burden will increase all that much. It may even benefit from
someone taking a fresh look and possibly refactoring org-table TBLFM
parser.


Most likely.


  


  
Orgtbl-fit as-is


It is also possible to include orgtbl-fit as-is into Org Mode core. It 
would sit side-by-side with the core without changing anything in its 
code and its unit-tests.

  
  
IMHO, it is not sufficiently integrated with org-table.el facilities in
its current state. I'm afraid that we will have code duplication if
include orgtbl-fit as is.


Yes. One of the benefits from a fresh look you were talking about,
would be avoiding code duplication.


  

BTW, the dollar replacement is something org-table can benefit from---a
number of people are confused because "$" is treated specially by Calc.


I'm not sure what you mean. The dollar in spreadsheet formulas?
Like:
#+TBLFM: $6=$5+1


  


  
Data-analysis toolkit
-

 From a higher perspective, we could give a consistent data-analysis 
toolkit to Org Mode (and call it org-data-analysis.el).

It would start with fitting, clustering & aggregation. Then, new 
algorithms would be added upon user requests.

Of course, there should be an interest among Org Mode users for such a 
toolkit.

  
  
We can, but it should be first and foremost added to GNU Calc. On Org
side, we just need appropriate integration. Maintaining generic data
analysis code in Org is out of Org's scope.


Absolutely

Although the latest Calc release seams to be 2.02f, dating back in
January 1992. Has it reached perfection 31 years ago?


  

Contributing to GNU Calc will also benefit GNU Calc users who don't use
Org mode.




  

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin  writes:

>>> Examples & documentation can be read here:
>>> https://github.com/tbanel/orgtblfit/blob/main/README.org
>> Interesting.
>> Could it be somehow integrated with TBLFM formulas?
>> I imagine something like
>>
>> ? +?*year +?*passengers +?*(year-2016)*passengers
>>
>> , when set as a column value in table formula, to be auto-updated with
>> actual coefficients upon re-calculating the table.
>> ...
> We need to specify the target column ("consumption" in this example). 
> Therefore, the formula could be something like that:
>
> $4 = fit (consumption = ? +?*year +?*passengers +?*(year-2016)*passengers)
> It would benefit from other spreadsheet features, like constants and 
> remote references.

Makes sense.

> On the development side, the TBLFM handling is already quite a big chunk 
> of code. We must take care that such an additional feature do not add 
> complexity and maintenance burden.

>From point of view of org-table.el, adding fitting functionality is
mostly delegating the work to GNU Calc. Org side is just parsing the
formula and transforming it to appropriate Calc function call.

So, given that the parsing is extended cleanly, I do not think that
maintenance burden will increase all that much. It may even benefit from
someone taking a fresh look and possibly refactoring org-table TBLFM
parser.

> Orgtbl-fit as-is
> 
>
> It is also possible to include orgtbl-fit as-is into Org Mode core. It 
> would sit side-by-side with the core without changing anything in its 
> code and its unit-tests.

IMHO, it is not sufficiently integrated with org-table.el facilities in
its current state. I'm afraid that we will have code duplication if
include orgtbl-fit as is.

BTW, the dollar replacement is something org-table can benefit from---a
number of people are confused because "$" is treated specially by Calc.

> Data-analysis toolkit
> -
>
>  From a higher perspective, we could give a consistent data-analysis 
> toolkit to Org Mode (and call it org-data-analysis.el).
>
> It would start with fitting, clustering & aggregation. Then, new 
> algorithms would be added upon user requests.
>
> Of course, there should be an interest among Org Mode users for such a 
> toolkit.

We can, but it should be first and foremost added to GNU Calc. On Org
side, we just need appropriate integration. Maintaining generic data
analysis code in Org is out of Org's scope.

Contributing to GNU Calc will also benefit GNU Calc users who don't use
Org mode.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [ANN] orgtbl-fit

[tbanelwebmin]

On 1/26/23 11:35, Ihor Radchenko wrote:

tbanelwebmin  writes:


Actually, orgtbl-fit is a bridge between Org Mode tables and Calc.

By the way, Org Mode table spreadsheet capabilities are also a bridge
with Calc.

Examples & documentation can be read here:
https://github.com/tbanel/orgtblfit/blob/main/README.org

Interesting.
Could it be somehow integrated with TBLFM formulas?
I imagine something like

? +?*year +?*passengers +?*(year-2016)*passengers

, when set as a column value in table formula, to be auto-updated with
actual coefficients upon re-calculating the table.



Hey! That's an awesome idea.


Expanding on the idea
-

We need to specify the target column ("consumption" in this example). 
Therefore, the formula could be something like that:


$4 = fit (consumption = ? +?*year +?*passengers +?*(year-2016)*passengers)

It would benefit from other spreadsheet features, like constants and 
remote references.


On the development side, the TBLFM handling is already quite a big chunk 
of code. We must take care that such an additional feature do not add 
complexity and maintenance burden.



Orgtbl-fit as-is


It is also possible to include orgtbl-fit as-is into Org Mode core. It 
would sit side-by-side with the core without changing anything in its 
code and its unit-tests.



Data-analysis toolkit
-

From a higher perspective, we could give a consistent data-analysis 
toolkit to Org Mode (and call it org-data-analysis.el).


It would start with fitting, clustering & aggregation. Then, new 
algorithms would be added upon user requests.


Of course, there should be an interest among Org Mode users for such a 
toolkit.

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin  writes:

> Actually, orgtbl-fit is a bridge between Org Mode tables and Calc.
>
> By the way, Org Mode table spreadsheet capabilities are also a bridge 
> with Calc.
>
> Examples & documentation can be read here:
> https://github.com/tbanel/orgtblfit/blob/main/README.org

Interesting.
Could it be somehow integrated with TBLFM formulas?
I imagine something like

? +?*year +?*passengers +?*(year-2016)*passengers

, when set as a column value in table formula, to be auto-updated with
actual coefficients upon re-calculating the table.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [ANN] orgtbl-fit

[tbanelwebmin]

Hi Ihor & the List

GnuPlot is a great software. If you feel confortable with it, continue 
using it! If others are used to R or Python, that is fine too.


Orgtbl-fit may be useful if:

 * - You want a pure Emacs process, without external dependencies.
 * - You know that Emacs-Calc can fit your data, but you are not
   familiar with it.
 * - Your data is already available as an Org Mode table.
 * - You don't want to write a script (just point at the target column
   and type M-x orgtbl-fit, that's all).

Actually, orgtbl-fit is a bridge between Org Mode tables and Calc.

By the way, Org Mode table spreadsheet capabilities are also a bridge 
with Calc.


Examples & documentation can be read here:
https://github.com/tbanel/orgtblfit/blob/main/README.org

Have fun!


On 1/24/23 20:55, Ihor Radchenko wrote:

tbanelwebmin  writes:


The new orgtbl-fit package has just been released on Melpa. It
does regression fitting on Org Mode tables.

Example. We suspect that `obs' depends on `x' and `y'.
...

Let us put the cursor on the `obs' column, and type
M-x orgtbl-fit

Two columns are added
- predicted obs column
- difference between obs and predicted

|  x | y |  obs | Best Fit | Fit Diff |
|+---+--+--+--|
| 32 | 7 | 38.3 | 38.2 | -0.1 |
| 18 | 3 | 11.4 | 11.6 |  0.2 |
| 43 | 9 | 47.3 | 47.2 | -0.1 |
| 11 | 2 |  8.9 |  8.7 | -0.2 |
| 35 | 8 | 45.1 | 45.3 |  0.2 |
#+TBLFM: $4=-0.289267886829 - 1.06613976706*$1 + 10.3668885192*$2;
%.1f::$5=$4-$3; %.1f

Are there situations when this package is actually useful for data
analysis? (I am usually using gnuplot for fitting)

Re: [ANN] orgtbl-fit

[Ihor Radchenko]

tbanelwebmin  writes:

> The new orgtbl-fit package has just been released on Melpa. It
> does regression fitting on Org Mode tables.
>
> Example. We suspect that `obs' depends on `x' and `y'.
> ...
>
> Let us put the cursor on the `obs' column, and type
> M-x orgtbl-fit
>
> Two columns are added
> - predicted obs column
> - difference between obs and predicted
>
> |  x | y |  obs | Best Fit | Fit Diff |
> |+---+--+--+--|
> | 32 | 7 | 38.3 | 38.2 | -0.1 |
> | 18 | 3 | 11.4 | 11.6 |  0.2 |
> | 43 | 9 | 47.3 | 47.2 | -0.1 |
> | 11 | 2 |  8.9 |  8.7 | -0.2 |
> | 35 | 8 | 45.1 | 45.3 |  0.2 |
> #+TBLFM: $4=-0.289267886829 - 1.06613976706*$1 + 10.3668885192*$2; 
> %.1f::$5=$4-$3; %.1f

Are there situations when this package is actually useful for data
analysis? (I am usually using gnuplot for fitting)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[ANN] orgtbl-fit

[tbanelwebmin]

Hi the List!

The new orgtbl-fit package has just been released on Melpa. It
does regression fitting on Org Mode tables.

Example. We suspect that `obs' depends on `x' and `y'.

|  x | y |  obs |
|+---+--|
| 32 | 7 | 38.3 |
| 18 | 3 | 11.4 |
| 43 | 9 | 47.3 |
| 11 | 2 |  8.9 |
| 35 | 8 | 45.1 |

Let us put the cursor on the `obs' column, and type
M-x orgtbl-fit

Two columns are added
- predicted obs column
- difference between obs and predicted

|  x | y |  obs | Best Fit | Fit Diff |
|+---+--+--+--|
| 32 | 7 | 38.3 | 38.2 | -0.1 |
| 18 | 3 | 11.4 | 11.6 |  0.2 |
| 43 | 9 | 47.3 | 47.2 | -0.1 |
| 11 | 2 |  8.9 |  8.7 | -0.2 |
| 35 | 8 | 45.1 | 45.3 |  0.2 |
#+TBLFM: $4=-0.289267886829 - 1.06613976706*$1 + 10.3668885192*$2; 
%.1f::$5=$4-$3; %.1f


So we discovered that
obs = -0.29 -1.07*x +10.37*y

Behind the scene, the calcFunc-fit function from Emacs-Calc is called.

Install through Melpa:
M-x package-install orgtbl-fit

Source and documentation here:
https://github.com/tbanel/orgtblfit/blob/master/README.org

Have fun
Thierry