Conceptually, I think an "array" whose "shape's rank" is higher than 1
should represent a collection of arrays of the same rank.
--
Raul
On Thu, Mar 3, 2022 at 11:50 AM 'Pascal Jasmin' via Programming
<programm...@jsoftware.com> wrote:
>
> I'm ok with the 0s, but another way to have 2 shapes (referring to 2 arrays)
> of different sizes ie. one of 3 2 2 the other shape 4, would be that the
> "shape of the shapes" is a "by 1 table":
>
> 3
> 1
>
> so
>
> (3 ,:1) $ 3 2 2 4 would be the shape of i. (3 ,:1) $ 3 2 2 4 or ((3 ,:1) $
> 3 2 2 4) $ i.16
>
> I'm ok with this as a formal definition for arrays of arrays, but I would
> definitely create a verb named shape to ease with typing and even
> processing/reshaping such that
>
> ((3 ,:1) $ 3 2 2 4) $ i.16
>
> would be equivalent to
>
> (3 2 2 ,: 4) shape i.16
>
> ie. in table shapes, trailing 0s (fills) are equivalent to axes not being
> present. I don't think this loses very much in expressibility, though I
> understand that it creates an extra computation step to properly understand a
> "shape item" with trailing 0s.
>
>
> On Thursday, March 3, 2022, 11:01:57 a.m. EST, Henry Rich
> <henryhr...@gmail.com> wrote:
>
>
>
>
>
> I like this, except for the trailing-0 bit. I would say having data with
> trailing 0 shape is 'common' rather than 'ultra-rare'.
>
> Henry Rich
>
> On 3/3/2022 10:15 AM, 'Pascal Jasmin' via Programming wrote:
> > I don't have a good grasp of order, but a different practical
> > interpretation of non-list shapes:
> >
> > 1. You mentioned distinguishing atoms from lists of 1 element. J already
> > manages this well, and making an atomic shape result would not be of any
> > help, IMO.
> >
> > 2. A table shape could be a code for ragged arrays, or arrays of arrays.
> > The following shape:
> >
> > 3 2 2
> > 4 0 0
> >
> > Would be 2 arrays, the first is shape 3 2 2, and the 2nd is shape 4.
> > Trailing 0s would be fills. This precludes the ultra rare need of needing
> > an "embedded" shape 4 0 or shape 4 0 0 in your data.
> >
> > A list of 3 strings/lists with individual lengths of 2 3 4 would have the
> > shape:
> >
> > 2
> > 3
> > 4
> >
> > If there is an extention of the meaning of shape, I'd prefer it be in this
> > direction rather than ultra high dimensional math that hurts my head.
> >
> >
> > On Thursday, March 3, 2022, 05:13:27 a.m. EST, Elijah Stone
> > <elro...@elronnd.net> wrote:
> >
> >
> >
> >
> >
> > A little thought exercise I embarked upon.
> > Probably not very clearly expressed, and certainly of no practical use,
> > but I think there are some interesting ideas:
> >
> > Why must the result of $y always be a vector? What would happen if we let
> > it be a higher-ranked array? What would that _mean_?
> >
> > Well, say that 2 3 4-:$y. That means that y is a mapping from arrays x
> > that have the same shape as 2 3 4 (and whose elements are natural and
> > bounded by 2 3 4). In other words, if (x -:&$ 2 3 4) *. -. 0 e. x<2 3 4
> > then (<x){y is well-defined as an atom. (I am intentially ignoring the
> > possibility of sentences such as ((<1 1){y) ((<,1){y) ((<1){y) and so
> > on; consider these sentences ill-formed for the time being.)
> > Generalising: if (x -:&$ $y) *. -. 0 e. x<$y then (<x){y is well-defined
> > as an atom.
> >
> > But this sentence makes no restrictions whatsoever on the rank of x! So
> > it would seem that we've discovered a way to make a y whose shape is an
> > array of any rank. But we must be careful. Really, all we've constructed
> > is a strange entity y--call it an array-as-we-do-not-know-it--and, on
> > arrays-as-we-do-not-know-them, we have defined that: $y is well-defined,
> > and evaluates to an array-as-we-know-it. x{y is well-defined for some
> > subset of arrays-as-we-know-them x, and evaluates to an
> > atom-as-we-know-it.
> >
> > I will call this strange creature a 4th-order array. And all
> > arrays-as-we-know-them are 3rd-order (this includes scalars, vectors, and
> > higher-rank arrays). A 4th-order array has a shape which is a 3rd-order
> > array, and maps from 3rd-order arrays to atoms. This suggests the
> > existence of lower-order arrays, and higher-order ones.
> >
> > Following the same logic, then, a 3rd-order array would map 2nd-order
> > arrays to atoms, and its shape would be a 2nd-order array. But--for
> > 3rd-order y, the x in (<x){y for 3rd-order y is currently also a 3rd-order
> > array, and $y is currently also 3rd-order.
> >
> > Or is it, really? We've already established that $y must always be a
> > vector for ordinary, familiar, 3rd-order y; and that x must be too for
> > (<x){y. (A vector is a 3rd-order array y which satisfies (,1)-:$$y.) I
> > propose that there is another strange creature, which is a 2nd-order
> > array, or _list_. This is no less strange than 4th-order array, and j
> > does not have them. A list is very similar to a vector, though, in that
> > it is an ordered sequence of atoms. And there is an obvious 1:1 mapping
> > between vectors and lists. Therefore, when you write (<x){y, and x is a
> > vector, it is first _converted_ to the corresponding list, and that list
> > is used to index y. Similarly, when you write $y, and y is a third-order
> > array, its shape is actually a list; that list is _converted_ to the
> > corresponding vector before being returned.
> >
> > Then, fairly straightforwardly, we can define the 1st-order array, or
> > _atom_, another strange creature. Atoms are used to index lists, and the
> > shape of a list is an atom. And every atom (1st-order array) also has a
> > counterpart in familiar 3rd-order land, as a _scalar_. (A scalar is a
> > 3rd-order array y which satisfies ((,0)$0)-:$y.)
> >
> > The obvious difference between a vector and a list is that, if we want
> > element 5 of y, and y is a list, we write (<5){y; but if y is a vector, we
> > must write (<,5){y. But to make the distinction that way misses the
> > point, I think. A 2nd-order array is a particular sort of array which is
> > restricted in that it can only contain unidirectional sequences of atoms.
> > A vector is of a different sort of array, which is less restricted, but it
> > happens to belong to a particular subset which maps to lists. (The same
> > relationship holds for scalars and atoms.)
> >
> > Now, imagine a hypothetical version of j which could represent arrays of
> > orders other than 3. It is difficult to talk about such a j, because--for
> > instance--there might be an atom value 3 and a scalar value 3, and these
> > would be distinct; if I simply write '3', it is ambiguous which is
> > referred to. So I will err on the side of verbosity rather than
> > ambiguity.
> >
> > One thing immediately becomes clear: the result of $y in such a j should
> > be the _actual_ shape of y; that is, it should have order one lower than
> > y. What, then is the shape of an atom? It might be an error, but I
> > should rather make it the _unit_. The unit is the singular value which is
> > an order-0 array. The unit is also its own shape. Then, we can also say
> > that an order n array is one which requires n applications of $ to reach a
> > fixed point. Personally, I think a fixed point of a unit is nicer than a
> > fixed point of ,1 (which is what we currently have, with all results
> > squashed into 3rd-order arrays), but this is a point of style.
> >
> > Another nice consistency that results is that the idiom $$y evaluates to
> > an atom when y is a 3rd-order array. That is, it gives you the rank of y,
> > rather than a 1-element vector containing the rank.
> >
> > Another observation: this provides a way to relate j to k; the difference
> > between them is one of degree, not kind. In k, everything is a list or a
> > scalar; that is, all values have order 1 or order 2. In j, all values
> > have order 3. (There is another difference, which is that k has nests
> > where j has boxes, but this is orthogonal.)
> >
> > We can think of 3rd-order arrays as functions; for instance, an array of
> > rank 4 is like a function of 4 parameters. If we index such an array
> > using a list of length 4, that is like applying the corresponding
> > function, where each element of the list is associated with the
> > corresponding argument. By convention, these functions are curried.
> > Even if y has rank 4, it is legal in j to write (<1 2 0){y, even though 1
> > 2 0 has length 3. (Previously, I said I was considering this illegal;
> > now, I am defining it again.) This works by currying y; producing an
> > array which has its first 3 arguments fixed (to 1, 2, and 0,
> > respectively), requiring only 1 more; because it only requires 1 more
> > argument, we say it has rank 1. This works well enough for 3rd-order
> > arrays, but what about higher-order ones? When is something a valid
> > 'partial index', and what are the 'first arguments'?
> >
> > A better definition comes from , . If p is a valid index into y (as
> > defined in the 3rd paragraph), then (<p){y applies the indexing procedure
> > as usual. Otherwise, it is the array for which (<q){(<p){y is equivalent
> > to (<p,q){y. (This only holds if there is q for which, given p and y, p,q
> > is a valid index into y; otherwise it is an index error.) This is
> > adequate to describe existing indexing behaviour for 3rd-order arrays .
> > And it works without a hitch for 4th-order arrays. For higher-order
> > arrays, it is simply necessary to extend , to work on higher-order arrays;
> > given such an extension, that, too will work without a hitch (and probably
> > (?) give sensible results).
> >
> > I think the foregoing definition goes most of the way towards explaining
> > why we may index arrays using scalars; it is more than a convenience
> > feature, and the scalar is not simply converted to a vector. (Going the
> > rest of the way would require a complete and general model of , .)
> >
> > Now, some questions:
> >
> > 1. It there any practical value to this scheme? In some respects, an
> > array with a non-list shape just seems like an array with a list shape,
> > and bells on. But I am a low-dimensional creature, and perhaps if I had
> > more use for very high-dimensional arrays, I would find it useful to
> > structure my axes (and then structure the structure of my axes, and so on,
> > leading to arrays of ever higher orders).
> >
> > The simplest 4th-order array which is 'interesting' has shape 2 2$2; that
> > is, it corresponds roughly to a 4-dimensional array; that is rare enough.
> > The simplest 5th-order array which is 'interesting' has shape (2 2$2)$2,
> > that is 16-dimensional-ish. I generally have little use for arrays of
> > such high dimension (4 is reasonable, 16 is quite unlikely), and have not
> > found a linear organization stifling or problematic.
> >
> > The jump from the unit to the atom seems most significant, as it allows
> > you to communicate any information at all; the unit carries no
> > information. Next in significance is the jump from atoms to lists, which
> > permit the communication of inherently _structured_ information. This is
> > good enough for many applications (and k gets along just fine); moving to
> > 3rd-order arrays simply increases the degree of structure available, which
> > can be useful in some (ok, many) situations. So it would not be
> > surprising if the added benefit of moving to higher orders continued to
> > peter out, such that 3 hit a good sweet spot.
> >
> > 2. Is there any theoretical value to this scheme, or anything interesting
> > which can be described only using it? I used it to explain a number of
> > small inconsistencies, but that is marginal at best.
> >
> > Is there any existing mathematical precedent? I know that what j calls
> > arrays the literature calls tensors. So I googled for 'higher-order
> > tensors', but only found information on what we would call higher-_rank_
> > arrays, not what I distinguished here as higher-_order_ arrays. I also
> > searched for non-cartesian coordinate systems, but could not find anything
> > where the components of the coordinates were not sequential and flat.
> >
> > 3. Are the different orders of arrays qualitatively different? E.G. a
> > 3rd-order array can approximate both a 1st-order array (using a scalar)
> > and a 2nd-order array (using a vector); but a 2nd-order array can not
> > approximate a 1st-order array. (This is why, to my understanding, a
> > scalar is an array in j, but it is not a list in k.) (On the other hand,
> > 3rd-order arrays can not represent the unit.) I think all higher-order
> > arrays can approximate any 3rd-order array; but do they gain any new
> > powers?
> >
> > 4. I called arrays of order 0, 1, and 2 units, atoms, and lists,
> > respectively. What should arrays of order 3 be called? Perhaps 'spaces'?
> > Is there a sensible name for 4th- or 5th-order arrays?
> >
> > Is the name 'shape' still sensible if extended to different orders?
> >
> > 5. How would the j primitives work if extended to work on arrays of order
> > other than 3? What are the rules of conformability? Can you add an atom
> > to a scalar? A list to a matrix? A list to a vector? When can you
> > catenate two 5th-order arrays? How does rank work on 4th-order arrays?
> >
> > 6. Is there a good intuition for higher-order arrays? I can more or less
> > imagine what 4th-dimensional space is like (though I can't visualize it),
> > but I have no idea what a 'space' of shape 2 2$y is like, even though it
> > has the same number of 'degrees of freedom'.
> >
> > Is there a well-defined notion of distance defined in higher-order
> > 'space'? If so, is it still scalar?
> >
> >
> > Thoughts?
> >
> > -E
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
>
>
> --
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
