I'm not so sure generalizing the shape vector is the best way of generalizing
arrays. To my mind, an array is a function f:I->V, where I is the index set
and V is the value set.
A vector with shape 3 has index set i.3. An array with shape 3 5 has index set
the Cartesian product (i.3) x (i.5).
The shape vector is shorthand for the index set for a rectangular array, and we
specify the index function by listing its values. Of course, shape has other
uses in J, such specifying frames and cells when considering rank.
The index set can be something different than i.n. For example, dictionaries
with strings as keys are arrays with index set the set of strings. Sparse
matrices have index set a finite subset of N x N, where N={0,1,2,...} is the
natural numbers, with the convention that the index function evaluates to a
zero element outside this finite set. Shape starts to be less useful here.
In addition to fixing an index set and allowing arbitrary functions to the
value set, certain constraints on the index function may be useful. For
example, for an identity matrix, f(i,j)=1 if i=j and 0 otherwise, and for a
symmetric matrix, f(i,j)=f(j,i).
I suggest focusing on the index function rather than the index set.
Best wishes,
John
________________________________
From: Programming <programming-boun...@forums.jsoftware.com> on behalf of Raul
Miller <rauldmil...@gmail.com>
Sent: Thursday, March 3, 2022 11:12 AM
To: Programming forum <programm...@jsoftware.com>
Subject: Re: [Jprogramming] Higher-order arrays?
A generic fix for the array padding problem lives in the sparse array
implementation (which lets us specify the fill element for an array).
But we have a lot of work ahead of us, if we want to bring sparse
arrays and dense arrays together. (Conceptually, we would want a
conjunction or conjunction family (maybe with a few adverbial cousins)
which lets n specify sparse or sparse-like characteristics for u.)
Anyways, right now, I don't think we have the internal interfaces to
pull that off. But it's something that might be nice to work towards.
Scary, but nice.
Thanks,
--
Raul
On Thu, Mar 3, 2022 at 11:02 AM Henry Rich <henryhr...@gmail.com> wrote:
>
> I like this, except for the trailing-0 bit. I would say having data with
> trailing 0 shape is 'common' rather than 'ultra-rare'.
>
> Henry Rich
>
> On 3/3/2022 10:15 AM, 'Pascal Jasmin' via Programming wrote:
> > I don't have a good grasp of order, but a different practical
> > interpretation of non-list shapes:
> >
> > 1. You mentioned distinguishing atoms from lists of 1 element. J already
> > manages this well, and making an atomic shape result would not be of any
> > help, IMO.
> >
> > 2. A table shape could be a code for ragged arrays, or arrays of arrays.
> > The following shape:
> >
> > 3 2 2
> > 4 0 0
> >
> > Would be 2 arrays, the first is shape 3 2 2, and the 2nd is shape 4.
> > Trailing 0s would be fills. This precludes the ultra rare need of needing
> > an "embedded" shape 4 0 or shape 4 0 0 in your data.
> >
> > A list of 3 strings/lists with individual lengths of 2 3 4 would have the
> > shape:
> >
> > 2
> > 3
> > 4
> >
> > If there is an extention of the meaning of shape, I'd prefer it be in this
> > direction rather than ultra high dimensional math that hurts my head.
> >
> >
> > On Thursday, March 3, 2022, 05:13:27 a.m. EST, Elijah Stone
> > <elro...@elronnd.net> wrote:
> >
> >
> >
> >
> >
> > A little thought exercise I embarked upon.
> > Probably not very clearly expressed, and certainly of no practical use,
> > but I think there are some interesting ideas:
> >
> > Why must the result of $y always be a vector? What would happen if we let
> > it be a higher-ranked array? What would that _mean_?
> >
> > Well, say that 2 3 4-:$y. That means that y is a mapping from arrays x
> > that have the same shape as 2 3 4 (and whose elements are natural and
> > bounded by 2 3 4). In other words, if (x -:&$ 2 3 4) *. -. 0 e. x<2 3 4
> > then (<x){y is well-defined as an atom. (I am intentially ignoring the
> > possibility of sentences such as ((<1 1){y) ((<,1){y) ((<1){y) and so
> > on; consider these sentences ill-formed for the time being.)
> > Generalising: if (x -:&$ $y) *. -. 0 e. x<$y then (<x){y is well-defined
> > as an atom.
> >
> > But this sentence makes no restrictions whatsoever on the rank of x! So
> > it would seem that we've discovered a way to make a y whose shape is an
> > array of any rank. But we must be careful. Really, all we've constructed
> > is a strange entity y--call it an array-as-we-do-not-know-it--and, on
> > arrays-as-we-do-not-know-them, we have defined that: $y is well-defined,
> > and evaluates to an array-as-we-know-it. x{y is well-defined for some
> > subset of arrays-as-we-know-them x, and evaluates to an
> > atom-as-we-know-it.
> >
> > I will call this strange creature a 4th-order array. And all
> > arrays-as-we-know-them are 3rd-order (this includes scalars, vectors, and
> > higher-rank arrays). A 4th-order array has a shape which is a 3rd-order
> > array, and maps from 3rd-order arrays to atoms. This suggests the
> > existence of lower-order arrays, and higher-order ones.
> >
> > Following the same logic, then, a 3rd-order array would map 2nd-order
> > arrays to atoms, and its shape would be a 2nd-order array. But--for
> > 3rd-order y, the x in (<x){y for 3rd-order y is currently also a 3rd-order
> > array, and $y is currently also 3rd-order.
> >
> > Or is it, really? We've already established that $y must always be a
> > vector for ordinary, familiar, 3rd-order y; and that x must be too for
> > (<x){y. (A vector is a 3rd-order array y which satisfies (,1)-:$$y.) I
> > propose that there is another strange creature, which is a 2nd-order
> > array, or _list_. This is no less strange than 4th-order array, and j
> > does not have them. A list is very similar to a vector, though, in that
> > it is an ordered sequence of atoms. And there is an obvious 1:1 mapping
> > between vectors and lists. Therefore, when you write (<x){y, and x is a
> > vector, it is first _converted_ to the corresponding list, and that list
> > is used to index y. Similarly, when you write $y, and y is a third-order
> > array, its shape is actually a list; that list is _converted_ to the
> > corresponding vector before being returned.
> >
> > Then, fairly straightforwardly, we can define the 1st-order array, or
> > _atom_, another strange creature. Atoms are used to index lists, and the
> > shape of a list is an atom. And every atom (1st-order array) also has a
> > counterpart in familiar 3rd-order land, as a _scalar_. (A scalar is a
> > 3rd-order array y which satisfies ((,0)$0)-:$y.)
> >
> > The obvious difference between a vector and a list is that, if we want
> > element 5 of y, and y is a list, we write (<5){y; but if y is a vector, we
> > must write (<,5){y. But to make the distinction that way misses the
> > point, I think. A 2nd-order array is a particular sort of array which is
> > restricted in that it can only contain unidirectional sequences of atoms.
> > A vector is of a different sort of array, which is less restricted, but it
> > happens to belong to a particular subset which maps to lists. (The same
> > relationship holds for scalars and atoms.)
> >
> > Now, imagine a hypothetical version of j which could represent arrays of
> > orders other than 3. It is difficult to talk about such a j, because--for
> > instance--there might be an atom value 3 and a scalar value 3, and these
> > would be distinct; if I simply write '3', it is ambiguous which is
> > referred to. So I will err on the side of verbosity rather than
> > ambiguity.
> >
> > One thing immediately becomes clear: the result of $y in such a j should
> > be the _actual_ shape of y; that is, it should have order one lower than
> > y. What, then is the shape of an atom? It might be an error, but I
> > should rather make it the _unit_. The unit is the singular value which is
> > an order-0 array. The unit is also its own shape. Then, we can also say
> > that an order n array is one which requires n applications of $ to reach a
> > fixed point. Personally, I think a fixed point of a unit is nicer than a
> > fixed point of ,1 (which is what we currently have, with all results
> > squashed into 3rd-order arrays), but this is a point of style.
> >
> > Another nice consistency that results is that the idiom $$y evaluates to
> > an atom when y is a 3rd-order array. That is, it gives you the rank of y,
> > rather than a 1-element vector containing the rank.
> >
> > Another observation: this provides a way to relate j to k; the difference
> > between them is one of degree, not kind. In k, everything is a list or a
> > scalar; that is, all values have order 1 or order 2. In j, all values
> > have order 3. (There is another difference, which is that k has nests
> > where j has boxes, but this is orthogonal.)
> >
> > We can think of 3rd-order arrays as functions; for instance, an array of
> > rank 4 is like a function of 4 parameters. If we index such an array
> > using a list of length 4, that is like applying the corresponding
> > function, where each element of the list is associated with the
> > corresponding argument. By convention, these functions are curried.
> > Even if y has rank 4, it is legal in j to write (<1 2 0){y, even though 1
> > 2 0 has length 3. (Previously, I said I was considering this illegal;
> > now, I am defining it again.) This works by currying y; producing an
> > array which has its first 3 arguments fixed (to 1, 2, and 0,
> > respectively), requiring only 1 more; because it only requires 1 more
> > argument, we say it has rank 1. This works well enough for 3rd-order
> > arrays, but what about higher-order ones? When is something a valid
> > 'partial index', and what are the 'first arguments'?
> >
> > A better definition comes from , . If p is a valid index into y (as
> > defined in the 3rd paragraph), then (<p){y applies the indexing procedure
> > as usual. Otherwise, it is the array for which (<q){(<p){y is equivalent
> > to (<p,q){y. (This only holds if there is q for which, given p and y, p,q
> > is a valid index into y; otherwise it is an index error.) This is
> > adequate to describe existing indexing behaviour for 3rd-order arrays .
> > And it works without a hitch for 4th-order arrays. For higher-order
> > arrays, it is simply necessary to extend , to work on higher-order arrays;
> > given such an extension, that, too will work without a hitch (and probably
> > (?) give sensible results).
> >
> > I think the foregoing definition goes most of the way towards explaining
> > why we may index arrays using scalars; it is more than a convenience
> > feature, and the scalar is not simply converted to a vector. (Going the
> > rest of the way would require a complete and general model of , .)
> >
> > Now, some questions:
> >
> > 1. It there any practical value to this scheme? In some respects, an
> > array with a non-list shape just seems like an array with a list shape,
> > and bells on. But I am a low-dimensional creature, and perhaps if I had
> > more use for very high-dimensional arrays, I would find it useful to
> > structure my axes (and then structure the structure of my axes, and so on,
> > leading to arrays of ever higher orders).
> >
> > The simplest 4th-order array which is 'interesting' has shape 2 2$2; that
> > is, it corresponds roughly to a 4-dimensional array; that is rare enough.
> > The simplest 5th-order array which is 'interesting' has shape (2 2$2)$2,
> > that is 16-dimensional-ish. I generally have little use for arrays of
> > such high dimension (4 is reasonable, 16 is quite unlikely), and have not
> > found a linear organization stifling or problematic.
> >
> > The jump from the unit to the atom seems most significant, as it allows
> > you to communicate any information at all; the unit carries no
> > information. Next in significance is the jump from atoms to lists, which
> > permit the communication of inherently _structured_ information. This is
> > good enough for many applications (and k gets along just fine); moving to
> > 3rd-order arrays simply increases the degree of structure available, which
> > can be useful in some (ok, many) situations. So it would not be
> > surprising if the added benefit of moving to higher orders continued to
> > peter out, such that 3 hit a good sweet spot.
> >
> > 2. Is there any theoretical value to this scheme, or anything interesting
> > which can be described only using it? I used it to explain a number of
> > small inconsistencies, but that is marginal at best.
> >
> > Is there any existing mathematical precedent? I know that what j calls
> > arrays the literature calls tensors. So I googled for 'higher-order
> > tensors', but only found information on what we would call higher-_rank_
> > arrays, not what I distinguished here as higher-_order_ arrays. I also
> > searched for non-cartesian coordinate systems, but could not find anything
> > where the components of the coordinates were not sequential and flat.
> >
> > 3. Are the different orders of arrays qualitatively different? E.G. a
> > 3rd-order array can approximate both a 1st-order array (using a scalar)
> > and a 2nd-order array (using a vector); but a 2nd-order array can not
> > approximate a 1st-order array. (This is why, to my understanding, a
> > scalar is an array in j, but it is not a list in k.) (On the other hand,
> > 3rd-order arrays can not represent the unit.) I think all higher-order
> > arrays can approximate any 3rd-order array; but do they gain any new
> > powers?
> >
> > 4. I called arrays of order 0, 1, and 2 units, atoms, and lists,
> > respectively. What should arrays of order 3 be called? Perhaps 'spaces'?
> > Is there a sensible name for 4th- or 5th-order arrays?
> >
> > Is the name 'shape' still sensible if extended to different orders?
> >
> > 5. How would the j primitives work if extended to work on arrays of order
> > other than 3? What are the rules of conformability? Can you add an atom
> > to a scalar? A list to a matrix? A list to a vector? When can you
> > catenate two 5th-order arrays? How does rank work on 4th-order arrays?
> >
> > 6. Is there a good intuition for higher-order arrays? I can more or less
> > imagine what 4th-dimensional space is like (though I can't visualize it),
> > but I have no idea what a 'space' of shape 2 2$y is like, even though it
> > has the same number of 'degrees of freedom'.
> >
> > Is there a well-defined notion of distance defined in higher-order
> > 'space'? If so, is it still scalar?
> >
> >
> > Thoughts?
> >
> > -E
> > ----------------------------------------------------------------------
> > For information about J forums see
> > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jsoftware.com%2Fforums.htm&data=04%7C01%7Crandall%40newark.rutgers.edu%7C2e94be69e97a4f2f571508d9fd30b1e8%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637819207865742748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wGsArk3qnqzexIdlTF45GF3HavLSPDMRNO9Us1xeVWY%3D&reserved=0
> > ----------------------------------------------------------------------
> > For information about J forums see
> > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jsoftware.com%2Fforums.htm&data=04%7C01%7Crandall%40newark.rutgers.edu%7C2e94be69e97a4f2f571508d9fd30b1e8%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637819207865742748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wGsArk3qnqzexIdlTF45GF3HavLSPDMRNO9Us1xeVWY%3D&reserved=0
>
>
> --
> This email has been checked for viruses by AVG.
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.avg.com%2F&data=04%7C01%7Crandall%40newark.rutgers.edu%7C2e94be69e97a4f2f571508d9fd30b1e8%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637819207865742748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=KOq5bMX237kjOUKaYtSIlzQ7sQ07xQFzB%2Bj48aJKvIE%3D&reserved=0
>
> ----------------------------------------------------------------------
> For information about J forums see
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jsoftware.com%2Fforums.htm&data=04%7C01%7Crandall%40newark.rutgers.edu%7C2e94be69e97a4f2f571508d9fd30b1e8%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637819207865742748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wGsArk3qnqzexIdlTF45GF3HavLSPDMRNO9Us1xeVWY%3D&reserved=0
----------------------------------------------------------------------
For information about J forums see
https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.jsoftware.com%2Fforums.htm&data=04%7C01%7Crandall%40newark.rutgers.edu%7C2e94be69e97a4f2f571508d9fd30b1e8%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637819207865742748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wGsArk3qnqzexIdlTF45GF3HavLSPDMRNO9Us1xeVWY%3D&reserved=0
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
