Yea, I've thought about it and given J's syntax (no special-case
syntax for indexing like APL) and semantics, the J authors went with
purity of language over special-case syntax kludges for efficiency.
They then made special-case parse exceptions to minimize the
performance issues. I now fully understand, mostly agree with their
choice, and feel the approach is reasonable.
Thanks.
Blake McBride
On Fri, Jan 6, 2012 at 11:58 AM, Dan Bron <j...@bron.us> wrote:
> It's also worth mentioning that amendment is a much rarer in J than one
> would expect coming from the non-array-oriented world. And iterated
> amendment (e.g., in a loop), where performance worries would be compounded,
> is rarer yet. Amendment, like almost every operation in J, is done in bulk.
> We don't think piecemeal here.
>
> For example, we would not write:
>
> x=:55 (3)} 44 (2)} x
>
> we would write:
>
> x=:55 44 (3 2)} x
>
> So, while amendment is common in scalar languages, and therefore a concern
> when implemented as copy-by-value, since it's infrequent in J, that worry is
> alleviated. But the flip side, as I mentioned in the previous email, is
> that almost every operation in J is copy-by-value. So if you're concerned
> about the costs of memcopies, you can't just restrict your focus to
> amendment. For example:
>
> BIG_DATA =: i. 1000 1000
>
> identity =: a:&{ NB. Identity function, returns input
>
> (%"1 <./) ts&> 'BIG_DATA';'identity BIG_DATA' NB. Compare
> performance, lower is better
> 1 1
> 5894 8193.88
>
> That said, rather than "copy-by-value", a better term for J's memory model
> is "copy-on-write". For example, if we use a less complex identity function
> (which is consequently easier to identify as a low-risk candidate for
> copy-by-reference):
>
> (%"1 <./) 2500000 ts&> 'BIG_DATA';'] BIG_DATA' NB. Compare
> performance, lower is better
> 1 1
> 1.09193 1.25
>
> Hope this helps,
>
> -Dan
>
> -----Original Message-----
> From: programming-boun...@jsoftware.com
> [mailto:programming-boun...@jsoftware.com] On Behalf Of Blake McBride
> Sent: Friday, January 06, 2012 11:45 AM
> To: Programming forum
> Subject: Re: [Jprogramming] Indexed assignment must be slow?
>
> Thanks. Got it.
>
> On Fri, Jan 6, 2012 at 10:39 AM, Dan Bron <j...@bron.us> wrote:
>> J has copy-by-value semantics. By default, all outputs occupy separate
>> memory spaces from their corresponding inputs. It's a safety net, which,
>> given that a lot of J programming is exploration, is very nice to have.
> Put
>> another way, this is one of the design decisions that contributes to J's
>> flexibility, responsiveness, and accommodating ("hey, what if?") feel.
>>
>> That said, for performance reasons, there are a few places where J
>> implements copy-by-reference (but only where this implementation detail
> can
>> be hidden from the programmer, so that it is indistinguishable from
>> copy-by-value). One such place is the specific formula you captured in
> (2).
>> Another is memory-mapped variables (mapped files). Some others are listed
>> on in the "Special Code" appendix [1], and in the release notes for
> various
>> versions of J [2].
>>
>> Anyway, I'll pass along some advice I was given as a J newcomer: don't
>> optimize prematurely.
>>
>> Don't worry about what "could" be slow in J. Build your application in
> the
>> clearest, most straightforward way, and try running it against realistic
>> data sets. If performance is unacceptable, profile it to find the
>> bottlenecks. If it isn't obvious how to remove the bottlenecks, or the
>> obvious workarounds are distasteful, ask here, on the Forums (include some
>> descriptive prose, working code, example inputs and expected outputs, and
>> possibly some benchmarks on your machine). Probably someone will suggest
> a
>> useful rephrasing (perhaps pointing out a special code phrase you hadn't
>> noticed before). But if not, at the very worst, you will have raised
>> awareness of a performance issue in J, which increases the chances it will
>> be addressed in a future version of the interpreter.
>>
>> -Dan
>>
>>
>> [1] Special J phrases recognized and optimized by the interpreter:
>> http://www.jsoftware.com/help/dictionary/special.htm
>>
>> [2] J release notes:
>> http://www.jsoftware.com/help/release/contents.htm
>>
>>
>>
>> -----Original Message-----
>> From: programming-boun...@jsoftware.com
>> [mailto:programming-boun...@jsoftware.com] On Behalf Of Blake McBride
>> Sent: Friday, January 06, 2012 10:56 AM
>> To: Programming forum
>> Subject: Re: [Jprogramming] Indexed assignment must be slow?
>>
>> So you are saying that J treats the following code 1 with a copy, and
>> code 2 without a copy?
>>
>> 1. y=:44(2})x
>>
>> 2. x=:44(2})x
>>
>> What about the following code:
>>
>> x=:55(3})44(2})x
>>
>> or:
>>
>> x=:55(3})(some verb returning a scalar)44(2})x
>>
>> Thanks for your response. It is very helpful!
>>
>> Blake McBride
>>
>>
>> On Fri, Jan 6, 2012 at 9:51 AM, Roger Hui <rogerhui.can...@gmail.com>
> wrote:
>>> David Mitchell already answered your question.
>>>
>>> x i}y is not indexed assign. (Where's the assign?) y=:x i}y *is
>> *indexed
>>> assign and is lean and fast.
>>>
>>>
>>>
>>> On Fri, Jan 6, 2012 at 7:42 AM, Blake McBride <bl...@mcbride.name> wrote:
>>>
>>>> This is a very important point. I'd like to get an official response to
>>>> this.
>>>>
>>>> Modifying an existing array could easily be 1,000 times faster than
>>>> creating a new, modified array. Naturally, this would depend on the
>>>> size and type of the array. Seeing how 'fast' it is in human or even
>>>> computer time means nothing! What is important is the relative time
>>>> between performing a real indexed update to a copy operation. Even a
>>>> small difference multiplied over a lot of operations can become
>>>> unbelievable significant. And, we are not talking about a small
>>>> difference!
>>>>
>>>> Imagine an application specific need to maintain a very large array
>>>> and make a lot of changes to individual elements. A program that does
>>>> only this on a large dataset could easily by 1,000 times more time
>>>> consuming. Imagine a situation where one program takes an hour to
>>>> complete a calculation and another takes 1,000 hours for the same
>>>> calculation. This is a significant issue.
>>>>
>>>> Blake McBride
>>>>
>>>>
>>>> On Fri, Jan 6, 2012 at 9:11 AM, David Mitchell <davidmitch...@att.net>
>>>> wrote:
>>>> > I believe that optimization is happening behind the scenes. See the
>>>> following
>>>> > tests actually timing indexed assignment, first without replacing the
>>>> original
>>>> > array and then replacing the original array:
>>>> >
>>>> > ts=: 6!:2 , 7!:2@]
>>>> > a=:i.1e6
>>>> > ts'_777 (2 9 43556)}a'
>>>> > 0.0238083 8.39078e6
>>>> > ts'a=:_777 (2 9 43556)}a'
>>>> > 1.00571e_5 2432
>>>> >
>>>> > On 1/6/2012 9:54, Blake McBride wrote:
>>>> >> Greetings,
>>>> >>
>>>> >> Going through J I found that:
>>>> >>
>>>> >> x=:1+i.5
>>>> >> x
>>>> >> 1 2 3 4 5
>>>> >> 44 (2}) x
>>>> >> 1 2 44 4 5
>>>> >> x
>>>> >> 1 2 3 4 5
>>>> >>
>>>> >> It seems that J doesn't actually make an indexed assignment. It
> seems
>>>> >> more like it creates a copy of the array with the specified
>>>> >> modifications. This is actually quite shocking. Index assignment to
>>>> >> a very large array in most languages is a very fast and simple
>>>> >> operation. It is also very memory efficient. Creating a modified
>>>> >> copy of a very large array has a much, much greater time and space
>>>> >> cost. Further, multiplying this (unnecessary) cost over a great
>>>> >> number of operations can drastically (and unnecessarily) affect
>>>> >> performance negatively.
>>>> >>
>>>> >> Am I missing something here?
>>>> >>
>>>> >> Thanks.
>>>>
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
