Raul,
I will offer one narrow use case where threading shines. I used the T
primitives recently with gethttp calls to speed a web crawling routine by a
factor of 60, which brought the problem from infeasible to near-trivial. So
while I agree that performance gains on CPU-bound tasks may be elusive and of
mostly academic interest, I think concurrency primitives have a lot of
potential with certain kinds of IO-bound tasks.
Ed
Sent from my iPad
> On Dec 20, 2022, at 2:30 PM, Raul Miller <rauldmil...@gmail.com> wrote:
>
> Knowing where to put the primitives is indeed the issue. That's why
> they're primitives, basically.
>
> Conceptually, the primitives are simple. T. is for configuration and
> setup. You always have one thread (the master thread), and you
> probably should have one additional thread for each cpu "core" (which
> you might think of as a register set -- a small set of variables that
> represent the machine itself).
>
> To accomplish this, you might use a routine like
>
> setthreadcount=: {{
> change=. y-1 T.''
> T.&''"0 (|change)#(*change){0 0 55
> 1 T.''
> }}
>
> setthreadcount <:{.8T.''
>
> Note that the value returned by this routine may be higher than
> expected if you had threads in use running tasks at the time. For
> example, if you had 32 threads running tasks on a 28 core machine, you
> might expect this routine to return 27. I could have named it
> 'setfuturethreadcount" to try to convey that it would eventually
> settle down to that number of threads, but as it is the name is
> already longer than I really like.
>
> ---------------
>
> But threads are not tasks. To fire off tasks you use (t.), and
> (besides timing issues), (t.'') is basically the same as (<@).
>
> To get a handle on the timing side of things, it's probably best to
> use timex. That said, I haven't found any good candidates for
> threading, yet. (By this I mean: simple code which gets improved
> performance using (t.'') (or (t.{{)n}})) vs using (<@)
>
> (That said, tasks can also be useful in some academic contexts, even
> without a performance gain.)
>
> I hope this helps,
>
> --
> Raul
>
>
>
>
>
>
>> On Tue, Dec 20, 2022 at 2:06 PM Ak O <akin...@gmail.com> wrote:
>>
>> Hi Devon,
>>
>> Thank you for the iterations of your exercise. I have studied these posts
>> along with your earliest proto examples in the multi-threading forum posts.
>> My challenge is understanding which Primitives go where in the context of
>> building up a threading routine for a directed task.
>>
>> Considering your photo flipping example:
>>
>> 1. I might structure a scheduling thread that:
>>
>> a) Assigns threads to process the files within a folder. Each of which
>> checking if there are remaining files within the folder. Alternatively
>> spinning down if none remain.
>>
>> b) Assigns threads to process folders. Each processing folders and
>> returning to check the scheduler for remaining unprocessed folders.
>> Alternatively spinning down in the absence of unprocessed folders.
>>
>> c) Batches files into groups of a given size class. Assigning subsets of
>> threads to unprocessed size classes.
>>
>> d) Along with many others.
>>
>> All of which having trade-offs based on the processing question.
>>
>>
>>
>> So knowing what the pieces are and how to properly place them is my issue.
>>
>>
>> Ak
>>
>>
>>
>>
>> On Tue., DecHi . 20, 2022, 10:34 Devon McCormick, <devon...@gmail.com>
>> wrote:
>>
>>> Hi Ak,
>>>
>>> That's a lot of questions. I, too, would like to better understand pyxs.
>>>
>>> In the meantime, here are some things I've written up on multithreading:
>>> https://code.jsoftware.com/wiki/NYCJUG/2022-05-10#Trying_to_Multithread,
>>>
>>> https://code.jsoftware.com/wiki/NYCJUG/2022-07-12#Multi-threaded_Photo_Flipping
>>> ,
>>> and
>>>
>>> https://code.jsoftware.com/wiki/NYCJUG/2022-08-09#A_New_Multi-threading_Project
>>> .
>>>
>>> It looks like I'm using the "Manager Model" in these exercises.
>>>
>>> Good luck!
>>>
>>> Devon
>>>
>>>> On Tue, Dec 20, 2022 at 12:13 PM Ak O <akin...@gmail.com> wrote:
>>>
>>>> I hope you are all well. Thank you for your hard work on this subject.
>>>>
>>>> I am having difficulty understanding how to correctly apply the T. and t.
>>>> (tcapdot/tdot) Primitives based on the vocabulary documentation pages.
>>>>
>>>>
>>>> The concept of threading is new to me. The documentation pages (T./t.)
>>> use
>>>> many terms that I am unclear on.
>>>>
>>>> I appreciate your efforts to explain them. Thank you for your help.
>>>>
>>>>
>>>> Ak.
>>>>
>>>>
>>>>
>>>> Vocabulary_tdot Page
>>>>
>>>>
>>>> Reading from the Vocabulary documentation page tdot (at indentation).
>>>>
>>>>
>>>> i. t. ''"0 i. 5
>>>> ++-+---+-----+-------+
>>>>
>>>> ||0|0 1|0 1 2|0 1 2 3|
>>>> ++-+---+-----+-------+
>>>>
>>>> In the context of threading, what is this statement meant to
>>> characterize?
>>>> Is there a sentence of the form ,'execute some operation in some distinct
>>>> thread (having thread attribute(s)..).'?
>>>>
>>>> [x] u t. n y runs ([x] <@:u y) in an idle thread and returns its
>>>> result, which is a pyx. A pyx is a special box. It looks like an atomic
>>> box
>>>> and can be used like any other box. When the pyx is opened, the opening
>>>> thread will block until the value is available. If the creation of the
>>> pyx
>>>> ended in error, that error will be signaled when the pyx is opened.
>>>>
>>>>
>>>> What is:
>>>> ->the purpose of a 'pyx'?
>>>> Where does a 'pyx' fit into the threading concept?
>>>> What is:
>>>> ->thread?
>>>> ->idle thread?
>>>> How is a thread identified?
>>>>
>>>>
>>>> The n argument to t. controls execution of the task. The format is
>>>>
>>>> [[<]important parameters][,< keyword[;value]]...
>>>> The important parameters are: [threadpool#]
>>>>
>>>> What is:
>>>> -> a task?
>>>> -> important parameters?
>>>> -> [threadpool#]
>>>> -> threadpool
>>>> -> i. t.(0;('worker'))"_1 i.5
>>>>
>>>>
>>>> threadpool# indicates which threadpool the task should be assigned
>>> to.
>>>>
>>>> How is a threadpool assigned?
>>>> How is a threadpool identified?
>>>>
>>>>
>>>> Threads are assigned to threadpools when they are created.
>>>>
>>>> How are assigned threadpools identified for created threads?
>>>>
>>>>
>>>> By default, threadpool 0 is used.
>>>>
>>>> What is the equivalent representation for this statement (i.e [x] u t. n
>>> y
>>>> runs ([x] <@:u y))?
>>>>
>>>>
>>>> One keyword has been defined.
>>>>
>>>> What is a 'keyword' (conceptually and type)?
>>>> Is it predefined?
>>>>
>>>> The worker keyword controls whether the task should be executed in
>>> the
>>>> originating thread if there is no waiting thread for it to run in.
>>>>
>>>> How does the keyword control the task executing thread?
>>>>
>>>>
>>>> u t. 'worker'
>>>> u t. (<'worker')
>>>> u t. ('worker';1)
>>>>
>>>> What differentiates these forms?
>>>> Is it always 'worker' a standin?
>>>> A few words on cases where one form might be chosen instead of another.
>>>>
>>>>
>>>>
>>>> --->
>>>> Reference questions
>>>>
>>>> In trying to understand how to build threading structures (threaded
>>>> program), three basic models are given:
>>>> (https://hpc-tutorials.llnl.gov/posix/designing_threaded_programs)
>>>> Quote:
>>>> Several common models for threaded programs exist:
>>>>
>>>> Manager/worker: a single thread, the manager assigns work to
>>> other
>>>> threads, the workers.
>>>> Typically, the manager handles all input and parcels out work to the
>>>> other
>>>> tasks. At least two forms of the manager/worker model are common:
>>>> static
>>>> worker pool and dynamic worker pool.
>>>>
>>>> Pipeline: a task is broken into a series of suboperations, each
>>> of
>>>> which is handled
>>>> in series, but concurrently, by a different thread. An automobile
>>>> assembly
>>>> line best describes this model.
>>>>
>>>> Peer: similar to the manager/worker model, but after the main
>>>> thread creates other
>>>> threads, it participates in the work.
>>>>
>>>> Can you please show or diagram the J primitives used to compose the three
>>>> models?
>>>>
>>>>
>>>> --->
>>>>
>>>>
>>>> - - - Please give a Primitives translation from C to J under the Creating
>>>> Pthreads heading, Reference Source (https://randu.org/tutorials/threads/
>>> ).
>>>> - - -
>>>>
>>>> - - - Please give a Primitives translation from C to J under the Pthreads
>>>> Attributes heading, Reference Source (
>>> https://randu.org/tutorials/threads/
>>>> ).
>>>> - -
>>>> ----------------------------------------------------------------------
>>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>>
>>>
>>>
>>> --
>>>
>>> Devon McCormick, CFA
>>>
>>> Quantitative Consultant
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
