Re: [Factor-talk] Startup time

[Timothy Hobbs]

Yes, the lockfile method is only if you cannot get startup times down 
otherwise and need to resort to starting only once, but do not want to 
have a "forever running daemon".


On 02/02/2017 10:09 AM, pet...@riseup.net wrote:
> On 2017-02-01 22:44, Jim Mack wrote:
>> Forgive me if I sounded like I was pushing.  I am working daily in OS
>> development where tools like grunt are commonly accepted, and
>> developers
>> accept choices that maximize their power/ease over many other concerns.
>>   I
>> never heard the slippery slope argument used against node.js, and it
>> would
>> surely lose!
>>
>> On Wed, Feb 1, 2017 at 2:20 PM, Timothy Hobbs  wrote:
>>
>>> Just so you understand the gif, what happens is a client starts up. It
>>> grabs a lock on the counter file. It reads the counter file. If the
>>> counter is zero, it launches the service. If it is greater than zero,
>>> then it connects to the service. After launching or connecting to the
>>> service, it increments the counter, and releases the lock on the lock
>>> file. When a client closes, it sends a command to the service, which
>>> then grabs a lock on the lock file. Decriments the counter, and if the
>>> counter is now zero, it shuts down, otherwise it merely releases the
>>> lock. There are no race conditions with this method. If two clients
>>> start up at the same time, they will have to wait in line for a
>>> connection to the lock file. If a client shuts down at the same time
>>> as
>>> another client starts up, this to means that the service and the
>>> client
>>> will have to wait their turns for the lock, and therefore, no races...
>>>
>>>
>>> On 02/01/2017 11:16 PM, Timothy Hobbs wrote:
>>>> You can use a dbus on-demand service or your own locking mechanism, if
>>>> you, like me, don't like dbus. Here is a gif which describes the process
>>>> for starting and stopping a race-free on demand service
>>>> http://timothy.hobbs.cz/subuser-client-service/lock-file.gif using
>>>> standard lock files. You can modify this method, so that the service
>>>> remains running for a certain number of seconds after the client counter
>>>> has reached zero, so that in a sequential shell script, you wouldn't be
>>>> launching and stopping the service over and over again.
>>>>
>>>> However, what I was refering to, with shared executable memory, has
>>>> nothing to do with background daemons. It is a feature that is built
>>>> into most modern OS kernels. Many kernels load memory pages that are
>>>> marked as executable as read-only, and share those pages between
>>>> processes. This greatly reduces startup times, and also improves
>>>> security (marking them read only that is). It has the dissadvantage,
>>>> that self modifying code is impossible. Factor, being a weird system
>>>> that self-modifies itself, cannot take advantage of this mechanism at
>>>> all. So we'll have to do something more advanced, like use criu, which I
>>>> linked to previously.
>>>>
>>>> On 02/01/2017 11:10 PM, pet...@riseup.net wrote:
>>>>> On 2017-02-01 19:40, Jim Mack wrote:
>>>>>> So why not create a separate small process that passes on its
>>>>>> parameters to
>>>>>> a running factor if it can find it, or starts a new one if it can't?
>>>>>>
>>>>> That's like running a server and sending requests to it. I take several
>>>>> issues with that:
>>>>>
>>>>> 1 - I need one instance to have *all* my libraries, present and future
>>>>> to be pre-loaded. But more importantly:
>>>>> 2 - a typical shell script can call a dozen external executables. Some
>>>>> will be in C, some in bash, some in python, some in perl etc. If every
>>>>> language would need a huge server to run, where would that leave us?
>>>>>
>>>>>> On Wed, Feb 1, 2017 at 7:51 AM, Timothy Hobbs 
>>> wrote:
>>>>>>> Have you tried loading the
>>>>>>> factor interpreter in the background and seeing if factor launches
>>>>>>> quicker while another factor process is running?
>>>>> I did what I think is fair - started it once so everything necessary
>>>>> gets cached in RAM and discard that run. 

Re: [Factor-talk] Startup time

[Timothy Hobbs]

Just so you understand the gif, what happens is a client starts up. It 
grabs a lock on the counter file. It reads the counter file. If the 
counter is zero, it launches the service. If it is greater than zero, 
then it connects to the service. After launching or connecting to the 
service, it increments the counter, and releases the lock on the lock 
file. When a client closes, it sends a command to the service, which 
then grabs a lock on the lock file. Decriments the counter, and if the 
counter is now zero, it shuts down, otherwise it merely releases the 
lock. There are no race conditions with this method. If two clients 
start up at the same time, they will have to wait in line for a 
connection to the lock file. If a client shuts down at the same time as 
another client starts up, this to means that the service and the client 
will have to wait their turns for the lock, and therefore, no races...


On 02/01/2017 11:16 PM, Timothy Hobbs wrote:
> You can use a dbus on-demand service or your own locking mechanism, if
> you, like me, don't like dbus. Here is a gif which describes the process
> for starting and stopping a race-free on demand service
> http://timothy.hobbs.cz/subuser-client-service/lock-file.gif using
> standard lock files. You can modify this method, so that the service
> remains running for a certain number of seconds after the client counter
> has reached zero, so that in a sequential shell script, you wouldn't be
> launching and stopping the service over and over again.
>
> However, what I was refering to, with shared executable memory, has
> nothing to do with background daemons. It is a feature that is built
> into most modern OS kernels. Many kernels load memory pages that are
> marked as executable as read-only, and share those pages between
> processes. This greatly reduces startup times, and also improves
> security (marking them read only that is). It has the dissadvantage,
> that self modifying code is impossible. Factor, being a weird system
> that self-modifies itself, cannot take advantage of this mechanism at
> all. So we'll have to do something more advanced, like use criu, which I
> linked to previously.
>
> On 02/01/2017 11:10 PM, pet...@riseup.net wrote:
>> On 2017-02-01 19:40, Jim Mack wrote:
>>> So why not create a separate small process that passes on its
>>> parameters to
>>> a running factor if it can find it, or starts a new one if it can't?
>>>
>> That's like running a server and sending requests to it. I take several
>> issues with that:
>>
>> 1 - I need one instance to have *all* my libraries, present and future
>> to be pre-loaded. But more importantly:
>> 2 - a typical shell script can call a dozen external executables. Some
>> will be in C, some in bash, some in python, some in perl etc. If every
>> language would need a huge server to run, where would that leave us?
>>
>>> On Wed, Feb 1, 2017 at 7:51 AM, Timothy Hobbs  wrote:
>>>
>>>> Have you tried loading the
>>>> factor interpreter in the background and seeing if factor launches
>>>> quicker while another factor process is running?
>> I did what I think is fair - started it once so everything necessary
>> gets cached in RAM and discard that run. As noted above I don't think
>> running a server for each possible language is a good solution.
>>
>>
>> Feel free to contradict me gentlemen, I'm open to discussion, but I do
>> have my own opinion of what is acceptable and transferable to other PCs
>> / colleagues. I'm not looking for some local hack to speed things up but
>> a general solution that doesn't put any more burden on the end users
>> than it is necessary.
>>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Startup time

[Timothy Hobbs]

You can use a dbus on-demand service or your own locking mechanism, if 
you, like me, don't like dbus. Here is a gif which describes the process 
for starting and stopping a race-free on demand service 
http://timothy.hobbs.cz/subuser-client-service/lock-file.gif using 
standard lock files. You can modify this method, so that the service 
remains running for a certain number of seconds after the client counter 
has reached zero, so that in a sequential shell script, you wouldn't be 
launching and stopping the service over and over again.

However, what I was refering to, with shared executable memory, has 
nothing to do with background daemons. It is a feature that is built 
into most modern OS kernels. Many kernels load memory pages that are 
marked as executable as read-only, and share those pages between 
processes. This greatly reduces startup times, and also improves 
security (marking them read only that is). It has the dissadvantage, 
that self modifying code is impossible. Factor, being a weird system 
that self-modifies itself, cannot take advantage of this mechanism at 
all. So we'll have to do something more advanced, like use criu, which I 
linked to previously.

On 02/01/2017 11:10 PM, pet...@riseup.net wrote:
> On 2017-02-01 19:40, Jim Mack wrote:
>> So why not create a separate small process that passes on its
>> parameters to
>> a running factor if it can find it, or starts a new one if it can't?
>>
> That's like running a server and sending requests to it. I take several
> issues with that:
>
> 1 - I need one instance to have *all* my libraries, present and future
> to be pre-loaded. But more importantly:
> 2 - a typical shell script can call a dozen external executables. Some
> will be in C, some in bash, some in python, some in perl etc. If every
> language would need a huge server to run, where would that leave us?
>
>> On Wed, Feb 1, 2017 at 7:51 AM, Timothy Hobbs  wrote:
>>
>>> Have you tried loading the
>>> factor interpreter in the background and seeing if factor launches
>>> quicker while another factor process is running?
> I did what I think is fair - started it once so everything necessary
> gets cached in RAM and discard that run. As noted above I don't think
> running a server for each possible language is a good solution.
>
>
> Feel free to contradict me gentlemen, I'm open to discussion, but I do
> have my own opinion of what is acceptable and transferable to other PCs
> / colleagues. I'm not looking for some local hack to speed things up but
> a general solution that doesn't put any more burden on the end users
> than it is necessary.
>


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Startup time

[Timothy Hobbs]

One thing that occures to me, is that bash is cheeting. Bash actually
takes longer than that to load to memory, but because it is already
running on your system, on linux at least, the new bash process shares
executable memory with the old bash processes. Factor probably loads
everything to DATA sections of memory (which are read-write) and
therefore linux cannot share all of that effort that has gone into
loading the factor process. I'm not an expert on how this works, so
please don't yell at me if I'm wrong ;). Have you tried loading the
factor interpreter in the background and seeing if factor launches
quicker while another factor process is running?


On 02/01/2017 16:31, pet...@riseup.net wrote:
> On 2017-02-01 15:16, John Benediktsson wrote:
>> I did that and it's not a huge amount, maybe 10-20%. We need to
>> profile and see what else it's doing. Like I said, low hanging fruit
>> are aplenty.
>>
>>
>>> On Feb 1, 2017, at 12:37 AM, pet...@riseup.net wrote:
>>>
>>>> On 2017-02-01 00:19, John Benediktsson wrote:
>>>> Part of the startup time is calling all of our "startup hooks", of
>>>> which I
>>>> have 35 right now in my Factor instance, in all of these 
>>>> vocabularies:
>>>>
>>>> {
>>>>"alien"
>>>>"destructors"
>>>>"alien.strings"
>>>>"io.backend"
>>>>"source-files.errors"
>>>>"compiler.units"
>>>>"vocabs"
>>>>"io.files"
>>>>"vocabs.loader"
>>>>"command-line"
>>>>"threads"
>>>>"cpu.x86.features"
>>>>"io.thread"
>>>>"core-foundation.run-loop"
>>>>"environment"
>>>>"io.backend.unix:signal-pipe-thread"
>>>>"io.launcher"
>>>>"random.unix"
>>>>"bootstrap.random"
>>>>"io.sockets:ipv6-supported?"
>>>>"openssl"
>>>>"tools.crossref"
>>>>"cocoa"
>>>>"io.files.temp"
>>>>"tools.deprecation"
>>>>"core-foundation"
>>>>"vocabs.cache"
>>>>"vocabs.refresh.monitor"
>>>>"opengl.gl"
>>>>"opengl"
>>>>"ui"
>>>>"core-text.fonts"
>>>>"core-text"
>>>>"tools.errors.model"
>>>>"ui.tools.error-list"
>>>> }
>>>>
>>>> I'm sure a lot of that could be delayed in smart ways to make startup
>>>> time
>>>> faster, that's what I meant by low hanging fruit, but also not
>>>> something
>>>> that anyone is working on right now.
>>>>
>>>> Best,
>>>> John.
>>>>
>>>>> On Mon, Jan 30, 2017 at 12:40 PM,  wrote:
>>>>>
>>>>>> On 2017-01-30 18:14, Timothy Hobbs wrote:
>>>>>> If you are on linux, you might try this very new feature:
>>>>>> https://criu.org/Main_Page
>>>>>>
>>>>>>> On 01/30/2017 17:29, Jim Mack wrote:
>>>>>>> You could also set up factor as a local web server, and send bash
>>>>>>> requests to it through a web page, and it would be able to 
>>>>>>> accomplish
>>>>>>> them locally (or remotely) and report you the results.  It would 
>>>>>>> be a
>>>>>>> quick round-trip cycle with great feedback and the ability to 
>>>>>>> bookmark
>>>>>>> commands.
>>>>>>>
>>>>>>> On Sun, Jan 29, 2017 at 1:39 PM, >>>>>> <mailto:pet...@riseup.net>> wrote:
>>>>>>>
>>>>>>>>On 2017-01-29 16:20, John Benediktsson wrote:
>>>>>>>> It would be nice to improve startup time. I imagine some low
>>>>>>> hanging
>>>>>>>> fruits if we looked deeply into it. I'll make a note to get that
>>>>>>>on my
>>>>>>>> list, unless someone looks into it sooner.
>>>>>>>>
>>>>>>>>
>>>>&g

Re: [Factor-talk] Startup time

[Timothy Hobbs]

If you are on linux, you might try this very new feature:
https://criu.org/Main_Page

On 01/30/2017 17:29, Jim Mack wrote:
> You could also set up factor as a local web server, and send bash
> requests to it through a web page, and it would be able to accomplish
> them locally (or remotely) and report you the results.  It would be a
> quick round-trip cycle with great feedback and the ability to bookmark
> commands.
>
> On Sun, Jan 29, 2017 at 1:39 PM,  > wrote:
>
> On 2017-01-29 16:20, John Benediktsson wrote:
> > It would be nice to improve startup time. I imagine some low hanging
> > fruits if we looked deeply into it. I'll make a note to get that
> on my
> > list, unless someone looks into it sooner.
> >
> >
> >
> >> On Jan 28, 2017, at 2:25 PM, pet...@riseup.net
>  wrote:
> >>
> >> I spend a lot of time writing small scripts, often in bash. That is
> >> becoming a more and more painful task as bash is a nice-enough
> >> language
> >> on the first look but in the end it seems to go out of it's way to
> >> trip
> >> you up with every character you type. I don't mean to rant, I know
> >> it's
> >> an old language that has to keep a lot of backward
> compatibility and
> >> it
> >> actually still serves very well for what it was designed for,
> as long
> >> as
> >> your script stays <100 lines. Still, there's a lot to be desired.
> >>
> >> Factor is a cool language, it's very expressive, mature, has a
> lot of
> >> libraries and has all sorts of tricks up its sleaves to bend it to
> >> your
> >> will (much like lisp in that regard). It would be a fun
> experiment to
> >> write a library or EDSL (embedded DSL) for bash-like scripting.
> >> However
> >> there's a bone to be picked:
> >>
> >> $ time bash -c ''
> >> bash -c ''  0.00s user 0.00s system 94% cpu 0.004 total
> >>
> >> $ time factor-vm -e=''
> >> factor-vm -e=''  0.12s user 0.05s system 99% cpu 0.178 total
> >>
> >> I know one can create a custom image and maybe cut down on the
> startup
> >> a
> >> bit, but my question is - would it be possible to cut it down to
> >> bash's
> >> startup time *and* still have all the necessary vocabularies in
> it? I
> >> don't want to know the startup time with a small image that has
> like
> >> nothing in it, I can quickly test that myself. I'd need help to
> answer
> >> -
> >> if you imagine the use case I'm talking about, include all the
> vocabs
> >> that use case would need and make all other possible
> optimizations (if
> >> there are) without sacrificing too much, can the startup reach
> similar
> >> times?
> >>
> >> --
> >> 
> >>   Peter Nagy
> >> 
> >>
> >>
> 
> --
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> >> ___
> >> Factor-talk mailing list
> >> Factor-talk@lists.sourceforge.net
> 
> >> https://lists.sourceforge.net/lists/listinfo/factor-talk
> 
> >
> >
> 
> --
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > ___
> > Factor-talk mailing list
> > Factor-talk@lists.sourceforge.net
> 
> > https://lists.sourceforge.net/lists/listinfo/factor-talk
> 
>
> Hi John,
>
> you think you can reach similar startup times? That would be really
> cool. I guess your todo list is rather long though.
>
> As a side note, are there other concatenative languages you know I
> could
> look at?
>
> --
> 
>Peter Nagy
> 
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> 
> https://lists.sourceforge.net/lists/listinfo/factor-talk
> 
>
>
>
>
> --
> Check o

Re: [Factor-talk] maybe{

[Timothy Hobbs]

> Maybe (hahah) because it is part of the class algebra system and it is
> a complicated part of Factor. So no one has spent the time to
> understand it fully and write documentation for it.
You write as though GOD has left us with a book of Runes and no knows 
what they mean or do, and it is up to theologians to work out their 
meanings.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Updating factor

[Timothy Hobbs]

Hi,

I tried to package the factor IDE using Docker on linux. That's why the 
directory was read only. In order for me to be able to package the IDE 
successfully, I need to be able to get factor to work properly when its 
directory is read only.

Regards,

Tim


On 01/29/2017 10:55 AM, Alexander Ilin wrote:
> Hello, Tim!
>
> 29.01.2017, 12:35, "Timothy Hobbs" :
>> I tried to follow
>> http://docs.factorcode.org/content/article-first-program-start.html and
>>
>> "palindrome" scaffold-work
>>
>> Tries to write to the "factor-install-dir/work" directory. This failed,
>> because that directory was read only. Is work the only directory that
>> factor tries to write to? Can I change the path to the work directory?
>First of all, there are other scaffold-* words, which will write to other 
> respective directories. It is entirely up to you whether to use those words, 
> or not. By calling those words you are explicitly asking Factor to create new 
> files in its subfolders.
>
>Second, I'm sure there are some unit-tests that would create temporary 
> files. You don't have to run those, unless you make changes to the Factor 
> itself (if you do, you need full write access anyway).
>
>To have write permissions to the work folder, you could make it a symbolic 
> link to a location that you prefer, like something in your home directory.
>
>You could avoid using all of the above and create your own folder wherever 
> you want, add it to the global .factor-roots file, and put your new vocabs in 
> there instead of the work directory.
>
>http://docs.factorcode.org/content/article-vocabs.roots.html
>http://docs.factorcode.org/content/article-.factor-roots.html
>
>Question: why was the work directory read-only? What OS are you on?
>
> ---=---
>   Александр
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Updating factor

[Timothy Hobbs]

I tried to follow 
http://docs.factorcode.org/content/article-first-program-start.html and

"palindrome" scaffold-work

Tries to write to the "factor-install-dir/work" directory. This failed, 
because that directory was read only. Is work the only directory that 
factor tries to write to? Can I change the path to the work directory?

Tim


On 01/28/2017 11:43 PM, Alexander Ilin wrote:
> Hello, Tim!
>
> 28.01.2017, 21:55, "Timothy Hobbs" :
>> I noticed that factor likes to write to its own source directory, creating 
>> factor files, by default in ./factor/work.
>I can't say Factor does much writing to its source directories. What did 
> you do to make it?
>
>It can create some files in its root folder, next to the main executable, 
> if you do file output operations without changing the current directory, or 
> if you save the image file, but other than that I haven't noticed much 
> writing going on.
>
>> How does this work when you update factor?
>In the Factor distribution the work folder is empty, except for the 
> README.txt that states that it's the folder for your own projects. It 
> contains only things you put in there (using scaffold-work, for example), so 
> when you update Factor, its contents will be preserved.
>
> ---=---
>   Александр
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

[Factor-talk] Updating factor

[Timothy Hobbs]

I noticed that factor likes to write to its own source directory, 
creating factor files, by default in ./factor/work. How does this work 
when you update factor?

Do you have to merge the trees by hand? Or is ./factor/work somehow 
special and the only directory that gets commonly written to?

Tim


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

[Factor-talk] High DPI displays

[Timothy Hobbs]

Hi, I cannot see the text on my relatively new laptop, when I launch the 
factor IDE. Is there any option to make it larger?

Thanks,
Tim

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk