Re: SSD's and many edits of a single file

[Gene Heskett]

On Thursday 12 April 2018 10:50:07 songbird wrote:

> Gene Heskett wrote:
> ...
>
> > Lots of other stuff that would be TL;DR for most here.
>
>   i'm actually facinated by this level of knowledge and
> wish i had such toys around for milling some rocks into
> carvings so i wouldn't have to do it by hand...

Thats often best done with diamond saw blades, running them wet to 
control the heating that if not controlled, will ablate the diamond 
until its too dull to cut.

A wet tile saw, would be a decent start. And a thumbler for finish 
polish.

>   good fodder for daydreaming.  :)

Don't knock that, creativity, if not used, tends to go away. Even at my 
age I still have an occasional flash of an idea. My problem is in 
getting it committed to code, or at least a paper outline, before the 
candle flames meet in the middle. When that happens, its both 
frustrating, and time for what passes for a beer around my place. 
Because I'm also a DM-II, that reduces to miller 64. Select 55 has been 
thru the horse 2 or 3 more times, and comes in alu cans in these here 
parts. That takes it off the table as I haven't willingly drunk from an 
alu can in 40 years. The connection between the alu can and the 
alzheimers epidemic is way too strong for me.

>
>   songbird



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[songbird]

Gene Heskett wrote:
...
> Lots of other stuff that would be TL;DR for most here.

  i'm actually facinated by this level of knowledge and
wish i had such toys around for milling some rocks into
carvings so i wouldn't have to do it by hand...

  good fodder for daydreaming.  :)


  songbird

Re: SSD's and many edits of a single file

[rhkramer]

Thanks!


On Thursday, April 12, 2018 08:43:50 AM Gene Heskett wrote:
> On Thursday 12 April 2018 07:44:39 rhkra...@gmail.com wrote:
> > On 2nd reading, I think the answer is yes, i.e., if I have a midi
> > program making use of the / a real time kernel, it will have total
> > control of the machine and other programs will have the "leftovers"?
> 
> Yes. Particularly noticeable if running the steppers by software step
> generation. There, one must run a "base thread", limited to integer only
> math, in a 20 to 30 nanosecond loop. Because the loop is so slow, this
> function is often off-loaded to an fpga smart card that is programmed at
> a 1 to 10 kilohertz rate with most using the default 1 kilohertz.  This
> can quadruple the speeds, from 2 effects. Because the motors see a much
> steadier heartbeat, they can often be moved much faster without
> stalling, and 2, this hardware can often generate the step signals 10x
> faster.
> 
> OTOH, doing that also frees up the parent cpu, allowing it to do much
> more without affecting the main process.
> 
> We can also setup slower "threads" for stuff that are the result of human
> activity for instance. Humans are rather glacial movers.
> 
> For less critical duties this works well, and in the case of running
> bigger machinery with something as puny-powered as an r-pi-3b, I have
> much of the not so time critical stuff running in a 200 hz thread, so
> that nearly half of the hal file only gets processed when the higher
> priority, faster loop is done. This includes the hand driven encoder
> dials that allow me to run the lathe by hand. Even at 200 hz I can still
> drive it at anything from the maximum speed of nearly 90 inches a
> minute, down to as small as .0001" per click of the dial. Thats a larger
> movement than the software itself can do since if I tell it to move 12"
> along the bed, while moving the cross-feed 3 thousandths of an inch, it
> will do what is effectively a dead straight line from that start to that
> finish. Errors in that motion are generally the coil current mappings of
> the stepper driver as it divides a full step, 1/200th of a turn, by 8 or
> 16 in order to get that smoother motion. Loading of course inserts an
> offset, but is under a .9 degree of the motor shaft rotation as long as
> the load doesn't cause a step slip. That much may break the micron
> claim, but not by enough to ruin the part.
> 
> Lots of other stuff that would be TL;DR for most here.

Re: SSD's and many edits of a single file

[Gene Heskett]

On Thursday 12 April 2018 07:44:39 rhkra...@gmail.com wrote:

> On Wednesday, April 11, 2018 03:27:48 AM Gene Heskett wrote:
> > What seems to be lost on you folks not running realtime patched
> > kernels, is that when linuxcnc is running, it has total control over
> > the hardware, and that linux becomes a client of the realtime
> > system, getting what cpu time is left after linuxcnc has finished
> > what it has to do to meet the timing constraints needed to run the
> > machine and meet the sub-micron positioning accuracy required.
>
> Hmm, a more careful 2nd reading may have answered my question, but
> I'll ask anyway--on first reading I wondered if that was true of any
> program running in / making use of a Linux realtime kernel--that is,
> that the realtime program (linuxcnc) (or is that the kernel) has total
> control over the  machine.
>
> On 2nd reading, I think the answer is yes, i.e., if I have a midi
> program making use of the / a real time kernel, it will have total
> control of the machine and other programs will have the "leftovers"?

Yes. Particularly noticeable if running the steppers by software step 
generation. There, one must run a "base thread", limited to integer only 
math, in a 20 to 30 nanosecond loop. Because the loop is so slow, this 
function is often off-loaded to an fpga smart card that is programmed at 
a 1 to 10 kilohertz rate with most using the default 1 kilohertz.  This 
can quadruple the speeds, from 2 effects. Because the motors see a much 
steadier heartbeat, they can often be moved much faster without 
stalling, and 2, this hardware can often generate the step signals 10x 
faster.

OTOH, doing that also frees up the parent cpu, allowing it to do much 
more without affecting the main process.

We can also setup slower "threads" for stuff that are the result of human 
activity for instance. Humans are rather glacial movers.
 
For less critical duties this works well, and in the case of running 
bigger machinery with something as puny-powered as an r-pi-3b, I have 
much of the not so time critical stuff running in a 200 hz thread, so 
that nearly half of the hal file only gets processed when the higher 
priority, faster loop is done. This includes the hand driven encoder 
dials that allow me to run the lathe by hand. Even at 200 hz I can still 
drive it at anything from the maximum speed of nearly 90 inches a 
minute, down to as small as .0001" per click of the dial. Thats a larger 
movement than the software itself can do since if I tell it to move 12" 
along the bed, while moving the cross-feed 3 thousandths of an inch, it 
will do what is effectively a dead straight line from that start to that 
finish. Errors in that motion are generally the coil current mappings of 
the stepper driver as it divides a full step, 1/200th of a turn, by 8 or 
16 in order to get that smoother motion. Loading of course inserts an 
offset, but is under a .9 degree of the motor shaft rotation as long as 
the load doesn't cause a step slip. That much may break the micron 
claim, but not by enough to ruin the part.

Lots of other stuff that would be TL;DR for most here.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[rhkramer]

On Wednesday, April 11, 2018 03:27:48 AM Gene Heskett wrote:
> What seems to be lost on you folks not running realtime patched kernels,
> is that when linuxcnc is running, it has total control over the
> hardware, and that linux becomes a client of the realtime system,
> getting what cpu time is left after linuxcnc has finished what it has to
> do to meet the timing constraints needed to run the machine and meet the
> sub-micron positioning accuracy required.

Hmm, a more careful 2nd reading may have answered my question, but I'll ask 
anyway--on first reading I wondered if that was true of any program running in 
/ making use of a Linux realtime kernel--that is, that the realtime program 
(linuxcnc) (or is that the kernel) has total control over the  machine.

On 2nd reading, I think the answer is yes, i.e., if I have a midi program 
making use of the / a real time kernel, it will have total control of the 
machine and other programs will have the "leftovers"?

Re: SSD's and many edits of a single file

[David Christensen]

On 04/11/18 00:27, Gene Heskett wrote:

On Tuesday 10 April 2018 22:24:39 David Christensen wrote:

Can you reproduce the bugs on a machine with straight-up Debian (e.g.
no linuxcnc)?


That to me would be rather pointless. The only places where I be doing
wholesale edits would be for linuxcnc use.


If geany and gedit fail on a machine with Debian plus LinuxCNC, but work 
correctly on a machine with Debian alone, that would indicate the bug is 
related to LinuxCNC.




As a safety net, I would suggest a versioning file system (e.g.
similar to the versioning feature of VMS).  STFW is pretty thin, but
copyfs might be worth a try:


This "fuse" would be on top of ext4? Or on an SSD that was a copy of this
one but using fuse as opposed to ext4?


Here is information about FUSE:

https://en.wikipedia.org/wiki/Filesystem_in_Userspace


See link below for information about copyfs.



What seems to be lost on you folks not running realtime patched kernels,
is that when linuxcnc is running, it has total control over the
hardware, and that linux becomes a client of the realtime system,
getting what cpu time is left after linuxcnc has finished what it has to
do to meet the timing constraints needed to run the machine and meet the
sub-micron positioning accuracy required.

Not your fault of course because to you its effectively real time if you
do not see a lag between touching a key and the response on screen. But
a 20ns response time is about 3 orders of magnitude faster than that
which is realtime to you.


I used to work as a embedded systems software engineer back in the 
dot-com days.  My niche was controlling electro-mechanical systems using 
8051, 68xx, and 68xxx micro-controllers.  I have written C and assembly 
language drivers and programs to meet soft and hard real-time 
requirements, including stepper motor sequencing.  I had heard of hard 
real-time kernels that used Linux as their idle loop.  I have since used 
Linux real-time patches for my audio and music hobbies.




https://packages.debian.org/search?keywords=copyfs=names
e=all=all


This link shows 2 versions, but only the older, 1.4 might be usable since
the newest system I'm running is jessie on an r-pi-3b. Stretch so far is
not stable on arm64, aka a rock64 yet. Even running an amanda client on
the arm64 can take it down. Downgrade its install to a jessie derived
version and its dead stable. And amanda is dead stable on 6 of the 7
systems with a net cable plugged into it here.

But this link does not contain a sales pitch describing what it can do
and why it should be used, 


Here is information about copyfs:

https://boklm.eu/copyfs/



I think that by changing how I work, so that an edit is done with a fresh
invocation of the editor each time, that I will not see this problem
again, so we should really put this thread to bed.


I used VMS systems back in the 1980's, and VMS's versioning file system 
was a killer feature that I would love to have on my Linux machines today:


https://mike632t.wordpress.com/2016/03/15/vms-file-operations-for-beginners/


Unfortunately, the semantics of copyfs appear to differ.



"Magic Twanger" and Froggie indeed, that kiddie tv show is now 68 years
old. I guess that does date me. 99% of the readers of this message have
no clue where that phrase came from.


STFW I see Andy's Gang (but it's before my time):

https://www.youtube.com/watch?v=G6a3fck0NBI



Thanks David.


Thanks for the interesting problem to think about.  :-)


David

Re: SSD's and many edits of a single file

[Gene Heskett]

On Tuesday 10 April 2018 22:24:39 David Christensen wrote:

> On 04/09/18 20:58, Gene Heskett wrote:
> > On Monday 09 April 2018 21:25:34 David Christensen wrote:
> >> On 04/09/18 07:30, Gene Heskett wrote:
> >>> On Monday 09 April 2018 09:51:37 Greg Wooledge wrote:
>  On Mon, Apr 09, 2018 at 09:46:07AM -0400, rhkra...@gmail.com
> 
>  wrote:
> > To your original problem, have you tried going to a command line
> > and throwing in a couple =sync=s?  I would try that, maybe after
> > saving in your editor, and again maybe after open and / or
> > saving in the cnc program.
> 
>  As others have explained, the OS (Linux) keeps a cache of file
>  contents that have been written by applications, but not yet
>  committed to permanent storage.  If you "save" from within the
>  text editor, then the saved contents should be immediately
>  visible to other processes reading the file, regardless of
>  whether it has been synced to disk. They'll simply get the cached
>  version.
> >>>
> >>> Which is not happening after several hours and a hundred or more
> >>> edits. Which is why its so intermittent.
> >>
> >> On 04/09/18 02:53, Gene Heskett wrote:
> >>> Lots of people seem to like gedit, but its saves are the cause of
> >>> important configuration files being written back to disk with the
> >>> line order totally trashed, as if you had thrown it on the floor
> >>> in 512 byte pieces, then picked it back up and reassembled it in
> >>> random order. Then try to recover a 1400 LOC configuration file...
> >>>
> >>> Thats happened using gedit enough, on several different machines
> >>> ...
> >>
> >> One editor failing would also make me suspect the editor.
> >>
> >>
> >> But two failing editors would make me suspect some common factor,
> >> such as a shared library and/or the kernel.
> >
> > I'd buy that if the failures were similar.  They are not.
>
> They both seem to involve getting blocks from an app to memory and
> blocks from memory to disk (and/or disk to memory).  They could be
> variations on a theme.
>
>
> Can you reproduce the bugs on a machine with straight-up Debian (e.g.
> no linuxcnc)?
>
That to me would be rather pointless. The only places where I be doing 
wholesale edits would be for linuxcnc use.
>
> As a safety net, I would suggest a versioning file system (e.g.
> similar to the versioning feature of VMS).  STFW is pretty thin, but
> copyfs might be worth a try:

This "fuse" would be on top of ext4? Or on an SSD that was a copy of this 
one but using fuse as opposed to ext4?

What seems to be lost on you folks not running realtime patched kernels, 
is that when linuxcnc is running, it has total control over the 
hardware, and that linux becomes a client of the realtime system, 
getting what cpu time is left after linuxcnc has finished what it has to 
do to meet the timing constraints needed to run the machine and meet the 
sub-micron positioning accuracy required.

Not your fault of course because to you its effectively real time if you 
do not see a lag between touching a key and the response on screen. But 
a 20ns response time is about 3 orders of magnitude faster than that 
which is realtime to you.

> https://packages.debian.org/search?keywords=copyfs=names
>e=all=all

This link shows 2 versions, but only the older, 1.4 might be usable since 
the newest system I'm running is jessie on an r-pi-3b. Stretch so far is 
not stable on arm64, aka a rock64 yet. Even running an amanda client on 
the arm64 can take it down. Downgrade its install to a jessie derived 
version and its dead stable. And amanda is dead stable on 6 of the 7 
systems with a net cable plugged into it here. 

But this link does not contain a sales pitch describing what it can do 
and why it should be used, a common fault, I think because git doesn't 
seem to have a "box" category to put this stuff into for ready 
presentation when browsing around looking for a solution to a problem. 
This lack of a descriptive sales pitch "box" for this info is, IMNSHO, a 
major failing of the linux genre. There may well be a magic twanger or 8 
out there, but searching the repo's is less than useful in many cases.

All this of course has drifted off-topic, debian moves a bit slow and 
does not actually support the arm64 yet. That I understand is a lack of 
manpower problem. Were I 40 years younger, I might be able to help a wee 
bit. But I'm not, and having had a pulmonary embolism that damned near 
put a ~30~ on my story 3 years back, I can feel the wet ram damage that 
caused.

I think that by changing how I work, so that an edit is done with a fresh 
invocation of the editor each time, that I will not see this problem 
again, so we should really put this thread to bed.

"Magic Twanger" and Froggie indeed, that kiddie tv show is now 68 years 
old. I guess that does date me. 99% of the readers of this message have 
no clue where that phrase came from.

> David 

Thanks David.

-- 
Cheers, 

Re: SSD's and many edits of a single file

[David Christensen]

On 04/09/18 20:58, Gene Heskett wrote:

On Monday 09 April 2018 21:25:34 David Christensen wrote:


On 04/09/18 07:30, Gene Heskett wrote:

On Monday 09 April 2018 09:51:37 Greg Wooledge wrote:

On Mon, Apr 09, 2018 at 09:46:07AM -0400, rhkra...@gmail.com

wrote:

To your original problem, have you tried going to a command line
and throwing in a couple =sync=s?  I would try that, maybe after
saving in your editor, and again maybe after open and / or saving
in the cnc program.


As others have explained, the OS (Linux) keeps a cache of file
contents that have been written by applications, but not yet
committed to permanent storage.  If you "save" from within the text
editor, then the saved contents should be immediately visible to
other processes reading the file, regardless of whether it has been
synced to disk. They'll simply get the cached version.


Which is not happening after several hours and a hundred or more
edits. Which is why its so intermittent.


On 04/09/18 02:53, Gene Heskett wrote:

Lots of people seem to like gedit, but its saves are the cause of
important configuration files being written back to disk with the
line order totally trashed, as if you had thrown it on the floor in
512 byte pieces, then picked it back up and reassembled it in random
order. Then try to recover a 1400 LOC configuration file...

Thats happened using gedit enough, on several different machines ...


One editor failing would also make me suspect the editor.


But two failing editors would make me suspect some common factor, such
as a shared library and/or the kernel.


I'd buy that if the failures were similar.  They are not.


They both seem to involve getting blocks from an app to memory and 
blocks from memory to disk (and/or disk to memory).  They could be 
variations on a theme.



Can you reproduce the bugs on a machine with straight-up Debian (e.g. no 
linuxcnc)?



As a safety net, I would suggest a versioning file system (e.g. similar 
to the versioning feature of VMS).  STFW is pretty thin, but copyfs 
might be worth a try:


https://packages.debian.org/search?keywords=copyfs=names=all=all


David

Re: SSD's and many edits of a single file

[Celejar]

On Mon, 9 Apr 2018 05:53:18 -0400
Gene Heskett  wrote:

...

> What else is a good editor? And lets not start yet another vim vs emacs 
> war. kwrite and kate come to mind. I'm running Trinity Desktop, maybe I 
> should check them out. Whatever, needs to be able to have several files 
> open so I can just click back and forth on the tabs as I usually 
> separate the files into what uses which cutting tool.

I use bluefish (and sometimes geany). Bluefish is probably more
complicated and heavier than you need, but you can take a look for
yourself.

Celejar

Re: SSD's and many edits of a single file

[Thomas Schmitt]

Hi,

Stefan Monnier wrote:
> I see no indication *at all* from Gene's description that the problem
> could have anything to do with the OS or the hardware.

I don't think so either. But since the discussion began to cycle,
i thought that it would be nice to have some tangible proof.

If the script runs fine for half a day while Gene does his editing and
CNCing and maybe experiences the problem with geany again, then we can
say that it's not the SSD or the kernel.


> My guess is rather that his editor ends up saving the file at location
> A and he looks at the state of a file at location B.

Yes. But who would want to explore the habits of a Gnome application ?
  https://github.com/geany/geany/issues/1774
  https://wiki.geany.org/config/all_you_never_wanted_to_know_about_file_saving


Have a nice day :)

Thomas

Re: SSD's and many edits of a single file

[Stefan Monnier]

> Following is a bash script which challenges the disk's fidelity in
> respect to small writes on a relatively small file over a long time.

I see no indication *at all* from Gene's description that the problem
could have anything to do with the OS or the hardware.

My guess is rather that his editor ends up saving the file at location
A and he looks at the state of a file at location B.


Stefan

Re: SSD's and many edits of a single file

[Thomas Schmitt]

Hi,

re-reading the original mail of Gene
  https://lists.debian.org/debian-user/2018/04/msg00237.html
i think it is time to collect some evidence.

Following is a bash script which challenges the disk's fidelity in
respect to small writes on a relatively small file over a long time.

Gene, please consider to run it and to report the outcome

Adjustable parameters are
  file_path , chunk_size , sleep_time , duration
Edit the script to change their values. No blanks or fancy characters.
(It is advisable to store it on an unsuspicious disk.)

With the given setting of chunk_size=8, the victim file will have a size
of 524,289 bytes.

The script will print a single dot to stdout for each successful write-read
cycle.  After 60 such dots, the cycle count and a newline character get
printed.

In case of a failed read-write cycle, an ALL-CAPS message will be emitted
to stderr and the script will abort immediately.

If there is no failure, then operating system and disk are working as
they should.

Start of script. Cut out after the "---" line, store as file
"random_writer.sh", and give execution permission by
  chmod u+x random_writer.sh
---
#!/bin/bash

# Which file to use
file_path=/home/test/random_writer_victim

# How many bytes to write in one try.
# (Write range is 65536 times chunk_size plus 1)
chunk_size=8

# Interval between writes
sleep_time=10

# Overall duration in seconds (here: half a day)
duration=43200

# Initialize file content by newline characters
echo "Victim file: $file_path"
length=$(expr 65536 '*' $chunk_size + 1)
yes "" | head -c $length | dd of="$file_path" conv=notrunc 2>/dev/null

start_time=$(date '+%s')
cycle=0
while test $(expr $(date '+%s') - $start_time) -lt $duration
do
  # Get a random write address in the range of 0 to 65535
  random_adr=$(dd if=/dev/urandom bs=1 count=2 2>/dev/null | od -d | \
   head -1 | awk '{print $2}' )
  cycle=$(expr $cycle + 1)
  text=$(printf '%'"$chunk_size"'d' $cycle)

  # Write to file (with trailing newline to keep the file editor friendly)
  echo "$text" | \
dd of="$file_path" seek="$random_adr" bs=$chunk_size count=1 conv=notrunc \
   2>/dev/null

  # Check-read
  reread=$(dd if="$file_path" skip="$random_adr" bs=$chunk_size count=1 \
  2>/dev/null)

  # <<< Simulated mismatch
  # test $cycle = 3 && reread=X_simulated_X

  if test "$text" != "$reread"
  then
echo
echo "CAUGHT MISMATCH OF WRITE AND READ !" >&2
echo "Written: '$text'" >&2
echo "Read   : '$reread'" >&2
exit 1
  fi 

  echo -n '.'
  if test $(expr $cycle % 60) = 0
  then
printf '%8d\n' $cycle
  fi
  sleep $sleep_time || break
done
printf '%8d\nDone. No errors detected.\n' $cycle
---
End of script. Cut out before the "---" line.

Have a nice day :)

Thomas

Re: SSD's and many edits of a single file

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, Apr 10, 2018 at 08:10:07AM -0400, Gene Heskett wrote:
> On Tuesday 10 April 2018 07:08:03 to...@tuxteam.de wrote:
> 
> > On Tue, Apr 10, 2018 at 06:49:54AM -0400, Gene Heskett wrote:

[...]

> No such clues were evident.

Then I'd look for troubles elsewhere...

> What good does it do to partition an SSD? Because of the wear leveling, 
> if it goes read-only, I expect the whole thing is read-only.  Including 
> swap I expect.

Hm. sooner or later, yes. But the OS takes its decisions for each file
system separately.

That said, I wouldn't partition an SSD without a compelling reason.

> I rest my case with this thread [...]

> Haveing fresh input from different angles helps my ancient wet ram, 
> thanks Thomas.

That's what wet RAM thrives on, I think that is valid for everyone.
So -- glad to help, and to receive help :)

Cheers
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlrMuDsACgkQBcgs9XrR2kYHfwCfaBahrZwupJwU69uDR4UKJv0U
1fIAn0nN4n+3Ye1QWQdW4+SudpCDa2hc
=2EJq
-END PGP SIGNATURE-

Re: SSD's and many edits of a single file

[Gene Heskett]

On Tuesday 10 April 2018 07:08:03 to...@tuxteam.de wrote:

> On Tue, Apr 10, 2018 at 06:49:54AM -0400, Gene Heskett wrote:
> > On Tuesday 10 April 2018 03:52:55 to...@tuxteam.de wrote:
>
> [...]
>
> > > The most typical symptom [of a failed write due to storage
> > > failure] would be that the file system gets remounted
> > > read-only because of write errors. And you tend to notice that.
> >
> > Does the mount command report that? It did not.
>
> If your logs are writable (because they are on a separate file
> system, or because they are remote),

They are neither.

> you'd see that there. If 
> not (because you have only one file system), you're bound to
> notice that anyway, because many applications are going to freak
> out on a failed write (*all* writes fail once the file system
> is read-only).

No such clues were evident.

> Cheers
> -- tomás

What good does it do to partition an SSD? Because of the wear leveling, 
if it goes read-only, I expect the whole thing is read-only.  Including 
swap I expect.

I rest my case with this thread, geany apparently is not taking care of 
its housekeeping and is running out of whatever is allocated to do the 
disk write, its the only conclusion that makes sense so far. There's a 
name for it, but oldtimers is at work here too. Dammit.

I'll change my default lcnc editor to nano --smooth, which gets rid of my 
scrolling complaint.  And see if it happens again. That will also tend 
to prevent my leaving it open forever if launched from lcnc as thats 
single-threaded. Either editor launched as a fresh action will probably 
solve the problem.

Haveing fresh input from different angles helps my ancient wet ram, 
thanks Thomas.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, Apr 10, 2018 at 06:49:54AM -0400, Gene Heskett wrote:
> On Tuesday 10 April 2018 03:52:55 to...@tuxteam.de wrote:

[...]

> > The most typical symptom [of a failed write due to storage
> > failure] would be that the file system gets remounted
> > read-only because of write errors. And you tend to notice that.

> Does the mount command report that? It did not.

If your logs are writable (because they are on a separate file
system, or because they are remote), you'd see that there. If
not (because you have only one file system), you're bound to
notice that anyway, because many applications are going to freak
out on a failed write (*all* writes fail once the file system
is read-only).

Cheers
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlrMmxMACgkQBcgs9XrR2kbtawCfRvDu3yYCEKvA69Nt+SXTW30h
dvYAnRqoB8pVenX4X8esrPavYDXstul5
=riS/
-END PGP SIGNATURE-

Re: SSD's and many edits of a single file

[Gene Heskett]

On Tuesday 10 April 2018 03:52:55 to...@tuxteam.de wrote:

> On Mon, Apr 09, 2018 at 06:25:34PM -0700, David Christensen wrote:
>
> [...]
>
> > One editor failing would also make me suspect the editor.
> >
> >
> > But two failing editors would make me suspect some common factor,
> > such as a shared library and/or the kernel.
> >
> >
> > Have you tested your hardware -- power supply, memory, and SSD?  Be
> > sure to test the SSD before you touch any cables.  If it fails,
> > re-seat and/or replace cables and test again.
>
> Folks: you're barking up the wrong tree. *If* some editor fails
> to write to a file because the hard disk goes on strike, chances
> are that you'll hear your OS *SCREAM LOUDLY* that something is
> wrong.
>
> The most typical symptom would be that the file system gets remounted
> read-only because of write errors. And you tend to notice that.
>
Does the mount command report that? It did not.

> Cheers
> -- tomás



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Apr 09, 2018 at 06:25:34PM -0700, David Christensen wrote:

[...]

> One editor failing would also make me suspect the editor.
> 
> 
> But two failing editors would make me suspect some common factor,
> such as a shared library and/or the kernel.
> 
> 
> Have you tested your hardware -- power supply, memory, and SSD?  Be
> sure to test the SSD before you touch any cables.  If it fails,
> re-seat and/or replace cables and test again.

Folks: you're barking up the wrong tree. *If* some editor fails
to write to a file because the hard disk goes on strike, chances
are that you'll hear your OS *SCREAM LOUDLY* that something is
wrong.

The most typical symptom would be that the file system gets remounted
read-only because of write errors. And you tend to notice that.

Cheers
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlrMbVcACgkQBcgs9XrR2kYDHQCbBbWJ05ZsCKnzao3k/F5/6VQG
Yd4An0o+M9R6tH8AOX0Uas7JvhwF0x9N
=Amez
-END PGP SIGNATURE-

Re: SSD's and many edits of a single file

[Gene Heskett]

On Monday 09 April 2018 21:25:34 David Christensen wrote:

> On 04/09/18 07:30, Gene Heskett wrote:
> > On Monday 09 April 2018 09:51:37 Greg Wooledge wrote:
> >> On Mon, Apr 09, 2018 at 09:46:07AM -0400, rhkra...@gmail.com
> >>
> >> wrote:
> >>> To your original problem, have you tried going to a command line
> >>> and throwing in a couple =sync=s?  I would try that, maybe after
> >>> saving in your editor, and again maybe after open and / or saving
> >>> in the cnc program.
> >>
> >> As others have explained, the OS (Linux) keeps a cache of file
> >> contents that have been written by applications, but not yet
> >> committed to permanent storage.  If you "save" from within the text
> >> editor, then the saved contents should be immediately visible to
> >> other processes reading the file, regardless of whether it has been
> >> synced to disk. They'll simply get the cached version.
> >
> > Which is not happening after several hours and a hundred or more
> > edits. Which is why its so intermittent.
>
> On 04/09/18 02:53, Gene Heskett wrote:
> > Lots of people seem to like gedit, but its saves are the cause of
> > important configuration files being written back to disk with the
> > line order totally trashed, as if you had thrown it on the floor in
> > 512 byte pieces, then picked it back up and reassembled it in random
> > order. Then try to recover a 1400 LOC configuration file...
> >
> > Thats happened using gedit enough, on several different machines ...
>
> One editor failing would also make me suspect the editor.
>
>
> But two failing editors would make me suspect some common factor, such
> as a shared library and/or the kernel.
>
I'd buy that if the failures were similar.  They are not.
>
> Have you tested your hardware -- power supply, memory, and SSD?  Be
> sure to test the SSD before you touch any cables.  If it fails,
> re-seat and/or replace cables and test again.

I saw this once when the drive was a 2T piece of spinning rust. Way 
overkill for that machine, but a 1T is being used for amanda, here on 
this machine and its around 90% right now.  So at some 
point "/amandatapes" is going to find another terabyte.
>
> Have you tested your SSD hypothesis?  Say, by removing the SSD,
> cloning the SSD to another device, installing the other device, and
> then testing for the bug while running the other device?

I have a twin to that one, but only one handy sata port. So it would be 
an extended project to clone it to the other drive.  And since I saw it 
on spinning rust too, I'm not inclined to sharpen that finger just yet.

That machine has also ran about 7 cycles of memtest86 with no errors 
since I noticed it the first time.

> Can you reproduce the bug on hardware with ECC memory with no memory
> errors reported?

I don't have anything with ECC memory in it. That usually runs the mobo 
cost up quite a ways.

> Can you reproduce the bug on RAID1 with no RAID errors reported?

That would also demand another handy sata port.  I'll see if I can cobble 
something up if and when it does it again. Possible if it has a back 
panel sata connector I think.
>
> If you want to make it possible for others to find the bug, I would
> suggest:
>
> 1.  Build a machine with the minimum hardware, software, and
> configuration required to demonstrate the bug.  Document everything.
>
> 2.  Write a program or script that invokes the bug every time it runs
> on the demonstration machine.  If a program, include a Makefile.
>
> 3.  Post the demonstrator document, program, script, Makefile, etc.,
> to the relevant support communities.
>
> David

Thanks David.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[David Christensen]

On 04/09/18 07:30, Gene Heskett wrote:

On Monday 09 April 2018 09:51:37 Greg Wooledge wrote:


On Mon, Apr 09, 2018 at 09:46:07AM -0400, rhkra...@gmail.com
wrote:

To your original problem, have you tried going to a command line
and throwing in a couple =sync=s?  I would try that, maybe after
saving in your editor, and again maybe after open and / or saving
in the cnc program.


As others have explained, the OS (Linux) keeps a cache of file 
contents that have been written by applications, but not yet

committed to permanent storage.  If you "save" from within the text
editor, then the saved contents should be immediately visible to
other processes reading the file, regardless of whether it has been
synced to disk. They'll simply get the cached version.


Which is not happening after several hours and a hundred or more
edits. Which is why its so intermittent.


On 04/09/18 02:53, Gene Heskett wrote:

Lots of people seem to like gedit, but its saves are the cause of
important configuration files being written back to disk with the
line order totally trashed, as if you had thrown it on the floor in
512 byte pieces, then picked it back up and reassembled it in random
order. Then try to recover a 1400 LOC configuration file...

Thats happened using gedit enough, on several different machines ...


One editor failing would also make me suspect the editor.


But two failing editors would make me suspect some common factor, such 
as a shared library and/or the kernel.



Have you tested your hardware -- power supply, memory, and SSD?  Be sure 
to test the SSD before you touch any cables.  If it fails, re-seat 
and/or replace cables and test again.



Have you tested your SSD hypothesis?  Say, by removing the SSD, cloning 
the SSD to another device, installing the other device, and then testing 
for the bug while running the other device?



Can you reproduce the bug on hardware with ECC memory with no memory 
errors reported?



Can you reproduce the bug on RAID1 with no RAID errors reported?


If you want to make it possible for others to find the bug, I would suggest:

1.  Build a machine with the minimum hardware, software, and 
configuration required to demonstrate the bug.  Document everything.


2.  Write a program or script that invokes the bug every time it runs on 
the demonstration machine.  If a program, include a Makefile.


3.  Post the demonstrator document, program, script, Makefile, etc., to 
the relevant support communities.



David

Re: SSD's and many edits of a single file

[songbird]

Gene Heskett wrote:
> On Monday 09 April 2018 06:18:18 songbird wrote:
>> Gene Heskett wrote:
>> > Greetings folks;
>> >
>> > Updodate Wheezy, realtime kernel because machine is running
>> > linuxcnc.
>> >
>> > Editor is geany and file is left open in the editor, and reloaded
>> > into linuxcnc as changes are made to the file, and saved but not
>> > closed.
>> >
>> > And eventually the updates made to the file are not actually saved,
>> > geany does not report an error and does say success by changing the
>> > filename font from red which tells me its been modified, back to
>> > black in the window top border, but does not actually update the
>> > file on the disk, and the only fix seems to be a reboot.
>> >
>> > Is this a known fault with some SSD's? Or???
>>
>>   i'm not familiar with geany or linuxcnc...  any options
>> in geany for save interval?
>
> I think 2 minutes.

  seems that isn't working, either a bug or you
have more than one copy of the same file open and
one hasn't changed and the other has?  or something
like that...


>>   what kind of file system? 
>
> ext4 w/journaling on.

  ok, so somewhat sane fs.  except you are using some
different hardware and perhaps that implementation
doesn't include all the fs options you think it may.
could be perhaps even ignoring some so...  ugh...
hard to debug.


>>   what are the mount options 
>
> defaults
>
>> in fstab?
> UUID=54273d27-b48f-4a73-9b4d-d3cd13c6ba79 / ext4 
> errors=remount-ro 01
>
>>type of SSD?
> kernel: [3.648318] scsi 0:0:0:0: Direct-Access ATA  SPCC 
> Solid State V3.7 PQ: 0 ANSI: 5
>
> Cheap on sale special from newegg. Lots quicker than spinning rust. 
> Currently 28% used ack df.

  yes, i'm on SSD finally too, love it so far.  by
the time i need to add more i should be able to get
a few TB for the same price of this one.


> Thanks songbird.

  curious though why no vi/vim.  :)


>>   a sync once in a while like other poster recommended
>> may be a help as it would also clear out any waiting io
>> but i'm not sure how much that may interfere with the
>> realtime aspects if you have some large chunks waiting.
>>
> I can only guess.  These files are generally not long, 100 lines or so if 
> I write it but if generated by a code generator, which has no knowledge 
> of the looping abilities of linuxcnc's RS-274-D translator, could make a 
> 40 meg file out of what should be less than 100 properly constructed 
> lines of gcode.

  no of course not, you'd not want to see flattened out 
loops like that!  :)

  for that small of files you are editing, a sync should
mean nothing to the system.  the other issue though is
how the system deals with cached io pages and that was
all i was concerned about if there was something else
going on that would have large io blocks waiting to get
written to disk.  doesn't sound like it to me.


>>   i've used ionice for some things to make sure they
>> don't interfere on an older system with limited
>> capacity.
>
> The heart of linuxcnc runs on one core of a multicore cpu, and something 
> like ionice would probably get locked out of that playpen and have no 
> effect. Any effect would probably encourage it to wreck an otherwise 
> purty part. It, LinuxCNC, supplies the motor signals to drive the 
> machine to sub-micron accuracy, but to do that it has to be able to 
> slice time into nanoseconds.

  sound like fun, but i'd be a bit worried to have
a machine that large driven by a relatively unshielded
pc of any kind.

  do they even have ECC memory in them?  eek!
shielded from cosmic rays?


>> my current machine is a screamer in comparison 
>> so i've not had to do that.
>
> Most of this stuff isn't the latest and greatest.  Old Dell's, or Atom 
> powered shoeboxes even. What matters is how steady a heartbeat it can 
> maintain when a beat of the heart might be needed every 25 nanoseconds, 
> and the atoms are the fastest kid in town for that. The biggest machine 
> is a 1500 lb lathe, and its being run by an r-pi-3b using mesa 
> interfaceing cards. Might be the only one on the planet.


> Thanks songbird.

  y.w. it does all sound like fun, but a bit too high
on the risk scale for me.  where something wrong might
involve spinning metal, i'd want to be a long ways away.
had someone behind me get hit by an exploding wood bowl
once in jr. high shop class.  not a fun time...


  songbird

Re: SSD's and many edits of a single file

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Apr 09, 2018 at 10:13:21AM -0400, Gene Heskett wrote:
> On Monday 09 April 2018 09:46:07 rhkra...@gmail.com wrote:

[...]

> > To your original problem, have you tried going to a command line and
> > throwing in a couple =sync=s?  I would try that, maybe after saving in
> > your editor, and again maybe after open and / or saving in the cnc
> > program.
>
> I wasn't schmardt enough to think of that before I rebooted it from here 
> in the house last night. I shoulda.

Rebooting should do a sync as well (in fact, any attempt at a clean
unmount). So this shouldn't be the problem either.

Cheers
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlrLlIMACgkQBcgs9XrR2kba5ACeMgPWZHAjWd3fDbIvS+OuHdXL
BvwAn3XSCiA6be2jxmzpf2mYN6wnyuPt
=btYw
-END PGP SIGNATURE-

Re: SSD's and many edits of a single file

[Greg Wooledge]

On Mon, Apr 09, 2018 at 10:30:58AM -0400, Gene Heskett wrote:
> On Monday 09 April 2018 09:51:37 Greg Wooledge wrote:
> > As others have explained, the OS (Linux) keeps a cache of file
> > contents that have been written by applications, but not yet committed
> > to permanent storage.  If you "save" from within the text editor, then
> > the saved contents should be immediately visible to other processes
> > reading the file, regardless of whether it has been synced to disk. 
> > They'll simply get the cached version.
> 
> Which is not happening after several hours and a hundred or more edits. 
> Which is why its so intermittent.

This is why some people are guessing that the problem lies in the
text editor -- it hasn't *actually* saved the content (via kernel
write() calls) yet.

Of course there are many other possible explanations.  You might have
thought you saved, but you actually didn't (classic PEBKAC).  You might
be catting the wrong file, different from the one that's being edited.
The editor might have been invoked to edit a temporary file within some
kind of version control environment, with your changes not actually
being submitted to the VCS until the editor terminates.

It's unlikely to be an issue with the actual storage hardware.

Re: SSD's and many edits of a single file

[Gene Heskett]

On Monday 09 April 2018 09:51:37 Greg Wooledge wrote:

> On Mon, Apr 09, 2018 at 09:46:07AM -0400, rhkra...@gmail.com wrote:
> > To your original problem, have you tried going to a command line and
> > throwing in a couple =sync=s?  I would try that, maybe after saving
> > in your editor, and again maybe after open and / or saving in the
> > cnc program.
>
> As others have explained, the OS (Linux) keeps a cache of file
> contents that have been written by applications, but not yet committed
> to permanent storage.  If you "save" from within the text editor, then
> the saved contents should be immediately visible to other processes
> reading the file, regardless of whether it has been synced to disk. 
> They'll simply get the cached version.

Which is not happening after several hours and a hundred or more edits. 
Which is why its so intermittent.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[Stefan Monnier]

>> A big, rapidly blinking BLOCK cursor would help these old eyes find
>> it a lot easier.  But in 20  years thats fallen out of style, dammit.

100% idle state is important to reduce power consumption, so blinking
while otherwise idle is to be avoided in general, yes.  But it's OK to
blink when there's other activity.  E.g. in Emacs, after execution of
a command by default the cursor blinks (by default) for upto
`blink-cursor-blinks` after which it stays solid.

> The cursor is a function of the terminal, not of the text editor
> running inside the terminal.

That's only for those editor that run within a text-terminal (and even
in those cases, the editor can decide to control the cursor's blinking.
Emacs currently doesn't, but I use a local patch which does let it do
so, so it obeys `blink-cursor-blinks`, `blink-cursor-interval`, and
`blink-cursor-delay` rather than).


Stefan

Re: SSD's and many edits of a single file

[Gene Heskett]

On Monday 09 April 2018 09:46:07 rhkra...@gmail.com wrote:

> On Monday, April 09, 2018 05:53:18 AM Gene Heskett wrote:
> > What else is a good editor? And lets not start yet another vim vs
> > emacs war. kwrite and kate come to mind. I'm running Trinity
> > Desktop, maybe I should check them out. Whatever, needs to be able
> > to have several files open so I can just click back and forth on the
> > tabs as I usually separate the files into what uses which cutting
> > tool.
>
> I use kate and kwrite (on Wheezy) for some of my important things (my
> askSam workalike mashup), nedit and some others for other purposes.
>
> kate and kwrite both allow multiple files to be open, but, instead of
> tabs, you can (with an option--maybe on the menu) have an additional
> panel on the left hand side of the screen showing you all the files
> open in that instance and allowing you to click on any one of them to
> switch to that file.
>
> To your original problem, have you tried going to a command line and
> throwing in a couple =sync=s?  I would try that, maybe after saving in
> your editor, and again maybe after open and / or saving in the cnc
> program.

I wasn't schmardt enough to think of that before I rebooted it from here 
in the house last night. I shoulda.

Thanks rhkramer.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[Gene Heskett]

On Monday 09 April 2018 06:18:18 songbird wrote:

> Gene Heskett wrote:
> > Greetings folks;
> >
> > Updodate Wheezy, realtime kernel because machine is running
> > linuxcnc.
> >
> > Editor is geany and file is left open in the editor, and reloaded
> > into linuxcnc as changes are made to the file, and saved but not
> > closed.
> >
> > And eventually the updates made to the file are not actually saved,
> > geany does not report an error and does say success by changing the
> > filename font from red which tells me its been modified, back to
> > black in the window top border, but does not actually update the
> > file on the disk, and the only fix seems to be a reboot.
> >
> > Is this a known fault with some SSD's? Or???
>
>   i'm not familiar with geany or linuxcnc...  any options
> in geany for save interval?

I think 2 minutes.
>
>   what kind of file system? 

ext4 w/journaling on.
>   what are the mount options 

defaults

> in fstab?
UUID=54273d27-b48f-4a73-9b4d-d3cd13c6ba79 / ext4 
errors=remount-ro 01

>type of SSD?
kernel: [3.648318] scsi 0:0:0:0: Direct-Access ATA  SPCC 
Solid State V3.7 PQ: 0 ANSI: 5

Cheap on sale special from newegg. Lots quicker than spinning rust. 
Currently 28% used ack df.

Thanks songbird.
>   a sync once in a while like other poster recommended
> may be a help as it would also clear out any waiting io
> but i'm not sure how much that may interfere with the
> realtime aspects if you have some large chunks waiting.
>
I can only guess.  These files are generally not long, 100 lines or so if 
I write it but if generated by a code generator, which has no knowledge 
of the looping abilities of linuxcnc's RS-274-D translator, could make a 
40 meg file out of what should be less than 100 properly constructed 
lines of gcode.

>   i've used ionice for some things to make sure they
> don't interfere on an older system with limited
> capacity.

The heart of linuxcnc runs on one core of a multicore cpu, and something 
like ionice would probably get locked out of that playpen and have no 
effect. Any effect would probably encourage it to wreck an otherwise 
purty part. It, LinuxCNC, supplies the motor signals to drive the 
machine to sub-micron accuracy, but to do that it has to be able to 
slice time into nanoseconds.

> my current machine is a screamer in comparison 
> so i've not had to do that.

Most of this stuff isn't the latest and greatest.  Old Dell's, or Atom 
powered shoeboxes even. What matters is how steady a heartbeat it can 
maintain when a beat of the heart might be needed every 25 nanoseconds, 
and the atoms are the fastest kid in town for that. The biggest machine 
is a 1500 lb lathe, and its being run by an r-pi-3b using mesa 
interfaceing cards. Might be the only one on the planet.

>
>   songbird

Thanks songbird.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[Greg Wooledge]

On Mon, Apr 09, 2018 at 09:46:07AM -0400, rhkra...@gmail.com wrote:
> To your original problem, have you tried going to a command line and throwing 
> in a couple =sync=s?  I would try that, maybe after saving in your editor, 
> and 
> again maybe after open and / or saving in the cnc program.

As others have explained, the OS (Linux) keeps a cache of file contents
that have been written by applications, but not yet committed to
permanent storage.  If you "save" from within the text editor, then the
saved contents should be immediately visible to other processes reading
the file, regardless of whether it has been synced to disk.  They'll
simply get the cached version.

Re: SSD's and many edits of a single file

[rhkramer]

On Monday, April 09, 2018 05:53:18 AM Gene Heskett wrote:
> What else is a good editor? And lets not start yet another vim vs emacs
> war. kwrite and kate come to mind. I'm running Trinity Desktop, maybe I
> should check them out. Whatever, needs to be able to have several files
> open so I can just click back and forth on the tabs as I usually
> separate the files into what uses which cutting tool.

I use kate and kwrite (on Wheezy) for some of my important things (my askSam 
workalike mashup), nedit and some others for other purposes.

kate and kwrite both allow multiple files to be open, but, instead of tabs, you 
can (with an option--maybe on the menu) have an additional panel on the left 
hand side of the screen showing you all the files open in that instance and 
allowing you to click on any one of them to switch to that file.

To your original problem, have you tried going to a command line and throwing 
in a couple =sync=s?  I would try that, maybe after saving in your editor, and 
again maybe after open and / or saving in the cnc program.

Re: SSD's and many edits of a single file

[Greg Wooledge]

On Mon, Apr 09, 2018 at 09:32:14AM -0400, Gene Heskett wrote:
> A big, rapidly blinking 
> BLOCK cursor would help these old eyes find it a lot easier. But in 20 
> years thats fallen out of style, dammit.

The cursor is a function of the terminal, not of the text editor
running inside the terminal.

rxvt-unicode (the terminal I use on Debian) has a solid block cursor by
default.  You can make it blink by running "rxvt -bc", or by setting the
"cursorBlink" resource in your X resources.  I wouldn't call it *rapidly*
blinking (looks to be about 1 Hz), but it's blinking.  (I've also decided
that I don't care for it, and won't be using this option, but hey,
that's what *options* are for.)

Other terminals may or may not have similar options.

Re: SSD's and many edits of a single file

[Gene Heskett]

On Monday 09 April 2018 09:02:55 Stefan Monnier wrote:

> >> So the actual disk isn't to blame (cat'ing a file that was just
> >> saved won't look at the disk anyway).
> >
> > Whats it look at 5+ minutes later?
>
> Same difference: if `cat` can't see it, then the change hasn't been
> received by the OS at all (even less so by the underlying disk).
>
> > The file had not been updated yet an hour later. Ext4's journal in
> > my understanding has a maximum of 5 minutes to sync?  Its now around
> > 10 hours later and has not yet been updated with the last two lines
> > I'd added.
>
> The changes you see in `cat`s output will be transferred to disk
> eventually, yes.  But those that aren't visible to `cat` just don't
> exist, as far as the OS is concerned.
>
> > What else is a good editor?
>
> Is that a trick question?
>
Twasn't intended to be.

However I did find (I actually read its man page!) an alias setting that 
makes nano do line at a time scrolling and installed it, so I'll try it 
when I get back to that project later today. nano hasn't made a mistake 
yet that I didn't do typeing in the wrong place. A big, rapidly blinking 
BLOCK cursor would help these old eyes find it a lot easier. But in 20 
years thats fallen out of style, dammit.
>
> Stefan



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[Stefan Monnier]

>> So the actual disk isn't to blame (cat'ing a file that was just saved
>> won't look at the disk anyway).
> Whats it look at 5+ minutes later?

Same difference: if `cat` can't see it, then the change hasn't been
received by the OS at all (even less so by the underlying disk).

> The file had not been updated yet an hour later. Ext4's journal in my 
> understanding has a maximum of 5 minutes to sync?  Its now around 10 
> hours later and has not yet been updated with the last two lines I'd 
> added.

The changes you see in `cat`s output will be transferred to disk
eventually, yes.  But those that aren't visible to `cat` just don't
exist, as far as the OS is concerned.

> What else is a good editor?

Is that a trick question?


Stefan

Re: SSD's and many edits of a single file

[songbird]

Gene Heskett wrote:
> Greetings folks;
>
> Updodate Wheezy, realtime kernel because machine is running linuxcnc.
>
> Editor is geany and file is left open in the editor, and reloaded into 
> linuxcnc as changes are made to the file, and saved but not closed.
>
> And eventually the updates made to the file are not actually saved, geany 
> does not report an error and does say success by changing the filename 
> font from red which tells me its been modified, back to black in the 
> window top border, but does not actually update the file on the disk, 
> and the only fix seems to be a reboot.
>
> Is this a known fault with some SSD's? Or???

  i'm not familiar with geany or linuxcnc...  any options
in geany for save interval?

  what kind of file system?  what are the mount options
in fstab?  type of SSD?

  a sync once in a while like other poster recommended
may be a help as it would also clear out any waiting io
but i'm not sure how much that may interfere with the
realtime aspects if you have some large chunks waiting.

  i've used ionice for some things to make sure they
don't interfere on an older system with limited
capacity.  my current machine is a screamer in comparison
so i've not had to do that.


  songbird

Re: SSD's and many edits of a single file

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Apr 09, 2018 at 05:53:18AM -0400, Gene Heskett wrote:
> On Sunday 08 April 2018 23:13:28 Stefan Monnier wrote:
> 
> > >> > And eventually the updates made to the file are not actually
> > >> > saved,
> > >>
> > >> Can you be more precise than "eventually"?
> > >
> > > Probably 100+ edits and saves over 4 or 5 hours.
> > >
> > >> More importantly: what makes you think they're not actually saved?
> > >
> > > Going to another shell and cat'ing the file shows the old contents.
> >
> > So the actual disk isn't to blame (cat'ing a file that was just saved
> > won't look at the disk anyway).
> 
> Whats it look at 5+ minutes later?
> 
> The file had not been updated yet an hour later. Ext4's journal in my 
> understanding has a maximum of 5 minutes to sync?  Its now around 10 
> hours later and has not yet been updated with the last two lines I'd 
> added.

If the file system stays mounted, this all is irrelevant: it's the
OS's job to present all applications a consistent view of the
file system. Once the writing application (Geany, or whatever) has
committed its write the modification should be visible, whether
the OS has committed its buffers to the storage or not.

> > Sounds like a bug in your text editor.

Yep, that would be my hunch too.

Cheers
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlrLTkEACgkQBcgs9XrR2kYxKQCdEe0tomEdxB/RbDSbhhlItssk
GPUAnjaWd8Uid0K4/iJsWE9D+hmapd9s
=TeCh
-END PGP SIGNATURE-

Re: SSD's and many edits of a single file

[Gene Heskett]

On Sunday 08 April 2018 23:13:28 Stefan Monnier wrote:

> >> > And eventually the updates made to the file are not actually
> >> > saved,
> >>
> >> Can you be more precise than "eventually"?
> >
> > Probably 100+ edits and saves over 4 or 5 hours.
> >
> >> More importantly: what makes you think they're not actually saved?
> >
> > Going to another shell and cat'ing the file shows the old contents.
>
> So the actual disk isn't to blame (cat'ing a file that was just saved
> won't look at the disk anyway).

Whats it look at 5+ minutes later?

The file had not been updated yet an hour later. Ext4's journal in my 
understanding has a maximum of 5 minutes to sync?  Its now around 10 
hours later and has not yet been updated with the last two lines I'd 
added.

> Sounds like a bug in your text editor.

Maybe, its geany. Lots of people seem to like gedit, but its saves are 
the cause of important configuration files being written back to disk 
with the line order totally trashed, as if you had thrown it on the 
floor in 512 byte pieces, then picked it back up and reassembled it in 
random order. Then try to recover a 1400 LOC configuration file...

Thats happened using gedit enough, on several different machines here 
that its been expunged from my systems, all of them.

Nano would do what I need, but its half a screen jump scroll leads to 
mistakes on my part because its too easy to loose track of the cursor 
when it does scroll.

What else is a good editor? And lets not start yet another vim vs emacs 
war. kwrite and kate come to mind. I'm running Trinity Desktop, maybe I 
should check them out. Whatever, needs to be able to have several files 
open so I can just click back and forth on the tabs as I usually 
separate the files into what uses which cutting tool.

Advice checked out.

Thanks Stefan.

>
> Stefan



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[Stefan Monnier]

>> > And eventually the updates made to the file are not actually saved,
>> Can you be more precise than "eventually"?
> Probably 100+ edits and saves over 4 or 5 hours.
>> More importantly: what makes you think they're not actually saved?
> Going to another shell and cat'ing the file shows the old contents. 

So the actual disk isn't to blame (cat'ing a file that was just saved
won't look at the disk anyway).

Sounds like a bug in your text editor.


Stefan

Re: SSD's and many edits of a single file

[Gene Heskett]

On Sunday 08 April 2018 22:17:33 Stefan Monnier wrote:

> > And eventually the updates made to the file are not actually saved,
>
> Can you be more precise than "eventually"?

Probably 100+ edits and saves over 4 or 5 hours.

> More importantly: what makes you think they're not actually saved?
>
>
> Stefan

Going to another shell and cat'ing the file shows the old contents. 
Linuxcnc also has a code display that tracks the execution, and new 
contents is not displayed in that window either.  Takes several hours to 
start failing though. I went out to install some keying screws in some 
brass slugs intended to hold threading taps, started at about 13:30, and 
it was doing the silent failure at 17:30.  Machine uptime was about 47 
days, and linuxcnc had not been shut down and restarted in a week.  I 
have SSR's to kill motor power when disabling linuxcnc, but leave it 
running so that I can restart the motor power without having to go thru 
a lengthy re-homing operation.

This seems to be related more to how busy the editor is rather than the 
uptime of linuxcnc.

Now I've rebooted, and the code won't get much more TLC until the next 
project. So I don't expect any more trouble until I start on the next 
phase of this project, probably at least a week down the log since I 
have about 50 more of this part to finish, taxes to do by the 15nth, and 
a safety rail beside a ramp I just built into the front deck so I can 
wheel the missus around in a wheel chair. We've both reached the golden 
years, and passed them by by 2 decades.

Yeah, I was a geek before the word was invented. :)

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: SSD's and many edits of a single file

[Stefan Monnier]

> And eventually the updates made to the file are not actually saved,

Can you be more precise than "eventually"?
More importantly: what makes you think they're not actually saved?


Stefan