Re: [gentoo-user] Re: jffs2 on gentoo

[dexters84]

Hi

In my system I didn't bother with any of embedded file systems - I've 
created 1 GB ext2 partition (journalising in ext3 increases read/write 
count), and it worked just like any other hard drive. Bios detected 
correct capacity - I was lucky with that, but in case where BIOS doesn't 
detect CF card properly google is Your friend.

I don't have all doc I've used during setup but I remember reading this one
http://silent.gumph.org/content/4/1/011-linux-on-cf.html

regards

Stroller pisze:


On 18 Mar 2008, at 10:33, Florian Philipp wrote:

On Tue, 2008-03-18 at 01:47 +, Stroller wrote:

On 17 Mar 2008, at 18:10, James wrote:

...
Wear leveling is *probably* built into the IDE to CF converter
carrier board?


Almost certainly not, I'd have thought. Aren't those boards just dumb
pin-convertors? CF cards "talk" IDE.


Yes  they are.

Another thought crossed my mind today: Does wear leveling work if I
create loopback devices (ext2-formatted) on FAT32?


Surely so. In this case you would be writing to the flash device's 
FAT32 filessystem. It doesn't matter if you're writing a .RAW picture 
file, an .iso or your loopback fs.



By the way: Why is wear leveling filesystem-dependent anyway?


No idea. Please note that in this thread I have stated that I 
_understand_ wear-levelling to be filesystem-dependent - it is others 
who have made replies stating this more confidently.



I would
have thought it were working on blocks (like device mapper, cryptsetup,
lvm and so on) and not on files.


Ah! But here we come back to the problem of recording how many times a 
given block has been written upon, in order not to kill that block. 
Most filesystems don't have to do that.


Stroller.



--
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 18 Mar 2008, at 10:33, Florian Philipp wrote:

On Tue, 2008-03-18 at 01:47 +, Stroller wrote:

On 17 Mar 2008, at 18:10, James wrote:

...
Wear leveling is *probably* built into the IDE to CF converter
carrier board?


Almost certainly not, I'd have thought. Aren't those boards just dumb
pin-convertors? CF cards "talk" IDE.


Yes  they are.

Another thought crossed my mind today: Does wear leveling work if I
create loopback devices (ext2-formatted) on FAT32?


Surely so. In this case you would be writing to the flash device's  
FAT32 filessystem. It doesn't matter if you're writing a .RAW picture  
file, an .iso or your loopback fs.



By the way: Why is wear leveling filesystem-dependent anyway?


No idea. Please note that in this thread I have stated that I  
_understand_ wear-levelling to be filesystem-dependent - it is others  
who have made replies stating this more confidently.



I would
have thought it were working on blocks (like device mapper,  
cryptsetup,

lvm and so on) and not on files.


Ah! But here we come back to the problem of recording how many times  
a given block has been written upon, in order not to kill that block.  
Most filesystems don't have to do that.


Stroller.

--
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Florian Philipp]

On Tue, 2008-03-18 at 01:47 +, Stroller wrote:
> On 17 Mar 2008, at 18:10, James wrote:
> > ...
> > Wear leveling is *probably* built into the IDE to CF converter
> > carrier board?
> 
> Almost certainly not, I'd have thought. Aren't those boards just dumb  
> pin-convertors? CF cards "talk" IDE.
> 
> Stroller.

Yes  they are.

Another thought crossed my mind today: Does wear leveling work if I
create loopback devices (ext2-formatted) on FAT32?

By the way: Why is wear leveling filesystem-dependent anyway? I would
have thought it were working on blocks (like device mapper, cryptsetup,
lvm and so on) and not on files.


signature.asc
Description: This is a digitally signed message part

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 17 Mar 2008, at 18:10, James wrote:

...
Wear leveling is *probably* built into the IDE to CF converter
carrier board?


Almost certainly not, I'd have thought. Aren't those boards just dumb  
pin-convertors? CF cards "talk" IDE.


Stroller.
--
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: jffs2 on gentoo

[James]

dexters84  gmail.com> writes:


> > I believe the size of the writes can be relevant as well.

> > Stroller.

> That was exactly my point. Systems based on cf card as hard drive are 
> usually small - one function focused devices, hence there is no need for 
> swap partition. To extend lifetime of cf card you have to minimize all 
> possible read/writes to card.


OK,


I agree.
minimize the size of the system and 
minimize writes to extend the
life of the CF.

Wear leveling is *probably* built into the IDE to CF converter
carrier board?

So no need to work about which File System to use or tuning the
file system (EXT2).

Is that it?


James



-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Jan Seeger]

On Sun, 16. Mar, W.Kenworthy spammed my inbox with 
> I believe that writing a file to a single location is not the way to do
> this: you need to write a byte to the usb key in the same location, but
> need to ensure it continually changes: perhaps rotating 1's/0's.
> Alternatively, the concern is that the FAT/inode table or the like is
> where the most wear will occur - perhaps concentrate there?

Yeah, if I have the stick mounted sync and always copy and delete a file, some
bytes should get flipped around regularly. *If* there is no internal wear
leveling, that is. On USB sticks with internal wear leveling, you will, from a
size of about 1 GB upwards, never (Well, perhaps after 10 years...)  see a
failure due to media wear.

For the record: My USB stick has now gone through 78560 read/write cycles and is
still happily copying.

Regards,
Jan

-- 
thenybble.de/blog/ -- four bits at a time


pgpbj4J4iDUdr.pgp
Description: PGP signature

Re: [gentoo-user] Re: jffs2 on gentoo

[Neil Bothwick]

On Mon, 17 Mar 2008 06:43:28 +0900, W.Kenworthy wrote:

> I believe that writing a file to a single location is not the way to do
> this: you need to write a byte to the usb key in the same location, but
> need to ensure it continually changes: perhaps rotating 1's/0's.
> Alternatively, the concern is that the FAT/inode table or the like is
> where the most wear will occur - perhaps concentrate there? (i.e., do a
> journelled FS like reiserfs with a fast update?

It used to be that writing a large file to a USB key mounted with the
sync option would update the FAT for each block written, so writing a
large file several times would soon kill it. I destroyed a 1GB key like
this by continually writing modified KNOPPIX images to it. That was a
couple of years ago, I've no idea if the kernel still writes FAT like
this because I've mounted flash devices with nosync ever since.


-- 
Neil Bothwick

Madness takes its toll. Exact change, please.


signature.asc
Description: PGP signature

Re: [gentoo-user] Re: jffs2 on gentoo

[W.Kenworthy]

I have been following this thread intermittantly and have not seen a
comment on the following:

I believe that writing a file to a single location is not the way to do
this: you need to write a byte to the usb key in the same location, but
need to ensure it continually changes: perhaps rotating 1's/0's.
Alternatively, the concern is that the FAT/inode table or the like is
where the most wear will occur - perhaps concentrate there? (i.e., do a
journelled FS like reiserfs with a fast update?

Do any USB keys do some kind of write minimisation in the controller? -
no change in the data/no write? - seems a logical way to extend the
life?

BillK


On Sun, 2008-03-16 at 19:03 +0100, Jan Seeger wrote:
> As a followup, I have actually written said script (in perl), and would 
> welcome
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Jan Seeger]

As a followup, I have actually written said script (in perl), and would welcome
any improvement comments. File size of the test file shouldn't matter, since
without wear leveling, the same cells should get written over and over again.

Only thing I need to do now is run it for a long time... Unfortunately, I need
linux for that since we need to mount the drive sync. And I have no live CD in
the house... Ts. What have I come to?

Anyway, here's the script: 


use strict;
use warnings; 

use Digest::MD5;
use Getopt::Std;
use File::Basename;
use File::Spec;
use File::Copy;

my %opts;
getopts("d:i:",\%opts);

if (! $opts{d} || ! $opts{i}) {
die <<"EOF"
-d: Mountpoint of drive to be tested. Should be mounted with "sync" mount 
option.
-i: Input file. Will be copied to mountpoint to test integrity.

Leave running for a long time to "test" your USB stick.
EOF
}

my $counter = 0;
my $originaldigest;
my $outfilename = File::Spec->catfile($opts{d},(fileparse($opts{i}))[0]);
my $digester = Digest::MD5->new();

open my $handle,"<$opts{i}";
binmode($handle);

$digester->addfile($handle);
close($handle);

$originaldigest = $digester->digest();

while (1) {
print "Running test $counter.\n";
copy($opts{i},$outfilename);
open my $outhandle,"<$outfilename";
binmode($outhandle);
$digester->addfile($outhandle);
if ($digester->digest() ne $originaldigest) {
die "Failed write at read $counter.\n";
}
close($outhandle);
unlink($outfilename);
$counter++;
}


-- 
thenybble.de/blog/ -- four bits at a time


pgp69UGUg3nle.pgp
Description: PGP signature

Re: [gentoo-user] Re: jffs2 on gentoo

[Alan McKinnon]

On Sunday 16 March 2008, Jan Seeger wrote:
> Yeah, it's the same here. I read an article in the german computer
> magazine c't, and they said that they have tried to break USB sticks
> with repeated writes, but have never succeeded (I think they ran
> 1 writes, but I could be wrong).

That test is probably insufficient. Somebody actually did this test on 
lkml some time ago, and found that the better devices were rated to 
100,000 writes to the same cell and the el-cheapo jobs were somewhere 
around 10,000.

IOW, the manufacturer of a cheapie says it should cope with 10,000 
writes *at least*, so in practice you could expect more.

I'd start to believe a test that does >1,000,000 writes to the same cell 
before drawing any conclusions.

-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Jan Seeger]

On Sun, 16. Mar, Stroller spammed my inbox with 
>>
>> Well, I've heard otherwise. Use jffs2 or the CF card will wear out
>> prematurely...
>
> I've "heard" lots about using flashdrives for filesystems, but I've never 
> read on a mailing list anything actually definitive on the subject. I find 
> many posts to be confused.

Yeah, it's the same here. I read an article in the german computer magazine c't,
and they said that they have tried to break USB sticks with repeated writes, but
have never succeeded (I think they ran 1 writes, but I could be wrong).

So why not just buy a cheap USB stick for 10 € (or whatever), mount it sync and
write a little script which writes to a file, deletes it and begins again. Have
it record the number of times the file was written and check the consistency
after every write (md5sum perhaps?). Leave it running for a (long?) time and
then you will probably encounter errors (if there are...).

Actually, this sounds interesting^^
Regards,
Jan


pgpEoKgGQZYYR.pgp
Description: PGP signature

Re: [gentoo-user] Re: jffs2 on gentoo

[dexters84]

Stroller pisze:


On 15 Mar 2008, at 20:17, James wrote:

dexters84  gmail.com> writes:

  Other things you have to remeber concern file system usage, you 
musn't

  create swap partition, disable local syslog, log rotation, turn
  everything except desired daemons etc.



Where did you get the idea not to use swap?


Too many writes.
(dexters writes "disable local syslog, log rotation" for the same reason)

I believe the size of the writes can be relevant as well.
In any case, swapfs was not designed for flash memory
(see also Windows Vista's ReadyBoost 
).


Stroller.
That was exactly my point. Systems based on cf card as hard drive are 
usually small - one function focused devices, hence there is no need for 
swap partition. To extend lifetime of cf card you have to minimize all 
possible read/writes to card.

--
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 15 Mar 2008, at 20:17, James wrote:

dexters84  gmail.com> writes:

  Other things you have to remeber concern file system usage, you  
musn't

  create swap partition, disable local syslog, log rotation, turn
  everything except desired daemons etc.



Where did you get the idea not to use swap?


Too many writes.
(dexters writes "disable local syslog, log rotation" for the same  
reason)


I believe the size of the writes can be relevant as well.
In any case, swapfs was not designed for flash memory
(see also Windows Vista's ReadyBoost ).


Stroller.
--
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: jffs2 on gentoo

[Stroller]

On 15 Mar 2008, at 20:08, James wrote:

You shouldn't need to do anything special - just copy all files over
exactly, and then set up GRUB on the CF card.


Well, I've heard otherwise. Use jffs2 or the CF card will wear out
prematurely...


I've "heard" lots about using flashdrives for filesystems, but I've  
never read on a mailing list anything actually definitive on the  
subject. I find many posts to be confused.


This comes up regularly on the mythtv-users list , and conclusions are vague.


_As I understand it_, previous posters are right though - either use  
a device with built in wear-levelling (and a file-system it's aware  
of), OR use a wear-levelling filesystem. It's not clear what happens  
if you use a wear-levelling filesystem on a device with built in wear- 
levelling.


Anecdotally, "it won't work, your flash drive will die in no  
time" (repeated often) and "we used these all the time in my last job  
and never had one die on us" (IIRC the latter guy worked on till  
systems, basically a PC with a touchscreen, and used SanDisk cards;  
it's unclear how much writing was done to the drive).


Stroller's law: ask about using compact-flash camera memory in Linux  
PC applications, and you will receive many anecdotal & hearsay replies.

Corollary: just try it. No-one else frikkin' is!


... When using 'fdisk'
to format the CF card: In section 4d of the handbook, I do not see
a 'mkjffs2fs' command.


$ eix -S jffs
* sys-fs/mtd
 Available versions:  20040825 20050519
 Homepage:http://sources.redhat.com/jffs2/
 Description: JFFS2 is a log-structured file system  
designed for use on flash devices in embedded systems.

$

if this doesn't provide the userspace tools, then the homepage prolly  
does.


Stroller.
--
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: jffs2 on gentoo

[James]

dexters84  gmail.com> writes:




 I did this sort of system a while ago. I've used 1GB card with gentoo
 and cf-ide adapter. There are some tricky parts that nobody mentions.
 One of them is that I wasn't able to boot from my 1GB "hard drive" when
 it was connected via 80 pin ide cable, I've dig up some old 40 pin ata
 cable and it worked.

  Other things you have to remeber concern file system usage, you musn't
  create swap partition, disable local syslog, log rotation, turn
  everything except desired daemons etc.



Where did you get the idea not to use swap?

What file system did you use?


James




-- 
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: jffs2 on gentoo

[James]

Dan Farrell  spore.ath.cx> writes:


> > I have a 4 gig Cf card (sandisk) and a ide-cf card that should
> > make the CF card look like an ide hard drive.

> > http://gentoo-wiki.com/Mounting_a_block_device_with_JFFS2

> > But it seems vague(outdated) and missing many steps. Or
> > am I confused? I'm just looking for some outline or verbose
> > steps to replace an ide drive on a system with a CF/ide drive
> > and jffs2, as I have many systems that I'd like to do this with, 
> > for core reliability on minimalistic gentoo servers. I plan
> > on having additional space on these systems (when needed)
> > via NFS.


> You shouldn't need to do anything special - just copy all files over
> exactly, and then set up GRUB on the CF card.  

Well, I've heard otherwise. Use jffs2 or the CF card will wear out 
prematurely...

I should have been more specific on my questions. When using 'fdisk'
to format the CF card: In section 4d of the handbook, I do not see
a 'mkjffs2fs' command. So I'm confused as to how you put the jffs2
file system on a partition.




-- 
gentoo-user@lists.gentoo.org mailing list