Re: [f2fs-dev] [PATCH RFC 1/5] f2fs: introduce inmem curseg

2020-08-03 Thread Jaegeuk Kim
On 08/04, Chao Yu wrote:
> On 2020/8/4 10:44, Jaegeuk Kim wrote:
> > On 08/04, Chao Yu wrote:
> > > On 2020/7/25 16:42, Chao Yu wrote:
> > > > On 2020/7/16 9:24, Chao Yu wrote:
> > > > > On 2020/7/16 3:07, Jaegeuk Kim wrote:
> > > > > > On 07/15, Chao Yu wrote:
> > > > > > > On 2020/7/7 11:51, Jaegeuk Kim wrote:
> > > > > > > > On 07/07, Chao Yu wrote:
> > > > > > > > > On 2020/7/7 11:21, Jaegeuk Kim wrote:
> > > > > > > > > > Hi Chao,
> > > > > > > > > > 
> > > > > > > > > > Do you have any brief design doc to present the idea?
> > > > > > > > > 
> > > > > > > > > Hi Jaegeuk,
> > > > > > > > > 
> > > > > > > > > You mean this whole patchset, right?
> > > > > > > > > 
> > > > > > > > > I can add a brief design description in patch 0/5.
> > > > > > > > 
> > > > > > > > Yeah, it's a bit hard to understand the whole flow.
> > > > > > > 
> > > > > > > Jaegeuk,
> > > > > > > 
> > > > > > > Do you have time to take a look at this idea summarized in
> > > > > > > [PATCH 0/5]'s cover letter?
> > > > > > 
> > > > > > Sorry, I couldn't afford to sitting down to review the design.
> > > > > > Let me give it a try soon.
> > > > > 
> > > > > Alright, let me know if you have any question about the idea.
> > > > 
> > > > Ping,
> > > > 
> > > > Jaegeuk, still be too busy...? :P
> > > 
> > > Ping again...
> > > 
> > > Any thoughts about left patches in patchset?
> > 
> > Ah, I was waiting for your another patch-set.
> 
> Oops, I thought that you have not finished reviewing all patches...
> 
> Anyway, let me send v2 w/ changes mentioned by you.
> 
> BTW, we can discuss about how to enable such feature, options can be:
> - enable when sb feature F2FS_FEATURE_ATGC was set
> - enable via mount option

Looks like mount option would be better.

> - enable via sysfs
> 
> > 
> > > 
> > > > 
> > > > Thanks,
> > > > 
> > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > Thanks,
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > Thanks,
> > > > > > > > > > 
> > > > > > > > > > On 06/30, Chao Yu wrote:
> > > > > > > > > > > Previous implementation of aligned pinfile allocation 
> > > > > > > > > > > will:
> > > > > > > > > > > - allocate new segment on cold data log no matter whether 
> > > > > > > > > > > last used
> > > > > > > > > > > segment is partially used or not, it makes IOs more 
> > > > > > > > > > > random;
> > > > > > > > > > > - force concurrent cold data/GCed IO going into warm data 
> > > > > > > > > > > area, it
> > > > > > > > > > > can make a bad effect on hot/cold data separation;
> > > > > > > > > > > 
> > > > > > > > > > > In this patch, we introduce a new type of log named 
> > > > > > > > > > > 'inmem curseg',
> > > > > > > > > > > the differents from normal curseg is:
> > > > > > > > > > > - it reuses existed segment type (CURSEG_XXX_NODE/DATA);
> > > > > > > > > > > - it only exists in memory, its segno, blkofs, summary 
> > > > > > > > > > > will not b
> > > > > > > > > > > persisted into checkpoint area;
> > > > > > > > > > > 
> > > > > > > > > > > With this new feature, we can enhance scalability of log, 
> > > > > > > > > > > special
> > > > > > > > > > > allocators can be created for purposes:
> > > > > > > > > > > - pure lfs allocator for aligned pinfile allocation or 
> > > > > > > > > > > file
> > > > > > > > > > > defragmentation
> > > > > > > > > > > - pure ssr allocator for later feature
> > > > > > > > > > > 
> > > > > > > > > > > So that, let's update aligned pinfile allocation to use 
> > > > > > > > > > > this new
> > > > > > > > > > > inmem curseg fwk.
> > > > > > > > > > > 
> > > > > > > > > > > Signed-off-by: Chao Yu 
> > > > > > > > > > > ---
> > > > > > > > > > > fs/f2fs/checkpoint.c |   7 ++-
> > > > > > > > > > > fs/f2fs/debug.c  |   6 ++-
> > > > > > > > > > > fs/f2fs/f2fs.h   |  12 +++--
> > > > > > > > > > > fs/f2fs/file.c   |   3 +-
> > > > > > > > > > > fs/f2fs/gc.c |   2 +-
> > > > > > > > > > > fs/f2fs/segment.c| 107 
> > > > > > > > > > > ++-
> > > > > > > > > > > fs/f2fs/segment.h|  17 ---
> > > > > > > > > > > fs/f2fs/super.c  |   9 ++--
> > > > > > > > > > > 8 files changed, 112 insertions(+), 51 deletions(-)
> > > > > > > > > > > 
> > > > > > > > > > > diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
> > > > > > > > > > > index 1bb8278a1c4a..644a914af25a 100644
> > > > > > > > > > > --- a/fs/f2fs/checkpoint.c
> > > > > > > > > > > +++ b/fs/f2fs/checkpoint.c
> > > > > > > > > > > @@ -1623,11 +1623,16 @@ int f2fs_write_checkpoint(struct 
> > > > > > > > > > > f2fs_sb_info *sbi, struct cp_control *cpc)
> > > > > > > > > > >   f2fs_flush_sit_entries(sbi, cpc);
> > > > > > > > > > > + /* save inmem log status */
> > > > > > > > > > > + f2fs_save_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
> > > > > > > > > > > +
> > > > > > > > > > >   err = do_checkpoint(sbi, cpc);
> > > > > > > > > > >   

Re: [f2fs-dev] [PATCH RFC 1/5] f2fs: introduce inmem curseg

2020-08-03 Thread Chao Yu

On 2020/8/4 10:44, Jaegeuk Kim wrote:

On 08/04, Chao Yu wrote:

On 2020/7/25 16:42, Chao Yu wrote:

On 2020/7/16 9:24, Chao Yu wrote:

On 2020/7/16 3:07, Jaegeuk Kim wrote:

On 07/15, Chao Yu wrote:

On 2020/7/7 11:51, Jaegeuk Kim wrote:

On 07/07, Chao Yu wrote:

On 2020/7/7 11:21, Jaegeuk Kim wrote:

Hi Chao,

Do you have any brief design doc to present the idea?


Hi Jaegeuk,

You mean this whole patchset, right?

I can add a brief design description in patch 0/5.


Yeah, it's a bit hard to understand the whole flow.


Jaegeuk,

Do you have time to take a look at this idea summarized in
[PATCH 0/5]'s cover letter?


Sorry, I couldn't afford to sitting down to review the design.
Let me give it a try soon.


Alright, let me know if you have any question about the idea.


Ping,

Jaegeuk, still be too busy...? :P


Ping again...

Any thoughts about left patches in patchset?


Ah, I was waiting for your another patch-set.


Oops, I thought that you have not finished reviewing all patches...

Anyway, let me send v2 w/ changes mentioned by you.

BTW, we can discuss about how to enable such feature, options can be:
- enable when sb feature F2FS_FEATURE_ATGC was set
- enable via mount option
- enable via sysfs







Thanks,









Thanks,





Thanks,

On 06/30, Chao Yu wrote:

Previous implementation of aligned pinfile allocation will:
- allocate new segment on cold data log no matter whether last used
segment is partially used or not, it makes IOs more random;
- force concurrent cold data/GCed IO going into warm data area, it
can make a bad effect on hot/cold data separation;

In this patch, we introduce a new type of log named 'inmem curseg',
the differents from normal curseg is:
- it reuses existed segment type (CURSEG_XXX_NODE/DATA);
- it only exists in memory, its segno, blkofs, summary will not b
persisted into checkpoint area;

With this new feature, we can enhance scalability of log, special
allocators can be created for purposes:
- pure lfs allocator for aligned pinfile allocation or file
defragmentation
- pure ssr allocator for later feature

So that, let's update aligned pinfile allocation to use this new
inmem curseg fwk.

Signed-off-by: Chao Yu 
---
fs/f2fs/checkpoint.c |   7 ++-
fs/f2fs/debug.c  |   6 ++-
fs/f2fs/f2fs.h   |  12 +++--
fs/f2fs/file.c   |   3 +-
fs/f2fs/gc.c |   2 +-
fs/f2fs/segment.c| 107 ++-
fs/f2fs/segment.h|  17 ---
fs/f2fs/super.c  |   9 ++--
8 files changed, 112 insertions(+), 51 deletions(-)

diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 1bb8278a1c4a..644a914af25a 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -1623,11 +1623,16 @@ int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, 
struct cp_control *cpc)
f2fs_flush_sit_entries(sbi, cpc);
+   /* save inmem log status */
+   f2fs_save_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
+
err = do_checkpoint(sbi, cpc);
if (err)
f2fs_release_discard_addrs(sbi);
else
f2fs_clear_prefree_segments(sbi, cpc);
+
+   f2fs_restore_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
stop:
unblock_operations(sbi);
stat_inc_cp_count(sbi->stat_info);
@@ -1658,7 +1663,7 @@ void f2fs_init_ino_entry_info(struct f2fs_sb_info *sbi)
}
sbi->max_orphans = (sbi->blocks_per_seg - F2FS_CP_PACKS -
-   NR_CURSEG_TYPE - __cp_payload(sbi)) *
+   NR_CURSEG_PERSIST_TYPE - __cp_payload(sbi)) *
F2FS_ORPHANS_PER_BLOCK;
}
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c
index 4276c0f79beb..41a91aa8c262 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -164,7 +164,7 @@ static void update_general_status(struct f2fs_sb_info *sbi)
* 100 / (int)(sbi->user_block_count >> sbi->log_blocks_per_seg)
/ 2;
si->util_invalid = 50 - si->util_free - si->util_valid;
-   for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_NODE; i++) {
+   for (i = CURSEG_HOT_DATA; i < NO_CHECK_TYPE; i++) {
struct curseg_info *curseg = CURSEG_I(sbi, i);
si->curseg[i] = curseg->segno;
si->cursec[i] = GET_SEC_FROM_SEG(sbi, curseg->segno);
@@ -393,6 +393,10 @@ static int stat_show(struct seq_file *s, void *v)
   si->dirty_seg[CURSEG_COLD_NODE],
   si->full_seg[CURSEG_COLD_NODE],
   si->valid_blks[CURSEG_COLD_NODE]);
+   seq_printf(s, "  - Pinned file: %8d %8d %8d\n",
+  si->curseg[CURSEG_COLD_DATA_PINNED],
+  si->cursec[CURSEG_COLD_DATA_PINNED],
+  si->curzone[CURSEG_COLD_DATA_PINNED]);
seq_printf(s, "\n  - Valid: %d\n  - Dirty: %d\n",
   si->main_area_segs - si->dirty_count -
  

Re: [f2fs-dev] [PATCH RFC 1/5] f2fs: introduce inmem curseg

2020-08-03 Thread Jaegeuk Kim
On 08/04, Chao Yu wrote:
> On 2020/7/25 16:42, Chao Yu wrote:
> > On 2020/7/16 9:24, Chao Yu wrote:
> > > On 2020/7/16 3:07, Jaegeuk Kim wrote:
> > > > On 07/15, Chao Yu wrote:
> > > > > On 2020/7/7 11:51, Jaegeuk Kim wrote:
> > > > > > On 07/07, Chao Yu wrote:
> > > > > > > On 2020/7/7 11:21, Jaegeuk Kim wrote:
> > > > > > > > Hi Chao,
> > > > > > > > 
> > > > > > > > Do you have any brief design doc to present the idea?
> > > > > > > 
> > > > > > > Hi Jaegeuk,
> > > > > > > 
> > > > > > > You mean this whole patchset, right?
> > > > > > > 
> > > > > > > I can add a brief design description in patch 0/5.
> > > > > > 
> > > > > > Yeah, it's a bit hard to understand the whole flow.
> > > > > 
> > > > > Jaegeuk,
> > > > > 
> > > > > Do you have time to take a look at this idea summarized in
> > > > > [PATCH 0/5]'s cover letter?
> > > > 
> > > > Sorry, I couldn't afford to sitting down to review the design.
> > > > Let me give it a try soon.
> > > 
> > > Alright, let me know if you have any question about the idea.
> > 
> > Ping,
> > 
> > Jaegeuk, still be too busy...? :P
> 
> Ping again...
> 
> Any thoughts about left patches in patchset?

Ah, I was waiting for your another patch-set.

> 
> > 
> > Thanks,
> > 
> > > 
> > > > 
> > > > > 
> > > > > > 
> > > > > > Thanks,
> > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > Thanks,
> > > > > > > > 
> > > > > > > > On 06/30, Chao Yu wrote:
> > > > > > > > > Previous implementation of aligned pinfile allocation will:
> > > > > > > > > - allocate new segment on cold data log no matter whether 
> > > > > > > > > last used
> > > > > > > > > segment is partially used or not, it makes IOs more random;
> > > > > > > > > - force concurrent cold data/GCed IO going into warm data 
> > > > > > > > > area, it
> > > > > > > > > can make a bad effect on hot/cold data separation;
> > > > > > > > > 
> > > > > > > > > In this patch, we introduce a new type of log named 'inmem 
> > > > > > > > > curseg',
> > > > > > > > > the differents from normal curseg is:
> > > > > > > > > - it reuses existed segment type (CURSEG_XXX_NODE/DATA);
> > > > > > > > > - it only exists in memory, its segno, blkofs, summary will 
> > > > > > > > > not b
> > > > > > > > >persisted into checkpoint area;
> > > > > > > > > 
> > > > > > > > > With this new feature, we can enhance scalability of log, 
> > > > > > > > > special
> > > > > > > > > allocators can be created for purposes:
> > > > > > > > > - pure lfs allocator for aligned pinfile allocation or file
> > > > > > > > > defragmentation
> > > > > > > > > - pure ssr allocator for later feature
> > > > > > > > > 
> > > > > > > > > So that, let's update aligned pinfile allocation to use this 
> > > > > > > > > new
> > > > > > > > > inmem curseg fwk.
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Chao Yu 
> > > > > > > > > ---
> > > > > > > > >fs/f2fs/checkpoint.c |   7 ++-
> > > > > > > > >fs/f2fs/debug.c  |   6 ++-
> > > > > > > > >fs/f2fs/f2fs.h   |  12 +++--
> > > > > > > > >fs/f2fs/file.c   |   3 +-
> > > > > > > > >fs/f2fs/gc.c |   2 +-
> > > > > > > > >fs/f2fs/segment.c| 107 
> > > > > > > > > ++-
> > > > > > > > >fs/f2fs/segment.h|  17 ---
> > > > > > > > >fs/f2fs/super.c  |   9 ++--
> > > > > > > > >8 files changed, 112 insertions(+), 51 deletions(-)
> > > > > > > > > 
> > > > > > > > > diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
> > > > > > > > > index 1bb8278a1c4a..644a914af25a 100644
> > > > > > > > > --- a/fs/f2fs/checkpoint.c
> > > > > > > > > +++ b/fs/f2fs/checkpoint.c
> > > > > > > > > @@ -1623,11 +1623,16 @@ int f2fs_write_checkpoint(struct 
> > > > > > > > > f2fs_sb_info *sbi, struct cp_control *cpc)
> > > > > > > > >   f2fs_flush_sit_entries(sbi, cpc);
> > > > > > > > > + /* save inmem log status */
> > > > > > > > > + f2fs_save_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
> > > > > > > > > +
> > > > > > > > >   err = do_checkpoint(sbi, cpc);
> > > > > > > > >   if (err)
> > > > > > > > >   f2fs_release_discard_addrs(sbi);
> > > > > > > > >   else
> > > > > > > > >   f2fs_clear_prefree_segments(sbi, cpc);
> > > > > > > > > +
> > > > > > > > > + f2fs_restore_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
> > > > > > > > >stop:
> > > > > > > > >   unblock_operations(sbi);
> > > > > > > > >   stat_inc_cp_count(sbi->stat_info);
> > > > > > > > > @@ -1658,7 +1663,7 @@ void f2fs_init_ino_entry_info(struct 
> > > > > > > > > f2fs_sb_info *sbi)
> > > > > > > > >   }
> > > > > > > > >   sbi->max_orphans = (sbi->blocks_per_seg - F2FS_CP_PACKS 
> > > > > > > > > -
> > > > > > > > > - NR_CURSEG_TYPE - __cp_payload(sbi)) *
> > > > > > > > > + NR_CURSEG_PERSIST_TYPE - 
> > > > > > > > > __cp_payload(sbi)) *
> > > > > > > > >   F2FS_ORPHANS_PER_BLOCK;
> > > > > > > > >}

Re: [f2fs-dev] [PATCH RFC 1/5] f2fs: introduce inmem curseg

2020-08-03 Thread Chao Yu

On 2020/7/25 16:42, Chao Yu wrote:

On 2020/7/16 9:24, Chao Yu wrote:

On 2020/7/16 3:07, Jaegeuk Kim wrote:

On 07/15, Chao Yu wrote:

On 2020/7/7 11:51, Jaegeuk Kim wrote:

On 07/07, Chao Yu wrote:

On 2020/7/7 11:21, Jaegeuk Kim wrote:

Hi Chao,

Do you have any brief design doc to present the idea?


Hi Jaegeuk,

You mean this whole patchset, right?

I can add a brief design description in patch 0/5.


Yeah, it's a bit hard to understand the whole flow.


Jaegeuk,

Do you have time to take a look at this idea summarized in
[PATCH 0/5]'s cover letter?


Sorry, I couldn't afford to sitting down to review the design.
Let me give it a try soon.


Alright, let me know if you have any question about the idea.


Ping,

Jaegeuk, still be too busy...? :P


Ping again...

Any thoughts about left patches in patchset?



Thanks,









Thanks,





Thanks,

On 06/30, Chao Yu wrote:

Previous implementation of aligned pinfile allocation will:
- allocate new segment on cold data log no matter whether last used
segment is partially used or not, it makes IOs more random;
- force concurrent cold data/GCed IO going into warm data area, it
can make a bad effect on hot/cold data separation;

In this patch, we introduce a new type of log named 'inmem curseg',
the differents from normal curseg is:
- it reuses existed segment type (CURSEG_XXX_NODE/DATA);
- it only exists in memory, its segno, blkofs, summary will not b
   persisted into checkpoint area;

With this new feature, we can enhance scalability of log, special
allocators can be created for purposes:
- pure lfs allocator for aligned pinfile allocation or file
defragmentation
- pure ssr allocator for later feature

So that, let's update aligned pinfile allocation to use this new
inmem curseg fwk.

Signed-off-by: Chao Yu 
---
   fs/f2fs/checkpoint.c |   7 ++-
   fs/f2fs/debug.c  |   6 ++-
   fs/f2fs/f2fs.h   |  12 +++--
   fs/f2fs/file.c   |   3 +-
   fs/f2fs/gc.c |   2 +-
   fs/f2fs/segment.c| 107 ++-
   fs/f2fs/segment.h|  17 ---
   fs/f2fs/super.c  |   9 ++--
   8 files changed, 112 insertions(+), 51 deletions(-)

diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 1bb8278a1c4a..644a914af25a 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -1623,11 +1623,16 @@ int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, 
struct cp_control *cpc)
   
   	f2fs_flush_sit_entries(sbi, cpc);
   
+	/* save inmem log status */

+   f2fs_save_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
+
err = do_checkpoint(sbi, cpc);
if (err)
f2fs_release_discard_addrs(sbi);
else
f2fs_clear_prefree_segments(sbi, cpc);
+
+   f2fs_restore_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
   stop:
unblock_operations(sbi);
stat_inc_cp_count(sbi->stat_info);
@@ -1658,7 +1663,7 @@ void f2fs_init_ino_entry_info(struct f2fs_sb_info *sbi)
}
   
   	sbi->max_orphans = (sbi->blocks_per_seg - F2FS_CP_PACKS -

-   NR_CURSEG_TYPE - __cp_payload(sbi)) *
+   NR_CURSEG_PERSIST_TYPE - __cp_payload(sbi)) *
F2FS_ORPHANS_PER_BLOCK;
   }
   
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c

index 4276c0f79beb..41a91aa8c262 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -164,7 +164,7 @@ static void update_general_status(struct f2fs_sb_info *sbi)
* 100 / (int)(sbi->user_block_count >> sbi->log_blocks_per_seg)
/ 2;
si->util_invalid = 50 - si->util_free - si->util_valid;
-   for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_NODE; i++) {
+   for (i = CURSEG_HOT_DATA; i < NO_CHECK_TYPE; i++) {
struct curseg_info *curseg = CURSEG_I(sbi, i);
si->curseg[i] = curseg->segno;
si->cursec[i] = GET_SEC_FROM_SEG(sbi, curseg->segno);
@@ -393,6 +393,10 @@ static int stat_show(struct seq_file *s, void *v)
   si->dirty_seg[CURSEG_COLD_NODE],
   si->full_seg[CURSEG_COLD_NODE],
   si->valid_blks[CURSEG_COLD_NODE]);
+   seq_printf(s, "  - Pinned file: %8d %8d %8d\n",
+  si->curseg[CURSEG_COLD_DATA_PINNED],
+  si->cursec[CURSEG_COLD_DATA_PINNED],
+  si->curzone[CURSEG_COLD_DATA_PINNED]);
seq_printf(s, "\n  - Valid: %d\n  - Dirty: %d\n",
   si->main_area_segs - si->dirty_count -
   si->prefree_count - si->free_segs,
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 7d6c5f8ce16b..f06c77066284 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -996,7 +996,9 @@ static inline void set_new_dnode(struct dnode_of_data *dn, 
struct inode *inode,
*/
   #define  NR_CURSEG_DATA_TYPE (3)
   #define NR_CURSEG_NODE_TYPE  (3)
-#define NR_CURSEG_TYPE (NR_CURSEG_DATA_TYPE + 

Re: [f2fs-dev] [PATCH RFC 1/5] f2fs: introduce inmem curseg

2020-07-25 Thread Chao Yu

On 2020/7/16 9:24, Chao Yu wrote:

On 2020/7/16 3:07, Jaegeuk Kim wrote:

On 07/15, Chao Yu wrote:

On 2020/7/7 11:51, Jaegeuk Kim wrote:

On 07/07, Chao Yu wrote:

On 2020/7/7 11:21, Jaegeuk Kim wrote:

Hi Chao,

Do you have any brief design doc to present the idea?


Hi Jaegeuk,

You mean this whole patchset, right?

I can add a brief design description in patch 0/5.


Yeah, it's a bit hard to understand the whole flow.


Jaegeuk,

Do you have time to take a look at this idea summarized in
[PATCH 0/5]'s cover letter?


Sorry, I couldn't afford to sitting down to review the design.
Let me give it a try soon.


Alright, let me know if you have any question about the idea.


Ping,

Jaegeuk, still be too busy...? :P

Thanks,









Thanks,





Thanks,

On 06/30, Chao Yu wrote:

Previous implementation of aligned pinfile allocation will:
- allocate new segment on cold data log no matter whether last used
segment is partially used or not, it makes IOs more random;
- force concurrent cold data/GCed IO going into warm data area, it
can make a bad effect on hot/cold data separation;

In this patch, we introduce a new type of log named 'inmem curseg',
the differents from normal curseg is:
- it reuses existed segment type (CURSEG_XXX_NODE/DATA);
- it only exists in memory, its segno, blkofs, summary will not b
  persisted into checkpoint area;

With this new feature, we can enhance scalability of log, special
allocators can be created for purposes:
- pure lfs allocator for aligned pinfile allocation or file
defragmentation
- pure ssr allocator for later feature

So that, let's update aligned pinfile allocation to use this new
inmem curseg fwk.

Signed-off-by: Chao Yu 
---
  fs/f2fs/checkpoint.c |   7 ++-
  fs/f2fs/debug.c  |   6 ++-
  fs/f2fs/f2fs.h   |  12 +++--
  fs/f2fs/file.c   |   3 +-
  fs/f2fs/gc.c |   2 +-
  fs/f2fs/segment.c| 107 ++-
  fs/f2fs/segment.h|  17 ---
  fs/f2fs/super.c  |   9 ++--
  8 files changed, 112 insertions(+), 51 deletions(-)

diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 1bb8278a1c4a..644a914af25a 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -1623,11 +1623,16 @@ int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, 
struct cp_control *cpc)
  
  	f2fs_flush_sit_entries(sbi, cpc);
  
+	/* save inmem log status */

+   f2fs_save_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
+
err = do_checkpoint(sbi, cpc);
if (err)
f2fs_release_discard_addrs(sbi);
else
f2fs_clear_prefree_segments(sbi, cpc);
+
+   f2fs_restore_inmem_curseg(sbi, CURSEG_COLD_DATA_PINNED);
  stop:
unblock_operations(sbi);
stat_inc_cp_count(sbi->stat_info);
@@ -1658,7 +1663,7 @@ void f2fs_init_ino_entry_info(struct f2fs_sb_info *sbi)
}
  
  	sbi->max_orphans = (sbi->blocks_per_seg - F2FS_CP_PACKS -

-   NR_CURSEG_TYPE - __cp_payload(sbi)) *
+   NR_CURSEG_PERSIST_TYPE - __cp_payload(sbi)) *
F2FS_ORPHANS_PER_BLOCK;
  }
  
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c

index 4276c0f79beb..41a91aa8c262 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -164,7 +164,7 @@ static void update_general_status(struct f2fs_sb_info *sbi)
* 100 / (int)(sbi->user_block_count >> sbi->log_blocks_per_seg)
/ 2;
si->util_invalid = 50 - si->util_free - si->util_valid;
-   for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_NODE; i++) {
+   for (i = CURSEG_HOT_DATA; i < NO_CHECK_TYPE; i++) {
struct curseg_info *curseg = CURSEG_I(sbi, i);
si->curseg[i] = curseg->segno;
si->cursec[i] = GET_SEC_FROM_SEG(sbi, curseg->segno);
@@ -393,6 +393,10 @@ static int stat_show(struct seq_file *s, void *v)
   si->dirty_seg[CURSEG_COLD_NODE],
   si->full_seg[CURSEG_COLD_NODE],
   si->valid_blks[CURSEG_COLD_NODE]);
+   seq_printf(s, "  - Pinned file: %8d %8d %8d\n",
+  si->curseg[CURSEG_COLD_DATA_PINNED],
+  si->cursec[CURSEG_COLD_DATA_PINNED],
+  si->curzone[CURSEG_COLD_DATA_PINNED]);
seq_printf(s, "\n  - Valid: %d\n  - Dirty: %d\n",
   si->main_area_segs - si->dirty_count -
   si->prefree_count - si->free_segs,
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 7d6c5f8ce16b..f06c77066284 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -996,7 +996,9 @@ static inline void set_new_dnode(struct dnode_of_data *dn, 
struct inode *inode,
   */
  #define   NR_CURSEG_DATA_TYPE (3)
  #define NR_CURSEG_NODE_TYPE   (3)
-#define NR_CURSEG_TYPE (NR_CURSEG_DATA_TYPE + NR_CURSEG_NODE_TYPE)
+#define NR_CURSEG_INMEM_TYPE   (1)
+#define NR_CURSEG_PERSIST_TYPE (NR_CURSEG_DATA_TYPE +