[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256549#comment-14256549 ] Jonathan Hsieh commented on HBASE-12332: Here's a version of the patch that builds on HBASE-12749. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253042#comment-14253042 ] Jiajia Li commented on HBASE-12332: --- upload the HBASE-12332-V2.patch, catch more exceptions when read the cell.[~j...@cloudera.com] > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246357#comment-14246357 ] Jiajia Li commented on HBASE-12332: --- hi, [~j...@cloudera.com], the case you mentioned is added in https://issues.apache.org/jira/browse/HBASE-12673, can you look at it? Thanks~ > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243847#comment-14243847 ] Jingcheng Du commented on HBASE-12332: -- Hi Jon [~jmhsieh]. We've added the test case in the patch of HBASE-12673, and make sure the current implementation could work well with this above case. Please help review and comment. Thanks. The mob ref cell in the HBase only contains the mob file name (not the HFileLink pattern), it's hard to know whether the mob file is a HFileLink in the read path. Instead reading the file via the value of the ref cell in HBase (which is a mob file name, not a HFileLink pattern) through the read paths(two possible locations) is much easier. How do you think about this? Thanks. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242079#comment-14242079 ] Jingcheng Du commented on HBASE-12332: -- To correct the description. bq. This case will be added in another JIRA to make sure the current implementation is right. Should be "The test case will be added in another JIRA to make sure the current implementation is right." > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242078#comment-14242078 ] Jingcheng Du commented on HBASE-12332: -- Thanks Jon [~jmhsieh], we will add such tests. >>The problem is that if we are in the middle of reading a mob file and it is >>moved to the archive, the read could fail and it is not clear if they read >>would be retried against the new location from the same spot. I think it's not a problem. When the read fails after the file is archived, the mob will try to read the file from the archive. This case will be added in another JIRA to make sure the current implementation is right. I call tell how the read works in mob based on the mentioned steps. 1) Create a table A with mob, add cells to it. These cells with tags of table A. 2) Snapshot it. 3) Clone it as a different table B. Now the cells in the table B still have tags of table A. 4) Read from the snapshot, the reader find the tag of table is A not B itself, will try to read the file from the candidate directories of A (working directory or archive of A), then it finds the file in the working directory of A. 5) Delete the original table A, the mob files are archived. The reader try to read the next candidate directory which is the archive of A, then it will find the file there. We will add test cases to prove it. Thanks. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241478#comment-14241478 ] Jonathan Hsieh commented on HBASE-12332: Do we have a test that shows that the mechanism you propose works when have a mob hfile move from the mob dir to the archive dir. Here's where I believe the we'll run into a problem 1) create a table with mobs, 2) snapshot it, 3) clone/restore it as a a different table 4) have a read workload on the snapshot 5) delete the original table The problem is that if we are in the middle of reading a mob file and it is moved to the archive, the read could fail and it is not clear if they read would be retried against the new location from the same spot. The hfile link would transparently fall over to alternate fie locations and continue the read from the same place it had encountered the exception on the new location. This avoids having to percolate an exception up as high. >From a quick scan of the patch, I believe it covers the case where reads are >not active or if a mob file move happens while reads are not in flight. I >didn't see the case I described being covered. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240558#comment-14240558 ] Jiajia Li commented on HBASE-12332: --- hi, [~j...@cloudera.com], do you have any idea? > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238899#comment-14238899 ] Jiajia Li commented on HBASE-12332: --- hi, [~jmhsieh], in reading, we don't directly open scanners to all the existing mob files which is easy to know a file link by the matching the name pattern, instead we follow steps below. # Read the file name from the HBase ( this is just a file name, not a file link pattern, we don't know the file link name in this cell). # Read mob cell from the candidate paths( mobworkingDir/filename, mobArchive/filename, the latter two are for cloned snapshot, srcTableMobWorkingDir/filename, srcTableArchive/filename). According to the above read path, it's not possible to know whether the current mob file in the working directory is a file link by the name which is just a mob file name (not a file link pattern). In the latest patch, the possible read path had been reduced from 4 to 2 by comparing the source table tag for the cloned snapshot. It means searching the cloned snapshot is as fast as the normal mob cells. Please advise. Thanks~ > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238092#comment-14238092 ] Jonathan Hsieh commented on HBASE-12332: Check out HBASE-12646. Inside it I've modified FileLink so that the mob path and archiver is considered as well. (I need to check more to make sure the mob data when it gets archived moves to the same archiver dir). However, with this we should be able to remove all the special logic and just use the FileLink to read mob files and transparently handle the case when the original moves. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231069#comment-14231069 ] Jiajia Li commented on HBASE-12332: --- hi, [~jmhsieh], can you give some advise on this patch? > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205814#comment-14205814 ] Jingcheng Du commented on HBASE-12332: -- Hi Jon, [~jmhsieh], do you have comments on this patch? Thanks. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1424#comment-1424 ] Jingcheng Du commented on HBASE-12332: -- Hi Jon [~jmhsieh], we will improve the way to read the cloned mob data. Previously we have four candidate locations, if it's a cloned mob data, it will have to find four files in the worst case, it really costs a lot. In the new implementation, we will decide the candidate locations by table name, the locations has two candidates at the most. The patch will be uploaded later. Thanks. With the new patch, the time cost of unit tests drops a lot. Most of them(except the case of exporting snapshot) are finished within 100 seconds. Do we still need to move these unit tests to integration tests? Please advise. Thanks. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191806#comment-14191806 ] Jingcheng Du commented on HBASE-12332: -- Hi [~jmhsieh], any ideas on the comments? Thanks! > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186540#comment-14186540 ] Jingcheng Du commented on HBASE-12332: -- Hi Jon [~jmhsieh], what's your opinion for this? > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182317#comment-14182317 ] Jingcheng Du commented on HBASE-12332: -- Do we need to do this for snapshot? Actually in the mob, the hfilelinks are not used in the read path. The Mob has four candidate paths in reading after the snapshot ( mob working, mob archive, source mob working and source mob archive), these paths are known without parsing the hfilelink. So is it necessary to detour through the hfilelink? One more thing is HFileLink can be only parsed its name to a hbase working/archive dir, it's a new problem to let it parse the file name to a mob dir. Please advise. Thanks. > [mob] use filelink instad of retry when resolving an hfilelink. > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh > Fix For: hbase-11339 > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
