Re: [Gluster-devel] How to solve the FSYNC() ERR

[Pranith Kumar Karampuri]

Is it possible to share the test you are running? As per your volume,
o-direct is not enabled on your volume, i.e. the file shouldn't be opened
with o-direct but as per the logs it is giving Invalid Argument as if there
is something wrong with the arguments when we do o-direct write with wrong
size. so I would like to test out why exactly is it giving this problem.
Please note that for o-direct write to succeed, both offset and size should
be page-aligned, something like multiple of 512 is one way to check it.

On Sun, Jul 10, 2016 at 5:19 PM, Keiviw  wrote:

> My volume info:
>
> Volume Name: test
> Type: Distribute
> Volume ID: 9294b122-d81e-4b12-9b5c-46e89ee0e40b
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: compute2:/home/brick1
> Brick2: compute2:/home/brick2
> Options Reconfigured:
> performance.flush-behind: off
> storage.linux-aio: off
> My brick logs(I have cleaned up the history log):
> [2016-07-10 11:42:50.577683] E [posix.c:2128:posix_writev]
> 0-test-posix: write failed: offset 0, Invalid argument
> [2016-07-10 11:42:50.577735] I
> [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569840: WRITEV 5
> (526a3118-9994-429e-afc0-4aa063606bde) ==> -1 (Invalid argument)
> [2016-07-10 11:42:54.583038] E [posix.c:2128:posix_writev]
> 0-test-posix: write failed: offset 0, Invalid argument
> [2016-07-10 11:42:54.583080] I
> [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569870: WRITEV 5
> (c3d28f34-8f43-446d-8d0b-80841ae8ec5b) ==> -1 (Invalid argument)
> My mnt-test-.logs:
> [2016-07-10 11:42:50.577816] W
> [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote
> operation failed: Invalid argument
> [2016-07-10 11:42:50.578508] W [fuse-bridge.c:968:fuse_err_cbk]
> 0-glusterfs-fuse: 12398282: FSYNC() ERR => -1 (Invalid argument)
> [2016-07-10 11:42:54.583156] W
> [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote
> operation failed: Invalid argument
> [2016-07-10 11:42:54.583762] W [fuse-bridge.c:968:fuse_err_cbk]
> 0-glusterfs-fuse: 12398317: FSYNC() ERR => -1 (Invalid argument)
>
>
>
>
>
>
> 在 2016-07-10 19:18:18，"Krutika Dhananjay"  写道：
>
>
> To me it looks like a case of a flush triggering a write() that was cached
> by write-behind and because the write buffer
> did not meet the page alignment requirement with o-direct write, it was
> failed with EINVAL and the trigger fop - i.e., flush() was failed with the
> 'Invalid argument' error code.
>
> Could you attach the brick logs as well, so that we can confirm the theory?
>
> -Krutika
>
> On Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee 
> wrote:
>
>> Pranith/Krutika,
>>
>> Your inputs please, IIRC we'd need to turn on some o_direct option here?
>>
>>
>> On Saturday 9 July 2016, Keiviw  wrote:
>>
>>> The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag
>>> in client protocol open() and create()! How to explain and solve the
>>> problem?
>>>
>>> 发自 网易邮箱大师 
>>> On 07/09/2016 17:58, Atin Mukherjee wrote:
>>>
>>> Any specific reason of using 3.3 given that its really quite old? We are
>>> at 3.6, 3.7 & 3.8 supportability matrix now.
>>>
>>>
>>> On Saturday 9 July 2016, Keiviw  wrote:
>>>
 hi,
 I have installed GlusterFS 3.3.0, and now I get Fsync failures when
 saving files with the O_DIRECT flag in open() and create().
 1, I tried to save a flie in vi and got this error:
 "test" E667: Fsync failed
 2, I see this in the logs:
 [2016-07-07 14:20:10.325400] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 102: FSYNC() ERR => -1 (Invalid argument)
 [2016-07-07 14:20:13.930384] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 137: FSYNC() ERR => -1 (Invalid argument)
 [2016-07-07 14:20:51.199448] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 174: FLUSH() ERR => -1 (Invalid argument)
 [2016-07-07 14:21:32.804738] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 206: FLUSH() ERR => -1 (Invalid argument)
 [2016-07-07 14:21:43.702146] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 276: FSYNC() ERR => -1 (Invalid argument)
 [2016-07-07 14:21:51.296809] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 314: FSYNC() ERR => -1 (Invalid argument)
 [2016-07-07 14:21:54.062687] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 349: FSYNC() ERR => -1 (Invalid argument)
 [2016-07-07 14:22:54.678960] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 429: FSYNC() ERR => -1 (Invalid argument)
 [2016-07-07 14:24:35.546980] W [fuse-bridge.c:968:fuse_err_cbk]
 0-glusterfs-fuse: 505: 

Re: [Gluster-devel] How to solve the FSYNC() ERR

[Keiviw]

My volume info:

Volume Name: test
Type: Distribute
Volume ID: 9294b122-d81e-4b12-9b5c-46e89ee0e40b
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: compute2:/home/brick1
Brick2: compute2:/home/brick2
Options Reconfigured:
performance.flush-behind: off
storage.linux-aio: off
My brick logs(I have cleaned up the history log):
[2016-07-10 11:42:50.577683] E [posix.c:2128:posix_writev] 0-test-posix: 
write failed: offset 0, Invalid argument
[2016-07-10 11:42:50.577735] I [server3_1-fops.c:1414:server_writev_cbk] 
0-test-server: 8569840: WRITEV 5 (526a3118-9994-429e-afc0-4aa063606bde) ==> -1 
(Invalid argument)
[2016-07-10 11:42:54.583038] E [posix.c:2128:posix_writev] 0-test-posix: 
write failed: offset 0, Invalid argument
[2016-07-10 11:42:54.583080] I [server3_1-fops.c:1414:server_writev_cbk] 
0-test-server: 8569870: WRITEV 5 (c3d28f34-8f43-446d-8d0b-80841ae8ec5b) ==> -1 
(Invalid argument)
My mnt-test-.logs:
[2016-07-10 11:42:50.577816] W [client3_1-fops.c:876:client3_1_writev_cbk] 
0-test-client-1: remote operation failed: Invalid argument
[2016-07-10 11:42:50.578508] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 12398282: FSYNC() ERR => -1 (Invalid argument)
[2016-07-10 11:42:54.583156] W [client3_1-fops.c:876:client3_1_writev_cbk] 
0-test-client-1: remote operation failed: Invalid argument
[2016-07-10 11:42:54.583762] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 12398317: FSYNC() ERR => -1 (Invalid argument)









在 2016-07-10 19:18:18，"Krutika Dhananjay"  写道：


To me it looks like a case of a flush triggering a write() that was cached by 
write-behind and because the write buffer

did not meet the page alignment requirement with o-direct write, it was failed 
with EINVAL and the trigger fop - i.e., flush() was failed with the 'Invalid 
argument' error code.


Could you attach the brick logs as well, so that we can confirm the theory?



-Krutika


On Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee  wrote:
Pranith/Krutika,


Your inputs please, IIRC we'd need to turn on some o_direct option here?



On Saturday 9 July 2016, Keiviw  wrote:

The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag in 
client protocol open() and create()! How to explain and solve the problem?


发自 网易邮箱大师
On 07/09/2016 17:58, Atin Mukherjee wrote:
Any specific reason of using 3.3 given that its really quite old? We are at 
3.6, 3.7 & 3.8 supportability matrix now.


On Saturday 9 July 2016, Keiviw  wrote:

hi,
I have installed GlusterFS 3.3.0, and now I get Fsync failures when saving 
files with the O_DIRECT flag in open() and create().
1, I tried to save a flie in vi and got this error:
"test" E667: Fsync failed
2, I see this in the logs:
[2016-07-07 14:20:10.325400] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 102: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:20:13.930384] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 137: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:20:51.199448] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 174: FLUSH() ERR => -1 (Invalid argument)
[2016-07-07 14:21:32.804738] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 206: FLUSH() ERR => -1 (Invalid argument)
[2016-07-07 14:21:43.702146] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 276: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:21:51.296809] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 314: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:21:54.062687] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 349: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:22:54.678960] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 429: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:24:35.546980] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 505: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:24:48.696888] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 538: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:24:55.824752] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 575: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:25:04.364629] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 612: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:27:08.579052] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 713: FSYNC() ERR => -1 (Invalid argument)
[2016-07-07 14:46:44.635169] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 859: FLUSH() ERR => -1 (Invalid argument)
[2016-07-07 14:46:55.892041] W [fuse-bridge.c:968:fuse_err_cbk] 
0-glusterfs-fuse: 989: FLUSH() ERR => 

Re: [Gluster-devel] How to solve the FSYNC() ERR

[Krutika Dhananjay]

To me it looks like a case of a flush triggering a write() that was cached
by write-behind and because the write buffer
did not meet the page alignment requirement with o-direct write, it was
failed with EINVAL and the trigger fop - i.e., flush() was failed with the
'Invalid argument' error code.

Could you attach the brick logs as well, so that we can confirm the theory?

-Krutika

On Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee  wrote:

> Pranith/Krutika,
>
> Your inputs please, IIRC we'd need to turn on some o_direct option here?
>
>
> On Saturday 9 July 2016, Keiviw  wrote:
>
>> The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag
>> in client protocol open() and create()! How to explain and solve the
>> problem?
>>
>> 发自 网易邮箱大师 
>> On 07/09/2016 17:58, Atin Mukherjee wrote:
>>
>> Any specific reason of using 3.3 given that its really quite old? We are
>> at 3.6, 3.7 & 3.8 supportability matrix now.
>>
>>
>> On Saturday 9 July 2016, Keiviw  wrote:
>>
>>> hi,
>>> I have installed GlusterFS 3.3.0, and now I get Fsync failures when
>>> saving files with the O_DIRECT flag in open() and create().
>>> 1, I tried to save a flie in vi and got this error:
>>> "test" E667: Fsync failed
>>> 2, I see this in the logs:
>>> [2016-07-07 14:20:10.325400] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 102: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:20:13.930384] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 137: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:20:51.199448] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 174: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:21:32.804738] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 206: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:21:43.702146] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 276: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:21:51.296809] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 314: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:21:54.062687] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 349: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:22:54.678960] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 429: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:24:35.546980] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 505: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:24:48.696888] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 538: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:24:55.824752] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 575: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:25:04.364629] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 612: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:27:08.579052] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 713: FSYNC() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:46:44.635169] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 859: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:46:55.892041] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 989: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:51:45.904233] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 148747: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:54:45.603557] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 148986: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 14:59:18.818972] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 292106: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 15:48:32.708207] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 6753592: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 16:24:10.476228] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 10733703: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 16:30:42.388220] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 11497698: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-07 16:44:18.247511] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 12037185: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-08 01:00:55.131141] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 12181783: FLUSH() ERR => -1 (Invalid argument)
>>> [2016-07-08 06:54:13.418607] W [fuse-bridge.c:968:fuse_err_cbk]
>>> 0-glusterfs-fuse: 12397060: FLUSH() ERR => -1 (Invalid argument)
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Atin
>> Sent from iPhone
>>