Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Jiffin Thottan]

- Original Message -
From: "hvjunk" 
To: "Adam Ru" 
Cc: gluster-users@gluster.org
Sent: Monday, June 5, 2017 9:29:03 PM
Subject: Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after 
reboot

Sorry, got sidetracked with invoicing etc. 

https://bitbucket.org/dismyne/gluster-ansibles/src/6df23803df43/ansible/files/?at=master
 

The .service files are the stuff going into SystemD, and they call the 
test-mounts.sh scripts. 
The playbook installing higher up in the directory 


I have submitted patch based on Hendrik scripts/systemd service file 
https://review.gluster.org/#/c/17339/ 

If everything works out, it can be included in next stable releases of 
gluster(3.8.13 and 3.10.3)

On 05 Jun 2017, at 17:45 , Adam Ru < ad.ruc...@gmail.com > wrote: 

Hi hvjunk, 

could you please tell me have you had time to check my previous post? 

Could you please send me mentioned link to your Gluster Ansible scripts? 

Thank you, 

Adam 


On Sun, May 28, 2017 at 2:47 PM, Adam Ru < ad.ruc...@gmail.com > wrote: 


Hi hvjunk (Hi Hendrik), 

"centos-release-gluster" installs "centos-gluster310". I assume it 
picks the latest version and install it. 

Would you be so kind and send me a link to your script & systemd 
service / Ansible scripts? I cannot find a way how to list your posts 
on lists.gluster.org (I assume it's not possible to list posts of a 
specific user). Or if you cannot find it could you please tell me when 
did you post it? I'll try to find it. 

Meantime I wrote something very simple but I assume your scripting 
skills are better. 

Thank you. 

Kind regards. 

Adam 

-- 

sudo sh -c 'cat > /root/gluster-run-ganesha << EOF 
#!/bin/bash 

while true; do 
echo "Wait" 
sleep 30 
if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf 
]]; then 
echo "Start Ganesha" 
systemctl start nfs-ganesha.service 
exit \$? 
else 
echo "Not mounted" 
fi 
done 
EOF' 

sudo chmod +x /root/gluster-run-ganesha 

sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF 
[Unit] 
Description=Start nfs-ganesha when Gluster shared storage is mounted 

[Service] 
Type=oneshot 
ExecStart=/root/gluster-run-ganesha 

[Install] 
WantedBy=multi-user.target 
EOF' 

sudo systemctl enable custom-gluster-ganesha.service 

-- 


On Mon, May 15, 2017 at 12:27 PM, hvjunk < hvj...@gmail.com > wrote: 





On 15 May 2017, at 12:56 PM, Soumya Koduri < skod...@redhat.com > wrote: 



On 05/12/2017 06:27 PM, Adam Ru wrote: 


Hi Soumya, 

Thank you very much for last response – very useful. 

I apologize for delay, I had to find time for another testing. 

I updated instructions that I provided in previous e-mail. *** means 
that the step was added. 

Instructions: 
- Clean installation of CentOS 7.3 with all updates, 3x node, 
resolvable IPs and VIPs 
- Stopped firewalld (just for testing) 
- *** SELinux in permissive mode (I had to, will explain bellow) 
- Install “centos-release-gluster" to get "centos-gluster310" repo 

should I also install the centos-gluster310, or will that be automagically 
chosen by the centos-release-gluster? 





and install following (nothing else): 
--- glusterfs-server 
--- glusterfs-ganesha 
- Passwordless SSH between all nodes 
(/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes) 
- systemctl enable and start glusterd 
- gluster peer probe  
- gluster volume set all cluster.enable-shared-storage enable 

After this step, I’ll advise (given my experience in doing this by Ansible) to 
make sure that the shared filesystem have propagated to all the nodes, as well 
as the needed entries made in fstab… safety check, and I’ll also load my 
systemd service and helper script to assist in cluster cold-bootstrapping. 





- systemctl enable and start pcsd.service 
- systemctl enable pacemaker.service (cannot be started at this moment) 
- Set password for hacluster user on all nodes 
- pcs cluster auth-u hacluster -p blabla 
- mkdir /var/run/gluster/shared_storage/nfs-ganesha/ 
- touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not 
sure if needed) 
- vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and 
insert configuration 
- Try list files on other nodes: ls 
/var/run/gluster/shared_storage/nfs-ganesha/ 
- gluster nfs-ganesha enable 
- *** systemctl enable pacemaker.service (again, since pacemaker was 
disabled at this point) 
- *** Check owner of "state", "statd", "sm" and "sm.bak" in 
/var/lib/nfs/ (I had to: chown rpcuser:rpcuser 
/var/lib/nfs/statd/state) 
- Check on other nodes that nfs-ganesha.service is running and "pcs 
status" shows started resources 
- gluster volume create mynewshare replica 3 transport tcp 
node1:/ node2:/ node3:/ 
- gluster volume start mynewshare 
- gluster vol set mynewshare ganesha.enable on 

At this moment, this is status of important (I think) services: 

-- corosync.service disabled 
-- corosync-notifyd.service disabled 
-- 

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[hvjunk]

Sorry, got sidetracked with invoicing etc.

https://bitbucket.org/dismyne/gluster-ansibles/src/6df23803df43/ansible/files/?at=master
 


The .service files are the stuff going into SystemD, and they call the 
test-mounts.sh scripts.
The playbook installing higher up in the directory

> On 05 Jun 2017, at 17:45 , Adam Ru  wrote:
> 
> Hi hvjunk,
> 
> could you please tell me have you had time to check my previous post?
> 
> Could you please send me mentioned link to your Gluster Ansible scripts?
> 
> Thank you,
> 
> Adam
> 
> 
> On Sun, May 28, 2017 at 2:47 PM, Adam Ru  wrote:
>> Hi hvjunk (Hi Hendrik),
>> 
>> "centos-release-gluster" installs "centos-gluster310". I assume it
>> picks the latest version and install it.
>> 
>> Would you be so kind and send me a link to your script & systemd
>> service / Ansible scripts? I cannot find a way how to list your posts
>> on lists.gluster.org (I assume it's not possible to list posts of a
>> specific user). Or if you cannot find it could you please tell me when
>> did you post it? I'll try to find it.
>> 
>> Meantime I wrote something very simple but I assume your scripting
>> skills are better.
>> 
>> Thank you.
>> 
>> Kind regards.
>> 
>> Adam
>> 
>> --
>> 
>> sudo sh -c 'cat > /root/gluster-run-ganesha << EOF
>> #!/bin/bash
>> 
>> while true; do
>>echo "Wait"
>>sleep 30
>>if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
>> ]]; then
>>echo "Start Ganesha"
>>systemctl start nfs-ganesha.service
>>exit \$?
>>else
>>echo "Not mounted"
>>fi
>> done
>> EOF'
>> 
>> sudo chmod +x /root/gluster-run-ganesha
>> 
>> sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF
>> [Unit]
>> Description=Start nfs-ganesha when Gluster shared storage is mounted
>> 
>> [Service]
>> Type=oneshot
>> ExecStart=/root/gluster-run-ganesha
>> 
>> [Install]
>> WantedBy=multi-user.target
>> EOF'
>> 
>> sudo systemctl enable custom-gluster-ganesha.service
>> 
>> --
>> 
>> 
>> On Mon, May 15, 2017 at 12:27 PM, hvjunk  wrote:
>>> 
 On 15 May 2017, at 12:56 PM, Soumya Koduri  wrote:
 
 
 
 On 05/12/2017 06:27 PM, Adam Ru wrote:
> Hi Soumya,
> 
> Thank you very much for last response – very useful.
> 
> I apologize for delay, I had to find time for another testing.
> 
> I updated instructions that I provided in previous e-mail. *** means
> that the step was added.
> 
> Instructions:
> - Clean installation of CentOS 7.3 with all updates, 3x node,
> resolvable IPs and VIPs
> - Stopped firewalld (just for testing)
> - *** SELinux in permissive mode (I had to, will explain bellow)
> - Install “centos-release-gluster" to get "centos-gluster310" repo
>>> 
>>> should I also install the centos-gluster310, or will that be automagically 
>>> chosen by the centos-release-gluster?
>>> 
> and install following (nothing else):
> --- glusterfs-server
> --- glusterfs-ganesha
> - Passwordless SSH between all nodes
> (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
> - systemctl enable and start glusterd
> - gluster peer probe 
> - gluster volume set all cluster.enable-shared-storage enable
>>> 
>>> After this step, I’ll advise (given my experience in doing this by Ansible) 
>>> to make sure that the shared filesystem have propagated to all the nodes, 
>>> as well as the needed entries made in fstab… safety check, and I’ll also 
>>> load my systemd service and helper script to assist in cluster 
>>> cold-bootstrapping.
>>> 
> - systemctl enable and start pcsd.service
> - systemctl enable pacemaker.service (cannot be started at this moment)
> - Set password for hacluster user on all nodes
> - pcs cluster auth-u hacluster -p blabla
> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
> sure if needed)
> - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
> insert configuration
> - Try list files on other nodes: ls
> /var/run/gluster/shared_storage/nfs-ganesha/
> - gluster nfs-ganesha enable
> - *** systemctl enable pacemaker.service (again, since pacemaker was
> disabled at this point)
> - *** Check owner of "state", "statd", "sm" and "sm.bak" in
> /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
> /var/lib/nfs/statd/state)
> - Check on other nodes that nfs-ganesha.service is running and "pcs
> status" shows started resources
> - gluster volume create mynewshare replica 3 transport tcp
> node1:/ node2:/ node3:/
> - gluster volume start mynewshare
> - gluster vol set mynewshare ganesha.enable on
> 
> At this moment, this is status 

Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[Adam Ru]

Hi hvjunk,

could you please tell me have you had time to check my previous post?

Could you please send me mentioned link to your Gluster Ansible scripts?

Thank you,

Adam


On Sun, May 28, 2017 at 2:47 PM, Adam Ru  wrote:
> Hi hvjunk (Hi Hendrik),
>
> "centos-release-gluster" installs "centos-gluster310". I assume it
> picks the latest version and install it.
>
> Would you be so kind and send me a link to your script & systemd
> service / Ansible scripts? I cannot find a way how to list your posts
> on lists.gluster.org (I assume it's not possible to list posts of a
> specific user). Or if you cannot find it could you please tell me when
> did you post it? I'll try to find it.
>
> Meantime I wrote something very simple but I assume your scripting
> skills are better.
>
> Thank you.
>
> Kind regards.
>
> Adam
>
> --
>
> sudo sh -c 'cat > /root/gluster-run-ganesha << EOF
> #!/bin/bash
>
> while true; do
> echo "Wait"
> sleep 30
> if [[ -f /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
> ]]; then
> echo "Start Ganesha"
> systemctl start nfs-ganesha.service
> exit \$?
> else
> echo "Not mounted"
> fi
> done
> EOF'
>
> sudo chmod +x /root/gluster-run-ganesha
>
> sudo sh -c 'cat > /etc/systemd/system/custom-gluster-ganesha.service << EOF
> [Unit]
> Description=Start nfs-ganesha when Gluster shared storage is mounted
>
> [Service]
> Type=oneshot
> ExecStart=/root/gluster-run-ganesha
>
> [Install]
> WantedBy=multi-user.target
> EOF'
>
> sudo systemctl enable custom-gluster-ganesha.service
>
> --
>
>
> On Mon, May 15, 2017 at 12:27 PM, hvjunk  wrote:
>>
>>> On 15 May 2017, at 12:56 PM, Soumya Koduri  wrote:
>>>
>>>
>>>
>>> On 05/12/2017 06:27 PM, Adam Ru wrote:
 Hi Soumya,

 Thank you very much for last response – very useful.

 I apologize for delay, I had to find time for another testing.

 I updated instructions that I provided in previous e-mail. *** means
 that the step was added.

 Instructions:
 - Clean installation of CentOS 7.3 with all updates, 3x node,
 resolvable IPs and VIPs
 - Stopped firewalld (just for testing)
 - *** SELinux in permissive mode (I had to, will explain bellow)
 - Install “centos-release-gluster" to get "centos-gluster310" repo
>>
>> should I also install the centos-gluster310, or will that be automagically 
>> chosen by the centos-release-gluster?
>>
 and install following (nothing else):
 --- glusterfs-server
 --- glusterfs-ganesha
 - Passwordless SSH between all nodes
 (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
 - systemctl enable and start glusterd
 - gluster peer probe 
 - gluster volume set all cluster.enable-shared-storage enable
>>
>> After this step, I’ll advise (given my experience in doing this by Ansible) 
>> to make sure that the shared filesystem have propagated to all the nodes, as 
>> well as the needed entries made in fstab… safety check, and I’ll also load 
>> my systemd service and helper script to assist in cluster cold-bootstrapping.
>>
 - systemctl enable and start pcsd.service
 - systemctl enable pacemaker.service (cannot be started at this moment)
 - Set password for hacluster user on all nodes
 - pcs cluster auth-u hacluster -p blabla
 - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
 - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
 sure if needed)
 - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
 insert configuration
 - Try list files on other nodes: ls
 /var/run/gluster/shared_storage/nfs-ganesha/
 - gluster nfs-ganesha enable
 - *** systemctl enable pacemaker.service (again, since pacemaker was
 disabled at this point)
 - *** Check owner of "state", "statd", "sm" and "sm.bak" in
 /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
 /var/lib/nfs/statd/state)
 - Check on other nodes that nfs-ganesha.service is running and "pcs
 status" shows started resources
 - gluster volume create mynewshare replica 3 transport tcp
 node1:/ node2:/ node3:/
 - gluster volume start mynewshare
 - gluster vol set mynewshare ganesha.enable on

 At this moment, this is status of important (I think) services:

 -- corosync.service disabled
 -- corosync-notifyd.service disabled
 -- glusterd.service enabled
 -- glusterfsd.service   disabled
 -- pacemaker.serviceenabled
 -- pcsd.service enabled
 -- nfs-ganesha.service  disabled
 -- nfs-ganesha-config.service   static
 -- nfs-ganesha-lock.service static

 -- corosync.service active (running)
 -- corosync-notifyd.service inactive (dead)
 -- glusterd.service active (running)
 

[Gluster-users] Rebalance failing on fix-layout

[Jarsulic, Michael [CRI]]

Hello,

The past couple of weeks I had some issues with firmware on the OS hard drives 
in my gluster cluster. I have recently fixed the issue, and am bringing my 
bricks back into the volume. I am running gluster 3.7.6 and am running into the 
following issue:

When I add the brick and rebalance, the operation fails after a couple minutes. 
The errors I find in the rebalance log is this:

[2017-06-05 13:38:40.441671] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C 
gfid not present
[2017-06-05 13:38:40.450341] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
/LV_Fitting/code/C/NoCov_NoImm gfid not present
[2017-06-05 13:38:40.450380] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
/LV_Fitting/code/C/simulate gfid not present
[2017-06-05 13:38:40.459365] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
/LV_Fitting/code/C/NoCov_NoImm/fits_generate gfid not present
[2017-06-05 13:38:40.468756] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
/LV_Fitting/code/C/NoCov_NoImm/fits_generate/N_0_vector.dat gfid not present
[2017-06-05 13:38:40.495645] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
LV_Fitting/code/C/simulate/RK45_Integrate.c gfid not present
[2017-06-05 13:38:40.512336] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
/LV_Fitting/output/line_search gfid not present
[2017-06-05 13:38:40.512373] E [MSGID: 109010] 
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:  
/LV_Fitting/output/mcmc gfid not present
[2017-06-05 13:38:40.517808] E [dht-rebalance.c:2992:gf_defrag_fix_layout] 
0-hpcscratch-dht: Setxattr failed for /LV_Fitting/output/line_search
[2017-06-05 13:38:40.518025] E [MSGID: 109016] 
[dht-rebalance.c:3006:gf_defrag_fix_layout] 0-hpcscratch-dht: Fix layout failed 
for /LV_Fitting/output
[2017-06-05 13:38:40.518136] E [MSGID: 109016] 
[dht-rebalance.c:3006:gf_defrag_fix_layout] 0-hpcscratch-dht: Fix layout failed 
for /LV_Fitting


There are about 102,000 of the gfid error, but only a few errors for the fix 
layout failed. Is there any way to recover from this issue?

--
Mike Jarsulic
Sr. HPC Administrator
Center for Research Informatics | University of Chicago
773.702.2066
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-Maintainers] [Gluster-devel] Backport for "Add back socket for polling of events immediately..."

[Shyam]

On 05/30/2017 08:44 PM, Zhang Huan wrote:

* Are there any existing users who need this enhancement?
https://bugzilla.redhat.com/show_bug.cgi?id=1358606#c27

Though not sure what branch Zhang Huan is on. @Zhang your inputs are
needed
here.

We are currently on 3.8. Thus the performance number is based on 3.8.
If you need more details, please let me know.


Thanks Zhang. The question was more on the lines whether you need
backport of the fix to 3.8.


Actually, we really need this backported to 3.8. I have seen the
backport of it to 3.8.
https://review.gluster.org/#/c/15046/
Once it gets merged, we will rebase to it and test it as a whole.


@Zang and @list, as this is a performance improvement feature and we do 
not backport features into releases (as a rule) that are already out in 
the field, hence we will not be backporting this to 3.8.


Further, 3.8 will EOL (end of life) from a maintenance standpoint when 
3.12 is released (scheduled around Aug 30th).


We would be merging this into 3.11.1 to provide early access for tests 
and such (Release date of June 20th), and this feature would be made 
generally available with 3.12.


We regret any inconvenience.




Can you upgrade to recent releases (say 3.11.x or 3.10.x)?


Sorry, I am afraid not. Gusterfs is one of the key components in our
product. An upgrade alone would break the whole thing.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance + VM corruption - current status and request for feedback

[Shyam]

Just to be clear, the release notes still carry the warning about this, 
and the code to use force when doing rebalance is still in place.


As we have received the feedback that this works, these will be removed 
in the subsequent minor release for the various streams as appropriate.


Thanks,
Shyam

On 06/05/2017 07:36 AM, Gandalf Corvotempesta wrote:

Great, thanks!

Il 5 giu 2017 6:49 AM, "Krutika Dhananjay" > ha scritto:

The fixes are already available in 3.10.2, 3.8.12 and 3.11.0

-Krutika

On Sun, Jun 4, 2017 at 5:30 PM, Gandalf Corvotempesta
> wrote:

Great news.
Is this planned to be published in next release?

Il 29 mag 2017 3:27 PM, "Krutika Dhananjay" > ha scritto:

Thanks for that update. Very happy to hear it ran fine
without any issues. :)

Yeah so you can ignore those 'No such file or directory'
errors. They represent a transient state where DHT in the
client process is yet to figure out the new location of the
file.

-Krutika


On Mon, May 29, 2017 at 6:51 PM, Mahdi Adnan
>
wrote:

Hello,


Yes, i forgot to upgrade the client as well.

I did the upgrade and created a new volume, same options
as before, with one VM running and doing lots of IOs. i
started the rebalance with force and after it completed
the process i rebooted the VM, and it did start normally
without issues.

I repeated the process and did another rebalance while
the VM running and everything went fine.

But the logs in the client throwing lots of warning
messages:


[2017-05-29 13:14:59.416382] W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk]
2-gfs_vol2-client-2: remote operation failed. Path:

/50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or
directory]
[2017-05-29 13:14:59.416427] W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk]
2-gfs_vol2-client-3: remote operation failed. Path:

/50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or
directory]
[2017-05-29 13:14:59.808251] W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk]
2-gfs_vol2-client-2: remote operation failed. Path:

/50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or
directory]
[2017-05-29 13:14:59.808287] W [MSGID: 114031]
[client-rpc-fops.c:2928:client3_3_lookup_cbk]
2-gfs_vol2-client-3: remote operation failed. Path:

/50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or
directory]



Although the process went smooth, i will run another
extensive test tomorrow just to be sure.

--

Respectfully*
**Mahdi A. Mahdi*



*From:* Krutika Dhananjay >
*Sent:* Monday, May 29, 2017 9:20:29 AM

*To:* Mahdi Adnan
*Cc:* gluster-user; Gandalf Corvotempesta; Lindsay
Mathieson; Kevin Lemonnier
*Subject:* Re: Rebalance + VM corruption - current
status and request for feedback

Hi,

I took a look at your logs.
It very much seems like an issue that is caused by a
mismatch in glusterfs client and server packages.
So your client (mount) seems to be still running 3.7.20,
as confirmed by the occurrence of the following log message:

[2017-05-26 08:58:23.647458] I [MSGID: 100030]
[glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started

Re: [Gluster-users] Rebalance + VM corruption - current status and request for feedback

[Gandalf Corvotempesta]

Great, thanks!

Il 5 giu 2017 6:49 AM, "Krutika Dhananjay"  ha scritto:

> The fixes are already available in 3.10.2, 3.8.12 and 3.11.0
>
> -Krutika
>
> On Sun, Jun 4, 2017 at 5:30 PM, Gandalf Corvotempesta <
> gandalf.corvotempe...@gmail.com> wrote:
>
>> Great news.
>> Is this planned to be published in next release?
>>
>> Il 29 mag 2017 3:27 PM, "Krutika Dhananjay"  ha
>> scritto:
>>
>>> Thanks for that update. Very happy to hear it ran fine without any
>>> issues. :)
>>>
>>> Yeah so you can ignore those 'No such file or directory' errors. They
>>> represent a transient state where DHT in the client process is yet to
>>> figure out the new location of the file.
>>>
>>> -Krutika
>>>
>>>
>>> On Mon, May 29, 2017 at 6:51 PM, Mahdi Adnan 
>>> wrote:
>>>
 Hello,


 Yes, i forgot to upgrade the client as well.

 I did the upgrade and created a new volume, same options as before,
 with one VM running and doing lots of IOs. i started the rebalance with
 force and after it completed the process i rebooted the VM, and it did
 start normally without issues.

 I repeated the process and did another rebalance while the VM running
 and everything went fine.

 But the logs in the client throwing lots of warning messages:


 [2017-05-29 13:14:59.416382] W [MSGID: 114031]
 [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2:
 remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c
 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
 (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]
 [2017-05-29 13:14:59.416427] W [MSGID: 114031]
 [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-3:
 remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c
 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
 (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]
 [2017-05-29 13:14:59.808251] W [MSGID: 114031]
 [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2:
 remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c
 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
 (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]
 [2017-05-29 13:14:59.808287] W [MSGID: 114031]
 [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-3:
 remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c
 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
 (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]



 Although the process went smooth, i will run another extensive test
 tomorrow just to be sure.

 --

 Respectfully
 *Mahdi A. Mahdi*

 --
 *From:* Krutika Dhananjay 
 *Sent:* Monday, May 29, 2017 9:20:29 AM

 *To:* Mahdi Adnan
 *Cc:* gluster-user; Gandalf Corvotempesta; Lindsay Mathieson; Kevin
 Lemonnier
 *Subject:* Re: Rebalance + VM corruption - current status and request
 for feedback

 Hi,

 I took a look at your logs.
 It very much seems like an issue that is caused by a mismatch in
 glusterfs client and server packages.
 So your client (mount) seems to be still running 3.7.20, as confirmed
 by the occurrence of the following log message:

 [2017-05-26 08:58:23.647458] I [MSGID: 100030] [glusterfsd.c:2338:main]
 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20
 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2
 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
 /rhev/data-center/mnt/glusterSD/s1:_testvol)
 [2017-05-26 08:58:40.901204] I [MSGID: 100030] [glusterfsd.c:2338:main]
 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20
 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2
 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
 /rhev/data-center/mnt/glusterSD/s1:_testvol)
 [2017-05-26 08:58:48.923452] I [MSGID: 100030] [glusterfsd.c:2338:main]
 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20
 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2
 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
 /rhev/data-center/mnt/glusterSD/s1:_testvol)

 whereas the servers have rightly been upgraded to 3.10.2, as seen in
 rebalance log:

 [2017-05-26 09:36:36.075940] I [MSGID: 100030] [glusterfsd.c:2475:main]
 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.2
 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/testvol
 

Re: [Gluster-users] Release 3.12: Scope and calendar!

[Sonal Arora]

On Thu, Jun 1, 2017 at 10:47 PM, Shyam  wrote:

> Hi,
>
> Here are some top reminders for the 3.12 release:
>
> 1) When 3.12 is released 3.8 will be EOL'd, hence users are encouraged to
> prepare for the same as per the calendar posted here.
>
> 2) 3.12 is a long term maintenance (LTM) release, and potentially the last
> in the 3.x line of Gluster!
>
> 3) From this release onward, the feature freeze date is moved ~45 days in
> advance, before the release. Hence, for this one release you will have
> lesser time to get your features into the release.
>
> Release calendar:
>
> - Feature freeze, or branching date: July 17th, 2017
>- All feature post this date need exceptions granted to make it into
> the 3.12 release
>
> - Release date: August 30th, 2017
>
> Release owners:
>
> - Shyam
> -  Any volunteers?
>
> Features and major changes process in a nutshell:
> 1) Open a github issue
>
> 2) Refer the issue # in the commit messages of all changes against the
> feature (specs, code, tests, docs, release notes) (refer to the issue as
> "updates gluster/glusterfs#N" where N is the issue)
>
> 3) We will ease out release-notes updates form this release onward. Still
> thinking how to get that done, but the intention is that a contributor can
> update release notes before/on/after completion of the feature and not
> worry about branching dates etc. IOW, you can control when you are done,
> than the release dates controlling the same for you.
>
> Thanks,
> Shyam
>

I am interested in this.

-Sonal

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Release 3.12: Scope and calendar!

[Ankit Raj]

On Mon, Jun 5, 2017 at 3:30 PM, Jiffin Tony Thottan 
wrote:

>
>
> On 01/06/17 22:47, Shyam wrote:
>
>> Hi,
>>
>> Here are some top reminders for the 3.12 release:
>>
>> 1) When 3.12 is released 3.8 will be EOL'd, hence users are encouraged to
>> prepare for the same as per the calendar posted here.
>>
>> 2) 3.12 is a long term maintenance (LTM) release, and potentially the
>> last in the 3.x line of Gluster!
>>
>> 3) From this release onward, the feature freeze date is moved ~45 days in
>> advance, before the release. Hence, for this one release you will have
>> lesser time to get your features into the release.
>>
>> Release calendar:
>>
>> - Feature freeze, or branching date: July 17th, 2017
>>- All feature post this date need exceptions granted to make it into
>> the 3.12 release
>>
>> - Release date: August 30th, 2017
>>
>> Release owners:
>>
>> - Shyam
>> -  Any volunteers?
>>
>>
> I am interested
> --
> Jiffin
>
>

  I am also interested
   Ankit

> Features and major changes process in a nutshell:
>> 1) Open a github issue
>>
>> 2) Refer the issue # in the commit messages of all changes against the
>> feature (specs, code, tests, docs, release notes) (refer to the issue as
>> "updates gluster/glusterfs#N" where N is the issue)
>>
>> 3) We will ease out release-notes updates form this release onward. Still
>> thinking how to get that done, but the intention is that a contributor can
>> update release notes before/on/after completion of the feature and not
>> worry about branching dates etc. IOW, you can control when you are done,
>> than the release dates controlling the same for you.
>>
>> Thanks,
>> Shyam
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
> ___
> Gluster-devel mailing list
> gluster-de...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.11 Retrospective

[Mohammed Rafi K C]

Great, it works perfectly. This is a good start.


Rafi KC


On 06/05/2017 01:44 PM, Amye Scavarda wrote:
> Resolved! 
> Try it again.
> - amye 
>
> On Mon, Jun 5, 2017 at 5:07 PM, Mohammed Rafi K C  > wrote:
>
> Hi Amye,
>
> The form is not accessible, it says
>
> Feedback for Gluster 3.11 release
>
> The form Feedback for Gluster 3.11 release is no longer accepting
> responses.
> Try contacting the owner of the form if you think this is a mistake.
>
>
> Regards
> Rafi KC
>
>
>
> On 06/05/2017 01:25 PM, Amye Scavarda wrote:
>> We're doing something new now with releases, running a
>> retrospective on what things we as a community should stop, what
>> we should start, and what we should continue.
>>
>> With last week's release, here's our quick form for 3.11
>> https://goo.gl/forms/OkhNZDFspYqdN00g2
>> 
>>
>> We'll keep this open until June 15th to give everyone time to
>> give us feedback. Thanks!
>>
>> -- 
>> Amye Scavarda | a...@redhat.com  |
>> Gluster Community Lead
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>> 
>
> -- 
> Amye Scavarda | a...@redhat.com  | Gluster
> Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.11 Retrospective

[Amye Scavarda]

Resolved!
Try it again.
- amye

On Mon, Jun 5, 2017 at 5:07 PM, Mohammed Rafi K C 
wrote:

> Hi Amye,
>
> The form is not accessible, it says
>
> Feedback for Gluster 3.11 release
> The form Feedback for Gluster 3.11 release is no longer accepting
> responses.
> Try contacting the owner of the form if you think this is a mistake.
>
>
> Regards
> Rafi KC
>
>
>
> On 06/05/2017 01:25 PM, Amye Scavarda wrote:
>
> We're doing something new now with releases, running a retrospective on
> what things we as a community should stop, what we should start, and what
> we should continue.
>
> With last week's release, here's our quick form for 3.11
> https://goo.gl/forms/
> OkhNZDFspYqdN00g2
>
> We'll keep this open until June 15th to give everyone time to give us
> feedback. Thanks!
>
> --
> Amye Scavarda | a...@redhat.com | Gluster Community Lead
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>


-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.11 Retrospective

[Mohammed Rafi K C]

Hi Amye,

The form is not accessible, it says

Feedback for Gluster 3.11 release

The form Feedback for Gluster 3.11 release is no longer accepting responses.
Try contacting the owner of the form if you think this is a mistake.


Regards
Rafi KC


On 06/05/2017 01:25 PM, Amye Scavarda wrote:
> We're doing something new now with releases, running a retrospective
> on what things we as a community should stop, what we should start,
> and what we should continue.
>
> With last week's release, here's our quick form for 3.11
> https://goo.gl/forms/OkhNZDFspYqdN00g2
>
> We'll keep this open until June 15th to give everyone time to give us
> feedback. Thanks!
>
> -- 
> Amye Scavarda | a...@redhat.com  | Gluster
> Community Lead
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] 3.11 Retrospective

[Amye Scavarda]

We're doing something new now with releases, running a retrospective on
what things we as a community should stop, what we should start, and what
we should continue.

With last week's release, here's our quick form for 3.11
https://goo.gl/forms/OkhNZDFspYqdN00g2

We'll keep this open until June 15th to give everyone time to give us
feedback. Thanks!

-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users