Re: Why does creating a repository fail?

[David Reagan]

> unless, of course, you were to detach the volume from host/vm-1 and
attached it to another system which had a different user table.
This is exactly the problem. For snapshot repositories, your repository
directory has to be accessible by all your data nodes. Thus, I mount the
same folder, in the same location, on all my data nodes. Thus, my uid/gid
issues. Which would also apply on iSCSI.

Anyway, I modified my Ansible elasticsearch role to support creating a
custom user and group. Tested it on my tiny test cluster, and it seems to
work ok. Now I just need to run it on my prod cluster and deal with a full
cluster restart. *sigh*


--David Reagan

On Wed, Mar 18, 2015 at 5:21 AM, Norberto Meijome  wrote:

> yes, that's the difference between a (network service which exposes )
> block storage like iSCSI and a network file system like NFS ( or glusterFS
> or Lustre... ).
>
> I don't see why on a local device (iSCSI) you'd have any issue with
> numeric uid not matching the 'name' of the user - unless, of course, you
> were to detach the volume from host/vm-1 and attached it to another system
> which had a different user table.
>
> centralised authentication systems like ldap would solve this for you too
> :) (though it seems overkill when all you'd have to do is plan your uid
> space properly across client + nfs servers ...). configuration systems like
> puppet / chef / ansible / salt should also help you ensure uids across
> multiple system...
>
> Anyway, Magnus' suggestion is on the money for the '1 server problem'
>
> On Wed, Mar 18, 2015 at 3:00 AM, David Reagan  wrote:
>
>> @Mark Walkom, So, I'm looking into iscsi. From what I have learned so
>> far, you actually format the LUN with whatever file system you want. So,
>> wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
>> ext4? Since Ubuntu would treat it like a normal partition and use typical
>> linux file perms on it.
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 5:37 PM, David Reagan  wrote:
>>
>>> If I were manually creating the elasticsearch user, that'd be easy. But
>>> I'm relying on apt to do the job for me. So, yeah...
>>>
>>> Hmm... I suppose I could manually create an elasticsearch2 user, then
>>> modify the defaults files to use it when running ES. Still seems clunky...
>>>
>>> --David Reagan
>>>
>>> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden 
>>> wrote:
>>>
 I’m not that familiar with iSCSI so I hesitate to say for sure, but
 anytime you are cross-mounting filesystems on Linux you have to take
 uid/gid consistency into account.

 - Andrew

 On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:

 Would an iSCSI mount have the same issue? I believe our SAN supports
 both.

 --David Reagan

 On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden 
 wrote:

> Hi David,
>
> This is a common problem with NFS. Unfortunately the protocol assumes
> identical uid/gid mappings across all machines. It’s just one of those
> annoying sys-admin tasks that one has to take into account when using NFS.
> To get your permissions back to less permissive settings you will have to
> edit the /etc/passwd and /etc/group files to keep them in sync.
>
> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
> for more context.
>
> - Andrew
>
>
> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>
> First, it is a file permissions issue. I did get snapshots to run when
> I chmoded to 777. As you can see from the ls output, /mounts/prod_backup 
> is
> 777. Prior to that it was 775 or 755. So, I could revise my question to
> "How can I get snapshots working without using insecure file permissions?"
>
> root@log-elasticsearch-01:~# mount
> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sda1 on /boot type ext2 (rw)
> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
> type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>
> root@log-elasticsearch-01:~# ls -ld /mounts
> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>
> root@log-elastic

Re: Why does creating a repository fail?

[Norberto Meijome]

yes, that's the difference between a (network service which exposes ) block
storage like iSCSI and a network file system like NFS ( or glusterFS or
Lustre... ).

I don't see why on a local device (iSCSI) you'd have any issue with numeric
uid not matching the 'name' of the user - unless, of course, you were to
detach the volume from host/vm-1 and attached it to another system which
had a different user table.

centralised authentication systems like ldap would solve this for you too
:) (though it seems overkill when all you'd have to do is plan your uid
space properly across client + nfs servers ...). configuration systems like
puppet / chef / ansible / salt should also help you ensure uids across
multiple system...

Anyway, Magnus' suggestion is on the money for the '1 server problem'

On Wed, Mar 18, 2015 at 3:00 AM, David Reagan  wrote:

> @Mark Walkom, So, I'm looking into iscsi. From what I have learned so far,
> you actually format the LUN with whatever file system you want. So,
> wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
> ext4? Since Ubuntu would treat it like a normal partition and use typical
> linux file perms on it.
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 5:37 PM, David Reagan  wrote:
>
>> If I were manually creating the elasticsearch user, that'd be easy. But
>> I'm relying on apt to do the job for me. So, yeah...
>>
>> Hmm... I suppose I could manually create an elasticsearch2 user, then
>> modify the defaults files to use it when running ES. Still seems clunky...
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden  wrote:
>>
>>> I’m not that familiar with iSCSI so I hesitate to say for sure, but
>>> anytime you are cross-mounting filesystems on Linux you have to take
>>> uid/gid consistency into account.
>>>
>>> - Andrew
>>>
>>> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
>>>
>>> Would an iSCSI mount have the same issue? I believe our SAN supports
>>> both.
>>>
>>> --David Reagan
>>>
>>> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden 
>>> wrote:
>>>
 Hi David,

 This is a common problem with NFS. Unfortunately the protocol assumes
 identical uid/gid mappings across all machines. It’s just one of those
 annoying sys-admin tasks that one has to take into account when using NFS.
 To get your permissions back to less permissive settings you will have to
 edit the /etc/passwd and /etc/group files to keep them in sync.

 See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
 for more context.

 - Andrew


 On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:

 First, it is a file permissions issue. I did get snapshots to run when
 I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
 777. Prior to that it was 775 or 755. So, I could revise my question to
 "How can I get snapshots working without using insecure file permissions?"

 root@log-elasticsearch-01:~# mount
 /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
 proc on /proc type proc (rw,noexec,nosuid,nodev)
 sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
 none on /sys/fs/fuse/connections type fusectl (rw)
 none on /sys/kernel/debug type debugfs (rw)
 none on /sys/kernel/security type securityfs (rw)
 udev on /dev type devtmpfs (rw,mode=0755)
 devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
 tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
 none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
 none on /run/shm type tmpfs (rw,nosuid,nodev)
 /dev/sda1 on /boot type ext2 (rw)
 rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
 nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
 (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
 nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

 root@log-elasticsearch-01:~# ls -ld /mounts
 drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts

 root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
 drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
 /mounts/prod_backup/

 --David Reagan

 On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
 wrote:

> Can you post the output from *mount* and *ls -ld /mounts
> /mounts/prod_backup*?
>
> On 16 March 2015 at 13:33, David Reagan  wrote:
>
>> Why does this happen?
>>
>>
>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true'
>>> -d '{
>>> > "type": "fs",
>>> > "settings": {
>>> > "location": "/mounts/prod_backup/my_backup",
>>> > "compress": true
>>> > }
>>> > }'
>>> {
>>>   "error" :
>>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]

Re: Why does creating a repository fail?

[Magnus Bäck]

On Tuesday, March 17, 2015 at 01:37 CET,
 David Reagan  wrote:

> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden 
> wrote:
> 
> > I'm not that familiar with iSCSI so I hesitate to say for sure, but
> > anytime you are cross-mounting filesystems on Linux you have to take
> > uid/gid consistency into account.
>
> If I were manually creating the elasticsearch user, that'd be easy.
> But I'm relying on apt to do the job for me. So, yeah...

You can create the elasticsearch user with the uid you want before the
Debian package gets a chance to do it.

> Hmm... I suppose I could manually create an elasticsearch2 user,
> then modify the defaults files to use it when running ES. Still
> seems clunky...

No need. If the elasticsearch user exists the package installation will
just skip that step.

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150318072807.GC15152%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Re: Why does creating a repository fail?

[David Reagan]

According to http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/
the method described in that post only applies to old, out of date,
systems.

I also found no mention of a map file in
http://linux.die.net/man/8/mount.nfs or http://linux.die.net/man/5/nfs

The closest I found to something I could use was
http://serverfault.com/questions/514118/mapping-uid-and-gid-of-local-user-to-the-mounted-nfs-share
But it seems to only apply to nfs version 4. We, for some reason, are on
version 3.

Hmm... Would adding the suid flag to the mount help?


As for iSCSI, it doesn't matter if the file system sees it as a local
device. Currently my file system sees my nfs mounts as pretty much local
mounts. But it still thinks that some of my elasticsearch owned files are
actually owned by my ntp user... I don't see how "formatting however I
want" will help with that kind of issue. Permissions are set by uid and
gid, not the name. Unless iSCSI has some feature that overrides that.

--David Reagan

On Tue, Mar 17, 2015 at 11:11 AM, Mark Walkom  wrote:

> iSCSI can be mounted as a block device that you can format however you
> want, if you do it that way the uid problem won't show up as the system
> sees it as a local FS.
>
> On 17 March 2015 at 09:00, David Reagan  wrote:
>
>> @Mark Walkom, So, I'm looking into iscsi. From what I have learned so
>> far, you actually format the LUN with whatever file system you want. So,
>> wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
>> ext4? Since Ubuntu would treat it like a normal partition and use typical
>> linux file perms on it.
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 5:37 PM, David Reagan  wrote:
>>
>>> If I were manually creating the elasticsearch user, that'd be easy. But
>>> I'm relying on apt to do the job for me. So, yeah...
>>>
>>> Hmm... I suppose I could manually create an elasticsearch2 user, then
>>> modify the defaults files to use it when running ES. Still seems clunky...
>>>
>>> --David Reagan
>>>
>>> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden 
>>> wrote:
>>>
 I’m not that familiar with iSCSI so I hesitate to say for sure, but
 anytime you are cross-mounting filesystems on Linux you have to take
 uid/gid consistency into account.

 - Andrew

 On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:

 Would an iSCSI mount have the same issue? I believe our SAN supports
 both.

 --David Reagan

 On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden 
 wrote:

> Hi David,
>
> This is a common problem with NFS. Unfortunately the protocol assumes
> identical uid/gid mappings across all machines. It’s just one of those
> annoying sys-admin tasks that one has to take into account when using NFS.
> To get your permissions back to less permissive settings you will have to
> edit the /etc/passwd and /etc/group files to keep them in sync.
>
> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
> for more context.
>
> - Andrew
>
>
> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>
> First, it is a file permissions issue. I did get snapshots to run when
> I chmoded to 777. As you can see from the ls output, /mounts/prod_backup 
> is
> 777. Prior to that it was 775 or 755. So, I could revise my question to
> "How can I get snapshots working without using insecure file permissions?"
>
> root@log-elasticsearch-01:~# mount
> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sda1 on /boot type ext2 (rw)
> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
> type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>
> root@log-elasticsearch-01:~# ls -ld /mounts
> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>
> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
> /mounts/prod_backup/
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
> wrote:
>
>> Can you post the output from *mount* and *ls -ld /mounts
>> /mounts/pr

Re: Why does creating a repository fail?

[Mark Walkom]

iSCSI can be mounted as a block device that you can format however you
want, if you do it that way the uid problem won't show up as the system
sees it as a local FS.

On 17 March 2015 at 09:00, David Reagan  wrote:

> @Mark Walkom, So, I'm looking into iscsi. From what I have learned so far,
> you actually format the LUN with whatever file system you want. So,
> wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
> ext4? Since Ubuntu would treat it like a normal partition and use typical
> linux file perms on it.
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 5:37 PM, David Reagan  wrote:
>
>> If I were manually creating the elasticsearch user, that'd be easy. But
>> I'm relying on apt to do the job for me. So, yeah...
>>
>> Hmm... I suppose I could manually create an elasticsearch2 user, then
>> modify the defaults files to use it when running ES. Still seems clunky...
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden  wrote:
>>
>>> I’m not that familiar with iSCSI so I hesitate to say for sure, but
>>> anytime you are cross-mounting filesystems on Linux you have to take
>>> uid/gid consistency into account.
>>>
>>> - Andrew
>>>
>>> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
>>>
>>> Would an iSCSI mount have the same issue? I believe our SAN supports
>>> both.
>>>
>>> --David Reagan
>>>
>>> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden 
>>> wrote:
>>>
 Hi David,

 This is a common problem with NFS. Unfortunately the protocol assumes
 identical uid/gid mappings across all machines. It’s just one of those
 annoying sys-admin tasks that one has to take into account when using NFS.
 To get your permissions back to less permissive settings you will have to
 edit the /etc/passwd and /etc/group files to keep them in sync.

 See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
 for more context.

 - Andrew


 On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:

 First, it is a file permissions issue. I did get snapshots to run when
 I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
 777. Prior to that it was 775 or 755. So, I could revise my question to
 "How can I get snapshots working without using insecure file permissions?"

 root@log-elasticsearch-01:~# mount
 /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
 proc on /proc type proc (rw,noexec,nosuid,nodev)
 sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
 none on /sys/fs/fuse/connections type fusectl (rw)
 none on /sys/kernel/debug type debugfs (rw)
 none on /sys/kernel/security type securityfs (rw)
 udev on /dev type devtmpfs (rw,mode=0755)
 devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
 tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
 none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
 none on /run/shm type tmpfs (rw,nosuid,nodev)
 /dev/sda1 on /boot type ext2 (rw)
 rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
 nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
 (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
 nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
 type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

 root@log-elasticsearch-01:~# ls -ld /mounts
 drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts

 root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
 drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
 /mounts/prod_backup/

 --David Reagan

 On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
 wrote:

> Can you post the output from *mount* and *ls -ld /mounts
> /mounts/prod_backup*?
>
> On 16 March 2015 at 13:33, David Reagan  wrote:
>
>> Why does this happen?
>>
>>
>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true'
>>> -d '{
>>> > "type": "fs",
>>> > "settings": {
>>> > "location": "/mounts/prod_backup/my_backup",
>>> > "compress": true
>>> > }
>>> > }'
>>> {
>>>   "error" :
>>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>>> nested: RepositoryVerificationException[[my_backup]
>>> [vxUQwUTCQwOaLyCy0eMK8A,
>>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>>> nested: RepositoryVerificationException[[my_backup] store location
>>> [/mounts/prod_backup/my_backup] is not accessible on the node
>>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>>> nested:
>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>>> (Permission denied)]; '], [GMTt6Y-3Qle1F

Re: Why does creating a repository fail?

[Mark Walkom]

As has been mentioned, use uid remapping when mounting.

On 16 March 2015 at 17:37, David Reagan  wrote:

> If I were manually creating the elasticsearch user, that'd be easy. But
> I'm relying on apt to do the job for me. So, yeah...
>
> Hmm... I suppose I could manually create an elasticsearch2 user, then
> modify the defaults files to use it when running ES. Still seems clunky...
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden  wrote:
>
>> I’m not that familiar with iSCSI so I hesitate to say for sure, but
>> anytime you are cross-mounting filesystems on Linux you have to take
>> uid/gid consistency into account.
>>
>> - Andrew
>>
>> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
>>
>> Would an iSCSI mount have the same issue? I believe our SAN supports
>> both.
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden  wrote:
>>
>>> Hi David,
>>>
>>> This is a common problem with NFS. Unfortunately the protocol assumes
>>> identical uid/gid mappings across all machines. It’s just one of those
>>> annoying sys-admin tasks that one has to take into account when using NFS.
>>> To get your permissions back to less permissive settings you will have to
>>> edit the /etc/passwd and /etc/group files to keep them in sync.
>>>
>>> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
>>> for more context.
>>>
>>> - Andrew
>>>
>>>
>>> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>>>
>>> First, it is a file permissions issue. I did get snapshots to run when I
>>> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
>>> 777. Prior to that it was 775 or 755. So, I could revise my question to
>>> "How can I get snapshots working without using insecure file permissions?"
>>>
>>> root@log-elasticsearch-01:~# mount
>>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
>>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>>> none on /sys/fs/fuse/connections type fusectl (rw)
>>> none on /sys/kernel/debug type debugfs (rw)
>>> none on /sys/kernel/security type securityfs (rw)
>>> udev on /dev type devtmpfs (rw,mode=0755)
>>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>>> /dev/sda1 on /boot type ext2 (rw)
>>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
>>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
>>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
>>> type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>>>
>>> root@log-elasticsearch-01:~# ls -ld /mounts
>>> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>>>
>>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
>>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
>>> /mounts/prod_backup/
>>>
>>> --David Reagan
>>>
>>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
>>> wrote:
>>>
 Can you post the output from *mount* and *ls -ld /mounts
 /mounts/prod_backup*?

 On 16 March 2015 at 13:33, David Reagan  wrote:

> Why does this happen?
>
>
> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d
>> '{
>> > "type": "fs",
>> > "settings": {
>> > "location": "/mounts/prod_backup/my_backup",
>> > "compress": true
>> > }
>> > }'
>> {
>>   "error" :
>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>> nested: RepositoryVerificationException[[my_backup]
>> [vxUQwUTCQwOaLyCy0eMK8A,
>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>> nested:
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
>> nested:
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
>> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
>> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]

Re: Why does creating a repository fail?

[David Reagan]

@Mark Walkom, So, I'm looking into iscsi. From what I have learned so far,
you actually format the LUN with whatever file system you want. So,
wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or
ext4? Since Ubuntu would treat it like a normal partition and use typical
linux file perms on it.

--David Reagan

On Mon, Mar 16, 2015 at 5:37 PM, David Reagan  wrote:

> If I were manually creating the elasticsearch user, that'd be easy. But
> I'm relying on apt to do the job for me. So, yeah...
>
> Hmm... I suppose I could manually create an elasticsearch2 user, then
> modify the defaults files to use it when running ES. Still seems clunky...
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden  wrote:
>
>> I’m not that familiar with iSCSI so I hesitate to say for sure, but
>> anytime you are cross-mounting filesystems on Linux you have to take
>> uid/gid consistency into account.
>>
>> - Andrew
>>
>> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
>>
>> Would an iSCSI mount have the same issue? I believe our SAN supports
>> both.
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden  wrote:
>>
>>> Hi David,
>>>
>>> This is a common problem with NFS. Unfortunately the protocol assumes
>>> identical uid/gid mappings across all machines. It’s just one of those
>>> annoying sys-admin tasks that one has to take into account when using NFS.
>>> To get your permissions back to less permissive settings you will have to
>>> edit the /etc/passwd and /etc/group files to keep them in sync.
>>>
>>> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
>>> for more context.
>>>
>>> - Andrew
>>>
>>>
>>> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>>>
>>> First, it is a file permissions issue. I did get snapshots to run when I
>>> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
>>> 777. Prior to that it was 775 or 755. So, I could revise my question to
>>> "How can I get snapshots working without using insecure file permissions?"
>>>
>>> root@log-elasticsearch-01:~# mount
>>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
>>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>>> none on /sys/fs/fuse/connections type fusectl (rw)
>>> none on /sys/kernel/debug type debugfs (rw)
>>> none on /sys/kernel/security type securityfs (rw)
>>> udev on /dev type devtmpfs (rw,mode=0755)
>>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>>> /dev/sda1 on /boot type ext2 (rw)
>>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
>>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
>>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01
>>> type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>>>
>>> root@log-elasticsearch-01:~# ls -ld /mounts
>>> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>>>
>>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
>>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
>>> /mounts/prod_backup/
>>>
>>> --David Reagan
>>>
>>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
>>> wrote:
>>>
 Can you post the output from *mount* and *ls -ld /mounts
 /mounts/prod_backup*?

 On 16 March 2015 at 13:33, David Reagan  wrote:

> Why does this happen?
>
>
> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d
>> '{
>> > "type": "fs",
>> > "settings": {
>> > "location": "/mounts/prod_backup/my_backup",
>> > "compress": true
>> > }
>> > }'
>> {
>>   "error" :
>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>> nested: RepositoryVerificationException[[my_backup]
>> [vxUQwUTCQwOaLyCy0eMK8A,
>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>> nested:
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:

Re: Why does creating a repository fail?

[David Reagan]

If I were manually creating the elasticsearch user, that'd be easy. But I'm
relying on apt to do the job for me. So, yeah...

Hmm... I suppose I could manually create an elasticsearch2 user, then
modify the defaults files to use it when running ES. Still seems clunky...

--David Reagan

On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden  wrote:

> I’m not that familiar with iSCSI so I hesitate to say for sure, but
> anytime you are cross-mounting filesystems on Linux you have to take
> uid/gid consistency into account.
>
> - Andrew
>
> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
>
> Would an iSCSI mount have the same issue? I believe our SAN supports both.
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden  wrote:
>
>> Hi David,
>>
>> This is a common problem with NFS. Unfortunately the protocol assumes
>> identical uid/gid mappings across all machines. It’s just one of those
>> annoying sys-admin tasks that one has to take into account when using NFS.
>> To get your permissions back to less permissive settings you will have to
>> edit the /etc/passwd and /etc/group files to keep them in sync.
>>
>> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
>> for more context.
>>
>> - Andrew
>>
>>
>> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>>
>> First, it is a file permissions issue. I did get snapshots to run when I
>> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
>> 777. Prior to that it was 775 or 755. So, I could revise my question to
>> "How can I get snapshots working without using insecure file permissions?"
>>
>> root@log-elasticsearch-01:~# mount
>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>> none on /sys/fs/fuse/connections type fusectl (rw)
>> none on /sys/kernel/debug type debugfs (rw)
>> none on /sys/kernel/security type securityfs (rw)
>> udev on /dev type devtmpfs (rw,mode=0755)
>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>> /dev/sda1 on /boot type ext2 (rw)
>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
>> nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>>
>> root@log-elasticsearch-01:~# ls -ld /mounts
>> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>>
>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
>> /mounts/prod_backup/
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
>> wrote:
>>
>>> Can you post the output from *mount* and *ls -ld /mounts
>>> /mounts/prod_backup*?
>>>
>>> On 16 March 2015 at 13:33, David Reagan  wrote:
>>>
 Why does this happen?


 curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d
> '{
> > "type": "fs",
> > "settings": {
> > "location": "/mounts/prod_backup/my_backup",
> > "compress": true
> > }
> > }'
> {
>   "error" :
> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
> nested: RepositoryVerificationException[[my_backup]
> [vxUQwUTCQwOaLyCy0eMK8A,
> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
> nested: RepositoryVerificationException[[my_backup] store location
> [/mounts/prod_backup/my_backup] is not accessible on the node
> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
> nested:
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
> nested: RepositoryVerificationException[[my_backup] store location
> [/mounts/prod_backup/my_backup] is not accessible on the node
> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
> nested:
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
> nested: RepositoryVerificationException[[my_backup] store location
> [/mounts/prod_backup/my_backup] is not accessible on the node
> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRg

Re: Why does creating a repository fail?

[Mark Walkom]

iSCSI is different, you shouldn't run into this.

On 16 March 2015 at 17:20, Andrew Selden  wrote:

> I’m not that familiar with iSCSI so I hesitate to say for sure, but
> anytime you are cross-mounting filesystems on Linux you have to take
> uid/gid consistency into account.
>
> - Andrew
>
> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
>
> Would an iSCSI mount have the same issue? I believe our SAN supports both.
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden  wrote:
>
>> Hi David,
>>
>> This is a common problem with NFS. Unfortunately the protocol assumes
>> identical uid/gid mappings across all machines. It’s just one of those
>> annoying sys-admin tasks that one has to take into account when using NFS.
>> To get your permissions back to less permissive settings you will have to
>> edit the /etc/passwd and /etc/group files to keep them in sync.
>>
>> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4
>> for more context.
>>
>> - Andrew
>>
>>
>> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>>
>> First, it is a file permissions issue. I did get snapshots to run when I
>> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
>> 777. Prior to that it was 775 or 755. So, I could revise my question to
>> "How can I get snapshots working without using insecure file permissions?"
>>
>> root@log-elasticsearch-01:~# mount
>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>> none on /sys/fs/fuse/connections type fusectl (rw)
>> none on /sys/kernel/debug type debugfs (rw)
>> none on /sys/kernel/security type securityfs (rw)
>> udev on /dev type devtmpfs (rw,mode=0755)
>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>> /dev/sda1 on /boot type ext2 (rw)
>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
>> nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>>
>> root@log-elasticsearch-01:~# ls -ld /mounts
>> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>>
>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
>> /mounts/prod_backup/
>>
>> --David Reagan
>>
>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom 
>> wrote:
>>
>>> Can you post the output from *mount* and *ls -ld /mounts
>>> /mounts/prod_backup*?
>>>
>>> On 16 March 2015 at 13:33, David Reagan  wrote:
>>>
 Why does this happen?


 curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d
> '{
> > "type": "fs",
> > "settings": {
> > "location": "/mounts/prod_backup/my_backup",
> > "compress": true
> > }
> > }'
> {
>   "error" :
> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
> nested: RepositoryVerificationException[[my_backup]
> [vxUQwUTCQwOaLyCy0eMK8A,
> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
> nested: RepositoryVerificationException[[my_backup] store location
> [/mounts/prod_backup/my_backup] is not accessible on the node
> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
> nested:
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
> nested: RepositoryVerificationException[[my_backup] store location
> [/mounts/prod_backup/my_backup] is not accessible on the node
> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
> nested:
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
> nested: RepositoryVerificationException[[my_backup] store location
> [/mounts/prod_backup/my_backup] is not accessible on the node
> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
> nested:
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
> (Permission denied)]; ']]]; ",
>   "status" : 500
> 

Re: Why does creating a repository fail?

[Andrew Selden]

I’m not that familiar with iSCSI so I hesitate to say for sure, but anytime you 
are cross-mounting filesystems on Linux you have to take uid/gid consistency 
into account.

- Andrew

> On Mar 16, 2015, at 4:46 PM, David Reagan  wrote:
> 
> Would an iSCSI mount have the same issue? I believe our SAN supports both.
> 
> --David Reagan
> 
> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden  > wrote:
> Hi David,
> 
> This is a common problem with NFS. Unfortunately the protocol assumes 
> identical uid/gid mappings across all machines. It’s just one of those 
> annoying sys-admin tasks that one has to take into account when using NFS. To 
> get your permissions back to less permissive settings you will have to edit 
> the /etc/passwd and /etc/group files to keep them in sync.
> 
> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 
>  for more 
> context.
> 
> - Andrew
> 
> 
>> On Mar 16, 2015, at 4:04 PM, David Reagan > > wrote:
>> 
>> First, it is a file permissions issue. I did get snapshots to run when I 
>> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is 
>> 777. Prior to that it was 775 or 755. So, I could revise my question to "How 
>> can I get snapshots working without using insecure file permissions?"
>> 
>> root@log-elasticsearch-01:~# mount
>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>> none on /sys/fs/fuse/connections type fusectl (rw)
>> none on /sys/kernel/debug type debugfs (rw)
>> none on /sys/kernel/security type securityfs (rw)
>> udev on /dev type devtmpfs (rw,mode=0755)
>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>> /dev/sda1 on /boot type ext2 (rw)
>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs 
>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type 
>> nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>> 
>> root@log-elasticsearch-01:~# ls -ld /mounts
>> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>> 
>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 
>> /mounts/prod_backup/
>> 
>> --David Reagan
>> 
>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom > > wrote:
>> Can you post the output from mount and ls -ld /mounts /mounts/prod_backup?
>> 
>> On 16 March 2015 at 13:33, David Reagan > > wrote:
>> Why does this happen?
>> 
>> 
>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true 
>> ' -d '{
>> > "type": "fs",
>> > "settings": {
>> > "location": "/mounts/prod_backup/my_backup",
>> > "compress": true
>> > }
>> > }'
>> {
>>   "error" : 
>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>>  nested: RepositoryVerificationException[[my_backup] 
>> [vxUQwUTCQwOaLyCy0eMK8A, 
>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>>  nested: RepositoryVerificationException[[my_backup] store location 
>> [/mounts/prod_backup/my_backup] is not accessible on the node 
>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>>  nested: 
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>>  (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 
>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>>  nested: RepositoryVerificationException[[my_backup] store location 
>> [/mounts/prod_backup/my_backup] is not accessible on the node 
>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
>>  nested: 
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
>>  (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, 
>> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
>>  nested: RepositoryVerificationException[[my_backup] store location 
>> [/mounts/prod_backup/my_backup] is not accessible on the node 
>> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
>>  nested: 
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
>>  (Permission deni

Re: Why does creating a repository fail?

[David Reagan]

Would an iSCSI mount have the same issue? I believe our SAN supports both.

--David Reagan

On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden  wrote:

> Hi David,
>
> This is a common problem with NFS. Unfortunately the protocol assumes
> identical uid/gid mappings across all machines. It’s just one of those
> annoying sys-admin tasks that one has to take into account when using NFS.
> To get your permissions back to less permissive settings you will have to
> edit the /etc/passwd and /etc/group files to keep them in sync.
>
> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 for
> more context.
>
> - Andrew
>
>
> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
>
> First, it is a file permissions issue. I did get snapshots to run when I
> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
> 777. Prior to that it was 775 or 755. So, I could revise my question to
> "How can I get snapshots working without using insecure file permissions?"
>
> root@log-elasticsearch-01:~# mount
> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sda1 on /boot type ext2 (rw)
> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
> nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>
> root@log-elasticsearch-01:~# ls -ld /mounts
> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>
> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
> /mounts/prod_backup/
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom  wrote:
>
>> Can you post the output from *mount* and *ls -ld /mounts
>> /mounts/prod_backup*?
>>
>> On 16 March 2015 at 13:33, David Reagan  wrote:
>>
>>> Why does this happen?
>>>
>>>
>>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{
 > "type": "fs",
 > "settings": {
 > "location": "/mounts/prod_backup/my_backup",
 > "compress": true
 > }
 > }'
 {
   "error" :
 "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
 nested: RepositoryVerificationException[[my_backup]
 [vxUQwUTCQwOaLyCy0eMK8A,
 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
 nested: RepositoryVerificationException[[my_backup] store location
 [/mounts/prod_backup/my_backup] is not accessible on the node
 [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
 nested:
 FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
 (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
 nested: RepositoryVerificationException[[my_backup] store location
 [/mounts/prod_backup/my_backup] is not accessible on the node
 [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
 nested:
 FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
 (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
 nested: RepositoryVerificationException[[my_backup] store location
 [/mounts/prod_backup/my_backup] is not accessible on the node
 [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
 nested:
 FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
 (Permission denied)]; ']]]; ",
   "status" : 500
 }
>>>
>>>
>>>
>>> /mounts/prod_backup is an NFS share mounted on all my ES nodes.
>>>
>>> The only thing I can think of is that file permissions are wrong. But
>>> I'm not sure how to fix that.
>>>
>>> On the two nodes not listed in the error, the elasticsearch gid and uid
>>> are different that on the other three nodes. So I can't just *chown -R
>>> elasticsearch:elasticsearch*.
>>>
>>> Do I have to make sure that the uid and gid are the same on all nod

Re: Why does creating a repository fail?

[Mark Walkom]

You can do a uid remap for NFS mounts, eg
http://www.kernelcrash.com/blog/nfs-uidgid-mapping/2007/09/10/


On 16 March 2015 at 16:04, David Reagan  wrote:

> First, it is a file permissions issue. I did get snapshots to run when I
> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
> 777. Prior to that it was 775 or 755. So, I could revise my question to
> "How can I get snapshots working without using insecure file permissions?"
>
> root@log-elasticsearch-01:~# mount
> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sda1 on /boot type ext2 (rw)
> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
> nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
>
> root@log-elasticsearch-01:~# ls -ld /mounts
> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
>
> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
> /mounts/prod_backup/
>
> --David Reagan
>
> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom  wrote:
>
>> Can you post the output from *mount* and *ls -ld /mounts
>> /mounts/prod_backup*?
>>
>> On 16 March 2015 at 13:33, David Reagan  wrote:
>>
>>> Why does this happen?
>>>
>>>
>>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{
 > "type": "fs",
 > "settings": {
 > "location": "/mounts/prod_backup/my_backup",
 > "compress": true
 > }
 > }'
 {
   "error" :
 "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
 nested: RepositoryVerificationException[[my_backup]
 [vxUQwUTCQwOaLyCy0eMK8A,
 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
 nested: RepositoryVerificationException[[my_backup] store location
 [/mounts/prod_backup/my_backup] is not accessible on the node
 [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
 nested:
 FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
 (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
 nested: RepositoryVerificationException[[my_backup] store location
 [/mounts/prod_backup/my_backup] is not accessible on the node
 [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
 nested:
 FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
 (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
 nested: RepositoryVerificationException[[my_backup] store location
 [/mounts/prod_backup/my_backup] is not accessible on the node
 [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
 nested:
 FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
 (Permission denied)]; ']]]; ",
   "status" : 500
 }
>>>
>>>
>>>
>>> /mounts/prod_backup is an NFS share mounted on all my ES nodes.
>>>
>>> The only thing I can think of is that file permissions are wrong. But
>>> I'm not sure how to fix that.
>>>
>>> On the two nodes not listed in the error, the elasticsearch gid and uid
>>> are different that on the other three nodes. So I can't just *chown -R
>>> elasticsearch:elasticsearch*.
>>>
>>> Do I have to make sure that the uid and gid are the same on all nodes?
>>> How am I supposed to do that when I install from the apt repositories?
>>>
>>> Also, log-estools-01 has *data.node = false*. Why does it still need
>>> the prod_backup mount?
>>>
>>> log-estools-01 is on Ubuntu 14.04.
>>>
>>> Everything else is Ubuntu 12.04.
>>>
>>> Running ES 1.4.4.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegro

Re: Why does creating a repository fail?

[Andrew Selden]

Hi David,

This is a common problem with NFS. Unfortunately the protocol assumes identical 
uid/gid mappings across all machines. It’s just one of those annoying sys-admin 
tasks that one has to take into account when using NFS. To get your permissions 
back to less permissive settings you will have to edit the /etc/passwd and 
/etc/group files to keep them in sync.

See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 for more 
context.

- Andrew


> On Mar 16, 2015, at 4:04 PM, David Reagan  wrote:
> 
> First, it is a file permissions issue. I did get snapshots to run when I 
> chmoded to 777. As you can see from the ls output, /mounts/prod_backup is 
> 777. Prior to that it was 775 or 755. So, I could revise my question to "How 
> can I get snapshots working without using insecure file permissions?"
> 
> root@log-elasticsearch-01:~# mount
> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sda1 on /boot type ext2 (rw)
> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs 
> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type nfs 
> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
> 
> root@log-elasticsearch-01:~# ls -ld /mounts
> drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts
> 
> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 
> /mounts/prod_backup/
> 
> --David Reagan
> 
> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom  > wrote:
> Can you post the output from mount and ls -ld /mounts /mounts/prod_backup?
> 
> On 16 March 2015 at 13:33, David Reagan  > wrote:
> Why does this happen?
> 
> 
> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true 
> ' -d '{
> > "type": "fs",
> > "settings": {
> > "location": "/mounts/prod_backup/my_backup",
> > "compress": true
> > }
> > }'
> {
>   "error" : 
> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>  nested: RepositoryVerificationException[[my_backup] [vxUQwUTCQwOaLyCy0eMK8A, 
> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>  nested: RepositoryVerificationException[[my_backup] store location 
> [/mounts/prod_backup/my_backup] is not accessible on the node 
> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>  nested: 
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>  (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 
> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>  nested: RepositoryVerificationException[[my_backup] store location 
> [/mounts/prod_backup/my_backup] is not accessible on the node 
> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
>  nested: 
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
>  (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, 
> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
>  nested: RepositoryVerificationException[[my_backup] store location 
> [/mounts/prod_backup/my_backup] is not accessible on the node 
> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
>  nested: 
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
>  (Permission denied)]; ']]]; ",
>   "status" : 500
> }
> 
> 
> /mounts/prod_backup is an NFS share mounted on all my ES nodes.
> 
> The only thing I can think of is that file permissions are wrong. But I'm not 
> sure how to fix that.
> 
> On the two nodes not listed in the error, the elasticsearch gid and uid are 
> different that on the other three nodes. So I can't just chown -R 
> elasticsearch:elasticsearch. 
> 
> Do I have to make sure that the uid and gid are the same on all nodes? How am 
> I supposed to do that when I install from the apt repositories?
> 
> Also, log-estools-01 has data.node = false. Why does it still need the 
> pr

Re: Why does creating a repository fail?

[David Reagan]

First, it is a file permissions issue. I did get snapshots to run when I
chmoded to 777. As you can see from the ls output, /mounts/prod_backup is
777. Prior to that it was 775 or 755. So, I could revise my question to
"How can I get snapshots working without using insecure file permissions?"

root@log-elasticsearch-01:~# mount
/dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs
(rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)
nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 type
nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip)

root@log-elasticsearch-01:~# ls -ld /mounts
drwxr-xr-x 6 root root 4096 Oct  1 13:43 /mounts

root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/
drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41
/mounts/prod_backup/

--David Reagan

On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom  wrote:

> Can you post the output from *mount* and *ls -ld /mounts
> /mounts/prod_backup*?
>
> On 16 March 2015 at 13:33, David Reagan  wrote:
>
>> Why does this happen?
>>
>>
>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{
>>> > "type": "fs",
>>> > "settings": {
>>> > "location": "/mounts/prod_backup/my_backup",
>>> > "compress": true
>>> > }
>>> > }'
>>> {
>>>   "error" :
>>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>>> nested: RepositoryVerificationException[[my_backup]
>>> [vxUQwUTCQwOaLyCy0eMK8A,
>>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>>> nested: RepositoryVerificationException[[my_backup] store location
>>> [/mounts/prod_backup/my_backup] is not accessible on the node
>>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>>> nested:
>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>>> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
>>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>>> nested: RepositoryVerificationException[[my_backup] store location
>>> [/mounts/prod_backup/my_backup] is not accessible on the node
>>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
>>> nested:
>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
>>> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
>>> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
>>> nested: RepositoryVerificationException[[my_backup] store location
>>> [/mounts/prod_backup/my_backup] is not accessible on the node
>>> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
>>> nested:
>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
>>> (Permission denied)]; ']]]; ",
>>>   "status" : 500
>>> }
>>
>>
>>
>> /mounts/prod_backup is an NFS share mounted on all my ES nodes.
>>
>> The only thing I can think of is that file permissions are wrong. But I'm
>> not sure how to fix that.
>>
>> On the two nodes not listed in the error, the elasticsearch gid and uid
>> are different that on the other three nodes. So I can't just *chown -R
>> elasticsearch:elasticsearch*.
>>
>> Do I have to make sure that the uid and gid are the same on all nodes?
>> How am I supposed to do that when I install from the apt repositories?
>>
>> Also, log-estools-01 has *data.node = false*. Why does it still need the
>> prod_backup mount?
>>
>> log-estools-01 is on Ubuntu 14.04.
>>
>> Everything else is Ubuntu 12.04.
>>
>> Running ES 1.4.4.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
>> 

Re: Why does creating a repository fail?

[Mark Walkom]

Can you post the output from *mount* and *ls -ld /mounts
/mounts/prod_backup*?

On 16 March 2015 at 13:33, David Reagan  wrote:

> Why does this happen?
>
>
> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{
>> > "type": "fs",
>> > "settings": {
>> > "location": "/mounts/prod_backup/my_backup",
>> > "compress": true
>> > }
>> > }'
>> {
>>   "error" :
>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>> nested: RepositoryVerificationException[[my_backup]
>> [vxUQwUTCQwOaLyCy0eMK8A,
>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>> nested:
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ,
>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
>> nested:
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
>> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw,
>> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
>> nested: RepositoryVerificationException[[my_backup] store location
>> [/mounts/prod_backup/my_backup] is not accessible on the node
>> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
>> nested:
>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
>> (Permission denied)]; ']]]; ",
>>   "status" : 500
>> }
>
>
>
> /mounts/prod_backup is an NFS share mounted on all my ES nodes.
>
> The only thing I can think of is that file permissions are wrong. But I'm
> not sure how to fix that.
>
> On the two nodes not listed in the error, the elasticsearch gid and uid
> are different that on the other three nodes. So I can't just *chown -R
> elasticsearch:elasticsearch*.
>
> Do I have to make sure that the uid and gid are the same on all nodes? How
> am I supposed to do that when I install from the apt repositories?
>
> Also, log-estools-01 has *data.node = false*. Why does it still need the
> prod_backup mount?
>
> log-estools-01 is on Ubuntu 14.04.
>
> Everything else is Ubuntu 12.04.
>
> Running ES 1.4.4.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Why does creating a repository fail?

[David Reagan]

Why does this happen?


curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' -d '{
> > "type": "fs",
> > "settings": {
> > "location": "/mounts/prod_backup/my_backup",
> > "compress": true
> > }
> > }'
> {
>   "error" : 
> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]];
>  
> nested: RepositoryVerificationException[[my_backup] 
> [vxUQwUTCQwOaLyCy0eMK8A, 
> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]];
>  
> nested: RepositoryVerificationException[[my_backup] store location 
> [/mounts/prod_backup/my_backup] is not accessible on the node 
> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300;
>  
> nested: 
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A
>  
> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, 
> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]];
>  
> nested: RepositoryVerificationException[[my_backup] store location 
> [/mounts/prod_backup/my_backup] is not accessible on the node 
> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]];
>  
> nested: 
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ
>  
> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, 
> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]];
>  
> nested: RepositoryVerificationException[[my_backup] store location 
> [/mounts/prod_backup/my_backup] is not accessible on the node 
> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300;
>  
> nested: 
> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw
>  
> (Permission denied)]; ']]]; ",
>   "status" : 500
> }



/mounts/prod_backup is an NFS share mounted on all my ES nodes.

The only thing I can think of is that file permissions are wrong. But I'm 
not sure how to fix that.

On the two nodes not listed in the error, the elasticsearch gid and uid are 
different that on the other three nodes. So I can't just *chown -R 
elasticsearch:elasticsearch*. 

Do I have to make sure that the uid and gid are the same on all nodes? How 
am I supposed to do that when I install from the apt repositories?

Also, log-estools-01 has *data.node = false*. Why does it still need the 
prod_backup mount?

log-estools-01 is on Ubuntu 14.04.

Everything else is Ubuntu 12.04.

Running ES 1.4.4.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.