[Gluster-devel] Unplanned Jenkins maintenance

[Nigel Babu]

Hello folks,

I did a quick unplanned Jenkins maintenance today to upgrade 3 plugins with
security issues in them. This is now complete. There was a brief period
where we did not start new jobs until Jenkins restarted. There should have
been no interruption of existing jobs or any jobs canceled. Please file a
bug if you notice something wrong post-upgrade.

-- 
nigelb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Branched and further dates

[Sanju Rakonde]

On Wed, Sep 26, 2018 at 7:53 PM Shyam Ranganathan 
wrote:

> Hi,
>
> Updates on the release and shout out for help is as follows,
>
> RC0 Release packages for testing are available see the thread at [1]
>
> These are the following activities that we need to complete for calling
> the release as GA (with no major regressions i.e):
>
> 1. Release notes (Owner: release owner (myself), will send out an
> initial version for review and to solicit inputs today)
>
> 2. Testing dashboard to maintain release health (new, thanks Nigel)
>   - Dashboard at [2]
>   - We already have 3 failures here as follows, needs attention from
> appropriate *maintainers*,
> (a)
>
> https://build.gluster.org/job/regression-test-with-multiplex/871/consoleText
> - Failed with core:
> ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t
> (b)
>
> https://build.gluster.org/job/regression-test-with-multiplex/873/consoleText
> - Failed with core: ./tests/bugs/snapshot/bug-1275616.t
> - Also test ./tests/bugs/glusterd/validating-server-quorum.t had
> to be
> retried
>

The test case ./tests/bugs/glusterd/validating-server-quorum.t had to be
retried since, it got timed out at the first run.
I went through the logs of first run, everything looks fine. Looking at all
the time stamps, got to know that cluster_brick_up_status took 45sec
(PROCESS_UP_TIMEOUT) most of the times when it is used. As we clubbed many
of the glusterd test cases into a single test case, the test case might
need some more time to execute. If this test case gets timed out
repeatedly, we will think of the actions need to be taken.

Definition of cluster_brick_up_status for your reference:
function cluster_brick_up_status {
local vol=$2
local host=$3
local brick=$4
eval \$CLI_$1 volume status $vol $host:$brick --xml | sed -ne
's/.*\([01]\)<\/status>/\1/p'
}

(c)
> https://build.gluster.org/job/regression-test-burn-in/4109/consoleText
> - Failed with core: ./tests/basic/mgmt_v3-locks.t
>
> 3. Upgrade testing
>   - Need *volunteers* to do the upgrade testing as stated in the 4.1
> upgrade guide [3] to note any differences or changes to the same
>   - Explicit call out on *disperse* volumes, as we continue to state
> online upgrade is not possible, is this addressed and can this be tested
> and the documentation improved around the same?
>
> 4. Performance testing/benchmarking
>   - I would be using smallfile and FIO to baseline 3.12 and 4.1 and test
> RC0 for any major regressions
>   - If we already know of any please shout out so that we are aware of
> the problems and upcoming fixes to the same
>
> 5. Major testing areas
>   - Py3 support: Need *volunteers* here to test out the Py3 support
> around changed python files, if there is not enough coverage in the
> regression test suite for the same
>
> Thanks,
> Shyam
>
> [1] Packages for RC0:
> https://lists.gluster.org/pipermail/maintainers/2018-September/005044.html
>
> [2] Release testing health dashboard:
> https://build.gluster.org/job/nightly-release-5/
>
> [3] 4.1 upgrade guide:
> https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_4.1/
>
> On 09/13/2018 11:10 AM, Shyam Ranganathan wrote:
> > Hi,
> >
> > Release 5 has been branched today. To backport fixes to the upcoming 5.0
> > release use the tracker bug [1].
> >
> > We intend to roll out RC0 build by end of tomorrow for testing, unless
> > the set of usual cleanup patches (op-version, some messages, gfapi
> > version) land in any form of trouble.
> >
> > RC1 would be around 24th of Sep. with final release tagging around 1st
> > of Oct.
> >
> > I would like to encourage everyone to test out the bits as appropriate
> > and post updates to this thread.
> >
> > Thanks,
> > Shyam
> >
> > [1] 5.0 tracker:
> https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-5.0
> > ___
> > maintainers mailing list
> > maintain...@gluster.org
> > https://lists.gluster.org/mailman/listinfo/maintainers
> >
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>


-- 
Thanks,
Sanju
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Niels de Vos]

On Thu, Sep 27, 2018 at 08:40:54AM -0400, Shyam Ranganathan wrote:
> On 09/27/2018 08:07 AM, Kaleb S. KEITHLEY wrote:
> >> The thought is,
> >> - Add a configure option "--enable-py-version-correction" to configure,
> >> that is disabled by default
> > "correction" implies there's something that's incorrect. How about
> > "conversion" or perhaps just --enable-python2
> > 
> 
> I would not like to go with --enable-python2 as that implies it is an
> conscious choice with the understanding that py2 is on the box. Given
> the current ability to detect and hence correct the python shebangs, I
> would think we should retain it as a more detect and modify the shebangs
> option name. (I am looking at this more as an option that does the right
> thing implicitly than someone/tool using this checking explicitly, which
> can mean different things to different people, if that makes sense)
> 
> Now "correction" seems like an overkill, maybe "conversion"?

Is it really needed to have this as an option? Instead of an option in
configure.ac, can it not be a post-install task in a Makefile.am? The
number of executable python scripts that get installed are minimal, so I
do not expect that a lot of changes are needed for this.

There do seem quite some Python files that have a shebang, but do not
need it (__init__.py, not executable, no __main__-like functions). This
should probably get reviewed as well. When those scripts get their
shebang removed, even fewer files need to be 'fixed-up'.

Is there a BZ or GitHub Issue that I can use to send some fixes?

Thanks,
Niels


$ for F in $(git ls-files) ; do if ( head -n1 $F | grep -q -E '^#.+python' 
) ; then echo "$F uses Python" ; fi ; done
api/examples/getvolfile.py uses Python
events/eventskeygen.py uses Python
events/src/gf_event.py uses Python
events/src/glustereventsd.py uses Python
events/src/peer_eventsapi.py uses Python
events/tools/eventsdash.py uses Python
extras/create_new_xlator/generate_xlator.py uses Python
extras/distributed-testing/distributed-test-runner.py uses Python
extras/failed-tests.py uses Python
extras/geo-rep/schedule_georep.py.in uses Python
extras/git-branch-diff.py uses Python
extras/gnfs-loganalyse.py uses Python
extras/hook-scripts/S40ufo-stop.py uses Python
extras/profiler/glusterfs-profiler uses Python
extras/quota/quota_fsck.py uses Python
extras/quota/xattr_analysis.py uses Python
extras/rebalance.py uses Python
extras/snap_scheduler/conf.py.in uses Python
extras/snap_scheduler/gcron.py uses Python
extras/snap_scheduler/snap_scheduler.py uses Python
geo-replication/src/peer_georep-sshkey.py.in uses Python
geo-replication/src/peer_mountbroker.in uses Python
geo-replication/src/peer_mountbroker.py.in uses Python
geo-replication/syncdaemon/changelogagent.py uses Python
geo-replication/syncdaemon/conf.py.in uses Python
geo-replication/syncdaemon/gsyncd.py uses Python
geo-replication/syncdaemon/gsyncdstatus.py uses Python
geo-replication/tests/__init__.py uses Python
geo-replication/tests/unit/__init__.py uses Python
geo-replication/tests/unit/test_gsyncdstatus.py uses Python
geo-replication/tests/unit/test_syncdutils.py uses Python
libglusterfs/src/gen-defaults.py uses Python
libglusterfs/src/generator.py uses Python
tools/gfind_missing_files/gfid_to_path.py uses Python
tools/glusterfind/S57glusterfind-delete-post.py uses Python
tools/glusterfind/glusterfind.in uses Python
tools/glusterfind/src/brickfind.py uses Python
tools/glusterfind/src/changelog.py uses Python
tools/glusterfind/src/main.py uses Python
tools/glusterfind/src/nodeagent.py uses Python
xlators/experimental/fdl/src/gen_dumper.py uses Python
xlators/experimental/fdl/src/gen_fdl.py uses Python
xlators/experimental/fdl/src/gen_recon.py uses Python
xlators/experimental/jbr-client/src/gen-fops.py uses Python
xlators/experimental/jbr-server/src/gen-fops.py uses Python
xlators/features/changelog/lib/examples/python/changes.py uses Python
xlators/features/cloudsync/src/cloudsync-fops-c.py uses Python
xlators/features/cloudsync/src/cloudsync-fops-h.py uses Python
xlators/features/glupy/src/__init__.py.in uses Python
xlators/features/utime/src/utime-gen-fops-c.py uses Python
xlators/features/utime/src/utime-gen-fops-h.py uses Python

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] gluster-ansible: status of the project

[Sachidananda URS]

Hi,

gluster-ansible project is aimed at automating the deployment and
maintenance of GlusterFS cluster.

The project can be found at:

* https://github.com/gluster/gluster-ansible
* https://github.com/gluster/gluster-ansible-infra
* https://github.com/gluster/gluster-ansible-features
* https://github.com/gluster/gluster-ansible-maintenance
* https://github.com/gluster/gluster-ansible-cluster
* https://github.com/gluster/gluster-ansible-repositories

We have the python bindings for GlusterD2 API, and can be found at:

https://github.com/gluster/python-gluster-mgmt-client

The goal is to use the python bindings in gluster_ansible module to make it
work with GlusterD2.

Current status of the project:

* We have the initial working roles, packages are available at:
   - https://copr.fedorainfracloud.org/coprs/sac/gluster-ansible/builds/

* The initial version supports:
   - End-to-end deployment of Gluster Hyperconverged Infrastructure.
   - GlusterFS volume management
   - GlusterFS brick setup
   - Packages and repository management

* Autogeneration of python bindings for GlusterD2 is being worked by
Sidharth (https://github.com/sidharthanup) and available at:
  -
https://github.com/sidharthanup/GD2_API/blob/master/testgen/glusterapi_README.md

The GD2 API python project will be merged into python-gluster-mgmt-client.

* Ansible modules (WIP):
   - New module: Facts module for self-heal and rebalance. Devyani is
working on these modules.
 https://github.com/ansible/ansible/pull/45997 - self-heal
   - Remove brick feature for gluster_ansible module:
 https://github.com/ansible/ansible/pull/38269

* Sheersha and Nigel are working on continuous integration, and PR is at:
   - https://github.com/gluster/gluster-ansible-infra/pull/29
   - https://github.com/gluster/gluster-ansible-infra/pull/26

The CI work is in progress and will be integrated soon. Which will help us
to keep the repository
in stable condition.

Questions and feedback are welcome.

-sac
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Gluster Weekly Report : Static Analyser

[Sunny Kumar]

Hello folks,

It's been a month since we shared latest update on coverity scan [1]
and clang-scan report, so the current status of static analyser is
below:

Coverity scan status:
We made further progress in reducing the coverity count from 269 (28th
Aug) to 180 (today).
Major contributors - Sunny (15 patches), Sanju (7 patches).

Clang-scan status:
We successfully reduced clang-scan bugs from 151(28th Aug)  to 119 (today).
Major contributors - Amar (11 fixes), Shwetha and Harpreet (3 patches each).

>From now on I will be sharing this report weekly. If you want to
contribute in fixing coverity and clang-scan bugs please follow these
instruction:
 * for coverity scan fixes:
https://lists.gluster.org/pipermail/gluster-devel/2018-August/055155.html
 * for clang-scan:
https://lists.gluster.org/pipermail/gluster-devel/2018-August/055212.html


 [1]. Coverity cleanup drive report
https://lists.gluster.org/pipermail/gluster-devel/2018-August/055342.html

Regards,
Sunny kumar
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Shyam Ranganathan]

On 09/28/2018 06:12 AM, Niels de Vos wrote:
> On Thu, Sep 27, 2018 at 08:40:54AM -0400, Shyam Ranganathan wrote:
>> On 09/27/2018 08:07 AM, Kaleb S. KEITHLEY wrote:
 The thought is,
 - Add a configure option "--enable-py-version-correction" to configure,
 that is disabled by default
>>> "correction" implies there's something that's incorrect. How about
>>> "conversion" or perhaps just --enable-python2
>>>
>> I would not like to go with --enable-python2 as that implies it is an
>> conscious choice with the understanding that py2 is on the box. Given
>> the current ability to detect and hence correct the python shebangs, I
>> would think we should retain it as a more detect and modify the shebangs
>> option name. (I am looking at this more as an option that does the right
>> thing implicitly than someone/tool using this checking explicitly, which
>> can mean different things to different people, if that makes sense)
>>
>> Now "correction" seems like an overkill, maybe "conversion"?
> Is it really needed to have this as an option? Instead of an option in
> configure.ac, can it not be a post-install task in a Makefile.am? The
> number of executable python scripts that get installed are minimal, so I
> do not expect that a lot of changes are needed for this.

Here is how I summarize this proposal,
- Perform the shebang "correction" for py2 in the post install
  - Keeps the git clone clean
- shebang correction occurs based on a configure time option
  - It is not implicit but an explicit choice to correct the shebangs to
py2, hence we need an option either way
- The configure option would be "--enable-python2"
  - Developers that need py2, can configure it as such
  - Regression jobs that need py2, either because of the platform they
test against, or for py2 compliance in the future, use the said option
  - Package builds are agnostic to these changes (currently) as they
decide at build time based on the platform what needs to be done.

> 
> There do seem quite some Python files that have a shebang, but do not
> need it (__init__.py, not executable, no __main__-like functions). This
> should probably get reviewed as well. When those scripts get their
> shebang removed, even fewer files need to be 'fixed-up'.

I propose maintainers/component-owner take this cleanup.

> 
> Is there a BZ or GitHub Issue that I can use to send some fixes?

See: https://github.com/gluster/glusterfs/issues/411

Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Shyam Ranganathan]

On 09/28/2018 12:36 AM, Kotresh Hiremath Ravishankar wrote:
> > - All regression hosts are currently py2 and so if we do not run
> the py
> > shebang correction during configure (as we do not build and test from
> > RPMS), we would be running with incorrect py3 shebangs (although this
> > seems to work, see [2]. @kotresh can we understand why?)
> 
> Is it because we don't test any of the python in the regression tests?
> 
> Geo-replication do have regression tests but not sure about glusterfind,
> events.
> 
> Or because when we do, we invoke python scripts with `python foo.py` or
> `$PYTHON foo.py` everywhere? The shebangs are ignored when scripts are
> invoked this way.
> 
> The reason why geo-rep is passing is for the same reason mentioned. Geo-rep
> python file is invoked from a c program always prefixing it with python
> as follows.
> 
> python = getenv("PYTHON");
>     if (!python)
>     python = PYTHON;
>     nargv[j++] = python;
>     nargv[j++] = GSYNCD_PREFIX "/python/syncdaemon/" GSYNCD_PY;

Thank you, makes sense now.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Niels de Vos]

On Fri, Sep 28, 2018 at 08:57:06AM -0400, Shyam Ranganathan wrote:
> On 09/28/2018 06:12 AM, Niels de Vos wrote:
> > On Thu, Sep 27, 2018 at 08:40:54AM -0400, Shyam Ranganathan wrote:
> >> On 09/27/2018 08:07 AM, Kaleb S. KEITHLEY wrote:
>  The thought is,
>  - Add a configure option "--enable-py-version-correction" to configure,
>  that is disabled by default
> >>> "correction" implies there's something that's incorrect. How about
> >>> "conversion" or perhaps just --enable-python2
> >>>
> >> I would not like to go with --enable-python2 as that implies it is an
> >> conscious choice with the understanding that py2 is on the box. Given
> >> the current ability to detect and hence correct the python shebangs, I
> >> would think we should retain it as a more detect and modify the shebangs
> >> option name. (I am looking at this more as an option that does the right
> >> thing implicitly than someone/tool using this checking explicitly, which
> >> can mean different things to different people, if that makes sense)
> >>
> >> Now "correction" seems like an overkill, maybe "conversion"?
> > Is it really needed to have this as an option? Instead of an option in
> > configure.ac, can it not be a post-install task in a Makefile.am? The
> > number of executable python scripts that get installed are minimal, so I
> > do not expect that a lot of changes are needed for this.
> 
> Here is how I summarize this proposal,
> - Perform the shebang "correction" for py2 in the post install
>   - Keeps the git clone clean
> - shebang correction occurs based on a configure time option
>   - It is not implicit but an explicit choice to correct the shebangs to
> py2, hence we need an option either way
> - The configure option would be "--enable-python2"
>   - Developers that need py2, can configure it as such
>   - Regression jobs that need py2, either because of the platform they
> test against, or for py2 compliance in the future, use the said option
>   - Package builds are agnostic to these changes (currently) as they
> decide at build time based on the platform what needs to be done.

I do not think such a ./configure option is needed. configure.ac can
find out the version that is available, and pick python3 if it has both.

Tests should just run with "$PYTHON run-the-test.py" instead of
./run-the-test.py with a #!/usr/bin/python shebang. The testing
framework can also find out what version of python is available.


> > There do seem quite some Python files that have a shebang, but do not
> > need it (__init__.py, not executable, no __main__-like functions). This
> > should probably get reviewed as well. When those scripts get their
> > shebang removed, even fewer files need to be 'fixed-up'.
> 
> I propose maintainers/component-owner take this cleanup.

That would be ideal!


> > Is there a BZ or GitHub Issue that I can use to send some fixes?
> 
> See: https://github.com/gluster/glusterfs/issues/411

Thanks,
Niels
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Shyam Ranganathan]

On 09/28/2018 09:11 AM, Niels de Vos wrote:
> On Fri, Sep 28, 2018 at 08:57:06AM -0400, Shyam Ranganathan wrote:
>> On 09/28/2018 06:12 AM, Niels de Vos wrote:
>>> On Thu, Sep 27, 2018 at 08:40:54AM -0400, Shyam Ranganathan wrote:
 On 09/27/2018 08:07 AM, Kaleb S. KEITHLEY wrote:
>> The thought is,
>> - Add a configure option "--enable-py-version-correction" to configure,
>> that is disabled by default
> "correction" implies there's something that's incorrect. How about
> "conversion" or perhaps just --enable-python2
>
 I would not like to go with --enable-python2 as that implies it is an
 conscious choice with the understanding that py2 is on the box. Given
 the current ability to detect and hence correct the python shebangs, I
 would think we should retain it as a more detect and modify the shebangs
 option name. (I am looking at this more as an option that does the right
 thing implicitly than someone/tool using this checking explicitly, which
 can mean different things to different people, if that makes sense)

 Now "correction" seems like an overkill, maybe "conversion"?
>>> Is it really needed to have this as an option? Instead of an option in
>>> configure.ac, can it not be a post-install task in a Makefile.am? The
>>> number of executable python scripts that get installed are minimal, so I
>>> do not expect that a lot of changes are needed for this.
>>
>> Here is how I summarize this proposal,
>> - Perform the shebang "correction" for py2 in the post install
>>   - Keeps the git clone clean
>> - shebang correction occurs based on a configure time option
>>   - It is not implicit but an explicit choice to correct the shebangs to
>> py2, hence we need an option either way
>> - The configure option would be "--enable-python2"
>>   - Developers that need py2, can configure it as such
>>   - Regression jobs that need py2, either because of the platform they
>> test against, or for py2 compliance in the future, use the said option
>>   - Package builds are agnostic to these changes (currently) as they
>> decide at build time based on the platform what needs to be done.
> 
> I do not think such a ./configure option is needed. configure.ac can
> find out the version that is available, and pick python3 if it has both.
> 
> Tests should just run with "$PYTHON run-the-test.py" instead of
> ./run-the-test.py with a #!/usr/bin/python shebang. The testing
> framework can also find out what version of python is available.

If we back up a bit here, if all shebangs are cleared, then we do not
need anything. That is not the situation at the moment, and neither do I
know if that state can be reached.

We also need to ensure we work against py2 and py3 for the near future,
which entails being specific in some regression job at least on the
python choice, does that correct the shebangs really depends on the
above conclusion.

> 
> 
>>> There do seem quite some Python files that have a shebang, but do not
>>> need it (__init__.py, not executable, no __main__-like functions). This
>>> should probably get reviewed as well. When those scripts get their
>>> shebang removed, even fewer files need to be 'fixed-up'.
>>
>> I propose maintainers/component-owner take this cleanup.
> 
> That would be ideal!
> 
> 
>>> Is there a BZ or GitHub Issue that I can use to send some fixes?
>>
>> See: https://github.com/gluster/glusterfs/issues/411
> 
> Thanks,
> Niels
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Niels de Vos]

On Fri, Sep 28, 2018 at 09:14:15AM -0400, Shyam Ranganathan wrote:
> On 09/28/2018 09:11 AM, Niels de Vos wrote:
> > On Fri, Sep 28, 2018 at 08:57:06AM -0400, Shyam Ranganathan wrote:
> >> On 09/28/2018 06:12 AM, Niels de Vos wrote:
> >>> On Thu, Sep 27, 2018 at 08:40:54AM -0400, Shyam Ranganathan wrote:
>  On 09/27/2018 08:07 AM, Kaleb S. KEITHLEY wrote:
> >> The thought is,
> >> - Add a configure option "--enable-py-version-correction" to configure,
> >> that is disabled by default
> > "correction" implies there's something that's incorrect. How about
> > "conversion" or perhaps just --enable-python2
> >
>  I would not like to go with --enable-python2 as that implies it is an
>  conscious choice with the understanding that py2 is on the box. Given
>  the current ability to detect and hence correct the python shebangs, I
>  would think we should retain it as a more detect and modify the shebangs
>  option name. (I am looking at this more as an option that does the right
>  thing implicitly than someone/tool using this checking explicitly, which
>  can mean different things to different people, if that makes sense)
> 
>  Now "correction" seems like an overkill, maybe "conversion"?
> >>> Is it really needed to have this as an option? Instead of an option in
> >>> configure.ac, can it not be a post-install task in a Makefile.am? The
> >>> number of executable python scripts that get installed are minimal, so I
> >>> do not expect that a lot of changes are needed for this.
> >>
> >> Here is how I summarize this proposal,
> >> - Perform the shebang "correction" for py2 in the post install
> >>   - Keeps the git clone clean
> >> - shebang correction occurs based on a configure time option
> >>   - It is not implicit but an explicit choice to correct the shebangs to
> >> py2, hence we need an option either way
> >> - The configure option would be "--enable-python2"
> >>   - Developers that need py2, can configure it as such
> >>   - Regression jobs that need py2, either because of the platform they
> >> test against, or for py2 compliance in the future, use the said option
> >>   - Package builds are agnostic to these changes (currently) as they
> >> decide at build time based on the platform what needs to be done.
> > 
> > I do not think such a ./configure option is needed. configure.ac can
> > find out the version that is available, and pick python3 if it has both.
> > 
> > Tests should just run with "$PYTHON run-the-test.py" instead of
> > ./run-the-test.py with a #!/usr/bin/python shebang. The testing
> > framework can also find out what version of python is available.
> 
> If we back up a bit here, if all shebangs are cleared, then we do not
> need anything. That is not the situation at the moment, and neither do I
> know if that state can be reached.

Not all shebangs need to go away, only the ones for the test-cases. A
post-install hook can modify the shebangs from python3 to python2
depending on what ./configure detected.

> We also need to ensure we work against py2 and py3 for the near future,
> which entails being specific in some regression job at least on the
> python choice, does that correct the shebangs really depends on the
> above conclusion.

Ok, so if both python2 and python3 are available, and you want to run
with python2, just run "python2 my-py2-test.py"?

Niels
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Kaleb S. KEITHLEY]

On 9/28/18 9:38 AM, Niels de Vos wrote:
>>>
>>> Tests should just run with "$PYTHON run-the-test.py" instead of
>>> ./run-the-test.py with a #!/usr/bin/python shebang. The testing
>>> framework can also find out what version of python is available.
>>
>> If we back up a bit here, if all shebangs are cleared, then we do not
>> need anything. That is not the situation at the moment, and neither do I
>> know if that state can be reached.
> 
> Not all shebangs need to go away, only the ones for the test-cases. A
> post-install hook can modify the shebangs from python3 to python2
> depending on what ./configure detected.

None of the .py files in .../tests/... have shebangs (master and
release-5 branch).

They are all invoked with ...$PYTHON some-test.py ...

All but one of them were always invoked that way. socket-as-fifo.py was
the only one that was not, and that has been fixed.

-- 

Kaleb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Python3 build process

[Kaleb S. KEITHLEY]

On 9/28/18 6:12 AM, Niels de Vos wrote:
> Is it really needed to have this as an option? Instead of an option in
> configure.ac, can it not be a post-install task in a Makefile.am? 

I don't fully understand how .pyc and .pyo files are used, or how the
new-in-python3 __cache__ files are used, but they seem to be created
during the build and/or the install.

What does it mean to build+install the .pyc, .pyo, and __cache__ files
and then go in after and whack the shebangs of the .py files?

-- 

Kaleb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Status update on "Hyperconverged Gluster oVirt support"

[Gobinda Das]

Hi All,
 Status update on "Hyperconverged Gluster oVirt support"

Features Completed:


  cockpit-ovirt
  -
  1- Asymmetric brick configuration.Brick can be configured per host basis
i.e. If the user wanted to make use of sdb from host1, sdc from host2, and
sdd from host3.
  2- Dedupe and Compression integration via VDO support (see
https://github.com/dm-vdo/kvdo). Gluster bricks are created on vdo devices
  3- LVM cache configuration support (Configure cache by using fast block
device such as SSD drive to imrove the performance of a larger and slower
logical volumes)
  4- Auto addition of 2nd and 3rd hosts in a 3 node setup during deployment
  5- Auto creation of storage domains based on gluster volumes created
during setup
  6- Single node deployment support via Cockpit UI. For details on single
node deployment -
https://www.ovirt.org/documentation/gluster-hyperconverged/chap-Single_node_hyperconverged/
  7- Gluster Management Dashboard (Dashboard will show the nodes in
cluster,Volumes and bricks. User can expand the cluster and also can create
new volume in existing cluster nodes )

  oVirt
  ---
  1- Reset brick support from UI to allow users to replace a faulty brick
  2- Create brick from engine now supports configuring an SSD device as
lvmcache device when bricks are created on spinning disks
  3- VDO monitoring

 GlusterFS
---
 Enhancements to performance with fuse by 15x
 1. Cluster after eager lock change for better detection of multiple clients
 2. Changing qemu option aio to "native" instead of "threads".

 end-to-end deployment:
 
 1- End to end deployment of a Gluster + Ovirt hyperconverged environment
using ansible roles (
https://github.com/gluster/gluster-ansible/tree/master/playbooks ). The
only pre-requisite is a CentOS node/oVirt node

Future Plan:
==
 cockpit-ovirt:

  1- ansible-roles integration for deployment
  2- Support for different volume types

 vdsm:
  1- Python3 compatibility of vdsm-gluster
  2- Native 4K support

-- 
Thanks,
Gobinda
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Memory overwrites due to processing vol files???

[FNU Raghavendra Manjunath]

I was looking into the issue and  this is what I could find while working
with shyam.

There are 2 things here.

1) The multiplexed brick process for the snapshot(s) getting the client
volfile (I suspect, it happened
 when restore operation was performed).
2) Memory corruption happening while the multiplexed brick process is
building the graph (for the client
 volfile it got above)

I have been able to reproduce the issue in my local computer once, when I
ran the testcase tests/bugs/snapshot/bug-1275616.t

Upon comparison, we found that the backtrace of the core I got and the core
generated in the regression runs was similar.
In fact, the victim information shyam mentioned before, is also similar in
the core that I was able to get.

On top of that, when the brick process was run with valgrind, it reported
following memory corruption

==31257== Conditional jump or move depends on uninitialised value(s)
==31257==at 0x1A7D0564: mdc_xattr_list_populate (md-cache.c:3127)
==31257==by 0x1A7D1903: mdc_init (md-cache.c:3486)
==31257==by 0x4E62D41: __xlator_init (xlator.c:684)
==31257==by 0x4E62E67: xlator_init (xlator.c:709)
==31257==by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
==31257==by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
==31257==by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
==31257==by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
==31257==by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
==31257==by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)
==31257==by 0x518084E: rpc_transport_notify (rpc-transport.c:525)
==31257==by 0x123273DF: socket_event_poll_in (socket.c:2504)
==31257==  Uninitialised value was created by a heap allocation
==31257==at 0x4C2DB9D: malloc (vg_replace_malloc.c:299)
==31257==by 0x4E9F58E: __gf_malloc (mem-pool.c:136)
==31257==by 0x1A7D052A: mdc_xattr_list_populate (md-cache.c:3123)
==31257==by 0x1A7D1903: mdc_init (md-cache.c:3486)
==31257==by 0x4E62D41: __xlator_init (xlator.c:684)
==31257==by 0x4E62E67: xlator_init (xlator.c:709)
==31257==by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
==31257==by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
==31257==by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
==31257==by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
==31257==by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
==31257==by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)

Based on the above observations, I think the below patch  by Shyam should
fix the crash.
https://review.gluster.org/#/c/glusterfs/+/21299/

But, I am still trying understand, why a brick process should get a client
volfile (i.e. the 1st issue mentioned above).

Regards,
Raghavendra

On Wed, Sep 26, 2018 at 9:00 PM Shyam Ranganathan 
wrote:

> On 09/26/2018 10:21 AM, Shyam Ranganathan wrote:
> > 2. Testing dashboard to maintain release health (new, thanks Nigel)
> >   - Dashboard at [2]
> >   - We already have 3 failures here as follows, needs attention from
> > appropriate *maintainers*,
> > (a)
> >
> https://build.gluster.org/job/regression-test-with-multiplex/871/consoleText
> >   - Failed with core:
> ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t
> > (b)
> >
> https://build.gluster.org/job/regression-test-with-multiplex/873/consoleText
> >   - Failed with core: ./tests/bugs/snapshot/bug-1275616.t
> >   - Also test ./tests/bugs/glusterd/validating-server-quorum.t had
> to be
> > retried
>
> I was looking at the cores from the above 2 instances, the one in job
> 873 is been a typical pattern, where malloc fails as there is internal
> header corruption in the free bins.
>
> When examining the victim that would have been allocated, it is often
> carrying incorrect size and other magic information. If the data in
> victim is investigated it looks like a volfile.
>
> With the crash in 871, I thought there maybe a point where this is
> detected earlier, but not able to make headway in the same.
>
> So, what could be corrupting this memory and is it when the graph is
> being processed? Can we run this with ASAN or such (I have not tried,
> but need pointers if anyone has run tests with ASAN).
>
> Here is the (brief) stack analysis of the core in 873:
> NOTE: we need to start avoiding flushing the logs when we are dumping
> core, as that leads to more memory allocations and causes a sort of
> double fault in such cases.
>
> Core was generated by `/build/install/sbin/glusterfsd -s
> builder101.cloud.gluster.org --volfile-id /sn'.
> Program terminated with signal 6, Aborted.
> #0  0x7f23cf590277 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> (gdb) bt
> #0  0x7f23cf590277 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x7f23cf591968 in __GI_abort () at abort.c:90
> #2  0x7f23cf5d2d37 in __l

Re: [Gluster-devel] [Gluster-Maintainers] Memory overwrites due to processing vol files???

[Shyam Ranganathan]

We tested with ASAN and without the fix at [1], and it consistently
crashes at the mdcache xlator when brick mux is enabled.
On 09/28/2018 03:50 PM, FNU Raghavendra Manjunath wrote:
> 
> I was looking into the issue and  this is what I could find while
> working with shyam.
> 
> There are 2 things here.
> 
> 1) The multiplexed brick process for the snapshot(s) getting the client
> volfile (I suspect, it happened
>      when restore operation was performed).
> 2) Memory corruption happening while the multiplexed brick process is
> building the graph (for the client
>      volfile it got above)
> 
> I have been able to reproduce the issue in my local computer once, when
> I ran the testcase tests/bugs/snapshot/bug-1275616.t
> 
> Upon comparison, we found that the backtrace of the core I got and the
> core generated in the regression runs was similar.
> In fact, the victim information shyam mentioned before, is also similar
> in the core that I was able to get.  
> 
> On top of that, when the brick process was run with valgrind, it
> reported following memory corruption
> 
> ==31257== Conditional jump or move depends on uninitialised value(s)
> ==31257==    at 0x1A7D0564: mdc_xattr_list_populate (md-cache.c:3127)
> ==31257==    by 0x1A7D1903: mdc_init (md-cache.c:3486)
> ==31257==    by 0x4E62D41: __xlator_init (xlator.c:684)
> ==31257==    by 0x4E62E67: xlator_init (xlator.c:709)
> ==31257==    by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
> ==31257==    by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
> ==31257==    by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
> ==31257==    by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
> ==31257==    by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
> ==31257==    by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)
> ==31257==    by 0x518084E: rpc_transport_notify (rpc-transport.c:525)
> ==31257==    by 0x123273DF: socket_event_poll_in (socket.c:2504)
> ==31257==  Uninitialised value was created by a heap allocation
> ==31257==    at 0x4C2DB9D: malloc (vg_replace_malloc.c:299)
> ==31257==    by 0x4E9F58E: __gf_malloc (mem-pool.c:136)
> ==31257==    by 0x1A7D052A: mdc_xattr_list_populate (md-cache.c:3123)
> ==31257==    by 0x1A7D1903: mdc_init (md-cache.c:3486)
> ==31257==    by 0x4E62D41: __xlator_init (xlator.c:684)
> ==31257==    by 0x4E62E67: xlator_init (xlator.c:709)
> ==31257==    by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
> ==31257==    by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
> ==31257==    by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
> ==31257==    by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
> ==31257==    by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
> ==31257==    by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)
> 
> Based on the above observations, I think the below patch  by Shyam
> should fix the crash.

[1]

> https://review.gluster.org/#/c/glusterfs/+/21299/
> 
> But, I am still trying understand, why a brick process should get a
> client volfile (i.e. the 1st issue mentioned above). 
> 
> Regards,
> Raghavendra
> 
> On Wed, Sep 26, 2018 at 9:00 PM Shyam Ranganathan  > wrote:
> 
> On 09/26/2018 10:21 AM, Shyam Ranganathan wrote:
> > 2. Testing dashboard to maintain release health (new, thanks Nigel)
> >   - Dashboard at [2]
> >   - We already have 3 failures here as follows, needs attention from
> > appropriate *maintainers*,
> >     (a)
> >
> 
> https://build.gluster.org/job/regression-test-with-multiplex/871/consoleText
> >       - Failed with core:
> ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t
> >     (b)
> >
> 
> https://build.gluster.org/job/regression-test-with-multiplex/873/consoleText
> >       - Failed with core: ./tests/bugs/snapshot/bug-1275616.t
> >       - Also test ./tests/bugs/glusterd/validating-server-quorum.t
> had to be
> > retried
> 
> I was looking at the cores from the above 2 instances, the one in job
> 873 is been a typical pattern, where malloc fails as there is internal
> header corruption in the free bins.
> 
> When examining the victim that would have been allocated, it is often
> carrying incorrect size and other magic information. If the data in
> victim is investigated it looks like a volfile.
> 
> With the crash in 871, I thought there maybe a point where this is
> detected earlier, but not able to make headway in the same.
> 
> So, what could be corrupting this memory and is it when the graph is
> being processed? Can we run this with ASAN or such (I have not tried,
> but need pointers if anyone has run tests with ASAN).
> 
> Here is the (brief) stack analysis of the core in 873:
> NOTE: we need to start avoiding flushing the logs when we are dumping
> core, as that leads to more memory allocations and causes a sort of
> double fault in such cases.
> 
> Core was generated by `/build/inst

Re: [Gluster-devel] query about one glustershd coredump issue

[Zhou, Cynthia (NSB - CN/Hangzhou)]

From: Ravishankar N 
Sent: Thursday, September 27, 2018 6:04 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
Subject: Re: query about one glustershd coredump issue


Hi,

I think it is better to send it on gluster-users mailing list to get more 
attention.
Regards,
Ravi
On 09/27/2018 01:10 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:
Hi,
In my test env(glusterfs 3.12.3 3brick config) when restart sn-0 sn-1 and sn-2 
at the same time, occasionally, I met glustershd on sn-0 coredump.
Could you help to shed some light on this issue? Thanks!


My gdb result of the coredump file

[root@sn-0:/root]
# gdb /usr/sbin/glusterfs 
core.glusterfs.0.c5f0c5547fbd4e5aa8f350b748e5675e.1812.153796707500
GNU gdb (GDB) Fedora 8.1-14.wf29
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/glusterfs...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 1818]
[New LWP 1812]
[New LWP 1813]
[New LWP 1817]
[New LWP 1966]
[New LWP 1968]
[New LWP 1970]
[New LWP 1974]
[New LWP 1976]
[New LWP 1814]
[New LWP 1815]
[New LWP 1816]
[New LWP 1828]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfs -s sn-0.local --volfile-id 
gluster/glustershd -p /var/run/g'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f1b5e5d7d24 in client3_3_lookup_cbk (req=0x7f1b44002300, 
iov=0x7f1b44002340, count=1,
myframe=0x7f1b4401c850) at client-rpc-fops.c:2802
2802  client-rpc-fops.c: No such file or directory.
[Current thread is 1 (Thread 0x7f1b5f00c700 (LWP 1818))]
Missing separate debuginfos, use: dnf debuginfo-install 
rcp-pack-glusterfs-1.2.0-RCP2.wf29.x86_64
(gdb) bt
#0  0x7f1b5e5d7d24 in client3_3_lookup_cbk (req=0x7f1b44002300, 
iov=0x7f1b44002340, count=1,
myframe=0x7f1b4401c850) at client-rpc-fops.c:2802
#1  0x7f1b64553d47 in rpc_clnt_handle_reply (clnt=0x7f1b5808bbb0, 
pollin=0x7f1b580c6620)
at rpc-clnt.c:778
#2  0x7f1b645542e5 in rpc_clnt_notify (trans=0x7f1b5808bde0, 
mydata=0x7f1b5808bbe0,
event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f1b580c6620) at rpc-clnt.c:971
#3  0x7f1b64550319 in rpc_transport_notify (this=0x7f1b5808bde0, 
event=RPC_TRANSPORT_MSG_RECEIVED,
data=0x7f1b580c6620) at rpc-transport.c:538
#4  0x7f1b5f49734d in socket_event_poll_in (this=0x7f1b5808bde0, 
notify_handled=_gf_true)
at socket.c:2315
#5  0x7f1b5f497992 in socket_event_handler (fd=25, idx=15, gen=7, 
data=0x7f1b5808bde0, poll_in=1,
poll_out=0, poll_err=0) at socket.c:2471
#6  0x7f1b647fe5ac in event_dispatch_epoll_handler (event_pool=0x230cb00, 
event=0x7f1b5f00be84)
at event-epoll.c:583
#7  0x7f1b647fe883 in event_dispatch_epoll_worker (data=0x23543d0) at 
event-epoll.c:659
#8  0x7f1b6354a5da in start_thread () from /lib64/libpthread.so.0
#9  0x7f1b62e20cbf in clone () from /lib64/libc.so.6
(gdb) info thread
  Id   Target Id Frame
* 1Thread 0x7f1b5f00c700 (LWP 1818) 0x7f1b5e5d7d24 in 
client3_3_lookup_cbk (req=0x7f1b44002300,
iov=0x7f1b44002340, count=1, myframe=0x7f1b4401c850) at 
client-rpc-fops.c:2802
  2Thread 0x7f1b64c83780 (LWP 1812) 0x7f1b6354ba3d in 
__pthread_timedjoin_ex ()
   from /lib64/libpthread.so.0
  3Thread 0x7f1b61eae700 (LWP 1813) 0x7f1b63554300 in nanosleep () from 
/lib64/libpthread.so.0
  4Thread 0x7f1b5feaa700 (LWP 1817) 0x7f1b635508ca in 
pthread_cond_timedwait@@GLIBC_2.3.2 
()
   from /lib64/libpthread.so.0
  5Thread 0x7f1b5ca2b700 (LWP 1966) 0x7f1b62dee4b0 in nanosleep () from 
/lib64/libc.so.6
  6Thread 0x7f1b4f7fe700 (LWP 1968) 0x7f1b6355050c in 
pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
  7Thread 0x7f1b4e7fc700 (LWP 1970) 0x7f1b6355050c in 
pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
  8Thread 0x7f1b4d7fa700 (LWP 1974) 0x7f1b62dee4b0 in nanosleep () from 
/lib64/libc.so.6
  9Thread 0x7f1b33fff700 (LWP 1976) 0x7f1b62dee4b0 in nanosleep () from 
/lib64/libc.so.6
  10   

Re: [Gluster-devel] Glusterfs v4.1.1 issue encountered while executing test case ./tests/features/trash.t

[Vijay Bellur]

Hi Abhay,

trash is not enabled by default and is considered an experimental feature
in Gluster. You can ignore the failure in trash.t while we fix the problem
for your architecture.

I will update through the bug if any other information is necessary from
you.

Thanks,
Vijay

On Thu, Sep 27, 2018 at 11:51 PM Abhay Singh  wrote:

> Hi,
>
> I have reported an issue
> on bugzilla
> regarding the test case failure of  ./tests/features/trash.t for the
> Glusterfs v4.1.1
> Do we have any updates on this?
>
> Please let us know if any additional info is needed from our side.
> The test failure is blocking our work on the package, so could you please
> take it on priority and give us any updates as soon as possible.
>
> Regards,
> Abhay Singh
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel