[Gluster-infra] [Bug 1558827] New: Unable to access cores/logs from regression runs

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558827

Bug ID: 1558827
   Summary: Unable to  access cores/logs from regression runs
   Product: GlusterFS
   Version: mainline
 Component: project-infrastructure
  Assignee: b...@gluster.org
  Reporter: spa...@redhat.com
CC: b...@gluster.org, gluster-infra@gluster.org



Description of problem:
Unable to download cores/logs from Gerrit regression run.

You can get the links from the below URL:
https://build.gluster.org/job/centos7-regression/365/consoleFull
https://build.gluster.org/job/centos7-regression/362/consoleFull

Similarly, log link is broken.

The output is shown like this:
filename="${ARCHIVED_LOGS}/glusterfs-logs-${UNIQUE_ID}.tgz"
16:18:54 sudo -E tar -czf "${ARCHIVE_BASE}/${filename}" /var/log/glusterfs
/var/log/messages*;
16:18:54 echo "Logs archived in http://${SERVER}/${filename};


Actual results:
403 error [ Forbidden ]

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=YZ7qBFP7af=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

Re: [Gluster-infra] [Gluster-devel] Announcing Softserve- serve yourself a VM

[Sankarshan Mukhopadhyay]

On Tue, Mar 20, 2018 at 1:57 PM, Sanju Rakonde  wrote:

> I have a suggestion here. It will be good if we have a option like request
> for extension of the VM duration, and the option will be automatically
> activated after 3 hours of usage of VM. If somebody is using the VM after 3
> hours and they feel like they need it for 2 more hours they will request to
> extend the duration by 1 more hour. It will save the time of engineering
> since if a machine is expired, one has to configure the machine and all
> other stuff from the beginning.
>

As Nigel states later on, please file an issue. I fear that very soon
we will require an enhancement to the issue which might have to handle
how many times an extension can be sought. Supplementing it would be
another enhancement around how often can a particular individual
request for extension on a particular machine instance within the
space of a work week. And thus, at some point, a simple machine
instance allocation system would need to be backed by a decision
making system.

This was supposed to be an easy-to-use infrastructure, not where the
Infra team members chase down individuals who are not freeing up
machines for as high as 5 weeks (I am not going to publicly cite this
instance).

That being said, I think it is worth a moment to think about what is
happening with Softserve.

The system is intended to provide a set of finite metered resources
available to developers in order to debug issues. The number of
available machine instances are not high (at this point) and the
resource allocation during this initial period has been designed to be
this way so as to arrive at a better understanding of usage patterns.

A 4 hour allocation is half of a working day. If we do think that this
is not enough, we want to also think deeply about the following (a)
all our issues require more than a day’s worth of debugging (b)
perhaps we are unable to allocate a dedicated 4 hour block on one
topic ie. there are distractions (c) there will not be enough machine
instances to make available to a backlog of developers waiting to
start off with their work

Any one or, a combination of the above possibilities are larger
challenges which need to be addressed. I understand why the
very-short-term approach of “just extend the darned time slice and be
done” is conducive to getting things done. I’d rather ask ourselves -
why do we have all issues that consume multiple days and what can we
do to improve that. Because I am convinced that it surely isn’t
enjoyable to always budget for multiple days when something comes up
to debug, even if it is a race condition (as Atin mentions in his
response).




-- 
sankarshan mukhopadhyay

___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1558335] gluster-block: discontinue gerrit

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558335

Pranith Kumar K  changed:

   What|Removed |Added

  Flags|needinfo?(pkarampu@redhat.c |
   |om) |
   |needinfo?(vbel...@redhat.co |
   |m)  |



--- Comment #5 from Pranith Kumar K  ---
(In reply to Prasanna Kumar Kalever from comment #3)
> (In reply to Atin Mukherjee from comment #1)
> > FWIW (and out of curiosity too), it'd be worth to mention why did we decide
> > to move away from gerrit :-)
> 
> 1. Currently the upstream RFE's/issues/bugs/milestones are tracked in
> github, where as we encourage patches to be pushed to gerrit only. It is
> good to have everything at one place (As a fact supporting both is not
> possible)
> 
> 2. Not everyone knows gerrit, for example we see new contributors directly
> send patches to github and asking them to send to gerrit looks an extra hop.
> 
> 3. Personally, I do not see a real benefit in using gerrit and not github.
> 
> 4. Also having everything at one place give us opportunity to address/answer
> the github issues faster (as we look at github more often than before)
> 
> Others can add/correct.

This was also asked multiple times by heketi team. So we also felt like it is
better.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=LichD3snYk=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1558335] gluster-block: discontinue gerrit

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558335

Prasanna Kumar Kalever  changed:

   What|Removed |Added

  Flags||needinfo?(vbel...@redhat.co
   ||m)



-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=42Qm1YIXO4=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1558335] gluster-block: discontinue gerrit

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558335

Prasanna Kumar Kalever  changed:

   What|Removed |Added

  Flags||needinfo?(pkarampu@redhat.c
   ||om)



--- Comment #4 from Prasanna Kumar Kalever  ---
(In reply to Nigel Babu from comment #2)
> Okay, I'm creating this checklist on the fly here.
> 
> 1. Do you have a team on github with the correct people having commit access?
> [Yes/No]

Yes

> 
> 2. Are all your existing review requests closed on github?
> [Yes/No]

do you mean to ask in gerrit ?

We have open issues in github and gerrit, but all the commits are from within
the team, so we can manage to send new review request and abandon the older
once.

Not sure if that answers your question though. 

> 
> 3. Do you expect new review requests on gerrit to be rejected?
> [Yes/No]

Yes, may be with some good message if possible.

> 
> 4. I will be switching your Gerrit repo to read-only and removing the sync
> to Github. Please acknowledge that this is okay.
> [Yes/No]

Yes.

@pranith, @vijay
Please acknowledge.

Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=fraMzfiJ0o=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1558335] gluster-block: discontinue gerrit

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558335



--- Comment #3 from Prasanna Kumar Kalever  ---
(In reply to Atin Mukherjee from comment #1)
> FWIW (and out of curiosity too), it'd be worth to mention why did we decide
> to move away from gerrit :-)

1. Currently the upstream RFE's/issues/bugs/milestones are tracked in github,
where as we encourage patches to be pushed to gerrit only. It is good to have
everything at one place (As a fact supporting both is not possible)

2. Not everyone knows gerrit, for example we see new contributors directly send
patches to github and asking them to send to gerrit looks an extra hop.

3. Personally, I do not see a real benefit in using gerrit and not github.

4. Also having everything at one place give us opportunity to address/answer
the github issues faster (as we look at github more often than before)

Others can add/correct.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=SVj5rWvNbk=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1558335] gluster-block: discontinue gerrit

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558335



--- Comment #2 from Nigel Babu  ---
Okay, I'm creating this checklist on the fly here.

1. Do you have a team on github with the correct people having commit access?
[Yes/No]

2. Are all your existing review requests closed on github?
[Yes/No]

3. Do you expect new review requests on gerrit to be rejected?
[Yes/No]

4. I will be switching your Gerrit repo to read-only and removing the sync to
Github. Please acknowledge that this is okay.
[Yes/No]

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=XnMz5JtCZO=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

Re: [Gluster-infra] [Gluster-devel] Announcing Softserve- serve yourself a VM

[Nigel Babu]

Please file an issue for this:
https://github.com/gluster/softserve/issues/new

On Tue, Mar 20, 2018 at 1:57 PM, Sanju Rakonde  wrote:

> Hi Nigel,
>
> I have a suggestion here. It will be good if we have a option like request
> for extension of the VM duration, and the option will be automatically
> activated after 3 hours of usage of VM. If somebody is using the VM after 3
> hours and they feel like they need it for 2 more hours they will request to
> extend the duration by 1 more hour. It will save the time of engineering
> since if a machine is expired, one has to configure the machine and all
> other stuff from the beginning.
>
> Thanks,
> Sanju
>
> On Tue, Mar 13, 2018 at 12:37 PM, Nigel Babu  wrote:
>
>>
>> We’ve enabled certain limits for this application:

1.

Maximum allowance of 5 VM at a time across all the users. User have
to wait until a slot is available for them after 5 machines allocation.
2.

User will get the requesting machines maximum upto 4 hours.


>>> IMHO ,max cap of 4 hours is not sufficient. Most of the times, the
>>> reason of loaning a machine is basically debug a race where we can't
>>> reproduce the failure locally what I have seen debugging such tests might
>>> take more than 4 hours. Imagine you had done some tweaking to the code and
>>> you're so close to understand the problem and then the machine expires,
>>> it's definitely not a happy feeling. What are the operational challenges if
>>> we have to make it for atleast 8 hours or max a day?
>>>
>>
>> The 4h cap was kept so that multiple people could have a chance to debug
>> their test failures on the same day. Pushing the cap to 8h means that if
>> you don't have a machine to loan when you start work one will not be
>> available until the next day. At this point, we'll not be increasing the
>> timeout. So far, we've had one person actually hit this. I'd like to see
>> more data points before we make an application level change.
>>
>> --
>> nigelb
>>
>> ___
>> Gluster-devel mailing list
>> gluster-de...@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>


-- 
nigelb
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1558335] gluster-block: discontinue gerrit

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1558335

Atin Mukherjee  changed:

   What|Removed |Added

 CC||amukh...@redhat.com



--- Comment #1 from Atin Mukherjee  ---
FWIW (and out of curiosity too), it'd be worth to mention why did we decide to
move away from gerrit :-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=G7I3dD2cOr=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

Re: [Gluster-infra] Distributed Testing and Memory issues

[Karthikeyan Radhakrishnan]

https://github.com/gluster/glusterfs/blob/master/extras/distributed-testing/distributed-test-runner.py#L334

This call can be expensive in terms of memory because it sends the entire zip 
of the files from client to server. This call be optimized to send in batches 
if there is memory issues. You can try the change or let me know if you want me 
to make the change. Otherwise this code should be very slim on memory.

Thanks!

From: Deepshikha Khandelwal 
Date: Monday, March 19, 2018 at 9:24 PM
To: Karthikeyan Radhakrishnan 
Cc: Nigel Babu , gluster-infra , 
Jeff Darcy 
Subject: Re: Distributed Testing and Memory issues



On Sun, Mar 18, 2018 at 12:12 PM, Karthikeyan Radhakrishnan 
> wrote:
Hi Nigel,

This is awesome!

MemoryError is very weird. We @Facebook have never seen that. The test 
server/client is super thin to cause memory pressure, but the tests they run 
can cause such issues. How much memory does the machine you are running have?
I'm running this on machines having 2GB memory. And I think this is enough to 
have this distributed test framework setup for us.
Is the machine under pressure when you see the errors? The best way would be to 
add a rpc to query memory stat and observe.
These are newly created machines running just XMLRPC server process. I checked 
with top and got to know that this process is utilizing about 77% of memory at 
initial stage itself when the tester part of code scans and skip kicking the 
host/server for availability.
RPC is a new thing for me, so I'm not aware of RPC query calls. If you can 
brief me more about this, it would be helpful.

Let me accelerate setting up some common space (like aws) where can re-pro such 
problems.
It would be great.

Thanks!
-Karthik

From: Nigel Babu >
Date: Saturday, March 17, 2018 at 7:03 AM
To: Karthikeyan Radhakrishnan >
Cc: gluster-infra 
>, Deepshikha 
Khandelwal >, Jeff Darcy 
>
Subject: Distributed Testing and Memory issues

Hey Karthik,

Deepshikha has been working on testing the distributed test framework that you 
contributed (thank you!). Instead of writing our own code to chunk the tests, 
we've decided to just consume what you've written so we can work on making it 
run both at FB and upstream.

We're running into MemoryError exception from the threads. Do you know what's 
the best way to debug or let us know how much memory your machines have? 
That'll help us figure out solving this sooner upstream.

PS: This email is CC'd to gluster-infra and is archived publicly.

--
nigelb

___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra