Lina, There were some leads on the Jenkins side about the issues. 1. All the tests were failing because the Hive/MR job couldn't find a file on the /tmp directory. 2. All the tests were failing only during the day (they were passing at night). 3. Reverting too old patches were sometimes passing or failing with the same failures. 4. Finally, we found that Jenkins slaves are shared with other non-sentry jobs. This was the last confirmation that something was not right with the slave environment.
Once I applied those patches to move the /tmp temporary files to a different one, then they worked, so the confirmation that something was deleting the /tmp space was correct. - Sergio On Mon, Nov 20, 2017 at 4:01 PM, Na Li <[email protected]> wrote: > Sergio, > > It's great you found the cause. How did you figure it out? > > Thanks, > > Lina > > On Mon, Nov 20, 2017 at 12:46 PM, Kalyan Kumar Kalvagadda < > [email protected]> wrote: > > > Sergio, > > > > I resolved SENTRY-2050. > > > > -Kalyan > > > > On Mon, Nov 20, 2017 at 12:40 PM, Sergio Pena <[email protected]> > > wrote: > > > > > We finally deciphered the reason for the Jenkins test failures, and > they > > > were not related to the Sentry codebase. For some unknown reason, the > > /tmp > > > directory was cleaned up at some point causing the Sentry hive tests to > > > fail on MR jobs because files on the /tmp weren't found. > > > > > > The fixes were done through these jiras and now tests are green: > > > SENTRY-2054 Unit tests must create temporary files under the Maven > > target > > > directory > > > SENTRY-2057: Set hadoop.tmp.dir to the maven build directory > configured > > > on java.io.tmpdir > > > > > > Kalyan, you're free to close the SENTRY-2050. > > > > > > - Sergio > > > > > > On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda < > > > [email protected]> wrote: > > > > > > > I have created SENTRY-2050 to investigate the situation. As a first > > step > > > > I'm reverting couple of commits and submit the patch to see how > Jenkins > > > > behaves. > > > > > > > > -Kalyan > > > > > > > > On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena < > [email protected] > > > > > > > wrote: > > > > > > > > > There are several tests failing currently and seems they're flaky. > > > > They're > > > > > passing locally but most of the time fail on Jenkins with some > > > exceptions > > > > > on some patches. > > > > > > > > > > We're looking at the old commits from the past week to see which > > commit > > > > is > > > > > the culprit of this. > > > > > > > > > > - Sergio > > > > > > > > > > > > > > >
