Bug#943509: libsqlite3 (Re: Bug#943509: python-django: FTBFS due to failed tests: failures=7, skipped=891, expected failures=4)

[ISHIKAWA,chiaki]

On Sun, 3 Nov 2019 08:19:20 +0900 "ISHIKAWA,chiaki" 
 wrote:

> Hi,
>
> If I am not mistaken, Ubuntu version used on Mozilla's build server
> farms uses rather very old version of sqlite3.
>
>
> > [task 2019-10-31T22:44:44.290Z] ++ NAME=Ubuntu
> > [task 2019-10-31T22:44:44.290Z] ++ VERSION='16.04.5 LTS (Xenial Xerus)'
> > [task 2019-10-31T22:44:44.290Z] ++ ID=ubuntu
>
>
> Maybe I should downgrade to this and then upgrade a bit until local 
test breaks.

>
> From: https://launchpad.net/ubuntu/xenial/+source/sqlite3
>
> Updates
>
> Package versions including new features after the distribution
> release has been made. Updates are usually turned on by default
> after a fresh install.
>
> * sqlite3 3.11.0-1ubuntu1.2
> 
> (main)
>

I downgraded the sqlite to 3.11.0 used in the first release of ubutno 
Xenial Xerus as noted above.


Unfortunately, the indexedDB errors I saw with mozilla Thunderbird test 
suite did not disappear.


This means that something else that changed between late August and 
middle October (quite likely some other packages) may be to blame.


I will continue investigating, but I *think* sqlite3 was not to blame 
for Thunderbird test failures.


Sorry for the noise.

TIA

Chiaki

Bug#943509: libsqlite3 (Re: Bug#943509: python-django: FTBFS due to failed tests: failures=7, skipped=891, expected failures=4)

[ISHIKAWA,chiaki]

Hi,

If I am not mistaken, Ubuntu version used on Mozilla's build server 
farms uses rather very old version of sqlite3.


  

[task 2019-10-31T22:44:44.290Z] ++ NAME=Ubuntu
[task 2019-10-31T22:44:44.290Z] ++ VERSION='16.04.5 LTS (Xenial Xerus)'
[task 2019-10-31T22:44:44.290Z] ++ ID=ubuntu



Maybe I should downgrade to this and then upgrade a bit until local test breaks.

From: https://launchpad.net/ubuntu/xenial/+source/sqlite3

Updates

   Package versions including new features after the distribution
   release has been made. Updates are usually turned on by default
   after a fresh install.

 * sqlite3 3.11.0-1ubuntu1.2
   
   (main)

Bug#943509: libsqlite3 (Re: Bug#943509: python-django: FTBFS due to failed tests: failures=7, skipped=891, expected failures=4)

[ISHIKAWA,chiaki]

Dear Laszló,

Thank you for your e-mail.

(Sorry my local system does not seem to have the correct font for the 
character in your name, and so I am not sure if I am writing the name 
correctly.

I hope it will show correctly on your end.)

My comments inline:

On 2019/11/01 23:44, László Böszörményi (GCS) wrote:

Hi Chiaki,

On Thu, Oct 31, 2019 at 9:09 PM ISHIKAWA,chiaki  wrote:

I vouch that there seems to be a serious issue in libsqlite3 3.30.1-1.

  It's not that fatal like it may seem so.

I wish that is  the case.

I came here because it seems that I have an issue with sqlite3 on my
linux installation.
The problem manifested as database operation failures during thunderbird
mail client regression testing (called mach xcpshell-tests|)

  Is there a documented way to get and run it locally?


There is a series of documents.
However, it is such a long winding path, it may take a few days to set 
up and  run correctly.

I can't really suggest that you take the steps outlined below casually.

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Thunderbird_build

My summary is as follows.

First you have to download the source for C-C (comm-central) AND M-C 
(mozilla-central.)

M-C should be installed somewhere with mozilla directory name.
C-C needs to be installed as comm immediately below mozilla directory.

Then you have to configure the development environment.
Setting environment variables and configuration file and set an 
environment variable, MOZCONFIG,

that points to the config file.
You have to specify the MOZ_OBJDIR that stores the generated binary: 
this object directory MUST be

outside the source tree.

Before building,
you have to download a series of debian package header files and 
libraries (i.e., development library packages) that are used by

the thunderbird build process.
There are a lot actually under Debian that we need to install on top of 
regular installation.


*THEN*
We run the compilation process using so-called |../mach build| command 
in the C-C topmost directory (comm) immediately below M-C (mozill)a 
directory.


This command will likely suggest that we need to install clang c++ 
compiler and rust compiler with many packages.

It prints out the commands how to perform these actions. So I follow it.

Once the compilation is successful, we can finally run
xpcshell test by

cd $MOZOBJ || exit 1

...    mozilla/mach --log-no-times xpcshell-test

where mach command is in the mozilla source directory (M-C).

Build takes about 1 hour on my Ryzen1700 with 16GB memory.
(Actually, Debian GNU/Linux amd64 image that runs inside VirtualBox that 
runs on top Windows10 Pro 64 bit)


If you really are interested, I can help.

There are all sorts of minor issues that pop up from time to time when 
we use open source tools and libraries.
Usually error messages are clear, but sometimes python and other script 
language errors pop up and they are difficult to analyze.




I looks someone on Ubuntu did not see it this month and mozilla's
compilation and testing farm machines do not seem to see it. They run
CentOs if I am not mistaken.

  Which version of CentOS? Its latest release, versioned 8 has SQLite3
3.26.0 if I'm not mistaken.


Oh, I was wrong. The compilation/test farm machines have transitioned to 
Ubuntu sometime before.
I did not know that. Maybe we can know what sqlite3 version is working 
there.

I see the following version info of OS. It seems a very old OS package.

[task 2019-10-31T22:44:44.290Z] ++ NAME=Ubuntu
[task 2019-10-31T22:44:44.290Z] ++ VERSION='16.04.5 LTS (Xenial Xerus)'
[task 2019-10-31T22:44:44.290Z] ++ ID=ubuntu
[task 2019-10-31T22:44:44.290Z] ++ ID_LIKE=debian
[task 2019-10-31T22:44:44.290Z] ++ PRETTY_NAME='Ubuntu 16.04.5 LTS'
[task 2019-10-31T22:44:44.291Z] ++ VERSION_ID=16.04

from 
https://taskcluster-artifacts.net/OtJtVGBiQkax0LeOVJAcbw/0/public/logs/live_backing.log

This is from a build/test job submission result:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central=e1c9f00a40034361a9a3b6c4abdf0807e22c5d7e

Mozilla has a rather nice build/test farm setup.




https://bugzilla.mozilla.org/show_bug.cgi?id=1589779

So I thought maybe the current installation of libsqlite3 on my linux PC
was to blame.

  Might be, but SQLite3 3.30.1 should be safe.

I am still trying to find out what changed between latest August and 
beginning of October.


It can be

- my PC environment changes (including package updates), or
- even testing environment change (but unlikely).

Since someone under Ubuntu told me the test works there without an 
issue, I think

local package version issue is more like it.


Well, it looks that is the case if I read the exchanges here correctly.
My libsqlite3 seems to be affected: the version is 3.30.1-1
(The problem was not observed in August. I am not sure if the upgrade of
libsqlite3 happened during then and now.)

What do people suggest that best course of action under the circumstances?

Bug#943509: libsqlite3 (Re: Bug#943509: python-django: FTBFS due to failed tests: failures=7, skipped=891, expected failures=4)

[GCS]

Hi Chiaki,

On Thu, Oct 31, 2019 at 9:09 PM ISHIKAWA,chiaki  wrote:
> I vouch that there seems to be a serious issue in libsqlite3 3.30.1-1.
 It's not that fatal like it may seem so.

> I came here because it seems that I have an issue with sqlite3 on my
> linux installation.
> The problem manifested as database operation failures during thunderbird
> mail client regression testing (called mach xcpshell-tests|)
 Is there a documented way to get and run it locally?

> I looks someone on Ubuntu did not see it this month and mozilla's
> compilation and testing farm machines do not seem to see it. They run
> CentOs if I am not mistaken.
 Which version of CentOS? Its latest release, versioned 8 has SQLite3
3.26.0 if I'm not mistaken.

> https://bugzilla.mozilla.org/show_bug.cgi?id=1589779
>
> So I thought maybe the current installation of libsqlite3 on my linux PC
> was to blame.
 Might be, but SQLite3 3.30.1 should be safe.

> Well, it looks that is the case if I read the exchanges here correctly.
> My libsqlite3 seems to be affected: the version is 3.30.1-1
> (The problem was not observed in August. I am not sure if the upgrade of
> libsqlite3 happened during then and now.)
>
> What do people suggest that best course of action under the circumstances?
>
> Downgrade (to which version and how) or install newer libsqlite3 (where)?
 All uploaded SQLite3 Debian packages should be archived[1]. Try
downgrading it to 3.29.0-2 first and see if it helps.

> All I can say is that the same tests that deal with database that worked
> for years suddenly broke in the last several weeks. So my bug report
> would not be that useful :-(
 May you have a date when you first experienced it? As sqlite3 3.30.0
is in the Debian archives since the fifth of October, this may narrow
down the issue a bit.

Regards,
Laszlo/GCS
[1] http://snapshot.debian.org/package/sqlite3/

Bug#943509: libsqlite3 (Re: Bug#943509: python-django: FTBFS due to failed tests: failures=7, skipped=891, expected failures=4)

[ISHIKAWA,chiaki]

Hi,

I vouch that there seems to be a serious issue in libsqlite3 3.30.1-1.

I came here because it seems that I have an issue with sqlite3 on my 
linux installation.
The problem manifested as database operation failures during thunderbird 
mail client regression testing (called mach xcpshell-tests|)


I looks someone on Ubuntu did not see it this month and mozilla's 
compilation and testing farm machines do not seem to see it. They run 
CentOs if I am not mistaken.


https://bugzilla.mozilla.org/show_bug.cgi?id=1589779

So I thought maybe the current installation of libsqlite3 on my linux PC 
was to blame.


Well, it looks that is the case if I read the exchanges here correctly.
My libsqlite3 seems to be affected: the version is 3.30.1-1
(The problem was not observed in August. I am not sure if the upgrade of 
libsqlite3 happened during then and now.)


What do people suggest that best course of action under the circumstances?

Downgrade (to which version and how) or install newer libsqlite3 (where)?

Your guidance is appreciated.

TIA

Chiaki

PS: if necessary, I can file a separate bug report to escalate this 
issue, but it would be weeks until I can come up with a very simplified 
test case since the problems occur inside the complex test harness: I 
don't understand the innards very well.


All I can say is that the same tests that deal with database that worked 
for years suddenly broke in the last several weeks. So my bug report 
would not be that useful :-(