Re: [sage-devel] Constructor of a very large matrix doesn't copy entries correctly

2021-12-02 Thread 'jonatha...@googlemail.com' via sage-devel 
FF = GF(next_prime(100))
M = Matrix(FF, [[1, 2], [3, 4]])
M.__init__??

reveals that the indices are casted into a long. This might be the problem.

Using Py_ssize_t or size_t might have been a better choice.

Jonathan
dim...@gmail.com schrieb am Mittwoch, 1. Dezember 2021 um 20:56:16 UTC+1:

>
>
> On Wed, Dec 1, 2021 at 4:39 PM Anton Mellit  wrote:
> >
> > When I create a very large matrix (the threshold may be 2^32 entries) 
> from a list of vectors, the elements are not copied over correctly.
> > Here is a "small" example:
> >
> > FF = GF(next_prime(100))
> > M = 7
> > N = 8
> > v1 = vector(FF, [FF(k) for k in range(N)])
> > v2 = vector(FF, [N] * N)
> > vecs = [v1 + v2*FF(i) for i in range(M)]
> > mat = matrix(FF, M, N, vecs)
> > mat[0] == vecs[0]
> >
> > the last line returns "false".
> > I understand the bug is not so easy to reproduce: the code above runs 
> for about 40 minutes and needs about 50G of RAM. In fact, it is also 
> strange that it takes so long: all the lines until "mat = matrix(FF, M, N, 
> vecs)" take only about a minute. It shouldn't take so much more time to 
> simply copy a bunch of vectors than it took to compute them.
> >
> Matrices in Sage are not just array, they come with parents - vector 
> spaces of matrices of this size.
> Perhaps it's where the time bottleneck is - in creating the parent, that 
> code wasn't obviously optimised for these kinds of dimensions.
>
> As to the error... Hard to say, one needs to look at the code (and find a 
> machine to reproduce the example). 
>
> Dima
>
>
> > Sage version: 9.3 running on a 64 bit linux system
> >
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "sage-devel" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to sage-devel+...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/d06f39c6-f066-4e83-a2c1-2bbcba345f36n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/9c38f81a-f471-41e1-a36f-07b618232f47n%40googlegroups.com.

Re: [sage-devel] Constructor of a very large matrix doesn't copy entries correctly

2021-12-02 Thread Dima Pasechnik 
On Thu, Dec 2, 2021 at 11:17 AM 'jonatha...@googlemail.com' via
sage-devel  wrote:
>
> FF = GF(next_prime(100))
> M = Matrix(FF, [[1, 2], [3, 4]])
> M.__init__??
>
> reveals that the indices are casted into a long. This might be the problem.

right, seems to be the right place

sage: M*N
56
sage: 2^32 # long is 4 bytes, i.e. maximal value it holds is 2^32-1
4294967296

It's probably not the only place where  is used for indexing of
matrices (which are stuffed into 1-dim. arrays)

Dima

>
> Using Py_ssize_t or size_t might have been a better choice.
>
> Jonathan
> dim...@gmail.com schrieb am Mittwoch, 1. Dezember 2021 um 20:56:16 UTC+1:
>>
>>
>>
>> On Wed, Dec 1, 2021 at 4:39 PM Anton Mellit  wrote:
>> >
>> > When I create a very large matrix (the threshold may be 2^32 entries) from 
>> > a list of vectors, the elements are not copied over correctly.
>> > Here is a "small" example:
>> >
>> > FF = GF(next_prime(100))
>> > M = 7
>> > N = 8
>> > v1 = vector(FF, [FF(k) for k in range(N)])
>> > v2 = vector(FF, [N] * N)
>> > vecs = [v1 + v2*FF(i) for i in range(M)]
>> > mat = matrix(FF, M, N, vecs)
>> > mat[0] == vecs[0]
>> >
>> > the last line returns "false".
>> > I understand the bug is not so easy to reproduce: the code above runs for 
>> > about 40 minutes and needs about 50G of RAM. In fact, it is also strange 
>> > that it takes so long: all the lines until "mat = matrix(FF, M, N, vecs)" 
>> > take only about a minute. It shouldn't take so much more time to simply 
>> > copy a bunch of vectors than it took to compute them.
>> >
>> Matrices in Sage are not just array, they come with parents - vector spaces 
>> of matrices of this size.
>> Perhaps it's where the time bottleneck is - in creating the parent, that 
>> code wasn't obviously optimised for these kinds of dimensions.
>>
>> As to the error... Hard to say, one needs to look at the code (and find a 
>> machine to reproduce the example).
>>
>> Dima
>>
>>
>> > Sage version: 9.3 running on a 64 bit linux system
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "sage-devel" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an 
>> > email to sage-devel+...@googlegroups.com.
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/sage-devel/d06f39c6-f066-4e83-a2c1-2bbcba345f36n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sage-devel+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sage-devel/9c38f81a-f471-41e1-a36f-07b618232f47n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq3sD5Sf3i%3DxV3fs29os7_4CC-28iGZPH9RJVDj_8npgzA%40mail.gmail.com.

Re: [sage-devel] Constructor of a very large matrix doesn't copy entries correctly

2021-12-02 Thread Dima Pasechnik 
I've opened https://trac.sagemath.org/ticket/32961 to deal with this.
At least we should check that the number of entries can be
handled by the current implementation.

On Thu, Dec 2, 2021 at 11:31 AM Dima Pasechnik  wrote:
>
> On Thu, Dec 2, 2021 at 11:17 AM 'jonatha...@googlemail.com' via
> sage-devel  wrote:
> >
> > FF = GF(next_prime(100))
> > M = Matrix(FF, [[1, 2], [3, 4]])
> > M.__init__??
> >
> > reveals that the indices are casted into a long. This might be the problem.
>
> right, seems to be the right place
>
> sage: M*N
> 56
> sage: 2^32 # long is 4 bytes, i.e. maximal value it holds is 2^32-1
> 4294967296
>
> It's probably not the only place where  is used for indexing of
> matrices (which are stuffed into 1-dim. arrays)
>
> Dima
>
> >
> > Using Py_ssize_t or size_t might have been a better choice.
> >
> > Jonathan
> > dim...@gmail.com schrieb am Mittwoch, 1. Dezember 2021 um 20:56:16 UTC+1:
> >>
> >>
> >>
> >> On Wed, Dec 1, 2021 at 4:39 PM Anton Mellit  wrote:
> >> >
> >> > When I create a very large matrix (the threshold may be 2^32 entries) 
> >> > from a list of vectors, the elements are not copied over correctly.
> >> > Here is a "small" example:
> >> >
> >> > FF = GF(next_prime(100))
> >> > M = 7
> >> > N = 8
> >> > v1 = vector(FF, [FF(k) for k in range(N)])
> >> > v2 = vector(FF, [N] * N)
> >> > vecs = [v1 + v2*FF(i) for i in range(M)]
> >> > mat = matrix(FF, M, N, vecs)
> >> > mat[0] == vecs[0]
> >> >
> >> > the last line returns "false".
> >> > I understand the bug is not so easy to reproduce: the code above runs 
> >> > for about 40 minutes and needs about 50G of RAM. In fact, it is also 
> >> > strange that it takes so long: all the lines until "mat = matrix(FF, M, 
> >> > N, vecs)" take only about a minute. It shouldn't take so much more time 
> >> > to simply copy a bunch of vectors than it took to compute them.
> >> >
> >> Matrices in Sage are not just array, they come with parents - vector 
> >> spaces of matrices of this size.
> >> Perhaps it's where the time bottleneck is - in creating the parent, that 
> >> code wasn't obviously optimised for these kinds of dimensions.
> >>
> >> As to the error... Hard to say, one needs to look at the code (and find a 
> >> machine to reproduce the example).
> >>
> >> Dima
> >>
> >>
> >> > Sage version: 9.3 running on a 64 bit linux system
> >> >
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google 
> >> > Groups "sage-devel" group.
> >> > To unsubscribe from this group and stop receiving emails from it, send 
> >> > an email to sage-devel+...@googlegroups.com.
> >> > To view this discussion on the web visit 
> >> > https://groups.google.com/d/msgid/sage-devel/d06f39c6-f066-4e83-a2c1-2bbcba345f36n%40googlegroups.com.
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "sage-devel" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to sage-devel+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/sage-devel/9c38f81a-f471-41e1-a36f-07b618232f47n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/CAAWYfq1SP17NR%3DnKjoRSspYhUfJ5wUtTKk4iFgCCEtZ5r_Ts0Q%40mail.gmail.com.

Re: [sage-devel] Docker images no longer being build and is gitlab still maintained?

2021-12-02 Thread Maarten Derickx 
In the meantime I managed to verify that aside from the gitlab CI/CD there 
are no other things that are broken. Meaning that I managed to build the 
docker file shipped with sage just fine on my laptop. I pushed a build of 
9.4 to https://hub.docker.com/r/mderickx/sagemath/ in case anyone is 
interested.

Op dinsdag 30 november 2021 om 18:57:51 UTC+1 schreef wst...@gmail.com:

> Thanks Thierry, I'll email you directly to figure out the details.
>
>
> On Tue, Nov 30, 2021 at 4:32 AM Thierry  
> wrote:
>
>> Hi,
>>
>> On Sun, Nov 28, 2021 at 07:32:18PM +, Dima Pasechnik wrote:
>> > On Sun, 28 Nov 2021, 15:38 William Stein,  wrote:
>> > > On Sun, Nov 28, 2021 at 2:32 AM Dima Pasechnik  
>> wrote:
>> > > > On Sun, Nov 28, 2021 at 10:12 AM Maarten Derickx
>> > > >  wrote:
>> > > > >
>> > > > > What are the specs that the dedicated builder would need? In 
>> terms of
>> > > cpu + ram + diskspace?
>> > > >
>> > > > nothing spectacularly big, but somehow it manages to run out of
>> > > > smallish disk space on an INRIA (or was it Paris Sud, aka Paris
>> > > > Saclay?).
>> > > >
>> > > > Like, in might need more than 20 Gb or 50Gb free space for some 
>> reason.
>> > > > Nothing too bug, a reasonable modern desktop would do the job ---if 
>> we
>> > > > don't let it run on every push into the repo.
>> > >
>> > > There is still a powerful server at Univ of Washington that I bought 
>> using
>> > > a grant specifically for Sage development
>> > > (nearly a decade ago!) that could be used for building Docker images. 
>> It's
>> > > very powerful and has tons of disk space.
>> > > The drawback is it could just go "poof" at any moment, and it could 
>> be a
>> > > very long time until it comes back, if ever.
>> > > But for building docker images, that's probably fine.
>> > 
>> > can it be used as a gitlab or/and github runner for sagemath org?
>> > 
>> > I guess I don't have an account there, but it would be good to get one.
>>
>> If this ressource is not fully used, i am also volonteering to get an
>> account and setup some specific (e.g. 32bit) patchbots there.
>>
>> Ciao,
>> Thierry
>>
>>
>>
>> > Dima
>> > 
>> > 
>> > > wstein@kucalc:~/cocalc-docker/linux$ df -h .
>> > > FilesystemSize  Used Avail Use% Mounted on
>> > > tank/home/wstein  2.7T  252G  2.5T  10% /home/wstein
>> > > wstein@kucalc:~/cocalc-docker/linux$ uptime
>> > >  07:30:53 up 904 days, 20:11,  9 users,  load average: 0.67, 0.61, 
>> 0.52
>> > > wstein@kucalc:~/cocalc-docker/linux$ free -g
>> > >   totalusedfree  shared  buff/cache
>> > > available
>> > > Mem:251  11  41   2 198
>> > >   198
>> > > Swap:   255   1 254
>> > > wstein@kucalc:~/cocalc-docker/linux$ cat /proc/cpuinfo | grep 
>> processor
>> > > |wc -l
>> > > 48
>> > >
>> > > -- William
>> > >
>> > > >
>> > > > Dima
>> > > >
>> > > >
>> > > >
>> > > > >
>> > > > > On Saturday, 27 November 2021 at 19:13:14 UTC+1 dim...@gmail.com
>> > > wrote:
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Sat, 27 Nov 2021, 17:07 Maarten Derickx, <
>> m.derick...@gmail.com>
>> > > wrote:
>> > > > >>>
>> > > > >>> I noticed on https://hub.docker.com/r/sagemath/sagemath/ that 
>> there
>> > > haven't been any updates of the official sagemath docker image since
>> > > 9.3.beta8-py3 about 9 months ago.
>> > > > >>>
>> > > > >>> On that page it mentions "Every push to our GitLab repository
>> > > triggers a pipeline in GitLab CI which pushes the actual images to 
>> Docker
>> > > Hub."
>> > > > >>>
>> > > > >>> Now I had a look at the pipelines on gitlab
>> > > 
>> https://gitlab.com/sagemath/sage/-/pipelines?page=1&scope=all&status=success
>> > > the last succesful run of the pipelines was about 8 months ago. So 
>> gitlab
>> > > pipelines being broken seems to explain why no new images are being 
>> pushed
>> > > to dockerhub.
>> > > > >>>
>> > > > >>> So I am wondering, is gitlab being phased out/abandoned, or do 
>> we
>> > > still support it and do we just need to show it some love?
>> > > > >>
>> > > > >>
>> > > > >> It needs quite a bit of love, in form of a gitlab runner that 
>> doesn't
>> > > run out of disk space (a dedicated runner at France runs of disk 
>> space).
>> > > > >>
>> > > > >> It's perhaps the time to switch over to GitHub.
>> > > > >> Mattias Koeppe should have an idea how much effort this would 
>> entail.
>> > > > >>
>> > > > >> Dima
>> > > > >>>
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Maarten
>> > > > >>>
>> > > > >>> --
>> > > > >>> You received this message because you are subscribed to the 
>> Google
>> > > Groups "sage-devel" group.
>> > > > >>> To unsubscribe from this group and stop receiving emails from 
>> it,
>> > > send an email to sage-devel+...@googlegroups.com.
>> > > > >>> To view this discussion on the web visit
>> > > 
>> https://groups.google.com/d/msgid/sage-devel/492af376-40ad-45b2-940c-f0ea4f49bf60n%40googlegroups.com
>> > > .
>> > > > >
>> > > >

Re: [sage-devel] Failed to build cysignals-1.10.3 and pyzmq-22.2.1 on a Fedora 35 (x86_64)

2021-12-02 Thread David Coudert 
I have the same issues after upgrade a desktop to fedora 35.
I succeed to compile using  https://trac.sagemath.org/ticket/32828
Thanks.
On Monday, November 22, 2021 at 8:22:48 AM UTC+1 enriqu...@gmail.com wrote:

> For cysignals it works using the package from this fork: 
> https://github.com/kliem/cysignals/tree/fix_non_constant_MINSIGSTKSZ 
> (attached) with the changed files for build/pkgs/cysignals
> For pyzmq it has been solved in https://trac.sagemath.org/ticket/32828.
> I got some failed tests but at least a usable build. 
> Following https://trac.sagemath.org/ticket/30767 it is also possible to 
> use Python 3.10 (in this ticket they do not use the system package).
>
> El lunes, 22 de noviembre de 2021 a las 7:26:47 UTC+1, 
> jonatha...@googlemail.com escribió:
>
>> Thanks for the report. For cysignals we have the ticket 
>> https://trac.sagemath.org/ticket/32576 and github issue 
>> https://github.com/sagemath/cysignals/issues/150.
>>
>> furutaka@gmail.com schrieb am Montag, 22. November 2021 um 04:17:55 
>> UTC+1:
>>
>>> Forgot to attach this...
>>>
>>> 2021年11月22日(月) 11:31 Kazuyoshi Furutaka :
>>>
 Hi,

 My build of sage-9.5.beta7 on a Fedora 35 system (x86_64) failed:
 * package: cysignals-1.10.3
   last build time: Nov 22 11:13

 * package: pyzmq-22.2.1
   last build time: Nov 22 11:13

 Logs attached to this message.

 By the way, they've been installed as Fedora packages:
 [root@peart-furu-or-jp furutaka]# dnf list installed \*zmq\* 
 \*cysignals\*
 Installed Packages
 czmq.x86_64  
 4.2.1-6.fc34@fedora
 czmq-devel.x86_64
 4.2.1-6.fc34@fedora
 python3-cysignals.x86_64 
 1.10.3-2.fc35   @fedora
 python3-cysignals-devel.x86_64   
 1.10.3-2.fc35   @fedora
 python3-zmq.x86_64   
 22.2.1-1.fc35   @fedora
 python3-zmq-tests.x86_64 
 22.2.1-1.fc35   @fedora

 -- 
 You received this message because you are subscribed to the Google 
 Groups "sage-devel" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to sage-devel+...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/sage-devel/392b7480-e729-48cb-ab98-8cb9af2ce7b0n%40googlegroups.com
  
 
 .

>>>
>>>
>>> -- 
>>> Kazuyoshi Furutaka
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sage-devel/cbf139da-6571-49e8-8e93-4c489c4a46ben%40googlegroups.com.