Bug#727085: An HHVM-in-Debian State of the Union (was Re: Bug#727085: Taking over packaging in Debian.)
On 3/3/14, 2:01 PM, David Martínez Moreno wrote: On 2/28/14, 10:10 PM, László Böszörményi (GCS) wrote: Hi Ender, all, On Fri, Feb 28, 2014 at 8:16 PM, David Martínez Moreno en...@debian.org wrote: Hello all. My name is Ender, I have been a Debian developer for quite some time and I work for Facebook, so I decided to do proper packaging of hhvm in Alioth, as having this done properly is a goal for the team in the first part of the year. Hello, I just wanted to update everyone on this thread with respect to the Debian packaging. I have been working (with some help from Faidon) to bring the 2.4.1 tarball to Debian standards. The git repo (remember, http://anonscm.debian.org/gitweb/?p=collab-maint/hhvm.git;a=summary) reflects right now three major operational changes: integration of system's libzip, integration of system's libsqlite3 and the removal of several libraries that added useless dependencies to the package. Apart from that, the debian/copyright work is, as you may imagine, a three-ring circus. Apart from the huge list of contributors and different licenses, there is a big showstopper that I found so far, which is that a couple of files are licensed under the (in)famous JSON.org license (Software has to be used for Good not Evil). I'm doing my best to convince the in-house developers that switching to pecl-json-c (from Remi Collet) is the best approach as we are not bug-compatible anymore with the two main Linux families out there (Debian and RedHat) since the end of May 2013, but at the same time they want to stay close to PHP for good reasons. There is an endless debate^W^W^Wmore information at PHP#63520. In the meantime, Facebook has released HHVM 3.0.0 with Hack support (a superset of PHP with gradual typing, collections and more stuff - http://hacklang.org). While I'm all for packaging that version, I'm trying to stay close to 2.4.1 as I already know my troubles with this version and I don't want to add more logs to the fire, so I'm not updating the upstream version yet. Also the 3.0.0 adds dependencies like some OCaml code analysis tools depending on other stuff that is not even nearly packaged, so...you know. All in all, the ball keeps rolling, and hopefully in another month or so I hope to have a package ready in the archives. Call me an optimist. Best regards, Ender. signature.asc Description: OpenPGP digital signature
Bug#727085: An HHVM-in-Debian State of the Union (was Re: Bug#727085: Taking over packaging in Debian.)
I have been working (with some help from Faidon) to bring the 2.4.1 tarball to Debian standards. The git repo (remember, https://urldefense.proofpoint.com/v1/url?u=http://anonscm.debian.org/gitwe b/?p%3Dcollab-maint/hhvm.git%3Ba%3Dsummaryk=ZVNjlDMF0FElm4dQtryO4A%3D%3D% 0Ar=KbCGvs3NEqqV%2FJZMWI0xxA%3D%3D%0Am=Cc%2BuMlf3PVOorM33Udvk4tQyoMy%2Bt %2BAWYxrtpFKq1%2BA%3D%0As=d572db8992a4b95f8b27e56fa11dde396e7881c8bda6de8 a2c441adc3e61b428) reflects right now three major operational changes: integration of system's libzip, integration of system's libsqlite3 and the removal of several libraries that added useless dependencies to the package. Yay. Apart from that, the debian/copyright work is, as you may imagine, a three-ring circus. Apart from the huge list of contributors and different licenses, there is a big showstopper that I found so far, which is that a couple of files are licensed under the (in)famous JSON.org license (Software has to be used for Good not Evil). I'm doing my best to convince the in-house developers that switching to pecl-json-c (from Remi Collet) is the best approach as we are not bug-compatible anymore with the two main Linux families out there (Debian and RedHat) since the end of May 2013, but at the same time they want to stay close to PHP for good reasons. There is an endless debate^W^W^Wmore information at PHP#63520. This one is fun. We've chatted a bunch and the general consensus is that the optimal outcome would be to be as compatible with php5 as possible. This means we would like to use remi's extension (assuming all our unit tests and the big framework tests pass). We actually don't care too much about staying close to PHP5 at the source level since many more people are using our packages as building from source. It also is pretty bad to have the developers using the source be using a different library than the uses who use the package. So Ender, if you can get remi's extension ported to us, then we'd take it upstream. In the meantime, Facebook has released HHVM 3.0.0 with Hack support (a superset of PHP with gradual typing, collections and more stuff - https://urldefense.proofpoint.com/v1/url?u=http://hacklang.org/k=ZVNjlDMF 0FElm4dQtryO4A%3D%3D%0Ar=KbCGvs3NEqqV%2FJZMWI0xxA%3D%3D%0Am=Cc%2BuMlf3PV OorM33Udvk4tQyoMy%2Bt%2BAWYxrtpFKq1%2BA%3D%0As=9398e7801cd780c0445cf51e09 28831d9065d9f94a050e88fe6c5477e456bb46). While I'm all for packaging that version, I'm trying to stay close to 2.4.1 as I already know my troubles with this version and I don't want to add more logs to the fire, so I'm not updating the upstream version yet. Also the 3.0.0 adds dependencies like some OCaml code analysis tools depending on other stuff that is not even nearly packaged, so...you know. As far as source goes, 3.0.0 isn't much of a change. It optionally builds the type checker if you have ocaml on your machine but happily ignores it if you don't. So I bet if you drop it into your workflow it will 'just work'. Paul -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727085: Taking over packaging in Debian.
On 03/04/14 00:01, David Martínez Moreno wrote: Don't act as we have some hidden agenda, please. There are several things that I'm putting work on like dropping support for the forked libevent HTTP server, using alternatives for the php5-cli/cgi binaries to be able to replace them, no init scripts for now, compiling folly statically (or at the very least not as a different package), getting rid of third-party stuff, releasing proper tarballs in hhvm.com, merging the nightly packages with my work, and those are off the top of my head. \o/ This list is awesome David, as is the rest of your work so far. Let's document this and others under debian/TODO. Thanks for this and apologies to you, Paul team for not having made much progress on this since we last spoke. I took a look at the collab-maint repository and submitted a bunch of commits -- I had already prepared a very similar list of Build-Depends myself in the early work I had locally, so I merged mine into yours and fixed some other collateral issues I found. Please review! Thanks again and I hope we can find ways to collaborate to make an awesome HHVM package in Debian :) Best, Faidon -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#727085: Taking over packaging in Debian.
On 2/28/14, 10:10 PM, László Böszörményi (GCS) wrote:
Hi Ender, all,
On Fri, Feb 28, 2014 at 8:16 PM, David Martínez Moreno en...@debian.org
wrote:
Hello all. My name is Ender, I have been a Debian developer for
quite some
time and I work for Facebook, so I decided to do proper packaging of hhvm in
Alioth, as having this done properly is a goal for the team in the first
part of
the year.
I was wondering if one of us is working for Facebook or not. I
suspected yes, as we are everywhere and I'm right.
Of course we are everywhere! :-)
I've read the entire thread to become familiar with everything
discussed so
far, and I will be incorporating some of the suggestions discussed here as
well
as some decisions that we made as a team.
Can you share those decisions?
Don't act as we have some hidden agenda, please. There are several
things that
I'm putting work on like dropping support for the forked libevent HTTP server,
using alternatives for the php5-cli/cgi binaries to be able to replace them, no
init scripts for now, compiling folly statically (or at the very least not as a
different package), getting rid of third-party stuff, releasing proper tarballs
in hhvm.com, merging the nightly packages with my work, and those are off the
top of my head.
All the development for now is happening on collab-maint/hhvm [1],
as the
current Github repo structure does not fit really well the layout of
git-buidpackage.
Quickly peeking into your packaging I do wonder if you read the
entire bug thread.
For example I've discussed architecture support with Paul Tarjan
(message 50)[1]:
-- cut --
From: Paul Tarjan p...@fb.com
Date: Wed, 6 Nov 2013 16:51:46 +
Last but not least that wiki forces HHVM to be amd64
only. Any reason to do that? Any known problem to run HHVM on
kfreebsd-*, mips{,el}, sparc or any other archs that Debian supports?
Absolutely. HHVM is a Just-In-Time compiler which emits assembly code into
memory then executes it. We currently only emit 64bit x86. We are looking
at other backends, but they require a lot of effort to implement and
optimize.
-- cut --
Still, you put the binary package architecture any, instead of only
amd64. Does Facebook supports for example mips now?
I hadn't pushed some of my changes to git... architecture and
additional fixes
are now in.
In that mail you can also read that I've identified most of the build
dependencies. May I ask why you don't use any of that?
Again the list was incomplete (not pushed) and I was just walking
through the
CMake packaging and getting familiar with it. Our list of dependencies are
different. Apart from the fact that you made yours 5 months ago (which is
actually a lot of time for the HHVM team), there are some differences, like you
depending on the 1.54 version of libboost vs. my packaging depending on the
general -dev packages.
Without wanting this to become a blame game, my list of dependencies is
much
more specific than yours, adding also missing libraries, for example:
chrpath
libexpat1-dev
libfreetype6-dev
libpng12-dev
libvpx-dev
pkg-config
unixodbc-dev
And those are only some of them. I'm not blaming you, because I didn't
take
the time to verify that at that time the dependencies were those, but on the
contrary I preferred to carefully add everything the HHVM sources depended on.
Can it be that you use the embedded sources, those as Paul acknowledged:
-- cut --
I've just realized that HHVM
embeds other codes under hphp/third_party/ , even folly which is an
other FB FOSS software and should be packaged separately.
[...]
I've asked Paul about other embedded libraries, but didn't get an answer.
No, I'm not. The code *wants* to use them but I'm as an example
writing a
patch to remove completely the libzip dependency (which is more harmful than
normal because we also install it in the make install phase) not only from the
open source one but from our internal code repository.
The rest of them (if they're released as normal packages) will follow.
But problems arise with other libs as well. For example libafdt is
version 0.1.0 (hardly a version number to call it final and stable),
which was last released in 2009 (yes, almost five years ago!). I've a
package ready, but it doesn't build a shared library, just a static
one and it has only one header file. I may upload it if you really
want, but I don't see a reason for that. I've a package of libmbfl as
well, version 1.3.2 , which is 'just' two years old. The version
included in HHVM is 1.0.2 (according to
hphp/third_party/libmbfl/mbfl.rc) and I don't want to think about how
old is it.
Let's go piece by piece. As I see it for libafdt, if no one has felt
the need
to package that, we can just embed that in the sources.
The issue with libmbfl is a bit more concerning, and I appreciate that
you
already made the work.
Bug#727085: Taking over packaging in Debian.
On Mon, Mar 3, 2014 at 11:01 PM, David Martínez Moreno en...@debian.org wrote: On 2/28/14, 10:10 PM, László Böszörményi (GCS) wrote: On Fri, Feb 28, 2014 at 8:16 PM, David Martínez Moreno en...@debian.org wrote: Can you share those decisions? Don't act as we have some hidden agenda, please. I referred only to the some decisions that we made as a team part, inside at Facebook and not yet public. I hadn't pushed some of my changes to git... architecture and additional fixes are now in. Will check them. It's midnight here. :( Again the list was incomplete (not pushed) [...] Referred to the fact there was no build dependencies at all. OK. I've asked Paul about other embedded libraries, but didn't get an answer. No, I'm not. The code *wants* to use them but I'm as an example writing a patch to remove completely the libzip dependency (which is more harmful than normal because we also install it in the make install phase) not only from the open source one but from our internal code repository. This is good news. The build system should check for proper versions installed and not including them. The rest of them (if they're released as normal packages) will follow. Several of them are packaged already. See double-conversion, sqlite3 and lz4. I've packaged libafdt and libmbfl and of course folly. I'm not sure all of them are correct and I know that folly doesn't stand on its own. Let's go piece by piece. As I see it for libafdt, if no one has felt the need to package that, we can just embed that in the sources. It's not just that. Its upstream is died five years ago and its 0.1.0 version number[1] doesn't suggest that it's stable and ready. The issue with libmbfl is a bit more concerning, and I appreciate that you already made the work. What I have to do as part of the packaging work i to make sure that we haven't patched those sources, then I'll update them here and will use the same logic in a backported patch. You have to think that the HHVM has quite a bunch of technical debt around some edges. Quoting Paul Tarjan, I think we just brought something from the outside to third-party and that gate kept open forever. This is fine, because what we are trying to do is not just a Debian packaging but also fix HHVM for the greater good. It's the HHVM team's desire and my own goal to make this monster more palatable. Sure, this should be done first before packaging. This is what I wanted from Paul in my private email. For me it seems Facebook just put the sources on GitHub, but don't play well that boxes can have the dependent libraries installed, maybe with newer versions that included in HHVM this time. Don't forget the Folly state (no ABI and neither source Dismissed, as folly is going compiled in the blob as it comes from the internal repos. Again, I meant it in a different context. Can HHVM team support Folly on the long run (say, Ubuntu LTS releases) when its not their work and it doesn't have any 'stable', versioned release (ie, even its team may not remember why something was solved that way in a moment of time) and someone may look for a mistic FTBFS problem or needs a security fix somewhen? stability), the history of old libevent version usage, makes me wonder Dismissed as well. Paul stated it before, but I will say it again: we won't support the libevent-based server and it won't be compiled in. I wrote _history_. Referred to how Facebook follows other upstream sources. Embedding and not updating them for more than a year is a problem. I know that the libevent 1.x problem is gone. Just a friendly reminder, under hphp/third_party/ there's still a patch for libevent 1.4.14 which can be removed. Because in Facebook we don't have the luxury of having up to date distributions in the servers. The last one available IIRC is 3.7.9, so the HHVM team decided to put an updated one and then no one took the time to update it, I guess. According to my records[2], sqlite3 has version 3.7.13 in stable and as a backport in oldstable as well. You may use backports as well I guess. But I doubt you use oldstable. It's not something that I am particularly proud of, but again, you don't seem to see that we are pushing these packages from internal repositories where they don't have the same level of scrutiny about updated versions of packages. It will take some time to get there. I'm mostly arguing against embed several other upstream packages in HHVM. Then it's hard to follow which one may have newer security / important bugfix release, what patches Facebook has on them, etc. That's maybe impossible in the current situation. I even doubt that the security team will allow HHVM in as it's today. Embedding several other sources and make an inside build is a security nightmare. I just replied to the bug report where all this people
Bug#727085: Taking over packaging in Debian.
Hello all. My name is Ender, I have been a Debian developer for quite some time and I work for Facebook, so I decided to do proper packaging of hhvm in Alioth, as having this done properly is a goal for the team in the first part of the year. I've read the entire thread to become familiar with everything discussed so far, and I will be incorporating some of the suggestions discussed here as well as some decisions that we made as a team. All the development for now is happening on collab-maint/hhvm [1], as the current Github repo structure does not fit really well the layout of git-buidpackage. Best regards, Ender. [1]: http://git.debian.org/?p=collab-maint/hhvm.git;a=summary signature.asc Description: OpenPGP digital signature
Bug#727085: Taking over packaging in Debian.
Hi Ender, all,
On Fri, Feb 28, 2014 at 8:16 PM, David Martínez Moreno en...@debian.org wrote:
Hello all. My name is Ender, I have been a Debian developer for
quite some
time and I work for Facebook, so I decided to do proper packaging of hhvm in
Alioth, as having this done properly is a goal for the team in the first part
of
the year.
I was wondering if one of us is working for Facebook or not. I
suspected yes, as we are everywhere and I'm right.
I've read the entire thread to become familiar with everything
discussed so
far, and I will be incorporating some of the suggestions discussed here as
well
as some decisions that we made as a team.
Can you share those decisions?
All the development for now is happening on collab-maint/hhvm [1], as
the
current Github repo structure does not fit really well the layout of
git-buidpackage.
Quickly peeking into your packaging I do wonder if you read the
entire bug thread.
For example I've discussed architecture support with Paul Tarjan
(message 50)[1]:
-- cut --
From: Paul Tarjan p...@fb.com
Date: Wed, 6 Nov 2013 16:51:46 +
Last but not least that wiki forces HHVM to be amd64
only. Any reason to do that? Any known problem to run HHVM on
kfreebsd-*, mips{,el}, sparc or any other archs that Debian supports?
Absolutely. HHVM is a Just-In-Time compiler which emits assembly code into
memory then executes it. We currently only emit 64bit x86. We are looking
at other backends, but they require a lot of effort to implement and
optimize.
-- cut --
Still, you put the binary package architecture any, instead of only
amd64. Does Facebook supports for example mips now?
In that mail you can also read that I've identified most of the build
dependencies. May I ask why you don't use any of that?
Can it be that you use the embedded sources, those as Paul acknowledged:
-- cut --
I've just realized that HHVM
embeds other codes under hphp/third_party/ , even folly which is an
other FB FOSS software and should be packaged separately.
Folly is pulled in with a git submodule, but yes, the others are copied in.
-- cut --
Those should be ironed out and use the packaged versions expect Folly
as an other Facebook employee wrote the following.
-- cut --
From: Jordan DeLong delon...@fb.com
Date: Sat, 4 Jan 2014 19:44:26 -0800
On Sat, Jan 04, 2014 at 11:53:25PM +0100, László Böszörményi (GCS) wrote:
Question is, does Folly maintain ABI compatibility? If it changes
from time-to-time, how often?
Yeah, it doesn't attempt to maintain ABI backward compatability, and
we haven't done much about tracking when we break source-level
backward compatability either. (As Sara said, we don't version it
currently... unless you count the submodule in hhvm ;)
There are changes probably a few times a week, although I'd suspect
few of the changes that aren't to new components (usually in
folly/experimental) actually break source backward compat.
-- cut --
As such, we have to trust that Folly is as up to date as possible in
HHVM and it will be supported for Debian/Ubuntu stable releases (at
least five years if I count in Ubuntu LTS releases).
I've asked Paul about other embedded libraries, but didn't get an answer.
-- cut --
But problems arise with other libs as well. For example libafdt is
version 0.1.0 (hardly a version number to call it final and stable),
which was last released in 2009 (yes, almost five years ago!). I've a
package ready, but it doesn't build a shared library, just a static
one and it has only one header file. I may upload it if you really
want, but I don't see a reason for that. I've a package of libmbfl as
well, version 1.3.2 , which is 'just' two years old. The version
included in HHVM is 1.0.2 (according to
hphp/third_party/libmbfl/mbfl.rc) and I don't want to think about how
old is it. Don't forget the Folly state (no ABI and neither source
stability), the history of old libevent version usage, makes me wonder
how HHVM works in the long run, how supportable is it. There's 'good'
news of course, like Sqlite3 is version 3.7.15.2 which is 'just' one
year old. While its homepage shows that the last release is 3.8.2 and
upgrading from versions before 3.8.1 is strongly recommended due to
the number of bugs fixed. OK, the only one database corruption bug
fixed (in 3.7.16.2) since the included 3.7.15.2 is a rare one.
Still, forgive me Paul for asking but why do you include and use well
rotten, quite old versions of libraries? Several of them are packaged
in modern distributions and kept fresh and clean.
-- cut --
May you answer this part? For example Sqlite3 became more than a year
old in HHVM now. Upstream version is 3.8.3.1 ATM. Paul didn't answer
this for a while now. :-(
I accept the fact that Facebook wants this packaged for Debian in the
first half of 2014 and you take over the packaging without asking me
(probably ignoring Ondřej and Faidon as I didn't see them as Cc in
your mail), but as you read in the mails you mentioned and I
