Re: Unable to push changes (access denied)
On 02/15/2018 07:27 AM, Antonio Trande wrote: > Hello. > > I can't push changes via GIT for 'coin-or-CoinUtils' package: > > $ git push > Counting objects: 3, done. > Delta compression using up to 2 threads. > Compressing objects: 100% (3/3), done. > Writing objects: 100% (3/3), 418 bytes | 418.00 KiB/s, done. > Total 3 (delta 2), reused 0 (delta 0) > remote: FATAL: W refs/heads/master rpms/coin-or-CoinUtils sagitter > DENIED by fallthru > remote: error: hook declined to update refs/heads/master > To ssh://pkgs.fedoraproject.org/rpms/coin-or-CoinUtils > ! [remote rejected] master -> master (hook declined) > error: failed to push some refs to > 'ssh://sagit...@pkgs.fedoraproject.org/rpms/coin-or-CoinUtils' I've regenerated the acls and you should be ok now. Let me know if not. kevin signature.asc Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: VDO
On Fri, Feb 16, 2018 at 03:47:08PM +0100, Miroslav Suchý wrote: > Dne 16.2.2018 v 14:56 Dusty Mabe napsal(a): > > Once we get it into Fedora this would be a great blog post for the > > magazine. What do you think? > > Yes. Sure. Would love to have a feature on this. Refer here for the process, if you haven't done an article before: https://fedoramagazine.org/writing-an-article-for-the-fedora-magazine/ -- Paul W. Frieldshttp://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Proposed zchunk file format
So here's my proposed file format for the zchunk file. Should I add some flags to facilitate possible different compression formats? +-+-+-+-+-+-+-+-+-+-+-+-+==+=+ | ID | Index size | Compressed Index | Compressed Dict | +-+-+-+-+-+-+-+-+-+-+-+-+==+=+ +===+===+ | Chunk | Chunk | ==> More chunks +===+===+ ID '\0ZCK', identifies file as zchunk file Index size This is a 64-bit unsigned integer containing the size of compressed index. Compressed Index This is the index, which is described in the next section. The index is compressed using standard zstd compression without a custom dictionary. Compressed Dict This is a custom dictionary used when compressing each chunk. Because each chunk is compressed completely separately from the others, the custom dictionary gives us much better overall compression. The custom dictionary is compressed using standard zstd compression without using a separate custom dictionary (for obvious reasons). Chunk This is a chunk of data, compressed using zstd with the custom dictionary provided above. The index: +++-+-+-+-+-+-+-+-+ | sha256sum | End of dict | +++-+-+-+-+-+-+-+-+ +++-+-+-+-+-+-+-+-+ | sha256sum | End of chunk | ==> More +++-+-+-+-+-+-+-+-+ sha256sum of compressed dict This is a binary sha256sum of the compressed chunk, used to detect whether two dicts are identical. End of dict This is the location of the end of the dict with 0 being the end of the index. This gives us the information we need to find and decompress the dict. sha256sum of compressed chunk This is a binary sha256sum of the compressed chunk, used to detect whether any two chunks are identical. End of chunk This is the location of the end of the chunk with 0 being the end of the index. This gives us the information we need to find and decompress each chunk. The index is designed to be able to be extracted from the file on the server and downloaded separately, to facilitate downloading only the parts of the file that are needed, but must then be re-embedded when assembling the file so the user only needs to keep one file. ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: A proof-of-concept for delta'ing repodata
On Tue, 2018-02-13 at 10:52 +0100, Igor Gnatenko wrote: > What about zstd? Also in latest version of lz4 there is support for > dictionaries too. So I've investigated zstd, and, here are my results: Latest F27 primary.gz - 3.1MB zlib zchunk (including custom dict) primary.zck - 4.2MB ~35% increase zstd zchunk (including dict generated from last three Fedora GA primaries) primary.zck - 3.7MB ~20% increase Using zstd for filelists.xml has roughly the same increase as with zlib, which is expected as the chunks are larger and thus get better compression even without a dict. I did also look briefly at lz4, but it seems that it's major advantage is speed, and I'm not sure that metadata decompression speed is our main bottleneck in dnf. With these numbers, I think it makes sense to move forward with zstd instead of zlib. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: Deploying Hubs to our Openshift instance
On 02/16/2018 07:37 AM, Pierre-Yves Chibon wrote: > On Fri, Feb 16, 2018 at 04:12:15PM +0100, Aurelien Bompard wrote: >>In the normal vm case, I have a couple questions: ...snip... >>Do we still require two DB users, one with CRUD permissions and one with >>full permissions? I haven't seen it used outside the hyperkitty playbook. >>If so, I'll need a password for the admin user too, and I'm interested in >>the way you give the privileges on the tables to the non-admin user. For >>HyperKitty I use a rather clumsy handwritten script, but there may be a >>better way. > > I know that I started using two users at one point but I ended up going back > to > a single one for most use-case as managing the permissions was clumsy indeed. > Updating the database schema required adjusting the permissions on table, on > indexes and forgetting one, getting a permission denied error and adjusting > again :s > Are you sure you want two users? > > I can create an user (hubs_db_user) and a database (hubs_db) and give the user > full access to the db if that works for you. Well, the idea was that we have a admin user that can change schema and drop things and the like and the 'normal' user that the app runs with that cannot do those things. That way if the application is compromised, they can only do things the normal user could do, not dropping entire tables or the like. I agree it's hard to setup perms just right for this. This would definitely be something it would be nice to have detailed docs on and I don't think we have any currently. kevin signature.asc Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: VDO
On 02/16/2018 05:47 AM, Miroslav Suchý wrote: > After I seen the talk about VDO: > https://www.youtube.com/watch?v=7CGr5LEAfRY > I went ahead and tried it. ...snip... > > I have to say that my experience is so far good. > > I hope that my notes help someone. Cool. Thanks for the notes/info. VDO indeed looks very interesting... kevin signature.asc Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: Deploying Hubs to our Openshift instance
> >- What's the Ipsilon instance I should register with? I used to > register > >on iddev.fedorainfracloud.org but I guess that's no good for staging > > I'll let Patrick answer, but I believe this is going to be id.stg.fp.o for > which > the registration is different (ie: not self-service) > OK, is there a way to register with it from the Ansible playbook? > >Do we still require two DB users, one with CRUD permissions and one > with > >full permissions? I haven't seen it used outside the hyperkitty > playbook. > >If so, I'll need a password for the admin user too, and I'm > interested in > >the way you give the privileges on the tables to the non-admin user. > For > >HyperKitty I use a rather clumsy handwritten script, but there may be > a > >better way. > > I know that I started using two users at one point but I ended up going > back to > a single one for most use-case as managing the permissions was clumsy > indeed. > Updating the database schema required adjusting the permissions on table, > on > indexes and forgetting one, getting a permission denied error and adjusting > again :s > Are you sure you want two users? > I would rather not, but I'll abide to any security policy we have. I can create an user (hubs_db_user) and a database (hubs_db) and give the > user > full access to the db if that works for you. > That would be the easiest. Aurélien ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: Deploying Hubs to our Openshift instance
On Fri, Feb 16, 2018 at 04:12:15PM +0100, Aurelien Bompard wrote: >In the normal vm case, I have a couple questions: > >- What will my URL be? https://hubs.stg.fedoraproject.org I guess? yes >- What's the Ipsilon instance I should register with? I used to register >on iddev.fedorainfracloud.org but I guess that's no good for staging I'll let Patrick answer, but I believe this is going to be id.stg.fp.o for which the registration is different (ie: not self-service) >- I need the following passwords set in the ansible private repo (proposed >ansible variable name) : >Â - for the hubs DB user (hubs_db_pass) >Â - for the Flask session secret key (hubs_session_secret), this can be >30-50 chars. Both created >Do we still require two DB users, one with CRUD permissions and one with >full permissions? I haven't seen it used outside the hyperkitty playbook. >If so, I'll need a password for the admin user too, and I'm interested in >the way you give the privileges on the tables to the non-admin user. For >HyperKitty I use a rather clumsy handwritten script, but there may be a >better way. I know that I started using two users at one point but I ended up going back to a single one for most use-case as managing the permissions was clumsy indeed. Updating the database schema required adjusting the permissions on table, on indexes and forgetting one, getting a permission denied error and adjusting again :s Are you sure you want two users? I can create an user (hubs_db_user) and a database (hubs_db) and give the user full access to the db if that works for you. Pierre ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: Deploying Hubs to our Openshift instance
In the normal vm case, I have a couple questions: - What will my URL be? https://hubs.stg.fedoraproject.org I guess? - What's the Ipsilon instance I should register with? I used to register on iddev.fedorainfracloud.org but I guess that's no good for staging - I need the following passwords set in the ansible private repo (proposed ansible variable name) : - for the hubs DB user (hubs_db_pass) - for the Flask session secret key (hubs_session_secret), this can be 30-50 chars. Do we still require two DB users, one with CRUD permissions and one with full permissions? I haven't seen it used outside the hyperkitty playbook. If so, I'll need a password for the admin user too, and I'm interested in the way you give the privileges on the tables to the non-admin user. For HyperKitty I use a rather clumsy handwritten script, but there may be a better way. I think that's all. Thanks! Aurélien ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: VDO
Dne 16.2.2018 v 14:56 Dusty Mabe napsal(a): > Once we get it into Fedora this would be a great blog post for the magazine. > What do you think? Yes. Sure. Miroslav ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Re: VDO
On 02/16/2018 08:47 AM, Miroslav Suchý wrote: > After I seen the talk about VDO: > https://www.youtube.com/watch?v=7CGr5LEAfRY > I went ahead and tried it. > I tried it on small (12GB) sample of Copr data and I saved 20-30% data. > I then deployed it on production server retrace.fedoraproject.org and saved > there 15% out of 2TB. Thanks for sharing! Once we get it into Fedora this would be a great blog post for the magazine. What do you think? I added Paul to this to see what he thinks as well. Dusty ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
VDO
After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
