[JIRA] (JENKINS-9190) deduplicating build artifacts
[ https://issues.jenkins-ci.org/browse/JENKINS-9190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=161613#comment-161613 ] yossis commented on JENKINS-9190: - This is possible in Artifactory using what we call [checksum deploy|http://wiki.jfrog.org/confluence/display/RTF/Artifactory%27s+REST+API#ArtifactorysRESTAPI-DeployArtifactbyChecksum]. Support in Jenkins plugin is coming in the next version. We'll probably use it only for artifacts bigger that 10KB. You can follow the following issue: https://issues.jfrog.org/jira/browse/BI-126. deduplicating build artifacts - Key: JENKINS-9190 URL: https://issues.jenkins-ci.org/browse/JENKINS-9190 Project: Jenkins Issue Type: Improvement Components: artifactory Affects Versions: current Reporter: Brian Murrell Assignee: yossis At least in our case, a project can produce quite a number of artifacts, some quite large and some which only change occasionally from one build to another (i.e. some artifacts change every time, some less frequently). It seems that both space and bandwidth could be saved by de-duplicating these seldom changed artifacts from one build to another. I imagine an algorithm where the server keeps a database of sums and sizes of stored artifacts and when a slave is going to send the artifacts of a build it first offers the sums and sizes of the artifacts. If the server finds potential matches, further verification of duplication could be performed (i.e. comparing random samples of the suspected duplicates) and once a duplicate has been confirmed, the server can either copy or link the artifact locally and tell the slave not to bother sending it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-9190) deduplicating build artifacts
[ https://issues.jenkins-ci.org/browse/JENKINS-9190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Murrell reopened JENKINS-9190: I don't find having to deploy an entirely new tool a reasonable resolution to this issue. The jenkins artifact archiving process is entirely suitable for our environment -- if a bit wasteful in the storage of numerous (copies even) of duplicate artifacts. I am sure Artifactory is a great tool for people who have a need for for it, but to simply de-duplicate artifacts it feels like using a sledgehammer to drive a finishing nail. deduplicating build artifacts - Key: JENKINS-9190 URL: https://issues.jenkins-ci.org/browse/JENKINS-9190 Project: Jenkins Issue Type: Improvement Components: artifactory Affects Versions: current Reporter: Brian Murrell Assignee: yossis At least in our case, a project can produce quite a number of artifacts, some quite large and some which only change occasionally from one build to another (i.e. some artifacts change every time, some less frequently). It seems that both space and bandwidth could be saved by de-duplicating these seldom changed artifacts from one build to another. I imagine an algorithm where the server keeps a database of sums and sizes of stored artifacts and when a slave is going to send the artifacts of a build it first offers the sums and sizes of the artifacts. If the server finds potential matches, further verification of duplication could be performed (i.e. comparing random samples of the suspected duplicates) and once a duplicate has been confirmed, the server can either copy or link the artifact locally and tell the slave not to bother sending it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-9190) deduplicating build artifacts
[ https://issues.jenkins-ci.org/browse/JENKINS-9190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] evernat updated JENKINS-9190: - Issue Type: Improvement (was: Bug) deduplicating build artifacts - Key: JENKINS-9190 URL: https://issues.jenkins-ci.org/browse/JENKINS-9190 Project: Jenkins Issue Type: Improvement Components: artifactory Affects Versions: current Reporter: Brian Murrell Assignee: yossis At least in our case, a project can produce quite a number of artifacts, some quite large and some which only change occasionally from one build to another (i.e. some artifacts change every time, some less frequently). It seems that both space and bandwidth could be saved by de-duplicating these seldom changed artifacts from one build to another. I imagine an algorithm where the server keeps a database of sums and sizes of stored artifacts and when a slave is going to send the artifacts of a build it first offers the sums and sizes of the artifacts. If the server finds potential matches, further verification of duplication could be performed (i.e. comparing random samples of the suspected duplicates) and once a duplicate has been confirmed, the server can either copy or link the artifact locally and tell the slave not to bother sending it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
