Re: Alternative directory implementation of ArtifactManager?

[Jesse Glick]

On Wed, Dec 5, 2018 at 4:33 PM Staffan Forsell  wrote:
> I managed to get around it using ItemListener.onDelete() to call 
> run.deleteArtifacts() but I kind of think that this should be jenkins own 
> responsibility while deleting a job.

Perhaps. `Job.delete` could call `ArtifactManager.delete` on every
`Run`. But this would be inefficient for `StandardArtifactManager`
(much slower than just deleting the whole job directory), and useless
overhead for `JCloudsArtifactManager` (which would typically not
delete anything). For now I would just be inclined to leave the
`ItemListener` workaround as is.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2L7merKrDFFJiyF6NYG8H_wRtk78CQtVu_8TDm08BxeQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Staffan Forsell]

Well for us such retention policys does not apply and the time to keep 
artifacts is better handled though jenkins since it created them, than by 
other time based policies. We typically keep a number of artifacts per job 
while some builds are kept forever, typically release builds for 
traceability.
Well now I got most of it working but I discovered in my unit tests that 
deleting the job does not remove the artifacts, but removing the build does.
It seems that removing the job only recursively removed the job directory, 
see hudson.model.Job#delete.
To me this seems like a bug, it makes it harder to clean clean artifacts 
for a custom ArtifactsManager.
I managed to get around it using ItemListener.onDelete() to call 
run.deleteArtifacts() but I kind of think that this should be jenkins own 
responsibility while deleting a job.
Should I file an issue or do you have an other opinion?

On Tuesday, 4 December 2018 16:37:26 UTC+1, Jesse Glick wrote:
>
> On Tue, Dec 4, 2018 at 3:48 AM Staffan Forsell  > wrote: 
> > How are moves and renames handled? E.g. if a admin moves a job to a 
> folder? How does the ArtifactManager handle this? 
>
> For `StandardArtifactManager`, that would be implicit in the job 
> directory itself moving. For others, including `artifact-manager-s3`, 
> this is not handled automatically. `JCloudsArtifactManager.key` could 
> perhaps be made non-`transient` to track the originally stored 
> location, or there could be an `ItemListener.onLocationChanged` to 
> physically relocate the blob(s), or the `unique-id` plugin could be 
> relied on for another key which persists across moves. For now we 
> decided to keep things simple and transparent—there is a predictable 
> blob location based on what you see in Jenkins—and just punt on the 
> use case of historical builds preceding a move/rename. 
>
> (On a related note, by default `artifact-manager-s3` declines to 
> remove artifacts after build deletion in Jenkins, on the grounds that 
> AWS retention policies and Glacier migration are likely better, and we 
> want the Jenkins master to require minimal IAM permissions: a 
> compromised master should not be able to destroy or alter cloud data.) 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/c4e02423-e915-44b2-a5bc-0f2280b3764b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Jesse Glick]

On Tue, Dec 4, 2018 at 3:48 AM Staffan Forsell  wrote:
> How are moves and renames handled? E.g. if a admin moves a job to a folder? 
> How does the ArtifactManager handle this?

For `StandardArtifactManager`, that would be implicit in the job
directory itself moving. For others, including `artifact-manager-s3`,
this is not handled automatically. `JCloudsArtifactManager.key` could
perhaps be made non-`transient` to track the originally stored
location, or there could be an `ItemListener.onLocationChanged` to
physically relocate the blob(s), or the `unique-id` plugin could be
relied on for another key which persists across moves. For now we
decided to keep things simple and transparent—there is a predictable
blob location based on what you see in Jenkins—and just punt on the
use case of historical builds preceding a move/rename.

(On a related note, by default `artifact-manager-s3` declines to
remove artifacts after build deletion in Jenkins, on the grounds that
AWS retention policies and Glacier migration are likely better, and we
want the Jenkins master to require minimal IAM permissions: a
compromised master should not be able to destroy or alter cloud data.)

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2Wii6KnaMpkz_oAsLyH74JbkP%3Da4MEk8-twJHgWRu7tA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Staffan Forsell]

I started and got the basic function and configuration working.

How are moves and renames handled? E.g. if a admin moves a job to a folder? 
How does the ArtifactManager handle this?
I was thinking of saving the redirected artifact build dir in the build.xml 
in order to be able to keep the artifacts working in case of a move.
The alternative is to move the artifacts so they match the new build dir 
structure in the alternative location, but I don't know where I would do 
this...
I found StashAwareArtifactManager.copyAllArtifactsAndStashes() from the 
s3-plugn implementation, but this implies handling stashes also which is 
not in the scope for my ArtifactManager.
Is there a correct way of handling artifacts when jobs are moved?

/S

On Friday, 30 November 2018 20:06:28 UTC+1, Jesse Glick wrote:
>
> On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell  > wrote: 
> > artifact-manager-s3 seems to contain examples of most of the things 
> needed. 
>
> Too complicated for your purpose. Creating a plugin with a lightly 
> edited clone of `StandardArtifactManager` would suffice. (Use the 
> Maven archetype for a plugin with global configuration, so the user 
> can pick the location in the GUI or using configuration-as-code.) 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/a4d2e037-43a1-4f2c-abe3-286717cce3bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Jesse Glick]

On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell  wrote:
> artifact-manager-s3 seems to contain examples of most of the things needed.

Too complicated for your purpose. Creating a plugin with a lightly
edited clone of `StandardArtifactManager` would suffice. (Use the
Maven archetype for a plugin with global configuration, so the user
can pick the location in the GUI or using configuration-as-code.)

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0zFubqsfZeoW3uKeGESuky957ABM-4wjVGMh7tXs15BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Staffan Forsell]

Well, we have lots of jobs, both freestyle and different styles of the new 
pipeline, not all for sw that's currently in active development but we still 
need to know we can build these.
We would rather invest our time in something that's useful for all jobs without 
changing all those old job definitions.
Also everything that we keep for a while on Jenkins as an artifact is not 
necessarily something that we would deploy to artifactory. Coupling  the 
artifact retention to the number of builds in Jenkins is often easier than 
having a "works for all cleanup policy" in artifactory.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/429aa64d-c305-4bbe-b6a9-67a4c08a232c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Jesse Glick]

On Fri, Nov 30, 2018 at 8:59 AM Staffan Forsell  wrote:
> Another way of doing this would be to just invest in lots of SSD storage for 
> our jenkins servers.

Or just changing your projects to deploy artifacts to a proper
repository manager. The `archiveArtifacts` step and its kin is
intended only as a simple built-in way to retain smallish files if you
have not yet bothered to set up a real system.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3dFiBDEAsv%2BgeoGAFckAwfv1h1h1_deE4pAgbnokCQ0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Staffan Forsell]

artifact-manager-s3 seems to contain examples of most of the things needed. 
I did not quite understand...do you recommend against trying to do this or 
something else? If we would implement the same tests and use the same key 
as artifact-manager-s3, I would thing it would not be unreasonable work?

I figure if there is infrastructure now for this to work on s3 or azure, 
then just mapping it to a different directory should be pretty simple in my 
mind.

Another way of doing this would be to just invest in lots of SSD storage 
for our jenkins servers... Might be cheaper if this would prove to be very 
complicated.


On Friday, 30 November 2018 13:30:37 UTC+1, Daniel Beck wrote:
>
>
>
> > On 30. Nov 2018, at 10:40, Staffan Forsell  > wrote: 
> > 
> > Is there any obvious reasons that just creating a system property that 
> switches the artifacts to a different dir would not work? 
>
> Given how many regressions the custom builds dir introduced due to moving 
> jobs between folders, renaming them, etc., and all those actions newly 
> having to be mirrored in a different structure, I'm not surprised nobody 
> bothered -- just having custom builds dir is good enough in many cases. 
> Would not recommend attempting it with using job names for the lookup. 
>
> Would look at artifact-manager-s3 as a sort of template for custom 
> implementation for artifact storage. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/5b9104a7-43b9-4980-b193-6c7a6b4d0cb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative directory implementation of ArtifactManager?

[Markus Winter]

Implementation is quite easy I would say.
You need an ArtifactManager (look at the StandardArtifactManager, you 
can probably just copy and replace the getArtifactsDir() method)
And an ArtifactManagerFactory where you can configure the root path for 
the artifacts



On 30.11.2018 10:40, Staffan Forsell wrote:

Hello,
I have been looking for a way moving the artifacts out of the builds 
directory and have been looking at the ArtifactManager.
Is there a reason that I can't find an implementation of this that 
just uses a different local directory on the master?

Would should an plugin be hard to implement and how would one start?
Is there any obvious reasons that just creating a system property that 
switches the artifacts to a different dir would not work?


Our use-case is that we create a lot of artifacts (zips/vms etc) that 
we want to save for testing. We do this by changing the whole 
buildsDir to a glusterfs. This gives us massive amounts of space but 
the less than ideal performance with many small files, so we would 
like to separate the rest of the jobs and build files from the artifacts.


Best regards Staffan
--
You received this message because you are subscribed to the Google 
Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to jenkinsci-dev+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/00cbdc00-ed36-4a4d-9187-4af486e44822%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/6852ca44-3eac-ce6e-63de-32e1ee921daf%40gmx.de.
For more options, visit https://groups.google.com/d/optout.

Alternative directory implementation of ArtifactManager?

[Staffan Forsell]

Hello,
I have been looking for a way moving the artifacts out of the builds 
directory and have been looking at the ArtifactManager. 
Is there a reason that I can't find an implementation of this that just 
uses a different local directory on the master?
Would should an plugin be hard to implement and how would one start?
Is there any obvious reasons that just creating a system property that 
switches the artifacts to a different dir would not work?

Our use-case is that we create a lot of artifacts (zips/vms etc) that we 
want to save for testing. We do this by changing the whole buildsDir to a 
glusterfs. This gives us massive amounts of space but the less than ideal 
performance with many small files, so we would like to separate the rest of 
the jobs and build files from the artifacts.

Best regards Staffan

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/00cbdc00-ed36-4a4d-9187-4af486e44822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.