-Archiver-multithreaded-Zip-edition-tp5822942p5826591.html
Sent from the Maven Developers mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h
ation should
have DMA access to that memory this way.
-
BR, tibor17
--
View this message in context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823268.html
Sent from the Maven Developers mailing list archive at Nabbl
byte
array, might be interesting too.
-
BR, tibor17
--
View this message in context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823241.html
Sent from the Maven Developers mailing list archive at Nabbl
in ThreadLocal copy for certainly
small thread pool.
I didn't measure the time spent on flushing ByteBuffer with 256KB byte
array, might be interesting too.
-
BR, tibor17
--
View this message in context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreade
t; Thread as it was before with more fancy join-functionality for us developers
> - nothing special - no direct selection of processor in Java code.
>
>
>
> -
> BR, tibor17
> --
> View this message in context:
> http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver
rkJoinPool is a pure pool with Java Threads - it's still the same
Thread as it was before with more fancy join-functionality for us developers
- nothing special - no direct selection of processor in Java code.
-
BR, tibor17
--
View this message in context:
http://maven.40175.n5.nabble.com/P
>
> TL;DR we should let the OS be responsible for applying resource limits, we
> should use every resource we are given to the full in order to reduce the
> build time as that shortens the feedback loop and makes developers mire
> productive
I guess I agree. Guidelines doesn't work anyways as the
2015-01-12 9:24 GMT+01:00 Kristian Rosenvold :
> Creating a 256MB zip file with typical jar content has a compression
> phase of 4 seconds on my 6 core machine, with 1.4 seconds in the
> "gather" phase. You'd need some really huge zips for this to become a
> real problem, and in that case I someho
2015-01-12 9:03 GMT+01:00 Anders Hammar :
> Do you foresee any issues with "use all available CPU cores" when used on a
> shared resource like CI server?
> When writing guidelines for my customers' devs I'm always very firm on that
> build jobs are not allowed to fork (or similar) freely and thereb
On Monday, 12 January 2015, Anders Hammar wrote:
> Do you foresee any issues with "use all available CPU cores" when used on a
> shared resource like CI server?
> When writing guidelines for my customers' devs I'm always very firm on that
> build jobs are not allowed to fork (or similar) freely a
vity" tool gave me this measurement results.
> The same test was done on Unix & Sun machine with RAID and the minimum
> latency was not such significant - even worse than with Windows.
>
>
>
> --
> View this message i
Do you foresee any issues with "use all available CPU cores" when used on a
shared resource like CI server?
When writing guidelines for my customers' devs I'm always very firm on that
build jobs are not allowed to fork (or similar) freely and thereby
occupy more than their share of the resources on
p; Sun machine with RAID and the minimum
latency was not such significant - even worse than with Windows.
--
View this message in context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823154.html
Sent from the Maven Developers mailing l
Nope, I did not miss it. DeferredFileOutputStream writes everything to
disk once it reaches the treshold, so it basically frees the allocated
memory buffers. Mine keeps the allocated buffers and creates a
SequenceInputStream over allocated buffers + disk file
Kristian
2015-01-10 23:14 GMT+01:00
Kristian Rosenvold wrote:
[snip]
> Inside commons-compress this target is always a
> tempfile. Inside plexus-archiver OffloadingOutputStream (a
> commons-compress ScatterOutputStream) is used. This writes to some
> pretty huge memory buffers, but when a certain treshold is reached it
> offloads t
DMA on
>> mapped memory really gains the performance.
>>
>>
>>
>> --
>> View this message in context:
>> http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823032.html
>> Sent from the Maven Developers ma
. So this DMA on
mapped memory really gains the performance.
--
View this message in context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823032.html
Sent from the Maven Developers mailing list archive at Nabble.com
omAccessFile on
> very large files 1GB got terribly fast throughput 400MB/s on ordinal hard
> drive however the flush/close operation was obviously slow. So this DMA on
> mapped memory really gains the performance.
>
>
>
> --
> View this message in context:
> http://maven.
/close operation was obviously slow. So this DMA on
mapped memory really gains the performance.
--
View this message in context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823032.html
Sent from the Maven Developers mailing list archive at
Where was the hotspot you gained the performance? Was it just the Java code
> you add ZipEntries in the stream, or parallel writes in file, or this
> improvement is specific on the hard drive? Does it apply to normal hard
> drive or SSD better maybe?
>
>
>
> --
> View this messa
context:
http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823013.html
Sent from the Maven Developers mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: dev-unsubscr
2015-01-10 16:46 GMT+01:00 Stefan Bodewig :
>> I originally had ConcurrentJarCreator in my c-compress fork. We discussed
>> this (arguably somewhat briefly) on the commons mailing list and to my
>> understanding Stefan wants c-c to be more of a toolkit (at a slightly lower
>> level) and did not wan
Cool. I'm going to try it on my project and a couple other tests and if it
makes things better I'll push a change for review.
On Jan 10, 2015, at 11:29 AM, Kristian Rosenvold
wrote:
> When I look at this code I see that there's probably a fair bit of
> work to do to bring this up to a level th
When I look at this code I see that there's probably a fair bit of
work to do to bring this up to a level that would fit for commons.
Since this does not really add any value for maven users I'm not
immediately willing to do this; I have other higher-value targets in
sight. Maybe sometime later. If
2015-01-10 16:58 GMT+01:00 Michael Osipov :
> the writeTo method uses currentTimeMillis to calculate elapsed time. You
> should rather turn that to nanoTime, see MNG-5626.
Damn. At my age System.currentTimeInMinutes() feels more appropriate
K
-
Am 2015-01-10 um 16:51 schrieb Kristian Rosenvold:
I'm probably mixing threads here. Better to do so in email than in code :)
The file we're talking about is
https://github.com/sonatype/plexus-archiver/blob/2.x/src/main/java/org/codehaus/plexus/archiver/zip/ConcurrentJarCreator.java
Kristian,
I'm probably mixing threads here. Better to do so in email than in code :)
The file we're talking about is
https://github.com/sonatype/plexus-archiver/blob/2.x/src/main/java/org/codehaus/plexus/archiver/zip/ConcurrentJarCreator.java
Kristian
2015-01-10 16:46 GMT+01:00 Stefan Bodewig :
> On 20
On 2015-01-10, Kristian Rosenvold wrote:
> 10. jan. 2015 15:19 skrev "Jason van Zyl" :
>> So I took a look and my feedback:
>> You have a blurb about how to use the parallel code in commons-compress
>> but there is no test or example that actually shows how to make one. The
>> actual working exa
10. jan. 2015 15:19 skrev "Jason van Zyl" :
> So I took a look and my feedback:
>
> You have a blurb about how to use the parallel code in commons-compress
> but there is no test or example that actually shows how to make one. The
> actual working example is in plexus-archiver and if you don't min
In the project I'm working on there is one module that has a very large JAR,
and the final application ZIP is super massive. So I'll try with those next and
report back with some metrics as well.
On Jan 10, 2015, at 9:06 AM, Kristian Rosenvold
wrote:
> Yes. Everything should be noticeably fas
So I took a look and my feedback:
You have a blurb about how to use the parallel code in commons-compress but
there is no test or example that actually shows how to make one. The actual
working example is in plexus-archiver and if you don't mind I'd like to put a
utility in commons-compress so
Yes. Everything should be noticeably faster. I have also been working on
features that allows transfer of compressed entries from one jar file to
another without packing/unpacking, which should be great for all kinds of
exploded archives inside zips/jars. I hope to get that into a second beta
soon.
Sounds like an awesome improvement! Will we see speed improvement on war
file creation with many megs of jars in the lib dir or only with
compressing of files into the archive (ignoring the web files for this
question)?
On Sat, Jan 10, 2015 at 7:43 AM, Kristian Rosenvold <
kristian.rosenv...@gma
I had 950% CPU usage on my 6 core + HT machine here the other day.
Kristian
2015-01-10 14:42 GMT+01:00 Kristian Rosenvold :
> It's faster; a lot faster - and it scales beautifully. But then
> again you probably need "war/ear/zip" heavy builds to really get max
> effect. The average "jar" plugi
It's faster; a lot faster - and it scales beautifully. But then
again you probably need "war/ear/zip" heavy builds to really get max
effect. The average "jar" plugin does not usually consume that large a
percentage the average build.
I'll try to make some nice graphs and a blog post some time th
Do you have any metrics? I use compress-compress directly so I can certainly
try it on some large archives. Anything special need to be done? Or do the same
code paths still work they just use the cores available with the new version?
On Jan 10, 2015, at 7:23 AM, Kristian Rosenvold
wrote:
> I
I just released plexus-archiver version 2.10-beta-1 to maven central.
This is a "technology preview" of the multithreaded Zip feature I have
been adding to commons-compress for the last few weeks, and will
basically use all available CPU cores when compressing the archive.
To test/use this featur
37 matches
Mail list logo
