GC issues with Java 1.7.0 u 21 (32 bit; Linux)

General discussion about Universal Media Server (no support or requests)
elsabio
Posts: 17
Joined: Thu Jan 31, 2013 7:52 am
Location: UK

GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by elsabio »

I only make this comment because I notice that the 1.6.2 release of UMS for Windows promotes usage of the 1.7.0 u 21 Java VM.

I find that u 21 has made significant changes to GC policy and as such, even if I double my max heap limit (to -Xmx1536M) I am unable to complete a scan of my media without Out of Heap Memory exceptions. This applies to 1.6.1 and 1.6.2 releases. The u 15 Java VM (from which I upgraded on this system) appears to have a more rapid GC cycle (or just a different policy) and completes my media scan with a 768M max heap.

This may be Linux/32 bit only of course.

I will look at the Java bug parade at some point to see if this is already understood. There have been several alternative GC policies introduced into Java 7, it may be that the default variant has changed or that some tuning has occurred.

Mark.
Last edited by elsabio on Tue Apr 23, 2013 10:11 pm, edited 1 time in total.
User avatar
Optimus_prime
Posts: 678
Joined: Fri Jun 01, 2012 6:39 pm
Location: Sydney, Australia

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by Optimus_prime »

can you give me your system spec's, ram etc.
How Ask For Support
Remember, Debug Log's Can/Will Help and Explain your issues, we're not mind reader's but here to help
OS's I Use And Can Assist With: Windows 7/8, Mac OS-X 10.8 & 10.9
Mac OS-X Java 7 Builds Mac OS-X Java 7 Forum
elsabio
Posts: 17
Joined: Thu Jan 31, 2013 7:52 am
Location: UK

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by elsabio »

Optimus_prime wrote:can you give me your system spec's, ram etc.
Sure - though I'm pretty sure this is a Java VM issue.

RAM: 3GB
Swap: 5GB
CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz
OS: Fedora fc14 32 bit
Kernel: 2.6.35.14-106.fc14.i686
Java: (Working) 1.7.0_15-b03
Java: (Failing) 1.7.0_21-b11

Collection size (held on external 6TB RAID 5 NAS):

Music: 841G in 21089 files
Video: 1882G in 1908 files
Pictures: 5.3G in 5667 files

Mark.
User avatar
Optimus_prime
Posts: 678
Joined: Fri Jun 01, 2012 6:39 pm
Location: Sydney, Australia

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by Optimus_prime »

ok whilst yes this can be thought of as a jvm issue it's not. The heap size is related to how much ram UMS can use at any one time. If the jvm cannot allocate itself what you have asked for then you will get errors in UMS starting up. That's why in windows for example we tried to allocate more if you had more than 4gb of ram but as it's a 32bit process it can only allocate a certain number, and we ran into big problems. When we moved to a 64bit jvm (in the new version it will automatically work this out for you, but still limited to prevent issues) you will inherently be able to allocate much more memory to the heap size as the amount of ram java can allocate is alot more. So long story short even on linux (fedora in your case) if your running a 32bit jvm you will only be able to allocate 800 to 900 max without problems. But to be safe i would recommend you stick to the default 768 allocated (-Xmx768 value on command line) so you don't run into problems.
How Ask For Support
Remember, Debug Log's Can/Will Help and Explain your issues, we're not mind reader's but here to help
OS's I Use And Can Assist With: Windows 7/8, Mac OS-X 10.8 & 10.9
Mac OS-X Java 7 Builds Mac OS-X Java 7 Forum
elsabio
Posts: 17
Joined: Thu Jan 31, 2013 7:52 am
Location: UK

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by elsabio »

Not entirely sure I agree with you ... :-)

These are not start-up failures - this is after a long period of scanning.

If we were discussing the initial heap size (set with -Xms), there would be limitations as you describe, and you are quite right that 64bit VMs would probably get around this (I can try this on another system later if I have time).
Setting a large max heap does mean that the GC becomes more lazy and won't free space with a zero refcount until it is within its threshold, the space is occupied for longer. So the "any one time" comment is a little more complex as the GC will not necessarily have reclaimed all the available heap, though the alternative GC policies (with which I could also experiment given time) address this in a variety of ways.
And you are right that the addressable space assigned to a 32 bit process is limited to 3G or so.

In my case ... leaving the max available heap at 768M fails to allocate enough after a while, and the change in behaviour (between completing a scan and failing with Out of Heap) is driven by only changing the VM, not by any parameter (I have so far tried) or UMS version change. It might be interesting to reduce the max heap if UMS only really needs a relatively small amount of heap. This would reduce performance because the GC would work harder: there is always a compromise :-)

Again, if I get time, I could do some profiling of this and see how the heap is being used, but that will have to wait.

[Update]...

It strikes me that the contiguous and reserve requirements of the heap would lead to a rapid failure when the jvm is started, and OutOfMemory is in that case usually fatal to the whole VM. What I did not say is that the scanner continues with a failure for each file it sees like this:

...
...

Code: Select all

ERROR 2013-04-23 00:22:17.408 [Library Scanner] Error adding child: Ms2e.jpg
java.lang.OutOfMemoryError: Java heap space
        at java.awt.image.DataBufferByte.<init>(DataBufferByte.java:76) ~[na:1.7.0_21]
        at java.awt.image.Raster.createInterleavedRaster(Raster.java:271) ~[na:1.7.0_21]
        at java.awt.image.BufferedImage.<init>(BufferedImage.java:382) ~[na:1.7.0_21]
        at net.coobird.thumbnailator.resizers.ProgressiveBilinearResizer.resize(Unknown Source) ~[ums.jar:2.6.1]
        at net.coobird.thumbnailator.resizers.Resizers.resize(Unknown Source) ~[ums.jar:2.6.1]
        at net.coobird.thumbnailator.makers.ThumbnailMaker.makeThumbnail(Unknown Source) ~[ums.jar:2.6.1]
        at net.coobird.thumbnailator.makers.FixedSizeThumbnailMaker.make(Unknown Source) ~[ums.jar:2.6.1]
        at net.coobird.thumbnailator.Thumbnailator.createThumbnail(Unknown Source) ~[ums.jar:2.6.1]
        at net.coobird.thumbnailator.Thumbnails$Builder.toFile(Unknown Source) ~[ums.jar:2.6.1]
        at net.pms.dlna.DLNAMediaInfo.parse(DLNAMediaInfo.java:702) ~[ums.jar:2.6.1]
        at net.pms.configuration.FormatConfiguration.parse(FormatConfiguration.java:270) ~[ums.jar:2.6.1]
        at net.pms.formats.Format.parse(Format.java:228) ~[ums.jar:2.6.1]
        at net.pms.dlna.RealFile.resolve(RealFile.java:188) ~[ums.jar:2.6.1]
        at net.pms.dlna.DLNAResource.run(DLNAResource.java:844) ~[ums.jar:2.6.1]
        at net.pms.dlna.RealFile.isValid(RealFile.java:61) ~[ums.jar:2.6.1]
        at net.pms.dlna.DLNAResource.addChild(DLNAResource.java:469) ~[ums.jar:2.6.1]
        at net.pms.dlna.MapFile.manageFile(MapFile.java:154) [ums.jar:2.6.1]
        at net.pms.dlna.MapFile.analyzeChildren(MapFile.java:207) [ums.jar:2.6.1]
        at net.pms.dlna.RootFolder.scan(RootFolder.java:215) [ums.jar:2.6.1]
        at net.pms.dlna.RootFolder.scan(RootFolder.java:225) [ums.jar:2.6.1]
        at net.pms.dlna.RootFolder.scan(RootFolder.java:173) [ums.jar:2.6.1]
        at net.pms.dlna.DLNAMediaDatabase.run(DLNAMediaDatabase.java:746) [ums.jar:2.6.1]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]
ERROR 2013-04-23 00:22:18.331 [Library Scanner] Error adding child: Nostalgia - sleeve.png
...
...

I wonder - could this be Native heap that is getting exhausted here?

Mark
elsabio
Posts: 17
Joined: Thu Jan 31, 2013 7:52 am
Location: UK

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by elsabio »

I know - replying to one's own posts is a bit naughty but ...

I've now tried UMS 1.6.2 with java 1.7.0_21 on a 64 bit (Fedora fc18) machine and ... it fails Out Of Heap in the same way too.
On that machine, the earlier jvm was 1.7.0_11, which does not have the same issue: it completes.

If it is of any relevance at all ... When I first tested this on 64 bit fc18, I did not have mediainfo installed. In that case, the scan completed rather quickly, as I would expect. I only see the Heap error after I had installed mediainfo.

Mark.
User avatar
Optimus_prime
Posts: 678
Joined: Fri Jun 01, 2012 6:39 pm
Location: Sydney, Australia

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by Optimus_prime »

hope you mean 2.6.1 and now i'm reading your post's, i think i jumped to a conclusion too fast. Your issue sounds strange but i'll dig around and see if i find an issue.
How Ask For Support
Remember, Debug Log's Can/Will Help and Explain your issues, we're not mind reader's but here to help
OS's I Use And Can Assist With: Windows 7/8, Mac OS-X 10.8 & 10.9
Mac OS-X Java 7 Builds Mac OS-X Java 7 Forum
User avatar
SubJunk
Lead Developer
Posts: 3705
Joined: Sun May 27, 2012 4:12 pm

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by SubJunk »

OK so the problem is occurring when a third-party program we use called thumbnailator attempts to create thumbnails of your image files. Do you have the cache enabled? Have you tried disabling it?

Edit: I have reproduced the bug now and will look into it
elsabio
Posts: 17
Joined: Thu Jan 31, 2013 7:52 am
Location: UK

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by elsabio »

Yes, but since my trigger to provoke this exception is by scanning my collection, no cache means I can't do that, so probably is just avoiding that path.

Oh - and yes, I do of course mean 2.6.1 (and 2.6.2, and 2.6.0, and 2.5.2 which all behave like this with Java 1.7u21, but not with u15).

BTW, though the stacktrace I posted corresponded to the processing of an image file, I see the same with flac files later too. But in that case, I don't get the stack trace, just the exception type and message so without some digging I don't know if this is also using thumnaillator to extract cover art (etc) from audio files.

Mark.
User avatar
SubJunk
Lead Developer
Posts: 3705
Joined: Sun May 27, 2012 4:12 pm

Re: GC issues with Java 1.7.0 u 21 (32 bit; Linux)

Post by SubJunk »

Yeah I have confirmed that it is caused by JRE 7u21, I was able to reproduce the bug with u21 but installing u17 fixed it. So I guess all we can do is recommend that users rollback to u17 since I can't see anything in our code that would cause that
Post Reply