Possible Memory Leak - Out of memory 98% processor

For help and support with Universal Media Server
Forum rules
Please make sure you follow the Problem Reporting Guidelines before posting if you want a reply
uxb
Posts: 7
Joined: Wed Jan 04, 2017 10:05 am

Possible Memory Leak - Out of memory 98% processor

Post by uxb »

Good day. I have been using UMS for several years now and never had an issue with any version. It has been running on a Win 2008 server and I was using it with the Western Digital WD TV Live streaming box. With version 6.X I switched to the 64Bit version and continued with no issues using the WD TV Live renderer. I am currently running version 6.5.2. The memory utilization was always very low never getting above the first tick mark on the UMS status tab processor utilization was negligible.

In December I purchased a new TV, A LG-UH6030 and now every couple of days the java process runs out of memory and the processor utilization for it spikes to 98% and then needs to be manually terminated. This is repeatable. This last time I ran UMS in trace mode.

I have enclosed what the packed debug information would have by manually zipping them because when the process runs away UMS can not create the zip file. Also enclosed are screenshots of the processes running at the time of UMS start and UMS failure.

I am not overly familiar with DLNA servers and clients but I am assuming that there must be something the LG TV is doing talking to the server that is different than the WD TV LIve streaming box since that is the only thing (other than the media in the folders) that has changed .

The file was too large to attach so you can download it here: http://www.uxb.net/ums_debug.zip
uxb
Posts: 7
Joined: Wed Jan 04, 2017 10:05 am

Re: Possible Memory Leak - Out of memory 98% processor

Post by uxb »

As I said this is repeatable. I installed version 6.5.3 (64bit Java 8 version) and after a day the UMS process went from negligible process utilization to 90-98% at which point it had exceeded the Java memory limitation and stoped serving videos. This time I was able to have UMS create the debug archive which was again too large to attach to this message (11mb) so you can download it from http://www.uxb.net/ums_dbg.zip

I looked at the log but can't really make sense of it but it appears as if there are a number of these errors occurring:

DEBUG 2017-01-11 10:09:51.318 [New I/O worker #3] net.pms.network.RequestHandlerV2 Connection error: java.io.IOException: An existing connection was forcibly closed by the remote host

And then at the point when the Processor goes to 98% on that process the log shows a number of java exceptions starting with:

DEBUG 2017-01-12 10:46:46.466 [New I/O worker #1] net.pms.network.RequestHandlerV2 Caught exception: GC overhead limit exceeded
TRACE 2017-01-12 10:46:46.466 [New I/O worker #1] net.pms.network.RequestHandlerV2
java.lang.OutOfMemoryError: GC overhead limit exceeded

Which I assume is when it runs out of memory. So my question is. Why is the memory constantly creeping up and not being released until it finally runs out? Is it something to do with my system or renderer? Is it the number of files I have crashing the database? or is it something else?

Again since version 2 I had been running UMS with a WD TV Live streaming box with no issues and this only occurred when I purchased an LG TV. Any assistance is appreciated.
uxb
Posts: 7
Joined: Wed Jan 04, 2017 10:05 am

Re: Possible Memory Leak - Out of memory 98% processor

Post by uxb »

Well it appears as if I have found what is causing the out of memory error then 98% CPU utilization. It does not appear to be the renderer or the TV.

It seems that it is the number of mp3 files or something wrong with one of more of them that I have in my music folder. Looking through the trace logs I noticed that it was taking up quite a lot of time indexing the files in my music folder and subfolders. That folder consists of over 22,000 files in over 1500 sub folders. After a day or so of indexing the the Java memory usage would continue to grow until it ran out which would then cause the CPU spike.

I removed that folder from shared folders configuration and the memory usage remains very low after 2 days of usage. Apparently there are limitations to the number of files that the media server can index. Is there any way to know other than by trial and error what the limit is?
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: Possible Memory Leak - Out of memory 98% processor

Post by Nadahar »

There isn't supposed to be any limit, but it seems there is a leak somewhere which adds up over time. I have no idea how to find it from the top of my head.
User avatar
Sami32
Posts: 851
Joined: Mon Apr 11, 2016 5:09 am

Re: Possible Memory Leak - Out of memory 98% processor

Post by Sami32 »

Hello uxb,

Do you have folders with more than 1000 files inside ? If withdrawing them from the shared folders solve your issue, you can add

Code: Select all

LimitFolders=1000
into your renderer configuration file.

But it seem that you need to increase you Java Heap memory size.
So, you can reintall UMS and choice a bigger value (for example: 2048M,3072M,4096M), or change you Heap memory size following these steps under UMS advanced section: https://github.com/UniversalMediaServer ... stallation

Additionally, you could enable the concurrent low pause collector

Code: Select all

-XX:+UseConcMarkSweepGC
and some Java Garbage collector optimization:

Code: Select all

-XX:+UseG1GC -XX:+UseStringDeduplication
Hoping that can help you.
Please, don't forget to return feedback.
Last edited by Sami32 on Wed Jan 18, 2017 1:33 pm, edited 9 times in total.
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: Possible Memory Leak - Out of memory 98% processor

Post by Nadahar »

@Sami he has already reported that the problem goes away if the music folder isn't shared.

@uxb Thinking some more about this, I'm not so sure there is a leak. I think this is more of a design issue. Unless I am mistaken, UMS will keep parsing information for all scanned or accessed elements in memory until UMS is restarted. Even though the parsing information isn't that big, it adds up. My guess is that this design simply can't handle that many files, and a fundamental redesign would be needed to address this.

When it comes to the memory utilization bar on the status page, that is actually very misleading and should either be improved a lot or removed imo. Unless I'm completely mistaken it only shows the memory used for buffering while transcoding and completely ignores all other memory used by UMS. I guess the author figured that transcoding buffering was "all that really mattered", but I doubt that is true for large shared libraries. If you look at memory utilization shown by the OS you should get a better idea of actual use.
User avatar
valib
Developer
Posts: 699
Joined: Fri Feb 08, 2013 3:11 am

Re: Possible Memory Leak - Out of memory 98% processor

Post by valib »

Nadahar the memory utilization bar shows memory which JVM is used not only which the UMS is using. It means all memory used for running the UMS. External engines used for transcoding are not included in it.
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: Possible Memory Leak - Out of memory 98% processor

Post by Nadahar »

@valib Ok, then I made a wrong assumption, but I know I've seen related code doing a manual calculation of used memory. Are you saying that is simply leftovers from an earlier implementation that's no longer used?
User avatar
valib
Developer
Posts: 699
Joined: Fri Feb 08, 2013 3:11 am

Re: Possible Memory Leak - Out of memory 98% processor

Post by valib »

I think that the manual memory calculation is used only in the PMS.java which actually should be renamed to UMS.java when keeping the backward compatiility with the PMS is no longer an issue.

EDIT: renaming from PMS to UMS should have an influence on the plugins.
uxb
Posts: 7
Joined: Wed Jan 04, 2017 10:05 am

Re: Possible Memory Leak - Out of memory 98% processor

Post by uxb »

Sami32 wrote:Hello uxb,

Do you have folders with more than 1000 files inside ? If withdrawing them from the shared folders solve your issue, you can add

Code: Select all

LimitFolders=1000
into your renderer configuration file.
The folder in question that caused the problem consists of 1500 sub folders that in total contain over 22,000 files. Removing it from the shared folders solved the issue and I have run for a week now without any memory issue.
Nadahar wrote: @uxb Thinking some more about this, I'm not so sure there is a leak. I think this is more of a design issue. [....] My guess is that this design simply can't handle that many files, and a fundamental redesign would be needed to address this.
I suspect the same. The index must be built and or sorted in memory before written to the internal database so the number of objects it could handle would be memory dependant. If I have some time I will play with the Java heap size and the number of files to see if I can anecdotally determine some limits.

Since I have "cut the cord" I mostly use the UMS application to serve video files and don't keep a running archive of them like you would for music. After all how many times can you watch the same TV show?
Post Reply