UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

For help and support with Universal Media Server
Forum rules
Please make sure you follow the Problem Reporting Guidelines before posting if you want a reply
wired4net
Posts: 5
Joined: Mon Apr 23, 2018 7:45 am

UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by wired4net »

Sirs,

I have a Pi3b running latest stretch and on which I installed UMS 7.0.1

It seemed to run pretty well for music when I used some small test collections of mp3s. (Oddly it does not work for me at all for streaming video to a Google chromecast, even though a prior version 7.0.0 rc2 worked fine in a VMWare virtual machine running Debian Cinnamon, but that is another story.)

Anyway, on the Pi3b, I then added a large collection of music. Something like 21,000+ mp3 files and about 120_GB. Every time I try to load a playlist using any one of a number of different clients (e.g. iPhone app, Foobar2000 on Windows 10) UMS crashes with not much fanfare. I tried running with Trace log (attached here), in which I notice lots of complaints about formats, but not ending with a fatal error as far as I can tell.

I wonder if this is just too large a collection for UMS to handle? I will say that minidlna on the same Pi has handled all of this music and then some without a problem, as well as streaming video fine to the Chromecast. I would have stuck with minidlna, but do not seem to able to get it to rescan additions without entirely rebuilding its db, which is a real pain). I was hoping to use minidlna for video and UMS for music.

Is there any chance you could take a look and let me know if you have any insights into what is going on? I would certainly appreciate it.

Regards,
Wired4Net
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by Nadahar »

The log isn't in "trace" mode, so it contains little of value. If you start UMS from the command line, simply use "./UMS.sh trace".

What little I see it that both mplayer and ffmpeg fails. UMS comes with bundled software for transcoding, and my guess is that these must be replaced with versions compiled for the Pi. The bundles ones are x86/x64.

You don't say if you have done so, but you should also install MediaInfo. Once those things are taken care of, post a trace log.
wired4net
Posts: 5
Joined: Mon Apr 23, 2018 7:45 am

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by wired4net »

Gee, I thought I ran it in trace mode. Of course, then it crashed and I had to restart it to Pack up the files. So the useful log should be debug.log.prev, which is very long and very detailed.

Plus, it (debug.log.prev) has the line: DEBUG 2018-05-11 11:52:03.492 [main] net.pms.PMS Forcing debug level to TRACE

and a page or two down from that, most of the lines start with "TRACE"

Any chance you looked at the wrong log?

Regards,

Wired4Net
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by Nadahar »

Sorry, I didn't look at the ".prev" file. It was indeed a trace log.

I didn't reveal much more though, but there are errors everytime UMS tries to run MPlayer, MEncoder or FFmpeg. You need to replace the binaries found in the "linux" folder with ARM compiled versions. Until that is done I don't think there is much point in troubleshooting, as UMS relies on external programs for both parsing and transcoding.

I see one potential issue though, and that is the low amount of RAM. It seems like you only have 1 GB, and 768 MB of those are assigned to the JVM (UMS). That leaves very little RAM for everything else, including the external programs UMS tries to run. I think you need to lower the amount of RAM dedicated to Java to maybe 512 MB. You can do this by editing UMS.sh.
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by Nadahar »

Ah, I just remembered what your original isse was. UMS should have no problem handling 21000 MP3s, but it does require a significant amount of RAM. To have a chance of handling that many files with so little RAM, you should turn off thumbnails completely. That means, turn off using Cover Art Archive, turn off "Generate thumbnails" and select "Disable thumbnail" (or something like that).
wired4net
Posts: 5
Joined: Mon Apr 23, 2018 7:45 am

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by wired4net »

I had already arrived at the idea of turning off generation of thumbnails, but hadn't turned off cover art, which I have now done.

I also have put a compatible version of ffmpeg in the Linux folder. Haven't installed MEncoder or MPlayer yet, although will do so soon. (I do have a compatible version of VLC on the machine, so would I still need MPlayer?) At least the ffmpeg error went away.

I also reduced the max memory allowed to 512MB, as you suggested. Also eliminated a shared folder with lots of .jpgs that I had included before, just to simplify things.

The app still crashes in the middle of processing when the client asks to stream a (fairly large) playlist of mp3 files.

Which brings me to a central question I have. When I start the UMS up with all these mp3s present and no database files, it does recreate the two db files (medias.mv.db and medias.trace.db), but they stay very small. Similarly, the log in the logs tab and the one I get when I click on "Open full debug log" both seem to indicate that the app starts up, registers transcoding engines and renderers, and then just stops doing anything more. No more log entries and cpu stays at about 1%.

It is only when the client (Foobar2000 in this case) accesses the UMS server and drills down to the playlists I have present that the app appears to start logging the mp3s and, apparently, adding them to the database. Is this normal behavior? The net effect is that when a client wants to stream a playlist, he has to wait while UMS rather slowly examines each mp3 file one by one -- you can see them go by slowly in the "live" log tab -- it would take forever for the client to get any streaming music. And, the UMS app on my RPi then always crashes somewhere along the line as it is responding to the request to stream the playlist. Isn't there a way to force the app to process all the files when it starts up, before any client asks to stream anything?

Thanks for helping me out with this. By the way, as you may see, I deleted the attachment I provided with the previous trace. When I looked at it again, it seemed to have a bit too much information about my network, devices, IP addresses, etc. to have it posted indefinitely on a public forum where it can be downloaded even by a visitor. If you decide you need further traces, I would rather PM them than post them publicly if that is ok.

Regards,

Wired4net
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by Nadahar »

MPlayer/MEncoder shouldn't be critical, it's mostly used for playing DVDs or videos with "tricky" subtitles that FFmpeg can't handle. The problem is that UMS expects it to be available (since it is bundled), so it will call it when needed. VLC isn't a substitute for the things MPlayer/MEncoder are especially good at.

The behavior you're describing is how UMS worked before 7.0.0. From 7.0.0 on however, the behavior is changed so that a full scan of the library starts when you start UMS. I suspect that this is causing the problem, especially if your storage is slow. The trace log you posted did show that this scan ran for a while before the log stopped. With such a large library and a relatively weak hardware, the library scan will probably take a considerable amount of time and it's probably still in the middle of this when you start playing. UMS 6.8.0 could be worth a try. The reason you don't see this in the log is probably because of the chosen log level. With trace logging enabled you should see the scanning taking place.

When it comes to playlists I don't think that code has gotten any TLC in a long time. The playlist library UMS use is quite old, and I don't how much knowledge about how well this works. Can you play without problems if you don't use playlists?

The thing I don't understand at all, is the crash itself. Usually if UMS has a fatal error, it will be logged. Something must be causing it to terminate so quickly that whatever is causing it isn't logged. This is why I suspect something very fundamental, like memory error or CPU incompatibility.

Running on Pi, I would think that you would run UMS headless (without GUI) to save resources. From your log it seems like you're running UMS with GUI. In case you're not aware of it, you can start UMS in headless mode by using "headless" as a command line argument. In any case, headless mode might help us find out what's crashing it, since UMS will log directly to console in headless mode - so the log messages immediately before the crash might shed some light on what happening. The console logging is filtered by default though, so you need to add "logging_filter_console = TRACE" in your UMS.conf (there's no GUI option for this) for trace log messages to make it to the console.

I would suggest that you set "logging_filter_console = TRACE" and then start UMS with "./UMS.sh headless trace" to try to find the cause. Every detail will then scroll over the terminal in realtime. Ctrl + C will make UMS shutdown. If you let is run like this until the crash happens, the last lines might reveal what is going wrong.
wired4net
Posts: 5
Joined: Mon Apr 23, 2018 7:45 am

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by wired4net »

Nadahar,

Thank you so much for your prompt and very detailed reply.

I have been looking at this today and found that because I had turned off the "Enable the Media Library" flag, since I didn't really use it, UMS apparently decided not to scan on start. When I re-enabled this flag it did scan on start. Unfortunately it did crash again when I tried to stream music. Admittedly, this was before it finished cataloging, so that may be part of the problem.

A couple of things I don't understand from the log before that crash:

1. It seems like even though I had turned off all of the thumbnail related flags (Generate thumbnails, Image thumbnails, Audio Thumbnails import:none, etc., it seemed to still be doing some thumbnail processing. There were a number of lines like:

TRACE 2018-05-12 13:41:48.868 [Library Scanner] Converting byte array image source to SOURCE format and type DLNAThumbnail using the following parameters: ScaleType = MAX, Width = 640, Height = 480, PadToSize = False
TRACE 2018-05-12 13:41:48.998 [Library Scanner] Image conversion DLNA compliance check: The source image is compliant
TRACE 2018-05-12 13:41:48.998 [Library Scanner] No conversion is needed, returning source image with width = 600, height = 460 and output profile: PNG_LRG.

2. Also, the UMS had problems with a number of mp3 files (but certainly not all), as evidenced by lines like:

TRACE 2018-05-12 13:41:50.908 [Library Scanner] Matched format MP3 to "/media/pi/VIDEO_01/aMusic/Fav_01a/Annie Lennox/Songs of Mass Destruction/02 Love Is blind.mp3"
TRACE 2018-05-12 13:41:52.405 [Library Scanner] Parsing results for file "02 Love Is blind.mp3": Size: 0, Mime Type: null
INFO 2018-05-12 13:41:52.406 [Library Scanner] The file /media/pi/VIDEO_01/aMusic/Fav_01a/Annie Lennox/Songs of Mass Destruction/02 Love Is blind.mp3 could not be parsed. It will be hidden

Then I restarted it in trace log mode and have been just leaving it alone as it slowly processed all the files. That was over 4 hours ago and it is still going. CPU usage on the RPi3B is generally not more than about 30% even though I am connected to the GUI and watching the application GUI, and memory usage has seemed to be never more than about 250 MB out of the available 500 MB. Perhaps the RPi3B is more up to the task than you might think?

Your comments about playlists are illuminating. I have been a music fan for a VERY long time, and since computers came along, being able to use a playlist has been a key part of how I listen. (After all, I spent a lot of years spinning LP's one at a time and I now relish the ability to just let the music run). Anyway, it is amazing how for many streaming server apps, software music players, and even music playing devices, playlists seem like a kind of afterthought. That has made my quest for a home music server that much more difficult. I am hoping I can make UMS serve that purpose adequately.

Anyway, I will continue this test and post any useful results. Thanks again for your help. BTW, may I know what time zone you are in? It would help me to focus any conversation on the right time of day. I am in the Pacific Daylight (GMT-8 but DST) time zone.

Regards,
Wired4Net

P. S. While writing this, the UMS has finished cataloging the files (after about 3.5 hours) without crashing! Now I am accessing it from Foobar2000. I opened the Playlists folder which contains about 5 playlists -- some large and some small. Lo and behold, the "live" log shows the server going through all of these playlists one file at a time and for each file entering in the log a line like:

INFO add global: [mp3 filename here]

This takes a long time and, in the meantime, some of the files in the playlist show up in Foobar.
OOPS, DAMN! I messed up the whole test by trying to simultaneously transfer a video file to the USB-attached drive from my windows machine!
This caused the whole RPi to drop off the network and I had to reboot. I'll have to continue this test tomorrow.
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by Nadahar »

I wrote an answer yesterday, but by the time I was ready to submit it, the server running the forum had crashed :cry: I guess there's nothing else to do than to start again..
wired4net wrote: Sun May 13, 2018 1:25 pm I have been looking at this today and found that because I had turned off the "Enable the Media Library" flag, since I didn't really use it, UMS apparently decided not to scan on start. When I re-enabled this flag it did scan on start.
Turning off the Media Library does in fact disable all use of the database for caching media information, and will make UMS a lot slower. Turning it off is mostly useful for debugging, and I wouldn't really recommended it in any other situation.
wired4net wrote: Sun May 13, 2018 1:25 pm 1. It seems like even though I had turned off all of the thumbnail related flags (Generate thumbnails, Image thumbnails, Audio Thumbnails import:none, etc., it seemed to still be doing some thumbnail processing. There were a number of lines like:

Code: Select all

TRACE 2018-05-12 13:41:48.868 [Library Scanner] Converting byte array image source to SOURCE format and type DLNAThumbnail using the following parameters: ScaleType = MAX, Width = 640, Height = 480, PadToSize = False
TRACE 2018-05-12 13:41:48.998 [Library Scanner] Image conversion DLNA compliance check: The source image is compliant
TRACE 2018-05-12 13:41:48.998 [Library Scanner] No conversion is needed, returning source image with width = 600, height = 460 and output profile: PNG_LRG.
This sounds very much like a bug. The code has to check the various settings many different places to respect them, and it seems like it has been forgotten somewhere. I'll see if I can find it.
wired4net wrote: Sun May 13, 2018 1:25 pm 2. Also, the UMS had problems with a number of mp3 files (but certainly not all), as evidenced by lines like:

Code: Select all

TRACE 2018-05-12 13:41:50.908 [Library Scanner] Matched format MP3 to "/media/pi/VIDEO_01/aMusic/Fav_01a/Annie Lennox/Songs of Mass Destruction/02 Love Is blind.mp3"
TRACE 2018-05-12 13:41:52.405 [Library Scanner] Parsing results for file "02 Love Is blind.mp3": Size: 0, Mime Type: null
INFO  2018-05-12 13:41:52.406 [Library Scanner] The file /media/pi/VIDEO_01/aMusic/Fav_01a/Annie Lennox/Songs of Mass Destruction/02 Love Is blind.mp3 could not be parsed. It will be hidden
It would be interesting to get either one of the problematic files or a MediaInfo report from one of them.
wired4net wrote: Sun May 13, 2018 1:25 pm Then I restarted it in trace log mode and have been just leaving it alone as it slowly processed all the files. That was over 4 hours ago and it is still going. CPU usage on the RPi3B is generally not more than about 30% even though I am connected to the GUI and watching the application GUI, and memory usage has seemed to be never more than about 250 MB out of the available 500 MB. Perhaps the RPi3B is more up to the task than you might think?
Hehe, I don't in which degree the RPi3B is up to the task, but I suspect you will hit some serious bumps in the road when it comes to transcoding ;) Except for transcoding, UMS is mostly I/O bound. It means that the speed is limited by the I/O speed most of the time, which means the transfer of data to/from disk, network etc. Although 21000 files is a lot, 4 hours sounds excessive to me. I'm wondering what the nature of the connection to the hard-drive is, and what speed this runs at.
wired4net wrote: Sun May 13, 2018 1:25 pm Your comments about playlists are illuminating. I have been a music fan for a VERY long time, and since computers came along, being able to use a playlist has been a key part of how I listen. (After all, I spent a lot of years spinning LP's one at a time and I now relish the ability to just let the music run). Anyway, it is amazing how for many streaming server apps, software music players, and even music playing devices, playlists seem like a kind of afterthought. That has made my quest for a home music server that much more difficult. I am hoping I can make UMS serve that purpose adequately.
For UMS I don't think this has ever been a conscious decision, but simply a result of how demand dictates what gets focus. There's no a lot of users that complaint about playlists, which tends to result in nobody looking at it.
wired4net wrote: Sun May 13, 2018 1:25 pm BTW, may I know what time zone you are in? It would help me to focus any conversation on the right time of day. I am in the Pacific Daylight (GMT-8 but DST) time zone.
I'm from Norway, so I'm in +1 although we're currently at +2 due to DST. I don't always adhere that much to what's considered "normal" hours though, but I have a lot of things on my plate so when I have the time to answer will vary.

wired4net wrote: Sun May 13, 2018 1:25 pm ...the "live" log shows the server going through all of these playlists one file at a time and for each file entering in the log a line like:

Code: Select all

INFO  add global: [mp3 filename here]
It sounds strange that this is logged as INFO, but all the files will have to be looked up by UMS to generate the "content" of the playlist as a folder. When the information about the files is in the database, it should be very quick though.
wired4net wrote: Sun May 13, 2018 1:25 pm OOPS, DAMN! I messed up the whole test by trying to simultaneously transfer a video file to the USB-attached drive from my windows machine!
This caused the whole RPi to drop off the network and I had to reboot. I'll have to continue this test tomorrow.
This part sparks some real interest from me and renews my suspicion that there are slow I/O. I suppose you have mapped this as a network drive? Does the RPi have any local storage at all? If so, what speed is it, and most important: Where is the database located, what speed does the drive with the database have?
Nadahar
Posts: 1990
Joined: Tue Jun 09, 2015 5:57 pm

Re: UMS 7.0.1 Crashing on Pi with Raspbian Stretch Lots of mp3s

Post by Nadahar »

It seems I was wrong about the thumbnails, we have no setting to disable all use of thumbnails. You should still save a lot of RAM by disabling all the available thumbnail options, and my guess is that those that still get thumbnails either have cover art in a file (JPEG or PNG) alongside the media file, or has embedded thumbnails.

I don't know if it's worth making it though, as it's really only an attempt to work around a "design problem" where media metadata is kept in memory.
Post Reply