Web Streams (with WAIT for download)

For help and support with Universal Media Server
Forum rules
Please make sure you follow the Problem Reporting Guidelines before posting if you want a reply
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Web Streams (with WAIT for download)

Post by AudioMan »

Some MP4's in webfeeds never play because the whole file [edit: or enough of the file if the header is at the start] needs to be downloaded first.
I'm wondering if there's a timeout in UMS to let it wait longer for the file to download, or if I can get Jumpy to do the downloading for me?

[Edit: thinking about a simple downloader script that you could use to save your http files for later playback. Will post at the bottom once I've tested it.]

-------
UMS seems to handle web streams quite well natively (so long as web.conf contains the actual streams, and not just a webpage -- use channels/jumpy plugin for web urls).
However, there are heaps of pseudo-streams out there that serve up a .MOV or .MP4 without the correct 'web optimised' format (they should put the MOOV containers at the start of the file).

In an ideal world you would use a better streaming format, to use the qt-faststart binary to optimise the mp4 for streaming, however we are not always in control of what is out there.

If we add say "videofeed.Web,TED=http://feeds.feedburner.com/tedtalks_video" to the web.conf file, UMS lists the correct streams, however it will always time out because UMS tells the encoder to send the http://...mp4 file to a named pipe on the local filesystem, but because of the container format, the encoder needs to download the entire file (let's say 50mb) before it sends any data to the pipe. It could take a couple of minutes to download the full file, by this time UMS has given up waiting to find any information in the named pipe, and gives up.

Is there somewhere we could set a (really large) timeout value so we could prevent this from happening?
Last edited by AudioMan on Sat Mar 14, 2015 7:00 pm, edited 3 times in total.
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Jumpy?

Post by AudioMan »

I thought about using Jumpy to add a 'player' that downloads the file (http://skeptical.github.io/jumpy/scripts.html).
You could then just play the local file in a client connected to UMS once is was downloaded.

The problem is that Jumpy scripts seems to show up on local files (file > # transcode # > [new-script]) but not for feeds listed in web.conf.
Last edited by AudioMan on Sun Feb 22, 2015 3:34 pm, edited 1 time in total.
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Download MP4 first (from Feed)

Post by AudioMan »

The question is, how can I get a download script (setup in Jumpy) to work on feeds in web.conf (see first post for TED example).
Should I be removing the listing from web.conf and get Jumpy to load the feed directly?
infidel
Developer
Posts: 571
Joined: Thu Jul 12, 2012 5:37 am

Re: Web Streams (with WAIT for download)

Post by infidel »

Hi, I've been following your posts, sorry for not responding sooner :).
AudioMan wrote:If we add say "videofeed.Web,TED=http://feeds.feedburner.com/tedtalks_video" to the web.conf file, UMS lists the correct streams, however it will always time out
I think you must be having some other timeout issue here, the TED feed should stream right away (it does for me) which is why it's included as an example in WEB.conf. Also its mp4s should already be optimized for streaming (moov atom frontloaded, etc).
AudioMan wrote:how can I get a download script (setup in Jumpy) to work on feeds in web.conf
Even though it may be moot now:
  • jumpy-scripts.ini one-liner:
    your original one-liner raised an interesting issue, namely that web items are initially assigned a generic 'web' format which is why 'f:mp4' wasn't working for web urls ending in '.mp4' ('f:http' would have worked, but this grabs all web media). Even though this likely isn't what you really want, out of curiosity I tried it and found a bug which is fixed in jumpy-0.3.4_player-patch.zip (replaces plugins/jumpy.jar), so now you can do stuff like this for example:

    Code: Select all

    [+flipflopweb]
    cmd = pms addPlayer flipflopweb "ffmpeg -y -i '${filename}' -vf 'vflip,hflip' -target ntsc-dvd ${outfile}" "f:http|mp4 m:video/mpeg" 
    
    You can control how long ums waits for output with the playback argument, but you have no control over your renderer's timeout.
  • An actual script:
    You could write a script to download the item and then add a local link to ums for separate playback, I'll go into it further if you think this is something you'd actually use :).
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Re: Web Streams (with WAIT for download)

Post by AudioMan »

Many thanks for your help! ;) I can confirm the MOOV is at the start (using qt-faststart --list), my assumption was from having tested ffmpeg from the command line, pointing it to the Ted url, and it only seemed to start processing once it had downloaded the entire file, which is the same command UMS uses during [Ffmpeg Web Video] transcodes, resulting in the timeout (maybe simply because of my download speed?).

jumpy-scripts.ini one-liner:
With your patch and using f:http I was able to get a jumpy-scripts.ini one-liner to work! (Am I right to think I can only set the wait time in a .py script using addPlayer?)

An actual script:
I'm not sure I'm ready to jump (pun intended) into using the API (yet)...
I would be comfortable chaining some commands together, but I think the jumpy-scripts.ini one-liner expects a single command. Any reason I couldn't say do this

Code: Select all

cd "$2" && nohup curl -o "$1" &
[[generate some fake ffmpeg output]] "$3"
#(ffmpeg -t 30 -s 640x480 -f rawvideo -pix_fmt rgb24 -r 25 -i /dev/zero -target ntsc-dvd) should do the trick...
Maybe into a bash script, and then using the jumpy-scripts.ini one-liner to add a player with

Code: Select all

scriptname.sh ${filename} /Media ${outfile}
And letting UMS find the files itself.

I think having an asynchronous download ability might still be handy, even if I can work out how to set the wait time?
infidel
Developer
Posts: 571
Joined: Thu Jul 12, 2012 5:37 am

Re: Web Streams (with WAIT for download)

Post by infidel »

AudioMan wrote:Am I right to think I can only set the wait time in a .py script using addPlayer?
No, you can supply all of the addPlayer() arguments in a one-liner, so a complete command like the one below would get you to the last argument 'delay:buffer', and should give you a 20 second delay in UMS. Note that in jumpy-scripts.ini you have to include the -1 part for buffer part (20:-1) otherwise you'll get a parse error.

Code: Select all

cmd = pms addPlayer flipflop2 "ffmpeg -y -i '${filename}' -vf 'vflip,hflip' -target ntsc-dvd ${outfile}" "f:http|mp4 m:video/mpeg" PMS_VIDEO PMS_MISC_PLAYER "Play stuff upside down" #raspberry 20:-1
But again, this will have no influence on your renderer.
AudioMan wrote:I would be comfortable chaining some commands together... into a bash script... [and] generate some fake ffmpeg output
Yeah, no reason to think it wouldn't work, you're supplying some output to keep UMS and the renderer happy and avoid unsightly errors :). For fake output try running 'jumpy/lib/vmsg.py' as a standalone script. To see a quick demo:

Code: Select all

 python vmsg.py -m "One moment please\n\n" -t 20 -x -b blue | ffplay -
which (with any luck :)) should generate:
onemoment.png
onemoment.png (22.01 KiB) Viewed 17371 times
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Re: Web Streams (with WAIT for download)

Post by AudioMan »

the last argument 'delay:buffer', and should give you a 20 second delay in UMS
Thanks very much! That worked brilliantly!

Something weird is happening if you use any single quotes near the edges of the double quotes - eg around ${outfile}:

Code: Select all

[+flipflop]
cmd = pms addPlayer flipflop "ffmpeg -y -i '${filename}' -vf 'vflip,hflip' -target ntsc-dvd '${outfile}'" "f:mpeg|vob m:video/mpeg"
Jumpy.log

Code: Select all

Py4JJavaError: An error occurred while calling t.addPlayer.
: java.lang.IllegalArgumentException: Unbalanced quotes in  , ffmpeg -y -i '${filename}' -vf 'vflip,hflip' -target ntsc-dvd '${outfile}
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Re: Web Streams (with WAIT for download)

Post by AudioMan »

[Edit: added URL redirection]
Here's the script:

Code: Select all

#!/bin/bash

# [downloadfirst.sh] A UMS/Jumpy Player for web.conf that downloads the URL first - 2015 AudioMan
# Works on URLS that can be downloaded with curl, and use HTTP. Eg: videofeed.Web,TED=http://feeds.feedburner.com/tedtalks_video
# Install by adding the following two lines to jumpy-scripts.ini
# ...    [+downloadfirst]
# ...    cmd = pms addPlayer downloadfirst "/Media/bin/downloadfirst.sh ${filename} /Media ${outfile} /Media/downloading.jpg" "f:http m:video/mpeg"
# And, make sure there's a an image or video local. Use anything you like as downloading.jpg (doesn't need to be jpg, just something ffmpeg recognises).
# Replace /Media with the appropriate location for you.  
# TODO - Doesn't seem to like spaces in the paths in jumpy-scripts.ini - something it eating single quotes near the double quotes.  
# If you don't use Jumpy, downloadfirst.sh might work as a standalone transcoding engine in UMS itself too, I'm not sure.  
#
#
# DESCRIPTION
# Saves a URL using curl in a separate process, and returns a dummy video message (eg downloading now), as well as attempt to return the requested file as it is downloading.  
#
# If your file doesn't play because not enough data was downloaded while the dummy message was being displayed, you could press play on your renderer again.  
# At this point more of the file will have been downloaded in the background.  
#
# If you stop the playback at any time, the file will continue downloading in the background. You could then play it as a normal local file
#
# REQUIREMENTS
# Make sure ffmpeg is in your system <path> variable.
# There was a bug in Jumpy 0.3.4 (and earlier?) - replace your jumpy.jar with a later one, or here https://sourceforge.net/projects/jumpy-plugin/files/util/jumpy-0.3.4_player-patch.zip/download
#
# RUNNING 
# Call with:
# downloadfirst.sh <url> <saveDirectory> <outputFile> <optional:imageOrVideoFile>
# If you pass an optional image file or videofile, we will display that instead of a blank screen 


echo "-Passed <url>: ${1}" ; echo "-Passed <saveDirectory>: ${2}" ; echo "-Passed <outputFile>: ${3}" ; echo "-Passed <imageOrVideoFile>: ${4}"
url=$1 ; saveDirectory=$2 ; outputFile=$3 ; imageOrVideoFile=$4
url=`curl $url -s -L -I -o /dev/null -w '%{url_effective}'` #Get the redirected URL
# Breakup URL parameter into domain/uri/filename:    '#' remove minimal matching prefixes,    '##' remove maximal matching prefixes,    '%' remove minimal matching suffixes,    '%%' remove maximal matching suffixes
domainLeft=${url#*//} ; domain=${domainLeft%%/*} 
uri=${url#*//*/}
uriFileName=${url##*/}


waitTime=15 #In seconds, give download enough time to get the video headers (eg MOOV atom in a 50MB MP4 could be 2MB)
subDirectory="${domain}"
# Uncomment if you want to save in a subdirectory for particular domain
saveDirectory="${saveDirectory}/${subDirectory}"
saveName="${uriFileName}"
# Uncomment if you want domain added to filename
#saveName="${domain} - ${uriFileName}"
save="${saveDirectory}/${saveName}"


#Check at least three parameters were passed
if [ $# -lt 3 ]
  then
    echo "Required arguments not supplied: <url> <saveDirectory> <outputFile> <optional:imageOrVideoFile>"
  else
  
  
    #Download (Non Blocking)
    #Letting Curl set the filename method - not used: cd "${saveDirectory}" && nohup curl --remote-time --remote-name "${url}" > /dev/null 2>&1 &
    #Grab URL using curl (with redirection), let us set the filename, including Domain Name if desired
    if [ ! -f "${save}" ]; then
      echo "Not previously downloaded - downloading now..."
      mkdir -p ${saveDirectory} && nohup curl --remote-time --output "${save}" -L "${url}" > /dev/null 2>&1 &
    fi


    #Video Message
    echo "Inserting blank or message video output..."
    #Generate a blank or message video in realtime (-re) so we give the file a chance to download
    #Start STDOUT ---------
    (
      if [ -z "${imageOrVideoFile}" ]
        then
          #TODO Look into using vmsg.py from Jumpy plugin
          #Create Message Using A Blank Video
          ffmpeg -re -y -t ${waitTime} -s 640x480 -f rawvideo -pix_fmt rgb24 -r 25 -i /dev/zero -f mpegts -target ntsc-dvd -
        else
          #Create Message Using Image or Video
          ffmpeg -re -y -loop 1 -i "${imageOrVideoFile}" -t ${waitTime} -f mpegts -target ntsc-dvd - || ffmpeg -re -y -i "${imageOrVideoFile}" -t ${waitTime} -f mpegts -target ntsc-dvd -
      fi
    
      #Try appending the video we've started downloading
      ffmpeg -re -i "${save}" -f mpegts -target ntsc-dvd - 
    #Finish STDOUT ---------
    ) > "${outputFile}"
    
fi



Last edited by AudioMan on Sun Mar 15, 2015 3:12 pm, edited 1 time in total.
infidel
Developer
Posts: 571
Joined: Thu Jul 12, 2012 5:37 am

Re: Web Streams (with WAIT for download)

Post by infidel »

AudioMan wrote:Here's the script:
Great ! I especially like that you've put put in the effort to get the non-blocking-full-download part right, so it's actually a nice watch-n-save solution. The main drawback (my fault not yours :)) is that downloadfirst becomes the default player for all things web, would be better if there was an option to make it secondary.

AudioMan wrote:Something weird is happening if you use any single quotes near the edges of the double quotes - eg around ${outfile}:
Jumpy has a no-quotes-required workaround for quote-escaping pitfalls, array notation syntax, e.g.

Code: Select all

[+downloadfirst]
cmd = pms addPlayer downloadfirst "[/path/to/downloadfirst.sh , ${filename} , /path/to/outdir , ${outfile}]" "f:http m:video/mpeg"
alternatively you can avoid passing $outfile on the command line entirely since there's already an $OUTFILE environment variable defined for you.

AudioMan wrote:#TODO Look into using vmsg.py from Jumpy plugin
Based on your nice work I added the two missing features, silent audio (-s) and realtime output (-n) to vmsg.py:
vmsg-1.1.zip
unzip to plugins/jumpy
(2.02 KiB) Downloaded 504 times
so this actually works now:

Code: Select all

#Grab URL using curl, let us set the filename, including Domain Name if desired
if [ ! -e "${save}" ]; then
	$pms log "Not previously downloaded - downloading now..."
	mkdir -p ${saveDirectory} && nohup curl --remote-time --output "${save}" "${url}" > /dev/null 2>&1 &
	downloading=yes
fi

#Video
(
	if [ "$downloading" = "yes" ] ; then
		$pms log "Inserting video message..."
		python "$($pms getHome)/lib/vmsg.py" -m "Starting\n\n${saveName}\n\n" -t ${waitTime} -b blue -nsx
	fi	
	#Try appending the video we've started downloading
	ffmpeg -re -i "${save}" -f mpegts -target ntsc-dvd -
) > "${outputFile}"
I also tried to see if I could trip things up with giant data-intensive download+transcoding by using bigger test files (added in jumpy-scripts.ini):

Code: Select all

[+]
cmd = pms addVideo /testing/bigbuckbunny.mp4 http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4
[+]
cmd = pms addVideo /testing/bigbuckbunny.mov http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov
[+]
cmd = pms addVideo /testing/bigbuckbunny.ogg http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_stereo.ogg
and succeeded a few times, though overall I was surprised at how robust it seemed :).
AudioMan
Posts: 7
Joined: Fri Feb 20, 2015 4:11 pm

Re: Web Streams (with WAIT for download)

Post by AudioMan »

Thanks for the feedback!
I made a change in the above post to the script so that it follows redirects correctly which feedproxy.google.com feeds were having problems with.
Post Reply