SurLaTablo.py - Python program to query and convert Tablo recordings

OS is FreeBSD. Here is the output.

plex@titan:/usr/local/surlatablo # ./surlatablo.py -n -q rec_id~=794024 -Q '${friendly_title}' Ed Sheeran ve at Wembley Stadium - s08e16 - Episode 2015-08-16

Here is what it looks like in Tablo’s GUI.

The problem is you have a non-utf8 compliant filesystem… which I guess I sort of should have guessed was a possibility even with a “good” OS.

Let me see if I can whip up a quick patch to at least let you set this on the command line somehow and/or config option.

SurLaTablo 1.6 Released (quick release)

###1.6 -
1.6 - (quick release)

  1. Fix for Mp4W transcode when width is “wrong”… there may be more problems than this.
  2. Added options[‘no_utf8_names’] for when an OS might handle it but the filesytem chosen cannot.

On #2, set options[‘no_utf8_names’] = ‘whatever’ in your surlatablo.conf or set the value on the command line with -o no_utf8_names=whatever

(whatever, because it simply needs to be something not null)

cjcox

Update - everything is working well. I have all my table recordings downloaded, converted to h.264 (not enough horsepower for h.265, yet), cron’d to execute when I won’t need the CPU resources, automatically imported into PLEX. Nothing could be better, except …

Any luck with the automatic delete?

:smile:

allen

Can’t get into all of it here… but there are some fundamental reasons why automatic delete will probably never be in surlatablo. Sorry.

Thanks for responding. If it’s a matter of time/availability, if I had some technology nudges, I’d consider exploring it myself. If that’s not the reason, that I certainly appreciate all that you’ve been. Either way, it’s already a terrific system :smile:

thx - allen

Running SurLaTablo.py on OSX 10.11.4 (beta).

Maybe I don’t understand how to use the Mp4z transcoder. No matter what I do, the -z.mp4 file is always zero bytes. My first thought is that my copy of ffmpeg doesn’t have the ultrafast preset and is failing. How can I tell?

joe

$ surlatablo.py --convert --zapcommercials --query "Big" Mp4z
Working on:                             [./TV Shows/The Big Bang Theory/Season 9/The Big Bang Theory - s09e14 - The Meemaw Materialization]
 Retrieving Tablo Data (11198):         [####################] 100% Elapsed seconds 141.0
 Searching for commercials:             [####################] 100% Elapsed seconds 89.0
 Removing commercials (z1):             [####################] 100% Elapsed seconds 4.0
Elapsed seconds 0.0

$ surlatablo.py --convert --query "Big" Mp4z
Working on:                             [./TV Shows/The Big Bang Theory/Season 9/The Big Bang Theory - s09e14 - The Meemaw Materialization]
 Retrieving Tablo Data (11198):         [####################] 100% Elapsed seconds 139.0
Elapsed seconds 0.0

$ surlatablo.py --convert --query "Big" +Mp4z
Working on:                             [./TV Shows/The Big Bang Theory/Season 9/The Big Bang Theory - s09e14 - The Meemaw Materialization]
 Retrieving Tablo Data (11198):         [####################] 100% Elapsed seconds 139.0
 Transcoding (Mp4, .mp4):               [####################] 100% Elapsed seconds 424.0
Elapsed seconds 0.0.

$ surlatablo.py --convert --zapcommercials --query "Big" +Mp4z
Working on:                             [./TV Shows/The Big Bang Theory/Season 9/The Big Bang Theory - s09e14 - The Meemaw Materialization]
 Retrieving Tablo Data (11198):         [####################] 100% Elapsed seconds 138.0
 Searching for commercials:             [####################] 100% Elapsed seconds 70.0
 Removing commercials (z1):             [####################] 100% Elapsed seconds 4.0
 Transcoding (Mp4, .mp4):               [####################] 100% Elapsed seconds 238.0
Elapsed seconds 0.0

Seems like the ffmpeg installation on my system is a little weird. I re-ran one of the tests above and captured the output (instead of the terminal progress bar). Got this output:

Mp4z
['/opt/local/bin/ffmpeg', '-threads', '0', '-i', '/var/folders/j5/hp16sj9j077g4c8cszk56pv80000gq/T/Tablo9Uoysf.ts', '-preset', 'ultrafast', '-y', '-metadata:s:a:0', 'language=eng', '-metadata', 'title=The Meemaw Materialization', '-metadata', 'date=2016-02-04', '-metadata', 'media_type=10', '-metadata', 'album=The Big Bang Theory', '-metadata', 'show=The Big Bang Theory', '-metadata', 'season_number=9', '-metadata', 'episode_sort=14', '-metadata', 'episode_id=s09e14', '-metadata', 'hd_video=1', '-metadata', 'genre=', '-metadata', "synopsis=Sheldon's excitement about his Meemaw's visit fades when she butts heads with Amy; Raj questions his relationship with Emily after meeting someone new.", '-metadata', "description=Sheldon's excitement about his Meemaw's visit fades when she butts heads with Amy; Raj questions his relationship with Emily after meeting someone new.", '-metadata', "comment=Sheldon's excitement about his Meemaw's visit fades when she butts heads with Amy; Raj questions his relationship with Emily after meeting someone new.", u'./TV Shows/The Big Bang Theory/Season 9/The Big Bang Theory - s09e14 - The Meemaw Materialization-z.mp4']
ffmpeg version 2.8.5 Copyright (c) 2000-2016 the FFmpeg developers
built with Apple LLVM version 7.0.2 (clang-700.1.81)
configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-indev=jack --disable-outdev=xv --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --enable-vda --enable-videotoolbox --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100
Input #0, mpegts, from '/var/folders/j5/hp16sj9j077g4c8cszk56pv80000gq/T/Tablo9Uoysf.ts':
Duration: 00:04:57.65, start: 1.405333, bitrate: 21442 kb/s
Program 1
Metadata:
service_name    : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 258 kb/s
[libx264 @ 0x7ff28b000c00] using SAR=1/1
[libx264 @ 0x7ff28b000c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x7ff28b000c00] profile Constrained Baseline, level 3.1
[libx264 @ 0x7ff28b000c00] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
[aac @ 0x7ff28b001e00] The encoder 'aac' is experimental but experimental codecs are not enabled, add '-strict -2' if you want to use it.

Seems like ultrafast doesn’t work as installed. So I added the ‘-strict -2’ arguments to the definition of the Mp4z transcode and it worked.

Glad you got it working. I tried to make things pretty easy to change, even through the conf file.

I had a question on this script - which I love, by the way! I’m using it with my little raspberry pi, and it’s just the best thing. It’s transcoding my daily game show fix into my NAS swimmingly for such a teensy little device.

But that’s the question then - I’m trying to set this up in a cron job to grab episodes daily, but I’ve only figured out how to use one query command at once? In my case, I’d like to (for example) pull today’s The Price is Right, which I’m doing right now with --query The\ Price\ Is\ Right, but then I’m not sure I’m able to also do a lair_date match as well.

If it’s impossible, I can figure out something else, but I figured it was worth an ask. And seriously, thanks for putting this together and out there, it’s a great resource.

So the query uses regular expression and it’s done across the whole meta data (like it were concatecated all together). I’ve inserted a probabalistic way of limiting search to specific metadata, so that you can do:

--query title~="Price is Right"

for example. By default, surlatablo will not clobber existing files, so runs that match multiple items won’t overwrite things that are already there.

Anyhow, right now you can’t do any combination query unless you consider that concatentation things I mentioned earlier. Personally, I would just rely on the noclobber thing. But you could try:

--query '"lair_date": "2016-03-12.*Price Is Right'

or more specifically that would be:

--query '"lair_date": "2016-03-12.*"series": "Price Is Right"'

But again, if all things are always going to same place… I’d just let it run and grab what isn’t there already.

Unfortuantely, I didn’t force order on the metadata, but you can look at your rec_ids_db.json file in the SURLATABLO_ROOT directory and see how the metadata is arraged for each show. From that you might be able to craft some regular expression searches like I did above that can match on multiple things.

I will totally experiment with your suggestions!

Noclobber is totally my fallback, and I’m only looking for a different solution because I’m great about deleting stuff off my NAS after I watch it, but deleting it off the Tablo will involve like, an extra thing that I’m basically guaranteed to forget. If I cron the transcode to run after the daily episode airs it’ll just put it back.

You can only keep so many 2.5GB files starring Drew Carey, you know? =p

But seriously, thanks again. Great tool!

Thanks for the thanks! I had fun writing it. And there might still be something else… yet to come…

Hi - I had an additional question about truncate_ts, if you don’t mind?

What does the value represent? You state in the changelog that “a value of options[‘truncate_ts’] = 75 should clip off about 5 min. off the original ts from the Tablo, and commercial processing will come after that” but my testing with that had the converted file (I left zapping off for testing) come up 13 minutes short.

I noticed in one of your examples that use use 25 for your truncate_ts value (and that would seem to come closer to removing 5 minutes if 75 = 13) but I’m just super curious what this value means. Edit: simple trial and error testing leads me to believe that a value of 40 is just perfect.

Potential bonus question - if I wanted to define two values from the command line with -o, is that possible, or does only one value get passed?

PS (no pun intended) - the postscript=true value helps so much to catch the winner of the showcase showdown.

Thanks again!

I don’t remember how I came up with the truncate_ts value, probably just via experimentation on my own setup. If 40 works right for you, then use it. I guess it’s really more important that the value is there for you to adjust.

And yes, you can do multiple options for -o, they would be done like so -o truncate_ts=75,someotheroption=whatever

Glad postscript works for you, it’s only valid on the default zap algorithm.

So, about postscript - I actually think it may have been broken.

I inserted some debugging checks, and found the statement was actually excepting with an out of bounds array error, which was silenced by the except: pass.

I spent the past four days learning Python, and humbly submit postscript 1.1:

                    if (options['postscript']):
                            seg_clips.pop(-1)
                            seg_clips += [ (start[lastsegend - 2], start[lastsegend]) ]

This removes the last (start, stop) value from seg_clips and replaces it with the start of the previous black frame from the list, and keeps the end time. I’m also considering a 1.2 like:

                    if (options['postscript']):
                            seg_clips.pop(-1)
                            seg_clips += [ (start[lastsegend - int(options['postscript']) + 1], start[lastsegend]) ]

so that postscript can have variable power, but I need to think that through more and actually see if that compiles or I stuffed the syntax.

But seriously, I’ve dived into your program like, danger deep, and learned so much doing it. Thanks again for your hard work.

Thanks again! Yeah… that z1 algorithm was just a random brain explosion turned code. I’m amazed when it does a good job and sometimes it guesses really well… which is why it’s the preferred algorithm, but as you just stated, it could have quite a few bugs. I’ll see about incorporating your ideas into the next version. A little sanity in the midst of the chaos that is z1!

SurLaTablo.py is reporting these recordings in progress:

TV Series/Program 27145
RECORDING/BUSY Marvel’s Agents of S.H.I.E.L.D. - s03e12 - The Inside Man
TV Series/Program 40350
RECORDING/BUSY NCIS: New Orleans - s02e18 - If It Bleeds, It Leads

But I know they’re not, they’re bad recordings from weeks ago. Is there any way to get SurLaTablo to delete these? Or some other app?

Thanks!

Use the -B option on a query that will find these. That will clear the status in SurLaTablo.

oh… and that just clears the meta data on SurLaTablo side. If you really want this off your Tablo, you can delete them from the Tablo. One of the closed source pullers has the capability as well (can’t remember which).

Thanks, that seems to have cleared them out.