SurLaTablo.py - Python program to query and convert Tablo recordings

Tablo Apps Third Party Apps (Rippers etc.)
627

Hi, CJCOX

I’m running 2.0b1. I think that’s the latest in this thread. Is there a newer version?

BTW, these problems just recently showed up. I run the scripts on multiple machines, and they both exhibited the problem simultaneously. Did tablo get an update?

Also, BTW, I have tweaked the compression commands to suit my plex needs. Thought you might be interested. I compress to h.265, embedding the .srt into a single “mkv” file. Figured out how to get surlatablo to delete the .srt when it’s done. Changed the frame rate to 24 fps, bit rate to 650K. I couldn’t do this if your code isn’t played out in a manner that I can understand. Kudos :slight_smile:

(I don’t, however, trust that the commercial zapper is 100% accurate, so I chose not to do that)

thx - allen

628

2.0 is the latest. 2.0b1 was the first of a string of betas leading up to 2.0. Sorry about the confusion.

Ideally, you should be able to produce a custom transcorder that you can place in your surlatablo.conf to do things differently. I tried to design it so that you didn’t have to edit the program itself.

Commercial zapping is based on black frame detection (thus it is error prone). In a future release, I will a feature that gives you chapter marks so that you can “skip” (if the client supports that). That makes things a bit better and not so much an “all or nothing” deal.

629

My weird way to use Surlatablo.py is to identify the shows (keep it’s local database up-to-date) and use a bash script and ffmpeg to pull the episodes I want. Like Allen, I do this to make a few changes as I save the file.

For removing commercials, I find the break-points manually and have another script that splices the show to remove commercials, adds metadata, and shoves it into the appropriate folder for Plex to pick-up.

I should probably dig through Surlatablo a bit more because I’m pretty sure I can skip the first bash script.

630

I see that TABLO has a pending update. Knowing that in the past, updates cause surlatablo to break. Has anyone tried this update with surlatablo?

thx - allen

631

I have been running the …14 update for a while now. Surlatablo works fine for me. No promise you will have the same experience.

632

It’s been running fine for me on my two Tablos since they were upgraded. I think the last time things broke only because Tablo made a huge change to their APIs.

633

Thanks for the feedback. Agreed - my experience could be different. But I have the comfort of knowing that at least ONE person is using it successfully with surlatablo.

thx - allen

634

Surprise, the author use it too!!
:slight_smile:

635

Do they have staggered release of the update? It sounds like y’all upgraded some time ago. My notice occurred just today.

allen

636

Not sure if they use a repeating pattern. Seems sometimes I will get an update quickly and sometimes I wait for many days. Patience is the key.

637

I guess somebody has to be last.

638

It’s always 100% random.

639

3 Likes
640

cjcox, thanks for putting this out there. I’ve been using it to backup my files and it works fine but have not had luck removing the commercials. If I use the -z option the audio is out of sync. I’ve tried using the Mp4z and Mp4zap1 but when I use these the file I get is empty? Not sure if I’m using them right. Can you give me an example of how I should specify the transcode options. Last one I tried was like this: python surlatablo.py --noupdate -i 140077 --convert -z Mp4zap1
Thanks again for all your work.

641

The -z is redundant when using a zapping transcoder option like Mp4zap1.

So ideally:

surlatablo.py -n -i 14077 -c Mp4zap1

Should have resulted in a -z.mp4 file being produced using zap algorithm 1. You can redirect the output of the command to a file if you like, that gives me (usually) everything I need to debug. It might show you where things went wrong and generate an “ah ha” moment. In other words, it should have worked. So try redirecting the output to a file and see if that provides any hints as to what went wrong. Feel free to private message me with the file if it’s not something obvious.

$ surlatablo.py -n -i 1112453 -c Mp4zap1
"/recordings/series/630324": {
    "object_id": 630324, 
    "path": "/recordings/series/630324", 
    "series": {
        "awards": null, 
        "background_image": {
            "has_title": false, 
            "image_id": 1092881
        }, 
        "cast": [
            "Patricia Heaton", 
            "Neil Flynn", 
            "Charlie McDermott", 
            "Eden Sher", 
            "Atticus Shaffer"
        ], 
        "cover_image": {
            "has_title": true, 
            "image_id": 1092880
        }, 
        "description": "In the Heck family, middle-age, middle-class, middle-America mom Frankie Heck (two-time Emmy winner Patricia Heaton) uses a sense of humor to try to steer her family through life's ups and downs as she tackles her career goals. Her unflappable husband, Mike (Neil Flynn), is a manager at the local quarry. Oldest son Axl is an obstinate young man; awkward daughter Sue cannot seem to find her niche \u2014 despite much enthusiasm in her attempts \u2014 and youngest son Brick is an unusual child whose best friend is his backpack.", 
        "episode_runtime": 1800, 
        "orig_air_date": "2009-09-30", 
        "thumbnail_image": {
            "has_title": true, 
            "image_id": 1092879
        }, 
        "title": "The Middle", 
        "tms_id": "SH011581190000", 
        "tms_series_id": "3560344"
    }, 
    "show_counts": {
        "airing_count": 1, 
        "failed_count": 0, 
        "protected_count": 0, 
        "unwatched_count": 1, 
        "watched_and_protected_count": 0
    }, 
    "user_info": {
        "up_next": "/recordings/series/episodes/1112453"
    }
},
"/recordings/series/episodes/1112453": {
    "airing_details": {
        "channel": {
            "channel": {
                "call_sign": "WFAA", 
                "call_sign_src": "WFAA", 
                "major": 8, 
                "minor": 1, 
                "network": "ABC", 
                "resolution": "hd_1080"
            }, 
            "object_id": 922280, 
            "path": "/recordings/channels/922280"
        }, 
        "channel_path": "/recordings/channels/922280", 
        "datetime": "2017-05-17T00:00Z", 
        "duration": 1800
    }, 
    "episode": {
        "description": "Frankie and Mike have different reactions when Axl tells them he plans to spend the summer in Europe; Sue panics when she doesn't hear back from the owner of an SUV she scrapes; Brick uncovers the source of his backaches.", 
        "number": 23, 
        "orig_air_date": "2017-05-16", 
        "season_number": 8, 
        "title": "Fight or Flight", 
        "tms_id": "EP011581190196"
    }, 
    "object_id": 1112453, 
    "path": "/recordings/series/episodes/1112453", 
    "qualifiers": [
        "cc"
    ], 
    "season_path": "/recordings/series/seasons/869202", 
    "series_path": "/recordings/series/630324", 
    "snapshot_image": {
        "has_title": false, 
        "image_id": 1126420
    }, 
    "user_info": {
        "position": 0, 
        "protected": false, 
        "watched": false
    }, 
    "video_details": {
        "clean": true, 
        "duration": 2721, 
        "height": 720, 
        "schedule_offsets": {
            "end": 906, 
            "start": -15
        }, 
        "size": 1559912448, 
        "state": "finished", 
        "width": 1280
    }
},
"/recordings/series/seasons/869202": {
    "object_id": 869202, 
    "path": "/recordings/series/seasons/869202", 
    "season": {
        "name": "8", 
        "number": 8
    }, 
    "season_counts": {
        "airing_count": 1, 
        "failed_count": 0, 
        "protected_count": 0, 
        "unwatched_count": 1, 
        "watched_and_protected_count": 0
    }, 
    "series_path": "/recordings/series/630324"
},
## (1112453) The Middle - s08e23 - Fight or Flight ##
        Tuesday, May 16, 2017 at 07:00 PM
        WFAA(ABC) - 1080i [0:45:21]

        Description:
        Frankie and Mike have different reactions when Axl tells them he plans to spend
        the summer in Europe; Sue panics when she doesn't hear back from the owner of
        an SUV she scrapes; Brick uncovers the source of his backaches.

        Patricia Heaton;Neil Flynn;Charlie McDermott;Eden Sher;Atticus Shaffer

Working on:                             [./TV/The Middle/Season 8/The Middle - s08e23 - Fight or Flight]
 Retrieving Tablo Data (1112453):       [####################] 100% Elapsed seconds 176.0
 Searching for commercials:             [####################] 100% Elapsed seconds 148.0
 Removing commercials (z1):             [####################] 100% Elapsed seconds 18.0
 Transcoding (Mp4zap1, -z.mp4):         [####################] 100% Elapsed seconds 366.0

$ mediainfo TV/The\ Middle/Season\ 8/The\ Middle\ -\ s08e23\ -\ Fight\ or\ Flight-z.mp4 
General
Complete name                            : TV/The Middle/Season 8/The Middle - s08e23 - Fight or Flight-z.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom
File size                                : 1.14 GiB
Duration                                 : 37mn 19s
Overall bit rate                         : 4 373 Kbps
Season                                   : 8
Movie name                               : The Middle
Album                                    : The Middle
Part                                     : 23
ContentType                              : TV Show
Description                              : Frankie and Mike have different reactions when Axl tells them he plans to spend the summer in Europe; Sue panics when she doesn't hear back from the owner of an SUV she scrapes; Brick uncovers the source of his backaches.
Writing application                      : Lavf57.25.100
Comment                                  : Frankie and Mike have different reactions when Axl tells them he plans to spend the summer in Europe; Sue panics when she doesn't hear back from the owner of an SUV she scrapes; Brick uncovers the source of his backaches.
LongDescription                          : Frankie and Mike have different reactions when Axl tells them he plans to spend the summer in Europe; Sue panics when she doesn't hear back from the owner of an SUV she scrapes; Brick uncovers the source of his backaches.
Part_ID                                  : s08e23
TVNetworkName                            : WFAA
HDVideo                                  : Yes

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Baseline@L3.1
Format settings, CABAC                   : No
Format settings, ReFrames                : 1 frame
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 37mn 19s
Bit rate                                 : 4 237 Kbps
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 29.970 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.153
Stream size                              : 1.10 GiB (97%)
Writing library                          : x264 core 148
Encoding settings                        : cabac=0 / ref=1 / deblock=0:0:0 / analyse=0:0 / me=dia / subme=0 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=250 / keyint_min=25 / scenecut=0 / intra_refresh=0 / rc=crf / mbtree=0 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=0

Audio
ID                                       : 2
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : 40
Duration                                 : 37mn 19s
Bit rate mode                            : Constant
Bit rate                                 : 129 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Compression mode                         : Lossy
Stream size                              : 34.5 MiB (3%)
Language                                 : English
642

Btw, since SurLaTablo doesn’t ovewrite by default, you can safely run a command over and over for patterns, including “everything” and it will just do the items you may have missed (as long as the destination files are still there).

So:

$ surlatablo.py -n -q 'The Middle' -c Mp4zap1

in my example case might have better than querying and using the record number (-i option).

Or do everything on the Tablo:

$ surlatablo.py -n -q '.' -c Mp4zap1

This assumes the surlatablo local database is up to date. If you run the above often, you may find it safer to leave out the -n option to ensure the local cache database is always up to date.

And of course you can now also force a delete. I usually do this separately in case I need to rerun a pull from the Tablo (for example you ran out of local disk space). So if you believe all the pulls from the Tablo succeeded, without updating the cache I’d do:

$ surlatablo.py -n -q '.' -c DeleteX

to remove them all from the Tablo. Again, once you’re satisfied. You don’t want an updated cache here because it needs to match the cache database as it was when you did the pull (for example you could have had additional things in flight recording, etc.).

643

Last thing… zapping is very problematic. So much so that I don’t use it anymore (and because it takes a long time). However, back when I did use it, I captured both the default and zapped version.

$ surlatablo.py ...query options... -C -c +Mp4zap1

which is to say, do the default TRANCODER_DEFAULT list of transcoders plus the Mp4zap1 one.

My plan for version 3 of SurLaTablo is to do chapter breaks (as commercial detects). I might even pull the transcode zap option and just go with chapter markers simply because the rate of error is too high on zapping. Of course, this assumes you end client can do chapter skipping easily. This is a great solution for interactive watching because when the commercials start, you can skip forward chapter by chapter and even go back a chapter if you believe it went too far due to bad commercial detection.

644

Thanks for the info. I tried again this morning. It never finishes the final transcoding after the commercials are removed. I may see the problem. Here is the last line of the output:
‘[aac @ 0x1635340] The encoder ‘aac’ is experimental but experimental codecs are not enabled, add ‘-strict -2’ if you want to use it.’ Thanks again.

645

You can try adding something like this to your surlatablo2.conf file:

xcoders_list = [ 'Mp4zap1', 'Mp4zap2', 'Mp4Wzap1', 'Mp4Wzap2', 'Mp4W', 'Mp4z', 'Mp4', 'Shrink', 'Glow', 'h265' ]
for xcoder in xcoders_list:
  if TRANSCODER_OPTS[xcoder]['options'][0:2] != [ '-strict', '-2' ]:
    TRANSCODER_OPTS[xcoder]['options'][:0] = [ '-strict', '-2' ]

And let me know how it goes.

646

Thanks CJ, that did work, at least I got a file that plays. I’ll play it all the way through and see if it’s in sync. Thanks again.