SurLaTablo.py - Python program to query and convert Tablo recordings

Tablo Apps Third Party Apps (Rippers etc.)
545

Starting to use SurLaTablo.py 2b1, and thank you very much for providing it @cjcox .

It’s having difficulty iterating through some of my recordings of meta_type Sports. I’m seeing:

Traceback (most recent call last): File "./surlatablo.py", line 2827, in <module> for team in teams: TypeError: 'NoneType' object is not iterable

I dug in a little bit and the offenders are from this past summer’s Olympics. The data structure for the teams list is set to “None”, which is breaking the iterator. Shown another way, if in the script I print the list:

Print (new_data_meta['event']['teams'])

It prints:

None

Can I send you some information to help reproduce or diagnose this further?

546

Are you using FreeNAS? In windows it works for me but not FreeNAS and there are others that posted in the Plex forums having this issue with Plex and FreeNAS.

Maybe I’m doing something wrong, I have PLEX running in a jail under FreeNAS if I copy anything to the folder that PLEX is monitoring it will not show unless I force an update. I just tried it now and nothing.

547

The auto scan function seemed to stop working on my Plex server a while back, then to mysteriously start working reliably again. I had several software components in my environment going through change at the time, so it could have very well been related to any number of them and not be a Plex issue. I guess I asked to try and gauge from other users if there was some robustness issues around the auto scanning functionality.

548

My box is running FreeBSD; serves up file share plus plex. I’ve been manually importing lately, but I’ll let it try to auto-scan and let you know.

549

FWIW, my Plex Server is running on a Mac mini, but the storage is on a Synology NAS (using AFP). I have Plex set to scan on timed intervals, and it works fine.

550

Consider adding the following lines to your surlatablo2.conf

TRANSCODER_DEFAULT = [ 'Mp4', 'PlexRefresh' ]
TRANSCODER_FUNCS += ['doPlexRefresh']
def doPlexRefresh(meta_type,inputfile):
    # Note: meta_type, TV, Movie, Sports, Manual is passed in.
    # You could add code here to select the Library number to make more
    # dynamic if you have things broken out into multiple libraries.
    #
    # inputfile is always passed, can be ignored
    print("Updating Plex Library 4")
    os.system("curl http://XXXXXXXX:32400/library/sections/4/refresh")

TRANSCODER_OPTS['PlexRefresh'] = {
                    'help': [ 'Refresh Plex Library' ],
                    'command': [ 'doPlexRefresh' ],
                    'options': [ '${meta_type}' ],
                    'ext': [ '.always' ]
}
551

Until I fix this.

find the code:

            try:
                teams = new_data_meta['event']['teams']
            except:
                teams = []
            for team in teams:
                teamnames.append(team['name'].encode('utf-8').strip())
            try:
                rec_val['teams_'] = ';'.join(teamnames).encode('utf-8').strip()
            except:
                rec_val['teams_'] = ''
            rec_val['home_team'] = ''
            for team in teams:
                try:
                    if (team['team_id'] == new_data_meta['event']['home_team_id']):
                        rec_val['home_team'] = team['name'].encode('utf-8').strip()
                except:
                    pass

Change it to:

            try:
                teams = new_data_meta['event']['teams']
            except:
                teams = []
            try:
                for team in teams:
                    teamnames.append(team['name'].encode('utf-8').strip())
                rec_val['teams_'] = ';'.join(teamnames).encode('utf-8').strip()
            except:
                rec_val['teams_'] = ''
            rec_val['home_team'] = ''
            try:
                for team in teams:
                    if (team['team_id'] == new_data_meta['event']['home_team_id']):
                        rec_val['home_team'] = team['name'].encode('utf-8').strip()
            except: 
                pass

I didn’t test this. I’d blow away the cache db files and redo after making this change.

552

I’ll give that a try, Thanks

553

The new exception handling worked great and fixes my symptom. Thanks @cjcox, appreciate it!

555

So, 2.0b2 will probably get pushed out tonight… in addition to some “fixes” it contains DELETE support. If you know anything about SurLaTablo and it’s ability to search the Tablo, then you now know how easy it will be to do, for example, a full season delete (and many other kinds of deletes of course).

It’s implemented as a “transcoder” thus it can be done automatically (add to the default transode list). But I don’t recommend that since there is a chance of a deletion happening and you wish you could re-transcode (though there are ways around that case since you can just ask SurLaTablo to keep the .ts).

Anyway, have more ideas for the next release already in my brain…

Look for the release tonight.

556

:+1:

Awesome!

557

SurLaTablo 2.0b2

2.0b2 - Fixes and Delete support

However there are two new transcoders, Delete (which prompts)
and DeleteX. If you haven’t guessed, this will delete the matched items. While you could tack
it onto your TRANSCODER_DEFAULT array or add in addition to whatever you have define by +DeleteX
on the command line, my preference would be to pull and then delete (just saying).

-i <rec_id> now works ok… Tablo separated the data, so it tries to do “right” things
when searching for the record. You can search using a full path or a number. I try to dump
all that is interesting out of the Tablo, however, it’s not just one big item anymore, it can
be 2 or 3 items now per “rec_id”.

558

So far the new 2.0b2 build is working good for me.

Thanks!

559

Hmmm. Maybe I’m crazy, but I think 2.0b2 processed the show, then deleted it when it finished.

560

Only if you add the DeleteX transcoder (or Delete if you want interactive prompting).

561

No - it’s me. I updated ffmpeg, but not fully. So THAT is the source of the problem. Sorry to add confusion.

562

2.0.b3 makes commercial zapping into a pure transcoder… that is, Mp4zap1 for algorithm 1 and Mp4zap2 for algorithm 2.

By default SurLaTablo just has the transcoder Mp4… but this would allow, for example, for you to add +Mp4zap1 to get the original fast mp4 and a version (hopefully) without commercials -z.mp4. SurLaTablo does not require extra software for commercial cutting.

2.0b3 probably coming out next week.

563

Example of extracting MacGyver s01e01 creating both a .mp4 with commercials and a -z.mp4 without commercials.

$ surlatablo.py -n -q 'MacGyver - s01e01' -c -C -k @ Gif Mp4 Mp4zap1 Json

(863062) MacGyver - s01e01 - The Rising

    Friday, September 23, 2016 at 07:00 PM
    KTVT(CBS) - 1080i [1:05:11]

    Description:
    Special agents for the Department of External Services, Angus "Mac" MacGyver and
    Jack Dalton, work to recover a missing bioweapon.

    Lucas Till;George Eads;Sandrine Holt;Justin Hires;Tristin Mays

Tablo (.ts) location:                   [./TV/MacGyver/Season 1/MacGyver - s01e01 - The Rising.ts]
Working on:                             [./TV/MacGyver/Season 1/MacGyver - s01e01 - The Rising]
 Retrieving Tablo Data (863062):        [####################] 100% Elapsed seconds 228.0
Srt file location:                      [./TV/MacGyver/Season 1/MacGyver - s01e01 - The Rising.eng.srt]
 Extracting CC as Subtitle:             [####################] 100% Elapsed seconds 13.0
 Transcoding (Gif, .gif):               [####################] 100% Elapsed seconds 2.0
 Transcoding (Mp4, .mp4):               [####################] 100% Elapsed seconds 9.0
 Searching for commercials:             [####################] 100% Elapsed seconds 214.0
 Removing commercials (z1):             [####################] 100% Elapsed seconds 24.0
Preserved (.ts) with commercials:       [./TV/MacGyver/Season 1/MacGyver - s01e01 - The Rising-orig.ts]
 Transcoding (Mp4zap1, -z.mp4):         [####################] 100% Elapsed seconds 405.0
Executing (dumpJson, .json)

Took about 15 minutes on a very very old CPU. Files created as a result of the above.

$ ls -lh TV/MacGyver/Season\ 1/
total 6.6G
-rw-r--r-- 1 ccox users 100K Nov 25 15:10 MacGyver - s01e01 - The Rising.eng.srt
-rw-r--r-- 1 ccox users 1.5M Nov 25 15:10 MacGyver - s01e01 - The Rising.gif
-rw-r--r-- 1 ccox users 2.1K Nov 25 15:21 MacGyver - s01e01 - The Rising.json
-rw-r--r-- 1 ccox users 1.9G Nov 25 15:10 MacGyver - s01e01 - The Rising.mp4
-rw-r--r-- 1 ccox users 2.0G Nov 25 15:10 MacGyver - s01e01 - The Rising-orig.ts
-rw-r--r-- 1 ccox users 1.5G Nov 25 15:14 MacGyver - s01e01 - The Rising.ts
-rw-r--r-- 1 ccox users 1.3G Nov 25 15:21 MacGyver - s01e01 - The Rising-z.mp4

The .gif file is a short animated GIF preview. The .json file is a dump of JSON from SurLaTablo describing the program. The -k @ option told SurLaTablo to preserve the .ts files.

The ffprobe (or you can use mediainfo) metadata of the resulting mp4 file.

"format": {
    "filename": "TV/MacGyver/Season 1/MacGyver - s01e01 - The Rising.mp4",
    "nb_streams": 3,
    "nb_programs": 0,
    "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
    "format_long_name": "QuickTime / MOV",
    "start_time": "0.000000",
    "duration": "3879.314000",
    "size": "2001526750",
    "bit_rate": "4127589",
    "probe_score": 100,
    "tags": {
        "major_brand": "isom",
        "minor_version": "512",
        "compatible_brands": "isomiso2avc1mp41",
        "title": "The Rising",
        "album": "MacGyver",
        "encoder": "Lavf57.25.100",
        "comment": "Special agents for the Department of External Services, Angus \"Mac\" MacGyver and Jack Dalton, work to recover a missing bioweapon.",
        "genre": "N/A",
        "description": "Special agents for the Department of External Services, Angus \"Mac\" MacGyver and Jack Dalton, work to recover a missing bioweapon.",
        "synopsis": "Special agents for the Department of External Services, Angus \"Mac\" MacGyver and Jack Dalton, work to recover a missing bioweapon.",
        "show": "MacGyver",
        "episode_id": "s01e01",
        "network": "KTVT",
        "episode_sort": "1",
        "season_number": "1",
        "media_type": "10",
        "hd_video": "1"
    }
}
   "streams": [
    {
        "index": 0,
        "codec_name": "h264",
        "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
        "profile": "High",
        "codec_type": "video",
        "codec_time_base": "1001/60000",
        "codec_tag_string": "avc1",
        "codec_tag": "0x31637661",
        "width": 1280,
        "height": 720,
        "coded_width": 1280,
        "coded_height": 720,
        "has_b_frames": 1,
        "sample_aspect_ratio": "1:1",
        "display_aspect_ratio": "16:9",
        "pix_fmt": "yuv420p",
        "level": 41,
        "chroma_location": "left",
        "refs": 2,
        "is_avc": "true",
        "nal_length_size": "4",
        "r_frame_rate": "30000/1001",
        "avg_frame_rate": "1787339657/59693558",
        "time_base": "1/90000",
        "start_pts": 2160,
        "start_time": "0.024000",
        "duration_ts": 348969621,
        "duration": "3877.440233",
        "bit_rate": "3866530",
        "bits_per_raw_sample": "8",
        "nb_frames": "116098",
        "disposition": {
            "default": 1,
            "dub": 0,
            "original": 0,
            "comment": 0,
            "lyrics": 0,
            "karaoke": 0,
            "forced": 0,
            "hearing_impaired": 0,
            "visual_impaired": 0,
            "clean_effects": 0,
            "attached_pic": 0
        },
        "tags": {
            "language": "und",
            "handler_name": "VideoHandler"
        }
    },
    {
        "index": 1,
        "codec_name": "aac",
        "codec_long_name": "AAC (Advanced Audio Coding)",
        "profile": "LC",
        "codec_type": "audio",
        "codec_time_base": "1/48000",
        "codec_tag_string": "mp4a",
        "codec_tag": "0x6134706d",
        "sample_fmt": "fltp",
        "sample_rate": "48000",
        "channels": 2,
        "channel_layout": "stereo",
        "bits_per_sample": 0,
        "r_frame_rate": "0/0",
        "avg_frame_rate": "0/0",
        "time_base": "1/48000",
        "start_pts": 0,
        "start_time": "0.000000",
        "duration_ts": 186096640,
        "duration": "3877.013333",
        "bit_rate": "254762",
        "max_bit_rate": "254762",
        "nb_frames": "181735",
        "disposition": {
            "default": 1,
            "dub": 0,
            "original": 0,
            "comment": 0,
            "lyrics": 0,
            "karaoke": 0,
            "forced": 0,
            "hearing_impaired": 0,
            "visual_impaired": 0,
            "clean_effects": 0,
            "attached_pic": 0
        },
        "tags": {
            "language": "eng",
            "handler_name": "SoundHandler"
        }
    },
    {
        "index": 2,
        "codec_name": "mov_text",
        "codec_long_name": "3GPP Timed Text subtitle",
        "codec_type": "subtitle",
        "codec_time_base": "1/1000",
        "codec_tag_string": "tx3g",
        "codec_tag": "0x67337874",
        "r_frame_rate": "0/0",
        "avg_frame_rate": "0/0",
        "time_base": "1/1000",
        "start_pts": 0,
        "start_time": "0.000000",
        "duration_ts": 3879314,
        "duration": "3879.314000",
        "bit_rate": "85",
        "nb_frames": "2261",
        "disposition": {
            "default": 1,
            "dub": 0,
            "original": 0,
            "comment": 0,
            "lyrics": 0,
            "karaoke": 0,
            "forced": 0,
            "hearing_impaired": 0,
            "visual_impaired": 0,
            "clean_effects": 0,
            "attached_pic": 0
        },
        "tags": {
            "language": "eng",
            "handler_name": "SubtitleHandler"
        }
    }
]
}

For players that support it the close captions are embedded in the mp4. However, the .srt is also produced for players that need that.

564

Just seeing this after the holiday but Awesome! I’ll give it a try in the next day or so.

565

I’ll try to get 2.0b3 out tonight.