SurLaTablo.py - Python program to query and convert Tablo recordings


#566

SurLaTablo 2.0b3

2.0b3 - Fixes. Addition of Mp4zap1 and Mp4zap2 “transcoders”.

Addition of a remove_format similar to query_format and a formatted_remove boolean to trigger it. This gives you the option of creating a better format for the db cache REMOVES than the normal full JSON dump.

This will likely become the official 2.0 release. So please test. Thank you.


#567

@cjcox: Had some time to check out your new release 2.0b3 this evening. It’s again, a fabulous script but I’m getting some errors early on when trying to create the initial cache files. It’s not a blocker as I can work around these issues but still wanted to let you know in case you want to properly catch these errors. I have a huge library and there currently are two offending episodes that your script chokes on when trying to process them:

Frequency - s01e03 - The Near Far Problem
Traceback (most recent call last):
  File "C:\TabloTV\surlatablo-2.0b3.py", line 2774, in <module>
    rec_val['description'] = new_data_meta['episode']['description'].encode('utf-8').strip()
AttributeError: 'NoneType' object has no attribute 'encode'

  Hogan's Heroes - s04e09 - Guess Who Came to Dinner?
Traceback (most recent call last):
  File "C:\TabloTV\surlatablo-2.0b3.py", line 2811, in <module>
    if (not rec_val['season_number']):
KeyError: 'season_number'

  Hogan's Heroes - s04e09 - Guess Who Came to Dinner?
Traceback (most recent call last):
  File "C:\TabloTV\surlatablo-2.0b3.py", line 2815, in <module>
    if (rec_val['season_number'] >= 0 and rec_val['episode_number'] >= 0):
KeyError: 'season_number'

  Hogan's Heroes - s04e09 - Guess Who Came to Dinner?
Traceback (most recent call last):
  File "C:\TabloTV\surlatablo-2.0b3.py", line 2955, in <module>
    print('  ' + rec_val['friendly_title'])
KeyError: 'friendly_title'`

I’ve also attached a screenshot of the offending shows’ meta data.


#568

Thanks, I’ll look into these and make fixes.


#569

BTW, I’m making an assumption that the traceback errors are from the show printed immediately before the error is shown. If that’s not correct, then I’ve attached the wrong meta data.


#570

Would you mind sending me the result of:

surlatablo.py -i 287642

I’m trying to get surlatablo’s view through the new API of the meta data (you’re giving me the WebSQL DB view I believe).


#571
"/recordings/series/episodes/287642": {
    "airing_details": {
        "channel": {
            "channel": {
                "call_sign": "METV-SD",
                "call_sign_src": "METV-SD",
                "major": 20,
                "minor": 2,
                "network": null,
                "resolution": "sd"
            },
            "object_id": 27142,
            "path": "/recordings/channels/27142"
        },
        "channel_path": "/recordings/channels/27142",
        "datetime": "2015-05-05T04:00Z",
        "duration": 1800
    },
    "episode": {
        "description": "Hogan is worried that an underground contact is a double agent.",
        "number": 9,
        "title": "Guess Who Came to Dinner?",
        "tms_id": "EP000021060101"
    },
    "object_id": 287642,
    "path": "/recordings/series/episodes/287642",
    "qualifiers": [],
    "season_path": "/recordings/series/seasons/27141",
    "series_path": "/recordings/series/27140",
    "snapshot_image": {
        "has_title": false,
        "image_id": 298342
    },
    "user_info": {
        "position": 0,
        "protected": false,
        "watched": false
    },
    "video_details": {
        "duration": 2120,
        "height": 480,
        "schedule_offsets": {
            "end": 305,
            "start": -15
        },
        "size": 531173376,
        "state": "finished",
        "width": 704
    }
},
"/recordings/series/seasons/27141": {
    "object_id": 27141,
    "path": "/recordings/series/seasons/27141",
    "season": {
        "name": "4",
        "number": 4
    },
    "season_counts": {
        "airing_count": 26,
        "failed_count": 0,
        "protected_count": 0,
        "unwatched_count": 23,
        "watched_and_protected_count": 0
    },
    "series_path": "/recordings/series/27140"
},
"/recordings/series/27140": {
    "object_id": 27140,
    "path": "/recordings/series/27140",
    "series": {
        "awards": null,
        "background_image": {
            "has_title": false,
            "image_id": 732686
        },
        "cast": [
            "Bob Crane",
            "Werner Klemperer",
            "John Banner",
            "Robert Clary",
            "Richard Dawson",
            "Ivan Dixon",
            "Larry Hovis",
            "Kenneth Washington",
            "Cynthia Lynn",
            "Sigrid Valdis"
        ],
        "cover_image": {
            "has_title": true,
            "image_id": 732685
        },
        "description": "Perhaps the funniest show ever made about fictional hilarity in a Nazi P.O.W. prison, this series focuses on Stalag 13, a camp set aside for U.S. resistance fighters and overseen by the strict but bumbling Col. Klink. Although the prisoners have an abnormal number of perks at the camp, they are constantly trying to mess with the Germans and help the Allied war effort from within.",
        "episode_runtime": 1800,
        "orig_air_date": "1965-09-17",
        "thumbnail_image": {
            "has_title": true,
            "image_id": 732684
        },
        "title": "Hogan's Heroes",
        "tms_id": "SH000021060000",
        "tms_series_id": "184204"
    },
    "show_counts": {
        "airing_count": 169,
        "failed_count": 0,
        "protected_count": 0,
        "unwatched_count": 136,
        "watched_and_protected_count": 0
    },
    "user_info": {
        "up_next": "/recordings/series/episodes/141557"
    }
},

#572

Well, the first one is certainly a “bug”. But I’m a bit confused on the season_number error. The meta data looks ok… and I think my code is ok… so not sure. Maybe that first bug is messing something up? (wouldn’t think so though).

I’ll put in two fixes… one that is more legit than the other. Because season_number should be there, in fact you see it in the friendly_title there. The last one is also puzzling.

Surlatablo should die at the first error and not try to continue. I’m thinking some of the errors are due to the program trying to continue. What version of python are you using?


#573

You could be right. Those style of failures should prevent a friendly_title from being successfully contructed for printing out.

Let me push out a 2.0b4.


#574

Just to be clear. Surlatablo exited out after each error and did not continue. I then hacked the script to skip the error. That way I got the next error and so on. Until I didn’t get any errors anymore.

Python 2.7.12


#575

Download and test with: http://endlessnow.com/ten/Temp/slt-test-py.txt

You’ll want to delete your SurLaTablo cache files before running.


#576

@cjcox: That worked. I deleted the cache file and ran this version. No errors this time.


#577

I have a couple more changes… then I’ll push out an official 2.0b4.


#578

SurLaTablo 2.0b4

You will want to remove the cache files under your SURLATABLO_ROOT and do a full reindex of your Tablo.

2.0b4 - Fixes. Meta data changes involving titles.

Addition of options[‘titles_from_description’] which defaults to false. If set, if there is a resource with no title, it will see if it can construct something from the description up to the first comma or semi-colon. Otherwise and by default title will set to lair_date YYYY-mm-dd. Sports with no season will no longer have empty parens appended to the title. As with TV, worst case for Sports will append lair_date YYYY-mm-dd.


#579

I may have to make title handling a bit more configurable… This means one more beta at least.


#580

@cjcox, i’ve got an interesting problem here.

When I use the normal MP4 transcoder I get my file just fine. When I try and use the new MP4zap1 transcoder it runs but I get a 0 size file as output. Any suggestions on where to look? I don’t get any errors and it processes just fine as far as I can see.


#581

What platform OS? What version of ffmpeg? When you say it “runs”… does it seem to be doing something? (that is, how many seconds do the steps take).


#582

I’ve got it running in a FreeNAS jail under FreeBSD:
[root@tablorip_1 /]# uname -a
FreeBSD tablorip_1 10.3-STABLE FreeBSD 10.3-STABLE #0 r295946+07c41cd(9.10-STABLE): Wed Nov 9 00:19:25 UTC 2016 root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64

Latest FFMPEG from repository:
[root@tablorip_1 /]# ffmpeg
ffmpeg version 2.8.8 Copyright © 2000-2016 the FFmpeg developers

It runs for about 5-8 mins and does all the extraction and such as I see the .ts file getting created. According to the log file I have it took about 259 seconds to get the file from the Tablo, 41 seconds to generate the first MP4 file and then 0 seconds when it goes to perform the commercial zap and generate the -z.MP4 file. But I am not seeing any error getting thrown.


#583

Might have found it but not 100% sure. Found the line below buried in my log file. Thinking this might be causing the issue.

[aac @ 0x80a896400] The encoder ‘aac’ is experimental but experimental codecs are not enabled, add ‘-strict -2’ if you want to use it.

Not sure if that’s it or not since it does actually generate the file just with 0 size.


#584

I had to do something similar with a previous version of SurLaTablo on Mac OS X.


#585

yep… since it requires a full transcode and not mere recontainerization, you’ll need to either create your own transcoder in surlatablo2.conf (recommended) or edit the code and add the corrected option (see below for what you would put in your surlatablo2.conf):

TRANSCODER_OPTS['Mp4zap1'] = {
                    'help': [ 'Ultrafast Mp4 re-transcode that force -z 1.' ],
                    'command': [ FFMPEG ],
                    'threads': [ '-threads', '0' ],
                    'inputfile': [ '-i', '${ts_filename}' ],
                    'subtitlefile': [ '-f', 'srt', '-i', '${srt_filename}' ],
                    'options': [ '-preset', 'ultrafast' ],
                    'subtitleoptions': [ '-c:s', 'mov_text', '-metadata:s:s:0', 'language=${lang3}' ],
                    'audiooptions': [ '-strict', '-2', '-metadata:s:a:0', 'language=${lang3}' ],
                    'metadata': [],
                    'zap': [ '1' ],
                    'ext': [ '-z.mp4' ]
                }

Can’t remember if the options go at the front of audioptions or at the rear… but you can test.

Alternatively, you could try a newer ffmpeg.