SurLaTablo.py - Python program to query and convert Tablo recordings

@lurker, I know Linux isn’t for everyone (though it should be!!)… with Linux (or Mac) you’ll get full UTF-8 filenames for shows with special characters.  It’s just a benefit that you don’t get with Windows.  And of course, it will run a bit faster if that matters.

There’s also some fringe cases using Python subprocess that are handled a bit better and buffer flushing, etc…

In other words, under Linux (or Mac) it’s also a bit more reliable.  Python may not have been the best choice… I had other reasons for writing it in Python.

I like Linux, over the past couple years anyways.  The two real things wrong with Linux is, it’s not support much by the major software/hardware vendors very well, the second is it’s FREE!  Which BTW, is way it isn’t support very well by 3rd parties.  At less then 1% market share in the computer OS world, manufacturers understand it will be hard to generate enough profit with so few users and they also know most of those users would not pay a penny for software – “It’s the principle of the thing”! That aside, it is far more stable, harder to crash, much better threading/multi tasking, and more.  But, I have to have windows for most of our business software.  Tried more than a couple times to switch, but the closest I have come is running a Linux server with Windows in a VM.  I have that on a few computers all the time and as long as the computer is strong enough, there is little real bottlenecks, just a bit of incompatibility work-a-rounds I have to do.  Actually, I am really curious as to why Microsoft is so stubborn with keeping Windows architecture, but I suppose they have their reasons.


I have had Linux boxes since it came out.  I started with computers in 1969 and have kept up with them since then.  Getting tired these days pretty easily and the old kanoggin ain’t what it used to be so I try not to burn it out thinking to hard these days, HA!

I’ve already put a Linux VM in the QNap and tried the VLC backup and it seems to work OK now.  Before it was clipping off the ends of the captures.  I had to reformat the 1.5TB tablo drive, just could not get Tablo stable with a 90% full drive.  Have it backed up on the NAS, so I have access to all of the recordings.  Just need to figure out the simplest way to merge the TS files, fined the title info, and move them to my PLEX server.

Thanks for the suggestions all.

-Rodger


The most popular software used today is free.  Just the way it is. :)  And IMHO, best free software is truly free software (FOSS). YMMV.  I would be bored out of my skull without Linux.  Without it there would be no web (practically speaking), no Roku, no Tablo, no Android (anything), arguably a lot less Mac… etc… etc…  tons of network switches would not exist, tons of load balancers, storage subsystems, etc…  We probably wouldn’t have had VMware and whole hypervisor movement.  Sure, it could have happened, just likely not when it happened… and you never know, it may never have happened!  Linux forced a lot of others to wake up and pay attention…

I cringe at a world built on WinCE :slight_smile:




I have no problem understanding that.  In my heyday, All I did from dawn to dawn (I would sleep maybe 15 hrs a week if lucky) was computers.  I built my first computer in 1966 with parts bought from Olsen Electronics in Whittier, CA, where I grew up.  The times where so fun so filled with learning and acquiring the ability to actually make something that I could control and have it do something at my command, it just seemed limitless. Today, I enjoy time doing some of this stuff, but I also have alot to do not related to computers.  Just the way life has become more and more over the past 20 years or so.  Now retired, I still do things, but mostly to help family and some friends.  And of course I do things I wasn’t able to do for one reason or another while running business’s.  Life is enjoyable, and it is nice to dream of times gone by.


-Rodger
SurLaTablo 1.0 released

There's also a Windows installation vid

@cjcox Thanks for this great and easy to follow tutorial !! I’m now enjoying watching my programs through my WDLive media player !!

@fzrxhq, Thanks for the kind words.

So how does a non-Windows guy produce a Windows video?  Fortunately it didn’t take too long… or else…



I have yet to get this going.  Are you still thinking of making an Web version?


-Rodger

I can’t even give an eta…

OK, thanks.  I am still trying various solutions to get the 1.3TB or so from my backup copy moved to PLEX.  I really don’t need all of those, but maybe about half or so.  I can do it, but it’s a job.  Hope your HTML/Web version works.  It would be nice to have any easier way to do backups/transfers to PLEX.


-Rodger

There probably won’t be any difference with regards to installation requirements though.

I think I have pieced together a solution that will work for to get recorded shows/movies off my backups.  It’s still a bit involved, but the file structure is fully intact and merged, so I guess it will do.


Still interested to try your system get backups off my active table HD and into PLEX.

-Rodger

SurLaTablo, like the other solutions found in these forums is not a backup solution (that is, not a way to backup Tablo raw data from one hard drive to the next in order to swap hard drives).

SurLaTablo is mainly for query Tablo data (which is hard with just the Tablo) and then it can take the results of a search and pull the mpeg4-ts files off the Tablo and do many different things with the data.  Main goal was Plex, so usually you would use it to pull the show and transcode it in a way that is suitable for your Plex environment.

Unlike some of the other tools, I’ve added some capabilities to take care of certain problematic cases when going to Plex (cases where you really need to create metadata for example).

I figure if somebody else can get something out of the tool great… and I’ll certainly look into features etc. that people want.

Next release will include my alternative commercial zapping (you can choose) algorithm and better truncation (to get rid of the overage on all shows).


I need to take the time to put your app on one of my linux VM’s.  I have a Linux Mint VM in my Video Editing Station which has bunches of power, so might put it in there.  I also have a personal notebook running Linux Mint as the boot OS, so could put it there.  Already mentioned the QNap NAS, which I did put Linux Mint in a few days back, but just tried that VLC script in it so far.  The QNap does transcode pretty fast, and really if you have set up your MP4 encode to do Unprocessed, it wouldn’t take much CPU power anyways.


Forgot – I would like a better commercial remover then the ones I have used in MCEBuddy.  I’ve only worked the that app a little, but seems ShowAnalyzer is fast, but misses a lot of them.  Comskip is too slow, but I guess I could donate and it would speedup.  But sounds like some work still has to be manually scripted in the ini file for Comskip to work right.

What do you use for commercial removal?

-Rodger

My commercial removal routine is very basic.  Unlike comskip (which is what is used by MCEbuddy I believe), I only look at black frames.  I was thinking of adding audio checks as well, but then I decided that maybe the best thing was to produce a tool to make commercial elimiation easy and use something more foolproof… that is, the human mind to make the important decisions.  I make continue to play with auto-zapping of commercials… but there are just way too many cases that cannot be easily determined.  Easier to just present a buch of options at someone and ask them to “click to select” what stays and what goes.

Yeah, it would be tough to do it any way I have thought about it.  I actually bought the ShowAnalyzer app and played with it, but it wasn’t the complete app for some reason.  Kinda think the coder more or less quit on it.  Got a refund, but would sure have liked the full app, which included an editor by the looks of it.


I’ve got a request into MCEBuddy asking for help with removing commercials with their software, so might get someplace with it if they have a practical way around this.  One thing about MCEBuddy is it works at a folder level.  Monitoring changes to a folder activates the process and then puts the finished work into another folder of choice.  When I was using it with Windows Media Center, it hardly had a hiccup.  

-Rodger

I’ll be honest, I haven’t really tried MCEBuddy.  I believe that auto commercial skipping using MCEBuddy relies on Comskip though.  So, apart from the extra features, when it comes to commercial skipping, it’s all Comskip.

Comskip uses many techniques to try to determine commercials.  As I implied, it’s more robust.  And it’s good to know that it didn’t make many mistakes.  The algorithms in SurLaTablo do make mistakes, by missing commercials that should be skipped and sometimes removing conent that’s part of the show.  It suffers from both problems.

As I understand it, many people use Comskip to create a sort of “skip list” that can be used with specific players to make it easy to “skip” and since it’s an optional skip, it won’t mess up.  The idea is that commercial starts, you skip to the next break and if that’s not the end of the commercial segment, you skip again to the next break until you’re back in the show.

What I want to do is develop a tool that presents the segments with preview information and allow you to choose the segments that stay vs. the ones to “zap”.

There are other problems though.  While commercial zapping (be that automatic or via “segment preview + choose tool”), right now, you lose the “closed caption as subtitles” feature.  So, lots more work to be done there.  It’s not hard just becase the time streams are easily parse in the .srt that comes from ccextractor… but still, you have to do the “math” so to speak.
 

1 Like

I have been using the ShowAnalyzer plug-in in MCEBuddy as it was 20 times faster than Comskip.  Just yesterday found out that if you donate to Comskip, they will turn-off throttling.  Thinking I might do a donation, just to see how well it can work.  I’m pretty good at figuring stuff, was my job in one way or another most of my adult life.  Maybe I’ll take a look-see at the Comskip app, which I also just found out about. 


-Rodger

Greetings,

I’m on a Mac running surlatablo.py but I’m getting the following error that I’m not able to track down.

Traceback (most recent call last):
File “surlatablo.py”, line 2230, in
jch_ids = urllib2.urlopen(ch_ids_url)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”, line 127, in urlopen
return _opener.open(url, data, timeout)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”, line 404, in open
response = self._open(req, data)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”, line 422, in _open
‘_open’, req)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”, line 382, in _call_chain
result = func(*args)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”, line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”, line 1184, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 61] Connection refused>

Make sure you have a surlatablo.conf setup in your home file system or you can edit the surlatablo.py and hardcode the values there.

In particular I supect you have a bad value for TABLO_IPS. It’s an array so it will look something like:
TABLO_IPS = [‘192.168.1.73’]

Where you would replace 192.168.1.73 with the IP of your Tablo.

SurLaTablo 1.1 Released