Is there a way to specify the max number of episodes to keep before deleting?
No. But it is an oft requested feature. You can achieve something running extra software alongside (might require some small programming).
Is there an API or some reference guide for how to talk to the tablo programatically?
Nuvyyo has an api document under NDA that is available if you request it. Not sure if the api covers everything (especially with regards to program guide and scheduling).
I didn’t sign the NDA, but yes, you could write something using what i know about the api.