Yeah I'm not expecting to be able to create the perfect bot, but I want to get close
One of the biggest challenges is the fact that a navmesh "wall" is simply the edge of the navmesh. There is no way to tell if there is any actual level geometry in the way, or if it's just a ledge that the bot could feasibly walk off.
I think my first port of call will be to stop the recast library from eroding edges, so that bots can walk right on the very edge of walkable areas if necessary, and also to prevent it discarding walkable spots it deems as being too small. This should help a little, in so far that it no longer erects a barrier along the edges of a ledge.