No - they can't be so smart to wait if there is at least 2 players - then they should leave the server (all).
The function autoadd/kick bot works this way - one player joins -> one bot is kicked; one player leaves the server -> one bot joins the server. So when You write max_bots 4 and You use fillserver command (or 4 addbot commands) - then at Your server You have 4 bots - one player joins -> one bot is kicked - they still can play 2:2. Second player joins - one bot is kicked again - You still have the situation 2:2. And if some of Your players decides to leave the server - bot is back to the server (still 2:2). But if You know AMX or AMX X and You know scripting langauge "small" You can write the plugin which will do the function You want. But it's the question to another forum..