When your voice is disconnected. Is Yahoo! Messenger also disconnected? Or do you just lose voice?
He might be booting your main ID. I think the main ID exploit is patched now.Try a brand new Yahoo! ID make an alias on it and join chat. If the problems stop then he was probably hitting your main ID.

Another thing is never add people to your main ID. Because it can leave you vulnerable to booting.