Why is husband so unhappy and always threatening to leave?

At first I was wondering if perhaps you were ignoring him and paying more attention to television programs or facebook. Then I read the part about him getting physically violent with you, as well as the verbal abuse. I don't think that is right. Whether the neglect is real or imagined, he doesn't have the right to treat you like that. As another has suggested here, relationship counseling is in order.