being bilingual

There are times for me when my English isn't crisp, sound terrible, and at the same time there are times where my Japanese isn't at it's best. I mumble, cannot say the word fluently, stutter and it irritates me when it happens because I feel like I'm giving the listeners a hard time. People hit me with the "huh what you say?", or "say that again", and I'm in need of how to make my English at it's best whenever I speak it. I make myself think in that specific language when I speak it, so it's not the problem in where I'm thinking in the opposite language when I speak the other. I'm just curious if anybody else had this experience, and how the fixed it.