Conversational speech recognition with multiple speakers?

I’m trying to use matrix for an application involving a couple speakers going back and forth in a small room with little noise. I haven’t really gotten good results so far on speech recognition - Google Speech is the best but it misses almost half of the words spoken. This only happens when there’s more than one person - i’ve tried IBM Watson and Amazon Transcribe but have gotten the same result.

Has someone been able to accomplish this and had success? If so, which libraries did you use?