bisexual
Your question isn't odd. In some mangas, I've read the term "only gay for x - the name of the person". I've read it in one of Sadahiro Mika's works (I think it was surf junkie or something like that).
I am not aware of any particular Japanese term for this situation but there is a more savant expression which is used to describe what you are talking about. It is "situational sexual behavior"
[A possibly common and transient example of situational sexuality is the person who, while self-identifying as heterosexual, will sexually interact with a member of the same sex (or a gay or lesbian self-identified person sexually interacting with a member of the opposite sex) - Wikipedia]
I found other interesting technical terms like "heteroflexible" and "androphilic". I think that both of them should be used with an adjective to limit their usage to the situation you are describing. I also find the second one quite interesting, it fits the situation you are describing perfectly if we say "situational androphilic" to emphasize the masculinity of both the subject and object of this situational attraction.
Heteroflexible means "mostly straight" but occasionally attracted to a same-sex person.
Is there an actual term used for when a straight guy falls for a gay guy? I mean I would normally use the term bi-sexual but it seems a case of I want to sleep with this one gay guy, not all gay guys. I don't think the term gay applies as well since obviously they've had relationships with women by choice and attraction in the past. But mangas seem to avoid the use of bisexual like it's a plague word, so I'm curious is there a term for it in yaois?