Who can protest?

Here's a thought......
Heading toward the finish with two boats clear in front of me - unsure if I'll catch them by the end to be fair but hey, i'll give it a go.
The two ahead are locked in a tacking duel with me, as I said clearly behind - 5 boat lengths maybe?
Going in to the line, boat A is on starboard, boat B on Port. Boat A calls 'Starboard!' quite clearly but boat B doesn't budge and A takes evasive action. Straightforward Port / Starboard.
However, A doesn't protest - the 'evasive' action saw them remain ahead(even though it was a considerable manouvre).
Now, B - in my mind, should have taken a 720 anyway, but didn't. Carrying on to the line ahead of me by around 2 maybe 3 lengths.
If B had carried out the required 720, I'd have gained a place and a point and, quite possibly, a higher place overall.
My question is, even though I wasn't directly infringed, could I have lodged a protest against B for not taking the required penalty?
I did discuss this at length with the infringee post-race, the answer being: "Well, I'd have had nowhere to go" My reply being: "Tough, then you shouldn't have been there"
Would I have been able to protest ands on what grounds? - is it as simple as I see it? -Straightforward Port/Starboard and no penalty taken?