If player A declared the end of his/her turn, like saying "I'm done," "Go ahead," etc, but then immediately after says, "Oh, never mind, wait!" before player B has untapped or cast any spells, done any actions, etc, at the end of player A's turn, can player B simply be like "No, sorry, you ended your turn and I'm going to hold you to it and go ahead with mine," or is s/he required to allow player A to "go back" and cast something in his/her postcombat mainphase if player A so desires? The situation here was in an informal, unsanctioned draft of Unhinged. Would it work differently in higher-level play? (my impression here is that it seems like player B is being pretty unreasonable given the circumstances, but the question is that is it within her/his rights under the rules to do so?)
In this situation, I don't think it's against the rules to hold them to their end of turn. They passed priority, you passed priority, and you agreed that the turn was changing to yours. Call a judge to be safe, though.