Quick dumb diplomacy question

If I stick an army on a rival's borders and they challenge me to declare war, and - for whatever reason - I opt not to, what do I have to do to avoid the diplomatic penalty for then later declaring war? Is there a time limit, as with promising not to settle nearby? Can I avoid it by withdrawing for a couple of turns? What's the mechanic?

I'm sure this has been asked before and I would search for it but I have no idea how to formulate it in a searchable way. Sorry, and thanks for any help.