Rhetorical strategies are used to support a claim in an argument or to convey meaning. Rhetorical devices are used to deliver on those strategies. One rhetorical device is called understatement, which is the one you are talking about.

The phrase, "Yeah, you could say I'm a little annoyed!" is likely being used to intentionally downplay the importance of the situation or emotion, when in reality the phrase is often used by someone feeling the opposite: very annoyed (which we know). Understatement is being used here to make a point using a certain style of writing.

You can refer to the links above and the Khan Academy grammar video on under- and overstatement for more information.

Let me know if you have any more questions!