I see a lot of people confused about this, so I figured I'd add in the necessary context.
As per the novel, the idea is that someone from a lower class background would never have the nerve to reject tea offered by their higher ups, especially when pressed. In a strictly hierarchical society, a random citizen with no backing rejecting the Duke/ Imperial family member/ Commander is unheard of. Thus, a boy from the mountains doing this says something about his character, and can also be seen as suspicious/ noteworthy etc.