Not to be one's cup of tea / Not to like someone
Literally 'not to be the saint of one's devotion'. Describes politely not liking someone or something without specifying why.
He's not my cup of tea, but I respect him.
—Do you like her? —Honestly, she's not my cup of tea.
That type of music is not my cup of tea.