It would, you're right, but there's always the option of rambling without giving him a chance to reply and then hanging up. I've found a lot of people you give your numbers to tend to do that so surely it wouldn't be seen as rude.

Hmm...do you have any psychic Pokemon yet?