Why is my kitten meoing but dont want to be loving?

i have had him 3weeks if i hold him and dont let him down and stoke him he will settle and go sleep but he wont come to me and dont really like being held he comes to me meos loads for attention but if i go hold him he runs off even when i hold him and stoke him he will meo and i have only hurd him purr 1 he hasnt got worms