It is so heartbreaking to watch our kids struggle socially isn't it?  Kids can be really cruel and often times it seems like there's no rhyme or reason behind who's "in" and who's "out" of the cliques at school.  This age (4th grade, 9 or 10 years old) seems to be when things get tough for kids.  

I think a good first step would be to talk to your son about how he's feeling about his situation.  I sometimes worry about my own son because he's very shy and quiet. But every time I talk to him about his social life he reminds me that he's really happy having just a few good friends and not being in the center of things on the playground.  He'd rather play by himself than be part of a big kick ball game and I constantly have to remind myself that that's OK because he's happy.  If you have an honest conversation with him, your son can probably help you understand whether he actually does feel lonely and the specific parts of his situation that he's unhappy about (if any).

If you find out that your son does want to have more friends at school but just doesn't feel confident about making them, maybe you can help him by setting up some play dates after school or on the weekends with kids he feels like he might enjoy spending time with.  I've seen with my son that he has a much easier time developing relationships one-on-one outside of school.  It may take a while, but eventually these out of school relationships can turn into in-school friendships too.  As far as the neighbor goes (who your son plays with outside of school but not in school) - you should ask your son whether he's ever talked to the neighbor about that.  It sounds crazy but it could honestly be that the neighbor doesn't know your son wants to play with him at school.  My son is so quiet that kids often think he doesn't want to play with them simply because he's never asked or because he gets nervous and walks away when they ask him.

I do think it's important to keep a close eye on whether or not your son is being bullied.  It sounds like he may have been the punchline of a cruel joke which makes me wonder if he's being treated this way more frequently than you may be aware of.  I'd encourage you to check out Education.com's Bullying Resource center which will give you great info about how to talk to your son about bullying, determine if he might be being bullied, and help stop it if he is.  I'll paste the link below.

I'm sorry you're both suffering in this way.  It really is heartbreaking when it seems like the rest of the world doesn't appreciate your child the way you do.

Kat

Hi yedna
Sor sorry for you little boy and your heartache,im in exactly the same boat as you right now, and its breaking my heart!!!
I dont know what to do or which way to turn for the best, i`m even thinking about changing school for his last year, im going to sit him down and talk to him tonight see if he would want to do that, his onfidence is so low, its awful listening to him..
It all begin when the leader of his friends didnt like him being friends with anyone else, and when that boy wasnt in school for a day or two, my son wouldnt play with anyone else,as Max made it clear if he did he wouldnt be his friend anymore..So now Max has newer friends my littlen has been pushed out and as theyve all established their little groups of friends, noone wants to invite him to join in, when he asks and they say no he simply walks away, and tells me its ok mum i just walk around and think of things in my head and it gets me thru..
As a parent im devasted, and heartbroken for him, i cant sleep or eat with worry, whis has been happening now for some time...
Its been some time since your post, so im wondering if your son managed to make some friends? any info and tips would be more than welcome.