Suzi, Ask yourself this question, Why does he like them?  Since he is only four, I am assuming it is a simple answer like, the shape, colors, or feel of the materials.  Ex, this fetish might be easily replaced by giving him a bandana or blanket with the same feel if he likes the feel of the material.
One thing that I have learned with young children is that sometimes they need to be introduced to different experiences or items to gain interest in something.  Introducing him to other things such as blowing up balloons, or practically anything else could draw his interest away from areas where you don't want it.  
I try to introduce my daughter to new thing constantly.  I am attempting to help shape her life by giving her a good group of interests to choose from.  She may not have much interest in anything that I have tried to show her, but I have been able to break bad habits by diverting her attention when I feel that she is heading in the wrong direction.  It can be hard to get through to the child when you tell them to stop something, and sometimes can be easier to get through by telling them to try something else... although not all children will respond the same to this stategy
Try a "focus" in on "raising a child".  Meaning it is our job to show and teach my child about life.  Not just to be around as they learn it for themselves and answer questions, but to actually contribute to the learning on a regular basis.  This attitude also forces you to learn with your child and reap the same rewards you child does from learning, also helping to create a stronger parent-child bond.

Hi,

When our daughter was a toddler, she too was interested in playing with our underwear. She'd find them in folded clothes piles, then put them on her head like hats, or use them like a flag. I don't think she understood their real use (she wasn't potty trained at that time), and instead was simply looking at them as a play objects.

If you're concerned that your grandson has an unhealthy attachment to your and your daughter's underwear (and isn't simply using them as play objects), I suggest discussing this with his pediatrician at the next visit. I'm including below some additional informational resources below you may also find helpful.

Thanks for asking!