If you mean overnight visits with you and you have been in a good co-parenting situation, go for it. Have him stay. At five months he doesn't know or care where he is except for feeding which is the trick. If your ex is breast feeding, it presents a different dynamic.
Wayne Yankus, MD, FAAP
expert panelist: pediatrics
2.) Attachment & Your Child info center -- "Infants have an instinctual need to identify adults in their life as safe, supportive figures. Being a socially responsive, available, and protective parent when your child is young can help to create a bond that lasts a lifetime.":