My opinion is that you cannot start too young.  Infant toys that make tones - bells, squeeze horns of different notes - a re available.  When your child gets a little older and can grasp things, a push button xylophone is a good one, then graduation to a toy piano, etc.

Lessons are essential.  Rare is the child that is eager to take music lessons, but every adult I know who took lessons, usually piano, as a child regrets giving it up.

In my music teaching career spanning over 30 years, I've found that 5 is the ideal age to start music lessons. The child has usually been in a daycare or nursery school setting and by 5 is in kindergarten, so the teacher-student relationship has been established. Plus at that age reading skills have begun, which is a great aid to learning notation, a language in itself. Especially with piano music, beginner songs include words, so singing enhances and reinforces the learning process. I have started kids as early as 3 and 4 but this requires intensive involvement from the parents during daily practice sessions, as a child of that age needs guidance to fulfill their weekly assignments. If your "baby" is not ready for formal lessons, buy him or her a toy piano or organ to play with. Some come with color-based note systems to play simple songs, so reading is not necessary. This will spark interest and hopefully lead to a desire to study with a teacher.

I think it's great to expose toddlers to musical instruments such as toy pianos, xylophones, etc.  I've included a link to Oriental Trading Company that has a great beginner's rhythm set.  As far as learning to read notes, etc. I think around 8 years old with the recorder as a beginning instrument.  It is an inexpensive instrument and I allow my 5 year old to play with one, but he does not yet grasp the notes.  Enjoy!