Hi Shannonmommy,
    I definitely wouldn't panic, it's natural for many children to take some time on symbol and number recognition to fully develop, but it never hurts to research other possibilities. Out of curiosity, does your son have issues with other symbol recognition such as letters?
 
 Regardless here is a reference article which shows the typical mathematical development of 4 to 5 year olds to see how your son compares in other math abilities:
http://www.education.com/reference/article/Ref_Tracker_Math_4_5/
 
Education.com also has a great number of resources to practice number recognition at home, here's a sample article:
http://www.education.com/magazine/article/preschool-number-recognition-counting-easy/
 
And finally, if all else fails you can see the symptoms of dyscalculia, a math based learning disability. You can read up more on dyscalculia here:
http://www.education.com/magazine/article/Does_Child_Hate_Math/
 
Hope this helps!

Thanks for reaching out with your concerns about your child's math learning. I would recommend finding ways to enhance your child's number sense using manipulatives (objects that learners can manipulate to understand math concepts). Here's one idea involving manipulatives:
 
http://www.education.com/activity/article/Backyard_Counting_Book_kindergarten/
 
Another variation of the backyard counting book would be to replace outdoor items with beans or cheerios or other fun, non-perishable food items. The concept is to glue down 1 bean (or non-perishable food item) on a card and write the “1” symbol on that same card and do the same for all the numbers you want to explore.
 
When you're ready for simple calculations, here's another idea, involving beans.
 
http://www.education.com/activity/article/bluebeangame_kindergarten/
 
You can also use other tools besides a pencil and paper for drawing numbers. You may try using sand or clay, to formulate the numeric symbols. Make the activities fun. You don't want to make your son dread math.
 
I would also look into Touch Math. It is a methodology that involves a specific number of touchpoints on a number that match the number itself, e.g., the number 3 has 3 specific touchpoints. This methodology has helped many of my students in the past, especially when it comes to calculations.
 
Math Touch does offer some free materials online. Here's a link to them explaining their methodology.
http://www.touchmath.com/index.cfm?fuseaction=about.how
 
Overall, I would recommend being patient with your son because he may not be motivated to engage in the number sense activities if he feels he's being pushed or there is concern about his capabilities. Find ways to make the number sense activities enjoyable and good quality time between the two of you.
 
Many children are hands-on learners and need kinesthetic activities to anchor specific academic concepts and skills in their brain. Here are some other education.com articles for fun, kinesthetic number sense activities:
 
http://www.education.com/activity/article/Number_Memory_kindergarten/
http://www.education.com/activity/article/Counting_Cups/
http://www.education.com/activity/article/countingbook_kindergarten/
 
If you wish to learn more about math manipulatives, check this site out. Math manipulatives are very useful and should be used in upper elementary classes, as well as in the younger grades.
http://en.wikipedia.org/wiki/Math_manipulatives
 
You are an excellent parent to be concerned about your son's math skills and researching ways to help him. Parenting is the hardest job in the world, but the most important.
 
Have fun exploring the world of numbers!
 
Best regards,
Lori

Looks like others have provided great resources for educational/development support.
 
On a more personal note: as an adult who remembers struggling with number recognition all the way through first-grade, I can connect.  I remember being consumed as a 5 year-old by "but what is a 3?? why does that shape mean this number? I know the order, but who decided the values? why?" Other kids were answering number questions so easily, while I just sat and wondered about the whole mystery, unable to relate 3 with three things.
 
I'm not suggesting you get a book on child metaphysics.  Just that some kids naturally get certain things, and are mystified by something else.  Perhaps the issue is not that he has difficulty with numbers, but that he is gifted with something that overshadows math.
 
For what it's worth, I eventually graduated from college with a degree in engineering. But I'm still not entirely satisfied with why the shape 3 means three things.