It is not unusual for a four year-old boy to not be ready for Kindergarten. Many parents choose to wait until their children are closer to 5 or 6 years old when they start school. Research has indicated that waiting until the child is older helps many children feel more confident, and gives them more opportunities to become leaders rather than followers. This seems to be even more important for boys because boys do not mature as quickly as girls. It has very little to do with intelliegence. Your child may be very advanced academically, but may not be ready for the social and emotional stress of formal schooling. It is more about emotional and social maturity.
Overall, most parents will say they do not regret waiting to send their child to school until they were closer to 6 years old. But on the other hand, more parents will report that they regret sending their child to school before they were emotionally ready.
Do not hesitate to take him out of Kindergarten and wait another year. This will not affect his development. But, it may be more traumatic if he has problems and has to be held back in a later grade. Waiting until your child is emotionally ready to start school is kind of an insurance policiy for your child's future. It does not guarantee your child will never have problems, but it may lesson the liklihood of him having to struggle as hard to "keep up" with his peers emotionally and socially as he grows older.
You also mentioned that he "doesn't hear anything." Please do discuss that with your son's pediatrician to rule out any hearing problems. His pediatrician may also have some suggestions to address the behavior issues you mentioned.