Good question. I don't know if there is a law that specially states the using the restroom is a privilege or not. I would check with your local school district about their policy.
However, as a teacher I think taking away bathroom privileges could be considered unreasonable. I had the rule (even with my first graders) that if they had to use the bathroom, then tell me and go. I was not going to stop them, but I did need to know where they were in case there was an emergency. I also told them that if I noticed that they went too often that I would have to contact their parents to make sure there wasn't a medical issue. That usually took care of those students who wanted to waste time in the bathroom.
If you have more details about what is happening in a particular situation, we might be able to give you some ideas of how it can be handled.