My son is going through a phase that he just won't drink milk. How bad is it to not drink milk, just get the nutrition form other dairy products?

From the post you didn't say how old your son is. My son breaks out with ezcema when he has milk or milk related products, so I give him soy, I give him a little milk or cheese every now and again to see if he's over his condition.