The woman allegedly fled to live off with another man in what the father of the child says is due to drug abuse