Hi CS,
That's a hard question. You should check your company policies on special circumstances, assuming there is one. Read your handbook, or talk to your HR dept.
I suspect, based on your stmt that you did not get a note from a doctor, that the time off is going to be on your back. Your manager's call will probably be categorized as allowing 'unpaid' comp time.
You could consult an attorney, but at a guess that will cost more than the lost couple of days wages.
Hope this helps.
Source: http://www.abusinessmentor.com