Air Vent dirt or Rust?

I have a 1996 Honda Prelude.

I bought it used a little over 2 years ago. I have noticed a slow
accumulation of brown dust in the air vent pockets. Specifically
where the air vent connects to the drivers front door. (you see the
vent when you open the door, it is in the dash. ).. But I noticed it
in other places as well.

I have cleaned it before (vacuum), but it always comes back.

I thought it might be an interior vent air filter deteriorating, but
the service manual doesn't mention one.

The particles are bigger than dust particles, and brown in color. A
friend said he had something similar with a home A/C, where it turned
out to be rust.

I rarely use the A/C, and usually use the heater, and/or the blower is
on a lot of the time, blowing unconditioned air much of the time.

I live in a fairly dry (low humidity) climate, northern california..

I suppose I will need to remove the blower/heater unit and inspect it,
but I am wondering if anyone has seen this before?

Thanks

Randall Fox