Yes I would do it. I have drilled a manifold with the turbo attached, but they were in a vise at the time.
I can't get a turbo off in 10 minutes. More like 30 as bolts are invariably difficult to swing a spanner on and sometimes seized. Minimum of 6 connections (air in, air out, exhaust in, exhaust out, oil in, oil out).

