I suppose its where you feel comfortable with your skills... pulling external cladding from the frame is a huge job, and chances are, if they are battered on the outside then the inside isnt much better. Like you I looked at loads of caravans and they were all in various states of neglect (not disrepair, neglect).
So yes, sometimes spending a bit more upfront means less headaches in the future, and you can use straight away.

