How long is a piece of string? Unless the surface is completely smooth, it will damage the seal and eventually result in a leak. Leaks can be minimised by using the special liquid grease specified for late model Defenders etc swivels.
How bad the surface is before you need to do something will depend on whether you are using this liquid grease or oil, and how short a seal life you will put up with, and how much of a minor leak you can accept.
John

