If the structure has been placed and painted, just have soap in your inventory, access the painted structure's pinwheel (hold E), and you'll see the option to remove paint.
This thread was added on July 10, 2019, with posts from Jatheish.