Here are the minimum requirements for Siege.[support.ubisoft.com]
Since the minimum asks for 6GB, you most likely will be running into a lot of performance problems.