If you have penetration, your hits treat the enemy's resistance as x% lower than it's actual value. There is no limit on how much penetration you can have, and therefore how much lower than their actual value you can treat their resistances.

It does not actually change their resistances, nor does it affect anything other than hits.

Vaxthrul

So it doesn't work with damage over time?

Bane or vortex for example

Correct

Halinn

There is no limit on how much penetration you can have, and therefore how much lower than their actual value you can treat their resistances.

So you could in theory go below -100%?

Yes