I think we need to worry about paperclip maximisation as much as we need to worry about “Jon is having a bad day so brews up a novel plague”. There are likely ways it can go wrong that we haven’t even considered, too.