The biggest downside of passwords is that it is a shared secret – shared between the user and the validator. Unfortunately, passphrases also share the same disadvantage, though they fare way better in so many other ways.
Systems designed to use passwords or passphrases can be made to not store the passwords in readable form or even make it inaccessible to users responsible for designing or running those systems. However, many fail to take steps to protect the end users identity, knowingly or unknowingly, and leave the systems vulnerable to insider attacks.
An even wiser step would be to completely forego authentication using passwords and implement passwordless systems.