I believe this. Servers should not infer any privileges over the admins of a server. They should just provide an extra piece of (essentially redundant) infrastructure in a network.
A big goal is to avoid leaning on servers whenever possible. Servers naturally gain a lot of importance as hosts and, like a system of government with multiple branches, we want to constrain that importance to hosting and disconnect it from content moderation. As soon as we ask the server for a list of all its communities, it becomes the server’s job to moderate that list. That makes server choice more important than it should be; they really should be background infrastructure.