Removes a server port and IPv6 brackets without resolving the host. Invalid addresses are left intact.