I've just checked and this error happens on Chrome 57. Here are the errors I've got from the different browsers:
ie 11.1066.14393.0: SecurityError
firefox 53.0: Firefox can't establish a connection to the server at wss://localjabrawebsocketserver:8080/.
Edge 38.14393.1066.0: SCRIPT12017: WebSocket Error: SECURITY_ERR, Cross zone connection not allowed
Chrome 57.0.2987.133: VM46:35 WebSocket connection to 'wss://localjabrawebsocketserver:8080/' failed: Error in connection establishment: net::ERR_SSL_PROTOCOL_ERROR--
I'm running Windows 10 pro version 10.0.14393
The service is installed and running. From all browsers I am able to connect to and communicate with the non-secure ws://localjabrawebsocketserver:8080
Could this be a some other issue. Windows blocking the connection? I've checked the firewall and all traffic is allowed to/from the service.