WebSocket security: Preventing Improper Subprotocol Negotiation

During a recent audit of a real-time messaging service, I found a gaping hole in how the server handled incoming client requests. The backend was blindly accepting any subprotocol requested by the client, which paved the way for a cross-protocol attack. It wasn't just a theoretical bug; it was an open door for connection hijacking that could have let an attacker bridge the gap between different application states.

Understanding WebSocket security and the Sec-WebSocket-Protocol

When a client initiates a WebSocket handshake, it sends a Sec-WebSocket-Protocol header. This is a list of subprotocols the client supports, like json, protobuf, or v1.myapp.com. The server is supposed to pick one from that list and confirm it in its response.

If your server doesn't explicitly validate this list, you're essentially letting the client dictate the communication contract. I’ve seen developers assume that because the connection is "upgraded," the protocol is implicitly safe. That’s a dangerous assumption. Much like when you're preventing HTTP parameter pollution, you have to treat every incoming header as potentially malicious or malformed.

The risks of improper negotiation

When you don't restrict subprotocols, you invite a few specific problems:

1. Cross-protocol attacks: An attacker might force the server to speak a protocol it wasn't intended to handle in a specific context.
2. Logic hijacking: If your backend logic branches based on the subprotocol, an attacker could trick the server into executing code paths intended for different API versions or internal features.
3. Data format confusion: If you expect protobuf but the client forces json, your serialization logic might crash or, worse, expose debug information.

I once worked on a system where the server would simply echo back the client's preferred protocol without checking if it was in an allow-list. It took about two days to realize that we could bypass some auth checks by switching to a legacy subprotocol that had weaker validation logic.

Implementing strict validation in Node.js

In Node.js, specifically when using ws (version 8.0+), you shouldn't rely on the default behavior. You need to provide a handleProtocols function.

JAVASCRIPT
const WebSocket = require(CE9178">'ws');

const wss = new WebSocket.Server({
  noServer: true,
  handleProtocols: (protocols, request) => {
    const allowed = new Set([CE9178">'v1.myapp.com', CE9178">'v2.myapp.com']);
    for (const protocol of protocols) {
      if (allowed.has(protocol)) {
        return protocol; // Accept the first valid match
      }
    }
    return false; // Reject the connection if no match
  }
});

By returning false, the server sends a 401 Unauthorized or a 400 Bad Request, effectively terminating the handshake before any sensitive logic executes. It’s cleaner than trying to filter headers manually after the connection is established.

Securing PHP WebSocket implementations

PHP is often a bit trickier because it’s usually sitting behind a proxy like Nginx. If you're using a library like Ratchet, you need to ensure your handshake logic validates the subprotocol before the connection is accepted.

In PHP, you should inspect the Sec-WebSocket-Protocol header during the request cycle. If you don't control the handshake, your proxy (Nginx/HAProxy) should be configured to strip or validate these headers before they reach your app. Much like how server-sent events security requires careful header inspection, your WebSocket gateway must be the gatekeeper.

Why negotiation matters for API development

When you're building out secure API development practices, you have to treat the WebSocket subprotocol as part of your API contract. If the contract isn't enforced, your versioning strategy falls apart.

I’ve seen teams ignore this because they think "it's just a string." But in a distributed environment, that string defines how your workers interpret incoming binary or text frames. If an attacker can switch your server into "debug mode" or "legacy mode" simply by changing a header, they’ve already won.

Final thoughts

The biggest mistake I see is developers focusing entirely on the payload and ignoring the handshake. You can have the most secure message parsing in the world, but if the protocol itself is hijacked, the payload doesn't matter.

I'm still cautious about how some third-party WebSocket libraries handle these headers by default. Always verify the library's source code to see if it implicitly trusts the client. If it does, you need to wrap the connection handler yourself. It’s a small amount of extra code that saves you from a massive headache down the road.