Mosh Was Painfully Slow, and I Almost Blamed the Tradeoff I'd Just Made
Shellby's from-scratch mosh-compatible transport shipped and was unusably laggy. I had just made a deliberate throughput-sacrificing tradeoff — stop-and-wait state sync — so it was the obvious suspect. But one metric didn't fit, and it exonerated the architecture. The real culprit was a mis-filled timestamp that made the server throttle itself to one packet per 100ms.
Shellby’s mosh-compatible transport had barely started working when the feedback arrived: unusably laggy.
My heart sank, because I knew it might be right — I had made a deliberate throughput-sacrificing tradeoff in this implementation.
The explanation I had ready for myself
Mosh’s SSP protocol syncs terminal state over UDP, not a byte stream. Each Instruction from the server carries a pair of state numbers, oldNum → newNum, meaning “this diff moves you from state oldNum to newNum.” The catch is that oldNum is the server’s assumption about the client’s state, and that assumption always lags by RTT/2. Under latency you inevitably hit “client is already at state 5, but the packet says 4 → 6.”
A complete mosh client absorbs that mismatch with a history frame buffer. Implementing one means building a Framebuffer, a constrained ANSI parser, and a frame differ — more work than OCB3 encryption, protobuf codec, and the SSP state machine combined, and neither SwiftTerm nor xterm.dart can be cheaply cloned.
So I skipped it: keep only the head state, and when oldNum ≠ head, drop the packet and don’t ack it. The server’s assumed_receiver_state therefore doesn’t advance, and it re-diffs from the last state we confirmed — convergent, and correct. Once you guarantee “only apply packets where oldNum == head,” whatever the server sends is a correct diff from the current screen and can be fed straight to the terminal. No frame model of my own required.
The cost is written in the design doc, by me: throughput degrades to a stop-and-wait protocol. The measurements are mine too:
| One-way delay | RTT | Hit rate | Refresh interval (median) |
|---|---|---|---|
| 0 | ~0 | 96.0% | 0.25s |
| 25ms | 50ms | 96.2% | 0.25s |
| 100ms | 200ms | 51.7% | 1.27s |
| 250ms | 500ms | 45.2% | 1.26s |
At 200ms RTT the hit rate halves and refreshes drop to once per 1.3 seconds. “Laggy” maps onto that table perfectly.
So I had a complete, self-consistent explanation backed by my own measurements. That is exactly what made it dangerous.
One number that didn’t fit
Before touching the architecture, I measured three things. The test setup was local loopback with per-character echo — RTT is approximately zero, which by the table above should land in the best row: 0.25s refresh, 96% hit rate.
- Median echo latency: 102ms
- First inbound packet latency: 102ms
- Packets dropped on
oldNummismatch: 0
The third number exonerated stop-and-wait.
There is exactly one mechanism by which stop-and-wait causes lag: on oldNum ≠ head, drop without acking, costing an extra round trip. A drop count of zero means that path was never taken. The packets weren’t being dropped by me — they weren’t arriving.
The second number filled in the other half. The “first inbound packet” is the first data packet after the handshake, when there is no state to sync yet and stop-and-wait doesn’t participate at all. It was also 102ms, so the slowness happened before arrival.
Both numbers point the same way: I wasn’t dropping too much. The other side was sending too little.
Who’s throttling
Why wouldn’t the server send? Reading mosh’s behavior: mosh-server’s send interval is clamp(SRTT/2, 20ms, 250ms). It deliberately throttles based on its estimate of round-trip time — the higher the RTT, the sparser it sends.
And where does its SRTT come from? From the packets we send it. Every mosh datagram’s plaintext header carries a pair of timestamps:
plaintext Packet = be16(timestamp) || be16(timestampReply) || fragmentBytestimestampReply is the echo field, and the rule is:
reply = peer's timestamp + (our send time - our receive time)That is, the peer’s timestamp plus how long the packet sat in our hands. The hold-time term is not optional: the peer computes the round trip as its own current time - reply, so if we don’t subtract our own processing time, that time gets counted as network latency.
There’s a second trap: before you’ve received anything, the sentinel must be 0xFFFF, not 0 — 0 is a valid timestamp, and sending it means reporting a real reading you never took.
My backfill was wrong. The server computed RTT ≈ 200ms, so clamp(200/2, 20, 250) = 100ms, and it dutifully throttled to one packet per 100ms. Of that 102ms median latency, 100ms was the server waiting out an interval it believed it should wait.
After the fix: 102ms → 16ms, six times faster. Over the same six-second interop test, cumulative bytes transferred went from 758 to 1074.
“Laggy” is a compound symptom
There wasn’t one culprit. The same investigation turned up two more independent causes, both of which present as lag:
Dropped keystrokes while typing fast. UserMessage.1 is a repeated Instruction — if a diff claims to cover oldNum → newNum, it must actually contain every user action in between. My original implementation overwrote pendingDiff each time while still incrementing newNum, so intermediate keystrokes were permanently lost. Typing ls delivered only s to the server. What the user feels is “lag,” but the data is simply gone. The fix is an unacked-action queue, pruned by ack.
A send throttle I had mistakenly added on the receive path. Worried about an ack storm, I had throttled sends on the receive path. But under stop-and-wait, the ack is the only thing that advances state — throttling acks throttles the entire session. And since we only reply when oldNum == head hits, no storm was possible in the first place. That throttle was a patch for a problem that never existed, and it cost real throughput on the way.
Three causes stacked together; fixing any one alone wouldn’t have made the experience normal. Which is also why the original symptom was so easy to pin on the architecture — the severity genuinely “deserved” an architecture-level defect.
Three things I’m keeping
One: the tradeoff you just made is the most dangerous suspect. Not because it’s often guilty, but because you know its rap sheet best and the explanation comes out smoothest. I had a performance-degradation table I’d measured myself, and the symptom slid into it without friction. A self-consistent explanation is easier to reach than a correct one.
Two: every tradeoff needs a metric that can exonerate it. What saved me here was the “oldNum mismatch drop count.” It wasn’t added for debugging — it was a counter I happened to keep while implementing stop-and-wait, and it happens to be the sole entrance to the degraded path: take that path, and the count is nonzero. If a tradeoff’s cost isn’t measurable, it will absorb every attribution, because no data can contradict you.
Three: before optimizing, confirm you aren’t compensating for a bug. That mistaken ack throttle is the archetype: worry about a problem that never occurs, add a limit, and the limit becomes the new bottleneck. Before adding throttling, caching, or batching, measure whether the problem it addresses actually exists.
All three fixes are now pinned by regression tests: testTimestampReplyEchoesPeerPlusHoldTime asserts the backfill equals “peer timestamp + hold time,” and testUnackedKeystrokesAccumulate plus testAckPrunesUserActions guard accumulation and pruning of the unacked queue. Stop-and-wait itself didn’t change by a single line — it was innocent the whole time.
Comments