Skip to content

sc_bytes_scheduled can underflow: pf_packout_size is not stable between schedule and send #652

Description

@AlexeyAtNol8

Summary. struct lsquic_send_ctl::sc_bytes_scheduled (an unsigned) can drift away from the true sum of scheduled packet sizes and underflow to ~UINT_MAX. Once it does, send_ctl_all_bytes_out() >= cwnd is permanently true, send_ctl_can_send() returns 0 forever, the connection is never tickable/writeable, and its streams' on_write is never dispatched — the connection wedges with an empty pipe (0 in-flight, 0 unacked).

Observed at the wedge: sc_n_scheduled == 0, sc_n_in_flight_all == 0, and sc_bytes_unacked_all == 0, yet sc_bytes_scheduled == 4294967295. Reproduces under high connection count with many short writes (we hit it at ~800+ concurrent connections with large per-stream transfers); it is timing-dependent, consistent with a small per-packet residual that accumulates.

Versions. Reproduced on v4.7.1; the root cause is still present on master (v4.8.0).

Root cause. sc_bytes_scheduled is maintained incrementally:

  • send_ctl_sched_Xpend_common does += packet_out_total_sz(p) at schedule-append, and
  • send_ctl_sched_remove does -= packet_out_total_sz(p) at send-remove.

But packet_out_total_sz()pf_packout_size() (ietf_v1_packout_size) is not a pure function of the packet — it reads mutable connection/path state that can change between the += and the -=:

  1. lconn->cn_flags & LSCONN_HANDSHAKE_DONE selects the short- vs long-header size. A 1-RTT (HETY_SHORT) packet scheduled before the handshake-done flip is costed with the long formula and, if sent after the flip, removed with the short formula → residual. There is no hook telling send_ctl that the flip changed the sizes of already-scheduled packets.
  2. p->po_path->np_dcid.len — has a hook (lsquic_send_ctl_cidlen_change) but it assumes every scheduled packet carries the DCID (diff *= sc_n_scheduled); short-header packets without PO_CONN_ID do not, so the hook can itself over/under-adjust.
  3. packno_bits (packet-number length) — also affects header size.

So += sz@schedule and -= sz@send use different values and do not cancel; the residual accumulates and, when the net is negative, the unsigned underflows.

Existing hint in the code: lsquic_send_ctl_cleanup keeps assert(0 == sc_n_scheduled) but has //assert(0 == ctl->sc_bytes_scheduled); commented out — i.e. the byte counter is already known not to reliably reach 0.

Suggested fix (PR to follow). Cache the exact size attributed at schedule time on the packet (a new po_acct_sz, mirroring how the unacked counter caches po_sent_sz) and subtract that cached value at remove, keeping it in lockstep in the few adjustment paths (incr_pack_sz, elide, update_for_resending, cidlen_change, rollback, and the gQUIC HELLO zero-pad in stream_write_to_packet_crypto). Schedule and remove then cancel by construction, regardless of any size-affecting state change between schedule and send.

This may be the same underlying bug as #301 ("after the client and server interact for a while, there is no traffic suddenly" — connection not tickable, send_ctl_can_send pinned against an exhausted cwnd) and #412 ("client stop sending data under pressure testing"), both of which match this symptom but were closed without an accounting root cause.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions