Summary. struct lsquic_send_ctl::sc_bytes_scheduled (an unsigned) can drift away from the true sum of scheduled packet sizes and underflow to ~UINT_MAX. Once it does, send_ctl_all_bytes_out() >= cwnd is permanently true, send_ctl_can_send() returns 0 forever, the connection is never tickable/writeable, and its streams' on_write is never dispatched — the connection wedges with an empty pipe (0 in-flight, 0 unacked).
Observed at the wedge: sc_n_scheduled == 0, sc_n_in_flight_all == 0, and sc_bytes_unacked_all == 0, yet sc_bytes_scheduled == 4294967295. Reproduces under high connection count with many short writes (we hit it at ~800+ concurrent connections with large per-stream transfers); it is timing-dependent, consistent with a small per-packet residual that accumulates.
Versions. Reproduced on v4.7.1; the root cause is still present on master (v4.8.0).
Root cause. sc_bytes_scheduled is maintained incrementally:
send_ctl_sched_Xpend_common does += packet_out_total_sz(p) at schedule-append, and
send_ctl_sched_remove does -= packet_out_total_sz(p) at send-remove.
But packet_out_total_sz() → pf_packout_size() (ietf_v1_packout_size) is not a pure function of the packet — it reads mutable connection/path state that can change between the += and the -=:
lconn->cn_flags & LSCONN_HANDSHAKE_DONE selects the short- vs long-header size. A 1-RTT (HETY_SHORT) packet scheduled before the handshake-done flip is costed with the long formula and, if sent after the flip, removed with the short formula → residual. There is no hook telling send_ctl that the flip changed the sizes of already-scheduled packets.
p->po_path->np_dcid.len — has a hook (lsquic_send_ctl_cidlen_change) but it assumes every scheduled packet carries the DCID (diff *= sc_n_scheduled); short-header packets without PO_CONN_ID do not, so the hook can itself over/under-adjust.
packno_bits (packet-number length) — also affects header size.
So += sz@schedule and -= sz@send use different values and do not cancel; the residual accumulates and, when the net is negative, the unsigned underflows.
Existing hint in the code: lsquic_send_ctl_cleanup keeps assert(0 == sc_n_scheduled) but has //assert(0 == ctl->sc_bytes_scheduled); commented out — i.e. the byte counter is already known not to reliably reach 0.
Suggested fix (PR to follow). Cache the exact size attributed at schedule time on the packet (a new po_acct_sz, mirroring how the unacked counter caches po_sent_sz) and subtract that cached value at remove, keeping it in lockstep in the few adjustment paths (incr_pack_sz, elide, update_for_resending, cidlen_change, rollback, and the gQUIC HELLO zero-pad in stream_write_to_packet_crypto). Schedule and remove then cancel by construction, regardless of any size-affecting state change between schedule and send.
This may be the same underlying bug as #301 ("after the client and server interact for a while, there is no traffic suddenly" — connection not tickable, send_ctl_can_send pinned against an exhausted cwnd) and #412 ("client stop sending data under pressure testing"), both of which match this symptom but were closed without an accounting root cause.
Summary.
struct lsquic_send_ctl::sc_bytes_scheduled(anunsigned) can drift away from the true sum of scheduled packet sizes and underflow to~UINT_MAX. Once it does,send_ctl_all_bytes_out() >= cwndis permanently true,send_ctl_can_send()returns 0 forever, the connection is never tickable/writeable, and its streams'on_writeis never dispatched — the connection wedges with an empty pipe (0 in-flight, 0 unacked).Observed at the wedge:
sc_n_scheduled == 0,sc_n_in_flight_all == 0, andsc_bytes_unacked_all == 0, yetsc_bytes_scheduled == 4294967295. Reproduces under high connection count with many short writes (we hit it at ~800+ concurrent connections with large per-stream transfers); it is timing-dependent, consistent with a small per-packet residual that accumulates.Versions. Reproduced on v4.7.1; the root cause is still present on
master(v4.8.0).Root cause.
sc_bytes_scheduledis maintained incrementally:send_ctl_sched_Xpend_commondoes+= packet_out_total_sz(p)at schedule-append, andsend_ctl_sched_removedoes-= packet_out_total_sz(p)at send-remove.But
packet_out_total_sz()→pf_packout_size()(ietf_v1_packout_size) is not a pure function of the packet — it reads mutable connection/path state that can change between the+=and the-=:lconn->cn_flags & LSCONN_HANDSHAKE_DONEselects the short- vs long-header size. A 1-RTT (HETY_SHORT) packet scheduled before the handshake-done flip is costed with the long formula and, if sent after the flip, removed with the short formula → residual. There is no hook tellingsend_ctlthat the flip changed the sizes of already-scheduled packets.p->po_path->np_dcid.len— has a hook (lsquic_send_ctl_cidlen_change) but it assumes every scheduled packet carries the DCID (diff *= sc_n_scheduled); short-header packets withoutPO_CONN_IDdo not, so the hook can itself over/under-adjust.packno_bits(packet-number length) — also affects header size.So
+= sz@scheduleand-= sz@senduse different values and do not cancel; the residual accumulates and, when the net is negative, theunsignedunderflows.Existing hint in the code:
lsquic_send_ctl_cleanupkeepsassert(0 == sc_n_scheduled)but has//assert(0 == ctl->sc_bytes_scheduled);commented out — i.e. the byte counter is already known not to reliably reach 0.Suggested fix (PR to follow). Cache the exact size attributed at schedule time on the packet (a new
po_acct_sz, mirroring how the unacked counter cachespo_sent_sz) and subtract that cached value at remove, keeping it in lockstep in the few adjustment paths (incr_pack_sz,elide,update_for_resending,cidlen_change,rollback, and the gQUIC HELLO zero-pad instream_write_to_packet_crypto). Schedule and remove then cancel by construction, regardless of any size-affecting state change between schedule and send.This may be the same underlying bug as #301 ("after the client and server interact for a while, there is no traffic suddenly" — connection not tickable,
send_ctl_can_sendpinned against an exhausted cwnd) and #412 ("client stop sending data under pressure testing"), both of which match this symptom but were closed without an accounting root cause.