PyPI · graphql-core
GraphQL-core: Denial of service via comment tokens bypassing the parser's max_tokens limit
The parser's max_tokens limit — the documented defence against oversized GraphQL
documents — did not count comment tokens. A document padded with comments could
therefore pass a max_tokens check of any size while still allocating one Token
object per comment, all of which stay reachable for the lifetime of the parsed
document. A 38 MB comment-padded payload allocated roughly 92 MB of additional
resident memory with max_tokens=5 in force.
Lexer.lookahead() reads tokens one at a time and skips over comments, returning only
the first non-comment token. Each skipped comment is nevertheless fully tokenized,
allocated as a Token object, and linked into the persistent token list via
token.next / token.prev — the chain that powers source-location reporting, and
which is retained as long as the document is.
Parser.advance_lexer() incremented the token counter only for the tokens the lexer
actually returned. Comments were therefore invisible to the accounting, and
max_tokens bounded only the non-comment tokens.
The practical consequence is a gap between what users reasonably believe max_tokens
protects and what it did protect. The limit was not broken for ordinary documents —
real tokens were still capped correctly — so the shortfall was not observable without
specifically testing comment padding.
As with the companion issue, the payload compresses extremely well (# c\n repeated),
so body-size limits and WAF inspection applied to the compressed request do not see
its true size, while the framework decompresses it in memory before the parser is
reached.
import gc
import resource
from graphql import parse
# 100,000 comments, a 5-token query, and max_tokens=5 — accepted.
query = "# c\n" * 100_000 + "{ a }"
parse(query, max_tokens=5) # no error raised
# Memory amplification.
big = "# padding comment used to amplify memory per token\n" * 500_000 + "{ a }"
gc.collect()
before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
parse(big, max_tokens=5) # ~38 MB source
after = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(f"RSS delta: +{(after - before) / 1024:.0f} MB")
Measured on GraphQL-core 3.2.8 / CPython 3.12 / Linux x86_64: the 391 KB payload parsed in 127 ms, and the 38.1 MB payload produced an RSS delta of +92 MB — about 2.4× the source size, on top of the source string itself.
Denial of service through memory exhaustion. A modest number of concurrent requests can push a worker past a typical 256 MB or 512 MB container limit and trigger an OOM kill. No authentication is required, and no data is disclosed or modified.
Depth limiters and complexity analyzers do not help: they run on the AST after parsing has already succeeded, and the document contains only five real tokens.
Fixed in 3.2.12 and 3.3.0rc1.
Parser.advance_lexer now counts every token consumed, including the comment tokens
the lexer skips between the previously focused token and the new one, and including
the advance onto EOF so that trailing comments are covered. max_tokens therefore now
bounds every Token the parser allocates.
Note the behaviour change: comments now count toward max_tokens. A
comment-heavy document that was previously accepted may now be rejected. If you parse
documents containing substantial comments — SDL files with documentation comments, for
instance — and you set max_tokens, you may need to raise the limit. The
token_count attribute of a parsed document likewise now includes comments.
If you cannot upgrade, apply a request body-size limit to the decompressed body
before the document reaches parse().
Such a limit is worth keeping after upgrading as well: max_tokens is one guard among
several and bounds the token count, not the overall cost of a request.
Reported by @BZHunt, who also reviewed the fixes before release. In memory of ZaX, who was the driving force behind this discovery.
Is your project exposed to this? Stateward checks every dependency on every pull request and flags it only if your code actually reaches it.
Check my repoSources: CISA KEV (public domain), OSV.dev & GitHub Advisory Database (CC-BY-4.0), FIRST EPSS, NVD/CWE (public domain). Served live from the Stateward advisory database.