GzipDecompressor: fix inflater memory leak#1134
Merged
pjfanning merged 4 commits intoJul 8, 2026
Merged
Conversation
He-Pin
reviewed
Jul 6, 2026
|
|
||
| private def cleanupInflater(): Unit = | ||
| if (!inflaterEnded) { | ||
| inflaterEnded = true |
Member
There was a problem hiding this comment.
Maybe we can mark it as null ? instead of an inflaterEnded
Member
Author
There was a problem hiding this comment.
I prefer the boolean to making inflater a var and then checking for nulls. The nulls approach makes cleanup harder.
private def cleanupInflater(): Unit =
if (!inflaterEnded) {
inflaterEnded = true
inflater.end()
}
This is not thread-safe.
private def cleanupInflater(): Unit =
if (inflater != null) {
inflater.end()
}
This is thread-safe but introduces a local variable.
private def cleanupInflater(): Unit = {
val localInflater = inflater
if (localInflater != null) {
localInflater.end()
}
}
Copilot AI
added a commit
to pjfanning/incubator-pekko-http
that referenced
this pull request
Jul 6, 2026
Motivation: DeflateDecompressor allocates an Inflater per stream but never explicitly releases its native memory. Under GCs with relaxed off-heap reclamation (e.g. ZGC), repeated request decompression can accumulate unreclaimed native buffers and eventually OOM. Modification: - Add createInflater(noWrap) factory method to DeflateDecompressor to allow test injection of a custom Inflater - Track the current inflater in a var within createLogic - Add idempotent cleanupInflater() that calls inflater.end() - Override postStop() to call cleanupInflater(), covering normal completion, failure, and cancellation - Add focused tests in DeflateSpec verifying the inflater is released on successful decode, early cancellation, and truncation Result: The Inflater's native memory is reclaimed promptly when the stage terminates, regardless of termination path. Tests: - Not run - sbt/builds skipped per user request to conserve credits References: Refs apache#1134
pjfanning
added a commit
that referenced
this pull request
Jul 7, 2026
* fix: DeflateDecompressor inflater memory leak Motivation: DeflateDecompressor allocates an Inflater per stream but never explicitly releases its native memory. Under GCs with relaxed off-heap reclamation (e.g. ZGC), repeated request decompression can accumulate unreclaimed native buffers and eventually OOM. Modification: - Add createInflater(noWrap) factory method to DeflateDecompressor to allow test injection of a custom Inflater - Track the current inflater in a var within createLogic - Add idempotent cleanupInflater() that calls inflater.end() - Override postStop() to call cleanupInflater(), covering normal completion, failure, and cancellation - Add focused tests in DeflateSpec verifying the inflater is released on successful decode, early cancellation, and truncation Result: The Inflater's native memory is reclaimed promptly when the stage terminates, regardless of termination path. Tests: - Not run - sbt/builds skipped per user request to conserve credits References: Refs #1134 * Update DeflateSpec.scala * close existing inflater * fix: await inflater.end() before asserting in cancellation test The Sink.ignore future completes when take(1) sends Complete downstream, but the Cancel signal that triggers postStop() (and thus end()) is a separate actor message dispatched afterwards. This means awaitResult can return before end() has been called, yielding endCalls = 0. Add a CountDownLatch to TrackingInflater that is counted down inside end(), and call inflater.awaitEnd() after awaitResult() in the cancellation test to eliminate the race. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Member
Author
|
@He-Pin do you think is ok to merge? |
pjfanning
added a commit
that referenced
this pull request
Jul 8, 2026
* fix: DeflateDecompressor inflater memory leak Motivation: DeflateDecompressor allocates an Inflater per stream but never explicitly releases its native memory. Under GCs with relaxed off-heap reclamation (e.g. ZGC), repeated request decompression can accumulate unreclaimed native buffers and eventually OOM. Modification: - Add createInflater(noWrap) factory method to DeflateDecompressor to allow test injection of a custom Inflater - Track the current inflater in a var within createLogic - Add idempotent cleanupInflater() that calls inflater.end() - Override postStop() to call cleanupInflater(), covering normal completion, failure, and cancellation - Add focused tests in DeflateSpec verifying the inflater is released on successful decode, early cancellation, and truncation Result: The Inflater's native memory is reclaimed promptly when the stage terminates, regardless of termination path. Tests: - Not run - sbt/builds skipped per user request to conserve credits References: Refs #1134 * Update DeflateSpec.scala * close existing inflater * fix: await inflater.end() before asserting in cancellation test The Sink.ignore future completes when take(1) sends Complete downstream, but the Cancel signal that triggers postStop() (and thus end()) is a separate actor message dispatched afterwards. This means awaitResult can return before end() has been called, yielding endCalls = 0. Add a CountDownLatch to TrackingInflater that is counted down inside end(), and call inflater.awaitEnd() after awaitResult() in the cancellation test to eliminate the race. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
pjfanning
added a commit
that referenced
this pull request
Jul 10, 2026
* GzipDecompressor: fix inflater memory leak (#1134) * Initial plan * Apply remaining changes * Update GzipSpec.scala * Update GzipCompressor.scala --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> * Ensure deflater/inflater end() is called in DeflateCompressor and GzipCompressor (#1142) * Ensure deflater/inflater end() is called in DeflateCompressor and GzipCompressor - DeflateCompressor: add deflaterEnded flag + idempotent endDeflater() method; call endDeflater() in finishWithBuffer() instead of deflater.end() directly - GzipDecompressor: add postStop() calling inflater.end() + add createInflater() factory method for testability - StreamUtils.byteStringTransformer: add optional cleanup callback, call it in postStop() when the stage is stopped before onUpstreamFinish() - Encoder.singleUseEncoderFlow: pass cleanup callback that calls endDeflater() on DeflateCompressor instances to handle stream cancellation/failure - DeflateSpec: add tests for deflater cleanup on normal finish and cancellation - GzipSpec: add tests for inflater cleanup on normal finish and cancellation, and deflater cleanup on normal finish and cancellation * compile issues * Add explicit cleanup() to Compressor/DeflateCompressor; GzipCompressor inherits it * revert GzipCompressor changes * Update GzipSpec.scala * Update GzipSpec.scala * Update GzipSpec.scala * review comments * Refactor onUpstreamFinish method for clarity --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> * compile issues * test compile issue * Update GzipCompressor.scala --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See #1133
GzipDecompressorallocates a newInflaterper gzip stream but did not explicitly release its native memory. Under collectors with relaxed off-heap reclamation such as ZGC, repeated request decompression can accumulate unreclaimed native buffers and eventually OOM.Inflater lifecycle cleanup
Inflater.end()cleanup forGzipDecompressorpostStop) to cover normal completion, failure, and cancellationTruncation handling
Truncated GZIP streambehavior while ensuring native resources are not left behindTargeted coverage