Skip to content

GzipDecompressor: fix inflater memory leak#1134

Merged
pjfanning merged 4 commits into
apache:mainfrom
pjfanning:copilot/fix-inflater-memory-leak
Jul 8, 2026
Merged

GzipDecompressor: fix inflater memory leak#1134
pjfanning merged 4 commits into
apache:mainfrom
pjfanning:copilot/fix-inflater-memory-leak

Conversation

@pjfanning

@pjfanning pjfanning commented Jul 6, 2026

Copy link
Copy Markdown
Member

See #1133

GzipDecompressor allocates a new Inflater per gzip stream but did not explicitly release its native memory. Under collectors with relaxed off-heap reclamation such as ZGC, repeated request decompression can accumulate unreclaimed native buffers and eventually OOM.

  • Inflater lifecycle cleanup

    • Add explicit Inflater.end() cleanup for GzipDecompressor
    • Make cleanup idempotent so termination paths can safely converge
    • Run cleanup from stage shutdown (postStop) to cover normal completion, failure, and cancellation
  • Truncation handling

    • Release the inflater before failing truncated gzip streams
    • Preserve the existing Truncated GZIP stream behavior while ensuring native resources are not left behind
  • Targeted coverage

    • Add focused tests that verify the inflater is released on:
      • successful decode completion
      • early downstream cancellation
      • truncation failure
private def cleanupInflater(): Unit =
  if (!inflaterEnded) {
    inflaterEnded = true
    inflater.end()
  }

override def postStop(): Unit = cleanupInflater()

override def onTruncation(): Unit = {
  cleanupInflater()
  failStage(new ZipException("Truncated GZIP stream"))
}


private def cleanupInflater(): Unit =
if (!inflaterEnded) {
inflaterEnded = true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can mark it as null ? instead of an inflaterEnded

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the boolean to making inflater a var and then checking for nulls. The nulls approach makes cleanup harder.

    private def cleanupInflater(): Unit =
      if (!inflaterEnded) {
        inflaterEnded = true
        inflater.end()
      }

This is not thread-safe.

    private def cleanupInflater(): Unit =
      if (inflater != null) {
        inflater.end()
      }

This is thread-safe but introduces a local variable.

    private def cleanupInflater(): Unit = {
      val localInflater = inflater
      if (localInflater != null) {
        localInflater.end()
      }
   }

@pjfanning pjfanning changed the title WIP: fix inflater memory leak fix inflater memory leak Jul 6, 2026
@pjfanning pjfanning requested a review from jrudolph July 6, 2026 13:05
@pjfanning pjfanning mentioned this pull request Jul 6, 2026
@pjfanning pjfanning changed the title fix inflater memory leak GzipDecompressor: fix inflater memory leak Jul 6, 2026
Copilot AI added a commit to pjfanning/incubator-pekko-http that referenced this pull request Jul 6, 2026
Motivation:
DeflateDecompressor allocates an Inflater per stream but never explicitly
releases its native memory. Under GCs with relaxed off-heap reclamation
(e.g. ZGC), repeated request decompression can accumulate unreclaimed
native buffers and eventually OOM.

Modification:
- Add createInflater(noWrap) factory method to DeflateDecompressor to
  allow test injection of a custom Inflater
- Track the current inflater in a var within createLogic
- Add idempotent cleanupInflater() that calls inflater.end()
- Override postStop() to call cleanupInflater(), covering normal
  completion, failure, and cancellation
- Add focused tests in DeflateSpec verifying the inflater is released
  on successful decode, early cancellation, and truncation

Result:
The Inflater's native memory is reclaimed promptly when the stage
terminates, regardless of termination path.

Tests:
- Not run - sbt/builds skipped per user request to conserve credits

References:
Refs apache#1134
pjfanning added a commit that referenced this pull request Jul 7, 2026
* fix: DeflateDecompressor inflater memory leak

Motivation:
DeflateDecompressor allocates an Inflater per stream but never explicitly
releases its native memory. Under GCs with relaxed off-heap reclamation
(e.g. ZGC), repeated request decompression can accumulate unreclaimed
native buffers and eventually OOM.

Modification:
- Add createInflater(noWrap) factory method to DeflateDecompressor to
  allow test injection of a custom Inflater
- Track the current inflater in a var within createLogic
- Add idempotent cleanupInflater() that calls inflater.end()
- Override postStop() to call cleanupInflater(), covering normal
  completion, failure, and cancellation
- Add focused tests in DeflateSpec verifying the inflater is released
  on successful decode, early cancellation, and truncation

Result:
The Inflater's native memory is reclaimed promptly when the stage
terminates, regardless of termination path.

Tests:
- Not run - sbt/builds skipped per user request to conserve credits

References:
Refs #1134

* Update DeflateSpec.scala

* close existing inflater

* fix: await inflater.end() before asserting in cancellation test

The Sink.ignore future completes when take(1) sends Complete downstream,
but the Cancel signal that triggers postStop() (and thus end()) is a
separate actor message dispatched afterwards. This means awaitResult can
return before end() has been called, yielding endCalls = 0.

Add a CountDownLatch to TrackingInflater that is counted down inside
end(), and call inflater.awaitEnd() after awaitResult() in the
cancellation test to eliminate the race.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
@pjfanning

Copy link
Copy Markdown
Member Author

@He-Pin do you think is ok to merge?

pjfanning added a commit that referenced this pull request Jul 8, 2026
* fix: DeflateDecompressor inflater memory leak

Motivation:
DeflateDecompressor allocates an Inflater per stream but never explicitly
releases its native memory. Under GCs with relaxed off-heap reclamation
(e.g. ZGC), repeated request decompression can accumulate unreclaimed
native buffers and eventually OOM.

Modification:
- Add createInflater(noWrap) factory method to DeflateDecompressor to
  allow test injection of a custom Inflater
- Track the current inflater in a var within createLogic
- Add idempotent cleanupInflater() that calls inflater.end()
- Override postStop() to call cleanupInflater(), covering normal
  completion, failure, and cancellation
- Add focused tests in DeflateSpec verifying the inflater is released
  on successful decode, early cancellation, and truncation

Result:
The Inflater's native memory is reclaimed promptly when the stage
terminates, regardless of termination path.

Tests:
- Not run - sbt/builds skipped per user request to conserve credits

References:
Refs #1134

* Update DeflateSpec.scala

* close existing inflater

* fix: await inflater.end() before asserting in cancellation test

The Sink.ignore future completes when take(1) sends Complete downstream,
but the Cancel signal that triggers postStop() (and thus end()) is a
separate actor message dispatched afterwards. This means awaitResult can
return before end() has been called, yielding endCalls = 0.

Add a CountDownLatch to TrackingInflater that is counted down inside
end(), and call inflater.awaitEnd() after awaitResult() in the
cancellation test to eliminate the race.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

@He-Pin He-Pin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@pjfanning pjfanning merged commit a06ad57 into apache:main Jul 8, 2026
6 checks passed
@pjfanning pjfanning deleted the copilot/fix-inflater-memory-leak branch July 8, 2026 08:32
pjfanning added a commit that referenced this pull request Jul 10, 2026
* GzipDecompressor: fix inflater memory leak (#1134)

* Initial plan

* Apply remaining changes

* Update GzipSpec.scala

* Update GzipCompressor.scala

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

* Ensure deflater/inflater end() is called in DeflateCompressor and GzipCompressor (#1142)

* Ensure deflater/inflater end() is called in DeflateCompressor and GzipCompressor

- DeflateCompressor: add deflaterEnded flag + idempotent endDeflater() method;
  call endDeflater() in finishWithBuffer() instead of deflater.end() directly
- GzipDecompressor: add postStop() calling inflater.end() + add createInflater()
  factory method for testability
- StreamUtils.byteStringTransformer: add optional cleanup callback, call it in
  postStop() when the stage is stopped before onUpstreamFinish()
- Encoder.singleUseEncoderFlow: pass cleanup callback that calls endDeflater()
  on DeflateCompressor instances to handle stream cancellation/failure
- DeflateSpec: add tests for deflater cleanup on normal finish and cancellation
- GzipSpec: add tests for inflater cleanup on normal finish and cancellation,
  and deflater cleanup on normal finish and cancellation

* compile issues

* Add explicit cleanup() to Compressor/DeflateCompressor; GzipCompressor inherits it

* revert GzipCompressor changes

* Update GzipSpec.scala

* Update GzipSpec.scala

* Update GzipSpec.scala

* review comments

* Refactor onUpstreamFinish method for clarity

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

* compile issues

* test compile issue

* Update GzipCompressor.scala

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants