-
Notifications
You must be signed in to change notification settings - Fork 524
Remove redundant BufReaders for decompressing tar archives #6317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6317 +/- ##
=========================================
- Coverage 82.8% 82.8% -0.1%
=========================================
Files 847 847
Lines 379488 379486 -2
=========================================
- Hits 314349 314309 -40
- Misses 65139 65177 +38 🚀 New features to boost your workflow:
|
Thanks for the link to where zstd creates its own BufReader. Can you also provide links for gzip and lz4 too, since this PR is changing them as well. |
Sure, I'm also switching bzip2 decoder as it has appropriate wrapper. Updated description. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. I think you'll need to rebase to pull in newer commits that fix the CI.
f9eacaf
to
57a86cc
Compare
57a86cc
to
71e2436
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also backport this to v2.3. We'll need to wait for #6430 to land, and then please add the "v2.3" label to kickoff the backport. |
Problem
ZSTD
Decoder::new
creates buf reader with tuned size already, other decoders handle buffering too:BzDecoder
https://docs.rs/bzip2/latest/src/bzip2/read.rs.html#87-91GzDecoder
https://docs.rs/flate2/latest/src/flate2/gz/read.rs.html#143-147lz4::Decoder
https://docs.rs/lz4/1.28.1/src/lz4/decoder.rs.html#33-43 (it uses a vec buffer directly)zstd::Decoder
https://github.com/gyscos/zstd-rs/blob/229054099aa73f7e861762f687d7e07cac1d9b3b/src/stream/read/mod.rs#L29In those cases wrapping reader with
BufReader
is counter-productive or unnecessary (bzip2 provides bothbufread
andread
decoders)Summary of Changes
Remove redundant BufReaders for decompressing tar archives when underlying decompressor is already doing buffering.