| Age | Commit message (Collapse) | Author |
|---|
|
Now it's based on calling fillMore rather than an illegal aliased stream
into the Reader buffer.
This commit also includes a disambiguation block inspired by #25162. If
`StreamTooLong` was added to `RebaseError` then this logic could be
replaced by removing the exit condition from the while loop. That error
code would represent when `buffer` capacity is too small for an
operation, replacing the current use of asserts.
|
|
Fix `takeDelimiter` and `takeDelimiterExclusive` tossing too many bytes
(#25132)
Also add/improve test coverage for all delimiter and sentinel methods,
update usages of `takeDelimiterExclusive` to not rely on the fixed bug,
tweak a handful of doc comments, and slightly simplify some logic.
I have not fixed #24950 in this commit because I am a little less
certain about the appropriate solution there.
Resolves: #25132
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
|
|
return 0
Previously, the logic in peekDelimiterInclusive (when the delimiter was not found in the existing buffer) used the `n` returned from `r.vtable.stream` as the length of the slice to check, but it's valid for `vtable.stream` implementations to return 0 if they wrote to the buffer instead of `w`. In that scenario, the `indexOfScalarPos` would be given a 0-length slice so it would never be able to find the delimiter.
This commit changes the logic to assume that `r.vtable.stream` can both:
- return 0, and
- modify seek/end (i.e. it's also valid for a `vtable.stream` implementation to rebase)
Also introduces `std.testing.ReaderIndirect` which helps in being able to test against Reader implementations that return 0 from `stream`/`readVec`
Fixes #25428
|
|
If `r.end` is updated in the `stream` implementation, then it's possible that `r.end += ...` will behave unexpectedly. What seems to happen is that it reverts back to its value before the function call and then the increment happens. Here's a reproduction:
```zig
test "fill when stream modifies `end` and returns 0" {
var buf: [3]u8 = undefined;
var zero_reader = infiniteZeroes(&buf);
_ = try zero_reader.fill(1);
try std.testing.expectEqual(buf.len, zero_reader.end);
}
pub fn infiniteZeroes(buf: []u8) std.Io.Reader {
return .{
.vtable = &.{
.stream = stream,
},
.buffer = buf,
.end = 0,
.seek = 0,
};
}
fn stream(r: *std.Io.Reader, _: *std.Io.Writer, _: std.Io.Limit) std.Io.Reader.StreamError!usize {
@memset(r.buffer[r.seek..], 0);
r.end = r.buffer.len;
return 0;
}
```
When `fill` is called, it will call into `vtable.readVec` which in this case is `defaultReadVec`. In `defaultReadVec`:
- Before the `r.end += r.vtable.stream` line, `r.end` will be 0
- In `r.vtable.stream`, `r.end` is modified to 3 and it returns 0
- After the `r.end += r.vtable.stream` line, `r.end` will be 0 instead of the expected 3
Separating the `r.end += stream();` into two lines fixes the problem (and this separation is done elsewhere in `Reader` so it seems possible that this class of bug has been encountered before).
Potentially related issues:
- https://github.com/ziglang/zig/issues/4021
- https://github.com/ziglang/zig/issues/12064
|
|
std.Io: delete GenericReader, AnyReader, FixedBufferStream; and related API breakage
|
|
This reverts commit 530cc2c1111699d9d02ad9ebef94efa6b99f5205.
The compiler bug has been fixed.
|
|
Also, breaking API changes to:
* std.fs.Dir.readFileAlloc
* std.fs.Dir.readFileAllocOptions
|
|
and delete deprecated alias std.io
|
|
There was some bug in this branch, and rather than diagnosing it, I
fully finished porting over to new Reader API. Did it fix the bug?
|
|
tracked by #25067 and I already have a fix cooking in another branch
|
|
perhaps these APIs have the defaults backwards, eh?
|
|
|
|
Before this commit, calling appendRemaining with an ArrayList where list.items.len != list.capacity could result in illegal behavior if the Writer.Allocating resized the list during the appendRemaining call.
Fixes #25057
|
|
* extend std.Io.Reader.peekDelimiterExclusive test to repeat successful end-of-stream path (fails)
* fix std.Io.Reader.peekDelimiterExclusive to not advance seek position in successful end-of-stream path
|
|
std: more reliable HTTP and TLS networking
|
|
This also makes initStreaming preemptively disable file size checking.
|
|
* std.Io.Reader: fix confused semantics of rebase. Before it was
ambiguous whether it was supposed to be based on end or seek. Now it
is clearly based on seek, with an added assertion for clarity.
* std.crypto.tls.Client: fix panic due to not enough buffer size
available. Also, avoid unnecessary rebasing.
* std.http.Reader: introduce max_head_len to limit HTTP header length.
This prevents crash in underlying reader which may require a minimum
buffer length.
* std.http.Client: choose better buffer sizes for streams and TLS
client. Crucially, the buffer shared by HTTP reader and TLS client
needs to be big enough for all http headers *and* the max TLS record
size. Bump HTTP header size default from 4K to 8K.
fixes #24872
I have noticed however that there are still fetch problems
|
|
* std.Io.Reader: appendRemaining no longer supports alignment and has
different rules about how exceeding limit. Fixed bug where it would
return success instead of error.StreamTooLong like it was supposed to.
* std.Io.Reader: simplify appendRemaining and appendRemainingUnlimited
to be implemented based on std.Io.Writer.Allocating
* std.Io.Writer: introduce unreachableRebase
* std.Io.Writer: remove minimum_unused_capacity from Allocating. maybe
that flexibility could have been handy, but let's see if anyone
actually needs it. The field is redundant with the superlinear growth
of ArrayList capacity.
* std.Io.Writer: growingRebase also ensures total capacity on the
preserve parameter, making it no longer necessary to do
ensureTotalCapacity at the usage site of decompression streams.
* std.compress.flate.Decompress: fix rebase not taking into account seek
* std.compress.zstd.Decompress: split into "direct" and "indirect" usage
patterns depending on whether a buffer is provided to init, matching
how flate works. Remove some overzealous asserts that prevented buffer
expansion from within rebase implementation.
* std.zig: fix readSourceFileToAlloc returning an overaligned slice
which was difficult to free correctly.
fixes #24608
|
|
fetch, tls, and http fixes
|
|
Running tar.pipeToFileSystem compressed_mingw_includes.tar file from #24732
finishes in infinite loop calling defaultReadVec with:
r.seek = 1024
r.end = 1024
r.buffer.len = 1024
first.len = 512
that combination calls vtable.stream with 0 capacity writer and loops
forever.
Comment is to use whichever has larger capacity, and this fix reflects that.
|
|
simplifies the logic & makes it respect limit
|
|
Now it avoids mutating `r` unnecessarily, allowing the `ending` Reader
to work.
|
|
|
|
zig fmt: canonicalize nested cast builtin order
|
|
readVec has two updated responsibilities:
1. it must respect any existing already buffered data.
2. it must write to the buffer if data is empty
|
|
|
|
because it may be used as a ring buffer
|
|
this is not necessary according to zig language, but works around a flaw
in the C backend
|
|
|
|
|
|
|
|
|
|
|
|
simplifies and fixes things
addresses a subset of #24608
|
|
addresses only one usage pattern in #24608
|
|
This eliminates a footgun and special case handling with fixed buffers,
as well as allowing decompression streams to keep a window in the output
buffer.
|
|
|
|
This passes tests but it doesn't provide as big a window size as is
required to decompress larger streams.
The next commit in this branch will work towards that, without
introducing an additional buffer.
|
|
|
|
|
|
closes #24516
|
|
std.zig: finish updating to new I/O API
|
|
tried to be too clever, wrote bad code
|
|
with `.fixed("")` you should still be able to do `fill(1)` and have it
return error.EndOfStream.
|
|
closes #24443
|
|
|
|
ensure that it issues a stream call that includes the buffer to detect
the end when needed, but otherwise does not offer Reader buffer to
append directly to the list.
|
|
it calls readVec which is a higher level function than was expected in
the previous implementation
|
|
closes #24418
|
|
std.Io: handle packed structs better
|
