Buffering in Standard StreamsI found it difficult to understand what type of buffering is applied to standard streams. To clarify this, let's consider a Unix shell pipeline:
```bash
$ command1 | command2
```
In this case, the shell forks off two processes and connects them using a pipe, as depicted in the diagram below. Note that there are three buffers involved in the connection.
The kernel buffer, created by the pipe system call from the shell, is sized based on the page size for the system. While we have no direct control over the kernel buffer size or operation, this is not an issue for us because it will efficiently copy data as soon as it's received (at least on Linux).
[Update: The Linux pipe buffers have changed to circular buffers (16 x 4KiB), and there is a proposed patch to make the size dynamic.]
The other buffers represented are associated with the standard streams, and they are only allocated on first use for efficiency purposes. This means that if you don't immediately write data to these streams, they won't consume any memory until you actually start writing.
The C library (libc) automatically creates three standard streams for nearly all unix programs at the start of program execution: stdin, stdout, and stderr. New streams can also be created to connect to various types of input/output devices such as files, sockets, pipes, and more. To control how data is read and written from these buffers, one can adjust both the buffer sizes and modes (Unbuffered, Buffered, Line Buffered). This allows for a high degree of flexibility when working with these standard streams.
To determine the characteristics of the buffering applied to standard streams in my system, I wrote a program that revealed some interesting findings. The following are the default buffering modes for each stream:
- **stdin** is buffered (line buffering doesn't affect stdin).
- **stdout** is buffered (line buffering is enabled if connected to a terminal).
- **stderr** is unbuffered.
When it comes to default buffer sizes, there are a few factors to consider. The buffer size directly affects whether a particular mode is used or not. The kernel uses page size (4096 bytes on my system) as the default size for buffers. If stdin or stdout are connected to a terminal, then the default size is set to 1024 bytes; otherwise, it is set to 4096 bytes. This means that when these streams are used without any connection to a terminal, they will generally have larger buffer sizes than when they are connected to a terminal.
One common problem that can arise when working with stdio output buffering is encountering issues with the timing of output. For example, if you write data to stdout but do not flush it immediately, this data may not be sent to the console until after the program has finished executing. This can lead to unexpected behavior or errors, especially in cases where other parts of your program depend on this output being displayed immediately. Therefore, it is important to ensure that you properly flush any output streams before程序结束运行， even if you don't need the output right away.
Consider a situation where the data source is generating intermittent output, and you want to both observe the data in its original form and filter it. For instance, you might be interested in filtering the output of tools like `tcpdump -l` or `tail -f`. It's important to note that certain filters, such as `sort`, require internal buffering and cannot be used in this application.
To illustrate this, let's take a concrete example involving a pipeline that displays IP addresses accessing a website while filtering out consecutive accesses from a particular IP. The following command demonstrates this:
```bash
tail -f access.log | cut -d' ' -f1 | uniq
```
However, there is a problem with this approach. One of the issues is that the hosts won't appear as they do in the log file due to the automatic stdio buffering applied by the `libc`. Given these constraints, applying the example pipeline above would result in buffering, as depicted in the following diagram.
The highlighted buffer presents a problem because it is connected to a pipe, which means it will automatically buffer up data into 4096 byte chunks before sending to `uniq`.
The following is the content of the original text:
Note tail's stdout buffer would also have this problem, but
tail -f
calls fflush
on the stdout stream when new data is received to alleviate this (as do
tcpdump -l
,
grep --line-buffered
and
sed --unbuffered
for example). Note also that uniq's stdout buffer is connected to a terminal and so will be automatically
flushed when a new line is written to it which is fine for our needs.
stdio input buffering problemsBuffering on stdin like stdout is used (to coalesce reads) for efficiency. One would have more control with byte by byte reads, but that would not be practical. Consider the following issue: $ printf "one
two
three
" | (sed 1q ; sed 1q ; sed 1q) one As you can see the first sed process reads all the data, starving the rest. Note if one could set the stdin buffer to line buffering mode it still would have no affect as that only controls when output is flushed. Reading lines from stdin is a common requirement but implemented above the stdin buffer.
SH clients, by default, will block if stdin is not closed when the remote command is running. This can be problematic because it may prevent the application from being responsive, especially in cases where graphical applications are used.
One way to avoid this issue is to disable buffering on stdin for the SSH process itself. However, this can lead to unexpected behavior if you have other processes that need to read stdin from the SSH client. For example, consider the following scenario:
```bash
$ printf "one
two
three
" | ssh localhost printf "zero\
" ; cat zero one two three
```
In this case, the remote `printf` command does not read stdin, but the local `ssh` client does not realize this and reads data on its behalf. To tell the `ssh` client that the remote command does not require any input, you can use the `-n` option:
```bash
$ printf "one
two
three
" | ssh -n localhost printf "zero\
" ; cat zero one two three
```
However, there is a common issue with this approach. If you know that the remote command will not need any input from stdin and you want to put the SSH client in the background, the SSH client will block trying to read from stdin, resulting in a stall in your application. To work around this issue, you can redirect stdin to `/dev/null` (on Linux systems) or `NUL` (on Windows systems):
```bash
$ printf "one
two
three
" | ssh localhost -t printf "zero\
" >/dev/null ; cat zero one two three
```
One can tell SSH to ignore standard input (stdin) and fork to the background when appropriate with the `-f` option. For example, running the command `ssh -fY localhost xterm` would achieve this.
Programmatically, one can bypass buffers by using read/write operations directly, but this approach is generally inefficient in most cases. Another way to set both the buffering mode and buffer size used is through the `setvbuf` library call, as demonstrated in my example program. However, it should be noted that changing the buffering for a stream may have unexpected consequences. For instance, glibc (version 2.3.5 or later) performs a `read(blksize)` after every `fseek()` if buffering is enabled. Currently, there is no direct way to control the buffering mode or buffer size of existing applications.
To address this issue, one can employ a hack where the unbuffer script, which comes as standard with "expect", is used to trick "cut" (as shown above) into believing that it is connected to a terminal. This technique relies on what libc does to manage buffering.
In conclusion, while SSH offers several options to control buffering behavior, it may not always be possible to achieve desirable results without resorting to more complex techniques or third-party libraries.
## Buffer Control with stdio
In Linux, you can use the `stdbuf` command in combination with a shell function to change the output buffering of standard input and output. However, there are some limitations on this method:
- You can only control whether line buffering is used or fully buffered mode is used.
- There are caveats associated with using this hack, which are documented in the man page for `unbuffer`.
- Additionally, there is an increased possibility of running out of ptys on the system when using this approach.
[Update] (Nov 2008)
Another method for controlling buffering was discovered by Brian Dessent, which uses `LD_PRELOAD` to modify the output buffering of the C library. This method was described in detail by Jim Meyering.
[Update] (Aug 2009)
We made the `LD_PRELOAD` method above easily available in coreutils 7.5 with the introduction of a new `stdbuf` command. This command works like this: tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq. Note that you should use `stdbuf -o0` if your data is not line oriented. For more information, please see the man page for `stdbuf`, or the `stdbuf info` manual.
[Update] (Dec 2012)
I've noticed that `stdbuf` has been added to FreeBSD 9.1 as well.
The only way to control the buffering mode and size of input and output streams is through environment variables. The format for these variables is: BUF_X_=Y, where X represents the stream (stdin, stdout, stderr, etc.) and Y represents the desired buffering mode or size. For example, to configure standard input (stdin) to use line-buffered mode with a buffer size of 1KB, you can set the environment variable as follows:
```bash
BUF_0_=1
```
Similarly, to control stdout's unbuffered mode, you would use:
```bash
BUF_1_=0
```
To control stderr's buffered mode with a size of 512 bytes, you would use:
```bash
BUF_2_=512
```
If you wanted to combine these settings in a single command, you could do so using a pipe. For instance, if you wanted to tail the contents of an access.log file while filtering and removing duplicate IP addresses from the output, you could run the following command:
```bash
tail -f access.log | cut -d' ' -f1 | uniq | BUF_1_=1 sort | BUF_0_=1
```
This pipeline would work well if it were added to the coreutils package as part of regular updates to the software. However, it was suggested that adding this functionality to libc might introduce potential security risks and could disrupt applications that rely on specific buffering setups. Additionally, since there is no straightforward way to map file descriptor numbers to stream pointers outside of libc, implementing this logic in applications may be more challenging. Despite these concerns, here is a patch against the cut.c file in coreutils-0.94 that demonstrates how such functionality could potentially be integrated into the program:
Many thanks to Bob Proulx for providing valuable feedback and ideas to enhance this page.
