How does a computer monitor work, OS-side?

[King Author]

I was thinking about the concept of creating an operating system the other day. Just for fun. I mean, don't get me wrong, I'd love to actually create one (not general-purpose, not for the public, just for me, just a simple, bare-bones, only-needs-to-run-the-programs-I-use OS), but I know damn well I don't have the tenacity and patience. Doesn't mean I can't ponder how I'd go about making one, though.

Anyway, rather than read some sort of "how to make an OS" thing, which I wouldn't understand and which I probably wouldn't like even if I would, I decided to just dream up how I might make an OS from scratch, using nothing but my human-level capacity for rational thought, extrapolation and critical thinking.

One thing I was thinking was, "how does an OS even send data to a monitor so it can display stuff in the first place?"

I came up with a way I'd do it (which, since I'm going off nothing, might be total hogwash, but that's not what this topic is about so I'll keep my dumb ideas to myself) but then I thought, "wait...I really, really have no idea how OSes actually send image data to monitors."

So I googled, but all I can come up with is stupid general-public crap like this HSW article and this eHow thing. I know what an arfing "resolution" is, I know the difference between LCD and CRT, that's not the info I'm looking for. I'm talking about, when those articles mention "the monitor interprets signals from the computer," what are those signals? What is it that the OS is actually doing? Sending RGB values to each pixel?

[jareds]

If you want to know about the signal between a computer and a monitor, the best strategy is not to look up information about monitors, but information about the particular signal standard you are interested in.

For example, let's take DVI. According to the article, red, green, and blue are each transmitted on their own pair of wires, 8 bits per color per pixel, in an uncompressed rasterized fashion. That is, the top line gets sent, left to right, pixel by pixel, then the next line, and so on, with the RGB values lining up because there is no compression. Each channel of data is encoded according to TMDS, which is designed to take into account various physical realities of high-frequency electrical signals. You could also look up VGA and HDMI for similar information.

However, the OS is far removed from sending the data to the monitor. That is handled by dedicated hardware in the video card. Roughly speaking, the video card driver gives the video card commands to draw the image. (In many 2D cases it will just be giving the video card the complete pixel by pixel image, while in 3D cases the video card will be doing a lot of the work itself.) In any event, I am reasonably confident that the video card ultimately places the pixel data for each frame in a buffer, from which a tiny piece of dedicated circuitry sends the signal to the monitor.

In general, it is considered good for dedicated hardware to handle I/O work under the control of the OS. Most of the OS's I/O work is just telling dedicated hardware what to do, and that hardware then knows the low-level protocol for external communication. As well as offloading work from the CPU, dedicated hardware means that a delay on the OS's part won't necessarily disrupt communication with the peripheral. For another example of this, take Ethernet. The OS tells the NIC where to put incoming packets, and the NIC signals the OS when it's placed a packet there, but if the OS doesn't do anything with the first packet right away, the NIC can still receive more packets until it runs out of space allocated to it for incoming packets.

[Carnildo]

For the most part, the OS doesn't talk to the monitor. It talks to the graphics card, and the card takes care of the fiddly details of talking to the monitor.

If you're writing your own OS, your best bet is to look for old "programming for MS-DOS" information. DOS doesn't do monitors, so every program had to have its own video driver. You should be able to find out how to set and use various text modes, and if you want graphics, you can use various VGA and VESA modes. Hardware acceleration (either 2D or 3D) is a much harder thing to do, and unlike the text, VGA, and VESA modes, it varies from card to card, so I wouldn't recommend trying it for a toy OS.

[Jplus]

In an attempt to add to jareds' explanation, here's what I believe to be the complete stack of operation between the application and the screen. I'm not exactly an expert so be warned that it may contain mistakes. You'll probably want to look up some of the words I'm going to mention.

1. The application invokes the toolkit API (for example Qt or GTK+) in order to have something drawn (for example a button), at a position relative to something else that has already been drawn (for example the window).
2. The toolkit code translates the request into lower level instructions, which are either sent directly to the OS API or to an intermediate API (like Direct3D) which in turn will invoke the OS API. This involves interprocess communication. In most cases, an OS service (such as the WindowServer in Mac OS X) will calculate the absolute position of the image data that are to be drawn.
3. The absolutely positioned drawing request is sent to the driver of the GPU. In simple cases this may be just pixel-for-pixel data, while in a typical 3D game it will be a parametric description of something that has to be projected in perspective (this might still contain bits of pixel-for-pixel data, e.g. textures). The driver translates the request into something that is digestible for the firmware of the GPU. In other words, this is just a layer of indirection that makes it possible for an OS to support any device and vice versa.
4. The firmware and hardware of the GPU together make it possible that the instructions are executed (note that support for an intermediate API like Direct3D and OpenGL often goes down all the way to the firmware level). The intermediate result will be some sort of raw pixel data that is specific to the GPU type.
5. A peripheral controller on the graphics card translates the intermediate result to some standard interface such as DVI (so that's just another encoding of raw pixel data). From there on the signal is sent to the screen, which will finally decode and display it. How the last step works is again totally specific to the screen type (tip: AMOLED and IMOD are very interesting technologies).

[Derek]

If all you want is a command prompt, you can somehow put the system in a text mode (there might be a couple of these, I'm not sure), and then you write to the monitor by just changing bytes at a specific location. When I was writing an OS for class, there were two bytes per pixel, one for the character and one for the foreground and background colors. We didn't have to write code to get this mode though. Actually I think the system starts in this mode, because I know the bootloader can write to the screen too.

If you want simple graphics, there are some other modes you can put the system in. Again you draw to the screen by writing to bytes, but this time the bytes represent pixels. I think they're based on a palette (which I believe you can modify, but I don't know how). When I was first learning programming with QBASIC, I used screen 13 to write games. It's 320x200 with 256 colors, close to an even 64K of memory. I learned a little bit of DOS-based x86 assembly, including writing to this screen. I think getting into screen 13 was a syscall to DOS though, so I don't know how it works on a low level.

This is all my understanding of it anyways. For more advanced graphics, you'll have to talk to the video card.

[Sc4Freak]

If all you want is a command prompt, you can somehow put the system in a text mode (there might be a couple of these, I'm not sure), and then you write to the monitor by just changing bytes at a specific location. When I was writing an OS for class, there were two bytes per pixel, one for the character and one for the foreground and background colors. We didn't have to write code to get this mode though. Actually I think the system starts in this mode, because I know the bootloader can write to the screen too.

If you want simple graphics, there are some other modes you can put the system in. Again you draw to the screen by writing to bytes, but this time the bytes represent pixels. I think they're based on a palette (which I believe you can modify, but I don't know how). When I was first learning programming with QBASIC, I used screen 13 to write games. It's 320x200 with 256 colors, close to an even 64K of memory. I learned a little bit of DOS-based x86 assembly, including writing to this screen. I think getting into screen 13 was a syscall to DOS though, so I don't know how it works on a low level.

This is all my understanding of it anyways. For more advanced graphics, you'll have to talk to the video card.

I wonder how many of those old-school trickses will even work any more, given that we're slowly but surely moving away from BIOS to UEFI.

[jareds]

I wonder how many of those old-school trickses will even work any more, given that we're slowly but surely moving away from BIOS to UEFI.

VGA is really a video card standard--the BIOS calls are just a small wrapper, and in fact it was pretty common to use the BIOS only to switch video modes, because making a BIOS call to draw each pixel would be slow--so using VGA without the BIOS at all wouldn't be a big deal. Pretty much any PC video card made in the last 20 years will support VGA, and I will go ahead and predict that they will continue to do so at least until desktop PCs without separately purchasable components such as video cards (or motherboards with integrated video) are no longer common. Removing it gains the vendor nothing (the fraction of the die supporting VGA in a modern GPU must be insignificant), but potentially upsets customers who want to use Linux/BSD/whatever in text mode, or just happen to be using an OS that falls back to VGA text mode with a video driver problem.

[LikwidCirkel]

At the lowest software level, you end up with a couple sections of memory for a "frame buffer" which some dedicated hardware will transparently draw this memory region to the screen.

This doesn't change whether you're using VGA/DVI/HDMI. The graphics hardware will deal with driving the actual signals.

So - as an extreme simplification, ignoring the GPU, so we have only DACs/drivers and a memory frame buffer, two things are required by the OS.

1) Configure the hardware to output at a particular resolution and format. This involves writing system registers related to the graphics hardware, and will be handled by drivers from the video device vendor.

2) Interact with the frame buffer, or swap/flip buffers as necessary. A very low level system call might have a "queue buffer" call, which passes a memory region from user space to be drawn (output on the wire) at the next refresh, and a corresponding "dequeue buffer" call which will return the buffer to the user. It's common practice to have two or three buffers so that the user can work on the next buffer while one is currently rendering to the screen. These are blocking calls, so the program will be naturally throttled to process exactly one buffer at each refresh. The OS will have to use interrupts provided by the video hardware to do this.

GPUs will provide some advanced functionality to indirectly write to frame buffers (through other commands), or selectively update certain regions and things like that.

The actual content of the memory regions could be in a few different formats - the most common are probably 24-bit RGB, 32-bit RGB-Alpha, where the extra 8 bits is alpha or is ignored. YCbCr 4:2:2 16 bit-per-pixel is also common, as that's the colour space which consumer video content works in.

[troyp]

It's common practice to have two or three buffers so that the user can work on the next buffer while one is currently rendering to the screen. These are blocking calls, so the program will be naturally throttled to process exactly one buffer at each refresh.

Why is that? It seems (naively) like the same buffer could be repeatedly rendered while a user writes to it willy-nilly.

[Xanthir]

Because that's a naive view. ^_^ It takes time to send a buffer through the graphics system; while that's happening, you don't want the buffer changing. In some systems, that might result in inconsistent state (only the latter half of a poly getting drawn, because the upper half of the buffer was already drawn), while in others this may simply result in lost data because the buffer is simply *unavailable* while the graphics system is using it.

So, double-buffering lets you deal with the graphics-system delay while still pretending to the user that the canvas is always available for writes.

[troyp]

Hmm...I did realize when I asked that the buffer might change as it was being read, but I was thinking it wouldn't be a problem. I think my implicit reasoning was that if half an X flashed on screen, it would only be for an instant anyway, so it would just be part of the "transition".

So maybe I'm overestimating the frequency of the reads?

[Carnildo]

Hmm...I did realize when I asked that the buffer might change as it was being read, but I was thinking it wouldn't be a problem. I think my implicit reasoning was that if half an X flashed on screen, it would only be for an instant anyway, so it would just be part of the "transition".

So maybe I'm overestimating the frequency of the reads?

With a typical monitor, the graphics system reads the framebuffer 60 times per second (once per screen refresh). For VGA and other analog connections, the "read" process is active for about 75% of each timeslice. For DVI and other digital connections, I don't know how much of each 1/60th of a second is spent transmitting data (for a low resolution, does the data get sent in a single burst at the beginning of the timeslice, or does it get sent spaced out over the entire thing?).

The main problem that double-buffering with vsync is designed to solve is "screen tearing": if you've got full-screen changes (eg. a movie or a video game) and your drawing is out of phase with the monitor's update cycle, it looks like the image is "torn", with the part of the monitor above the discontinuity displaying one frame, while the part below is displaying a different frame.

[EvanED]

You can see an illustration of tearing at Wikipedia (there's a whole article on it). In an FPS-style game, you can often see it pretty clearly if you turn off vsync and look around rapidly.

[LikwidCirkel]

Yes, Xanthir and Carnildo have got it.

The hardware will have a dedicated DMA engine and related logic which transfers data in memory buffers to signals on the wire. VGA/DVI/HDMI/DisplayPort/Whatever interfaces ALL have similar timings where some 70-90% of the time is "active region" data where data is being read out of memory and used to drive the wire. The rest of the time is "blanking", where the signals are ignored by the monitor and don't have associated video data in memory. This historically comes from CRT monitors where the electron gun needed time to move to the other end of the screen, but it is preserved at the interface level to keep everything similar going forward.

All this really means is that for a 60FPS display, every 17 ms, the video hardware/DMAs will be ready for another buffer. Some 14ms of that, the DMA will be reading out of memory, and the remaining 3ms is when the user should swap buffers for smooth video rendering. At the driver-level, this all breaks down to writing registers which start the DMA transfer (requires a memory address and size), and then waiting on an interrupt which signals completion of the transfer. For optimal niceness, the interrupt should be handled within a few milliseconds by swapping buffers if necessary and triggering the next DMA transfer.

The DMA engine is pretty standard on every platform that I've ever heard of. Otherwise, the user would have to write an I/O word for every single pixel, and obviously that's problematic if the wire is transferring data 70% of the time.

Of course, there are all kinds of extras features/functionality which have been added over time. Monitors have a ROM with E-DID data which the OS can read to determine the capabilities of the monitor. This is related, but mostly separate from actually outputting video.

I won't get into GPUs, because they're much more like math-coprocessors than integral parts of the basic video output system.

[troyp]

Thanks for the replies, guys. I have to admit, I'm still a bit confused about this stuff, like the screen tearing. I mean the two images that are combined in that Wikipedia picture aren't from successive "frames", are they? They seem way too different for that. So how does the discrepancy become so large?

The funny thing is, the rare times I've seen this (I don't really play computer games), it always struck me as a pretty simple phenomonon. I figured it was just frames being sent from X to Y where the frequency of X and Y was mismatched for a while, causing them to get out of phase and display stationary tearing. If they remain mismatched, it would cause the tearing to cycle vertically up or down (depending on whether X or Y was faster). The short description in Wikipedia seems to match that pretty well, so maybe I was right, but it doesn't *seem* right to me anymore (or at least not that simple). Now that I'm actually thinking about it and trying to pin down exactly what "X" and "Y" are and how it works, I find I can't and just become increasingly confused the more I think about it :-S.

Surprisingly, it's hard to find a good "how tearing works" article on Google (although I haven't made a really concerted effort yet). The Wikipedia article has a bit of information on preventative techniques and whatnot, but the explanation of tearing itself is only a brief sentence which doesn't really tell me much.

[Sizik]

Thanks for the replies, guys. I have to admit, I'm still a bit confused about this stuff, like the screen tearing. I mean the two images that are combined in that Wikipedia picture aren't from successive "frames", are they? They seem way too different for that. So how does the discrepancy become so large?

Fast motion.

[EvanED]

Yeah. The Wikipedia image is simulated, but it's pretty easy to wind up with tearing that's actually far more pronounced if you spin around fast and stuff.

[troyp]

Oh, well that makes a bit more sense. Is the fact that the shifted portion in the image occurs in the middle of the image (rather than the top or bottom) also just a liberty they've taken? That seemed odd as well (unless the "scanning" starts in the middle of the screen for some reason).

[EvanED]

Oh, well that makes a bit more sense. Is the fact that the shifted portion in the image occurs in the middle of the image (rather than the top or bottom) also just a liberty they've taken? That seemed odd as well (unless the "scanning" starts in the middle of the screen for some reason).

No, and no.

The point is that the top third of the image has been sent to the monitor, but then the next frame replaces the contents of the video buffer, so as the card is sending the middle third of the image, it gets the next frame. And then the frame after that replaces the contents of the buffer, so the bottom third of the screen "sees" that frame. (Note that there are two tear points in that image.)

[troyp]

The fact that there are two tear points was what I meant, but I thought the top and bottom were from the same frame, which why I was talking about the middle being replaced. Clearly I'm crazy, though, because I can see now the two shifts are in the same direction. Okay, so the Wikipedia image isn't confusing me anymore. Hooray! (still not feeling like a graphics expert though ;-) )